I know this is old, but in case anyone else stumbles across this. I also use pipewire (with both pipewire-alsa and pipewire-pulse) and had the same experience. I use SMAPI as well. I found the issue was caused by a mod, LowHealthWarning. Removing this mod fixed the issue for me. I am guessing it has some audio-specific logic which is not as robustly coded as the main game, or is specifically not compatible with Linux at all. I had switched from playing on Windows recently so I never thought that some mods may not work properly on Linux.
If others are running into audio issues on Linux and also using mods, I recommend first trying to remove all mods to see if it solves the issue (just move the mod folders temporarily out of the folder). You can then add them back if audio is fixed until it breaks, so you know which mod broke audio.
