Made an account to say I'm having similar issues. As of recently my game started to drop frames/stutter every so often, sometimes every couple seconds, sometimes every minute. It's not huge but I'm very sensitive to such things and I notice it every time it happens and it bothers me more than I would like to admit :(
I didn't have this problem before and there was no update on either Windows, Nvidea, Drivers etc which may have caused this, apart from the latest Stardew update. Tried a whole lot of things, including what is being mentioned here about settings the affinity and priority, repairing XNA, changing resolution from windowed to borderless to fullscreen.
What I also noticed, whenever I boot up Stardew, in borderless mode, he boots it in 3620 x 2036, even though my resolution is 2560 x 1440. I don't know if this has anything to do with it, seeing as I've never had this problem before with the lag/stuttering, but wanted to mention it anyway. Changing resolution is also not easy, as I first need to go to windowed mode, then to fullscreen which messes up my whole screen, then to some random resolution cause the 2560 x 1440 is then off the screen and I can't reach it, and then after that select the right resolution xD It's a bit of work haha.
I normally play with mods but even when I remove all the mods the problem is still there, also reinstalling the game did not fix it.
Also for me hardware is not the issue.
AMD Ryzen 9 5900X
32GB RAM
Gigabyte GeForce RTX 3080Ti