In my personal experience, if you have too many mods + complex mods it sometimes leads to lag. There was a point that I was running the game with a little over 300 mods and some of them were many instances of custom crops + Ridgeside + stardew expanded + a bunch of new npcs. I did downgrade a little and the stuttering went away. I currently have no expanded areas except for deepwoods and fewer custom crops. I don't know if there is another cause or if there is another solution, maybe it was just me being a little short-sighted and trying something that worked in other games (I'm a avid The Sims 2 player, so 30-50 minutes loading screens used to be common for me)
And if your game is installed on a traditional hard drive or SSD drive can also make a lot of difference.