This same issue happens to me as well. Pretty much every time we sit down to play 4p local split screen, we have a day that crashes out. It usually is closer to the end of the day, but not exclusively. We've tried to play around it, but it gets really frustrating after really productive days.
Device: Xbox Series X
Build: 1.6.15 build 1