I believe this issue is caused by SDL2 within MonoGame, not exactly Stardew Valley itself. The bug only affects Mac OSX users. I have yet to hear Linux users report the issue.
The bug occurs when a hot-unplug (that is, the controller disconnects improperly) happens. At this point, SDL2 now has a pointer value pointing to an incorrect address in memory. The moment it tries to poll the controller, the moment it'll produce a SIGSEGV (Segementation Fault)
If TheTreePrince uses SMAPI with no mods, and has his wireless controller disconnect, upon reconnect they should see the following:
[Thumbnail version, click to enlarge]
Within the SDL2 Repo, there seems to be fixes to this:
http://hg.libsdl.org/SDL/log?rev=joystick+Mac
So the fix *maybe* as simple as recompiling SDL2 to a newer version. (Granted, this would need to be tested against MonoGame)