Are you maximized in that top screenshot? You can try to hit Alt-Enter to switch between Fullscreen and Windowed mode and maybe if you get lucky the screen resolution will toggle back to something you can actually use? Try that in the initial Game menu too, it could help.
A workaround which sounds like it would work for you is to install SMAPI and then run the following command from the in-game chat interface:
debug us 60
which will set your UI scale to 60% (lower than ordinarily available). Set the number to your target and then use the regular UI (if you can access it now!) to adjust it back to your preference.