Stardew Valley clearly lives in a Pocket Dimension, like in Brooks' Landover or Anthony's Xanth novels. Some people can navigate the Border but there's necessarily a change in the rules once you enter SDV. Animals and other things only exist there which are not willing or able to exist in the Outside World. One of the rules is entry by invitation only, family preferred, so that the current residents can maintain their quality of life. Another is that you start with a clean slate, so the passage by a new prospective candidate is only allowed on Spring 0, a date only accessible by Lewis and Robin when processing the new candidate's invitation and credentials.
SDV opens up the Border on special occasions for Festivals, and then only a few people make it though. It seems they slowly forget about SDV until next year, so it's existence isn't acknowledged as more than a legend. A Fairy Tale. The Governor and his missus have no chance of ever landing a home here, The Forgetting is too strong for them. Lewis and Pierre will never get those tax breaks.
Some life forms have found a way to cross by purity of spirit or by simply being refugees in need of a safe harbor (Moonlight Jellies, Krobus, Dwarf). SDV residents of a particularly strong will can cross when necessary but not many choose to do so (Sebastian, Sam), and they always come back. The Outside World is nice to visit, but you wouldn't want to live there.
However on rare occasions there seem to be lapses in The Forgetting and outside interests are able to set up shop in town (Joja Mart). The Valley slowly reforms itself to close off those cancerous parts towards the edges of town and if necessary, entice a newcomer with the requisite pastoral attitude and annoyance with the Outside World to hasten this process. It's not always successful, but most of the time Jodi can still scrape up the funds at Pierre's to make boiled Beets for Vincent.
The true threat is from the other Pocket Dimension, the Gotoro Empire. Even the Gotoros know that the Outside World is simply not worth the effort but they have identified Stardew Valley as an excellent target for conquest. Inter-dimensional battles such as these take a toll on the human mind, which is why Kent rarely speaks of it. And the Junimos merely muse on this foolishness as the Gotoros are utterly unaware of their nirvana.
Stardew isn't quite a one way door, but even those who don't seem like a perfect fit: the Wizard, Witch, Shane, Haley still have something to offer and to expand the cultural sphere of the Valley and never seem to leave, though they threaten to on occasion.
Not sure what was up with Grandpa, though. He was always rather a nut.
