For such a large area, I'd like to see it have a bit more commitment than just being able to come and go as you please. Let's break my additions to your idea into a couple of sections for improving the depth of it.
For the sake of keeping with your 'man from the city' scenario, when you first initiate the event you get a big cutscene where you travel to the island by seaplane) with the Man from the City (if married your spouse and kids come with you, if single you go with whoever you've got the most hearts with. Or just Willy or Lewis). On the island is the Plantation Manor house, which is in disrepair, a tropical forest full of fruits (but not many trees for woodworking), a pier and a small village that supported the Manor. This village is filled with local traders and artisans that made up the Plantation's former staff and workers. There's also the larger farming area, but you can only plant tropical crops or Summer crops from Stardew Valley. And the cave itself, which would, in my mind, be similar to the Mountain Mines than the Skull Cave. I wouldn't include the option to grow the island.
When the even ends and whenever you purchase the island instead of by dingy or a smaller boat whenever you want, you have to help Willy repair an old fishing boat of his (maybe have it dry docked or just docked on the far side of the beach with the second pier) and you have to complete a another bundle (this time focused more on crafting goods, battery packs, etc) as well as spending gold in order to be able to go to the island on your own. This would be a significant investment, much like unlocking the bus was, on top of buying the island itself. Again, this is a major expansion to the game so gameplay wise it makes more sense to make unlocking the island a very, very large time/gold sink. Unless this was to be a paid expansion, then I could see it being you buy the island so that the content isn't blocked off by the bundle.
Time on the island.
For the sake of the island not being another Calico Desert distraction, once you arrive on the island you discover that a storm has destroyed the pier where you docked before. While Willy can let you off, he can't stay because there's no place to moor his fishing boat and he returns to Stardew Valley. You and your spouse (or just you) are left alone on the island with a largely empty Plantation Manor, no real supplies, and a village full of strangers. As Willy heads back to Stardew Valley, he promises to return at the end of the season and that he'll take care of the farm for you. Your immediate goal is to repair the dock so that you can get back to Stardew Valley when Willy returns.
While you do own the islanders, the islanders will treat you and your possible family with distrust. The shopkeeper will trade with you, but you'll have to go the usual route of earning the villager's trust not by spamming gifts, but completing quests. Lower heart levels would be more reliant of fetching items from the island, stepping up to cleaning out the island cave, etc. Eventually you'd be able to hire islanders to farm the Plantation, but to keep you from swimming in money from the higher yield rates of having a much larger farming space, you have to pay these workers as well as investing in repairs to the Manor and island village in general.
Once you repair the dock, you can contact Willy. Willy informs you that he's on his way, but that since he only knows about fishing, that he wasn't able to grow anything on the farm. Everything is still standing, your animals are still alive and the farm is relatively clear of debris, but it just isn't producing.
When Willy returns at the end of the season, you will have to leave the island for at least one full season (this makes the old overworld still relevant) and If you've cultivated your relationships on the island enough, you can assign a manager to take care of the Plantation for you. The downside is that while you're gone, you can't make direct money from the island. Instead, you have to use a import/export system between your farm and the plantation,, importing goods from the island every three days to sell, and exporting goods to the island to complete Manor repairs or other projects you left on the island.
