A difference of one week is a quarter million in my pocket, which far outweighs everything else you've mentioned, so don't knock it.
You'll get maybe one or two life elixirs in Spring of Year 1, which isn't relevant amounts. Especially not if you've got plenty of Salmonberries. Life Elixirs are still entirely and completely useless to attempt. Heck, you'll get more purple mushrooms out of finding a single random mushroom level while delving than you would in probably a year of waiting for one to pop up in your mushroom cave.
Let's do the math on your anecdotal evidence, shall we?
Purple Mushrooms are a 2.5% chance to spawn, which is 1 in 40. Multiply by six, now you have 6 in 40 chance, which is 15% that on any given harvest, at least one of those mushrooms will be purple. With 3-4 per week... you're lucky to get around a fifty-fifty odds of getting a purple mushroom per week. Comes out to roughly 52% and change, if you do the multiplications out for graduating probability that you will see at least one. Sure, you can get lucky, but odds are against you seeing one before Summer.
Meanwhile, let's look at the math on fruit bats by contrast
Assuming no fruit at all, there's a 48% chance of no fruit at all, so slightly better odds that you will get at least one fruit per day than mushrooms, although the odds of more than that go down. The odds that the fruit in question will be one of the artisan bundle tree fruits is 20%. Even assuming a 52% chance of getting a single fruit (it's weighted higher because you can get up to four fruits, but let's simplify things conservatively), you're looking at just over a 10% chance *per day* of getting your tree fruit. At which point the cave has done its job, and you no longer need to bother with it anymore unless you want to rake in profits from either Botanist or by processing the fruits, which are going to be lackluster at best, but still roughly comparable to mushroom cave at that point.
As far as the Skull Cave? About the only buffs I need come from Coffee and maybe Pepper Poppers. But then, there's not much point in bothering with the Skull Cave to begin with. I mean... why? What do you gain from the Skull Cave? Iridium? Fine, but outside of the six iridium sprinklers for your greenhouse, what else do you want, practically, with Iridium? Tool upgrades are pointless by then. You've got all the sprinklers down on your farm, so watering can and hoe upgrades are pointless. You're not going to be diving skull caves so the mining pick is pointless. There's no need for chopping in bulk anymore so the axe upgrade is pointless. There's nothing to spend it on. Unless you're specifically trying to get some of the new 1.4 content stuff from the new Skull Cave floor generation for museum completion or the other thing that item (or rather, its processed version) is used for, there's no point in bothering with Skull Caves all that much.
Life Elixirs are simply not reliable at the point in the game where they are remotely useful. You'll never have more than a handful of them, and you'll already have superior options by the time they come online. Meanwhile, it's virtually a given that you will get at least one tree fruit to help unlock the Greenhouse, which will be making you a million and a half per season once it is completed and filled, so the sooner you can get to that, the better off you are.