It all depends on how much iridium you want.
1. You can get enough for a couple of bars before you get to the Skull Caverns, mostly from magma geodes and fishing treasure chests.
2. If you take a chill (happy with whatever and wherever you get) approach to the Skull Cavern and always go with about 30 Salads or Cheese, 50+ Coal (kilns are helpful for this), a few emergency staircases, and whatever Spicy Eel and Triple Shot Espresso you can muster (5 would be nice), you can both
a) have a lot more fun there than you would at the bottom of The Mines and
b) gradually build up your stream of goodies including jade and iridium ore.
While you won't see much Iridium Ore until you are around 50 deep, you will build up jade, gold, rubies, (Spicy Eel), Diamonds (Espresso), and other fun goodies when on short visits.
I've had plenty of fun starting my Skull Cavern snowball even with the Obsidian Edge or Insect Head (evasion and avoidance are viable combat strategies) as soon as I repair the bus, maybe even before I can craft Mega Bombs or get the fanciest sword. The key to this is food and bombs (coal in hand plus mined iron ore).
Once you have enough bombs and staircases to get over 100 deep and you
a) develop a good intuition for which time-waster floors to skip,
b) take luck and time management more seriously
you can get enough iridium ore to sell for reasonably serious income around your first Winter if that's what you want to do.
The Skull Cavern is fun! So many great prizes!