Honestly, this question is a bit like "What's the best way to get tomato seeds, other than Pierre's or Jojamart?" You CAN do it, but... the Skull Cavern is great for finding Prismatic Shards. Even if you've only got a humble Lava Katana, cook or find some food with a halfway decent buff, get a big ol' stack of gold-quality cheese or large eggs for healing, wait for a good luck day, and even a casual trip down is way more likely to see results than anything else you can do.
If you really dislike combat (and the Skull Cavern pre-Galaxy Sword IS a little rough), one alternative is to make five or six crystalariums, set them all to Jade, trade them all for a pile stairs on a lucky Sunday. Use, like, half your stairs to go deep ASAP, and keep the rest to avoid any fights that might give you trouble.
If you're patient, then sure, set up a few Rainbow Trout Fish Ponds, check the quarry every day, and kill random monsters in the mines hoping for a lucky drop. But I'll bet a full set of prismatic clothing that you'll get much faster results with cautious stair-aided forays into the Skull Cavern than any other route. Especially if you spec mining for extra geodes and bust open the omni-geodes you get.