I don't know about the formula and did not ever see a formula about it. Maybe
@FilthyGorilla or
@BlaDe can help you. However, if the objective is to get a prismatic shard without diving into the Skull Caverns, my advice is to clear the Quarry every day until a mystic stone spawn there.
Then save the day by going to bed!
The next morning, head to the quarry and open the mystic stone. If there is a prismatic shard in it: lucky you. If there isn't: restart the day and go through it without opening the mystic stone. Try again next morning and rinse and repeat until you get a prismatic shard from the mystic stone. This works because the drops of a mystic stone in the Quarry are rolled every morning, not at spawn.
This is how I got all my pre-SC prismatic shards.
Also, killing monsters in the Mines once floor 120 is reached has a drop rate of 0.05% for a prismatic shard (this is 1 prismatic shard for 2000 monsters killed).