I would actually suggest that you try some different Qi quests to get gems. The prismatic shard quest is extremely difficult if you don't routinely do deep skull cavern dives, just because that is the fastest way to obtain a lot of them. I usually rely on random loot drops, omni geodes, or mystic stones in the quarry, so I don't give the ones I get to Mr. Qi.
Instead, I collect lots of cheap items that can be used for the prismatic grange quest.
Red - salmon berries, hot peppers, apples
Orange - copper ore, apricots, earth crystals
Yellow - sap, wheat
Green - fiber, tea leaves
Blue - blueberries, broken CDs, broken glasses, blackberries
Purple - refined quartz, grapes, amethyst, sweet pea, bug meat
Some of these are literal trash you just have to collect as you play; some you can grow; some you can pop into a dedicated crystallarium. It can take some time to collect enough, but all you need to do is remember to put junk in a chest. Usually, by the time I get the quest, I can immediately fill it, sometimes twice over.
Another one is giving 50 loved items in one week. This is extremely easy if you have a stash of rabbit feet (and emeralds for Penny). For anyone who says that rabbit feet are hard to get, I say, you clearly need to pet more rabbits.
You can actually cheese the dangerous mines or dangerous skull cavern quests with a stack of 100 staircases and some food to eat if you get too injured.