The gem birds can repeat, so the next time you see one, take note of which area you are in (N, S, E, W). I'm still seeing the birds every now and then when it's raining, even after completing the puzzle. If you need more rain, take rain totems to the island.
It doesn't tell you if you got the gems wrong at all, so you'll only know when all four are correct. I waited until I had 3 out of 4 and then checked the remaining 2 gems.
If you only want to unlock the level 10 shortcut, you can double up speed boost with triple espresso and any speed-boosting food. Bring a stack of something like gold cheese and hold it in your hands. Eat some any time your health drops below 1/3, and only switch to pick or watering can if your way is completely blocked. Then just run like mad. But I don't have any advice for actually fighting off the enemies so you can collect actual loot, sorry.