Bombs (Cherry Bomb, Bomb, Mega Bomb) will clear many Rocks at once and are the fastest way to find Ladders.
If you don't have access to Bombs yet or they're too expensive, killing Monsters also gives a chance to find Ladders.
You can also craft Ladders from 99 Stone each which saves loads of time, but that's also expensive.
You can buy Ladders from the Desert Trader for 5 Quartz each, and many people (like me!) set up Crystalariums to duplicate Quartz. You get 5 Quartz every 2-3 days so not too bad.