You can go to the desert only when:
1) Pam is present at the bus.
2) Strictly at a time frame, between 10:00 am and 17:00 pm.
3) She has left a sign behind, reading; "you can use the bus yourself" (something along those lines).
4) You have built the Desert Obelisk, from the wizard tower.
5) There is no festival day, during which most NPCs aren't accessible, or cannot be found in their normal routes... Exception to that is obviously the desert festival!
P.S. It might be obvious, but you don't board the bus or talk to Pam directly; you must buy a ticket from the machine, southwest of pam.