No, but there is a late-game item added in 1.5 that lets you
dodge works hours and go into any house at any time. (
wiki)
To be fair though, the Wizard is open from 6:00am to 11:00pm, so unless you want to do a midnight shuffle, you'll be good. He is a little out of the way, but I like to make it sort of a "forest excursion": I'll chop all the hardwood, pick up any foragables, and let my horse stretch its legs.
If you prefer ease of transit over any time of day, then Mountain Warp Totems or the
Obelisk (
wiki) might be better for you.
Hope this helps!