Is this on a particular platform? I haven't run into this issue myself. Note I'm a PC player.
I play PC too, it's a problem I mostly run into if I take a ladder down and then immediately try to eat something. It may give you the prompt to exit the cavern, and if you're stressing out and trying to be quick, you often don't notice what the bubble says. Would be super helpful if it was a different colour.
To be fair, it doesn't happen all the time, and depending on what your tactic is or how you like to descend, you may never run into an issue with this. Especially if you tend to eat before taking a ladder down.
It's not the biggest deal to me, but I know my SO loves the mines as the main part of the game, and getting so far only to accidentally leave has really taken some of the joy out of playing the game for him, unfortunately.
It may also be because we play on a laptop with a touch pad, making a misclick easier. But I don't have any issues playing like that in any places, eeeeexcept for the skull cavern.
I've also seen players complain that using the return scepter in the caverns doesn't give you a prompt, so you don't accidentally teleport yourself out when trying to reach for a snack. Just many small fixes could be done to this area that would make massive differences imo.