I had the exact same thing. Got stuck after a failure. Did the quest again but still had the question mark over my head at the shrine. Here is what worked for me! Just did it tonight and it fixed it!
I am the hosting player. I accepted the quest from Qi in co-op mode. Then I left co-op and loaded the farm single-player (NOT in co-op mode). I blew through the levels with staircases to complete the quest. Leaving the mines, I had access to the button finally! I didn't mess with it and went to sleep. I also closed the game completely here just to ensure, but I don't think I needed to do that. (You can rejoin co-op mode here.) Then the next day I went back to the cavern and made it dangerous mode (because it hadn't swapped back yet by itself). Again, slept and came back the next day to turn off dangerous mode. Slept, and voila! The mines were back to normal. Let me know if you try it or have any questions! It fixed mine perfectly. Feels amazing to have the normal mine back!
