How do I get to the third floor of Colourous cave?

I feel like an idiot. I've wandered all through the 2nd floor a dozen times, and I can't figure out how to get to floor three. I have the key, but I cant find a door/ hole in the wall/ tunnel/ whatever that gives me the option to use it. Help?