I wasted several games before getting the Wooden Key, but seriously, it's really worth it. The Glacial Key isn't very hard to get, if that is a consolation.
If you find it too hard, then I guess you can wait until you complete all the BoS quests. I did it without the BoS stuff, so I bet you can with it.

Also, the Rigorous Key is really easy to get... Once you know where to land on the ice.