After getting the Ice Fruit in Polyandrium (the area after defeating the Plant and Skeleton boss), you are able to access the Ogre Fortress. The Ogre Fortress is at the top of the map, the provided image shows how it is accessed. The areas in Polyandrium after getting the Ice Fruit are optional, though it is possible to find a map there that hints at this route into the Ogre Fortress.