East of the Heraion and Pelopion stood the great altar of Zeus. No trace of it has survived, but the large quantities of ash and bronze votives discovered inside the Pelopion may come from this altar. According to myth, Zeus himself indicated the building spot of his altar by striking the ground with a thunderbolt. The altar was destroyed under Theodosius I, who abolished the Olympic Games, and under his grandson, Theodosius II.

Pausanias, who saw the impressive altar in the second century AD, describes it in detail (V, 13, 8-10). It was a circular or elliptical structure, approximately six and a half metres high, and consisted of a platform and the altar proper. The platform was three metres high with steps on the sides. On the platform stood the conical altar, nine and a half metres in diameter, made of the ashes of the sacrificed animals. A narrow staircase carved into the ash led to the top. Blood sacrifices in honour of Zeus were performed daily. The sacrifice took place on the platform and the legs of the sacrificial animal were then taken by the priests to the top of the altar to be burnt. Women were allowed on the platform but not on the altar where only priests and men had access. On the ninth day of the month Elaphion (late March) the sanctuary soothsayers would bring ash from the altar of Hestia inside the Prytaneion, mix it with water from the river Alpheios, and use it to coat the altar of Zeus. According to Pausanias only water from Alpheios, the god's favoured river, would do. Also only poplar was used for the sacrificial fire, since this was the wood chosen by Hercules when he sacrificed to Zeus.