The remains of the gymnasium are on the steep slope between the Castalian fountain and the temple of Athena Pronaea. This is one of the most complete examples of an ancient gymnasium complex, which included the gymnasium proper, a palaestra and baths.

The gymnasium dates to the fourth century BC, but was rebuilt in the Roman period, when the baths were added to it. Originally it was used exclusively for training athletes. Track and field were practiced inside the gymnasium proper, with events like wrestling, boxing and the pankration taking place inside the palaestra. However, in the Hellenistic period the gymnasium became a centre for intellectual development and housed cultural events, including lectures by orators, sophists, philosophers and poets.

The gymnasium is built on two levels. On the upper terrace is the xystos, or covered colonnade, seven metres wide and a hundred and seventy eight metres long, where the athletes practiced in bad weather, with a open air, six metres wide parallel track. In the fourth century BC, the xystos had a Doric limestone colonnade, the fa?ade of which was replaced by an Ionic marble one in Roman times. The xystos, which owes its name to the fact that its floor had to be regularly scraped and leveled (xystos=scraped), was recently cleared in its entire length. The lower terrace is occupied by the palaestra, a court twelve metres square surrounded by a colonnade and rooms on all four sides. Inscriptions in each of these rooms state their use: ball court, changing room, wrestling pit, and possibly a sanctuary of Hermes or Hercules. The court was used for wrestling or boxing practice. West of the palaestra is a circular (cold) pool, ten metres in diameter and 1.80 metres deep. A series of douche baths for the athletes, consisting of faucets which poured water from the Castalian spring into ten communicating clay basins is in the retaining wall at the back. The hot baths to the west are a Roman (AD 120) addition.

Several centuries later the gymnasium precinct was occupied by a Byzantine monastery; its main church, built on top of the palaestra, was demolished in 1898 by the excavators. During a visit to Delphi, Lord Byron inscribed his name on one of the Doric columns re-used in the monastery.