The so-called south-east building, probably a shrine of the goddess Hestia, formed the south-east limit of the Altis enclosure together with the Echo-hall, which was built to its north in the fifth century BC. Raised in the first half of the fifth century BC, the south-east building continued to function until the first century BC, when it was demolished to make way for new buildings. When Pausanias visited Olympia in the second century AD, the shrine was no longer visible as it had been replaced by the House of Nero and other buildings.

The shrine was built in stages. Originally it comprised two corner rooms, a backing wall and an atrium, twenty-nine metres long and twenty-three metres wide. Very few traces of this first building phase survive. Two central rooms and a Doric colonnade surrounding the earlier building on three sides were added later, probably in the early fourth century BC after the earthquake of 373 BC. The colonnade, with eighteen columns along the front and eight columns at each side, was probably constructed as the building's main fa?ade towards the Altis. In its final form the building measured 36.42 by 14.66 metres. Fragments of the building's architectural members and terracotta gutter with palmette decoration have survived.

The building is not visible today because of the superimposed Roman structures.