Scientists Use AI to Turn 134-Year-Old Photo Into 3D Model of Lost Temple Relief

A team of computer scientists recently made 3D reconstructions of lost relief panels at a UNESCO World Heritage Site using artificial intelligence.

The researchers developed a neural network that can take a single-2D photo of a three-dimensional object and produce a digital reconstruction in three dimensions. In effect, they developed a stereoscope for the 21st century. The team presented its proof-of-concept at the 32nd rendition of the ACM Multimedia conference last month.

For the purposes of their research, the scientists used images of reliefs in Indonesia’s Borobudur temple, a UNESCO World Heritage Site. The temple is covered in 2,672 bas reliefs, making it the largest collection of Buddhist reliefs in the world. In the late 19th century, the temple’s foot encasement was reinstalled, concealing 156 of the reliefs behind stone walls, and they remain buried today. But before they were buried, grayscale photographs were taken of each panel. The recent team’s neural network managed to reconstruct one of those now-hidden reliefs using an old black-and-white photo from 134 years ago.

Image: Pan et al. 2024

Previous attempts had been made, but these earlier reconstructions couldn’t replicate the finer details of the reliefs. Those details were lost because of the compression of depth values; in other words, these three-dimensional reliefs have detail from the carvings closest to the viewer and farthest from the viewer, and previous reconstruction attempts flattened out the details at these varying depths. The team  referred to the lost characteristics as “soft edges,” and developed a map of those edges based on the calculated curvature changes in the 3D space.

In the new paper, the team posited that the edge map as it existed was reducing the accuracy of the model, it failed to convey the changes in 3D curvature properly, and the way it was incorporated into the network limited its impact on estimating depth in the physical objects.

A soft-edge map (left) and semantic map (right) of the 2d relief image. Pan et al. 2024

“Although we achieved 95% reconstruction accuracy, finer details such as human faces and decorations were still missing,” said Satoshi Tanaka, a researcher at Ritsumeikan University in Japan and co-author of the study, in a university release. “This was due to the high compression of depth values in 2D relief images, making it difficult to extract depth variations along edges. Our new method tackles this by enhancing depth estimation, particularly along soft edges, using a novel edge-detection approach.”

The images above represent the team’s best experimental results (bottom row) for a soft-edge map (left) and a semantic map (right) of the sample relief, compared to the ground truth data (top row). The edge map is just that—it tracks the points where curves in the relief give it depth, which confused earlier models.

The semantic map—which is vaguely reminiscent of Ellsworth Kelly’s Blue Green Red—shows how the model’s knowledge base associates related concepts. In this image, the model distinguishes foreground features (blue), human figures (red), and background. The researchers also included how their model compared with other state-of-the-art models in relation to the ground truth images.

AI gets its share of flak, but in the sciences it is proving remarkably adept at solving issues in image recognition and cultural heritage preservation. In September, a different team used a neural network to identify previously unseen details in panels painted by Raphael, and a different team used a convolutional neural network to nearly double the number of known Nazca lines—famous geoglyphs in Peru.

The model is capable of multi-modal understanding, meaning it is able to intake multiple channels of data to make sense of its target object. In this case, the soft-edge detector used to measure curves in the relief doesn’t only see slight changes in brightness to perceive depth, but the curves in the carvings themselves. Using both channels of information allowed the new model to recreate a sharper, more detailed reconstruction of the relief than previous attempts.

“Our technology holds vast potential for preserving and sharing cultural heritage,” Tanaka said. “It opens new opportunities not only for archeologists but also for immersive virtual experiences through VR and metaverse technologies, preserving global heritage for future generations.”

Cultural heritage needs to be preserved. But some cultural heritage is particularly at risk, and while these AI-generated reconstructions can’t replace the real McCoy, they have their uses. Neural networks like the one described the recent paper could resurrect lost heritage that only exists in images—for example, the Bamiyan Buddhas, monumental statues blown up by the Taliban in 2001—if only in an augmented or virtual reality environment.

The models could also be used to preserve cultural heritage on the brink of destruction, like the centuries-old aboriginal carvings on boab trees in Australia’s Tanami Desert.

Cultural heritage defines who we are by way of the communities and cultures that came before us. If these AI models help art historians and preservationists save just one piece of history, they’ve done good. Of course, AI models also require a huge amount of energy, which can contribute to the loss of cultural heritage in tangential ways. But even if the ways AI is powered remain problematic, using the technology for good causes is to be on the right side of history—especially when it comes to artifacts.