Khoury College students win award for holographic project at MIT Hackathon

Khoury College of Computer Sciences students Ezekiel D’Ascoli, Madeline Hulse, and Michael Steinberg, along with New York City-based graphic designer Elizabeth Sheffield, won the Best Use of Looking Glass award at the MIT Reality Hack this past spring for Terrariam, their unique, holographic experience of a user’s emotional state.

Left to right: Ezekiel D’Ascoli, Michael Steinberg, Madeline Hulse, and Elizabeth Sheffield.

Reality Hack, an annual community-run extended reality (XR) hackathon, brings together thought leaders, brand mentors, creators, participants, students, and technology lovers from diverse backgrounds and skill levels. For five days, these participants attend tech workshops, talks, discussions, collaborations, hacking demonstrations, and more. This year, more than 80 teams — roughly 260 people — participated.

“The hackathon was broken into two parts: the first was a conference for all things XR, and the second was the main hacking event,” D’Ascoli said. Teams were given 72 hours to devise and finish a project, with several elimination rounds. “By the final round, there were only fifteen teams left. Each time we made it to the next tier in the judging, we were screaming.”

Although D’Ascoli, Hulse, and Steinberg were in Caglar Yildirim’s Mixed Reality class together, the three Khoury College students did not initially plan to collaborate at Reality Hack.

“It came from a casual conversation during the hackathon,” Steinberg said. Hulse stressed that they all went into the event intending to find a project they really wanted to work on, regardless of who their collaborators might be. Sheffield, who they met there, rounded out the team.

Looking Glass Factory, one of the event’s sponsors, is a Brooklyn-based startup specializing in software and tools that allow users to render holograms from 3D content in programs Unity, Blender or Unreal. Hulse likened the company’s holograms to the moving newspaper pictures in Harry Potter.

“Each of us were blown away by their work — particularly their Portrait’s stunning holographic visuals — when we first saw them, and we adamantly wanted to work with them,” Hulse said. “Unlike other VR/AR hardware, you didn’t have to put anything on to experience the 3D visuals, and they could be viewed at the same time with other people.”

One of the team’s original ideas was a visualizer that displayed the current time and temperature outside. It was D’Ascoli who came up with the alternative idea of creating a project dealing with emotions. Using the stressful impact of the global pandemic on mental health as their motivation, the group created Terrariam, a holographic experience that visualizes the user’s emotional state. The goal: to encourage personal growth, acceptance, and attachment to community.

“We used the Python Deepface library, which takes in data from your computer’s webcam, analyzes it, and determines what emotion you’re experiencing based on your current facial expression,” Hulse said.

The team had to “translate real-time data from a computer vision into different emotions to read facial expressions,” in order to turn that data into a three-dimensional landscape, Steinberg said.

“If you’re sad,” Hulse added, “the tree in the landscape turns blue and the Ready Player Me avatar, which looks like the user, does a corresponding animation. The tree will bloom according to the intensity of the emotion.”

Of the six teams using Looking Glass Portrait or Looking Glass 4K Gen2 to build their projects, Terrariam was chosen as grand prize finalists, winning top ten in the competition and scoring first place in the Best Use of Looking Glass category.

“Looking Glass [Factory] liked how the team used the glass not just as an image, but more as a mirror, reflecting something personally insightful,” D’Ascoli said.

Terrariam takes its name from the word “terrarium,” a glass container for plants; the name is a play on words.

“We changed the ending to read “I am” to indicate how the product caters to and visualizes the individual,” Hulse said.

All three students credit Yildirim’s class with helping them with their project at Reality Hack.

“His class taught us how to use Unity, a 3D game engine used to develop a wide range of applications such as VR and AR,” Hulse said. “We also learned about design guidelines for usability in XR, which helped with coming up with the design of the application.” Steinberg, who graduated from Khoury College in May and now works as a full-time software engineer at Meta, noted that the class taught him how to complete projects quickly, something that was essential at Hackathon.

D’Ascoli elaborated on the benefits of Yildirim’s class, which is set up to give students theory and product experience. “The event and that class shaped what I want to do in my career, which is to be a graphic engineer in virtual reality,” he said.

Will they come back to MIT’s Hackathon next year? “Absolutely!” D’Ascoli and Hulse said. Steinberg would welcome the opportunity to come back as a mentor.

“Meta wasn’t there this year,” he said, “so it would be great to come back and represent it.”

Subscribe to the Khoury College newsletter