Khoury inclusive AI team places third in Amazon’s Alexa Prize TaskBot Challenge

Khoury College professor Malihe Alikhani and her doctoral student Anthony Sicilia led a team that recently finished third in the Alexa Prize TaskBot Challenge 2. The international Amazon-run competition aims to build AI assistants called “TaskBots” that help users through detailed tasks like cooking or do-it-yourself work, with winners determined by user ratings, research contributions, and a judged final event.

After nine months of building a TaskBot that interacted with real Alexa users, the team not only achieved the third-place prize, but did so by centering the Khoury ethos of “computer science for everyone.” Their proposal was for an inclusive and equitable Alexa system, which they called ISABEL (IncluSive And collaBorativE aLexa).

“We promised that we would make these devices accessible and usable by communities that historically haven’t been able to use them as well,” Alikhani says, “such as people with speech impediments, African American Vernacular English speakers, and the American Sign Language community.”

Alikhani’s lab, which migrated from the University of Pittsburgh to Khoury College last year, designs inclusive and equitable language technologies for health, education, and social justice uses, and seeks to understand and mitigate biases in language models. Sicilia is an integral part of that lab, which he joined after the first year of his doctoral program. He’d studied math as an undergraduate because of an early interest in chaos theory, but while building an app for a professor, he saw the wide benefits computer science can create — benefits he saw as vital for ISABEL.

Malihe Alikhani

“Our goal set us apart,” he says. “We were trying to reach more Amazon customers who have different preferences or capabilities without excluding anyone in the way the bot was designed. All of our choices were built from that principle.”

ISABEL’s context-aware, multi-modal communication relies on state-of-the-art machine learning techniques to enable flexible, human-like collaboration. Users with different preferences and capabilities can tailor their experience, making the TaskBot more accessible for diverse groups. To achieve this, the team needed to rectify the sorts of biases that limited previous TaskBots; this, Sicilia notes, required a “mathematical explanation of why AI fails when working with underrepresented communities.” So Team ISABEL based their design on their recent mathematical analyses of bias prevention and collaboration in large AI language models.

As the team lead, there were many firsts for Sicilia. He’d captained lab projects before, but this was a larger effort, and one intended for consumer use.

“If your code breaks in a lab setting, you lose a few hours. It’s lower stakes,” he explains. “Moving to this setting, where if your bot does something it’s not supposed to, it would impact a real person … The threshold for quality is a lot higher.” He adds that it was his first time doing “multithreaded processing” across diverse projects, referencing a form of computer architecture that involves parallel computing on one central processing unit core.

Anthony Sicilia

Alikhani expresses pride in Sicilia’s work over the last few years on difficult applied and theoretical problems, lauding him as a bright, driven, and high-potential researcher. But she’s also quick to mention the eight-student team Sicilia led, a team which also included Kate Atwell and Mert Inan. That group coordinated with Amazon scientists, the Alexa team, and community partners to bring their vision to fruition.

“The spirit was there, and they were always working hard,” Alikhani says. “We often design and evaluate systems but don’t see them deployed in the wild. It was rewarding for them to see a tangible result, and it changed their perspectives on the impact they can have.”

It’s that impact, Sicilia and Alikhani say, that fueled the effort they put into ISABEL.

“We have these technologies advancing extremely fast. I know friends who aren’t in CS who are using ChatGPT and improving their daily lives,” Sicilia says. “It’s one of those things where if we don’t pay attention to marginalized communities, we’re leaving them behind.”

“I care about how I take this technology and reach out to communities that may not have access to it,” Alikhani adds. “I want natural language processing technology to be for everyone, and I’m glad to be part of a college that can support this vision for my lab and my research. This is a very exciting time for conversational AI and natural language processing, and our lab will focus on the ethics, accessibility, and inclusivity of this technology as we move forward.”