Awards, ethics, cross-college collaboration: Northeastern at CHI 2023
Wed 04.12.23 / Madelaine Millar, Milton Posner, and Matty Wasserman
Awards, ethics, cross-college collaboration: Northeastern at CHI 2023
Wed 04.12.23 / Madelaine Millar, Milton Posner, and Matty Wasserman
Wed 04.12.23 / Madelaine Millar, Milton Posner, and Matty Wasserman
Wed 04.12.23 / Madelaine Millar, Milton Posner, and Matty Wasserman
Awards, ethics, cross-college collaboration: Northeastern at CHI 2023
Wed 04.12.23 / Madelaine Millar, Milton Posner, and Matty Wasserman
Awards, ethics, cross-college collaboration: Northeastern at CHI 2023
Wed 04.12.23 / Madelaine Millar, Milton Posner, and Matty Wasserman
Wed 04.12.23 / Madelaine Millar, Milton Posner, and Matty Wasserman
Wed 04.12.23 / Madelaine Millar, Milton Posner, and Matty Wasserman
The most prestigious human–computer interaction conference in the world is taking place in Hamburg, Germany this month, and Khoury College’s researchers — along with their collaborators in the College of Engineering (CoE) and the College of Arts, Media and Design (CAMD) — are ready. Their work, which touches on everything from parrot socialization to deceptive dark patterns to social media misinformation, has made Northeastern the seventh-ranked institution in the world in terms of CHI publications, as well as sixth in the United States and first in Massachusetts. For a full schedule of their papers, panels, and other appearances at the conference, click here.
Click on one of the linked summaries below to learn about the corresponding paper, or simply read on.
Bold = Khoury College
() = another Northeastern college
(+) = interdisciplinary appointee
* = Khoury College courtesy appointee
Corsetto: A Kinesthetic Garment for Designing, Composing for, and Experiencing an Intersubjective Haptic Voice
or “How to feel music … literally”
Ozgun Kilic Afsar, Yoav Luft, Kelsey Cotton, Ekaterina R. Stepanova, Claudia Núñez-Pacheco, Rébecca Kleinberger (+CAMD), Fehmi Ben Abdesslem, Hiroshi Ishii, Kristina Höök
There is nothing quite like an immersive live music performance. When the composition meets the performance, and when the audience is fully engaged, the building feels like it’s lifting off its studs. The experience is not just visceral; it’s physical.
While the ambience of live music can naturally elicit this sensation, a team of researchers hailing from Sweden, the US, and Canada looked to deepen it. Their creation is Corsetto, a robotic upper-body garment which uses haptic gestures to transfer the physicality of a singer’s performance to the audience members wearing the device.
“This project explored the strange question of what it feels like to wear someone else’s voice,” said Rébecca Kleinberger, a professor jointly appointed between Khoury College and CAMD. “This collaboration was truly multidisciplinary, with elements of art, design, fabrication, software, HCI, experience design, and philosophy.”
The team created Corsetto with the aid of a classically trained opera singer who, through sessions of vocal exercises, helped the team to understand the biomechanical elements of singing. Once the team had incorporated those lessons into a functional device, they invited audiences to wear it during a performance of Morton Feldman’s piece “Three Voices,” then interviewed them afterwards. The result, they say, was an “intersubjective haptic voice,” a connective tissue between the auditory and physical elements of the performance that can help to convey the gravity and meaning of the music.
“[Corsetto] blurred boundaries between inner and outer aspects of participants’ bodies,” the researchers wrote, “at times rendering a fusion of all the bodies in the room into one holistic experience.”
“Who is the right homeless client?”: Values in Algorithmic Homelessness Service Provision and Machine Learning Research
or “Fair and ethical research practices for homeless populations”
Dilruba Showkat, Angela D. R. Smith, Wang Lingqing, Alexandra To (+CAMD)
As emphasis on ethical and equitable research practices has increased in recent years, so too have new innovations and diverse perspectives across the field of CS. But homeless populations have remained marginalized, and according to Khoury College doctoral student Dilruba Showkat, the field’s failure to produce ethical, balanced research of homeless communities isn’t due to a lack of intent or effort. Rather, it’s because homeless people are often difficult to access and don’t always cooperate with researchers, so their point of view often gets left out of the picture.
“Homeless populations’ values are well known in HCI research,” Showkat said. “However, during my analysis, I found a disconnect between these marginalized communities’ values and the machine learning systems that are developed to support them.”
Among the issues Showkat and her collaborators found were breaches of homeless peoples’ privacy and autonomy, as well as their lack of technological savvy and ability to self-advocate — each of which made them susceptible to being nullified or misrepresented by the predictors, categories, and algorithms used in machine learning technology.
“In several research papers, I’ve found social security numbers of homeless people, as well as other private information, unintentionally passed when sharing data sets,” Showkat said. “This is something really wrong with the system, and we should change that by conducting research that is held accountable. ”
The paper argues that researchers must actively protect the privacy and autonomy of homeless people, since they often cannot voice their concerns themselves. One proposed solution is to design machine learning tools using novel experimental techniques that don’t rely on vulnerable communities’ data, and instead prioritize human values. Another recommendation is to develop reproducible and replicable models by sharing a chunk of data anonymously, which would enable researchers to use the information without violating privacy.
Why, When, and From Whom: Considerations for Collecting and Reporting Race and Ethnicity Data in HCI
or “How should we ask about race?”
CHI “Honorable Mention” winner
Yiqun T. Chen, Angela D.R. Smith, Katharina Reinecke, Alexandra To (+CAMD)
Last year, Alexandra To and her fellow researchers reviewed six years of CHI proceedings and determined that just three percent of research papers on human–computer interaction collect race or ethnicity data. This year, they interviewed 15 authors to better understand how they’re collecting race and ethnicity data now, and to begin to develop best practices for collection in the future.
This research team doesn’t recommend collecting race and ethnicity data for every study, as privacy concerns, the risk of further entrenching “scientific racism”— the pseudoscientific belief that there is empirical evidence justifying racism — and participant retention could all be legitimate reasons not to. Instead, they encourage researchers to make such data collection a deliberate, rather than a default, decision.
If a study does collect racial data, the researchers recommend allowing participants to select multiple racial categories, including an open-ended way to self-identify, using more granular categories than the US Census default categories if appropriate, and considering where race is acting as a stand-in for another attribute (like socioeconomic status) and asking about that attribute directly instead.
Birds of a Feather Video-Flock Together: Design and Evaluation of an Agency-Based Parrot-to-Parrot Video-Calling System for Interspecies Ethical Enrichment
or “If you give a bird a Zoom call”
CHI “Honorable Mention” winner
Rébecca Kleinberger (+CAMD), Jennifer Cunha, Megha M. Vemuri, Ilyena Hirskyj-Douglas
Parrots have earned their reputation as the brainiacs of the aviary world, but such brainpower entails a tricky reality for the millions kept as pets. They need social and cognitive stimulation like their counterparts in the wild, but seldom get enough of it in captivity.
Seeking to rectify that imbalance, a new study first-authored by Khoury College–CAMD professor Rébecca Kleinberger aimed to provide the birds with the same social solution many humans jumped to in March 2020: video calling. In a three-month study involving 18 domesticated parrots, the team facilitated parrot-to-parrot calls using Facebook Messenger, all while monitoring the birds’ agency and motivation in initiating the calls and their engagement once the call began.
“Once the parrots learned to trigger the calls, it was hard to stop them, and some started exploring their tablets a bit too much,” Kleinberger recalled. “It was particularly challenging, but also truly rewarding, to involve so many animal participants for such a long period. To this day, I still sometimes receive calls from some of our bird participants.”
All 18 birds participated, most of them showing high motivation and intent while on the video call. Additionally, their human caretakers reported that not only did some of the birds learn new behaviors from their remote buddies — including flying and foraging — but the experience also helped the humans connect with their pets and learn more about them.
Through Their Eyes and In Their Shoes: Providing Group Awareness During Collaboration Across Virtual Reality and Desktop Platforms
or “How to work together when your work looks very different”
David Saffo, Andrea Batch, Cody Dunne, Niklas Elmqvist
To collaborate on a data exploration project, coworkers need to share an understanding of the parameters of their collaboration, which is difficult when one partner has a whole extra dimension to play in. These researchers used a tool they developed called VRxD to help a person on a desktop computer and a person using a VR headset to explore a data set about baseball pitchers; the researchers studied how they worked together and developed design recommendations for cross-platform collaborative tools.
“One of the potential benefits of AR and VR devices is the unique ways they can enable multi-user experiences and collaboration. However, I do not like that these experiences require all users to also use AR and VR devices,” lead researcher and doctoral student David Saffo said. “What excited me was thinking of all the design possibilities of a cross-platform system.”
By varying what features were available to the study’s six pairs of participants, the researchers determined that certain features — like the ability to quickly toggle into a collaborator’s view or leave the collaborator’s view open in another window — influenced how much time the pairs spent discussing their work versus working on the task itself, and who tended to lead a given activity.
Based on their findings, the researchers developed four design recommendations, which they call “through their eyes and in their shoes,” emphasizing how important it is for systems to allow collaborators to understand one another’s perspectives. Per the recommendations, VR–desktop collaboration systems should:
1. build in shared points of reference
2. encourage leadership by the user whose interface is best suited to the task
3. incorporate information cues (such as causing data to light up in both views when either participant clicks on it)
4. build in multiple ways to share perspectives so collaborators can quickly check on each other, present to one another, or sustain longer periods of collaboration
Auditory Seasoning Filters: Altering Food Perception via Augmented Sonic Feedback of Chewing Sounds
or “What can Pringles and sound effects teach us about our eating impulses?”
Rébecca Kleinberger (+CAMD), Akito Oshiro van Troyer, Qian Janice Wang
Aside from their snackability, Pringle chips are uniquely useful experimental stimuli for food-tech research, as their uniform size, texture, and flavor allows researchers to create consistent food samples and investigate how environmental conditions affect the eating experience. In a new study co-led by Rébecca Kleinberger, the team gave Pringles to study subjects while modulating the sound of their chewing using precise volume tweaks, delays, reverbs, and frequency filters. Then they determined how that modulation changed the subjects’ perceptions of both the food and their own hunger.
Among a slew of findings, the team discovered that amplified chewing sounds made participants feel greater sensations of satiety, texture, and flavor. Delay slowed eating speed. Large-room reverb diminished sourness and cravings. Filtering of high and low frequencies each bolstered fullness.
“This technology is quite fun and it helps you understand what’s going on in your brain, but is there a way to apply it and make an impact, especially regarding foods that aren’t especially healthy for you?” Kleinberger mused. “Can we make you eat slower? Can we make you chew more? Can we make you perceive more flavor and saltiness even if there’s actually less, so you’d eat food with less salt and artificial flavor?”
To that end, the team believes that future efforts can package these audio filters into an easy-to-use mobile app that helps users to regulate their food perception and cravings in real time — all without the need for the external hardware, explicit self-regulation, or specially prepared foods involved in previous approaches. But apart from the intricacies of the sound effects and neurology at play, Kleinberger believes there’s an underlying question of perspective.
“All of our participants told us they’d never had an experience like this, that it was really fun and transformative,” Kleinberger said. “Even just the notion of gaining awareness of your chewing, of becoming more attuned to it and having this stronger experience of sound — is interesting in itself. It can make people eat more mindfully.”
Model Sketching: Centering Concepts in Early-Stage Machine Learning Model Design
or “Let’s give machine learning designers a sketchbook”
Michelle S. Lam, Zixian Ma, Anne Li, Izequiel Freitas, Dakuo Wang (+CAMD), James A. Landay, Michael S. Bernstein
When designing a machine learning (ML) model, it’s easy to get stuck on the details of how to execute a solution at the expense of thinking about what kinds of solutions to pursue in the first place. These researchers want to get ML designers focused on the big picture, so they created ModelSketchBook: a Python package for quickly “sketching” minimal models and testing different design choices.
ModelSketchBook uses zero-shot learning and pretrained models like GPT-3 to let users create concepts, link them logically, and see how a machine learning model interprets those concepts on real data. For example, someone trying to build a model to identify hateful memes might use ModelSketchBook to explore how ML interprets the interactions of concepts like racism, nudity, and violence, then iterate several sketches to refine their initial model.
The researchers had 17 ML practitioners sketch models that would determine whether memes from Meta AI’s Hateful Memes Challenge should be removed from social media. They found that ModelSketchBook helped the practitioners discover gaps in their models, come up with a greater diversity of concepts to include, and produce quality outcomes more quickly. Their data also suggests that iterating on sketches — which the method encourages — produces better-performing models.
Thought Bubbles: A Proxy into Players’ Mental Model Development
or “How can we think about how we think?”
Omid Mohaddesi (CoE), Noah Chicoine (CoE), Min Gong (CoE), Ozlem Ergun (CoE), Jacqueline Griffin (CoE), David Kaeli* (CoE), Stacy Marsella (+CoS), Casper Harteveld* (CAMD)
In a work honored at CHI last year, this research team designed a game. Players role-played as wholesalers attempting to source medication from factories and supply hospitals during a supply chain disruption. Some hoarded medication from the beginning, others reacted to the shortage with panic-buying, and still others followed their computer’s recommendations through the whole game.
“(Last year), we identified Hoarders, Reactors, and Followers using observable behavior, and now we attempted to investigate differences in the development of these groups’ mental models,” said lead author and doctoral student Omid Mohaddesi, referring to the representation of a real-world situation that someone holds in their head. “We can see different patterns in the mental model development of groups of people and how information sharing impacts their mental model development … all this insight can potentially help us make better decision-support tools.”
To get contextualized insights into how their subjects’ mental models developed over time, the research team added a recurring, open-ended reflection prompt called a “thought bubble” to the supply chain game. Through two related studies, they coded participants’ reflections based on whether they were perceiving elements of the environment (“the backlog has gone up”), comprehending the meaning of those elements (“this is bad for our business”), or projecting a future state (“we will have to buy more medicine”). Researchers could then spot patterns between the ways people modeled the world and the decisions they made when managing a supply chain.
The study found that distinctions in participants’ supply chain management decisions are the result of differences in the development of their mental models. For instance, hoarders are proactive because they struggle to make sense of the situation — leading to uncertainty about the future — while “followers” trust order suggestions because they interpret the situation positively. The study also presents thought bubbles as a method for further studying mental models.
Rapid Convergence: The Outcomes of Making PPE during a Healthcare Crisis
or “PPE innovation soared during the pandemic, but how did it hold up?”
Kelly Mack, Megan Hofmann, Udaya Lakshmi, Jerry Cao, Nayha Auradkar, Rosa I. Arriaga, Scott E. Hudson, Jennifer Mankoff
In 2018, when Megan Hofmann and her partners began researching “medical making” — the application of 3D-printing models to the medical device and healthcare fields — she estimates there were only a few thousand medical makers worldwide.
But with the onset of the COVID-19 pandemic in 2020, the once-niche field was upended overnight. Suddenly, there was an urgent need to develop personal protective equipment (PPE) to halt the virus’ spread, and the field ballooned to hundreds of thousands of researchers. Hofmann and the other veteran researchers had a unique vantage point to evaluate this rapid escalation of PPE and medical device innovation, having been well-versed in the space before the spotlight found it.
READ: Megan Hofmann wins SIGCHI award for disability-device dissertation
Now studying the surge’s impact two years later, Hofmann and her co-researchers evaluated 623 3D-printable designs of new PPE technology submitted to the National Institutes of Health during the pandemic’s early stages. With so much new work and so many new researchers emerging so rapidly, the team was searching for patterns and noteworthy trends.
The study reinforced that while new PPE technology was developed impressively fast, there remains a fundamental importance to documenting the research process and sharing it publicly, along with evaluating risk and passing standard safety benchmarks. Hofmann believes the future of medical making lies in developing technology to partially automate the adjustment and verification process of the designs, creating a more efficient and streamlined process.
Simphony: Enhancing Accessible Pattern Design Practices among Blind Weavers
or “Using computing to help blind weavers weave”
Maitraye Das (+CAMD), Darren Gergle, Anne Marie Piper
Maitraye Das was introduced to a group of blind and low-vision weavers in 2019 while she was a graduate student at Northwestern University. She has immersed herself in the community ever since, and this paper is the final phase of her years-long work to help the weavers to design patterns and produce fabrics, cloths, artistic tapestries, and much more — despite pattern generation being a highly visual form of work.
To accomplish this, Das and her fellow researchers built Simphony, a system which combines musical notes, spoken audio, and tactile feedback to help the weavers envision weave patterns using their non-visual senses.
“Sighted people understand what they are seeing as a whole on a pattern,” Das said. “Whereas blind and low-vision people who rely on either auditory or tactile feedback, they sequentially explore what they notice and then cognitively create an imagery of the pattern.”
For the weavers, the ability to craft real-life products is a source of immense pride and self-expression. It also provides income and social capital, as their work is often sold, donated, or gifted to friends and family. In the future, Das hopes this work can help researchers to broadly understand the relationship between audio and visual–spatial perception, and to extend this technology to blind communities that stretch well beyond weaving.
“That’s important, but…”: How Computer Science Researchers Anticipate Unintended Consequences of Their Research Innovations
Or “Are CS researchers accounting for the ethical ramifications of their work?”
Kimberly Do, Rock Yuren Pang, Jiachen Jiang, Katharina Reinecke
Look no further than the dozens of works presented at CHI itself, and it’s clear that the computer science research community is rapidly developing powerful new technology with capabilities and potential that even the creators don’t fully comprehend. Often lost in the all-encompassing innovation race are the unintended consequences of the work, such as biases against marginalized groups, potential misuses of the technology, or fundamental flaws in the research process.
Kimberly Do and her collaborators interviewed 20 researchers across various CS disciplines to identify how they consider these consequences. What they found was a prevailing disconnect; researchers said they wanted to consider the ethical ramifications of their work, but in practice were often unwilling to take the necessary actions to do so.
“The biggest reason is the infrastructural barriers,” Do said. “So many people have this ‘publish or perish’ mindset where thinking about unintended consequences of the research isn’t necessarily as important as just getting it out there. People mostly view [the ethical component] as an accessory to publishing more.”
Do hopes that with heightened awareness of these oversights and an emphasis on community participation and feedback, researchers will be better equipped to make good on their stated intent.
OPTIMISM: Enabling Collaborative Implementation of Domain-Specific Metaheuristic Optimization
or “Bringing programming capabilities to non-programmers”
Megan Hofmann, Nayha Auradkar, Jessica Birchfield, Jerry Cao, Autumn G. Hughes, Gene S-H Kim, Shriya Kurpad, Kathryn J. Lum, Kelly Mack, Anisha Nilakantan, Margaret Ellen Seehorn, Emily Warnock, Jennifer Mankoff, Scott E. Hudson
Optimization is a complex, rapidly developing area of computer science research which uses complex equations and algorithms to “optimize” solutions across a wide range of fields, including software, medical machinery, and physical tools. But although its diverse array of applications presents great opportunity, it comes with a catch-22: only experienced programmers can implement advanced optimization techniques, but they are not experts in the fields they’re optimizing for.
After years of research, Megan Hofmann and her collaborators have developed a cutting-edge solution: OPTIMISM, a toolkit which enables domain-specific experts to collaborate with programmers to build and optimize new designs.
“A disabled person can be an expert in their navigational practices, someone can be a medical expert, or anything else,” Hofmann said. “And we wanted to build tools that allow people to work at their specific point of expertise, at a more complex dimension. Because you want to use their expertise, but you still need them to be able to do some programming tasks or to work with programmers in an effective way.”
The research has already produced results. An ophthalmologist and computer programmer collaboratively developed a program that could select the proper cataract lenses for doctors, streamlining cataract surgery. Likewise, a community of blind people worked with programmers to create customized tactile maps.
Is “Categorical Imperative” Metaversal?: A Kantian Ethical Framework for Social Virtual Reality
or “Real-world VR needs real-world ethics”
Eyup Engin Kucuk, Caglar Yildirim
Today, virtual reality is used for everything from firefighter training regimens to immersive video games to “sitting courtside” at an NBA game from the comfort of your living room. While the technology is innovative and rapidly improving, modern applications of VR tech are largely relegated to specific niches and too expensive for most to afford.
But imagine a world where large swaths of our daily activities operate in VR. Friends don’t travel to see each other, but instead socialize virtually. Real life and the virtual world blur together. It sounds dystopian, but Eyup Engin Kucuk and Caglar Yildirim are among the many who believe the world could be headed there one day — and that it is crucial to establish a precedent of ethical conduct while VR technology is still in its relative infancy.
By combining Kucuk’s philosophy background with Yildirim’s CS background, the pair researched and developed an ethical framework for social morality based on Immanuel Kant’s 200-year-old Theory of Morality, still among the most frequently used philosophical models for explaining human morality.
“Just as these ethical standards are valid in the real world, we applied Kant’s theory to social VR and discussed how these ethical standards are also applicable to virtual worlds,” Kucuk said. “We cannot think ‘Oh, it’s just the game world so we can do whatever we want.’”
And while the intersection of ethics and technology will evolve as the tech does, by proving the merits of human-based philosophy to the VR world at such an early stage, the researchers hope they’ll pave the way for further work in the space and bring awareness to the R&D underpinning VR products.
Exploratory Thematic Analysis of Crowdsourced Photosensitivity Warnings
or “Making flash warnings shine”
Laura South, Caglar Yildirim, Amy Pavel, and Michelle Borkin
Warnings about the presence, frequency, and duration of flashing lights are an important tool to ensure films are accessible to people with photosensitive epilepsy, for whom the lights might trigger seizures. However, warnings often don’t exist, leading to awkward and dangerous scenes in movie theaters. Additionally, many existing warnings are incredibly broad, prompting epileptic people to skip a film entirely when only a small portion of it would be risky.
To improve the warnings, these researchers sought to pinpoint what makes a photosensitivity warning good in the first place. They analyzed 265 crowdsourced warnings from the online trigger warning forum DoesTheDogDie to discover what elements were included in warnings that epileptic people wrote for one another. They found that scene warnings like “there are flashing lights immediately after the main character goes into a school” were more common than timestamped warnings like “there are flashing lights at 53:36.” They also found that descriptors like duration and color were more common than descriptors of frequency or size, and that qualitative warnings outnumbered quantitative ones. The researchers say that future work could automate qualitative, conversational photosensitivity warnings.
Improving Multiparty Interactions with a Robot using Large Language Models
or “Using robots to moderate group meetings”
Prasanth Murali, Ian Steenstra, Hye Sun Yun, Ameneh Shamekhi, Timothy Bickmore
Think back to your last group meeting. Did one or two speakers dominate the conversation, while no one else could get a word in? Did a conflict arise, or did an unhelpful tangent derail the meeting? While a meeting moderator or point person can help to facilitate productive conversation, studies have shown that a social robot can effectively facilitate such meetings as well. However, training such a robot is challenging with so many voices in the room.
That’s where Prasanth Murali and his co-authors come in. With the help of recently launched ChatGPT and other open-source AI machines, they aimed to identify and differentiate between speakers, and interpret those conversations to provide productive feedback. For example, the human-like robot can provide feedback such as “Hiring manager one, it seems like you did not have a lot to say, would you like time now?” or “Hiring manager two, it seems like you said this, but others did not quite pick up on your points. Can you clarify this a bit more?”
The technology is still in the early development stage, and it remains to be seen exactly how it will be delivered to users. However, this research further shows the promise of language models to identify speakers, and Murali hopes researchers can continue building on the technology, ultimately building social robots that make meetings more efficient, effective, and equitable.
Understanding Dark Patterns in IoT Devices
or “How smart home devices are invading your privacy”
Monica Kowalczyk, Johanna T. Gunawan, David Choffnes, Daniel J. Dubois, Woodrow Hartzog (external affiliate at Northeastern’s Cybersecurity and Privacy Institute), Christo Wilson
Smart fridges. Smart lights. Smart thermostats. Smart speakers. Smart TVs. Smart doorbells. Smart security systems. All consumer devices which apply internet capabilities to traditionally “dumb” (or non-internet-enabled) everyday objects. Known as Internet of Things (IoT) devices, this evolving category interacts with the internet much like our phones or laptops, but helps to streamline daily needs through physical hardware.
But as IoT devices become more ingrained in our lives, concerns have mounted over their pervasive use of “dark patterns” — deceptive user interfaces that manipulate users into taking actions or sharing information they don’t intend to. Though dark patterns are a major issue across technology in general, their application within IoT devices is particularly concerning because of their unique ability to breach users’ privacy.
“[IoT devices] might be placed in intimate spaces such as bedrooms or bathrooms, have sensors to collect very specific data, or have hardware that is constantly recording and listening, giving them access to sensitive data not as easily accessible to websites and apps,” said Monica Kowalczyk, a fourth-year Khoury College undergraduate who worked with a team of researchers to undercover these patterns in widely used smart-home devices.
The researchers reset 57 devices to their bare-bones settings, then closely examined the devices, their features and settings, their account deletion processes, and their user data and legal terms. They found that speakers, doorbells, and camera devices contained the most dark patterns among IoT devices, and that large-scale manufacturers such as Amazon and Google generally had more dark patterns than other vendors.
What Do We Mean When We Talk about Trust in Social Media? A Systematic Review
or “How can we tell if we trust social media if we can’t agree on what trust is?”
CHI “Best Paper” winner
Yixuan Zhang, Joseph D. Gaggiano, Nutchanon Yongsatianchot, Nurul M. Suhaimi, Miso Kim (CAMD), Yifan Sun, Jacqueline Griffin (CoE), Andrea G. Parker
A lot of research has examined trust and social media in recent years. How is trust in social media defined and measured? What factors influence its development and its consequences? And do researchers define trust cohesively enough for these conclusions to aggregate meaningfully?
This research team reviewed 70 papers examining social media and trust concepts, and found that less than half of those papers defined trust at all. Those that did often understood it as a willingness to be vulnerable to another party, but who was being trusted — the platform, the information, or the other users — was inconsistently defined. That said, there were still meaningful conclusions to be drawn, for instance that a user’s location, education, income, and political ideology were better predictors of their trust in social media than their gender or age.
The researchers encourage future work to clearly specify the context and the trustee when defining trust, and for that definition to align with other works’ definitions. The researchers also advocate for further research into distrust and mistrust, and into the conditions that create or result from trust in social media.
Exploring the Use of Personalized AI for Identifying Misinformation on Social Media
or “Can we personalize AI content moderation without it personalizing us back?”
Farnaz Jahanbakhsh, Yannis Katsis, Dakuo Wang (+CAMD), Lucian Popa, Michael Muller
It’s legendarily difficult to deal with misinformation on social media. Users disagree about what constitutes misinformation, who gets to decide, whether automation should be used, and what happens to posts deemed inaccurate. But what if a personalized AI could learn what types of posts you specifically were likely to deem as misinformation? Would it help you avoid falling victim to untruths? Or would it influence what you believed to be true in the first place?
These researchers studied how users perceive a personalized AI that helps them spot content they would likely deem misinformation, and whether seeing the AI’s predictions impacted how a user assessed the content’s accuracy for themselves. The team had 50 participants rate the accuracy of tweets about COVID-19 and their confidence in their predictions with and without AI assistance, then solicited feedback about the users’ experience.
They found that when a user saw the AI’s prediction about whether the user would find a tweet accurate, the user was more likely to agree with the AI’s prediction. However, asking the user why they believed a tweet was true or untrue eliminated this bias. The participants varied widely when it came to how they perceived the AI’s usefulness; some believed it would worsen political bubbles, while others liked how it guided them to inspect content more closely.
“Everyone is Covered”: Exploring the Role of Online Interactions in Facilitating Connection and Social Support in Black Churches
or “What do Black churches need to bring faith online?”
Darley Sackitey, Teresa K. O’Leary, Michael Paasche-Orlow, Timothy Bickmore, Andrea G. Parker
Thanks to their unique history navigating American colonialism, institutional neglect, and white supremacy, Black churches have become critical community gathering places and centers of social support. By extending their reach online, they could make the health benefits of faith and community more available to more people. A poorly designed tool could do more harm than good though, so as part of an ongoing collaboration with two Black churches in Boston, these researchers studied the features and design considerations most critical to building effective spiritual technology for Black communities.
These researchers worked with nine church members in leadership roles through interviews, focus groups, and a trial technical probe called Church Connect Community. To understand which features were important in faith-related technology, they discussed how the church members used the platform, how they wanted to use the platform, and what their concerns about the platform were. The research team identified several important design recommendations, including building leadership opportunities, staying conscious of inclusivity, integrating with offline church programs, prioritizing the spiritual over the material, and offering multiple types of spiritual support, from one-on-ones to larger community groups.
The most prestigious human–computer interaction conference in the world is taking place in Hamburg, Germany this month, and Khoury College’s researchers — along with their collaborators in the College of Engineering (CoE) and the College of Arts, Media and Design (CAMD) — are ready. Their work, which touches on everything from parrot socialization to deceptive dark patterns to social media misinformation, has made Northeastern the seventh-ranked institution in the world in terms of CHI publications, as well as sixth in the United States and first in Massachusetts. For a full schedule of their papers, panels, and other appearances at the conference, click here.
Click on one of the linked summaries below to learn about the corresponding paper, or simply read on.
Bold = Khoury College
() = another Northeastern college
(+) = interdisciplinary appointee
* = Khoury College courtesy appointee
Corsetto: A Kinesthetic Garment for Designing, Composing for, and Experiencing an Intersubjective Haptic Voice
or “How to feel music … literally”
Ozgun Kilic Afsar, Yoav Luft, Kelsey Cotton, Ekaterina R. Stepanova, Claudia Núñez-Pacheco, Rébecca Kleinberger (+CAMD), Fehmi Ben Abdesslem, Hiroshi Ishii, Kristina Höök
There is nothing quite like an immersive live music performance. When the composition meets the performance, and when the audience is fully engaged, the building feels like it’s lifting off its studs. The experience is not just visceral; it’s physical.
While the ambience of live music can naturally elicit this sensation, a team of researchers hailing from Sweden, the US, and Canada looked to deepen it. Their creation is Corsetto, a robotic upper-body garment which uses haptic gestures to transfer the physicality of a singer’s performance to the audience members wearing the device.
“This project explored the strange question of what it feels like to wear someone else’s voice,” said Rébecca Kleinberger, a professor jointly appointed between Khoury College and CAMD. “This collaboration was truly multidisciplinary, with elements of art, design, fabrication, software, HCI, experience design, and philosophy.”
The team created Corsetto with the aid of a classically trained opera singer who, through sessions of vocal exercises, helped the team to understand the biomechanical elements of singing. Once the team had incorporated those lessons into a functional device, they invited audiences to wear it during a performance of Morton Feldman’s piece “Three Voices,” then interviewed them afterwards. The result, they say, was an “intersubjective haptic voice,” a connective tissue between the auditory and physical elements of the performance that can help to convey the gravity and meaning of the music.
“[Corsetto] blurred boundaries between inner and outer aspects of participants’ bodies,” the researchers wrote, “at times rendering a fusion of all the bodies in the room into one holistic experience.”
“Who is the right homeless client?”: Values in Algorithmic Homelessness Service Provision and Machine Learning Research
or “Fair and ethical research practices for homeless populations”
Dilruba Showkat, Angela D. R. Smith, Wang Lingqing, Alexandra To (+CAMD)
As emphasis on ethical and equitable research practices has increased in recent years, so too have new innovations and diverse perspectives across the field of CS. But homeless populations have remained marginalized, and according to Khoury College doctoral student Dilruba Showkat, the field’s failure to produce ethical, balanced research of homeless communities isn’t due to a lack of intent or effort. Rather, it’s because homeless people are often difficult to access and don’t always cooperate with researchers, so their point of view often gets left out of the picture.
“Homeless populations’ values are well known in HCI research,” Showkat said. “However, during my analysis, I found a disconnect between these marginalized communities’ values and the machine learning systems that are developed to support them.”
Among the issues Showkat and her collaborators found were breaches of homeless peoples’ privacy and autonomy, as well as their lack of technological savvy and ability to self-advocate — each of which made them susceptible to being nullified or misrepresented by the predictors, categories, and algorithms used in machine learning technology.
“In several research papers, I’ve found social security numbers of homeless people, as well as other private information, unintentionally passed when sharing data sets,” Showkat said. “This is something really wrong with the system, and we should change that by conducting research that is held accountable. ”
The paper argues that researchers must actively protect the privacy and autonomy of homeless people, since they often cannot voice their concerns themselves. One proposed solution is to design machine learning tools using novel experimental techniques that don’t rely on vulnerable communities’ data, and instead prioritize human values. Another recommendation is to develop reproducible and replicable models by sharing a chunk of data anonymously, which would enable researchers to use the information without violating privacy.
Why, When, and From Whom: Considerations for Collecting and Reporting Race and Ethnicity Data in HCI
or “How should we ask about race?”
CHI “Honorable Mention” winner
Yiqun T. Chen, Angela D.R. Smith, Katharina Reinecke, Alexandra To (+CAMD)
Last year, Alexandra To and her fellow researchers reviewed six years of CHI proceedings and determined that just three percent of research papers on human–computer interaction collect race or ethnicity data. This year, they interviewed 15 authors to better understand how they’re collecting race and ethnicity data now, and to begin to develop best practices for collection in the future.
This research team doesn’t recommend collecting race and ethnicity data for every study, as privacy concerns, the risk of further entrenching “scientific racism”— the pseudoscientific belief that there is empirical evidence justifying racism — and participant retention could all be legitimate reasons not to. Instead, they encourage researchers to make such data collection a deliberate, rather than a default, decision.
If a study does collect racial data, the researchers recommend allowing participants to select multiple racial categories, including an open-ended way to self-identify, using more granular categories than the US Census default categories if appropriate, and considering where race is acting as a stand-in for another attribute (like socioeconomic status) and asking about that attribute directly instead.
Birds of a Feather Video-Flock Together: Design and Evaluation of an Agency-Based Parrot-to-Parrot Video-Calling System for Interspecies Ethical Enrichment
or “If you give a bird a Zoom call”
CHI “Honorable Mention” winner
Rébecca Kleinberger (+CAMD), Jennifer Cunha, Megha M. Vemuri, Ilyena Hirskyj-Douglas
Parrots have earned their reputation as the brainiacs of the aviary world, but such brainpower entails a tricky reality for the millions kept as pets. They need social and cognitive stimulation like their counterparts in the wild, but seldom get enough of it in captivity.
Seeking to rectify that imbalance, a new study first-authored by Khoury College–CAMD professor Rébecca Kleinberger aimed to provide the birds with the same social solution many humans jumped to in March 2020: video calling. In a three-month study involving 18 domesticated parrots, the team facilitated parrot-to-parrot calls using Facebook Messenger, all while monitoring the birds’ agency and motivation in initiating the calls and their engagement once the call began.
“Once the parrots learned to trigger the calls, it was hard to stop them, and some started exploring their tablets a bit too much,” Kleinberger recalled. “It was particularly challenging, but also truly rewarding, to involve so many animal participants for such a long period. To this day, I still sometimes receive calls from some of our bird participants.”
All 18 birds participated, most of them showing high motivation and intent while on the video call. Additionally, their human caretakers reported that not only did some of the birds learn new behaviors from their remote buddies — including flying and foraging — but the experience also helped the humans connect with their pets and learn more about them.
Through Their Eyes and In Their Shoes: Providing Group Awareness During Collaboration Across Virtual Reality and Desktop Platforms
or “How to work together when your work looks very different”
David Saffo, Andrea Batch, Cody Dunne, Niklas Elmqvist
To collaborate on a data exploration project, coworkers need to share an understanding of the parameters of their collaboration, which is difficult when one partner has a whole extra dimension to play in. These researchers used a tool they developed called VRxD to help a person on a desktop computer and a person using a VR headset to explore a data set about baseball pitchers; the researchers studied how they worked together and developed design recommendations for cross-platform collaborative tools.
“One of the potential benefits of AR and VR devices is the unique ways they can enable multi-user experiences and collaboration. However, I do not like that these experiences require all users to also use AR and VR devices,” lead researcher and doctoral student David Saffo said. “What excited me was thinking of all the design possibilities of a cross-platform system.”
By varying what features were available to the study’s six pairs of participants, the researchers determined that certain features — like the ability to quickly toggle into a collaborator’s view or leave the collaborator’s view open in another window — influenced how much time the pairs spent discussing their work versus working on the task itself, and who tended to lead a given activity.
Based on their findings, the researchers developed four design recommendations, which they call “through their eyes and in their shoes,” emphasizing how important it is for systems to allow collaborators to understand one another’s perspectives. Per the recommendations, VR–desktop collaboration systems should:
1. build in shared points of reference
2. encourage leadership by the user whose interface is best suited to the task
3. incorporate information cues (such as causing data to light up in both views when either participant clicks on it)
4. build in multiple ways to share perspectives so collaborators can quickly check on each other, present to one another, or sustain longer periods of collaboration
Auditory Seasoning Filters: Altering Food Perception via Augmented Sonic Feedback of Chewing Sounds
or “What can Pringles and sound effects teach us about our eating impulses?”
Rébecca Kleinberger (+CAMD), Akito Oshiro van Troyer, Qian Janice Wang
Aside from their snackability, Pringle chips are uniquely useful experimental stimuli for food-tech research, as their uniform size, texture, and flavor allows researchers to create consistent food samples and investigate how environmental conditions affect the eating experience. In a new study co-led by Rébecca Kleinberger, the team gave Pringles to study subjects while modulating the sound of their chewing using precise volume tweaks, delays, reverbs, and frequency filters. Then they determined how that modulation changed the subjects’ perceptions of both the food and their own hunger.
Among a slew of findings, the team discovered that amplified chewing sounds made participants feel greater sensations of satiety, texture, and flavor. Delay slowed eating speed. Large-room reverb diminished sourness and cravings. Filtering of high and low frequencies each bolstered fullness.
“This technology is quite fun and it helps you understand what’s going on in your brain, but is there a way to apply it and make an impact, especially regarding foods that aren’t especially healthy for you?” Kleinberger mused. “Can we make you eat slower? Can we make you chew more? Can we make you perceive more flavor and saltiness even if there’s actually less, so you’d eat food with less salt and artificial flavor?”
To that end, the team believes that future efforts can package these audio filters into an easy-to-use mobile app that helps users to regulate their food perception and cravings in real time — all without the need for the external hardware, explicit self-regulation, or specially prepared foods involved in previous approaches. But apart from the intricacies of the sound effects and neurology at play, Kleinberger believes there’s an underlying question of perspective.
“All of our participants told us they’d never had an experience like this, that it was really fun and transformative,” Kleinberger said. “Even just the notion of gaining awareness of your chewing, of becoming more attuned to it and having this stronger experience of sound — is interesting in itself. It can make people eat more mindfully.”
Model Sketching: Centering Concepts in Early-Stage Machine Learning Model Design
or “Let’s give machine learning designers a sketchbook”
Michelle S. Lam, Zixian Ma, Anne Li, Izequiel Freitas, Dakuo Wang (+CAMD), James A. Landay, Michael S. Bernstein
When designing a machine learning (ML) model, it’s easy to get stuck on the details of how to execute a solution at the expense of thinking about what kinds of solutions to pursue in the first place. These researchers want to get ML designers focused on the big picture, so they created ModelSketchBook: a Python package for quickly “sketching” minimal models and testing different design choices.
ModelSketchBook uses zero-shot learning and pretrained models like GPT-3 to let users create concepts, link them logically, and see how a machine learning model interprets those concepts on real data. For example, someone trying to build a model to identify hateful memes might use ModelSketchBook to explore how ML interprets the interactions of concepts like racism, nudity, and violence, then iterate several sketches to refine their initial model.
The researchers had 17 ML practitioners sketch models that would determine whether memes from Meta AI’s Hateful Memes Challenge should be removed from social media. They found that ModelSketchBook helped the practitioners discover gaps in their models, come up with a greater diversity of concepts to include, and produce quality outcomes more quickly. Their data also suggests that iterating on sketches — which the method encourages — produces better-performing models.
Thought Bubbles: A Proxy into Players’ Mental Model Development
or “How can we think about how we think?”
Omid Mohaddesi (CoE), Noah Chicoine (CoE), Min Gong (CoE), Ozlem Ergun (CoE), Jacqueline Griffin (CoE), David Kaeli* (CoE), Stacy Marsella (+CoS), Casper Harteveld* (CAMD)
In a work honored at CHI last year, this research team designed a game. Players role-played as wholesalers attempting to source medication from factories and supply hospitals during a supply chain disruption. Some hoarded medication from the beginning, others reacted to the shortage with panic-buying, and still others followed their computer’s recommendations through the whole game.
“(Last year), we identified Hoarders, Reactors, and Followers using observable behavior, and now we attempted to investigate differences in the development of these groups’ mental models,” said lead author and doctoral student Omid Mohaddesi, referring to the representation of a real-world situation that someone holds in their head. “We can see different patterns in the mental model development of groups of people and how information sharing impacts their mental model development … all this insight can potentially help us make better decision-support tools.”
To get contextualized insights into how their subjects’ mental models developed over time, the research team added a recurring, open-ended reflection prompt called a “thought bubble” to the supply chain game. Through two related studies, they coded participants’ reflections based on whether they were perceiving elements of the environment (“the backlog has gone up”), comprehending the meaning of those elements (“this is bad for our business”), or projecting a future state (“we will have to buy more medicine”). Researchers could then spot patterns between the ways people modeled the world and the decisions they made when managing a supply chain.
The study found that distinctions in participants’ supply chain management decisions are the result of differences in the development of their mental models. For instance, hoarders are proactive because they struggle to make sense of the situation — leading to uncertainty about the future — while “followers” trust order suggestions because they interpret the situation positively. The study also presents thought bubbles as a method for further studying mental models.
Rapid Convergence: The Outcomes of Making PPE during a Healthcare Crisis
or “PPE innovation soared during the pandemic, but how did it hold up?”
Kelly Mack, Megan Hofmann, Udaya Lakshmi, Jerry Cao, Nayha Auradkar, Rosa I. Arriaga, Scott E. Hudson, Jennifer Mankoff
In 2018, when Megan Hofmann and her partners began researching “medical making” — the application of 3D-printing models to the medical device and healthcare fields — she estimates there were only a few thousand medical makers worldwide.
But with the onset of the COVID-19 pandemic in 2020, the once-niche field was upended overnight. Suddenly, there was an urgent need to develop personal protective equipment (PPE) to halt the virus’ spread, and the field ballooned to hundreds of thousands of researchers. Hofmann and the other veteran researchers had a unique vantage point to evaluate this rapid escalation of PPE and medical device innovation, having been well-versed in the space before the spotlight found it.
READ: Megan Hofmann wins SIGCHI award for disability-device dissertation
Now studying the surge’s impact two years later, Hofmann and her co-researchers evaluated 623 3D-printable designs of new PPE technology submitted to the National Institutes of Health during the pandemic’s early stages. With so much new work and so many new researchers emerging so rapidly, the team was searching for patterns and noteworthy trends.
The study reinforced that while new PPE technology was developed impressively fast, there remains a fundamental importance to documenting the research process and sharing it publicly, along with evaluating risk and passing standard safety benchmarks. Hofmann believes the future of medical making lies in developing technology to partially automate the adjustment and verification process of the designs, creating a more efficient and streamlined process.
Simphony: Enhancing Accessible Pattern Design Practices among Blind Weavers
or “Using computing to help blind weavers weave”
Maitraye Das (+CAMD), Darren Gergle, Anne Marie Piper
Maitraye Das was introduced to a group of blind and low-vision weavers in 2019 while she was a graduate student at Northwestern University. She has immersed herself in the community ever since, and this paper is the final phase of her years-long work to help the weavers to design patterns and produce fabrics, cloths, artistic tapestries, and much more — despite pattern generation being a highly visual form of work.
To accomplish this, Das and her fellow researchers built Simphony, a system which combines musical notes, spoken audio, and tactile feedback to help the weavers envision weave patterns using their non-visual senses.
“Sighted people understand what they are seeing as a whole on a pattern,” Das said. “Whereas blind and low-vision people who rely on either auditory or tactile feedback, they sequentially explore what they notice and then cognitively create an imagery of the pattern.”
For the weavers, the ability to craft real-life products is a source of immense pride and self-expression. It also provides income and social capital, as their work is often sold, donated, or gifted to friends and family. In the future, Das hopes this work can help researchers to broadly understand the relationship between audio and visual–spatial perception, and to extend this technology to blind communities that stretch well beyond weaving.
“That’s important, but…”: How Computer Science Researchers Anticipate Unintended Consequences of Their Research Innovations
Or “Are CS researchers accounting for the ethical ramifications of their work?”
Kimberly Do, Rock Yuren Pang, Jiachen Jiang, Katharina Reinecke
Look no further than the dozens of works presented at CHI itself, and it’s clear that the computer science research community is rapidly developing powerful new technology with capabilities and potential that even the creators don’t fully comprehend. Often lost in the all-encompassing innovation race are the unintended consequences of the work, such as biases against marginalized groups, potential misuses of the technology, or fundamental flaws in the research process.
Kimberly Do and her collaborators interviewed 20 researchers across various CS disciplines to identify how they consider these consequences. What they found was a prevailing disconnect; researchers said they wanted to consider the ethical ramifications of their work, but in practice were often unwilling to take the necessary actions to do so.
“The biggest reason is the infrastructural barriers,” Do said. “So many people have this ‘publish or perish’ mindset where thinking about unintended consequences of the research isn’t necessarily as important as just getting it out there. People mostly view [the ethical component] as an accessory to publishing more.”
Do hopes that with heightened awareness of these oversights and an emphasis on community participation and feedback, researchers will be better equipped to make good on their stated intent.
OPTIMISM: Enabling Collaborative Implementation of Domain-Specific Metaheuristic Optimization
or “Bringing programming capabilities to non-programmers”
Megan Hofmann, Nayha Auradkar, Jessica Birchfield, Jerry Cao, Autumn G. Hughes, Gene S-H Kim, Shriya Kurpad, Kathryn J. Lum, Kelly Mack, Anisha Nilakantan, Margaret Ellen Seehorn, Emily Warnock, Jennifer Mankoff, Scott E. Hudson
Optimization is a complex, rapidly developing area of computer science research which uses complex equations and algorithms to “optimize” solutions across a wide range of fields, including software, medical machinery, and physical tools. But although its diverse array of applications presents great opportunity, it comes with a catch-22: only experienced programmers can implement advanced optimization techniques, but they are not experts in the fields they’re optimizing for.
After years of research, Megan Hofmann and her collaborators have developed a cutting-edge solution: OPTIMISM, a toolkit which enables domain-specific experts to collaborate with programmers to build and optimize new designs.
“A disabled person can be an expert in their navigational practices, someone can be a medical expert, or anything else,” Hofmann said. “And we wanted to build tools that allow people to work at their specific point of expertise, at a more complex dimension. Because you want to use their expertise, but you still need them to be able to do some programming tasks or to work with programmers in an effective way.”
The research has already produced results. An ophthalmologist and computer programmer collaboratively developed a program that could select the proper cataract lenses for doctors, streamlining cataract surgery. Likewise, a community of blind people worked with programmers to create customized tactile maps.
Is “Categorical Imperative” Metaversal?: A Kantian Ethical Framework for Social Virtual Reality
or “Real-world VR needs real-world ethics”
Eyup Engin Kucuk, Caglar Yildirim
Today, virtual reality is used for everything from firefighter training regimens to immersive video games to “sitting courtside” at an NBA game from the comfort of your living room. While the technology is innovative and rapidly improving, modern applications of VR tech are largely relegated to specific niches and too expensive for most to afford.
But imagine a world where large swaths of our daily activities operate in VR. Friends don’t travel to see each other, but instead socialize virtually. Real life and the virtual world blur together. It sounds dystopian, but Eyup Engin Kucuk and Caglar Yildirim are among the many who believe the world could be headed there one day — and that it is crucial to establish a precedent of ethical conduct while VR technology is still in its relative infancy.
By combining Kucuk’s philosophy background with Yildirim’s CS background, the pair researched and developed an ethical framework for social morality based on Immanuel Kant’s 200-year-old Theory of Morality, still among the most frequently used philosophical models for explaining human morality.
“Just as these ethical standards are valid in the real world, we applied Kant’s theory to social VR and discussed how these ethical standards are also applicable to virtual worlds,” Kucuk said. “We cannot think ‘Oh, it’s just the game world so we can do whatever we want.’”
And while the intersection of ethics and technology will evolve as the tech does, by proving the merits of human-based philosophy to the VR world at such an early stage, the researchers hope they’ll pave the way for further work in the space and bring awareness to the R&D underpinning VR products.
Exploratory Thematic Analysis of Crowdsourced Photosensitivity Warnings
or “Making flash warnings shine”
Laura South, Caglar Yildirim, Amy Pavel, and Michelle Borkin
Warnings about the presence, frequency, and duration of flashing lights are an important tool to ensure films are accessible to people with photosensitive epilepsy, for whom the lights might trigger seizures. However, warnings often don’t exist, leading to awkward and dangerous scenes in movie theaters. Additionally, many existing warnings are incredibly broad, prompting epileptic people to skip a film entirely when only a small portion of it would be risky.
To improve the warnings, these researchers sought to pinpoint what makes a photosensitivity warning good in the first place. They analyzed 265 crowdsourced warnings from the online trigger warning forum DoesTheDogDie to discover what elements were included in warnings that epileptic people wrote for one another. They found that scene warnings like “there are flashing lights immediately after the main character goes into a school” were more common than timestamped warnings like “there are flashing lights at 53:36.” They also found that descriptors like duration and color were more common than descriptors of frequency or size, and that qualitative warnings outnumbered quantitative ones. The researchers say that future work could automate qualitative, conversational photosensitivity warnings.
Improving Multiparty Interactions with a Robot using Large Language Models
or “Using robots to moderate group meetings”
Prasanth Murali, Ian Steenstra, Hye Sun Yun, Ameneh Shamekhi, Timothy Bickmore
Think back to your last group meeting. Did one or two speakers dominate the conversation, while no one else could get a word in? Did a conflict arise, or did an unhelpful tangent derail the meeting? While a meeting moderator or point person can help to facilitate productive conversation, studies have shown that a social robot can effectively facilitate such meetings as well. However, training such a robot is challenging with so many voices in the room.
That’s where Prasanth Murali and his co-authors come in. With the help of recently launched ChatGPT and other open-source AI machines, they aimed to identify and differentiate between speakers, and interpret those conversations to provide productive feedback. For example, the human-like robot can provide feedback such as “Hiring manager one, it seems like you did not have a lot to say, would you like time now?” or “Hiring manager two, it seems like you said this, but others did not quite pick up on your points. Can you clarify this a bit more?”
The technology is still in the early development stage, and it remains to be seen exactly how it will be delivered to users. However, this research further shows the promise of language models to identify speakers, and Murali hopes researchers can continue building on the technology, ultimately building social robots that make meetings more efficient, effective, and equitable.
Understanding Dark Patterns in IoT Devices
or “How smart home devices are invading your privacy”
Monica Kowalczyk, Johanna T. Gunawan, David Choffnes, Daniel J. Dubois, Woodrow Hartzog (external affiliate at Northeastern’s Cybersecurity and Privacy Institute), Christo Wilson
Smart fridges. Smart lights. Smart thermostats. Smart speakers. Smart TVs. Smart doorbells. Smart security systems. All consumer devices which apply internet capabilities to traditionally “dumb” (or non-internet-enabled) everyday objects. Known as Internet of Things (IoT) devices, this evolving category interacts with the internet much like our phones or laptops, but helps to streamline daily needs through physical hardware.
But as IoT devices become more ingrained in our lives, concerns have mounted over their pervasive use of “dark patterns” — deceptive user interfaces that manipulate users into taking actions or sharing information they don’t intend to. Though dark patterns are a major issue across technology in general, their application within IoT devices is particularly concerning because of their unique ability to breach users’ privacy.
“[IoT devices] might be placed in intimate spaces such as bedrooms or bathrooms, have sensors to collect very specific data, or have hardware that is constantly recording and listening, giving them access to sensitive data not as easily accessible to websites and apps,” said Monica Kowalczyk, a fourth-year Khoury College undergraduate who worked with a team of researchers to undercover these patterns in widely used smart-home devices.
The researchers reset 57 devices to their bare-bones settings, then closely examined the devices, their features and settings, their account deletion processes, and their user data and legal terms. They found that speakers, doorbells, and camera devices contained the most dark patterns among IoT devices, and that large-scale manufacturers such as Amazon and Google generally had more dark patterns than other vendors.
What Do We Mean When We Talk about Trust in Social Media? A Systematic Review
or “How can we tell if we trust social media if we can’t agree on what trust is?”
CHI “Best Paper” winner
Yixuan Zhang, Joseph D. Gaggiano, Nutchanon Yongsatianchot, Nurul M. Suhaimi, Miso Kim (CAMD), Yifan Sun, Jacqueline Griffin (CoE), Andrea G. Parker
A lot of research has examined trust and social media in recent years. How is trust in social media defined and measured? What factors influence its development and its consequences? And do researchers define trust cohesively enough for these conclusions to aggregate meaningfully?
This research team reviewed 70 papers examining social media and trust concepts, and found that less than half of those papers defined trust at all. Those that did often understood it as a willingness to be vulnerable to another party, but who was being trusted — the platform, the information, or the other users — was inconsistently defined. That said, there were still meaningful conclusions to be drawn, for instance that a user’s location, education, income, and political ideology were better predictors of their trust in social media than their gender or age.
The researchers encourage future work to clearly specify the context and the trustee when defining trust, and for that definition to align with other works’ definitions. The researchers also advocate for further research into distrust and mistrust, and into the conditions that create or result from trust in social media.
Exploring the Use of Personalized AI for Identifying Misinformation on Social Media
or “Can we personalize AI content moderation without it personalizing us back?”
Farnaz Jahanbakhsh, Yannis Katsis, Dakuo Wang (+CAMD), Lucian Popa, Michael Muller
It’s legendarily difficult to deal with misinformation on social media. Users disagree about what constitutes misinformation, who gets to decide, whether automation should be used, and what happens to posts deemed inaccurate. But what if a personalized AI could learn what types of posts you specifically were likely to deem as misinformation? Would it help you avoid falling victim to untruths? Or would it influence what you believed to be true in the first place?
These researchers studied how users perceive a personalized AI that helps them spot content they would likely deem misinformation, and whether seeing the AI’s predictions impacted how a user assessed the content’s accuracy for themselves. The team had 50 participants rate the accuracy of tweets about COVID-19 and their confidence in their predictions with and without AI assistance, then solicited feedback about the users’ experience.
They found that when a user saw the AI’s prediction about whether the user would find a tweet accurate, the user was more likely to agree with the AI’s prediction. However, asking the user why they believed a tweet was true or untrue eliminated this bias. The participants varied widely when it came to how they perceived the AI’s usefulness; some believed it would worsen political bubbles, while others liked how it guided them to inspect content more closely.
“Everyone is Covered”: Exploring the Role of Online Interactions in Facilitating Connection and Social Support in Black Churches
or “What do Black churches need to bring faith online?”
Darley Sackitey, Teresa K. O’Leary, Michael Paasche-Orlow, Timothy Bickmore, Andrea G. Parker
Thanks to their unique history navigating American colonialism, institutional neglect, and white supremacy, Black churches have become critical community gathering places and centers of social support. By extending their reach online, they could make the health benefits of faith and community more available to more people. A poorly designed tool could do more harm than good though, so as part of an ongoing collaboration with two Black churches in Boston, these researchers studied the features and design considerations most critical to building effective spiritual technology for Black communities.
These researchers worked with nine church members in leadership roles through interviews, focus groups, and a trial technical probe called Church Connect Community. To understand which features were important in faith-related technology, they discussed how the church members used the platform, how they wanted to use the platform, and what their concerns about the platform were. The research team identified several important design recommendations, including building leadership opportunities, staying conscious of inclusivity, integrating with offline church programs, prioritizing the spiritual over the material, and offering multiple types of spiritual support, from one-on-ones to larger community groups.
The most prestigious human–computer interaction conference in the world is taking place in Hamburg, Germany this month, and Khoury College’s researchers — along with their collaborators in the College of Engineering (CoE) and the College of Arts, Media and Design (CAMD) — are ready. Their work, which touches on everything from parrot socialization to deceptive dark patterns to social media misinformation, has made Northeastern the seventh-ranked institution in the world in terms of CHI publications, as well as sixth in the United States and first in Massachusetts. For a full schedule of their papers, panels, and other appearances at the conference, click here.
Click on one of the linked summaries below to learn about the corresponding paper, or simply read on.
Bold = Khoury College
() = another Northeastern college
(+) = interdisciplinary appointee
* = Khoury College courtesy appointee
Corsetto: A Kinesthetic Garment for Designing, Composing for, and Experiencing an Intersubjective Haptic Voice
or “How to feel music … literally”
Ozgun Kilic Afsar, Yoav Luft, Kelsey Cotton, Ekaterina R. Stepanova, Claudia Núñez-Pacheco, Rébecca Kleinberger (+CAMD), Fehmi Ben Abdesslem, Hiroshi Ishii, Kristina Höök
There is nothing quite like an immersive live music performance. When the composition meets the performance, and when the audience is fully engaged, the building feels like it’s lifting off its studs. The experience is not just visceral; it’s physical.
While the ambience of live music can naturally elicit this sensation, a team of researchers hailing from Sweden, the US, and Canada looked to deepen it. Their creation is Corsetto, a robotic upper-body garment which uses haptic gestures to transfer the physicality of a singer’s performance to the audience members wearing the device.
“This project explored the strange question of what it feels like to wear someone else’s voice,” said Rébecca Kleinberger, a professor jointly appointed between Khoury College and CAMD. “This collaboration was truly multidisciplinary, with elements of art, design, fabrication, software, HCI, experience design, and philosophy.”
The team created Corsetto with the aid of a classically trained opera singer who, through sessions of vocal exercises, helped the team to understand the biomechanical elements of singing. Once the team had incorporated those lessons into a functional device, they invited audiences to wear it during a performance of Morton Feldman’s piece “Three Voices,” then interviewed them afterwards. The result, they say, was an “intersubjective haptic voice,” a connective tissue between the auditory and physical elements of the performance that can help to convey the gravity and meaning of the music.
“[Corsetto] blurred boundaries between inner and outer aspects of participants’ bodies,” the researchers wrote, “at times rendering a fusion of all the bodies in the room into one holistic experience.”
“Who is the right homeless client?”: Values in Algorithmic Homelessness Service Provision and Machine Learning Research
or “Fair and ethical research practices for homeless populations”
Dilruba Showkat, Angela D. R. Smith, Wang Lingqing, Alexandra To (+CAMD)
As emphasis on ethical and equitable research practices has increased in recent years, so too have new innovations and diverse perspectives across the field of CS. But homeless populations have remained marginalized, and according to Khoury College doctoral student Dilruba Showkat, the field’s failure to produce ethical, balanced research of homeless communities isn’t due to a lack of intent or effort. Rather, it’s because homeless people are often difficult to access and don’t always cooperate with researchers, so their point of view often gets left out of the picture.
“Homeless populations’ values are well known in HCI research,” Showkat said. “However, during my analysis, I found a disconnect between these marginalized communities’ values and the machine learning systems that are developed to support them.”
Among the issues Showkat and her collaborators found were breaches of homeless peoples’ privacy and autonomy, as well as their lack of technological savvy and ability to self-advocate — each of which made them susceptible to being nullified or misrepresented by the predictors, categories, and algorithms used in machine learning technology.
“In several research papers, I’ve found social security numbers of homeless people, as well as other private information, unintentionally passed when sharing data sets,” Showkat said. “This is something really wrong with the system, and we should change that by conducting research that is held accountable. ”
The paper argues that researchers must actively protect the privacy and autonomy of homeless people, since they often cannot voice their concerns themselves. One proposed solution is to design machine learning tools using novel experimental techniques that don’t rely on vulnerable communities’ data, and instead prioritize human values. Another recommendation is to develop reproducible and replicable models by sharing a chunk of data anonymously, which would enable researchers to use the information without violating privacy.
Why, When, and From Whom: Considerations for Collecting and Reporting Race and Ethnicity Data in HCI
or “How should we ask about race?”
CHI “Honorable Mention” winner
Yiqun T. Chen, Angela D.R. Smith, Katharina Reinecke, Alexandra To (+CAMD)
Last year, Alexandra To and her fellow researchers reviewed six years of CHI proceedings and determined that just three percent of research papers on human–computer interaction collect race or ethnicity data. This year, they interviewed 15 authors to better understand how they’re collecting race and ethnicity data now, and to begin to develop best practices for collection in the future.
This research team doesn’t recommend collecting race and ethnicity data for every study, as privacy concerns, the risk of further entrenching “scientific racism”— the pseudoscientific belief that there is empirical evidence justifying racism — and participant retention could all be legitimate reasons not to. Instead, they encourage researchers to make such data collection a deliberate, rather than a default, decision.
If a study does collect racial data, the researchers recommend allowing participants to select multiple racial categories, including an open-ended way to self-identify, using more granular categories than the US Census default categories if appropriate, and considering where race is acting as a stand-in for another attribute (like socioeconomic status) and asking about that attribute directly instead.
Birds of a Feather Video-Flock Together: Design and Evaluation of an Agency-Based Parrot-to-Parrot Video-Calling System for Interspecies Ethical Enrichment
or “If you give a bird a Zoom call”
CHI “Honorable Mention” winner
Rébecca Kleinberger (+CAMD), Jennifer Cunha, Megha M. Vemuri, Ilyena Hirskyj-Douglas
Parrots have earned their reputation as the brainiacs of the aviary world, but such brainpower entails a tricky reality for the millions kept as pets. They need social and cognitive stimulation like their counterparts in the wild, but seldom get enough of it in captivity.
Seeking to rectify that imbalance, a new study first-authored by Khoury College–CAMD professor Rébecca Kleinberger aimed to provide the birds with the same social solution many humans jumped to in March 2020: video calling. In a three-month study involving 18 domesticated parrots, the team facilitated parrot-to-parrot calls using Facebook Messenger, all while monitoring the birds’ agency and motivation in initiating the calls and their engagement once the call began.
“Once the parrots learned to trigger the calls, it was hard to stop them, and some started exploring their tablets a bit too much,” Kleinberger recalled. “It was particularly challenging, but also truly rewarding, to involve so many animal participants for such a long period. To this day, I still sometimes receive calls from some of our bird participants.”
All 18 birds participated, most of them showing high motivation and intent while on the video call. Additionally, their human caretakers reported that not only did some of the birds learn new behaviors from their remote buddies — including flying and foraging — but the experience also helped the humans connect with their pets and learn more about them.
Through Their Eyes and In Their Shoes: Providing Group Awareness During Collaboration Across Virtual Reality and Desktop Platforms
or “How to work together when your work looks very different”
David Saffo, Andrea Batch, Cody Dunne, Niklas Elmqvist
To collaborate on a data exploration project, coworkers need to share an understanding of the parameters of their collaboration, which is difficult when one partner has a whole extra dimension to play in. These researchers used a tool they developed called VRxD to help a person on a desktop computer and a person using a VR headset to explore a data set about baseball pitchers; the researchers studied how they worked together and developed design recommendations for cross-platform collaborative tools.
“One of the potential benefits of AR and VR devices is the unique ways they can enable multi-user experiences and collaboration. However, I do not like that these experiences require all users to also use AR and VR devices,” lead researcher and doctoral student David Saffo said. “What excited me was thinking of all the design possibilities of a cross-platform system.”
By varying what features were available to the study’s six pairs of participants, the researchers determined that certain features — like the ability to quickly toggle into a collaborator’s view or leave the collaborator’s view open in another window — influenced how much time the pairs spent discussing their work versus working on the task itself, and who tended to lead a given activity.
Based on their findings, the researchers developed four design recommendations, which they call “through their eyes and in their shoes,” emphasizing how important it is for systems to allow collaborators to understand one another’s perspectives. Per the recommendations, VR–desktop collaboration systems should:
1. build in shared points of reference
2. encourage leadership by the user whose interface is best suited to the task
3. incorporate information cues (such as causing data to light up in both views when either participant clicks on it)
4. build in multiple ways to share perspectives so collaborators can quickly check on each other, present to one another, or sustain longer periods of collaboration
Auditory Seasoning Filters: Altering Food Perception via Augmented Sonic Feedback of Chewing Sounds
or “What can Pringles and sound effects teach us about our eating impulses?”
Rébecca Kleinberger (+CAMD), Akito Oshiro van Troyer, Qian Janice Wang
Aside from their snackability, Pringle chips are uniquely useful experimental stimuli for food-tech research, as their uniform size, texture, and flavor allows researchers to create consistent food samples and investigate how environmental conditions affect the eating experience. In a new study co-led by Rébecca Kleinberger, the team gave Pringles to study subjects while modulating the sound of their chewing using precise volume tweaks, delays, reverbs, and frequency filters. Then they determined how that modulation changed the subjects’ perceptions of both the food and their own hunger.
Among a slew of findings, the team discovered that amplified chewing sounds made participants feel greater sensations of satiety, texture, and flavor. Delay slowed eating speed. Large-room reverb diminished sourness and cravings. Filtering of high and low frequencies each bolstered fullness.
“This technology is quite fun and it helps you understand what’s going on in your brain, but is there a way to apply it and make an impact, especially regarding foods that aren’t especially healthy for you?” Kleinberger mused. “Can we make you eat slower? Can we make you chew more? Can we make you perceive more flavor and saltiness even if there’s actually less, so you’d eat food with less salt and artificial flavor?”
To that end, the team believes that future efforts can package these audio filters into an easy-to-use mobile app that helps users to regulate their food perception and cravings in real time — all without the need for the external hardware, explicit self-regulation, or specially prepared foods involved in previous approaches. But apart from the intricacies of the sound effects and neurology at play, Kleinberger believes there’s an underlying question of perspective.
“All of our participants told us they’d never had an experience like this, that it was really fun and transformative,” Kleinberger said. “Even just the notion of gaining awareness of your chewing, of becoming more attuned to it and having this stronger experience of sound — is interesting in itself. It can make people eat more mindfully.”
Model Sketching: Centering Concepts in Early-Stage Machine Learning Model Design
or “Let’s give machine learning designers a sketchbook”
Michelle S. Lam, Zixian Ma, Anne Li, Izequiel Freitas, Dakuo Wang (+CAMD), James A. Landay, Michael S. Bernstein
When designing a machine learning (ML) model, it’s easy to get stuck on the details of how to execute a solution at the expense of thinking about what kinds of solutions to pursue in the first place. These researchers want to get ML designers focused on the big picture, so they created ModelSketchBook: a Python package for quickly “sketching” minimal models and testing different design choices.
ModelSketchBook uses zero-shot learning and pretrained models like GPT-3 to let users create concepts, link them logically, and see how a machine learning model interprets those concepts on real data. For example, someone trying to build a model to identify hateful memes might use ModelSketchBook to explore how ML interprets the interactions of concepts like racism, nudity, and violence, then iterate several sketches to refine their initial model.
The researchers had 17 ML practitioners sketch models that would determine whether memes from Meta AI’s Hateful Memes Challenge should be removed from social media. They found that ModelSketchBook helped the practitioners discover gaps in their models, come up with a greater diversity of concepts to include, and produce quality outcomes more quickly. Their data also suggests that iterating on sketches — which the method encourages — produces better-performing models.
Thought Bubbles: A Proxy into Players’ Mental Model Development
or “How can we think about how we think?”
Omid Mohaddesi (CoE), Noah Chicoine (CoE), Min Gong (CoE), Ozlem Ergun (CoE), Jacqueline Griffin (CoE), David Kaeli* (CoE), Stacy Marsella (+CoS), Casper Harteveld* (CAMD)
In a work honored at CHI last year, this research team designed a game. Players role-played as wholesalers attempting to source medication from factories and supply hospitals during a supply chain disruption. Some hoarded medication from the beginning, others reacted to the shortage with panic-buying, and still others followed their computer’s recommendations through the whole game.
“(Last year), we identified Hoarders, Reactors, and Followers using observable behavior, and now we attempted to investigate differences in the development of these groups’ mental models,” said lead author and doctoral student Omid Mohaddesi, referring to the representation of a real-world situation that someone holds in their head. “We can see different patterns in the mental model development of groups of people and how information sharing impacts their mental model development … all this insight can potentially help us make better decision-support tools.”
To get contextualized insights into how their subjects’ mental models developed over time, the research team added a recurring, open-ended reflection prompt called a “thought bubble” to the supply chain game. Through two related studies, they coded participants’ reflections based on whether they were perceiving elements of the environment (“the backlog has gone up”), comprehending the meaning of those elements (“this is bad for our business”), or projecting a future state (“we will have to buy more medicine”). Researchers could then spot patterns between the ways people modeled the world and the decisions they made when managing a supply chain.
The study found that distinctions in participants’ supply chain management decisions are the result of differences in the development of their mental models. For instance, hoarders are proactive because they struggle to make sense of the situation — leading to uncertainty about the future — while “followers” trust order suggestions because they interpret the situation positively. The study also presents thought bubbles as a method for further studying mental models.
Rapid Convergence: The Outcomes of Making PPE during a Healthcare Crisis
or “PPE innovation soared during the pandemic, but how did it hold up?”
Kelly Mack, Megan Hofmann, Udaya Lakshmi, Jerry Cao, Nayha Auradkar, Rosa I. Arriaga, Scott E. Hudson, Jennifer Mankoff
In 2018, when Megan Hofmann and her partners began researching “medical making” — the application of 3D-printing models to the medical device and healthcare fields — she estimates there were only a few thousand medical makers worldwide.
But with the onset of the COVID-19 pandemic in 2020, the once-niche field was upended overnight. Suddenly, there was an urgent need to develop personal protective equipment (PPE) to halt the virus’ spread, and the field ballooned to hundreds of thousands of researchers. Hofmann and the other veteran researchers had a unique vantage point to evaluate this rapid escalation of PPE and medical device innovation, having been well-versed in the space before the spotlight found it.
READ: Megan Hofmann wins SIGCHI award for disability-device dissertation
Now studying the surge’s impact two years later, Hofmann and her co-researchers evaluated 623 3D-printable designs of new PPE technology submitted to the National Institutes of Health during the pandemic’s early stages. With so much new work and so many new researchers emerging so rapidly, the team was searching for patterns and noteworthy trends.
The study reinforced that while new PPE technology was developed impressively fast, there remains a fundamental importance to documenting the research process and sharing it publicly, along with evaluating risk and passing standard safety benchmarks. Hofmann believes the future of medical making lies in developing technology to partially automate the adjustment and verification process of the designs, creating a more efficient and streamlined process.
Simphony: Enhancing Accessible Pattern Design Practices among Blind Weavers
or “Using computing to help blind weavers weave”
Maitraye Das (+CAMD), Darren Gergle, Anne Marie Piper
Maitraye Das was introduced to a group of blind and low-vision weavers in 2019 while she was a graduate student at Northwestern University. She has immersed herself in the community ever since, and this paper is the final phase of her years-long work to help the weavers to design patterns and produce fabrics, cloths, artistic tapestries, and much more — despite pattern generation being a highly visual form of work.
To accomplish this, Das and her fellow researchers built Simphony, a system which combines musical notes, spoken audio, and tactile feedback to help the weavers envision weave patterns using their non-visual senses.
“Sighted people understand what they are seeing as a whole on a pattern,” Das said. “Whereas blind and low-vision people who rely on either auditory or tactile feedback, they sequentially explore what they notice and then cognitively create an imagery of the pattern.”
For the weavers, the ability to craft real-life products is a source of immense pride and self-expression. It also provides income and social capital, as their work is often sold, donated, or gifted to friends and family. In the future, Das hopes this work can help researchers to broadly understand the relationship between audio and visual–spatial perception, and to extend this technology to blind communities that stretch well beyond weaving.
“That’s important, but…”: How Computer Science Researchers Anticipate Unintended Consequences of Their Research Innovations
Or “Are CS researchers accounting for the ethical ramifications of their work?”
Kimberly Do, Rock Yuren Pang, Jiachen Jiang, Katharina Reinecke
Look no further than the dozens of works presented at CHI itself, and it’s clear that the computer science research community is rapidly developing powerful new technology with capabilities and potential that even the creators don’t fully comprehend. Often lost in the all-encompassing innovation race are the unintended consequences of the work, such as biases against marginalized groups, potential misuses of the technology, or fundamental flaws in the research process.
Kimberly Do and her collaborators interviewed 20 researchers across various CS disciplines to identify how they consider these consequences. What they found was a prevailing disconnect; researchers said they wanted to consider the ethical ramifications of their work, but in practice were often unwilling to take the necessary actions to do so.
“The biggest reason is the infrastructural barriers,” Do said. “So many people have this ‘publish or perish’ mindset where thinking about unintended consequences of the research isn’t necessarily as important as just getting it out there. People mostly view [the ethical component] as an accessory to publishing more.”
Do hopes that with heightened awareness of these oversights and an emphasis on community participation and feedback, researchers will be better equipped to make good on their stated intent.
OPTIMISM: Enabling Collaborative Implementation of Domain-Specific Metaheuristic Optimization
or “Bringing programming capabilities to non-programmers”
Megan Hofmann, Nayha Auradkar, Jessica Birchfield, Jerry Cao, Autumn G. Hughes, Gene S-H Kim, Shriya Kurpad, Kathryn J. Lum, Kelly Mack, Anisha Nilakantan, Margaret Ellen Seehorn, Emily Warnock, Jennifer Mankoff, Scott E. Hudson
Optimization is a complex, rapidly developing area of computer science research which uses complex equations and algorithms to “optimize” solutions across a wide range of fields, including software, medical machinery, and physical tools. But although its diverse array of applications presents great opportunity, it comes with a catch-22: only experienced programmers can implement advanced optimization techniques, but they are not experts in the fields they’re optimizing for.
After years of research, Megan Hofmann and her collaborators have developed a cutting-edge solution: OPTIMISM, a toolkit which enables domain-specific experts to collaborate with programmers to build and optimize new designs.
“A disabled person can be an expert in their navigational practices, someone can be a medical expert, or anything else,” Hofmann said. “And we wanted to build tools that allow people to work at their specific point of expertise, at a more complex dimension. Because you want to use their expertise, but you still need them to be able to do some programming tasks or to work with programmers in an effective way.”
The research has already produced results. An ophthalmologist and computer programmer collaboratively developed a program that could select the proper cataract lenses for doctors, streamlining cataract surgery. Likewise, a community of blind people worked with programmers to create customized tactile maps.
Is “Categorical Imperative” Metaversal?: A Kantian Ethical Framework for Social Virtual Reality
or “Real-world VR needs real-world ethics”
Eyup Engin Kucuk, Caglar Yildirim
Today, virtual reality is used for everything from firefighter training regimens to immersive video games to “sitting courtside” at an NBA game from the comfort of your living room. While the technology is innovative and rapidly improving, modern applications of VR tech are largely relegated to specific niches and too expensive for most to afford.
But imagine a world where large swaths of our daily activities operate in VR. Friends don’t travel to see each other, but instead socialize virtually. Real life and the virtual world blur together. It sounds dystopian, but Eyup Engin Kucuk and Caglar Yildirim are among the many who believe the world could be headed there one day — and that it is crucial to establish a precedent of ethical conduct while VR technology is still in its relative infancy.
By combining Kucuk’s philosophy background with Yildirim’s CS background, the pair researched and developed an ethical framework for social morality based on Immanuel Kant’s 200-year-old Theory of Morality, still among the most frequently used philosophical models for explaining human morality.
“Just as these ethical standards are valid in the real world, we applied Kant’s theory to social VR and discussed how these ethical standards are also applicable to virtual worlds,” Kucuk said. “We cannot think ‘Oh, it’s just the game world so we can do whatever we want.’”
And while the intersection of ethics and technology will evolve as the tech does, by proving the merits of human-based philosophy to the VR world at such an early stage, the researchers hope they’ll pave the way for further work in the space and bring awareness to the R&D underpinning VR products.
Exploratory Thematic Analysis of Crowdsourced Photosensitivity Warnings
or “Making flash warnings shine”
Laura South, Caglar Yildirim, Amy Pavel, and Michelle Borkin
Warnings about the presence, frequency, and duration of flashing lights are an important tool to ensure films are accessible to people with photosensitive epilepsy, for whom the lights might trigger seizures. However, warnings often don’t exist, leading to awkward and dangerous scenes in movie theaters. Additionally, many existing warnings are incredibly broad, prompting epileptic people to skip a film entirely when only a small portion of it would be risky.
To improve the warnings, these researchers sought to pinpoint what makes a photosensitivity warning good in the first place. They analyzed 265 crowdsourced warnings from the online trigger warning forum DoesTheDogDie to discover what elements were included in warnings that epileptic people wrote for one another. They found that scene warnings like “there are flashing lights immediately after the main character goes into a school” were more common than timestamped warnings like “there are flashing lights at 53:36.” They also found that descriptors like duration and color were more common than descriptors of frequency or size, and that qualitative warnings outnumbered quantitative ones. The researchers say that future work could automate qualitative, conversational photosensitivity warnings.
Improving Multiparty Interactions with a Robot using Large Language Models
or “Using robots to moderate group meetings”
Prasanth Murali, Ian Steenstra, Hye Sun Yun, Ameneh Shamekhi, Timothy Bickmore
Think back to your last group meeting. Did one or two speakers dominate the conversation, while no one else could get a word in? Did a conflict arise, or did an unhelpful tangent derail the meeting? While a meeting moderator or point person can help to facilitate productive conversation, studies have shown that a social robot can effectively facilitate such meetings as well. However, training such a robot is challenging with so many voices in the room.
That’s where Prasanth Murali and his co-authors come in. With the help of recently launched ChatGPT and other open-source AI machines, they aimed to identify and differentiate between speakers, and interpret those conversations to provide productive feedback. For example, the human-like robot can provide feedback such as “Hiring manager one, it seems like you did not have a lot to say, would you like time now?” or “Hiring manager two, it seems like you said this, but others did not quite pick up on your points. Can you clarify this a bit more?”
The technology is still in the early development stage, and it remains to be seen exactly how it will be delivered to users. However, this research further shows the promise of language models to identify speakers, and Murali hopes researchers can continue building on the technology, ultimately building social robots that make meetings more efficient, effective, and equitable.
Understanding Dark Patterns in IoT Devices
or “How smart home devices are invading your privacy”
Monica Kowalczyk, Johanna T. Gunawan, David Choffnes, Daniel J. Dubois, Woodrow Hartzog (external affiliate at Northeastern’s Cybersecurity and Privacy Institute), Christo Wilson
Smart fridges. Smart lights. Smart thermostats. Smart speakers. Smart TVs. Smart doorbells. Smart security systems. All consumer devices which apply internet capabilities to traditionally “dumb” (or non-internet-enabled) everyday objects. Known as Internet of Things (IoT) devices, this evolving category interacts with the internet much like our phones or laptops, but helps to streamline daily needs through physical hardware.
But as IoT devices become more ingrained in our lives, concerns have mounted over their pervasive use of “dark patterns” — deceptive user interfaces that manipulate users into taking actions or sharing information they don’t intend to. Though dark patterns are a major issue across technology in general, their application within IoT devices is particularly concerning because of their unique ability to breach users’ privacy.
“[IoT devices] might be placed in intimate spaces such as bedrooms or bathrooms, have sensors to collect very specific data, or have hardware that is constantly recording and listening, giving them access to sensitive data not as easily accessible to websites and apps,” said Monica Kowalczyk, a fourth-year Khoury College undergraduate who worked with a team of researchers to undercover these patterns in widely used smart-home devices.
The researchers reset 57 devices to their bare-bones settings, then closely examined the devices, their features and settings, their account deletion processes, and their user data and legal terms. They found that speakers, doorbells, and camera devices contained the most dark patterns among IoT devices, and that large-scale manufacturers such as Amazon and Google generally had more dark patterns than other vendors.
What Do We Mean When We Talk about Trust in Social Media? A Systematic Review
or “How can we tell if we trust social media if we can’t agree on what trust is?”
CHI “Best Paper” winner
Yixuan Zhang, Joseph D. Gaggiano, Nutchanon Yongsatianchot, Nurul M. Suhaimi, Miso Kim (CAMD), Yifan Sun, Jacqueline Griffin (CoE), Andrea G. Parker
A lot of research has examined trust and social media in recent years. How is trust in social media defined and measured? What factors influence its development and its consequences? And do researchers define trust cohesively enough for these conclusions to aggregate meaningfully?
This research team reviewed 70 papers examining social media and trust concepts, and found that less than half of those papers defined trust at all. Those that did often understood it as a willingness to be vulnerable to another party, but who was being trusted — the platform, the information, or the other users — was inconsistently defined. That said, there were still meaningful conclusions to be drawn, for instance that a user’s location, education, income, and political ideology were better predictors of their trust in social media than their gender or age.
The researchers encourage future work to clearly specify the context and the trustee when defining trust, and for that definition to align with other works’ definitions. The researchers also advocate for further research into distrust and mistrust, and into the conditions that create or result from trust in social media.
Exploring the Use of Personalized AI for Identifying Misinformation on Social Media
or “Can we personalize AI content moderation without it personalizing us back?”
Farnaz Jahanbakhsh, Yannis Katsis, Dakuo Wang (+CAMD), Lucian Popa, Michael Muller
It’s legendarily difficult to deal with misinformation on social media. Users disagree about what constitutes misinformation, who gets to decide, whether automation should be used, and what happens to posts deemed inaccurate. But what if a personalized AI could learn what types of posts you specifically were likely to deem as misinformation? Would it help you avoid falling victim to untruths? Or would it influence what you believed to be true in the first place?
These researchers studied how users perceive a personalized AI that helps them spot content they would likely deem misinformation, and whether seeing the AI’s predictions impacted how a user assessed the content’s accuracy for themselves. The team had 50 participants rate the accuracy of tweets about COVID-19 and their confidence in their predictions with and without AI assistance, then solicited feedback about the users’ experience.
They found that when a user saw the AI’s prediction about whether the user would find a tweet accurate, the user was more likely to agree with the AI’s prediction. However, asking the user why they believed a tweet was true or untrue eliminated this bias. The participants varied widely when it came to how they perceived the AI’s usefulness; some believed it would worsen political bubbles, while others liked how it guided them to inspect content more closely.
“Everyone is Covered”: Exploring the Role of Online Interactions in Facilitating Connection and Social Support in Black Churches
or “What do Black churches need to bring faith online?”
Darley Sackitey, Teresa K. O’Leary, Michael Paasche-Orlow, Timothy Bickmore, Andrea G. Parker
Thanks to their unique history navigating American colonialism, institutional neglect, and white supremacy, Black churches have become critical community gathering places and centers of social support. By extending their reach online, they could make the health benefits of faith and community more available to more people. A poorly designed tool could do more harm than good though, so as part of an ongoing collaboration with two Black churches in Boston, these researchers studied the features and design considerations most critical to building effective spiritual technology for Black communities.
These researchers worked with nine church members in leadership roles through interviews, focus groups, and a trial technical probe called Church Connect Community. To understand which features were important in faith-related technology, they discussed how the church members used the platform, how they wanted to use the platform, and what their concerns about the platform were. The research team identified several important design recommendations, including building leadership opportunities, staying conscious of inclusivity, integrating with offline church programs, prioritizing the spiritual over the material, and offering multiple types of spiritual support, from one-on-ones to larger community groups.
The most prestigious human–computer interaction conference in the world is taking place in Hamburg, Germany this month, and Khoury College’s researchers — along with their collaborators in the College of Engineering (CoE) and the College of Arts, Media and Design (CAMD) — are ready. Their work, which touches on everything from parrot socialization to deceptive dark patterns to social media misinformation, has made Northeastern the seventh-ranked institution in the world in terms of CHI publications, as well as sixth in the United States and first in Massachusetts. For a full schedule of their papers, panels, and other appearances at the conference, click here.
Click on one of the linked summaries below to learn about the corresponding paper, or simply read on.
Bold = Khoury College
() = another Northeastern college
(+) = interdisciplinary appointee
* = Khoury College courtesy appointee
Corsetto: A Kinesthetic Garment for Designing, Composing for, and Experiencing an Intersubjective Haptic Voice
or “How to feel music … literally”
Ozgun Kilic Afsar, Yoav Luft, Kelsey Cotton, Ekaterina R. Stepanova, Claudia Núñez-Pacheco, Rébecca Kleinberger (+CAMD), Fehmi Ben Abdesslem, Hiroshi Ishii, Kristina Höök
There is nothing quite like an immersive live music performance. When the composition meets the performance, and when the audience is fully engaged, the building feels like it’s lifting off its studs. The experience is not just visceral; it’s physical.
While the ambience of live music can naturally elicit this sensation, a team of researchers hailing from Sweden, the US, and Canada looked to deepen it. Their creation is Corsetto, a robotic upper-body garment which uses haptic gestures to transfer the physicality of a singer’s performance to the audience members wearing the device.
“This project explored the strange question of what it feels like to wear someone else’s voice,” said Rébecca Kleinberger, a professor jointly appointed between Khoury College and CAMD. “This collaboration was truly multidisciplinary, with elements of art, design, fabrication, software, HCI, experience design, and philosophy.”
The team created Corsetto with the aid of a classically trained opera singer who, through sessions of vocal exercises, helped the team to understand the biomechanical elements of singing. Once the team had incorporated those lessons into a functional device, they invited audiences to wear it during a performance of Morton Feldman’s piece “Three Voices,” then interviewed them afterwards. The result, they say, was an “intersubjective haptic voice,” a connective tissue between the auditory and physical elements of the performance that can help to convey the gravity and meaning of the music.
“[Corsetto] blurred boundaries between inner and outer aspects of participants’ bodies,” the researchers wrote, “at times rendering a fusion of all the bodies in the room into one holistic experience.”
“Who is the right homeless client?”: Values in Algorithmic Homelessness Service Provision and Machine Learning Research
or “Fair and ethical research practices for homeless populations”
Dilruba Showkat, Angela D. R. Smith, Wang Lingqing, Alexandra To (+CAMD)
As emphasis on ethical and equitable research practices has increased in recent years, so too have new innovations and diverse perspectives across the field of CS. But homeless populations have remained marginalized, and according to Khoury College doctoral student Dilruba Showkat, the field’s failure to produce ethical, balanced research of homeless communities isn’t due to a lack of intent or effort. Rather, it’s because homeless people are often difficult to access and don’t always cooperate with researchers, so their point of view often gets left out of the picture.
“Homeless populations’ values are well known in HCI research,” Showkat said. “However, during my analysis, I found a disconnect between these marginalized communities’ values and the machine learning systems that are developed to support them.”
Among the issues Showkat and her collaborators found were breaches of homeless peoples’ privacy and autonomy, as well as their lack of technological savvy and ability to self-advocate — each of which made them susceptible to being nullified or misrepresented by the predictors, categories, and algorithms used in machine learning technology.
“In several research papers, I’ve found social security numbers of homeless people, as well as other private information, unintentionally passed when sharing data sets,” Showkat said. “This is something really wrong with the system, and we should change that by conducting research that is held accountable. ”
The paper argues that researchers must actively protect the privacy and autonomy of homeless people, since they often cannot voice their concerns themselves. One proposed solution is to design machine learning tools using novel experimental techniques that don’t rely on vulnerable communities’ data, and instead prioritize human values. Another recommendation is to develop reproducible and replicable models by sharing a chunk of data anonymously, which would enable researchers to use the information without violating privacy.
Why, When, and From Whom: Considerations for Collecting and Reporting Race and Ethnicity Data in HCI
or “How should we ask about race?”
CHI “Honorable Mention” winner
Yiqun T. Chen, Angela D.R. Smith, Katharina Reinecke, Alexandra To (+CAMD)
Last year, Alexandra To and her fellow researchers reviewed six years of CHI proceedings and determined that just three percent of research papers on human–computer interaction collect race or ethnicity data. This year, they interviewed 15 authors to better understand how they’re collecting race and ethnicity data now, and to begin to develop best practices for collection in the future.
This research team doesn’t recommend collecting race and ethnicity data for every study, as privacy concerns, the risk of further entrenching “scientific racism”— the pseudoscientific belief that there is empirical evidence justifying racism — and participant retention could all be legitimate reasons not to. Instead, they encourage researchers to make such data collection a deliberate, rather than a default, decision.
If a study does collect racial data, the researchers recommend allowing participants to select multiple racial categories, including an open-ended way to self-identify, using more granular categories than the US Census default categories if appropriate, and considering where race is acting as a stand-in for another attribute (like socioeconomic status) and asking about that attribute directly instead.
Birds of a Feather Video-Flock Together: Design and Evaluation of an Agency-Based Parrot-to-Parrot Video-Calling System for Interspecies Ethical Enrichment
or “If you give a bird a Zoom call”
CHI “Honorable Mention” winner
Rébecca Kleinberger (+CAMD), Jennifer Cunha, Megha M. Vemuri, Ilyena Hirskyj-Douglas
Parrots have earned their reputation as the brainiacs of the aviary world, but such brainpower entails a tricky reality for the millions kept as pets. They need social and cognitive stimulation like their counterparts in the wild, but seldom get enough of it in captivity.
Seeking to rectify that imbalance, a new study first-authored by Khoury College–CAMD professor Rébecca Kleinberger aimed to provide the birds with the same social solution many humans jumped to in March 2020: video calling. In a three-month study involving 18 domesticated parrots, the team facilitated parrot-to-parrot calls using Facebook Messenger, all while monitoring the birds’ agency and motivation in initiating the calls and their engagement once the call began.
“Once the parrots learned to trigger the calls, it was hard to stop them, and some started exploring their tablets a bit too much,” Kleinberger recalled. “It was particularly challenging, but also truly rewarding, to involve so many animal participants for such a long period. To this day, I still sometimes receive calls from some of our bird participants.”
All 18 birds participated, most of them showing high motivation and intent while on the video call. Additionally, their human caretakers reported that not only did some of the birds learn new behaviors from their remote buddies — including flying and foraging — but the experience also helped the humans connect with their pets and learn more about them.
Through Their Eyes and In Their Shoes: Providing Group Awareness During Collaboration Across Virtual Reality and Desktop Platforms
or “How to work together when your work looks very different”
David Saffo, Andrea Batch, Cody Dunne, Niklas Elmqvist
To collaborate on a data exploration project, coworkers need to share an understanding of the parameters of their collaboration, which is difficult when one partner has a whole extra dimension to play in. These researchers used a tool they developed called VRxD to help a person on a desktop computer and a person using a VR headset to explore a data set about baseball pitchers; the researchers studied how they worked together and developed design recommendations for cross-platform collaborative tools.
“One of the potential benefits of AR and VR devices is the unique ways they can enable multi-user experiences and collaboration. However, I do not like that these experiences require all users to also use AR and VR devices,” lead researcher and doctoral student David Saffo said. “What excited me was thinking of all the design possibilities of a cross-platform system.”
By varying what features were available to the study’s six pairs of participants, the researchers determined that certain features — like the ability to quickly toggle into a collaborator’s view or leave the collaborator’s view open in another window — influenced how much time the pairs spent discussing their work versus working on the task itself, and who tended to lead a given activity.
Based on their findings, the researchers developed four design recommendations, which they call “through their eyes and in their shoes,” emphasizing how important it is for systems to allow collaborators to understand one another’s perspectives. Per the recommendations, VR–desktop collaboration systems should:
1. build in shared points of reference
2. encourage leadership by the user whose interface is best suited to the task
3. incorporate information cues (such as causing data to light up in both views when either participant clicks on it)
4. build in multiple ways to share perspectives so collaborators can quickly check on each other, present to one another, or sustain longer periods of collaboration
Auditory Seasoning Filters: Altering Food Perception via Augmented Sonic Feedback of Chewing Sounds
or “What can Pringles and sound effects teach us about our eating impulses?”
Rébecca Kleinberger (+CAMD), Akito Oshiro van Troyer, Qian Janice Wang
Aside from their snackability, Pringle chips are uniquely useful experimental stimuli for food-tech research, as their uniform size, texture, and flavor allows researchers to create consistent food samples and investigate how environmental conditions affect the eating experience. In a new study co-led by Rébecca Kleinberger, the team gave Pringles to study subjects while modulating the sound of their chewing using precise volume tweaks, delays, reverbs, and frequency filters. Then they determined how that modulation changed the subjects’ perceptions of both the food and their own hunger.
Among a slew of findings, the team discovered that amplified chewing sounds made participants feel greater sensations of satiety, texture, and flavor. Delay slowed eating speed. Large-room reverb diminished sourness and cravings. Filtering of high and low frequencies each bolstered fullness.
“This technology is quite fun and it helps you understand what’s going on in your brain, but is there a way to apply it and make an impact, especially regarding foods that aren’t especially healthy for you?” Kleinberger mused. “Can we make you eat slower? Can we make you chew more? Can we make you perceive more flavor and saltiness even if there’s actually less, so you’d eat food with less salt and artificial flavor?”
To that end, the team believes that future efforts can package these audio filters into an easy-to-use mobile app that helps users to regulate their food perception and cravings in real time — all without the need for the external hardware, explicit self-regulation, or specially prepared foods involved in previous approaches. But apart from the intricacies of the sound effects and neurology at play, Kleinberger believes there’s an underlying question of perspective.
“All of our participants told us they’d never had an experience like this, that it was really fun and transformative,” Kleinberger said. “Even just the notion of gaining awareness of your chewing, of becoming more attuned to it and having this stronger experience of sound — is interesting in itself. It can make people eat more mindfully.”
Model Sketching: Centering Concepts in Early-Stage Machine Learning Model Design
or “Let’s give machine learning designers a sketchbook”
Michelle S. Lam, Zixian Ma, Anne Li, Izequiel Freitas, Dakuo Wang (+CAMD), James A. Landay, Michael S. Bernstein
When designing a machine learning (ML) model, it’s easy to get stuck on the details of how to execute a solution at the expense of thinking about what kinds of solutions to pursue in the first place. These researchers want to get ML designers focused on the big picture, so they created ModelSketchBook: a Python package for quickly “sketching” minimal models and testing different design choices.
ModelSketchBook uses zero-shot learning and pretrained models like GPT-3 to let users create concepts, link them logically, and see how a machine learning model interprets those concepts on real data. For example, someone trying to build a model to identify hateful memes might use ModelSketchBook to explore how ML interprets the interactions of concepts like racism, nudity, and violence, then iterate several sketches to refine their initial model.
The researchers had 17 ML practitioners sketch models that would determine whether memes from Meta AI’s Hateful Memes Challenge should be removed from social media. They found that ModelSketchBook helped the practitioners discover gaps in their models, come up with a greater diversity of concepts to include, and produce quality outcomes more quickly. Their data also suggests that iterating on sketches — which the method encourages — produces better-performing models.
Thought Bubbles: A Proxy into Players’ Mental Model Development
or “How can we think about how we think?”
Omid Mohaddesi (CoE), Noah Chicoine (CoE), Min Gong (CoE), Ozlem Ergun (CoE), Jacqueline Griffin (CoE), David Kaeli* (CoE), Stacy Marsella (+CoS), Casper Harteveld* (CAMD)
In a work honored at CHI last year, this research team designed a game. Players role-played as wholesalers attempting to source medication from factories and supply hospitals during a supply chain disruption. Some hoarded medication from the beginning, others reacted to the shortage with panic-buying, and still others followed their computer’s recommendations through the whole game.
“(Last year), we identified Hoarders, Reactors, and Followers using observable behavior, and now we attempted to investigate differences in the development of these groups’ mental models,” said lead author and doctoral student Omid Mohaddesi, referring to the representation of a real-world situation that someone holds in their head. “We can see different patterns in the mental model development of groups of people and how information sharing impacts their mental model development … all this insight can potentially help us make better decision-support tools.”
To get contextualized insights into how their subjects’ mental models developed over time, the research team added a recurring, open-ended reflection prompt called a “thought bubble” to the supply chain game. Through two related studies, they coded participants’ reflections based on whether they were perceiving elements of the environment (“the backlog has gone up”), comprehending the meaning of those elements (“this is bad for our business”), or projecting a future state (“we will have to buy more medicine”). Researchers could then spot patterns between the ways people modeled the world and the decisions they made when managing a supply chain.
The study found that distinctions in participants’ supply chain management decisions are the result of differences in the development of their mental models. For instance, hoarders are proactive because they struggle to make sense of the situation — leading to uncertainty about the future — while “followers” trust order suggestions because they interpret the situation positively. The study also presents thought bubbles as a method for further studying mental models.
Rapid Convergence: The Outcomes of Making PPE during a Healthcare Crisis
or “PPE innovation soared during the pandemic, but how did it hold up?”
Kelly Mack, Megan Hofmann, Udaya Lakshmi, Jerry Cao, Nayha Auradkar, Rosa I. Arriaga, Scott E. Hudson, Jennifer Mankoff
In 2018, when Megan Hofmann and her partners began researching “medical making” — the application of 3D-printing models to the medical device and healthcare fields — she estimates there were only a few thousand medical makers worldwide.
But with the onset of the COVID-19 pandemic in 2020, the once-niche field was upended overnight. Suddenly, there was an urgent need to develop personal protective equipment (PPE) to halt the virus’ spread, and the field ballooned to hundreds of thousands of researchers. Hofmann and the other veteran researchers had a unique vantage point to evaluate this rapid escalation of PPE and medical device innovation, having been well-versed in the space before the spotlight found it.
READ: Megan Hofmann wins SIGCHI award for disability-device dissertation
Now studying the surge’s impact two years later, Hofmann and her co-researchers evaluated 623 3D-printable designs of new PPE technology submitted to the National Institutes of Health during the pandemic’s early stages. With so much new work and so many new researchers emerging so rapidly, the team was searching for patterns and noteworthy trends.
The study reinforced that while new PPE technology was developed impressively fast, there remains a fundamental importance to documenting the research process and sharing it publicly, along with evaluating risk and passing standard safety benchmarks. Hofmann believes the future of medical making lies in developing technology to partially automate the adjustment and verification process of the designs, creating a more efficient and streamlined process.
Simphony: Enhancing Accessible Pattern Design Practices among Blind Weavers
or “Using computing to help blind weavers weave”
Maitraye Das (+CAMD), Darren Gergle, Anne Marie Piper
Maitraye Das was introduced to a group of blind and low-vision weavers in 2019 while she was a graduate student at Northwestern University. She has immersed herself in the community ever since, and this paper is the final phase of her years-long work to help the weavers to design patterns and produce fabrics, cloths, artistic tapestries, and much more — despite pattern generation being a highly visual form of work.
To accomplish this, Das and her fellow researchers built Simphony, a system which combines musical notes, spoken audio, and tactile feedback to help the weavers envision weave patterns using their non-visual senses.
“Sighted people understand what they are seeing as a whole on a pattern,” Das said. “Whereas blind and low-vision people who rely on either auditory or tactile feedback, they sequentially explore what they notice and then cognitively create an imagery of the pattern.”
For the weavers, the ability to craft real-life products is a source of immense pride and self-expression. It also provides income and social capital, as their work is often sold, donated, or gifted to friends and family. In the future, Das hopes this work can help researchers to broadly understand the relationship between audio and visual–spatial perception, and to extend this technology to blind communities that stretch well beyond weaving.
“That’s important, but…”: How Computer Science Researchers Anticipate Unintended Consequences of Their Research Innovations
Or “Are CS researchers accounting for the ethical ramifications of their work?”
Kimberly Do, Rock Yuren Pang, Jiachen Jiang, Katharina Reinecke
Look no further than the dozens of works presented at CHI itself, and it’s clear that the computer science research community is rapidly developing powerful new technology with capabilities and potential that even the creators don’t fully comprehend. Often lost in the all-encompassing innovation race are the unintended consequences of the work, such as biases against marginalized groups, potential misuses of the technology, or fundamental flaws in the research process.
Kimberly Do and her collaborators interviewed 20 researchers across various CS disciplines to identify how they consider these consequences. What they found was a prevailing disconnect; researchers said they wanted to consider the ethical ramifications of their work, but in practice were often unwilling to take the necessary actions to do so.
“The biggest reason is the infrastructural barriers,” Do said. “So many people have this ‘publish or perish’ mindset where thinking about unintended consequences of the research isn’t necessarily as important as just getting it out there. People mostly view [the ethical component] as an accessory to publishing more.”
Do hopes that with heightened awareness of these oversights and an emphasis on community participation and feedback, researchers will be better equipped to make good on their stated intent.
OPTIMISM: Enabling Collaborative Implementation of Domain-Specific Metaheuristic Optimization
or “Bringing programming capabilities to non-programmers”
Megan Hofmann, Nayha Auradkar, Jessica Birchfield, Jerry Cao, Autumn G. Hughes, Gene S-H Kim, Shriya Kurpad, Kathryn J. Lum, Kelly Mack, Anisha Nilakantan, Margaret Ellen Seehorn, Emily Warnock, Jennifer Mankoff, Scott E. Hudson
Optimization is a complex, rapidly developing area of computer science research which uses complex equations and algorithms to “optimize” solutions across a wide range of fields, including software, medical machinery, and physical tools. But although its diverse array of applications presents great opportunity, it comes with a catch-22: only experienced programmers can implement advanced optimization techniques, but they are not experts in the fields they’re optimizing for.
After years of research, Megan Hofmann and her collaborators have developed a cutting-edge solution: OPTIMISM, a toolkit which enables domain-specific experts to collaborate with programmers to build and optimize new designs.
“A disabled person can be an expert in their navigational practices, someone can be a medical expert, or anything else,” Hofmann said. “And we wanted to build tools that allow people to work at their specific point of expertise, at a more complex dimension. Because you want to use their expertise, but you still need them to be able to do some programming tasks or to work with programmers in an effective way.”
The research has already produced results. An ophthalmologist and computer programmer collaboratively developed a program that could select the proper cataract lenses for doctors, streamlining cataract surgery. Likewise, a community of blind people worked with programmers to create customized tactile maps.
Is “Categorical Imperative” Metaversal?: A Kantian Ethical Framework for Social Virtual Reality
or “Real-world VR needs real-world ethics”
Eyup Engin Kucuk, Caglar Yildirim
Today, virtual reality is used for everything from firefighter training regimens to immersive video games to “sitting courtside” at an NBA game from the comfort of your living room. While the technology is innovative and rapidly improving, modern applications of VR tech are largely relegated to specific niches and too expensive for most to afford.
But imagine a world where large swaths of our daily activities operate in VR. Friends don’t travel to see each other, but instead socialize virtually. Real life and the virtual world blur together. It sounds dystopian, but Eyup Engin Kucuk and Caglar Yildirim are among the many who believe the world could be headed there one day — and that it is crucial to establish a precedent of ethical conduct while VR technology is still in its relative infancy.
By combining Kucuk’s philosophy background with Yildirim’s CS background, the pair researched and developed an ethical framework for social morality based on Immanuel Kant’s 200-year-old Theory of Morality, still among the most frequently used philosophical models for explaining human morality.
“Just as these ethical standards are valid in the real world, we applied Kant’s theory to social VR and discussed how these ethical standards are also applicable to virtual worlds,” Kucuk said. “We cannot think ‘Oh, it’s just the game world so we can do whatever we want.’”
And while the intersection of ethics and technology will evolve as the tech does, by proving the merits of human-based philosophy to the VR world at such an early stage, the researchers hope they’ll pave the way for further work in the space and bring awareness to the R&D underpinning VR products.
Exploratory Thematic Analysis of Crowdsourced Photosensitivity Warnings
or “Making flash warnings shine”
Laura South, Caglar Yildirim, Amy Pavel, and Michelle Borkin
Warnings about the presence, frequency, and duration of flashing lights are an important tool to ensure films are accessible to people with photosensitive epilepsy, for whom the lights might trigger seizures. However, warnings often don’t exist, leading to awkward and dangerous scenes in movie theaters. Additionally, many existing warnings are incredibly broad, prompting epileptic people to skip a film entirely when only a small portion of it would be risky.
To improve the warnings, these researchers sought to pinpoint what makes a photosensitivity warning good in the first place. They analyzed 265 crowdsourced warnings from the online trigger warning forum DoesTheDogDie to discover what elements were included in warnings that epileptic people wrote for one another. They found that scene warnings like “there are flashing lights immediately after the main character goes into a school” were more common than timestamped warnings like “there are flashing lights at 53:36.” They also found that descriptors like duration and color were more common than descriptors of frequency or size, and that qualitative warnings outnumbered quantitative ones. The researchers say that future work could automate qualitative, conversational photosensitivity warnings.
Improving Multiparty Interactions with a Robot using Large Language Models
or “Using robots to moderate group meetings”
Prasanth Murali, Ian Steenstra, Hye Sun Yun, Ameneh Shamekhi, Timothy Bickmore
Think back to your last group meeting. Did one or two speakers dominate the conversation, while no one else could get a word in? Did a conflict arise, or did an unhelpful tangent derail the meeting? While a meeting moderator or point person can help to facilitate productive conversation, studies have shown that a social robot can effectively facilitate such meetings as well. However, training such a robot is challenging with so many voices in the room.
That’s where Prasanth Murali and his co-authors come in. With the help of recently launched ChatGPT and other open-source AI machines, they aimed to identify and differentiate between speakers, and interpret those conversations to provide productive feedback. For example, the human-like robot can provide feedback such as “Hiring manager one, it seems like you did not have a lot to say, would you like time now?” or “Hiring manager two, it seems like you said this, but others did not quite pick up on your points. Can you clarify this a bit more?”
The technology is still in the early development stage, and it remains to be seen exactly how it will be delivered to users. However, this research further shows the promise of language models to identify speakers, and Murali hopes researchers can continue building on the technology, ultimately building social robots that make meetings more efficient, effective, and equitable.
Understanding Dark Patterns in IoT Devices
or “How smart home devices are invading your privacy”
Monica Kowalczyk, Johanna T. Gunawan, David Choffnes, Daniel J. Dubois, Woodrow Hartzog (external affiliate at Northeastern’s Cybersecurity and Privacy Institute), Christo Wilson
Smart fridges. Smart lights. Smart thermostats. Smart speakers. Smart TVs. Smart doorbells. Smart security systems. All consumer devices which apply internet capabilities to traditionally “dumb” (or non-internet-enabled) everyday objects. Known as Internet of Things (IoT) devices, this evolving category interacts with the internet much like our phones or laptops, but helps to streamline daily needs through physical hardware.
But as IoT devices become more ingrained in our lives, concerns have mounted over their pervasive use of “dark patterns” — deceptive user interfaces that manipulate users into taking actions or sharing information they don’t intend to. Though dark patterns are a major issue across technology in general, their application within IoT devices is particularly concerning because of their unique ability to breach users’ privacy.
“[IoT devices] might be placed in intimate spaces such as bedrooms or bathrooms, have sensors to collect very specific data, or have hardware that is constantly recording and listening, giving them access to sensitive data not as easily accessible to websites and apps,” said Monica Kowalczyk, a fourth-year Khoury College undergraduate who worked with a team of researchers to undercover these patterns in widely used smart-home devices.
The researchers reset 57 devices to their bare-bones settings, then closely examined the devices, their features and settings, their account deletion processes, and their user data and legal terms. They found that speakers, doorbells, and camera devices contained the most dark patterns among IoT devices, and that large-scale manufacturers such as Amazon and Google generally had more dark patterns than other vendors.
What Do We Mean When We Talk about Trust in Social Media? A Systematic Review
or “How can we tell if we trust social media if we can’t agree on what trust is?”
CHI “Best Paper” winner
Yixuan Zhang, Joseph D. Gaggiano, Nutchanon Yongsatianchot, Nurul M. Suhaimi, Miso Kim (CAMD), Yifan Sun, Jacqueline Griffin (CoE), Andrea G. Parker
A lot of research has examined trust and social media in recent years. How is trust in social media defined and measured? What factors influence its development and its consequences? And do researchers define trust cohesively enough for these conclusions to aggregate meaningfully?
This research team reviewed 70 papers examining social media and trust concepts, and found that less than half of those papers defined trust at all. Those that did often understood it as a willingness to be vulnerable to another party, but who was being trusted — the platform, the information, or the other users — was inconsistently defined. That said, there were still meaningful conclusions to be drawn, for instance that a user’s location, education, income, and political ideology were better predictors of their trust in social media than their gender or age.
The researchers encourage future work to clearly specify the context and the trustee when defining trust, and for that definition to align with other works’ definitions. The researchers also advocate for further research into distrust and mistrust, and into the conditions that create or result from trust in social media.
Exploring the Use of Personalized AI for Identifying Misinformation on Social Media
or “Can we personalize AI content moderation without it personalizing us back?”
Farnaz Jahanbakhsh, Yannis Katsis, Dakuo Wang (+CAMD), Lucian Popa, Michael Muller
It’s legendarily difficult to deal with misinformation on social media. Users disagree about what constitutes misinformation, who gets to decide, whether automation should be used, and what happens to posts deemed inaccurate. But what if a personalized AI could learn what types of posts you specifically were likely to deem as misinformation? Would it help you avoid falling victim to untruths? Or would it influence what you believed to be true in the first place?
These researchers studied how users perceive a personalized AI that helps them spot content they would likely deem misinformation, and whether seeing the AI’s predictions impacted how a user assessed the content’s accuracy for themselves. The team had 50 participants rate the accuracy of tweets about COVID-19 and their confidence in their predictions with and without AI assistance, then solicited feedback about the users’ experience.
They found that when a user saw the AI’s prediction about whether the user would find a tweet accurate, the user was more likely to agree with the AI’s prediction. However, asking the user why they believed a tweet was true or untrue eliminated this bias. The participants varied widely when it came to how they perceived the AI’s usefulness; some believed it would worsen political bubbles, while others liked how it guided them to inspect content more closely.
“Everyone is Covered”: Exploring the Role of Online Interactions in Facilitating Connection and Social Support in Black Churches
or “What do Black churches need to bring faith online?”
Darley Sackitey, Teresa K. O’Leary, Michael Paasche-Orlow, Timothy Bickmore, Andrea G. Parker
Thanks to their unique history navigating American colonialism, institutional neglect, and white supremacy, Black churches have become critical community gathering places and centers of social support. By extending their reach online, they could make the health benefits of faith and community more available to more people. A poorly designed tool could do more harm than good though, so as part of an ongoing collaboration with two Black churches in Boston, these researchers studied the features and design considerations most critical to building effective spiritual technology for Black communities.
These researchers worked with nine church members in leadership roles through interviews, focus groups, and a trial technical probe called Church Connect Community. To understand which features were important in faith-related technology, they discussed how the church members used the platform, how they wanted to use the platform, and what their concerns about the platform were. The research team identified several important design recommendations, including building leadership opportunities, staying conscious of inclusivity, integrating with offline church programs, prioritizing the spiritual over the material, and offering multiple types of spiritual support, from one-on-ones to larger community groups.
Khoury Social
Contact Khoury
Khoury Social
Contact Khoury
Khoury Social
Contact Khoury
Khoury Social
Contact Khoury