by Julia Renner

A computer science and mathematics major, a guitar-playing graduate student, and a dual-major CCIS/CAMD student sounds like an unlikely combination, but the result? A sheet-music-reading, piano-playing robot.

Kevin Luo, computer science/mathematics ’18, developed the robot with collaborators across the College of Computer and Information Science (CCIS). He has held an interest in robotics since joining his elementary school Lego robotics team, and was involved in competitive robotics through high school. “I’ve been playing around with robotics-related applications while learning computer science since,” he says.

The project is based on enabling CCIS Professor Platt’s Baxter robot to play piano from sheet music. Luo and his collaborators taught the robot to use its camera to read a line of sheet music placed in front of it, then play that music on a keyboard.

Luo became acquainted with Platt while taking his Robotic Sciences and Systems class. The group he worked with had the goal of programming Platt’s Baxter robot to perform an everyday task for their final project in this course. They discussed ideas like a table-setting robot or a laundry-folding robot before developing the idea of a robot that could play piano from sheet music.

Like any complicated project, the robot presented challenges for the team to tackle. One of the most significant was getting the robot to consistently hit piano keys. “It’s a problem of integrating sensing with controls, which seems intuitive, but is actually pretty difficult,” Luo says, pointing out that “a lot of the things we do naturally and don’t think about, like ignoring a shadow that gets cast on a page that we’re reading, aren’t obvious to the robot and need to be programmed explicitly.”

“We thought of this mostly as an exercise in manipulation,” Luo explains, “since it basically boils down to a synthesizer. But I bet it would be really cool to have a live performance with a robot if it becomes capable of playing more complicated pieces of music.”

The project was a team effort between Luo, graduate student Sreejith Vu, a guitar player whose idea it was to have the robot play piano, and Jeff Weng, a dual major in the College of Arts, Media and Design who was responsible for creating a written report with graphics. “Overall it was a solid team,” Luo says, “and we all stayed pretty late a couple of nights to get the project completed.

“Professor Platt gave a lot of help with getting a general direction for the implementation. None of us had a lot of experience with robotic manipulation, so we had a lot of different control and planning algorithms we were considering. Professor Platt helped us decide on a few that would be best for this particular application.”

Luo, graduating this May, plans to stay in Boston and hopes to eventually work in robotics.