Eliana Willis

The sentient Magic Carpet from “Aladdin” might have a new competitor. While it can’t fly or speak, a new tactile sensing carpet from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) can estimate human poses without using cameras, in a step toward improving self-powered personalized health care, smart homes, and gaming.

Many of our daily activities involve physical contact with the ground: walking, exercising, or resting. These embedded interactions contain a wealth of information that help us better understand people’s movements. 

Previous research has leveraged use of single RGB cameras, (think Microsoft Kinect), wearable omnidirectional cameras, and even plain old off-the-shelf webcams, but with the inevitable byproducts of camera occlusions and privacy concerns. 

The CSAIL team’s system only used cameras to create the dataset the system was trained on, and only captured the moment of the person performing the activity. To infer the 3D pose, a person would simply have to get on the carpet, perform an action, and then the team’s deep neural network, using just the tactile information, could determine if the person was doing situps, stretching, or doing another action. 

Play video

Intelligent Carpet: Estimating a person’s 3D pose using only tactile sensors

“You can imagine leveraging this model to enable a seamless health-monitoring system for high-risk individuals, for fall detection, rehab monitoring, mobility, and more,” says Yiyue Luo, a lead author on a paper about the carpet. 

The carpet itself, which is low-cost and scalable, was made of commercial, pressure-sensitive film and conductive thread, with over 9,000 sensors spanning 36-by-2 feet. (Most living room rug sizes are 8-by-10 and 9-by-12.) 

Each of the sensors on the carpet converts the human’s pressure into an electrical signal, through the physical contact between people’s feet, limbs, torso, and the carpet. The system was specifically trained on synchronized tactile and visual data, such as a video and corresponding heat map of someone doing a pushup. 

The model takes the pose extracted from the visual data as the ground truth, uses the tactile data as input, and finally outputs the 3D human pose.

This might look something like when, after stepping onto the carpet and doing a set up of pushups, the system is able to produce an image or video of someone doing a pushup. 

In fact, the model was able to predict a person’s pose with an error margin (measured by the distance between predicted human body key points and ground truth key points) by less than 10 centimeters. For classifying specific actions, the system was accurate 97 percent of the time. 

“You could envision using the carpet for workout purposes. Based solely on tactile information, it can recognize the activity, count the number of reps, and calculate the amount of burned calories,” says MIT CSAIL PhD student Yunzhu Li, a co-author on the paper.

Since much of the pressure distributions were prompted by movement of the lower body and torso, that information was more accurate than the upper-body data. Also, the model was unable to predict poses without more explicit floor contact, like free-floating legs during situps, or a twisted torso while standing up. 

While the system can understand a single person, the scientists, down the line, want to improve the metrics for multiple users, where two people might be dancing or hugging on the carpet. They also hope to gain more information from the tactical signals, such as a person’s height or weight. 

Luo wrote the paper alongside Li and MIT CSAIL PhD student Yunzhu Pratyusha Sharma, MIT CSAIL mechanical engineer Michael Foshey, MIT CSAIL postdoc Wan Shou, and MIT professors Tomas Palacios, Antonio Torralba, and Wojciech Matusik. The work is funded by the Toyota Research Institute.  More

The U.S. Department of Energy (DoE) recently announced the names of 83 scientists who have been selected for their 2021 Early Career Research Program. The list includes four faculty members from MIT: Riccardo Comin of the Department of Physics; Netta Engelhardt of the Department of Physics and Center for Theoretical Physics; Philip Harris of the Department of Physics and Laboratory for Nuclear Science; and Mingda Li of the Department of Nuclear Science and Engineering.

Each year, the DoE selects researchers for significant funding the “nation’s scientific workforce by providing support to exceptional researchers during crucial early career years, when many scientists do their most formative work.”

Resonant coherent diffractive imaging of quantum solids

The quantum technologies of tomorrow –– more powerful computing, better navigation systems, and more precise imaging and magnetic sensing devices –– rely on understanding the properties of quantum materials. Quantum materials contain unique physical characteristics, and can lead to phenomena like superconductivity. Detecting and visualizing these materials at the nanoscale will enable scientists to understand and harness the properties of quantum materials.

Riccardo Comin, the Class of 1947 Career Development Assistant Professor of Physics, leads the Comin Photon Scattering Lab at MIT. The group uses high-energy electromagnetic waves, or X-rays, to observe how new collective states emerge at the nanoscale in quantum materials. This is a difficult feat, as the lenses in cameras and in the human eye do not work for X-rays as they do for visible light. Conventional microscopy techniques are not well-suited for visualizing these complex phenomena.

To overcome this technical limitation, the Comin group has worked on a “lensless” X-ray microscopy approach to image these electronic textures.

“These new imaging techniques are really fascinating and deeply challenge our traditional ways of performing X-ray microscopy,” Comin says. “We now rely on special algorithms that can perform computationally the task of image reconstruction that is normally taken care of by a lens.”

The support from the DoE Early Career Research program will be instrumental to the group’s work developing and applying these novel techniques to study the nanoscale organization of quantum materials of interest. Looking beyond the horizon of quantum materials, the availability of lensless X-ray imaging methods provides a new powerful tool set for the characterization of catalysts, batteries, data storage devices, soft matter, and biological systems.

Spacetime emergence from quantum gravity

Few phenomena in modern physics remain as mysterious as the black hole interior. Black holes seem to wreck the objects that fall into them, as well as information about what those objects once were. Yet according to basic principles of quantum mechanics (the study of subatomic particle behavior), knowing the current state of a given system should mean knowing everything about its past and future.

General relativity and quantum mechanics are two highly tested theories. When it comes to black holes, general relativity and quantum mechanics disagree on a fundamental point: whether information about the region behind the event horizon can escape and be decoded by an observer outside of the black hole. The clash between general relativity and quantum mechanics on this matter results in what is termed the “black hole information paradox.” In recent years, scientists have drawn numerous connections between gravity and quantum information.

Netta Engelhardt, the Biedenharn Career Development Assistant Professor of physics and member of the Center for Theoretical Physics, researches quantum gravity and the black hole information paradox.

“With a recent leap in our understanding of the black hole information paradox, the connection between gravity, quantum computational complexity, and black holes has newfound potential to shed light on some of the most foundational questions about quantum gravity, starting with ‘What really happens inside a black hole?’” Engelhardt says.

With support from the DoE award, her project aims to move toward resolving the black hole information paradox using some of the novel tools and insights at the intersection of the two theories.

Harnessing the Large Hadron Collider with new insights in real-time data processing and artificial intelligence

Particle accelerators help scientists learn more about the particles that comprise matter.

Nearly 17 miles in circumference, the Large Hadron Collider (LHC) at the European Center for Nuclear Research is the largest and most powerful particle accelerator in the world, producing valuable information for researchers.

Researchers have spent much time using LHC data to investigate novel particle interactions at the highest energies. But over the next two decades, they anticipate shifting their focus, directing their efforts toward precision measurements that target physics processes with small interaction strengths and extensive background rates.

As a result of these more detailed observations, physicists expect additional rare and hidden processes within the Standard Model (SM) of particle physics, and potentially beyond the SM, to emerge as more data mounts.

Philip Harris, assistant professor of physics and researcher in the Laboratory for Nuclear Science, is working on a physics program to measure those smaller, more inconspicuous processes. Specifically, with the support of DoE funding, his research aims to exploit a new measurement technique he created to identify light resonances that decay into quarks –– the particles that eventually combine to create subatomic particles.

“In conjunction with advanced artificial intelligence algorithms, this new technique can open up a wealth of unique measurements and searches,” Harris says. “The fully developed state-of-the-art system will empower new measurements of the Higgs boson, new searches for dark matter, and analyses of a multitude of unexplored scientific phenomena.”

Machine learning-augmented multimodal neutron scattering for emergent topological materials

Topological materials are a class of quantum materials whose electronic properties have robust protection against outside influences. This robustness enables a wide range of promising applications, such as next-generation electronics without energy loss, and error-tolerant quantum computers.

But it’s difficult to directly test materials for their topological properties. Rather, scientists usually use methods that measure manifestations of topology. One such method is neutron scattering, or neutron spectroscopy, a process used by scientists to assess materials.

Neutron scattering has particular advantages when it comes to evaluating topological quantum materials, but more information is needed to understand exactly how massive amounts of data gathered during neutron spectroscopy map onto topology.

The DoE Early Career Research Program Award will support Mingda Li, the Norman C. Rasmussen Assistant Professor of Nuclear Science and Engineering, in his machine-learning approach to analyzing high-dimensional neutron scattering spectra in quantum materials.

“The new approach will augment existing neutron scattering probes by measuring things that were not measurable before,” Li says. By doing so, “it will enable a broader discovery of hidden materials states that may have electronics applications, and identify topological solutions that can be used for computer memory.”  More

Hovering 100 meters above a densely populated urban residential area, the drone takes a quiet breath. Its goal is singular: to systematically measure air quality across the metropolitan landscape, providing regular updates to a central communication module where it docks after its patrol, awaiting a new set of instructions. The central module integrates each new data point provided by a small drone fleet, processing them against wind and traffic patterns and historical pollution hot spot information. Then the fleet is assigned new sampling waypoints and relaunched.

This simulated autonomous air pollution monitoring system is the capstone project of recently graduated New Engineering Education Transformation (NEET) program alumni Chloe Nelson-Arzuaga ’21, Jeana Choi ’21, Daniel Gonzalez-Diaz ’21, Leilani Trautman ’21, Rima Rebei ’21, and Berke Saat ’21, who have together studied autonomous robotics since they entered NEET during their sophomore year at MIT.

“We all converged on this, because we felt that the social impact would be more powerful than any of the other projects we were looking at,” says Trautman, who majored in electrical engineering and computer science.

Their system represents a fundamentally different approach to air quality monitoring compared with the stationary systems routinely used in urban areas, which the group says often fail to detect spatial heterogeneity in pollution levels across a landscape. Given their limited distribution and lack of mobility, these systems are really only a reliable indicator of the air quality directly surrounding each monitoring point, but their data are reported as though they were representative of air quality across the entire city, say the recent graduates.

“So even though they might say that your air quality is somewhat good, that may not be the case for the park right next to your home,” says Gonzalez-Diaz.

The NEET cohort’s drone system is designed to provide real-time air quality data with a 15-meter resolution that is publicly accessible through a user-friendly interface.

The NEET program was launched in 2017 in an effort to fundamentally reconceptualize the way that engineering is taught at MIT. It emphasizes interdisciplinary scholarship, cross-departmental community, and project-based learning to prepare students to engage the major engineering challenges of the 21st century. The program is actively growing and is the fourth-largest undergraduate academic community, with more than 186 students participating. Twenty-three majors from 13 departments are represented. Sixty-four percent of the students are women and 28 percent are members of underrepresented groups. This year over 39% of first-years who applied to this program heard about it from current NEET students.

Nelson-Arzuaga, Choi, Gonzalez-Diaz, Trautman, Rebei, and Saat graduated from MIT with a certificate from NEET’s Autonomous Machines “thread,” or area of concentration. The NEET program has five threads that students can choose from, each emphasizing a class of contemporary or futuristic engineering problems. For example, the Advanced Materials thread explores the future of materials technologies and manufacturing, while the Digital Cities thread integrates computer science with urban planning to prepare students to build more idealized cities. NEET also offers a biotechnology-focused Living Machines thread as well as the Renewable Energy Machines thread, which emphasizes green energy systems design. The Autonomous Machines thread teaches students to design, build, and program autonomous robots. “A common feature across all five threads is that NEET students want to create an impact while they are still students,” says NEET Executive Director Babi Mitra, “by doing projects that tackle critical societal problems.”

The NEET curriculum structure is progressive, building on the previous year’s lessons to ultimately prepare the students for real-world application.

“[Sophomore year] we give them an individual project … and then, during junior year, they have their first small group project. And then the senior year, they have a class project. So it progressively gets more complicated,” says NEET Lead Instructor Greg Long. “This senior project is supposed to mimic something they would do if they were going to do a startup company.”

It was during the cohort’s junior year, when they were tasked with building autonomous vehicles that could race other vehicles and avoid obstacles, that the pandemic forced the closure of MIT’s campus and the NEET cohort was scattered across the globe.

“[The curriculum] is so hands-on and such a huge time commitment that we really thought the classes would end at that point when we had to leave campus,” says Saat. “But then they kept going. They set up the simulation for us and everything, but the time commitment was still there and we were in five different time zones.”

The NEET program demands roughly 20 hours per week on top of the rest of the students’ course load and the students say that meeting this demand with Choi in Cambridge, Trautman and Nelson-Arzuaga in California, Saat in Turkey, Rebei in Illinois, Gonzalez-Diaz in Puerto Rico, and another teammate in Taiwan required creative and exhausting schedule coordination.

“Literally [for] some of us the sun was rising and [for] the others the sun was going down [while working together],” says Choi. “We’ve definitely bonded and learned so much, and I think it would not have been possible if even one of us was not very interested in robotics.”

A unique feature of the NEET Autonomous Machines thread is that students take a senior fall semester class during which they discuss and decide the senior spring project they want to work on. It was also the pandemic that helped inform the group’s specific choice to tackle air pollution as a final project their senior year, because it further exposed “the racial and economic disparities that air pollution causes in the United States,” says Rebei.  

The cohort’s drone project is designed to monitor a form of pollution called PM 2.5, which are essentially pollution particles small enough to enter the bloodstream when inhaled, potentially resulting in lung and heart disease over time, according to the students.

“Low-income communities of color, at the end of the day, are the ones who are disproportionately impacted by air pollution, and … air pollution is what contributes to a lot of these deadly respiratory illnesses … in these particular communities,” says Rebei. “People who already have these preexisting conditions … are at higher risk of getting very sick from Covid-19.”

In addition to designing a drone system capable of effectively capturing neighborhood-level variation in pollution exposure levels, the NEET cohort created a web interface that could layer this information with area socioeconomic data, such as income, race, household composition, disability status, housing types, and modes of transportation. This would make patterns and disparities in air pollution exposure public and easier to prove, something the students say could help affected communities advocate for change.

The complexity of the drone project, remote learning notwithstanding, required the entire cohort to step out of their comfort zone and learn new skills quickly.

“I’m a mechanical design student, so I do a lot of 3D modeling,” says Nelson-Arzuaga. However, for the drone project she was in charge of learning how to operate a cellular communications network so that the drones would be able to talk to the central communication module. “[It was] very different from anything that I learned in all of my other design classes.”

The recent graduates say that NEET prepared them to engage these challenges, but expressed that the most valuable part of the program for them was the community that they built and the experiences they shared working together.

“When we took the sophomore robot class, we were all like, ‘What is a robot? How do we do anything?’” jokes Trautman. “[And now we’re] doing very high-quality development work on this project. I think it’s been exciting to see, to see where everyone is going in the future and just seeing how everyone’s progressed. It’s been a really cool journey.”

The NEET program is continuing to develop and fine-tune its curriculum, learning environment, and community, and student engagement is an integral part of that process. Spaces are still available in the NEET Class of 2024 cohort. For more information about applying visit the NEET website. More