Artificial Intelligence

Isaac Newton may have met his match.

For centuries, engineers have relied on physical laws — developed by Newton and others — to understand the stresses and strains on the materials they work with. But solving those equations can be a computational slog, especially for complex materials.

MIT researchers have developed a technique to quickly determine certain properties of a material, like stress and strain, based on an image of the material showing its internal structure. The approach could one day eliminate the need for arduous physics-based calculations, instead relying on computer vision and machine learning to generate estimates in real time.

The researchers say the advance could enable faster design prototyping and material inspections. “It’s a brand new approach,” says Zhenze Yang, adding that the algorithm “completes the whole process without any domain knowledge of physics.”

The research appears today in the journal Science Advances. Yang is the paper’s lead author and a PhD student in the Department of Materials Science and Engineering. Co-authors include former MIT postdoc Chi-Hua Yu and Markus Buehler, the McAfee Professor of Engineering and the director of the Laboratory for Atomistic and Molecular Mechanics.

Engineers spend lots of time solving equations. They help reveal a material’s internal forces, like stress and strain, which can cause that material to deform or break. Such calculations might suggest how a proposed bridge would hold up amid heavy traffic loads or high winds. Unlike Sir Isaac, engineers today don’t need pen and paper for the task. “Many generations of mathematicians and engineers have written down these equations and then figured out how to solve them on computers,” says Buehler. “But it’s still a tough problem. It’s very expensive — it can take days, weeks, or even months to run some simulations. So, we thought: Let’s teach an AI to do this problem for you.”

The researchers turned to a machine learning technique called a Generative Adversarial Neural Network. They trained the network with thousands of paired images — one depicting a material’s internal microstructure subject to mechanical forces,  and the other depicting that same material’s color-coded stress and strain values. With these examples, the network uses principles of game theory to iteratively figure out the relationships between the geometry of a material and its resulting stresses.

“So, from a picture, the computer is able to predict all those forces: the deformations, the stresses, and so forth,” Buehler says. “That’s really the breakthrough — in the conventional way, you would need to code the equations and ask the computer to solve partial differential equations. We just go picture to picture.”

That image-based approach is especially advantageous for complex, composite materials. Forces on a material may operate differently at the atomic scale than at the macroscopic scale. “If you look at an airplane, you might have glue, a metal, and a polymer in between. So, you have all these different faces and different scales that determine the solution,” say Buehler. “If you go the hard way — the Newton way — you have to walk a huge detour to get to the answer.”

But the researcher’s network is adept at dealing with multiple scales. It processes information through a series of “convolutions,” which analyze the images at progressively larger scales. “That’s why these neural networks are a great fit for describing material properties,” says Buehler.

The fully trained network performed well in tests, successfully rendering stress and strain values given a series of close-up images of the microstructure of various soft composite materials. The network was even able to capture “singularities,” like cracks developing in a material. In these instances, forces and fields change rapidly across tiny distances. “As a material scientist, you would want to know if the model can recreate those singularities,” says Buehler. “And the answer is yes.”

The advance could “significantly reduce the iterations needed to design products,” according to Suvranu De, a mechanical engineer at Rensselaer Polytechnic Institute who was not involved in the research. “The end-to-end approach proposed in this paper will have a significant impact on a variety of engineering applications — from composites used in the automotive and aircraft industries to natural and engineered biomaterials. It will also have significant applications in the realm of pure scientific inquiry, as force plays a critical role in a surprisingly wide range of applications from micro/nanoelectronics to the migration and differentiation of cells.”

In addition to saving engineers time and money, the new technique could give nonexperts access to state-of-the-art materials calculations. Architects or product designers, for example, could test the viability of their ideas before passing the project along to an engineering team. “They can just draw their proposal and find out,” says Buehler. “That’s a big deal.”

Once trained, the network runs almost instantaneously on consumer-grade computer processors. That could enable mechanics and inspectors to diagnose potential problems with machinery simply by taking a picture.

In the new paper, the researchers worked primarily with composite materials that included both soft and brittle components in a variety of random geometrical arrangements. In future work, the team plans to use a wider range of material types. “I really think this method is going to have a huge impact,” says Buehler. “Empowering engineers with AI is really what we’re trying to do here.”

Funding for this research was provided, in part, by the Army Research Office and the Office of Naval Research. More

“MIT’s work to understand and improve human health spans decades and covers the Institute,” said W. Eric L. Grimson PhD ’80, at MIT Better World (Health), a virtual gathering in February. “More than a third of the faculty representing every department at MIT engage in research directly related to health science and innovation.” Grimson, who is MIT’s chancellor for academic advancement and the Bernard M. Gordon Professor of Medical Engineering, spoke of the many achievements of Institute scholars in the human health arena: “Serving as the hub of the densest innovation cluster in the world, MIT is nimble and inventive, particularly when it comes to the life sciences.”

MIT alumni and friends from around the globe were invited to attend the online event, which featured presentations from Institute leaders, faculty, and alumni about human health-related research at the Institute. With more than 1,000 participants from 27 countries, the evening began with video greetings from nearly a dozen alumni working in a range of health-care roles all over the world. Their graduation years spanned five decades, from 1967 to 2019.

Play video

Innovations in Human Health Main Session and Q&A

Grimson then turned the spotlight over to the presenting speakers: Daniel P. Huttenlocher SM ’84 PhD ’88, dean of the MIT Stephen A. Schwarzman College of Computing and Henry Ellis Warren (1894) Professor of Electrical Engineering and Computer Science; Mariana Arcaya MCP ’08, associate professor of urban planning and public health; and Steven Truong ’20, a Marshall Scholar studying computational biology at the University of Cambridge in England.

Huttenlocher spoke about the role of artificial intelligence in health research. Last year, he said, faculty at MIT’s Abdul Latif Jameel Clinic for Machine Learning in Health identified a new antibiotic candidate capable of killing drug-resistant bacteria. “In the search for new antibiotics, there are so many possibilities that it’s not practical to try even a small fraction of them,” he explained. “This is where machine learning comes in.”

He also discussed the Schwarzman College’s mission of educating “computing bilinguals” — “people [who] are equipped with knowledge about computing and AI in addition to their field of expertise” — and emphasized the need for experts in different disciplines to collaborate. “By truly integrating computing across MIT — that’s how we’ll make unparalleled leaps in making a better world.”

“The work we heard about tonight embodies the MIT commitment to curiosity and discovery in the pursuit of a better, healthier world.”

When the Covid-19 pandemic struck, according to Arcaya, “everyone could guess who would suffer first and most.” She explained that social epidemiologists have repeatedly demonstrated that socially vulnerable people face elevated disease risk. Through participatory action research in Massachusetts cities like Chelsea and Everett, Arcaya’s students learned that the high cost of Boston-area housing has forced many community members to live in overcrowded apartments or become transient, increasing their likelihood of exposure. Concluding that rapidly increasing home values in previously affordable neighborhoods also increased Covid-19 infection rates, Arcaya’s team made a compelling case for public policy that protects affordable housing. “Putting residents at the center of place-based research improves social science,” she said.

Truong offered a sobering statistic: People of Asian descent are three times more likely than their white counterparts to have undiagnosed diabetes, because they often lack the obesity commonly associated with the disease. “My dad was a perfect example of this,” he said. “Because he didn’t look like the ‘typical’ American with diabetes, the doctors didn’t test him for it. So he was diagnosed so late in his disease that his body had already been seriously damaged.” While his father’s death reinforced Truong’s determination to study the genetic basis of diabetes in Vietnamese people, he noted the limitations of large data resources such as the UK Biobank, which includes genetic information representative of the demographic breakdown of the UK as it currently is: 95 percent white. “I was able to kickstart something in Vietnam; hopefully, it not only sheds a little light onto these questions but also brings more awareness to this issue of representation in general,” he told the audience. “I hope you uplift those underrepresented in whatever fields you represent.”

“The work we heard about tonight,” remarked Grimson as the main program concluded, “embodies the MIT commitment to curiosity and discovery in the pursuit of a better, healthier world.” More

Strategic use of data is vital for progress in science, commerce, and even politics, but at the same time, citizens are demanding more responsible, respectful use of personal data. Internet users have never felt more helpless about how their data are being used: Surveys show that the vast majority of U.S. adults feel that they have little to no control over the data that the government and private companies collect about them. In response to these concerns, new privacy laws are being enacted in Europe, California, Virginia, and elsewhere around the world.

To conduct more-focused research and analysis of these issues, last week MIT launched a new initiative to bring state-of-the-art computer science research together with public policy expertise and engagement.

Launched on April 6, the MIT Future of Data, Trust, and Privacy initiative (FOD) will involve collaboration between experts specializing in five distinct technical areas:

database systems
applied cryptography
AI and machine learning
data portability and new information architectures; and
human-computer interaction.

In addition to technical research, FOD will provide forums for dialogue amongst MIT researchers, policymakers, and industry consortium members, with a structure similar to MIT’s 2019 AI Policy Congress, which included members of the Organization for Economic Cooperation and Development. 

Future of Data: Law, Technology and Policy

FOD is a collaboration between MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) and the MIT Internet Policy Research Initiative (IPRI). Co-director Daniel Weitzner is both a researcher at CSAIL and founding director of IPRI, and previously served as the White House deputy CTO under President Obama.

Weitzner says that one of the larger goals is to reduce the cycle time between the development of new policies and new software systems. He also hopes to work with industry to develop new privacy-preserving tools and to help steer conversations focused on “shaping the future of data governance.”

Founding member companies include American Family Insurance, Capital One, and MassMutual. Initiative Co-director Srini Devadas, a professor at MIT, says that the effort will draw on expertise across MIT in the fields of cryptography, machine learning, systems security, and public policy.

“The goal is to solve challenging problems of collaborative data analytics and machine learning where sharing data provides significant benefit to all participants, while also preserving strong privacy protections,” says Devadas.

At the launch event, CSAIL Director Daniela Rus cited MIT’s long history of work in the privacy space, from foundational work on cryptography, to IPRI and the Trust:Data Consortium, which has created tools and architectures that foster the development of a secure internet-based network of trusted data.

Member companies stressed the benefits they see in being part of this initiative as not only helping navigate a changing policy landscape but also developing technical tools to help manage the new policies, laws, and regulations more efficiently. Speaking at the launch were MassMutual’s Head of Data Adam Fox, Capital One’s Machine Learning Research Director Bayan Bruss, and American Family Insurance’s Enterprise Chief Data Officer Brad Burke.

Companies interested in participating in the new initiative can visit the CSAIL site for more information. More

Whatever business a company may be in, software plays an increasingly vital role, from managing inventory to interfacing with customers. Software developers, as a result, are in greater demand than ever, and that’s driving the push to automate some of the easier tasks that take up their time. 

Productivity tools like Eclipse and Visual Studio suggest snippets of code that developers can easily drop into their work as they write. These automated features are powered by sophisticated language models that have learned to read and write computer code after absorbing thousands of examples. But like other deep learning models trained on big datasets without explicit instructions, language models designed for code-processing have baked-in vulnerabilities.

“Unless you’re really careful, a hacker can subtly manipulate inputs to these models to make them predict anything,” says Shashank Srikant, a graduate student in MIT’s Department of Electrical Engineering and Computer Science. “We’re trying to study and prevent that.”

In a new paper, Srikant and the MIT-IBM Watson AI Lab unveil an automated method for finding weaknesses in code-processing models, and retraining them to be more resilient against attacks. It’s part of a broader effort by MIT researcher Una-May O’Reilly and IBM-affiliated researcher Sijia Liu to harness AI to make automated programming tools smarter and more secure. The team will present its results next month at the International Conference on Learning Representations.

A machine capable of programming itself once seemed like science fiction. But an exponential rise in computing power, advances in natural language processing, and a glut of free code on the internet have made it possible to automate at least some aspects of software design. 

Trained on GitHub and other program-sharing websites, code-processing models learn to generate programs just as other language models learn to write news stories or poetry. This allows them to act as a smart assistant, predicting what software developers will do next, and offering an assist. They might suggest programs that fit the task at hand, or generate program summaries to document how the software works. Code-processing models can also be trained to find and fix bugs. But despite their potential to boost productivity and improve software quality, they pose security risks that researchers are just starting to uncover.

Srikant and his colleagues have found that code-processing models can be deceived simply by renaming a variable, inserting a bogus print statement, or introducing other cosmetic operations into programs the model tries to process. These subtly altered programs function normally, but dupe the model into processing them incorrectly, rendering the wrong decision.

The mistakes can have serious consequences for code-processing models of all types. A malware-detection model might be tricked into mistaking a malicious program for benign. A code-completion model might be duped into offering wrong or malicious suggestions. In both cases, viruses may sneak by the unsuspecting programmer. A similar problem plagues computer vision models: Edit a few key pixels in an input image and the model can confuse pigs for planes, and turtles for rifles, as other MIT research has shown. 

Like the best language models, code-processing models have one crucial flaw: They’re experts on the statistical relationships among words and phrases, but only vaguely grasp their true meaning. OpenAI’s GPT-3 language model, for example, can write prose that veers from eloquent to nonsensical, but only a human reader can tell the difference. 

Code-processing models are no different. “If they’re really learning intrinsic properties of the program, then it should be hard to fool them,” says Srikant. “But they’re not. They’re currently relatively easy to deceive.”

In the paper, the researchers propose a framework for automatically altering programs to expose weak points in the models processing them. It solves a two-part optimization problem; an algorithm identifies sites in a program where adding or replacing text causes the model to make the biggest errors. It also identifies what kinds of edits pose the greatest threat. 

What the framework reveals, the researchers say, is just how brittle some models are. Their text summarization model failed a third of the time when a single edit was made to a program; it failed more than half of the time when five edits were made, they report. On the flip side, they show that the model is able to learn from its mistakes, and in the process potentially gain a deeper understanding of programming.

“Our framework for attacking the model, and retraining it on those particular exploits, could potentially help code-processing models get a better grasp of the program’s intent,” says Liu, co-senior author of the study. “That’s an exciting direction waiting to be explored.”

In the background, a larger question remains: what exactly are these black-box deep-learning models learning? “Do they reason about code the way humans do, and if not, how can we make them?” says O’Reilly. “That’s the grand challenge ahead for us.” More

Over the past decade, hospitals and other health care providers have put massive amounts of time and energy into adopting electronic health care records, turning hastily scribbled doctors’ notes into durable sources of information. But collecting these data is less than half the battle. It can take even more time and effort to turn these records into actual insights — ones that use the learnings of the past to inform future decisions.

Cardea, a software system built by researchers and software engineers at MIT’s Data to AI Lab (DAI Lab), is built to help with that. By shepherding hospital data through an ever-increasing set of machine learning models, the system could assist hospitals in planning for events as large as global pandemics and as small as no-show appointments.

With Cardea, hospitals may eventually be able to solve “hundreds of different types of machine learning problems,” says Kalyan Veeramanchaneni, principal investigator of the DAI Lab and a principal research scientist in MIT’s Laboratory for Information and Decision Systems (LIDS). Because the framework is open-source, and uses generalizable techniques, they can also share these solutions with each other, increasing transparency and enabling teamwork.

Automated for the people

Cardea belongs to a field called automated machine learning, or AutoML. Machine learning is increasingly common, used for everything from drug development to credit card fraud detection. The goal of AutoML is to democratize these predictive tools, making it easier for people — including, eventually, non-experts — to build, use, and understand them, says Veeramachaneni.

Instead of requiring people to design and code an entire machine learning model, AutoML systems like Cardea surface existing ones, along with explanations of what they do and how they work. Users can then mix and match modules to accomplish their goals, like going to a buffet rather than cooking a meal from scratch.

For instance, data scientists have built a number of machine learning tools for health care, but most of them aren’t very accessible — even to experts. “They’re written up in papers and hidden away,” says Sarah Alnegheimish, a graduate student in LIDS. To build Cardea, she and her colleagues have been unearthing these tools and bringing them together, aiming to form “a powerful reference” for hospital problem-solvers, she says.

Step by step

To turn reams of data into useful predictions, Cardea walks users through a pipeline, with choices and safeguards at each step. They are first greeted by a data assembler, which ingests the information they provide. Cardea is built to work with Fast Healthcare Interoperability Resources (FHIR), the current industry standard for electronic health care records.

Hospitals vary in exactly how they use FHIR, so Cardea has been built to “adapt to different conditions and different datasets seamlessly,” says Veeramachaneni. If there are discrepancies within the data, Cardea’s data auditor points them out, so that they can be fixed or dismissed.

Next, Cardea asks the user what they want to find out. Perhaps they would like to estimate how long a patient might stay in the hospital. Even seemingly small questions like this one are crucial when it comes to day-to-day hospital operations — especially now, as health care facilities manage their resources during the Covid-19 pandemic, says Alnegheimish. Users can choose between different models, and the software system then uses the dataset and models to learn patterns from previous patients, and to predict what could happen in this case, helping stakeholders plan ahead.

Currently, Cardea is set up to help with four types of resource-allocation questions. But because the pipeline incorporates so many different models, it can be easily adapted to other scenarios that might arise. As Cardea grows, the goal is for stakeholders to eventually be able to use it to “solve any prediction problem within the health care domain,” Alnegheimish says.

The team presented their paper describing the system at the IEEE International Conference on Data Science and Advanced Analytics in October 2020. The researchers tested the accuracy of the system against users of a popular data science platform, and found that it out-competed 90 percent of them. They also tested its efficacy, asking data analysts to use Cardea to make predictions on a demo health care dataset. They found that Cardea significantly improved their efficiency — for example, feature engineering, which the analysts said usually takes them an average of two hours, took them five minutes instead.

Trust the process

Hospital workers are often tasked with making high-stakes, critical decisions. It’s vital that they trust any tools they use along the way, including Cardea. It’s not enough for users to plug in some numbers, press a button, and get an answer: “They should get some sense of the model, and they should know what is going on,” says Dongyu Liu, a postdoc in LIDS.

To build in even more transparency, Cardea’s next step is a model audit. Like all predictive apparatuses, machine learning models have strengths and weaknesses. By laying these out, Cardea gives the user the ability to decide whether to accept this model’s results, or to start again with a new one.

Cardea was released to the public earlier this year. Because it’s open source, users are welcome to integrate their own tools. The team also took pains to ensure that the software system is not only available, but understandable and easy to use. This will also help with reproducibility, Veeramachaneni says, so that predictions made on models built with the software can be understood and checked by others.  

The team also plans to build in more data visualizers and explanations, to provide an even deeper view, and make the software system more accessible to non-experts, Liu says.

“The hope is for people to adopt it, and start contributing to it,” Alnegheimish says. “With the help of the community, we can make it something much more powerful.”  More

Melanoma is a type of malignant tumor responsible for more than 70 percent of all skin cancer-related deaths worldwide. For years, physicians have relied on visual inspection to identify suspicious pigmented lesions (SPLs), which can be an indication of skin cancer. Such early-stage identification of SPLs in primary care settings can improve melanoma prognosis and significantly reduce treatment cost.

The challenge is that quickly finding and prioritizing SPLs is difficult, due to the high volume of pigmented lesions that often need to be evaluated for potential biopsies. Now, researchers from MIT and elsewhere have devised a new artificial intelligence pipeline, using deep convolutional neural networks (DCNNs) and applying them to analyzing SPLs through the use of wide-field photography common in most smartphones and personal cameras.

How it works: A wide-field image, acquired with a smartphone camera, shows large skin sections from a patient in a primary-care setting. An automated system detects, extracts, and analyzes all pigmented skin lesions observable in the wide-field image. A pre-trained deep convolutional neural network (DCNN) determines the suspiciousness of individual pigmented lesions and marks them (yellow = consider further inspection, red = requires further inspection or referral to dermatologist). Extracted features are used to further assess pigmented lesions and to display results in a heatmap format.

Animation courtesy of the researchers.

Previous item
Next item

DCNNs are neural networks that can be used to classify (or “name”) images to then cluster them (such as when performing a photo search). These machine learning algorithms belong to the subset of deep learning.

Using cameras to take wide-field photographs of large areas of patients’ bodies, the program uses DCNNs to quickly and effectively identify and screen for early-stage melanoma, according to Luis R. Soenksen, a postdoc and a medical device expert currently acting as MIT’s first Venture Builder in Artificial Intelligence and Healthcare. Soenksen conducted the research with MIT researchers, including MIT Institute for Medical Engineering and Science (IMES) faculty members Martha J. Gray, W. Kieckhefer Professor of Health Sciences and Technology, professor of electrical engineering and computer science; and James J. Collins, Termeer Professor of Medical Engineering and Science and Biological Engineering.

Soenksen, who is the first author of the recent paper, “Using Deep Learning for Dermatologist-level Detection of Suspicious Pigmented Skin Lesions from Wide-field Images,” published in Science Translational Medicine, explains that “Early detection of SPLs can save lives; however, the current capacity of medical systems to provide comprehensive skin screenings at scale are still lacking.”

The paper describes the development of an SPL analysis system using DCNNs to more quickly and efficiently identify skin lesions that require more investigation, screenings that can be done during routine primary care visits, or even by the patients themselves. The system utilized DCNNs to optimize the identification and classification of SPLs in wide-field images.

Using AI, the researchers trained the system using 20,388 wide-field images from 133 patients at the Hospital Gregorio Marañón in Madrid, as well as publicly available images. The images were taken with a variety of ordinary cameras that are readily available to consumers. Dermatologists working with the researchers visually classified the lesions in the images for comparison. They found that the system achieved more than 90.3 percent sensitivity in distinguishing SPLs from nonsuspicious lesions, skin, and complex backgrounds, by avoiding the need for cumbersome and time-consuming individual lesion imaging. Additionally, the paper presents a new method to extract intra-patient lesion saliency (ugly duckling criteria, or the comparison of the lesions on the skin of one individual that stand out from the rest) on the basis of DCNN features from detected lesions.

“Our research suggests that systems leveraging computer vision and deep neural networks, quantifying such common signs, can achieve comparable accuracy to expert dermatologists,” Soenksen explains. “We hope our research revitalizes the desire to deliver more efficient dermatological screenings in primary care settings to drive adequate referrals.”

Doing so would allow for more rapid and accurate assessments of SPLS and could lead to earlier treatment of melanoma, according to the researchers.

Gray, who is senior author of the paper, explains how this important project developed: “This work originated as a new project developed by fellows (five of the co-authors) in the MIT Catalyst program, a program designed to nucleate projects that solve pressing clinical needs. This work exemplifies the vision of HST/IMES devotee (in which tradition Catalyst was founded) of leveraging science to advance human health.” This work was supported by Abdul Latif Jameel Clinic for Machine Learning in Health and by the Consejería de Educación, Juventud y Deportes de la Comunidad de Madrid through the Madrid-MIT M+Visión Consortium. More

In recent years, robots have gained artificial vision, touch, and even smell. “Researchers have been giving robots human-like perception,” says MIT Associate Professor Fadel Adib. In a new paper, Adib’s team is pushing the technology a step further. “We’re trying to give robots superhuman perception,” he says.

The researchers have developed a robot that uses radio waves, which can pass through walls, to sense occluded objects. The robot, called RF-Grasp, combines this powerful sensing with more traditional computer vision to locate and grasp items that might otherwise be blocked from view. The advance could one day streamline e-commerce fulfillment in warehouses or help a machine pluck a screwdriver from a jumbled toolkit.

The research will be presented in May at the IEEE International Conference on Robotics and Automation. The paper’s lead author is Tara Boroushaki, a research assistant in the Signal Kinetics Group at the MIT Media Lab. Her MIT co-authors include Adib, who is the director of the Signal Kinetics Group; and Alberto Rodriguez, the Class of 1957 Associate Professor in the Department of Mechanical Engineering. Other co-authors include Junshan Leng, a research engineer at Harvard University, and Ian Clester, a PhD student at Georgia Tech.

Play video

As e-commerce continues to grow, warehouse work is still usually the domain of humans, not robots, despite sometimes-dangerous working conditions. That’s in part because robots struggle to locate and grasp objects in such a crowded environment. “Perception and picking are two roadblocks in the industry today,” says Rodriguez. Using optical vision alone, robots can’t perceive the presence of an item packed away in a box or hidden behind another object on the shelf — visible light waves, of course, don’t pass through walls.

But radio waves can.

For decades, radio frequency (RF) identification has been used to track everything from library books to pets. RF identification systems have two main components: a reader and a tag. The tag is a tiny computer chip that gets attached to — or, in the case of pets, implanted in — the item to be tracked. The reader then emits an RF signal, which gets modulated by the tag and reflected back to the reader.

The reflected signal provides information about the location and identity of the tagged item. The technology has gained popularity in retail supply chains — Japan aims to use RF tracking for nearly all retail purchases in a matter of years. The researchers realized this profusion of RF could be a boon for robots, giving them another mode of perception.

“RF is such a different sensing modality than vision,” says Rodriguez. “It would be a mistake not to explore what RF can do.”

RF Grasp uses both a camera and an RF reader to find and grab tagged objects, even when they’re fully blocked from the camera’s view. It consists of a robotic arm attached to a grasping hand. The camera sits on the robot’s wrist. The RF reader stands independent of the robot and relays tracking information to the robot’s control algorithm. So, the robot is constantly collecting both RF tracking data and a visual picture of its surroundings. Integrating these two data streams into the robot’s decision making was one of the biggest challenges the researchers faced.

“The robot has to decide, at each point in time, which of these streams is more important to think about,” says Boroushaki. “It’s not just eye-hand coordination, it’s RF-eye-hand coordination. So, the problem gets very complicated.”

The robot initiates the seek-and-pluck process by pinging the target object’s RF tag for a sense of its whereabouts. “It starts by using RF to focus the attention of vision,” says Adib. “Then you use vision to navigate fine maneuvers.” The sequence is akin to hearing a siren from behind, then turning to look and get a clearer picture of the siren’s source.

With its two complementary senses, RF Grasp zeroes in on the target object. As it gets closer and even starts manipulating the item, vision, which provides much finer detail than RF, dominates the robot’s decision making.

RF Grasp proved its efficiency in a battery of tests. Compared to a similar robot equipped with only a camera, RF Grasp was able to pinpoint and grab its target object with about half as much total movement. Plus, RF Grasp displayed the unique ability to “declutter” its environment — removing packing materials and other obstacles in its way in order to access the target. Rodriguez says this demonstrates RF Grasp’s “unfair advantage” over robots without penetrative RF sensing. “It has this guidance that other systems simply don’t have.”

RF Grasp could one day perform fulfilment in packed e-commerce warehouses. Its RF sensing could even instantly verify an item’s identity without the need to manipulate the item, expose its barcode, then scan it. “RF has the potential to improve some of those limitations in industry, especially in perception and localization,” says Rodriguez.

Adib also envisions potential home applications for the robot, like locating the right Allen wrench to assemble your Ikea chair. “Or you could imagine the robot finding lost items. It’s like a super-Roomba that goes and retrieves my keys, wherever the heck I put them.”

The research is sponsored by the National Science Foundation, NTT DATA, Toppan, Toppan Forms, and the Abdul Latif Jameel Water and Food Systems Lab (J-WAFS). More

In recent years there have been exciting breakthroughs in wearable technologies, like smartwatches that can monitor your breathing and blood oxygen levels. 

But what about a wearable that can detect how you move as you do a physical activity or play a sport, and could potentially even offer feedback on how to improve your technique? 

And, as a major bonus, what if the wearable were something you’d actually already be wearing, like a shirt of a pair of socks?

That’s the idea behind a new set of MIT-designed clothing that use special fibers to sense a person’s movement via touch. Among other things, the researchers showed that their clothes can actually determine things like if someone is sitting, walking, or doing particular poses.

The group from MIT’s Computer Science and Artificial Intelligence Lab (CSAIL) says that their clothes could be used for athletic training and rehabilitation. With patients’ permission, they could even help passively monitor the health of residents in assisted-care facilities and determine if, for example, someone has fallen or is unconscious.  

The researchers have developed a range of prototypes, from socks and gloves to a full vest. The team’s “tactile electronics” use a mix of more typical textile fibers alongside a small amount of custom-made functional fibers that sense pressure from the person wearing the garment.

According to CSAIL graduate student Yiyue Luo, a key advantage of the team’s design is that, unlike many existing wearable electronics, theirs can be incorporated into traditional large-scale clothing production. The machine-knitted tactile textiles are soft, stretchable, breathable, and can take a wide range of forms. 

“Traditionally it’s been hard to develop a mass-production wearable that provides high-accuracy data across a large number of sensors,” says Luo, lead author on a new paper about the project that is appearing in this month’s edition of Nature Electronics. “When you manufacture lots of sensor arrays, some of them will not work and some of them will work worse than others, so we developed a self-correcting mechanism that uses a self-supervised machine learning algorithm to recognize and adjust when certain sensors in the design are off-base.”

The team’s clothes have a range of capabilities. Their socks predict motion by looking at how different sequences of tactile footprints correlate to different poses as the user transitions from one pose to another. The full-sized vest can also detect the wearers’ pose, activity, and the texture of the contacted surfaces.

The authors imagine a coach using the sensor to analyze people’s postures and give suggestions on improvement. It could also be used by an experienced athlete to record their posture so that beginners can learn from them. In the long term, they even imagine that robots could be trained to learn how to do different activities using data from the wearables. 

“Imagine robots that are no longer tactilely blind, and that have ‘skins’ that can provide tactile sensing just like we have as humans,” says corresponding author Wan Shou, a postdoc at CSAIL. “Clothing with high-resolution tactile sensing opens up a lot of exciting new application areas for researchers to explore in the years to come.”

The paper was co-written by MIT professors Antonio Torralba, Wojciech Matusik, and Tomás Palacios, alongside PhD students Yunzhu Li, Pratyusha Sharma, and Beichen Li; postdoc Kui Wu; and research engineer Michael Foshey. 

The work was partially funded by Toyota Research Institute. More