As many as 100,000 breast cancer patients have one or both breasts removed in mastectomies every year in the United States. This surgery frequently leads to loss of breast sensation, which is thought to contribute to the high rates of sexual dysfunction among breast cancer survivors. But a bionic breast now under development could restore these important tactile sensations.
The project is the brainchild of Stacy Lindau, a gynecologist at the University of Chicago who specialises in the sexual function of women with cancer. She says she’s dealt with countless women over the past decade whose sex lives have been dramatically impacted by this loss of sensation.
Since the outbreak of the COVID-19 pandemic, national and local governments have come to realize that although sheltering in place helps to “flatten the curve” of new cases, economic and other considerations mean we can’t stay home indefinitely. But if we are to regain some sense of normalcy, early detection of the novel coronavirus, SARS-CoV-2, will be essential to containing it—at least during the period between the easing of restrictions on gatherings and commercial activity and the development of a vaccine.
At this moment, the best diagnostic tests for SARS-CoV-2, based on real-time RT-PCR (rRT-PCR) assays, are sensitive, but they require expensive equipment and trained technicians. What’s more, there’s a relatively long turnaround time for getting an answer (at least 24 hours prep time after receipt of a sample, plus 6 hours to complete the actual test procedure).
Now a multidisciplinary group of researchers at the University of Minnesota, with expertise in magnetics, microbiology and electrical engineering, says it has developed a test that is relatively cheap, easy to use, and quick to deliver results. The group, led by Prof. Jian-Ping Wang, chair of the Department of Electrical & Computer Engineering, and Associate Professor Maxim C. Cheeran, from the Department of Veterinary Population Medicine, says it looks to have a commercially viable version of its prototype system available soon enough to help speed up the pace of pre-vaccine testing. IEEE Spectrum interviewed Wang, an IEEE Fellow, about his group’s portable testing system.
IEEE Spectrum: How would you describe your virus detection system in a sentence or two?
Wang: It’s a portable platform based on magnetic particle spectroscopy (MPS) that allows for rapid, sensitive detection of SARS-CoV-2 at the point of care. Eventually, it will be used for routine diagnosis in households.
Spectrum: Give us a quick primer on the science underlying the MPS testing technique.
Wang: MPS is a versatile platform for different bioassays that uses artificially designed magnetic nanoparticles that act as magnetic tracers when their surfaces are functionalized with test reagents such as antibodies, aptamers or peptides. Imagine them as tiny probes capturing target analytes from biofluid samples. For COVID-19 antigen detection, we functionalized our nanoparticles with polyclonal antibodies to two of the four structural proteins that are components of the coronavirus (nucleocapsid and spike proteins). The antibodies allow the nanoparticles to bind to epitopes, or receptor sites, on these particular proteins. The more binding that occurs, the greater the presence of the virus.
Spectrum: So, how does the MPS system answer the question “Does this person have coronavirus?”
Wang: The MPS platform’s job is to monitor and assess the real-time specific binding of the nanoparticles with these proteins. The quantity or concentration of the target analyte—in this case, the aforementioned proteins indicating the presence of coronavirus—directly affects the responses of the nanoparticles to the system’s magnetic field. The magnitude of the difference in their behavior before and after the addition of the biofluid sample tells the tale. Because the biological tissues and fluids are nonmagnetic, there is negligible magnetic background noise from the biological samples. As a result, this volumetric-based immunoassay tool not only uncomplicated, but also accurate and effective with minimal sample preparation.
Spectrum: Take us through the steps of a testing cycle using this device.
Wang: No problem. But let’s back up a bit and note that our group has developed the MagiCoil Android app that processes information from the device’s microcontroller in real-time. And crucially, it also guides users on how to conduct a test from start to finish. The user interacts with the MPS handheld device using their smart phone, which communicates with the device via Bluetooth. Testing results are securely transmitted to cloud storage and could be readily shared between a patient and clinicians.
To run a test, the user begins by inserting a testing vial into the device and waiting as the system collects a baseline signal for 10 seconds. Then the user adds a biofluid sample into vial and waits again as the antigen and antibody bind for 10 minutes. The system will automatically read the ending signal for 10 seconds, then display the results.
Spectrum: How did you know this would work?
Wang: This epitope binding of the nanoparticles and target analytes form viral protein clusters, similar to what occurred when we applied this technique to the detection of the H1N1 nucleoprotein reported in our previous work.
Spectrum: What makes your team sure there won’t be a shortage of testing vials containing the functionalized nanoparticles (a situation comparable to the current shortage of reagents for COVID testing)?
Wang: For each test we use only a microgram of nanoparticles and a nanogram of reagents (antibodies, RNA fragments). We are already collaborating with nanoparticle companies, who supply us with high quality iron nitride nanoparticles for this application. Our group has also been seeking collaborations with biotechnology companies to secure sources of chemical reagents.
Spectrum: What challenges has your team faced and how did you overcome them?
Wang: Over the past decade, there was very little attention paid to filling the need in the market for a low noise, easy-to-use, portable bioassay kit for the detection of viruses—not until we began working on detecting Influenza A Virus subtype H1N1 in past year. We have steadily grown the team, accumulated experience in this research area, and optimized the MPS platform. In addition to full-time researchers, the MPS and MagiCoil team has benefited from the efforts of many talented graduate and undergraduate students from different areas. This work has resulted in the current version of the MPS device and the accompanying app.
I feel proud of my students for forging ahead on a project because they see it as an important bit of research and never giving up. Since the outbreak of COVID-19 pandemic, we saw how critical it was for us to speed up work on our project so we could contribute to the fight against the virus by making our MPS device available as soon as possible.
Spectrum: Your team’s aim has been to get this in doctors’ offices so anyone could walk in, get tested, and walk out knowing whether they’ve contracted the coronavirus. But in places like New York City, people have been urged to stay away from hospitals and clinics unless they are experiencing acute symptoms. How important is it to go beyond the clinical setting to household use so even asymptomatic people know their status?
Wang: From the outset, we wanted to make it inexpensive and easy to use so untrained people could conduct tests at home or out in the field in remote areas that are the antithesis of clinical situations. It will let large portions of the population afford to get regular updates on whether they have contracted the virus. And because it is capable of transmitting test results collected from distant locations to centrally located data analysis units, governments can have real-time epidemiological data at their fingertips. This would also significantly reduce the costs associated with tracking the spread of a disease and help health authorities more quickly evaluate and refine their disease control protocols.
Spectrum: How long before the system is commercially available?
Wang: We have just transformed the benchtop version of the system into a handheld version [which, at 212- by-84-by-72-millimeters is about the size of an old-school brick cellphone]. We have carried out preliminary tests such as characterizing the minimum amount of magnetic nanoparticles detectable and the system’s overall antigen sensitivity. And we’re still homing in on the optimum concentration of antibody to be functionalized on the nanoparticles so they’ll be most effective.
We anticipate that clinical trials will take an additional 3 to 5 months. At that point, we will work with local companies in Minnesota to mass produce MPS devices. The University of Minnesota Office for Technology Commercialization has been helping us lay the groundwork for founding a startup company in order to accelerate the process of commercializing this handheld device the instant we receive the necessary government approvals.
Spectrum: You mentioned antigen sensitivity. How much virus must there be in a sample for the MPS system to detect it?
Wang: We are currently evaluating what is the lowest concentration of virus our test can detect. But based on our experience with the H1N1 flu virus, it will be less than 150 virus particles.
Spectrum: I know it’s hard to say definitively, but give us a ballpark figure for the eventual price of the MPS testing system.
Wang: Based on our first prototype MSP device, we foresee the unit price starting at roughly US $100; the MagiCoil app, which is already completed and available for download from the Google Play store, is free. Testing vials containing the functionalized magnetic nanoparticles targeting the coronavirus will cost between $2 and $5 each.
Eventually, we plan to make a second-generation MPS device that’s as small as today’s smart phones. That step will require expertise in the areas of microfluid channel design, printed microcoils, high moment magnetic nanoparticles, automatic biofluid sample loading and filtering, optimized circuit layouts, etc. We are open to collaborations with other groups (including, but certainly not limited to IEEE members) to make a better, lighter weight, sensitive, fully automatic MPS device.
In Maryland, restaurant patrons stand inside bumper-style tables to keep six feet apart. In New York, sunbathers maintain distance by lounging in white chalk circles painted on a grassy field.
As the United States slowly begins opening public spaces, organizations are getting creative about how to encourage social distancing. But two new studies on the airborne spread of saliva droplets, which can harbor virus particles from respiratory diseases like COVID-19, suggest those six feet alone are not always enough.
In a rare act of cooperation, Google and Apple this month released specifications for software developers to build digital contact tracing apps for Apple and Google mobile operating systems, which jointly encompass the majority of smartphones around the world.
Digital contact tracing, which can automatically notify an individual if they’ve crossed paths with someone who tested positive for COVID-19, has been proposed as a way to augment manual contact tracing, which requires the painstaking work of thousands of trained workers per state to identify, track, and assist individuals exposed to the virus.
As digital contact tracing technologies advance, two questions rise to the surface: Will state health officials and individuals opt to use the technology? And, if so, how well will it work?
As shuttered businesses make plans to resume on-site operations, many plan to outfit their employees with new, anti-pandemic gear: wearable tech that could prevent the spread of COVID-19 inside the workplace.
Other employers are considering equipping their workforces with wearables—separate from their phones—that are capable of granular on-site and indoor location tracking and contact tracing. CarePredict recently rolled out such devices for senior living facilities.
In fact, in a survey of 871 finance executives at companies in 24 countries, 21 percent said they were eyeing location tracking and contact tracing for their workforces, according to PwC, which conducted the survey and posted it online this week.
When the last global pandemic broke out, in 1918, it ravaged a population with essentially no technological countermeasures. There were no diagnostic tests, no mechanical ventilators, and no antiviral or widely-available anti-inflammatory medications other than aspirin. The first inactivated-virus vaccines would not become available until 1936. An estimated 50 million people died.
For the current outbreak, a best-case scenario could limit fatalities to 1.3 million, according to projections by Imperial College London. That in a world with 7.8 billion people—more than four times as many as in 1918. Many factors will lessen mortality this time, chief among them better, more consistent implementation of social-distancing measures. But technology will also be a primary bulwark. Enormous sums are being spent to ramp up testing, diagnosis, modeling, treatment, vaccination, and other tech-based responses.
In reaction to the rapid spread of COVID-19, over 100 countries worldwide have instituted lockdowns, restricting movement for billions of people. Those restrictions are having major effects, such as a global food crisis, on economically vulnerable populations.
Now, a group of 73 volunteer engineers, students, and policy experts are working to identify and quantify unintended consequences of the COVID-19 lockdown on vulnerable populations.
For two months, the group, organized by Palo Alto-based Omdena, will scour publicly available data sources and apply data visualization and AI tools to investigate how government policies are impacting four effects of the pandemic lockdowns: reduced access to healthcare, wage loss, employment loss, and domestic abuse.
“All over the world, from the media to state leaders, we are hearing a lot of noise and a lot of propaganda, amidst a lot of facts,” says Baidurja Ray, a computational scientist and engineer based in Texas who is participating in the effort. “I hope this project provides a purely factual and data-driven look into the effect of government policies on their citizens.”
The COVID-19 epidemic in China was gathering terrible force when 11 overworked Chinese physicians found time to write a research paper. At the time, in mid February, confirmed cases of the disease were inching towards 60,000 and the death toll stood at nearly 1,400. The doctors, most of them pulmonary specialists at the Xi’an Chest Hospital, had already treated scores of people seriously stricken with the disease.
Weary as they were, they had something they wanted to tell the world: “Ultrasound is playing an indispensable role in the diagnosis, treatment and efficacy evaluation of severe acute pneumonia,” the life-threatening illness associated with the most severe cases of COVID-19.
While the response from these well-intentioned innovators is inspiring, there remains some issues around the use these new designs—one in particular being that very few were designed to meet minimum safety standards or vetted at an early enough stage to help these pop-up equipment manufacturers make the necessary adjustments.
One group, called Collaborating to Address Shortages of Medical Supplies (CASMS), has formed to address the issue. Mike Dempsey, Director of the CIMIT Accelerator Program, helped create the group after his own experience of designing a face shield for a hospital. Upon coming up with the design, it became evident that the hospital may not be able to use the face shields without assurance that basic standards were met.
“So that was very frustrating,” say Dempsey. “Because we had this supply of emergency shields and we had a solution, but we can’t get it into the hands of the caregivers.
“But then I realized that it’s [necessary] that the things that do get into the hands of caregivers are safe. You don’t want sub-quality stuff to be used. So that was the genesis of CASMS. We had to figure out a way to have both speed and safety.”
Dempsey was already a member of a consortium for innovative medicine, called CIMIT, which comprises experts at the intersection of medicine, academia, and business. From CIMIT, the CASMS working group was formed. The team looks at existing specifications and testing protocols for the type of medical supply or device at hand—say, a face shield. Existing specifications for a face shield may be very detailed and include features that are not applicable to the COVID-19 crisis, but are related to different tasks, such as welding for example. The CASMS group pares down these specifications to make them applicable to the pandemic.
Adjusting the specifications also helps by eliminating the need for expensive or specialized equipment. The standard test rig for a face shield typically analyzes multiple features of the shield; in contrast, CASMS may suggest a simpler test fixture that only tests the features important to healthcare workers. “It tests the same thing, but in a much simpler way,” Dempsey explains.
With many existing designs already out there and CASMS in action, Dempsey is emphasizing the need for people to shift the focus from designing to testing. “I will say that it has been a very positive experience because—since the word has gotten out that we’re doing this—so many people have reached out and said, ‘I want to help,’ When we say things like we really need more testers and not more people making new designs, everyone is willing to step up and do that,” he says.
While the pandemic will remain the focus of this initiative in the immediate future, Dempsey has some ideas on how CASMS can continue in the long-term. He has previously worked in impoverished countries and those with middling economies—exactly the types that tend to have medical supply shortages.
“We are interested in keeping the CASMS website going after we’ve conquered COVID, to help low- and middle-income countries deal with their persistent equipment shortages. They are lacking medical equipment all of the time, and if there can be minimally viable specs, we think that could help with the development of newer, lower cost things, and hence safety of things that are produced,” he says.
As we write these words, several billion people, the majority of the world’s population, are confined to their homes or subject to physical-distancing policies in an attempt to contain one of the worst pandemics of modern times. Economic activity has plummeted, countless people are out of work, and entire industries have ground to a halt.
Quite understandably, a couple of questions are on everyone’s mind: What is the exit strategy? How will we know when it’s safe to implement it?
Around the globe, epidemiologists, statisticians, biologists, and health officials are grappling with these questions. Though engineering perspectives are uncommon in epidemiological modeling, we believe that in this case public officials could greatly benefit from one. Of course, the COVID-19 pandemic isn’t an obvious or typical engineering problem. But in its basic behavior it is an unstable, open-loop system. Left alone, it grows exponentially, as we have all been told repeatedly. However, there’s good news, too: Like many such systems, it can be stabilized effectively and efficiently by applying the principles of control theory, most notably the use of feedback.
Inspired by the important work of epidemiologists and others on the front lines of this global crisis, we have explored how feedback can help stabilize and diminish the rate of propagation of this deadly virus that now literally plagues us. We’ve drawn on proven engineering principles to come up with an approach that would offer policymakers concrete guidance, one that takes into account both medical and socioeconomic considerations. We relied on feedback-based mechanisms to devise a system that would bring the outbreak under control and then adeptly manage the longer-term caseload.
It is during this longer-term phase, the inevitable relaxing of physical distancing that is required for a functioning society, that the strengths of a response grounded in control theory are most crucial. Using one of the widely available computer models of the disease, we tested our proposal and found that it could help officials manage the enormous complexity of trade-offs and unknowns that they will face, while saving perhaps hundreds of thousands of lives.
Our goal here is to share some of our key findings and to engage a community of control experts in this vital and fascinating problem. Together, we can contribute vitally to the international efforts to manage this outbreak.
The COVID-19 pandemic is unlike any other recent disease outbreak for several reasons. One is that its basic reproduction number, or R0 (“R naught”), is relatively high. R0 is an indication of how many people, on average, an infected person will infect during the course of her illness. R0 is not a fixed number, depending as it does on such factors as the density of a community, the general health of its populace, its medical infrastructure and resources, and countless details of the community’s response. But a commonly cited R0 figure for ordinary seasonal influenza is 1.3, whereas a figure calculated for the experience in Wuhan, China, where COVID-19 is understood to have originated, is 2.6 [PDF]. Figures for some outbreaks in Italy range from 2.76 to 3.25 [PDF].
The goal of infectious-disease intervention is reducing the R0 to below 1, because such a value means that new infections are in decline and will eventually reach zero. But with the COVID-19 outbreak, the level of urgency is extraordinarily high due to the disease’s relatively high fatality rate. Fatality rates, too, are quite variable and depend on such factors as age, physical fitness, present pathologies, region, and access to health care. But in general they are much higher for COVID-19 than for ordinary influenza. A surprisingly large percentage of people who contract the disease develop a form of viral pneumonia that sometimes proves fatal. Many of those patients require artificial ventilation, and if their number exceeds the capacity of intensive care units to accommodate them, some number of them, perhaps a majority, will die.
For that reason, enormous worldwide efforts have focused on “flattening the curve” of infections against time. A high, sharp curve indicating a surge of infections in a short time period, as occurred in China, Italy, Spain, and elsewhere, means that the number of serious cases will swamp the ability of hospitals to treat them and result in mass fatalities. So to reduce the peak demand on health care, the first priority must be to bring the caseload under control. Once that’s done, the emphasis shifts to managing a long-term return to normalcy while minimizing both death rates and economic impact.
The two basic approaches to controlling the spread of disease are mitigation, which focuses on slowing but not necessarily stopping the spread, and suppression, which aims to reverse epidemic growth. For mitigation, R0 is reduced but remains greater than 1, while for suppression, R0 is smaller than 1. Both obviously require changing R0. Officials accomplish that by introducing social measures such as restricted travel, home confinement, social distancing, and so on. These restrictions are referred to as nonpharmaceutical interventions, or NPIs. What we are proposing is a systematically designed strategy, based on feedback, to change R0 through modulation of NPIs. In effect, the strategy alternates between suppression and mitigation in order to maintain the spread at a desired level.
It may sound straightforward, but there are many challenges. Some of them arise from the fact that COVID-19 is a very peculiar disease. Despite enormous efforts to characterize the virus, biologists still do not understand why some people experience fairly mild symptoms while others spiral into a massive, uncontrolled immune response and death. And no one can explain why, among fatalities, men predominate. Other mysteries include the disease’s long incubation period—up to 14 days between infection and symptoms—and even the question of whether a person can get re-infected.
These perplexities have helped bog down efforts to deal with the pandemic. As a recent Imperial College research paper [PDF] notes: “There are very large uncertainties around the transmission of this virus, the likely effectiveness of different policies, and the extent to which the population spontaneously adopts risk reducing behaviours.” Consider the long incubation time and apparent spreading of the virus before symptoms are experienced. These undoubtedly contributed to the relatively high R0 values, because people who were infectious continued to interact with others and transmitted the virus without being aware that they were doing so.
This lag before the onset of symptoms corresponds to time delay in control-system theory. It is notorious for introducing oscillations into closed-loop systems, particularly when combined with substantial uncertainty in the model itself.
In addition to delays, there are very significant uncertainties. Testing, for example, has been spotty in some countries, and that inconsistency has obscured the number of actual cases. That, in turn, made it impossible for officials to know the true level of contagion. Even NPIs are not immutable. The extent to which the public is complying with policies is never 100 percent and may not even be knowable with a high degree of accuracy; people may follow directives less strictly over time. Also, health care capacity can go up because of an increase in available beds due to capacity additions, or down because of a decrease due to a natural disaster.
The point is, a pandemic is a dynamic, fast-moving situation, and inadequate local attempts to monitor and control it can be disastrous. In the Spanish flu pandemic of 1918, cities took widely varying approaches to the lockdown and release of their citizens, with wildly varying results. Some recovered straightforwardly, others had rebound spikes larger than the initial outbreak, and still others had multiple outbreaks after the initial lockdown.
A commonly cited proposal for relaxing social-distancing measures is an on-off approach, where some restrictions are lifted when the number of new cases requiring intensive care is below a threshold and are put back into place when it exceeds a certain number. The research paper [PDF] by the Imperial College COVID-19 Response Team showed how such a strategy is “robust to uncertainty in both the reproduction number, R0, and in the severity of the virus” and offers “greater robustness to uncertainty than fixed duration interventions and can be adapted for regional use.”
This on-off approach is an example of the use of feedback, where the feedback variable is the number of cases in hospital intensive care units. A major drawback of this type of on-off control is that it can lead to oscillations—which, if this strategy is too aggressive, may overwhelm the capacity of the health care system to treat serious cases.
A major advantage of feedback here is that it lessens the impact of model uncertainty—meaning that if carefully designed, a strategy can be effective even if the models it is based on are not accurate. We do not yet have an accurate epidemiological model of COVID-19 and will likely not have one for at least several months, if at all. Furthermore, the physical distancing and confinement regimes that have been put into place are new, so we don’t really know yet exactly how effective they’ll be or even the extent to which people are complying with them.
In the absence of widespread immunity or vaccination, the only way to suppress the disease is total confinement—obviously not a viable long-term solution. A reasonable middle ground is to implement a feedback policy designed to keep R0 close to 1, with perhaps small oscillations on either side. In so doing we would maintain the critical caseload within the capacity of health care institutions while slowly and safely building immunity in our communities, and returning to normal social and economic conditions as quickly as is safely possible.
A key point is that the design of the policy be rigorous from a control-engineering viewpoint while remaining comprehensible to epidemiologists, policymakers, and others without deep knowledge of control theory. It should also be capable of generating restriction regimes that can be translated into practical public policies. If the tuning mechanism is too aggressive—for example, switching between full and zero social distancing—it will lead to severe oscillations and overwhelmed hospitals, and very likely frustration and social-distancing fatigue among the people who need to follow dramatically changing rules.
On the other hand, tuning that is too timid also courts fiasco. An example of such tuning might be a policy requiring a full month in which no new cases are recorded before officials relax restrictions. Such a hypercautious approach risks needlessly prolonging the pandemic’s economic devastation, creating a catastrophe of a different sort.
But a properly designed feedback-based policy that takes into account both dynamics and uncertainty can deliver a stable result while keeping the hospitalization rate within a desired approximate range. Furthermore, keeping the rate within such a range for a prolonged period allows a society to slowly and safely increase the percentage of people who have some sort of antibodies to the disease because they have either suffered it or they have been vaccinated—preferably the latter.
Eventually, as the percentage of the population who have suffered the disease and recovered from it becomes high enough, the number of susceptible people becomes small enough that the virus’s rate of spread is naturally lowered. This phenomenon is called herd immunity, and it is how pandemics have generally died out in the past. However, the question of whether or how such immunity to COVID-19 can be built up is still under investigation, which makes it particularly important to monitor and manage nonpharmaceutical interventions as a function of the actual spread of the virus and of hospitalization rates.
In designing our control system, we relied strongly on the fact that most nonpharmaceutical interventions are not binary, on-off quantities but instead can take on a range of values and can be implemented that way by policymakers. For example, stay-at-home directives can be applied disproportionately to specific groups of people who are particularly at risk because of age or a preexisting health condition. Then the number of people who are affected by the directive can be increased or decreased by simply changing the guidelines on who is “at risk.” We can similarly widen or narrow the definition of people who are designated “essential” and are therefore exempt from the directives. Public meetings, too, may be banned for n participants, where the value of n can be increased as things loosen up.
Restrictions on travel, too, can be quite variable. Full lockdown limits people to moving within the boundaries of their property. But as conditions improve, officials might grant access to businesses within a couple, and then perhaps a dozen, kilometers of their home, and so on. These are all obviously important levers.
To explore how such variability can save lives, we devised a series of scenarios, each indicative of a recovery strategy with a different level of feedback, and simulated the resulting policies against a commonly used infectious-disease computer model. We plotted the results in a series of graphs showing COVID-19 hospital cases as a function of time. Hospital occupancy may be a more reliable and tangible measure than total case count, which depends on extensive testing that many countries (such as the United States) do not have at the moment. Furthermore, hospital ICU bed occupancy or ventilator availability is arguably an important measure of the ability of the local health care system to treat those who are suffering from respiratory distress acute enough to require intensive care and perhaps assisted breathing.
The model we used was created by Jeffrey Kantor, professor of chemical engineering at the University of Notre Dame (Kantor’s model is available on GitHub). The model assumes we can suppress disease transmission to a very low level by choosing appropriate policy levers. Although worldwide experience with COVID-19 is still limited, at the time of this writing this assumption appears to be a realistic one.
To make the model more reflective of our current understanding of COVID-19, we added two types of uncertainty. We assumed different values of R0 to see how they affected outcomes. To consider how noncompliance with nonpharmaceutical interventions would affect results, we programmed for a range of effectiveness of these NPIs.
Our first, simplest simulation confirms what we all know by now, which is that not doing anything was not an option [Figure 1].
The large and lengthy peak well above the available bed capacity in the intensive care unit indicates a huge number of cases that will likely result in death. This is why, of course, most countries have put aggressive measures in place to flatten the curve.
So what do we do when the number of infections comes down? As this second simulation clearly shows [Figure 2], relaxing all restrictions when the number of infections has come down will only lead to a second surge in infections. Not only could this second surge overwhelm our hospitals, it could also lead to an even higher mortality rate than the first surge, as occurred repeatedly in several U.S. cities during the Spanish flu epidemic of 1918.
Now let’s consider the simple on-off approach to confinement, in which most of the usual restrictions on gatherings, travel, and social interaction are lifted entirely when the number of new ICU cases drops below a lower threshold, and then are put back into place when this number exceeds a higher threshold. In this case, the R0 swings sharply between two levels, a high above 2 and a low below 1, as shown in blue in the graph [Figure 3]. This approach leads to oscillations, and if it is applied too aggressively, the high points of these oscillations will exceed the health care system’s capacity to treat patients. Another likely problem with this approach has been labeled “social distance fatigue.” People become weary of the repeated changes to their routine—going back to work for a couple of weeks, then being told to stay at home for a few weeks, then given the all-clear to go back to work, and so on.
We can do better. For our third experiment, we developed a scenario in which we targeted 90 percent occupancy of hospital intensive care units. To achieve this, we designed a simple feedback-based policy using the principles of control systems theory.
When R0 is high, many restrictions are put into place. People are largely confined to their homes and services are limited to the bare minimum needed for society to function—utilities, police, sanitation, and food distribution, for example. Then, as conditions begin to improve, as revealed by our feedback measure of hospital-bed occupancy, other services are gradually phased in. Recovered people are allowed to move freely as they can no longer contract, or transmit, the virus. Perhaps people are allowed to visit restaurants within walking distance, some small businesses are allowed to reopen under certain conditions, or certain age groups are subject to less-stringent restrictions. Then geographical mobility might be loosened in other ways. The point is that restrictions are eased gradually, with each new gradation based carefully on feedback.
This strategy results in a stable response that maximizes the rate of recovery. Furthermore, the demand for hospital ICU beds never exceeds a threshold, thanks to a “set point” target below that threshold [Figure 4]. The health care capacity limit is never breached. In addition, note the general upward trend for the release of restrictions, as the number of recovered and immune people grows and nonpharmaceutical interventions are gradually phased out.
This simplified example shows how using feedback to modulate the restrictions imposed on a population to modify R0 leads to a policy that is robust. For example, early on in the outbreak, there will be a great deal of uncertainty about R0 because testing will still be spotty, and because an unknown number of people may have the disease without realizing it. That uncertainty will inevitably fuel a surge in initial cases. However, once the case count is stabilized by the initial restrictive regime, a policy based on feedback will prove very tolerant of variations in R0, as illustrated in Figure 5. As the graph shows, after a few months it doesn’t matter whether the R0 is 2 or 2.6 because the total case count stays well below the number of available hospital beds due to the use of feedback.
As an added benefit, using feedback makes the policy effective even in the face of likely degrees of noncompliance. In practice, what noncompliance means is that a given level of restrictions will result in an R0 that is slightly higher than expected, which in turn causes fluctuations in the number of people who are infected. Noncompliance might, for instance, result in the restrictions being 10 percent less effective than intended [Figure 6]. However, through feedback, the policy will automatically tighten to compensate.
In reality, various factors that the model treats as invariable, such as health care capacity, might be anything but. However, variations of this sort can typically be accommodated in a policy, for example by changing the threshold on ICU occupancy.
Clearly, tried-and-true principles of control theory, particularly feedback, can help officials plot more robust and optimal strategies as they attempt to deal with the devastating COVID-19 pandemic. But how to make officials aware of these powerful tools?
Imagine an online interactive tool offering detailed, specific guidance in plain language and aimed at public officials and others charged with mounting a response to the pandemic in their communities. The guidance would be based on strategies developed by a small group of control theorists, epidemiologists, and people with policy experience. The site could review the now-familiar initial response, in which nonessential workers are confined to their homes except for essential needs. Then the site could go on to give some guidance on how and when the tightest restrictions could be lifted.
The biggest challenge to the designers of this Web-based tool will be enabling nonspecialists to visualize how the various components of the epidemiological model interact with the various feedback policy options and model uncertainty. How exactly should the main feedback measure—likely some aspect of hospital or intensive care occupancy—be implemented? Which restrictions should be lifted in the first round of easing? How should they be eased in the first round? In the second round? While monitoring the feedback measure, how frequently should officials consider whether to implement another round of easing? Feedback will help officials determine when to time various phases of interventions. An interactive tool that could assess different policy approaches, illustrating what conditions must be in place to alleviate uncertainty and shrink the projected caseload, would be very valuable indeed.
Working with political officials, epidemiologists, and others, control engineers can systematically design policies that take these constraints and trade-offs into account. It comes down to this: In the many months of struggle ahead, such a collaboration could save countless lives.
Greg Stewart is vice president of data science for the agriculture technology startup Ecoation and an adjunct professor at the University of British Columbia, in Vancouver. A Fellow of the IEEE, he has led the research, development, and deployment of control and machine-learning technology in such applications as microalgae cultivation, large-scale data centers, automotive power-train control, and semiconductor fabrication. He is currently developing models and strategies for controlling the spread of pests and disease in agriculture.
Klaske van Heusden is a research associate at the University of British Columbia in Vancouver. An IEEE Senior Member, her research interests include modeling, prediction, and control, with applications in medical devices, mechatronics, and robotics. Lately she has been working on a robust and provably safe automated drug-delivery device for use in operating rooms.
Guy A. Dumont is a professor of electrical and computer engineering at the University of British Columbia in Vancouver and a principal investigator at BC Children’s Hospital Research Institute. An IEEE Fellow, he has 40 years of experience applying advanced control theory in the process industries, in particular pulp and paper and, for the last 20 years, in biomedical applications such as automated drug delivery for closed-loop control of anesthesia.
Global regulators have largely stepped out of diagnostics manufacturers’ way to enable them to quickly bring COVID-19 tests to the public. That has led to a deluge of testing options on the market, and in many cases, the reliability and accuracy of these tests is unclear.
That led us to wonder: Is anyone testing the tests?
We found one organization that’s on it. The Foundation for Innovative New Diagnostics, or FIND, headquartered in Geneva, Switzerland, is evaluating its way through a list of over 300 COVID-19 tests manufactured globally, and today published its first results.
FIND’s effort, which it is undertaking in collaboration with the World Health Organization (WHO), involves running thousands of coronavirus samples through the tests and comparing their performance against a gold standard. The organization will rank the tests based on sensitivity.
In late March, the FDA approved the use of Cepheid’s GeneXpert rapid molecular diagnostic machines to test for the new coronavirus. The automated modules—5000 of which are already installed in U.S. health facilities, while 18,000 are in operation in other countries—don’t require a lab facility or special training to operate. What’s more, they generate accurate results in about 45 minutes. The modules use disposable cartridges, pre-filled with the required chemicals that are channeled around test chambers using microfluidics.
While the cartridges had to be adapted to test for COVID-19, the microfluidic system itself dates back to the late 1990s, when Cepheid was cofounded in Silicon Valley by IEEE Fellow Kurt Petersen, a MEMS pioneer and winner of the 2019 IEEE Medal of Honor. (Cepheid went public in 2000 and Danaher acquired it in 2016.)
The system was just a prototype in September 2001, when letters containing anthrax spores began to arrive at the offices of U.S. senators and journalists. Cepheid won a set of competitions held by the U.S. Postal Service aimed at helping it prevent anthrax from getting into the mail system. To this day, Cepheid systems, attached to mail sorting machines, screen most mail in the U.S.
Since then, the company’s GeneXpert has been adapted to test for flu, strep, norovirus, chlamydia, tuberculosis, MRSA—and now, COVID-19.
Says Petersen: “The technology we developed is very powerful, because you can have all different kinds of cartridges for the instrument. Each cartridge can have a different design of its microfluidics and hold different chemicals, creating a PCR (polymerase chain reaction) specific to any DNA sequence you want, and so sensitive that you can detect just a few segments of that DNA in a milliliter of liquid.”
The GeneXpert test, like most COVID-19 tests to date, starts with a nasal sample taken with a swab. The person collecting the sample drops the swab into a liquid-filled specimen transfer tube. To start the test, liquid containing the sample is pipetted into a disposable test cartridge, and the cartridge is inserted into the test machine; this takes no special training. After this, the process is automatic.
Petersen explains how it works:
The test cartridge contains microfluidic channels; these are made out of plastic using high-precision injection molding. All the chemicals needed for the process are stored in chambers within the system. In the center of the cartridge, a rotary valve turns to open different pathways, while a tiny plunger—like a syringe–moves fluids in and out as needed.
So, the plunger pulls the sample into the center, the valve rotates, and the plunger pushes it into another region of the cartridge to do an operation on it.The system can do that multiple times, moving the sample to different regions with different chemicals, extracting RNA, mixing it with the reverse transcriptase that synthesizes complementary DNA that matches the RNA, and eventually pushing it into PCR reaction tube, where rapid heating and cooling speeds up the process of copying the DNA. Each new copy of the DNA gets a fluorescent molecule attached, which allows an optical system to determine whether or not the targeted gene sequence is in the sample.
Petersen, now an angel investor, says he’s gratified that a technology he worked to develop is being used to help address the pandemic. “The instrument hasn’t changed that much,” he says; “it’s pretty much what we designed 20 years ago.”
While Cepheid’s test for COVID-19 was the first approved in the U.S., Abbott has also received FDA approval for a five-minute test that runs on its ID Now analyzers. Roughly 18,000 of those are installed in U.S. healthcare facilities. According to a recent study on flu virus identification that compared the Cepheid and Abbott systems along with a similar technology from Roche, “the Cepheid test showed the best performance,” and was generally more sensitive. Such comparisons with respect to identifying COVID-19 are not yet available.
Months into the COVID-19 pandemic, the United States has finally moved from relying entirely on a single, flawed diagnostic test to having what may soon be an onslaught of testing options available from private entities. The U.S. Food and Drug Administration over the last three weeks has authorized the emergency use of more than 20 diagnostic tests for the novel coronavirus known as SARS-CoV-2.
The sudden arrival of the new coronavirus has caused shortages of ventilators, face masks, and respirators. And those shortages have now sparked automakers, quite unexpectedly, to enter the business of manufacturing critical medical equipment.
On the surface, it makes little sense: What do the makers of Mustangs and Chevrolet Volts know about ventilators? Automakers have been first to admit—not much at all.
“We’re not the experts here, but we can help the experts,” Mike Levine, Ford spokesman, said in a phone interview.
Instead of reinventing the wheel, automakers are leveraging their expertise in fast manufacturing, logistics, and supply-chain operations. Ford CEO Jim Hackett has said it currently takes GE about 27 hours to build a ventilator, but estimates Ford can cut production time in half, to around 13 hours.
Over the past week, companies around the world announced a flurry of AI-based systems to detect COVID-19 on chest CT or X-ray scans. Already, these deep learning tools are being used in hospitals to screen mild cases, triage new infections, and monitor advancing disease.
AI-powered analysis of chest scans has the potential to alleviate the growing burden on radiologists, who must review and prioritize a rising number of patient chest scans each day, experts say. And in the future, the technology might help predict which patients are most likely to need a ventilator or medication, and which can be sent home.
They made their first delivery of 97 shields on 22 March, and currently have enough capacity to produce between 500 and 1,000 shields a day, according to Jake Lee, a computer science masters student at Columbia University and spokesperson for the coalition called NYC Makes PPE. And they’ve some tips for other engineers and makers itching to put their skills and facilities to use.
Last Wednesday, Todd Goldstein was working on other projects. Then physicians in the New York-based hospital system where he works, hard hit by a surge in COVID-19 cases, told him they were worried about running out of supplies.
Specifically, they needed more nasal test swabs. A nasopharyngeal swab for COVID-19 is no ordinary Q-tip. These specialty swabs cannot be made of cotton, nor have wood handles. They must be long and skinny to fit up behind the nose into the upper part of the throat.
Goldstein, director of 3D Design and Innovation at Northwell Health, a network of 23 hospitals and 800 outpatient facilities, thought, “Well, we can make that.” He quickly organized a collaboration with Summer Decker and Jonathan Ford of the University of South Florida, and 3D-printing manufacturer Formlabs. In one week, the group designed, made, tested, and are now distributing 3D-printed COVID-19 test swabs.
France and wine—what an iconic link, and for centuries, how immutable! Wine was introduced by Greeks before the Romans conquered Gaul. Production greatly expanded during the Middle Ages, and since then the very names of the regions—Bordeaux, Bourgogne, Champagne—have become a symbol of quality everywhere. Thus has the French culture of wine long been a key signifier of national identity.
Statistics for French wine consumption begin in 1850 with a high mean of 121 liters per capita per year, which is nearly two glasses per day. By 1890, a Phylloxera infestation had cut the country’s grape harvest by nearly 70 percent from its 1875 peak, and French vineyards had to be reconstituted by grafting on resistant rootstocks from the United States. Although annual consumption of wine did fluctuate, rising imports prevented any steep decline in the total supply. Vineyard recovery brought the per capita consumption to a pre-World War I peak of 125 L in 1909, equaled again only in 1924. The all-time record of 136 L was set in 1926, after which the rate fell only slightly to 124 liters per capita in 1950.
Postwar, the French standard of living remained surprisingly low: According to the 1954 census, only 25 percent of homes had an indoor toilet. But rapidly rising incomes during the 1960s brought dietary shifts, notably a decline in wine drinking per capita. It fell to about 95 L in 1980, to 71 L in 1990, and then to 58 L in 2000—about half what it had been a century before. The latest available data shows the mean at just 40 L.
France’s wine consumption survey of 2015 shows deep gender and generational divides that explain the falling trend. Forty years ago, more than half of French adults drank wine nearly every day; now it’s just 16 percent, with 23 percent among men and only 11 percent among women. Among people over 65, the rate is 38 percent; for people 25 to 34 years of age, it is 5 percent, and for 15- to 24-year-olds, it’s only 1 percent. The same divides apply to all alcoholic drinks, as beer, liquors, and cider have also seen gradual consumption declines, while the beverages with the highest average per capita gains include mineral and spring water, roughly doubling since 1990, as well as fruit juices and carbonated soft drinks.
Alcoholic beverages are thus fast disappearing from French culture. And although no other traditional wine-drinking country has seen greater declines in absolute or relative terms, Italy comes close, and wine consumption has also decreased in Spain and Greece.
Only one upward trend persists: French exports of wine set a new record, at about €9.7 billion, in 2018. Premium prices and exports to the United States and China are the key factors. American drinkers have been the largest importers of French wines, and demand by newly rich Chinese has also claimed a growing share of sales. But in the country that gave the world countless vins ordinaires as well as exorbitantly priced Grand Crus Classés, the clinking of stemmed glasses and wishes of santé have become an endangered habit.
This article appears in the April 2020 print issue as “(Not) Drinking Wine.”
More than 100 COVID-19 patients at a hospital in Beijing are receiving injections of mesenchymal stem cells to help them fend off the disease. The experimental treatment is part of an ongoing clinical trial, which coordinators say has shown early promise in alleviating COVID-19 symptoms.
However, other experts criticize the trial’s design and caution that there’s not sufficient evidence to show that the treatment works for COVID-19. They say other treatments have far greater potential than stem cells in aiding patients during the pandemic.
Researchers have so far reported results from only seven patients treated with stem cells at Beijing You’an Hospital. Each patient suffered from COVID-19 symptoms including fevers and difficulty breathing. They each received a single infusion of mesenchymal stem cells sometime between 23 January and 16 February. A few days later, investigators say, all symptoms disappeared in all seven patients. They reported no side effects.
Jahar Bhattacharya, a professor of physiology and cellular biophysics and medicine at Columbia University, who was not involved in the work, says injecting mesenchymal stem cells into a patient’s bloodstream remains an unproven treatment for COVID-19 patients and could cause harmful side effects.
“You are injecting large numbers of cells in a patient’s veins,” Bhattacharya says. “If those cells go and clog the lungs, and cause damage because of the clogging—well, that’s not good at all.”
He adds that the study’s sample size is much too small to draw any meaningful conclusions about the treatment’s efficacy at this stage. “Folks do all kinds of things and they’ll say—we got a result,” Bhattacharya says. “It’s very risky to go by any of those.”
Kunlin Jin, a lead author in the trial and professor of pharmacology and neuroscience at the University of North Texas Health Science Center, says his group now has unpublished data from 31 additional COVID-19 patients who received the treatment. In every case, he claims, their symptoms improved after treatment. “I think the results are very promising,” he says.
According to Jin, 120 COVID-19 patients are now receiving mesenchymal stem cell injections in Beijing for the trial.
Jin’s team isn’t alone in considering the use of stem cells to treat COVID-19 patients. Another mesenchymal stem cell trial registered to clinicaltrials.gov aims to enroll 20 COVID-19 patients across four hospitals in China. The Australia-based firm Mesoblast says it’s evaluating its stem cell therapy for use against COVID-19. And in the United States, the Biomedical Advanced Research and Development Authority recently contacted the company Athersys to request information about its stem cell treatment called MultiStem for its potential as a COVID-19 therapy.
Mesenchymal stem cells (a term some experts criticize as too broad) can be isolated from different kinds of tissues and, once injected into a patient, grow into a wide variety of cells. They have not been approved for COVID-19 therapeutic use by the U.S. Food and Drug Administration.
The new coronavirus invades the body through a spike protein that lives on the surface of virus cells. The S protein, as it’s called, binds to a receptor called angiotensin-converting enzyme 2 (ACE2) on a healthy cell’s surface. Once attached, the cells fuse and the virus is able to infect the healthy cell.
ACE2 receptors are present on cells in many places throughout the body, and especially in the lungs. Cells in the lungs are also some of the first to encounter the virus, since the primary form of transmission is thought to be breathing in droplets after an infected person has coughed or sneezed.
However, cells from other parts of the body—including those which produce mesenchymal stem cells—lack ACE2 receptors, which makes them immune to the virus.
In many COVID-19 cases, a patient’s immune system responds to the virus so strongly, it harms healthy cells in the process. Jin explains that, once mesenchymal stem cells are injected into the blood, these cells can travel to the lungs and secrete growth factor and other cytokines—anti-inflammatory substances that modulate the immune system so it doesn’t go into overdrive.
But Lawrence Goldstein, director of UC San Diego’s stem cell program, says it’s not clear from the trial how many of the injected cells actually made it to the lungs, or how long they stayed there. He criticized the classification of patients in the study as “common,” “severe,” or “critically severe,” saying those categories weren’t well defined (Jin says these labels are defined by the National Health Commission of China). And Goldstein noted the lack of information about the properties of the stem cells used in the trial.
“It’s pretty weak,” Goldstein says of the trial design.
Steven Peckman, deputy director of UCLA’s Broad Stem Cell Research Center, adds: “Researchers and clinicians should use a critical eye when reviewing such reports and avoid the ‘therapeutic misconception,’ namely, a willingness to view experimental interventions as both safe and effective without the support of compelling scientific evidence.”
Jin himself doesn’t think most COVID-19 patients should receive stem cell infusions. “I think for the moderate patients, maybe don’t need the stem cell treatment,” he says. “For life-threatening cases, I think it’s essential to use mesenchymal stem cell treatment if no other drug is available.”
Goldstein says other potential treatments for COVID-19—such as drugs that modulate the body’s immune system—appear much more promising than stem cells. Many such drugs have been shown to be safe and effective at regulating the immune system and are already approved by regulatory authorities. It’s also easier to use drugs to treat a large number of patients compared with stem cell infusions.
“When you’ve got a hundred things you want to try, it’s not obvious that this one is on the short list,” Goldstein says of stem cell trials for COVID-19. “It’s a higher priority to test well-known immune modulators than to test these cells.”
Imagine this scene: It’s nearly dinnertime, and little Jimmy is in the kitchen. His mom is rushing to get dinner on the table, and she puts all the silverware in a pile on the counter. Jimmy, who’s on the autism spectrum, wants the silverware to be more orderly, and while his mom is at the stove he carefully begins to put each fork, knife, and spoon back in its slot in the silverware drawer. Suddenly Jimmy hears shouting. His mom is loud; her face looks different. He continues what he’s doing.
Now imagine that Jimmy is wearing a special kind of Google Glass, the augmented-reality headset that Google introduced in 2013. When he looks up at his mom, the head-up display lights up with a green box, which alerts Jimmy that he’s “found a face.” As he focuses on her face, an emoji pops up, which tells Jimmy, “You found an angry face.” He thinks about why his mom might be annoyed. Maybe he should stop what he’s doing with the silverware and ask her.
Our team has been working for six years on this assistive technology for children with autism, which the kids themselves named Superpower Glass. Our system provides behavioral therapy to the children in their homes, where social skills are first learned. It uses the glasses’ outward-facing camera to record the children’s interactions with family members; then our software detects the faces in those videos and interprets their expressions of emotion. Through an app, caregivers can review auto-curated videos of social interactions.
Over the years we’ve refined our prototype and run clinical trials to prove its beneficial effects: We’ve found that its use increases kids’ eye contact and social engagement and also improves their recognition of emotions. Our team at Stanford University has worked with coauthor Dennis Wall’s spinoff company, Cognoa, to earn a “breakthrough therapy” designation for Superpower Glass, which puts the technology on a fast track toward approval by the U.S. Food and Drug Administration (FDA). We aim to get health insurance plans to cover the costs of the technology as an augmented-reality therapy.
When Google Glass first came out as a consumer device, many people didn’t see a need for it. Faced with lackluster reviews and sales, Google stopped making the consumer version in 2015. But when the company returned to the market in 2017 with a second iteration of the device, Glass Enterprise Edition, a variety of industries began to see its potential. Here we’ll tell the story of how we used the technology to give kids with autism a new way to look at the world.
Vivaan’s mother, Deepali Kulkarni, and his father, V.R. Ferose, use the Superpower Glass system to play games with their son. Photo: Gabriela Hasbun
The system is built around the second version of the Google Glass system, the Glass Enterprise Edition. Photo: Gabriela Hasbun
The glasses’ head-up display gives Vivaan information about the emotions on his parents’ faces. Photo: Gabriela Hasbun
The system includes games such as “Capture the Smile” and “Guess the Emotion” that encourage Vivaan to interact with his parents and experiment with facial expressions. Photo: Gabriela Hasbun
Many families with autistic children struggle to get the behavioral therapy their kids need. The Superpower Glass system enables them to take the therapy into their own hands. Photo: Gabriela Hasbun
Vivaan, who is nonverbal, uses laminated cards to indicate which emotion he has identified. Most autistic kids who use the Superpower Glass system are verbal and don’t use such cards. Photo: Gabriela Hasbun
When Jimmy puts on the glasses, he quickly gets accustomed to the head-up display (a prism) in the periphery of his field of view. When Jimmy begins to interact with family members, the glasses send the video data to his caregiver’s smartphone. Our app, enabled by the latest artificial-intelligence (AI) techniques, detects faces and emotions and sends the information back to the glasses. The boundary of the head-up display lights up green whenever a face is detected, and the display then identifies the facial expression via an emoticon, emoji, or written word. The users can also choose to have an audio cue—a voice identifying the emotion—from the bone-conducting speaker within the glasses, which sends sound waves through the skull to the inner ear. The system recognizes seven facial expressions—happiness, anger, surprise, sadness, fear, disgust, and contempt, which we labeled “meh” to be more child friendly. It also recognizes a neutral baseline expression.
To encourage children to wear Superpower Glass, the app currently offers two games: “Capture the Smile,” in which the child tries to elicit happiness or another emotion in others, and “Guess the Emotion,” in which people act out emotions for the child to name. The app also logs all activity within a session and tags moments of social engagement. That gives Jimmy and his mom the ability to watch together the video of their conflict in the kitchen, which could prompt a discussion of what happened and what they can do differently next time.
The three elements of our Superpower Glass system—face detection, emotion recognition, and in-app review—help autistic children learn as they go. The kids are motivated to seek out social interactions, they learn that faces are interesting, and they realize they can gather valuable information from the expressions on those faces. But the glasses are not meant to be a permanent prosthesis. The kids do 20-minute sessions a few times a week in their own homes, and the entire intervention currently lasts for six weeks. Children are expected to quickly learn how to detect the emotions of their social partners and then, after they’ve gained social confidence, stop using the glasses.
Our system is intended to ameliorate a serious problem: limited access to intensive behavioral therapy. Although there’s some evidence that such therapy can diminish or even eliminate core symptoms associated with autism, kids must start receiving it before the age of 8 to see real benefits. Currently the average age of diagnosis is between 4 and 5, and waitlists for therapy can stretch over 18 months. Part of the reason for the shortage is the shocking 600 percent rise since 1990 in diagnoses of autism in the United States, where about one in 40 kids is now affected; less dramatic surges have occurred in some parts of Asia and Europe.
Because of the increasing imbalance between the number of children requiring care and the number of specialists able to provide therapy, we believe that clinicians must therefore look to solutions that can scale up in a decentralized fashion. Rather than relying on the experts for everything, we think that data capture, monitoring, and therapy—the tools needed to help all these children—must be placed in the hands of the patients and their parents.
Efforts to provide in situ learning aids for autistic children date back to the 1990s, when Rosalind Picard, a professor at MIT, designed a system with a headset and minicomputer that displayed emotional cues. However, the wearable technology of the day was clunky and obtrusive, and the emotion-recognition software was primitive. Today, we have discreet wearables, such as Google Glass, and powerful AI tools that leverage massive amounts of publicly available data about facial expressions and social interactions.
The design of Google Glass was an impressive feat, as the company’s engineers essentially packed a smartphone into a lightweight frame resembling a pair of eyeglasses. But with that form factor comes an interesting challenge for developers: We had to make trade-offs among battery life, video streaming performance, and heat. For example, on-device processing can generate too much heat and automatically trigger a cutback in operations. When we tried running our computer-vision algorithms on the device, that automatic system often reduced the frame rate of the video being captured, which seriously compromised our ability to quickly identify emotions and provide feedback.
Our solution was to pair Glass with a smartphone via Wi-Fi. The glasses capture video, stream the frames to the phone, and deliver feedback to the wearer. The phone does the heavy computer-vision work of face detection and tracking, feature extraction, and facial-expression recognition, and also stores the video data.
But the Glass-to-phone streaming posed its own problem: While the glasses capture video at a decent resolution, we could stream it only at low resolution. We therefore wrote a protocol to make the glasses zoom in on each newly detected face so that the video stream is detailed enough for our vision algorithms.
Our computer-vision system originally used off-the-shelf tools. The software pipeline was composed of a face detector, a face tracker, and a facial-feature extractor; it fed data into an emotion classifier trained on both standard data sets and our own data sets. When we started developing our pipeline, it wasn’t yet feasible to run deep-learning algorithms that can handle real-time classification tasks on mobile devices. But the past few years have brought remarkable advances, and we’re now working on an updated version of Superpower Glass with deep-learning tools that can simultaneously track faces and classify emotions.
This update isn’t a simple task. Emotion-recognition software is primarily used in the advertising industry to gauge consumers’ emotional responses to ads. Our software differs in a few key ways. First, it won’t be used in computers but rather in wearables and mobile devices, so we have to keep its memory and processing requirements to a minimum. The wearable form factor also means that video will be captured not by stable webcams but by moving cameras worn by kids. We’ve added image stabilizers to cope with the jumpy video, and the face detector also reinitializes frequently to find faces that suddenly shift position within the scene.
Failure modes are also a serious concern. A commercial emotion-recognition system might claim, for example, a 98 percent accuracy rate; such statistics usually mean that the system works well on most people but consistently fails to recognize the expressions of a small handful of individuals. That situation might be fine for studying the aggregate sentiments of people watching an ad. But in the case of Superpower Glass, the software must interpret a child’s interactions with the same people on a regular basis. If the system consistently fails on two people who happen to be the child’s parents, that child is out of luck.
We’ve developed a number of customizations to address these problems. In our “neutral subtraction” method, the system first keeps a record of a particular person’s neutral-expression face. Then the software classifies that person’s expressions based on the differences it detects between the face he or she currently displays and the recorded neutral estimate. For example, the system might come to learn that just because Grandpa has a furrowed brow, it doesn’t mean he’s always angry. And we’re going further: We’re working on machine-learning techniques that will rapidly personalize the software for each user. Making a human–AI interaction system that adapts robustly, without too much frustration for the user, is a considerable challenge. We’re experimenting with several ways to gamify the calibration process, because we think the Superpower Glass system must have adaptive abilities to be commercially successful.
We realized from the start that the system would be imperfect, and we’ve designed feedback to reflect that reality. The green box face-detection feature was originally intended to mitigate frustration: If the system isn’t tracking a friend’s face, at least the user knows that and isn’t waiting for feedback that will never come. Over time, however, we came to think of the green box as an intervention in itself, as it provides feedback whenever the wearer looks at a face, a behavior that can be noticeably different for children on the autism spectrum.
To evaluate Superpower Glass, we conducted three studies over the past six years. The first one took place in our lab with a very rudimentary prototype, which we used to test how children on the autism spectrum would respond to wearing Google Glass and receiving emotional cues. Next, we built a proper prototype and ran a design trial in which families with autistic kids took the devices home for several weeks. We interacted with these families regularly and made changes to the prototype based on their feedback.
With a refined prototype in hand, we then set out to test the device’s efficacy in a rigorous way. We ran a randomized control trial in which one group of children received typical at-home behavioral therapy, while a second group received that therapy plus a regimen with Superpower Glass. We used four tests that are commonly deployed in autism research to look for improvement in emotion recognition and broader social skills. As we described in our 2019 paper in JAMA Pediatrics, the intervention group showed significant gains over the control group in one test (the socialization portion of the Vineland Adaptive Behavior Scales [PDF]).
We also asked parents to tell us what they had noticed. Their observations helped us refine the prototype’s design, as they commented on technical functionality, user frustrations, and new features they’d like to see. One email from the beginning of our at-home design trial stands out. The parent reported an immediate and dramatic improvement: “[Participant] is actually looking at us when he talks through google glasses during a conversation…it’s almost like a switch was turned.… Thank you!!! My son is looking into my face.”
This email was extremely encouraging, but it sounded almost too good to be true. Yet comments about increased eye contact continued throughout our studies, and we documented this anecdotal feedback in a publication about that design study. To this day, we continue to hear similar stories from a small group of “light switch” participants.
We’re confident that the Superpower Glass system works, but to be honest, we don’t really know why. We haven’t been able to determine the primary mechanism of action that leads to increased eye contact, social engagement, and emotion recognition. This unknown informs our current research. Is it the emotion-recognition feedback that most helps the children? Or is our device mainly helping by drawing attention to faces with its green box? Or are we simply providing a platform for increased social interaction within the family? Is the system helping all the kids in the same way, or does it meet the needs of various parts of the population differently? If we can answer such questions, we can design interventions in a more pointed and personalized way.
The startup Cognoa, founded by coauthor Dennis Wall, is now working to turn our Superpower Glass prototype into a clinical therapy that doctors can prescribe. The FDA breakthrough therapy designation for the technology, which we earned in February 2019, will speed the journey toward regulatory approval and acceptance by health insurance companies. Cognoa’s augmented-reality therapy will work with most types of smartphones, and it will be compatible not only with Google Glass but also with new brands of smart glasses that are beginning to hit the market. In a separate project, the company is working on a digital tool that physicians can use to diagnose autism in children as young as 18 months, which could prepare these young kids to receive treatment during a crucial window of brain development.
Ultimately, we feel that our treatment approach can be used for childhood concerns beyond autism. We can design games and feedback for kids who struggle with speech and language, for example, or who have been diagnosed with attention deficit hyperactivity disorder. We’re imagining all sorts of ubiquitous AI-powered devices that deliver treatment to users, and which feed into a virtuous cycle of technological improvement; while acting as learning aids, these devices can also capture data that helps us understand how to better personalize the treatment. Maybe we’ll even gain new scientific insights into the disorders in the process. Most important of all, these devices will empower families to take control of their own therapies and family dynamics. Through Superpower Glass and other wearables, they’ll see the way forward.
This article appears in the April 2020 print issue as “Making Emotions Transparent.”
About the Authors
When Stanford professor Dennis Wall met Catalin Voss and Nick Haber in 2013, it felt like a “serendipitous alignment of the stars,” Wall says. He was investigating new therapies for autism, Voss was experimenting with the Google Glass wearable, and Haber was working on machine learning and computer vision. Together, the three embarked on the Superpower Glass project to encourage autistic kids to interact socially and help them recognize emotions.
The collective thoughts of the interwebz
The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.