Creating a virtual experience that’s as close to real life as possible is one of virtual reality’s (VR’s) ultimate goals, as companies and researchers find ways to accurately mimic a person’s facial expressions and body gestures. But if the lighting and shadows on faces—notoriously tricky to simulate with accuracy—are all wrong, expressions and gestures don’t matter. It just looks wrong. So how to simulate light, shadows, and reflections on the virtual human face?
“Oftentimes, when you take a photo of someone and add lighting to it, the image looks unnatural because the lighting on the person and the lighting on the environment don’t match,” says Victor Lempitsky, associate professor and head of the computer vision group at Skoltech. “This mismatch in lighting becomes a problem. It affects realism, suspending the impression of looking at a real person. That’s why we decided to investigate how to make those portraits relightable.”
The team’s process involves taking a video-recorded orbit around a person standing still, with the flash going off at periodic intervals. A point cloud is then generated, and an algorithm called neural point-based graphics takes care of the 3D reconstruction. In other words, a deep neural network processes images and predicts lighting-related properties such as albedo and shadows based on the room lighting. These maps then help to relight images from different viewpoints and under various lighting conditions. As an example, using this technique, a 3D face might be lit and shadowed via a point light source, or an ambient source from light in all directions, or a directional source coming from, perhaps, a large window.
“The main appeal of neural point-based graphics is its robustness and suitability to a wide variety of geometries,” says Lempitsky. Polygonal meshes are still the most common geometric representation of 3D objects in traditional computer graphics—meshes are well supported and typically allow for fast rendering. But mesh-based approaches often fail on thin objects or those with a small diameter, such as hair threads, fingers, or small sheets of cloth. Point-based graphics have a similar problem: The light and shadows they generate on a face can still have small holes in it, because the mesh is not dense enough.
“Combining point-based graphics with neural rendering helps overcome these challenges, with the neural network deciding how to connect points locally during rendering,” Lempitsky says.
Unlike other approaches which focus on a single view, require specific lighting, or call for the use of sophisticated equipment such as lidar instruments or light stages, the process devised by Skoltech needs only a smartphone. “Not everyone has access to high-end setups in studios or has knowledge of complex techniques like photogrammetry,” says Artem Sevastopolsky, a Ph.D. student at Skoltech and primary author of the research. “One of our motivations was to make data acquisition simpler.”
These photorealistic portraits veer into uncanny valley territory while potentially sowing the seeds for future deepfake mischief. “Our work is in line with the uncanny valley concept, but we quite accurately simulate the face. In this way, it’s hard to frighten anyone with the 3D portraits we create,” Sevastopolsky says. “This field of synthesizing facial images does have a connection to deepfakes, but I don’t think our process can be used for that because it doesn’t simulate lip motions or facial expressions.”
To further advance their research, Sevastopolsky and the computer vision group at Skoltech are looking into applying their process to full body relighting. And aside from creating more realistic VR experiences, the team’s process can also be used for 3D reconstruction of objects or environments.
“This is important for cultural heritage preservation, for example,” says Evgeny Burnaev, associate professor and head of the advanced data analytics in science and engineering group at Skoltech. “We can use simpler equipment instead of costly and complicated scanners to quickly capture things. The corresponding 3D model can then be put on a website to attract more people to a certain historical place, or allow people with limited abilities or who can’t otherwise visit that place to view and experience it virtually.”
Using a 3D printer to make tiny translucent lotuses with millimeter-size petals: It’s eye-catching, but then again, printers are pretty good these days.
But to do it—the “it” being the process of building objects using a solid which becomes liquid after squeezing, flows through a needle, and reverts to a solid as it dries—with aerogel, a material that’s more than 99 percent air is something else. Swiss researchers recently announced an additive manufacturing breakthrough with the material after three years of working on it. Potential applications include building tiny, thin superinsulators that save space on circuit boards and creating bespoke shapes that would help keep medical implants cool inside the body.
The researchers pulled back the curtain this August on a set of intricate demonstrator aerogel grids, cubes, and lotuses made using 3D microprinting techniques delivering structures as thin as one-tenth of a millimeter.
Aerogels are nanoporous solids; NASA sometimes calls them liquid or frozen smoke (the space agency’s made extensive use of aerogels including a mission that used them to capture interstellar comet dust). They can made from silica, graphene or any number of other base materials.
Silica aerogels are among the world’s best thermal insulators and a go-to object of Internet attention. Among the most dramatic illustrations of aerogels’ abilities are clips of flowers shielded from blowtorches or people protected from flamethrowers. Different base materials yield different aerogel properties: graphene or gold aerogels yield conductive paper, but make lousy insulators. Silica is what’s called for to ward off heat. But aerogels are as finicky as they are fascinating. They’re difficult to handle and expensive to make in large quantities. Touch them too often and they crumble. They can be molded to more or less small sizes, but heretofore couldn’t be machined into tiny objects.
But teams of researchers—from Empa, the Swiss materials science and technology lab, along with colleagues at ETH Zürich, the federal tech research institute, and at the Paul Scherrer Institute in Aargau—were aware of the potential applications and wanted to see if they could shape aerogels at micro size. These teams have been working with aerogels in different capacities for the past decade, and developed an aerogel-based plaster for insulating restored historic or listed buildings.
“If you can miniaturize aerogel, the cost aspects become much different. With one cubic meter, you could make a million components for a cell phone or something—the materials cost doesn’t matter anymore,” says Wim Malfait, leader of Empa’s superinsulation materials group. “The issue is how to make them. How do you make the components in the shape, size and format that you need them?”
The Empa team’s recipe for 3D printing takes off-the-shelf silica aerogel powder and adds it to a pentanol solvent with a silica precursor to bind the powder together. They first make an ink by using a spatula, then spin it in a food mixer at 3,000 rpm for five minutes. The result is a paste. Putting this paste under pressure—squeezing it—turns it into a liquid slurry that can flow easily through a printing needle. This is called shear thinning. After flowing into the needle, it can be printed out in the desired shape, layer by layer. Once the tiny object has been formed, they use ammonia gas to make the object into a gel. Then, a special drying process removes the solvent. There you have your aerogel lotus with millimeter-size petals. The group recently laid out its results and a few samples in a paper in the journal Nature.
Malfait says the Swiss institutes are now working with commercial industry partners on feasibility studies for the insulators.
Thermal insulation is a complex field, and there are other ways to diffuse or control heat without custom engineering. There are also other aerogel efforts underway, for example in Singapore, where Hai Duong’s group is using waste products including cotton waste and pineapple leaf in aerogel-making. Aerogels’ 90-year-history is full of false starts down the road toward hoped-for applications, but engineers keep at it: Their properties look too good for them to remain a novelty.
Germany’s BASF and Massachusetts-based partner Aspen Aerogels are building a line of aerogel-based insulator mats. And the doors are open for tiny, custom-made, inexpensive insulators—if industry wants it. Richard Collins, a tech analyst at IDTechEx, offers the reminder that market pull is required for real commercial success, rather than material push.
Tiny, heat-resistant lotus petals, though—they’ll get people looking.
Greg stood at the base of the tall granite crag, his climbing harness arrayed with carabiners, anchors, and rope, his ever-present Tilly hat crumpled under an orange climbing helmet. As usual, he had a flock of novice climbers in attendance, all nervous about what the day would hold. It was July of 2017, and I was one of them. The most reassuring item by far was not any of the safety gear, but the impish grin on Greg’s face, which conveyed a reassuring message: It’s okay; this will be fun; I’ve got you.
We were spending a week on California’s Pacific Crest Trail, traversing a part of the Sierras north and west of Lake Tahoe. The terrain was rugged, the weather was hot and dry, and both the fitness level and experience of the 20 people involved varied greatly. Greg was our leader.
Some context is probably useful. Every year, the American Leadership Forum curates a class of 20 to 25 people. The goal is to have people with a mélange of skills, backgrounds, and perspectives study and interact in hopes that in the end the group will be more enlightened, aware, and cohesive—and thus more capable of effecting positive change in the community.
Our cohort was typical: four civil servants, three business people, three health care professionals, three religious leaders of varied faiths, three entrepreneurs, three non-profit executive directors, two educators, and one mayor. Class XXXV was deliberately diverse in race, gender, religion, sexual orientation, and lifestyle.
Greg was not only our guide up the mountain, he was also the leader of our broader journey. Together we learned about mindfulness, aikido, compassion, drumming, the structure of cities, culture in America, and the obligation of the individual in a collective. He rang the gong, gently nudged the conversation, and slowly but surely became a trusted and dear friend.
One night I was helping with dinner preparation after a long day on the trail. Greg sat down next to me and began massaging his right arm. He was one of the most vigorous people I knew—whether biking, hiking, or climbing—so I assumed he just had a bit of muscle soreness. I asked him about it, and he said that it had been bothering him for a few weeks. Greg went on to say that he had a surgical procedure scheduled to address the underlying pinched nerve when he returned home to Oregon.
I didn’t see Greg for a couple of months, until well after his surgery. When we reconnected in class, I was surprised to see him squeezing an exercise ball and reporting that his arm wasn’t feeling better. If anything, the pain and weakness were getting progressively worse. More ominously, he said, the doctors wanted to run some additional tests.
A week later we all heard the devastating news: Greg had ALS, and it was the faster moving of the two varieties.
Amyotrophic lateral sclerosis is a disease that affects nerve cells in the brain and spinal cord, causing progressive loss of muscle control. In North America, it’s often called Lou Gehrig’s disease, after the baseball player of the 1920s and ’30s who was famously diagnosed with it in the prime of his career. Medical professionals usually don’t know why ALS strikes, although some cases are inherited. It often begins with muscle twitching and weakness in a limb, or slurred speech. Eventually it affects the muscles needed to move, speak, eat, and breathe. There is no cure: It is invariably fatal.
This diagnosis, obviously, was an enormous gut-wrenching shock, and the exigencies of Greg’s life were instantly and inexorably altered. He would need increasing amounts of care, his livelihood was in doubt, and his family had to prepare for his inevitable downward trajectory.
Greg’s disease progressed quickly. The next time I saw him he couldn’t lift either arm and needed assistance balancing. His daughter moved home to accompany him on select work assignments, but it was clear that his life as an independent person was drawing to a close.
Being an engineer, a tech entrepreneur, and most recently a venture capitalist, I was maddeningly unqualified to assist with anything medical. But I felt compelled nonetheless to try and improve Greg’s quality of life. My initial idea was vague: to help Greg regain a bit of independence and get him back into the outdoors, where in the past he had so thrived.
After I ruminated for a few days, a solution began to emerge. I would build Greg a recumbent tricycle that could accommodate his declining abilities. It would be a trike rather than a bike, so balance wouldn’t be needed. It would be electrified to augment his weakened but still functioning leg muscles. And (here’s where my special contribution came it), it would have joystick controls, because his only remaining granular dexterity was in two fingers on his left hand. A higher-order goal was to create an integrated solution that was simple to operate and maintain, so that he’d be able to use it as much as possible.
After researching the various recumbent trikes available, I settled on a frame from Inspired Cycle Engineering (ICE) in the UK. I chose the company’s “Adventure” model which has underslung steering linkages, and a fairly traditional drum brake setup. I ordered one, which showed up at my doorstep three weeks later.
At this early point of the design phase, I enlisted some help from Greg, who was very clear about the extent and nature of his disability. The right side of his body was almost completely impaired, but the large muscle groups in the shoulders still functioned and would remain working for the longest time in the future. We settled on the idea that gear selection would somehow be controlled by a shift of his shoulder. Steering and braking would be handled by two fingers on his left hand.
My original idea for shifting was to adapt an electronic derailleur from Shimano—the company’s Di2 system. But I discovered that it was totally proprietary, which would have made it difficult to reverse engineer. Fortunately, I found a company in California named Archer Components that sells a system to automate a manual derailleur. It comes complete with open APIs and Bluetooth, so it was easy to interface wirelessly with paddles on the left arm rest that Greg could operate by moving his shoulder.
I next turned to consider steering and braking. Fortunately, the force profiles and absolute travel required for both are well within the range of off-the-shelf linear actuators, which I purchased from Firgelli Automations. Still, I had a bunch of decisions to make: Should the control of steering and braking be open or closed loop; should the brake on each front wheel be operated by its own actuator, or should a single one control both. And what would be the best way to translate the motion of the joystick into braking and steering action so as to give Greg a safe and fully useful trike.
After numerous design iterations and test rides around the parking lot, I settled on open-loop controls with one actuator operating both front brakes following a fairly linear displacement curve. That is, the amount each brake cable is pulled is roughly proportional to amount Greg displaces the joystick. For steering, though, I programmed in an exponential response curve, meaning that small deflections of the joystick to the left and right would give Greg fine-grained steering control, but he’d still be able to generate plenty of throw when he moved the joystick farther to the side. This arrangement seemed to best match the dexterity of his left hand. And in any case, these curves could be quickly adjusted with a software update.
For the electric-assist, I used Bafang’s mid-drive system. (It’s called “mid drive” because in a normal bike, the motor is mounted in the middle where it turns the chain, rather than powering the front or back wheel directly.) This choice came with several advantages. First, it allowed Greg to select a level of pedal assistance from almost nothing, to fully electric. Second, it has a relatively large battery pack, which could be used to power the entire system and still provide a 50-mile cruising range. Powering steering and braking off the same battery, though, could have created a special kind of range anxiety. So I engineered a battery-management system that cut power to the drive motor when the battery was depleted, ensuring there’d always be enough power for steering and braking.
Integrating these various components was straightforward. I flashed the software onto the boards, packaged the electronics in a water-resistant case, wired things up, and fabricated a custom seat, armrest, and left-hand controller. The only thing left was to ship it to Greg in Oregon and hope for the best.
Three days later, I received a message from Greg saying that he had received the big box and was getting some help assembling the trike, which I had to dismantle in part for shipping. A few days after that, he reported the results of his first ride: “OMG! I am still basking in the glow of that first test drive! You guys don’t know how grateful I am. The freedom, ability to move, adventure I am imagining on the bike fills me with so much joy. Thank you. Thank you. Thank you.”
Greg’s ALS has continued to progress. His speech is now slurred, and walking is becoming a challenge. But he is still able to take his trike out on nearby trails for an hour a day, thanks to its various cycle-by-wire modifications.
This project is the perfect example of why I’m proud to be an engineer. To identify a problem, conceptualize a solution, build it, refine it through testing, and deliver something of value to the end user is a reward unto itself. In this particular case, I was also able to do a much-appreciated favor for a friend in need. While I wish could have somehow helped Greg medically, that was beyond my skill set. Getting him back cycling, though, was right up my alley.
I should mention that after Greg posted some images of the finished trike on social media I was immediately deluged with expressions of interest from other people with ALS and also from some manufacturers of recumbent tricycles. While I have no desire to produce such trikes commercially, I have offered my CAD files, source code, and thoughts to anyone who has approached me in hopes that other people with disabilities may benefit from this project. I know it would please Greg no end if it did.
Think back to when you were a young and an eager beginner in technology. Remember the first time you took apart your first PC, wrote your first line of code, learned how to hack Doom. The easiest way to learn technology was (and is) by being hands-on.
“Hands-on learning is 15X more effective than passive learning (ie. lectures).”
Learning Technology Today
Getting started in technology can be intimidating. If you want to learn technology these days, there aren’t many great options.
Since these are purely digital, you don’t get the hands-on experience of building electronics. These often end up being just coding lessons.
Most schools don’t even offer electronics and programming in their curriculum. Even fewer engage students in hands-on learning. If you’re lucky, you might find a school that has an afterschool program led by a passionate STEM educator.
“There are nearly 500,000 open Computing jobs in the U.S. alone.”
Educational Hands-on Products
You can find some hands-on projects that use block code and snap-on parts, but these are often oversimplified to the point that you don’t even learn the fundamentals. When you remove the potential of making mistakes, you lose the connection to how things work in the real world.
What happened to the good ole days of getting your hands dirty with real hardware and programming?
The truth is, people learn best by doing.
That’s why we created Creation Crate, atech subscription boxthat prepares learners for the jobs of the future by teaching them how to build awesome DIY electronic projects!
Here are 5 reasons why Creation Crate is the perfect gift for your aspiring engineer!
An Educational Hands-on learning curriculum
Would you rather build your own bluetooth speaker, or read a textbook on electronics? Hands-on learning is not only more engaging, but fun too. Learning shouldn’t be a chore, and Creation Crate makes sure of that.
Creation Crate combines hands-on learning with educational electronics courses to teach electronics, circuits, coding, critical thinking, problem solving, and more!
“I am majoring in STEM (physics and computer science double-major), and I ordered this mainly for the purpose of tinkering and expanding my computer engineering knowledge through independent projects, and even for me, this bundle ended up being incredibly handy and interesting. Thank you guys, and good luck in the future!” – Roman F.
Introduces real-world hardware
If there was ever a time to be an aspiring engineer, it’s now! The cost of components is a fraction of what it was ten years ago. Anyone can get access to the hardware and software used in everyday tech careers.
So why settle for anything but the real thing?
With Creation Crate, everything necessary is delivered in a kit to your door. Kits include all the components needed to build your project. You will also find access to an online classroom with detailed step-by-step video tutorials.
Each project uses an Uno R3 (Arduino-compatible) Microcontroller, a small programmable computer that acts as the brain of the project.
They’ll also learn how to use components like a Breadboard, Ultrasonic Sensor, LED Matrix, 7-Segment Display, Accelerometer, Distance, Pressure, Temperature, & Humidity Sensors, LCD Screen, Keypad, Microphone Module, Resistors, Servo Motors, Motor Driver Board, and more!
Teaches popular programming languages
Learning how to program is a tedious but rewarding process. Most engineering careers in technology require an understanding of programming languages like Java, C++, Python, Ruby, and others.
With Creation Crate, students will learn how to write their own computer programs in the Arduino language (C/C++) to make their projects come to life!
Each project will introduce different lessons in programming C++. Here are a few examples of what they’ll learn:
What are Comments and Variables?
Arrays and Functions
Detecting variable input values
As your aspiring engineer progresses through the curriculum, they’ll learn how to build and program electronic projects that become more challenging as they learn new lessons. They’ll learn how to build things like…
A color-changing mood lamp that activates when the lights are off
An optical theremin that let you create music simply by waving your hands
A Bluetooth speaker that plays music from your phone
A rover bot that avoids obstacles and follows lines
By the end of the curriculum, they’ll have more hands-on learning experience in hardware and programming than many students receive in a four year degree!
“Creation Crate fills a void that has existed in the Tech Subscription box world. Most kits are aimed at the very young or adults with extra income. Creation Crate is affordable and challenges tweens and teens (and at least one adult!). I especially appreciate the manner in which the project challenges build from month to month.” – Justin D.
Great family activity
Family activities create everlasting memories. The key to a great family activity is to do something that everyone enjoys and is interested in.
Unlike every other “learn electronics kit” out there, this isn’t just for kids and teens. Even adults will find the projects fun and challenging! That’s why Creation Crate makes the perfect family activity for parents looking to spend more quality time with their child.
“By high school, a child will have used up 90% of in-person parent time” – Tim Urban (Author Wait But Why)
Unpack Their Potential
Americans spend almost $13 billion on unwanted presents each year. Why not gift something that’s not only fun, but will help develop a lifelong skill?
As IEEE Spectrum reported at the end of last year, pinball is having a revival, driven in part by the shift to e-commerce, which is turning erstwhile big-box retail stores into cheap real estate for family entertainment centers. Modern pinball machines, with enhancements like upgradable software, are vastly more sophisticated than their electromechanical ancestors. Stern Pinball is in the vanguard of this renaissance, making home and arcade versions of many of its games. The latest title available in a home version is the US $4,500 Star Wars Pin, based on comics artwork and models inspired by the original movie trilogy.
Classic Microprocessor Kit
Wichit Sirichote is a professor at King Mongkut’s Institute of Technology, in Bangkok. He’s also the maker of a terrific line of single-board computers based on classic CPUs such as the 6502, Z80, and 8088. Prices range from $85 to $175, depending on the CPU. They are bare-bones machines designed for learning and protoyping, but they are very flexible: You can upload code through an RS-232 port, plug in a standard LCD character display directly using an onboard connector, and add other custom hardware via a bus-expansion slot. Sirichote wrote his own monitor software for the boards that lets you, for example, examine the contents of CPU registers, and extensive documentation is available.
Musical Tesla Coil Kit
It’s not going to win awards for the quality of its sound, but it is a crowd pleaser. OneTesla’s $400 Musical Tesla Coil Kit can be driven directly by a MIDI-enabled instrument or play a MIDI file. The frequency of the notes is used to modulate the output of the coil with a square wave, producing buzzing notes and impressive sparks over half a meter long. A smaller version is also available for $230.
Doppel is worn like a watch, except you wear it on the inside of your wrist rather than on the outside. A rotating weight inside creates a rhythmic vibration. The purpose of the $280 wearable is to improve focus and reduce stress, by using rhythms that are faster or slower, respectively, than your normal resting heart rate. (The accompanying smartphone app measures your heart rate when you place your finger over your phone’s camera lens and looks at changes in the ambient light that gets filtered through.) It did help reduce my anxiety levels somewhat, but I found it worked best with deliberate mindfulness techniques, so if you’re not already familiar with those, your mileage may vary.
Piper Computer Kit
Minecraft is already used to introduce children to writing software. The $300 Piper Computer Kit, aimed at 8- to 13-year-olds, extends that idea to hardware. Kids first assemble the wooden case and plug together the basic components of the kit, which is based on a Raspberry Pi and comes with its own screen. While the kit includes a mouse, there is no keyboard. Instead a breadboarding module is provided, which can be used to, for example, wire up buttons to control events in Minecraft through a series of game levels.
If you’re looking for a nonfiction book to give, try one from the Platform Studies series from MIT Press. These books describe influential platforms in the history of digital media, examining how the specific technical details and hardware capabilities of each platform (or, in academia-speak, “affordances”) shaped the software that ran on them and how that combination in turn affected the industry and wider culture. The most recent 2019 title (The Media Snatcher) dissects the PC Engine/TurboGrafx-16 console. The highlights of the series so far for me are Racing the Beam, about the Atari 2600, and Minitel.
In the fiction department, Spectrum’s recommendation is Fall, or Dodge in Hell (William Morrow, 2019) by Neal Stephenson, author of the cyberpunk classic Snow Crash. Fall is a sequel of sorts to his 2011 novel, Reamde, but it can be read completely independently of Reamde (and is in fact a much better book). Reamde is an entertaining enough technothriller, but Fall is Stephenson at his best, weaving together deep philosophical questions against the background of a compelling vision of the future (a chapter featuring a journey across an America that’s been utterly fragmented by competing social-media feeds is plausibly chilling). In Fall, the lead character awakens in a digital afterlife, in which his first order of business is to create a universe to live in.
This article appears in the December 2019 print issue as “2019 Holiday Gift Guide.”
The collective thoughts of the interwebz
The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.