Adobe Character Animator Lets You Make Cartoons Speak With Face Tracking
- View Original
- April 9th, 2015
Facial recognition is one of the earliest skills that baby humans learn. Considering how important paying attention to faces is to our survival it is no surprise that computer vision researchers have put a lot of effort into this field of research. The applications of static facial recognition have gone from enabling digital cameras to improve the focus on multiple faces in an image to Facebook’s auto-identification of people in photo galleries. Face tracking is the ability to recognize the geometry of facial features in moving images and follow them despite changes in angle or lighting. Adobe’s new Character Animator app, announced today for its Creative Cloud service, uses advanced face tracking to create animated effects that are downright playful.
I first saw an early version of the app, code-named “Animal,” demoed as part of a high-profile collaboration with Microsoft Microsoft at Adobe’s MAX conference last October. The presenter showed how he could use the camera on a Surface Pro 3 Tablet to capture his facial expressions. He then applied these expressions in real time to a 2d graphic image of a spiky (though cute) red “animal.” This demo was part of the same session where Microsoft CEO Satya Nadella and Adobe president and CEO Shantanu Narayen shared the stage to announce a partnership that involved giving a fully-loaded Surface Pro 3 to each attendee. (N.B. this did not include press attendees.)
Even more striking than the motion tracking was the ability to lip-synch the character’s mouth to the animator’s speech. Seeing all of this work so smoothly in real time—on a Microsoft tablet no less—was pretty breathtaking. The audience was wowed.
Six months later, Adobe has taken what seemed a cute tablet app and released it as a full desktop companion to After Effects, a key part of its first-rate video production toolbox. I recently sat in on a demo of Character Animator along with a bunch of other new Creative Cloud video features that Adobe is premiering at the upcoming National Association of Broadcasters (NAB) Show in Las Vegas. Many of the new features improve the capture and sharing of sophisticated color looks, but Character Animator is my personal favorite.
The powers of machine intelligence have inspired a lot of interest in how algorithms can push human creativity. Dan Buczaczer, who manages creative partnerships at VivaKi/Publicis in San Francisco, wrote a great post on the subject last month in advance of a similarly themed panel at SXSW. Buczaczer finds that the most satisfying applications address open-ended creativity in partnerships between humans and computers. Far from evil AI robots stealing our jobs, these are benevolent AIs that help us do our jobs better.
Character Animator falls into this quadrant of applications in a subtle way. The intelligence here is that the program has an invariant idea of a face that it can recognize and track across the range of human faces. The software can also map the behavior of the human face that it captures onto the moving parts of an illustrated image. (Note that the human needs to construct the image in layers and label them according to set conventions for this to work.) The human collaborator both creates the graphic character and the facial movements and speech. These functions, of course, can be performed by different humans, but part of the power here is that many creatives are good enough at both to create a highly personal expression with this app.
This last point is key to what Adobe is trying to achieve with its Creative Cloud overall. By giving an illustrator (or even an actor) the ability to create character animations with face tracking, Adobe is helping to break down the barriers between creative disciplines. By incorporating sophisticated machine vision into these products, Adobe is removing technical barriers as well. Character Animator is a democratized form of the same kinds of technology that power major Hollywood animated movies. Face tracking is what makes us recognize Rihanna, for instance, as the feisty girl in the forthcoming Home from Dreamworks. Traditionally, creating character animations has involved either painstaking drawing and redrawing or complicated “rigging” of digital models. Like many other recent front end innovations, Character Animator makes things happen declaratively instead of imperatively. You no longer have to worry about keyframes. You just label your model properly and start performing.
It is easy to imagine that the behavior data that Character Animator captures will one day allow creatives to use machine learning on the front end as well. In Buczaczer’s post, he includes an amazing video of robotic music researcher Mason Bretan at Georgia Tech playing a Miles Davis-inspired fusion tune with a collection of adorable robot collaborators. Watch the video (below) and think about how a later version of Character Animator might be able to improvise the behavior of a set of background characters that follow the human-driven character’s lead. Such an app would be an interesting animal indeed.