Intro

Today’s shot breakdown is a complete CG face replacement, similar to the technique used for the “Two Face” character in “The Dark Knight”, except it took one day and $0. And it’s infinitely simpler and less impressive- but, hey, $0! Hard to beat that.

The footage is courtesy of FXPhD, which is a fantastic resource.

MoCap in PFTrack

Camera 1 of 2 as matchmoved in PFTrack - Click to play

The first stop for this shot was PFTrack to capture the motion of the tracking dots on the actor’s face. The footage was filmed on two HD cameras; the image above is from the left camera. The right camera was timed to match the footage from the left camera. When the facial dots are tracked and identified in both cameras it allows the matchmoving software to “triangulate” (although with only two points in this example- so what’s a two sided triangle?) the location of the dots in 3D space and follow them as the movie. These cameras are called “witness” cameras, a moving main camera could also have been used, although in this case it wasn’t provided.

A lot of the dots here are useless because the cameras’ placements don’t allow for them to be seen by both cameras at the same time. For production purposes I would really recommend 3 to 4 cameras to ensure 2 cameras on each dot at all times. I’d also have liked more dots around the lips to better capture their motion as he delivers his dialogue. I got good results for the center dots, which is perfect since I was only really interested in modifying that portion of his face.

Geometry Placement in Maya

Base geometry in Maya- Click to Play

I exported the resulting camera and point locations to Maya as a .MA file. In Maya I constructed quick geometry that combined the tracking points into a single mesh, then removed the nose for the purposes of my shot. I rigged my mesh so that the motion of the tracking locators exported by PFTrack were controlling the motion of the geometry. This could be done through bones and skinning or a series of constraints and deformers.

I then added geometry for my Pigman’s nostrils into the mesh. I projected the UV’s from the camera in order to make texture generation simpler in Photoshop. I also exported the UV’s themselves to help choose where to fade out my painting.

Texture creation in Photoshop

Skin texture as generated in Photoshop - Click to enlarge.

In Photoshop I used a combination of the clone tool and various brushes to paint out the nose and try to match as close as possible the lighting and skin tones of the original frame. I also removed the eyebrows in the same fashion.

Next I started adding in wrinkles and skin texture, largely by multiplying in and deforming photographs of elephant skin.

The final step was to create and alpha, which I did by consulting the UVs I had exported from Maya (to make sure my texture faded softly before leaving the geometry boundaries).

The final image was exported from Photoshop as a TIF with an alpha channel and reimported as a texture into Maya.

The texture with alpha mapped onto geometry - Click to play

The shader in Maya was really simple- since I wasn’t lighting the model, but simply reusing the lighting from the texture, I simply created a light in Maya and set it’s intensity to zero. This override’s Maya’s default lights, which are generally on if there are no user created lights in the scene. Then I attached the image to the shader with three connections- transparency (to bring in our alpha channel), and incandescence and ambient color. These last two allowed the object to render at exactly the original brightness. I’m sure there’s a slicker way to do this, but this was is so easy I’ve never bothered to research a more proper technique. I’m all about cheats and workarounds.

I also rendered an ambient occlusion pass to capture the nostrils (honestly, the nostrils were kind of an afterthought that was tacked on at the very end).

AfterEffects

The complete node tree in AfterEffects- Click to enlarge.

I actually used Nuke to do some targeted color correction to match the shadows and highlights on the face as the actor moves in relation to the scene lights, but other than that the entire comp was in AfterEffects.

The first step, as show in the splitscreen video, was to eliminate the tracking dots that would be visible in the final shot. I did this by tracking the dot, applying the tracking info to a solid with a dot mask prepared on it, then using that as an alpha track matte for a duplicate of the original footage. That duplicate was then moved around until a nice clean piece of skin was found that matched to the area around it. Sometimes a few quick levels keyframes were needed to match the lighting changes. This work represents the large, repetitive section in the upper right of the node tree above.

Once the dots were satisfactorily removed I brought in the rendered face texture and nostril ambient occlusion from Maya and combined them with the shot. This included, as mentioned before, some targeted color correction to match the lighting changes in the shot, as well as ReelSmart Motion Blur Pro to duplicate the motion blur, plus a little Add Grain. The result was luma keyed in order to replace the background.

With a shiny new Pigman I decided that it’d be creepy to have him peering out of some kind of decrepit looking jail cell. (Makes him look dangerous, you know…) The jail cell/broken wall with rebar exposed, was cut apart into foreground and background layers. All three layers were divided in 3D space and attached to a Wiggler-controlled camera to add some subtle movement and parallax to the shot.

The final result was color corrected and fine tuned by adding some vignetting, blurring, and lens distortion.

Conclusion

The final Pigman shot - Click to play

Overall, this shot was a blast. As mentioned earlier, the entire shot was completed in a single day, which is pretty decent for something like this. I feel that the results were pretty promising for similar future work, although I would like more cameras. I would also like to have better positioned tracking markers to facilitate removal and also better capture the nuances of lip movements.

To request a bid for your project or for any questions please use my contact form.