PIRATES OF THE CARIBBEAN – DEAD MEN TELL NO TALES: Rudy Grossman – Digital Effects Supervisor – Atomic Fiction

Rudy Grossmann has nearly 20 years of experience in the visual effects. He has worked in many studios such as Industrial Light & Magic, The Orphanage, Weta Digital or Digital Domain before joining Atomic Fiction in 2014. At Atomic Fiction, he has worked on different movies such as THE WALK, DEADPOOL or ALIEN: COVENANT.

How did you and Atomic Fiction get involved on this show?
Atomic Fiction was in talks with Gary, Tamara, David, and the rest of the team at Disney since the early days of the project. Our founders, Ryan and Kevin, as well as Marc Sadeghi, myself and many others at Atomic had worked on several of the ‘Pirates’ films previously, so we were really eager to get back into that world and deliver some amazing shots. We all really wanted to play a role on the film, and were given the opportunity to do so!

What was your feeling to be in the Pirates of the Caribbean universe?
I’ve loved the PIRATES OF THE CARIBBEAN universe since the first film. The ‘Pirates’ world is a beautiful blend of vast and inspiring visuals with fantastical pirate lore.

How was this collaboration with directors Joachim Rønning and Espen Sandberg?
Joachim Ronning and Espen Sandberg are fantastic to work with. Both directors have a really great eye for visual effects. Their feedback was always clear and to the point. The attention to detail is a clear indication of how much of their heart is put into the project.

What was their approaches and expectations about the visual effects?
The filming approach expected a lot of collaboration between visual effects, the SFX team, and editorial. I loved this about the project, all the work done on the SFX side really helped establish a rooted and consistent look to the world that we could ground our digital visual effects to. On the editorial side, several shots were filmed with multiple plates. After the filming wrapped, a lot of effort was put into the PreVis and editorial cut references. This provided us with a clear understanding of how we were expected to fit all these different pieces together into a final shot.

How did you work with VFX Supervisor Gary Brozenich?
Gary was great to work with! He brings with him a wealth of experience and insight, and he was also very open to hearing our thoughts and suggestions. While a lot of our reviews were run remotely, Gary also made it a point to drop by our Montreal Studio frequently, so we could have reviews with him in person.

What are the sequences made by Atomic Fiction?
Our major sequences are the opening ‘Bank Heist’, in which the bank is dragged through the town of St. Martin, the execution escape/guillotine sequence in the town square of St. Martin, and the dilapidated ship yard where the Dying Gull is docked.

How did you split the work with Jim Gibbs and organize the work at Atomic Fiction?
Jim Gibbs was Atomic Fiction’s VFX Supervisor and I was the DFX Supervisor for PIRATES 5. As VFX Sup, Jim handled the overall creative delivery which included everything from the heavy CG environment shots, to the comp heavy shots, and numerous paint cleanup shots. My role was creative and technical supervision focused mainly on the heavy CG environment shots, digital doubles, and crowds, essentially the shots which relied on the most CG rendered elements, animation, and matte paintings. 

Can you explain in detail about the creation of the bank for the excellent bank heist sequence?
The on-set bank, used in most of the shots, was actually drivable! A fully CG bank was needed for the big destruction shots in the end of the sequences and also to augment areas of the on-set bank for several other shots. The CG bank was created from scan data and photographs of the on-set bank. Throughout the sequence we had to augment the bank with CG elements so it felt like it was really being dragged through the ground, and not driving on top of it. Several shots had Jack filmed on a static bank in front of a blue screen. Those shot needed to be composited onto a moving background plate shot from a drone. Using our CG bank, we were able to render additional lighting and shadowing passes for the bank rooftop to integrate its foreground plate into the background plates.

How did you work with the PreVis for this sequence?
The PreVis provided a fantastic starting point for our shots. In general, the PreVis is the guide representing the directors’ overall plan and intention of the shot. There were a number of shots which required complex combinations of multiple plates with CG animation and environments. The PreVis provided a roadmap that illustrated how the directors envisioned those pieces coming together and also visualized the key action beats they had in mind when formulating the shot.

Can you tell us more about the environment creation?
We wanted to really capture the imaginative fantasy of a lush tropical Caribbean island as it would exist in the ‘Pirates’ universe. We spent a lot of time researching different Caribbean islands, comparing the rock formations, and the transition from sandy beaches to jungle foliage. Also, the different color hues along the beach shoreline where the shallows drop off to deeper depths. Right before we started working on the environment, I had a trip planned traveling through the southern jungles and islands of Thailand. This turned out to be the perfect opportunity to capture everything from tons of high resolution texture reference to motion studies of palm trees in different wind conditions. Using this motion reference, we animated our trees and plants to match the way they moved in the wind. Next we developed some tools which allowed us to easily customize each shot’s amount of wind depending on the weather conditions and camera moves. The directors really liked the look of a dense canopy surrounding the Caribbean island town. One of our biggest challenges was optimizing the render times to handle so many CG trees without sacrificing the visual quality. St. Martin and the Shipyard environments required rendering over 20,000 trees and plants. Our lookdev, lighting, and amazing dev support worked closely together thoroughly the production to optimize our shader and render settings. All those CG elements combined with some gorgeous matte paintings and a lot of hard work from the compositing team fit together to create our final environments.

How did you create the various digi-double and especially the one for Jack Sparrow?
Our main actor digital doubles were Henry and Jack Sparrow. Jack Sparrow was a shared asset. MPC provided us with a fantastic model and textures, this allowed us to jump in right away and start focusing on Jack’s rigging, lookdev, cloth, and hair for our sequences. Henry’s digital double was created based off scan data and texture photo reference of the actor. We also created 16 interchangeable classifications types of villager digital doubles that represented a wide diversity of economic and social backgrounds to give the townspeople a deeper sense of reality. These villager digital doubles could be animated as stand-alone digital doubles or simulated as crowd characters. The costumes were fit to adjustable body types, ie: skinny to heavy set, child to adult, even male to female.

How did you handle all the destructions of this sequence?
Our talented FX team created the destruction and volumetric simulations in Houdini. The destruction differs depending on what that material would be made of in real life. Terracotta shingles break apart on impact, but will often keep breaking apart into smaller pieces on additional impacts. Wooden boards can bend and splinter, but also get restrained by any nails holding them together. To hit the final look of the destructed model after simulation we used a combination of techniques.

For some scenarios, the FX software was used to create the final look of the fractured model, in other scenarios we’d actually sculpt the destructed look of the CG model first and then apply the FX simulation to that. Throughout the sequence there was a lot of dust getting kicked up from the bank being dragged through the streets. We wanted to make sure there was realistic lighting interaction between the debris and volumes of dust and smoke. To achieve this, we used OpenVDB caches to bring the dust volumes into Katana so they could be rendered with VRay shaders alongside all our other CG elements.

For the big finale bank collapse, Joachim, Espen, and Gary all had a very specific vision about how they wanted the internal structure to buckle upwards and collapse. To make sure we were delivering exactly what the directors and Gary wanted, animation collaborated really closely with FX to rough out the overall timing so we could nail the creative target. The overall bank destruction simulation was so heavy and so complex that this allowed FX to focus on that, and not get caught up in running heavy simulation iterations only for timing approval.

The sequence ends with a beautiful slow-mo shot. Can you tell us more about it?
That shot was a lot of fun! It was also one of the first shots turned over, and the last to finish, so it was quite complex. Jack Sparrow jumps off the top of an archway and lands on a bank vault at the exact time when it is pulled underneath him, all while the bank behind him is demolished into a shower of wooden splinters and terra cotta fragments. The beginning and end had been shot with different cameras moves, framing different action, and shot on different days. The first plate framed Jack running across and jumping off the archway. The second plate framed Jack on the vault getting pulled across ground.

One of our biggest challenges for this shot was creating a camera move to smoothly transition between the beginning and end. First we tracked both plates, recreating them as camera moves in 3D. Then using a CG reconstruction of the set, we animated a third camera to seamlessly link into the first and second cameras while also creating a new middle of the shot that never actually existed. The environment used a combination of projected plates and paint work along with fully CG elements to complete the seamless illusion. A fully CG Jack Sparrow was used to blend into both plates and fill the gap between. For the vault bursting through the bank wall, we referenced videos of wooden structures getting demolished, so we could study how the wooden boards bend and break. The final shot required the integration of so many different elements that we split the work between a few different compositors.

Later in the movie, Jack Sparrow is escaping from Saint Martin. How did you created this big place?
St. Martin’s evolved a lot from when we first started working on the show to the final film footage. We started off planning out the town layout with some really gorgeous illustrations from our art department. It was very important to Gary that we base the layout of the town on historic developments. A few blocks of buildings were built on set for shooting the live action. We used the design of those buildings to set the tone for the rest of the town.

The final layout of St. Martin town was over 600 CG buildings and over 2,000 CG set dressing props. Gary had worked on large CG environment shots before and wanted us to add some more subtle motion throughout the town, but we didn’t want to use the old trick of having a few birds fly by! To give it a last kiss of love we added in several simulated cloth canopies throughout the town and created a system that allowed us to quickly drop in clothing lines that had a random distribution of hanging cloth simulation. The garments on the clotheslines were created from the same wardrobes as our crowd villagers.

How did you populate the place and handle the crowd animation?
We used Golaem to create our crowd simulations in St. Martin. The crowd digital doubles were used to extend live-action crowds in closer shots, and populate the entire town in the large CG environment shots. Gary has a background in crowd work, so we knew he would have an acute eye for CG crowds. It was really important to us that the performance of the crowds had a sense of purpose the same way extras would be directed on set, and not just uniformly milling around. Some areas were food markets where villagers were buying and selling fruit in the street. Other areas were being rebuilt after the bank heist. Some villagers were going in and out of shops, one couple was strolling casually out on a date. There are even farmers tending crops outside the city wall. Because our crowd characters could also be animated as standalone characters, we animated fun interactions between them. For example, crowd soldiers on the wall were responding to the orders shouted by an animated captain. We also animated another villager who manages to get ahead of the soldiers, who are chasing the bank, because he was so determined not to lose his life savings!

Jack Sparrow is attached to a guillotine. How did you create it and especially when he is spinning around?
A stunt version of the guillotine platform was built on set which they actually strapped Johnny Depp into and spun him around. For those shots, we added an animated CG guillotine blade structure with rope simulation, and replaced the lunette. We also added the wooden platform destruction to tie in with the digital destruction from the previous shots. A few of the wider shots used a fully CG Jack digital double and spinning CG guillotine rig. The comic timing of the guillotine blade animation throughout the sequence was crucial for hitting the beats that sell the joke. These shots were especially difficult, due to the importance and prominence of the VFX work. Our fearless comp and CG teams spent many late nights getting them to look just right.

How you extend the Shipyard and its environment?
The shipyard environment is one of my personal favorites. The actual location was in the midst of several hotels. I was able to paint and explore the concept art for the shipyard environment at the beginning of the project. It’s not often a show’s schedule works out so a supervisor can also work on concept art, but it’s really great when it does. I was trying to keep in mind all the shots, cameras angles, and methods that would be used to hit the look. Also, the process of approving the concept art with Gary and the directors really helps us understand exactly what they want. That artwork is then used as guide for each department to create the result the directors are expecting. The different ship wrecks, debris, shoreline, and mountain structure from the artwork were then created as CG assets.

The shipyard takes place on the outskirts of St. Martin, so the same trees and plants were populated along the coastline to create the sense of a dense canopy. The water interacting along the shoreline was created using a clever compositing projection technique, so we didn’t need to create any fluid simulation. The finishing details were added by matte painting.

Was there a shot or a sequence that prevented you from sleep?
I have to admit, there were a few times where a particular shot or an idea got me so excited that I woke up early and rushed back in the office!

What was the main challenge on this show and how did you achieve it?
The big St. Martin environment shots were a lot of work, and the complexity evolved as key changes were made to best suit the story. Creating the buildings, the set dressing, the digital doubles, the dense tropical canopy. It all takes a lot of talented artists working really hard to give it a sense of depth and life. We put a lot of effort into making the building blocks for this early on, all this work up front paid off well in providing the flexibility to handle dramatic changes quickly later in the schedule as the scope and complexity of the environment grew.

What is your best memory on this show?
There were so many good ones, it’s hard to pick. The team as a whole is such a talented group to work with, and we all really enjoyed diving into the Pirates world. While helping to make this film, it was an absolutely gorgeous summer in Montreal. Some evenings after work, we’d step outside and enjoy the summer festivals or just hang out together.

How long have you worked on this show?
The bulk of the work happened over a period of 10 months, but we were doing some tests and early art work around a year before delivering our final shots.

What’s the VFX shots count?
We delivered over 180 shots for PIRATES.

What was the size of your team?
The Atomic team consisted of around 100 artists spread between multiple departments, including the stereo packaging.

What is your next project?
Atomic Fiction has several really exciting projects underway: BLADE RUNNER 2049, THE PREDATOR, STRANGER THINGS, and Bob Zemeckis’ next film, which is inspired by the 2010 documentary, ‘Marwencol’.

A big thanks for your time.

// WANT TO KNOW MORE?

Atomic Fiction: Dedicated page about PIRATES OF THE CARIBBEAN: DEAD MEN TELL NO TALES on Atomic Fiction website.

© Vincent Frei – The Art of VFX – 2017