AI in animation

This last week I’ve been at the Manchester Animation Festival - a week-long festival for industry, education, and public alike. The panels and short films lit my mind up with an inspiration and drive that I haven’t felt in a while.

On the Industry Day, one of the panels I’d most anticipated was about AI: the fears, the predictions, the ways we’re already using it to improve our work… I’m certainly using AI tools in my photo editing, and as a base for subtitling any fast turnaround work for YouTube etc

For my money, a lot of the practical uses will end up being based on personal/client preferences, past the obvious ethical and legal conversations that are already taking place.

Take voice acting. When we change a line / scene in a script, we usually scratch in the dialogue by getting someone in the studio to record it. The actor then records the line at their next session, or ADRs it if it has already been animated to the scratch.

The current ethical debate, prompted by existing “machine learning” in which AI systems have been “trained” on an actor’s voice and then prompted to generate new speech from a written prompt, is “we can replace that by training an AI on the actor’s voice” vs “actors should be paid, and given the chance to produce the most authentic work”.

Obviously I agree with the latter- audiences deserve the authenticity, and suggesting that AI can replace their training and talent is horrible. But. How about using the AI as a temporary placeholder, instead of the scratch of a completely different voice just jumping in?

+ you get a smoother viewing experience

- the output is misrepresenting what you actually have “in the can” (burn-ins are easily overlooked)

+ it joins up more neatly, and you can judge timing better with a more accurate placeholder

+ if animation for the shot must happen before the actor is next available, a smooth read in the correct cadence and pronunciation will help lipsync

- the actor may be unnerved by hearing their own voice at ADR/ sense the uncanny valley more readily

… all of this assumes that the technology allows for cadence, tone etc. But existing technology such as VocAlign/ Revoice Pro (where you can match an audio clip by the actor to a reading by anyone else) take us a long way towards that already. We can use a written “find” command to find words or syllables in an audio library in Avid’s PhraseFind or DaVinci Resolve Studio. I feel confident that I could get some way towards a useable voice performance… if given enough time. If AI can be used as a tool to cut out that time, and present options to a human, it could be very useful indeed. IF it is desired.

It’s going to be an interesting time. But whether people currently view it as the dawn of a new era or the opening of a new Pandora’s Box… AI is happening. It’s worth being part of the conversation.

Avid Blogs - "Editor Judith Allen Illustrates the Difference Between Cutting Animation and Live Action Films"

Excerpt:

In live action filming, every action on set is accompanied by the corresponding sound. Often these are replaced or enhanced, but the sound is there and used within the rhythm of the cut whether consciously or unconsciously. In animation, every single sound needs to be purposefully placed – and often created from scratch by a foley team. It’s often a delicate balance between knowing which temp sound effects are best to place during the animatic to help tell the story and do the best cut, and which ones should be saved until after animation so as not to restrict the animators.

At the moment I tend to make my sound choices based on how the characters need to respond to them in their headspaces more than how the final sound should be, because I know that the sound’s going to be completely replaced once we have the final picture, and because it will help the animation before that.

Read more

Cinema 4DX

4DX cinema is programmed by a South Korean company, building on the older theme-park style immersive experience (I have memories of an Alien 4D experience at a Disney park in which you could feel the simulated breath on the back of your neck as the speakers in the chair implied it was right behind you), but expanded out to mainstream feature films. The chairs move, and various effects of light, water, fragrance, and air surround the audience in complement to the on-screen action. Having recently moved out of London to the suburbs (greener, MUCH more affordable, ample opportunity to listen to podcasts on the rail commute), I find myself now living near the UK's first 4DX cinema in Milton Keynes.
Read more

The future's 3D?

I'm beginning to understand the appeal a bit more now that I've been able to view things such as football and rugby matches in 3D, and now that people are becoming more familiar with the techniques to correctly shoot for 3D.

I'm not bothered about the new technologies and terminologies so much. I generally enjoy the challenges and opportunities to see how new methodologies can be used in storytelling. It's the creative implications that this could have for editors, as well as the cast and crew of any given production, that concerns me.

Read more

The editors' toolkit of the future

Today another editor on my course came into my room sounding rather excited about Microsoft's latest announcement for user interfacing - the Windows 7 multi-touch:


Video: Multi-Touch in Windows 7


This of course is a subject we've all discussed before - what if editing could have the user interface of Minority Report?





In the film, Tom Cruise reviews video footage of a crime (that has yet to happen, but that's irrelevant here) and utilises all manner of time-lapse, zoom and selection tools via a pair of gloves and a projector.

Minority Report cap


The potential has always been obvious - mapping functions on an editing system to certain gestures (in the same way they can be mapped to buttons on the shuttlepro) to start with, with more specialist applications being developed to relate those gestures to certain areas of the screen/room or depending on the toolset you'd like to access at the time.... but with the wii remote and now the promised multi-touch mapping to multiple features and becoming a part of regular useage, the future seems to look bright. We're getting there. Now just to combine them and not actually need a physical connection to input data.

Another 'future' development that came to my attention recently was in the 20th June issue of Broadcast - the UK TV trade paper. Within a more general section hypothesising the technology available in 2012 (with a fair number of mentions given to stereoscopic techniques, also known as 3D and previously discussed in this blog), there was a mention of the implications for post-production involved in tapeless filming. If rushes/ dailies are recorded straight to a disc, could that disc be part of a network which also includes an edit suite? How soon could editing theoretically begin after filming starts? And if an editor is on set/ receiving footage in real-time even in a remote location, how would that alter their role - as well as that of the director and cinematographer?

As a recent entrant to the post-production business, it's easy for me to think of the current processes as how they've always been. My technological progression has more or less been from Premiere to various versions of Final Cut and Avid. As part of the Editing MA at the NFTS we do two excercises on Steenbecks with 16mm film, and I have a vague recollection of witnessing linear editing at Oxford Road in Manchester during some BBC outreach programme I participated in during my school years. Unless an article or book is specifically referring to the physical techniques involved, the view seems to be that the editor's role and input has stayed largely the same over the conversion to digital even if the techniques have changed. But as I prepare to start my professional career (with any luck) with the tools that I think will see me through for the time being at least, I do wonder how different the job will look and be in 40 years' time.

ShuttlePRO v.2


Shuttlepro


This is my latest toy.  It's a Contour ShuttlePRO v.2, USB interface, 15 customisable buttons (plus jog/shuttle), a shed load of application pre-loads, plus import/export on your custom settings whenever you take it anywhere else.

In Avid, the inner dial will move your playhead along one frame at a time (at just the right sensitivity), and the outer will play your video at normal speed, increased speed, reverse speed.... basically the same as tapping the J and L keys multiple times, but in a far more ergonomic and inutitive way. One minor grumble that there's no lock to keep the video playing at normal speed as you'd get on a Steenbeck or a Lightworks console, but it's easy enough to adjust your hand and/or map a button or two to the relevant function.

I've put it on the left of the keyboard so that I can simultaneously use the shuttlepro and the mouse, though I'm not sure it's possible to entirely omit the need for the keyboard - I have settings for digitising, assembling/ editing and trimming (see below the cut), and rather than go up to the drop down menu to select the exact setting I need for a certain splice, it's easier to just press the corresponding key.

I'm still changing some of the settings and buttons since I got it a fortnight or so ago to find the best positions and keystroke replacements, but so long as I remember it's there I've found that my editing speed has significantly increased as I've become more used to the shuttlepro interface.  I'd definitely recommend it - and for video editing, the 15 button shuttlePRO is a must - its smaller cousin the ShuttleXpress has just 5 buttons in addition to the jog/ shuttle wheel, and finding your way around the larger version isn't as intimidating as it may look at first.