Amazing animated AI image processing/transformation

[FrameSniffer]

Corridor Crew recently posted a video covering advances in animated image processing and pulled off something rather special (IMO).

I would love to leverage it in my edits as the technology matures beyond animation into realistic live action image processing and transformation. Has anyone played around with this kind of technology yet?

 

[StarkillerAG]

I saw that video too. It was the most interesting thing Corridor Crew has done in a long time.

They were definitely exaggerating about "changing animation forever", of course, but it does seem like an interesting way to retouch live-action footage for a more "artistic" effect. MoviesRemastered on Reddit is already planning to use this tech for his edit of Netflix's Cowboy Bebop, which is a series that definitely needs a bit more "style".

 

[FrameSniffer]

I saw that video too. It was the most interesting thing Corridor Crew has done in a long time.

They were definitely exaggerating about "changing animation forever", of course, but it does seem like an interesting way to retouch live-action footage for a more "artistic" effect. MoviesRemastered on Reddit is already planning to use this tech for his edit of Netflix's Cowboy Bebop, which is a series that definitely needs a bit more "style".

Agree, it was quite a bold statement. This tech is progress so quickly at the moment, I'm really excited to see what's next. i've not seen or heard of Cowboy Bebop. Will check MoviesRemastered version out when its done.

 

[StarkillerAG]

I've not seen or heard of Cowboy Bebop. Will check MoviesRemastered version out when its done.

Just so you know, Cowboy Bebop is a live-action adaptation of a late-90s anime series. MoviesRemastered's goal with the rotoscoping is to make certain flashback scenes in the adaptation more close to the original series's style, like what Scott Pilgrim did with the comic-panel flashbacks.

 

[That One Guy]

This is an interesting technique and quite cool that they've gotten this far with the tools available, but IMO for someone trying to produce an animated feature it's still not as good as actual animation - heck, even one of the frames where they did the comparison between the original footage and the generated image, it is clear that the general problem many image GANs have with human hands is still present. So for me this is a case of "good progress so far, but a long way to go yet".

(As someone who is a big fan of hand-drawn animation, I am ambivalent about this development - because on the one hand studios are already moving away from hand-drawn animation as few are willing to invest in it and this will contribute to that, but on the other hand it is possible that we will eventually hit a point where animation-specific iterations of these tools will be developed which can be trained on the animation team's work and thus achieve results like hand-drawn animation without the high overhead of traditional hand-drawn animation... which would also mean that animator's work is still valued in some way because the training data to establish the visual style is still required.)

I also found it extremely jarring that, as independent professionals working in media creation, they would throw out an off-hand "You can watch this anime free on Youtube" when the only "free" uploads on Youtube are blatant copyright infringements - as far as I can tell the film is still in copyright and has not been distributed for free by any of the rights holders...

 

[StarkillerAG]

As someone who is a big fan of hand-drawn animation, I am ambivalent about this development - because on the one hand studios are already moving away from hand-drawn animation as few are willing to invest in it and this will contribute to that, but on the other hand it is possible that we will eventually hit a point where animation-specific iterations of these tools will be developed which can be trained on the animation team's work and thus achieve results like hand-drawn animation without the high overhead of traditional hand-drawn animation... which would also mean that animator's work is still valued in some way because the training data to establish the visual style is still required.

Yeah, I hear you. This tech, like anything AI, could be used either for good or for evil.

My main hope with AI animation is that we'll eventually be able to get AI-generated in-betweener frames. For those who don't know, in-betweeners are frames in hand-drawn animation intended to create a fluid sense of movement between two "key frames", or important poses that the art is mainly focused on. If we can simplify the process so that 2D animators only need to draw the key frames, with everything in between being filled in by AI, then that would make hand-drawn animation immensely faster, cheaper, and easier. It's still almost as artistic as drawing the entire thing, just much more appealing to the studios that generally consider 2D too much of a risk.

 

[asterixsmeagol]

This seems like a high-tech version of what Ralph Bakshi did for his 1978 adaptation of The Lord of the Rings.

 

[WilliamRedRobin]

Great potential if it can be mastered. I'd love to be able to attempt it personally, but I'm not sure I'd be able to do it well.

 

[Jrzag42]

All I want is for someone to apply this to Hellraiser. I don't know why, but for a while now I've longed for an animated version of Hellraiser, and this seems to be the best chance of making it happen.

Also, has anyone tested this on cgi footage rather than live action? Imagine a 2d version of the 3d Clone Wars.

 

[Editzilla]

Know this video is making a lot of freakouts on YouTube, especially in animation circles for them using Vampire Hunter D directly for the AI.

 

[Gaith]

^ They did blow past that detail pretty casually, it's true. Otherwise, a pretty amazing video.

 

[FrameSniffer]

^ They did blow past that detail pretty casually, it's true. Otherwise, a pretty amazing video.

Agree. They made it look easy. Reality is you still need a lot of talent to pull this off.

 

[addiesin]

What would be cool is creating your own style model with a few hundred drawings may or may not already have made (if you draw you probably know what I mean). Then act out a film alone and overlay different characters. Rapid production cartooning.

Best part is, if it's your own style, you can fix the AI weirdness by redrawing better than anyone.

 

[That One Guy]

Agree. They made it look easy. Reality is you still need a lot of talent to pull this off.

If they didn't make it look easy how would they sell those subscriptions to their "see how we do it" club, or whatever it's called?

I'm impressed at what they've managed to do here, but pretty appalled at their casual disregard for the copyright on the work they've lifted from and the complete lack of mention of any ethical or legal discussion around what sources are used as training input sours me on the whole thing somewhat. I feel like if someone was uploading their chargeable material to youtube constantly and talking about how it can be watched for free there, they wouldn't be too happy about it...

 

[Gaith]

^ I agree the casualness is a bit troubling, but I'm not sure I find the use of technology as concerning as you do. It seems similar to a band writing a song that "sounds like" a Beatles song, albeit with a big machine learning assist, but the short's faces, costumes, story, and dialogue were original content (or, rather, the visuals were in large sourced from an Unreal environment and Etsy costumes, rather than the anime the algorithm took stylistic "inspiration" from). Inspiration and homage is always a part of art-making, but machine learning certainly adds a new and complicating element. It's a complex matter, no doubt.