A new creative AI system called ModelScope is now pumping out short videos in response to text prompts. The early results are wonderfully bizarre and thoroughly memeworthy – but it’s immediately clear how immensely powerful these tools will become.

Developed by a collaborative team at Huggingface, Modelscope is a “multi-stage text-to-video diffusion model,” which takes plain English text prompts, attempts to understand what you’re hoping to see, then generates and de-noises a short video for you. You can play with it online through a very simple interface. It’s very early days for this sort of thing, making it the perfect time to marvel both at its incredible capabilities and at its bizarre misunderstanding of the world. Read More

Try It

Try it out

#vfx

Learn how we created an AI generated video with a ChatGPT script, a Synthesia avatar and voice, and stock footage from Descript.

There is a lot of buzz around new and exciting artificial intelligence (AI) and machine learning (ML) tools for video production and video creation. So, I wanted to see first-hand how some of these tools perform! As an experiment, I set out to create a high quality video using generative AI in less than 15 minutes. Read More

Text-guided generative diffusion models unlock powerful image creation and editing tools. While these have been extended to video generation, current approaches that edit the content of existing footage while retaining structure require expensive re-training for every input or rely on error-prone propagation of image edits across frames.

In this work, we present a structure and content-guided video diffusion model that edits videos based on visual or textual descriptions of the desired output. Conflicts between user-provided content edits and structure representations occur due to insufficient disentanglement between the two aspects. As a solution, we show that training on monocular depth estimates with varying levels of detail provides control over structure and content fidelity. Our model is trained jointly on images and videos which also exposes explicit control of temporal consistency through a novel guidance method. Our experiments demonstrate a wide variety of successes; fine-grained control over output characteristics, customization based on a few reference images, and a strong user preference towards results by our model. Read More

The media spectacle of generative AI (in which AI companies’ breathless claims of their software’s sorcerous powers are endlessly repeated) has understandably alarmed many creative workers, a group that’s already traumatized by extractive abuse by media and tech companies.

Even though the claims about “AI” are overblown and overhyped, creators are right to be alarmed. Their bosses would like nothing more than to fire them and replace them with pliable software. The “creative” industries talk a lot about how audiences should be paying for creative works, but the companies that bring creators’ works to market treat their own payments to creators as a cost to be minimized. Read More

Stars like Tom Hanks and Harrison Ford are being rendered younger digitally but voices in the industry express concern about where we might be heading

Craggy, grey-haired and 80 years old, Harrison Ford might seem a bit old to don his brown Fedora-style hat or crack his whip as Indiana Jones. But a trailer for his upcoming film Indiana Jones and the Dial of Destiny offers a flashback to Indy in his swashbuckling glory days.

“That is my actual face at that age,” the actor explained on CBS’s The Late Show with Stephen Colbert. “They have this artificial intelligence (AI) programme. It can go through every foot of film that Lucasfilm owns because I did a bunch of movies for them and they have all this footage including film that wasn’t printed: stock. They could mine it from where the light is coming from, the expression. But that’s my actual face. Then I put little dots on my face and I say the words and they make it. It’s fantastic.” Read More