Deforum Stable Diffusion is a version of Stable Diffusion focussing on creating videos and transitions of images created with Stable Diffusion.
Here is an example made with Deforum Stable Diffusion on Linux AWS GPU server:
With the Stable Diffusion (SD) Linux Cloud Server created in cooperation with AI-SP you can instantly render stunning Stable Diffusion images and videos independently on your own Cloud server with great performance. Stable Diffusion is a machine learning model developed by StabilityAI, in collaboration with EleutherAI and LAION, to generate digital images from natural language descriptions.
The neural network model can be used for other tasks too, like generating image-to-image translations guided by a text prompt. Here you can find more background about What is Stable Diffusion. Please login to Hugging Face hosting Stable Diffusion and request access to stable-diffusion-v1-4 to comply with license and access regulation.
The CreativeML OpenRAIL License specifies: You can’t use the model to deliberately produce nor share illegal or harmful outputs or content CompVis claims no rights on the outputs you generate, you are free to use them and are accountable for their use which must not go against the provisions set in the license. You may re-distribute the weights and use the model commercially and/or as a service. If you do, please be aware you have to include the same use restrictions as the ones in the license and share a copy of the CreativeML OpenRAIL-M to all your users (please read the license entirely and carefully).
Deforum Stable Diffusion Resources:
- Detailed Guide to Deforum Stable Diffusion (check out the section “Notebook Overview”)
- Background on Animation Parameters in Deforum
- Background on Math in Deforum Guide and Youtube guide: How to use Maths For Stable Diffusion Video Movement Keys With Deforum Diffusion
- Discord Server: https://discord.gg/aMhZ2Djtf9
Here is the Table of Content of this page:
1. Start and Access the AWS Stable Diffusion GPU cloud server
In the AWS Marketplace please subscribe to the Stable Diffusion Cloud Server (Linux) and start the instance on an AWS GPU server (actually the Stable Diffusion neural network was trained on AWS GPU instances with nVidia A100; don’t forget to turn off the server after using). The default server type is g4dn.xlarge which will give you access to a nVidia T4 GPU with 16 GB GPU memory and 16 GB system memory which is a great platform to start with Stable Diffussion. The server also supports g5 instances with A10 GPUs offering 24 GB of GPU memory.
For the remote desktop connection we will use the NICE DCV high-end remote desktop software and login to our cloud server with the following steps (please also see our video: NICE DCV Remote 3D on AWS – NI SP DCV AMI):
- Connect to the Stable Diffusion server:
- Open a web browser window at
https://PUBLIC_DNS_NAME:8443 -or- https://EXTERNAL_IP_ADDRESS:8443
replacing the capital part with the address of your Stable Diffusion server
—– OR —- - Download the NICE DCV client for your OS from https://download.nice-dcv.com/. E.g. for Windows download: DCV Windows Client
- Open the DCV client and enter the “public DNS name” of your server or “IP address”.
- Open a web browser window at
Accept the security warning related to the dynamically created certificate (the connection is secure) clicking “Trust and connect” and login with user Administrator and the password retrieved from the AWS “Connect” information (please follow steps 1-5 at the AWS Windows login guide to retrieve the password):
2. How to Run the Stable Diffusion GUI in Your Server
Our Stable Diffusion Linux Cloud server supports the following neural network solutions:
- Stable Diffusion Deforum repository focussing on video creation (this guide)
- Stable Diffusion AUTOMATIC repository with many powerful features (SD Guide)
- NeRF – Create 3d scenes with neural networks yourself (NeRF Guide)
How to run Deforum Stable Diffusion:
2.1 Deforum Extension to Automatic Stable Deforum Webui
The Deforum extension can be found in the upper tab section of the Automatic Stable Deforum Webui (please see our guide how to start the Automatic Stable Deforum Webui):
The settings are available in their own tab section – see below for more information about Deforum settings:
Here are Deforum for AUTO guides for further background: https://www.youtube.com/watch?v=R52hxnpNews.
Configure the animation prompts e.g. with the following animation prompts for frame 0, 100 and 200:
animation_prompts = {
0: "Elves with their pumpkins in the forrest, search all the oaks with a tiny light beam, trending on artstation HQ, 8k, indirect sunlight, (high detail), colorful, realistic, photorealistic, 8k, (artgerm)",
100: "haunted house, trees in the distance, tree leaves on the ground, mischievous and gorgeous, vibrantcolors, awardwinning, intricate, insanely detailed, digitalpainting, conceptart",
200: "A highly realistic, true to life portrait of a young woman looking similar to grace kelly, by karol bak, james jean, tom bagshaw, rococo, sharp focus, trending on artstation, cinematic lighting, hyper realism, octane render, 8 k, hyper detailed",
}
If you like you can adapt the width and height of the video images.
At the end of the run the images will be compiled into a video in the same output directory with the specified FPS. You can double-click in the file browser on the video to view the results.
5. Stable Diffusion License, Tips and Tricks
More information on Stable Diffusion from the Stable Diffusion github page:
Stable Diffusion is a latent text-to-image diffusion model. Thanks to a generous compute donation from Stability AI and support from LAION, we were able to train a Latent Diffusion Model on 512×512 images from a subset of the LAION-5B database. Similar to Google’s Imagen, this model uses a frozen CLIP ViT-L/14 text encoder to condition the model on text prompts. With its 860M UNet and 123M text encoder, the model is relatively lightweight and runs on a GPU with at least 10GB VRAM. See this section below and the model card.
The weights are available via the CompVis organization at Hugging Face under a license which contains specific use-based restrictions to prevent misuse and harm as informed by the model card, but otherwise remains permissive. While commercial use is permitted under the terms of the license, we do not recommend using the provided weights for services or products without additional safety mechanisms and considerations, since there are known limitations and biases of the weights, and research on safe and ethical deployment of general text-to-image models is an ongoing effort. The weights are research artifacts and should be treated as such.
The CreativeML OpenRAIL M license is an Open RAIL M license, adapted from the work that BigScience and the RAIL Initiative are jointly carrying in the area of responsible AI licensing. See also the article about the BLOOM Open RAIL license on which our license is based.
You can use the built-in file up/download functionality of the remote desktop DCV software to e.g. download rendered images. The button is located to the upper left in the DCV window.
Deforum Stable Diffusion Feature Guide with Examples
The Stable Diffusion Deforum Google Doc has a detailed overview of Deforum features.
Examples of videos created with Deforum SD: https://www.youtube.com/channel/UCKhP12shDrUkFtuGUMmqCAw.
Prompt Weights
Currently, prompts weights are not implemented yet in deforum, however following a template should yield fair results:
[Medium] [Subject] [Artist] [Details] [Repository]
Ex. “A Sculpture of a Purple Fox by Alex Grey, with tiny ornaments, popular on CGSociety”
6. Example prompts for Stable Diffusion
- A well-preserved library among hidden ruins, matte painting, trending on artstation
- A highly realistic, true to life portrait of a young woman, by karol bak, james jean, tom bagshaw, rococo, sharp focus, trending on artstation, cinematic lighting, hyper realism, octane render, 8 k, hyper detailed
- Tiny room, isometric diorama cutaway render of Fallout 2, by Curzio Ferrara, octane render, 100mm, 3D blender render, 8k, vibrant, incredibly detailed, high contrast, grungy, colorful, contest winner
- A distant futuristic city full of tall buildings inside a huge transparent glass dome, In the middle of a barren desert full of large dunes, Sun rays, Artstation, Dark sky full of stars with a shiny sun, Massive scale, Fog, Highly detailed, Cinematic, Colorful
- haunted house, trees in the distance, tree leaves on the ground, mischievous and gorgeous, vibrantcolors, awardwinning, intricate, insanely detailed, digitalpainting, conceptart, horrorvibes
- Ultra realistic photo, red hair girl working in a bar, beautiful face, intricate, highly detailed, smooth, sharp focus, art by artgerm and greg rutkowski and alphonse mucha — (try replacing red hair with ginger, fire hair, …. )
- A place in wales, tucked out of view magic happens, only seen by a few. for just one day, for only just one hour. The last summer’s day break at Gelli aur. there you must follow a winding trout stream. search all the oaks with a tiny light beam, inspired by ( greg rutkowski ) and charlie bowater
- Asian Brad Pitt, high quality, trending on Artstation
- A beautiful view of hogwarts school of witchcraft and wizardry and the great lake, concept art, by Thomas Kinkade, architecture, atmospheric, sense of awe and scale, artstation HQ
- Ice goddess with beautiful face with a glowing blue crystal on her forehead, frosty white eyes, winter mist around her, white plated armor, pale skin, white smoke:: photorealism, octane render, frostbite, 8k, cinematic, 35mm
- A levitating and floating beautiful detailed haussmannian palace villa by Neil Blevins and Gilles Beloeil, M C Escher and Lee Madgwick over a lake by Cyril Rolando, colorful, geometric, Vray, illuminated windows, Transcended Beyond Physics, Gravitational Anomaly, Detailed Realistic, Detailed Digital Painting, vibrantcolors, 5-Dimensional, Assassin’s Creed, Color Grading, Ektachrome
- Blonde-haired beautiful Warrior Queen, in fantasy armor, with Iron crown, cross symbolism, with a fit body, dark forest background, hopeful light, photorealistic, painted by artgerm, Akihiko yoshida, sakimichan, krenz cushart, low angle shot, digital painting
- Ultra realistic photo, princess peach in the mushroom kingdom, beautiful face, intricate, highly detailed, smooth, sharp focus, art by artgerm and greg rutkowski and alphonse mucha
- portrait of a woman made of cracked marble. high contrast. macro photography
- A beautiful neon cyberpunk city street at night, apartment, skyscrapers, by alphonse mucha caravaggio monet ,4K resolution, 8K resolution, a lot of Decoration and embellishments, sci-fi, photorealistic, highly detailed, sharp focus, clean 8k, volumetric lighting, octane render, ceramic
- A modern teenage girl’s bedroom, indirect sunlight, high detail, lush decor, realistic, photorealistic, 8k
- Ana de Armas as Red Sonja, portrait, detailed features, intricate, highly detailed, sharp focus
- (painting of girl from behind looking a fleet of imperial ships in the sky, in a meadow of flowers. ) by donato giancola and Eddie Mendoza, elegant, dynamic lighting, beautiful, poster, trending on artstation, poster, anato finnstark, wallpaper, 4 k, award winning, digital art, imperial colors, fascinate view
- A painting of a beautiful woman in the middle of a city street by Daniel F Gerhartz, William Adolphe Bouguereau, John William Waterhouse, and Thomas Kinkade. City skyline. Chiaroscuro. Volumetric Lighting
Other resources for prompts: