Deforum Stable Diffusion is a version of Stable Diffusion focussing on creating videos and transitions of images created with Stable Diffusion.

Here is an example made with Deforum Stable Diffusion on Linux AWS GPU server:

With the Stable Diffusion (SD) Linux Cloud Server created in cooperation with AI-SP you can instantly render stunning Stable Diffusion images and videos independently on your own Cloud server with great performance. Stable Diffusion is a machine learning model developed by StabilityAI, in collaboration with EleutherAI and LAION, to generate digital images from natural language descriptions.

The neural network model can be used for other tasks too, like generating image-to-image translations guided by a text prompt. Here you can find more background about What is Stable Diffusion. Please login to Hugging Face hosting Stable Diffusion and request access to stable-diffusion-v1-4 to comply with license and access regulation.

The CreativeML OpenRAIL License specifies: You can’t use the model to deliberately produce nor share illegal or harmful outputs or content CompVis claims no rights on the outputs you generate, you are free to use them and are accountable for their use which must not go against the provisions set in the license. You may re-distribute the weights and use the model commercially and/or as a service. If you do, please be aware you have to include the same use restrictions as the ones in the license and share a copy of the CreativeML OpenRAIL-M to all your users (please read the license entirely and carefully).

Deforum Stable Diffusion Resources:

Here is the Table of Content of this page:

1. Start and Access the AWS Stable Diffusion GPU cloud server

In the AWS Marketplace please subscribe to the Stable Diffusion Cloud Server (Linux) and start the instance on an AWS GPU server (actually the Stable Diffusion neural network was trained on AWS GPU instances with nVidia A100; don’t forget to turn off the server after using). The default server type is g4dn.xlarge which will give you access to a nVidia T4 GPU with 16 GB GPU memory and 16 GB system memory which is a great platform to start with Stable Diffussion. The server also supports g5 instances with A10 GPUs offering 24 GB of GPU memory.

For the remote desktop connection we will use the NICE DCV high-end remote desktop software and login to our cloud server with the following steps (please also see our video: NICE DCV Remote 3D on AWS – NI SP DCV AMI):

  1. Connect to the Stable Diffusion server:
    • Open a web browser window at

      https://PUBLIC_DNS_NAME:8443 -or- https://EXTERNAL_IP_ADDRESS:8443

      replacing the capital part with the address of your Stable Diffusion server
      —– OR —-
    • Download the NICE DCV client for your OS from https://download.nice-dcv.com/. E.g. for Windows download: DCV Windows Client
    • Open the DCV client and enter the “public DNS name” of your server or “IP address”.
DCV Connection window

Accept the security warning related to the dynamically created certificate (the connection is secure) clicking “Trust and connect” and login with user Administrator and the password retrieved from the AWS “Connect” information (please follow steps 1-5 at the AWS Windows login guide to retrieve the password):

DCV Login Window

2. How to Run the Stable Diffusion GUI in Your Server

Our Stable Diffusion Linux Cloud server supports the following neural network solutions:

  • Stable Diffusion Deforum repository focussing on video creation (this guide)
  • Stable Diffusion AUTOMATIC repository with many powerful features (SD Guide)
  • NeRF – Create 3d scenes with neural networks yourself (NeRF Guide)

Deforum Stable DIffusion can be used in 4 different ways:

  1. Deforum Extension to Automatic Stable Deforum Webui
  2. Deforum Jupyter Notebook (visual interface)
  3. Deforum Python script
  4. Google Colab Interface where GPU server is used as local environment

Below we explain the different usage scenarios of Deforum Stable Diffusion on the server:

2.1 Deforum Extension to Automatic Stable Deforum Webui

The Deforum extension can be found in the upper tab section of the Automatic Stable Deforum Webui (please see our guide how to start the Automatic Stable Deforum Webui):

The settings are available in their own tab section – see below for more information about Deforum settings:

Here are Deforum for AUTO guides for further background: https://www.youtube.com/watch?v=R52hxnpNews.

2.2 Jupyter Notebook – Deforum Stable Diffusion Startup and GUI

After login to the desktop open a Linux terminal from the left bar (see icon to the right) and start the Stable Diffusion GUI:

cd deforum-SD
./start-deforum.sh

Wait 1-2 minutes for the Deforum SD – GUI to automatically open in the firefox browser after loading has completed:

Deforum Stable Diffusion jupyter notebook

In the Jupyter Notebook you can run the different steps and adapt the settings. You can ignore the error messages related to the environment components already installed (e.g. “destination path ‘stable-diffusion’ already exists”).

Relevant parts of the configuration start in “Settings” (cell 6):

Deforum Notebook Settings
Deforum Notebook Settings

Please configure to your Settings (you can save the notebook with Ctrl-S, best is to backup the Deforum_Stable_Diffusion.ipynb notebook before making changes):

  • animation_mode = ‘2D’
  • max_frames = 300
  • zoom = “0:(1.01), 100:(0.999),200:(1.01)” # zoom in at frame 0, zoom out at frame 100, zoom in at frame 200

And configure the animation prompts in the following cell e.g. with the following animation prompts for frame 0, 100 and 200:

animation_prompts = {
    0: "Elves with their pumpkins in the forrest, search all the oaks with a tiny light beam, trending on artstation HQ, 8k, indirect sunlight, (high detail), colorful, realistic, photorealistic, 8k, (artgerm)",
    100: "haunted house, trees in the distance, tree leaves on the ground, mischievous and gorgeous, vibrantcolors, awardwinning, intricate, insanely detailed, digitalpainting, conceptart",
    200: "A highly realistic, true to life portrait of a young woman looking similar to grace kelly,  by karol bak, james jean, tom bagshaw, rococo, sharp focus, trending on artstation, cinematic lighting, hyper realism, octane render, 8 k, hyper detailed",
}

If you like you can adapt the width and height of the images in the “Run” cell at:

W = 512 #@param
H = 512 #@param

Then move your cursor down to cell “Disconnect when finished” and start the run with “Run” -> Run All Above Selected Cell”.

The images should start to be rendered and will be shown after a few moments above that cell so you can monitor the progress of image generation.

When all images have been created in “content/output/2022-10/” (or the respective month) then you can “Run -> Run Selected Cells”” the present cell to create the final video (default is 8 FPS).

2.3 Python Script – Deforum Stable Diffusion Startup and GUI

You can edit the python script with your favorite editor to make the configuration changes highlighted above regards max_frames, zoom, animation prompts, … and then start the python script with the following command (ignore the initial error message related to the setup of the enviroment, you can try with the unmodified script for a first test in case using the default values):

python3 Deforum_Stable_Diffusion.py

The output images should be visible in “content/output/2022-10/StableFun/” (or respective year-month) where you can navigate to with the file manager ( in the left vertical bar) to see the progress and have a look at the images created.

Sample output of the Stable Diffusion script:

Rendering animation frame 18 of 30
haunted house, trees in the distance, tree leaves on the ground, mischievous and gorgeous, vibrantcolors, awardwinning, intricate, insanely detailed, digitalpainting, conceptart 3808252226
Angle: 0.0 Zoom: 1.0006
Tx: 0.0 Ty: 0.0 Tz: 10.0
Rx: 0.0 Ry: 0.0 Rz: 0.0
Global seed set to 3808252226
100%|████████████████████████████████████████████████████████████████████████| 17/17 [00:04<00:00,  3.49it/s]
<PIL.Image.Image image mode=RGB size=512x512 at 0x7FF4D864C1C0>

At the end of the run the images will be compiled into a video in the same output directory with the specified FPS. You can double-click in the file browser on the video to view the results.

2.4 Use Colab Notebook with local GPU Server runtime

Google Colab offers to run a Colab Notebook via a local runtime. Here are the steps (Please note: We have not been able to get this connecting successfully yet, if you find out please let us know ;)):

1. In a terminal start the local notebook to enable access from the Colab hosted notebook (assuming you are in the directory /home/ubuntu/deforum-SD):

./start-colab.sh

which will show a connection string similar to this: http://localhost:8888/?token=fff032cf090de66bce3cde91172accffe886a52758b9dc17 which we will use below.

2. Open the Deforum Stable Diffusion Colab Notebook: https://colab.research.google.com/github/deforum/stable-diffusion/blob/main/Deforum_Stable_Diffusion.ipynb
and connect the Colab notebook to the local server with the runtime option to the upper right: “Connect -> Connect to local runtime…”. In the following popup enter the URL from above. Here is the Google Colab Guide to connect to a local runtime.

5. Stable Diffusion License, Tips and Tricks

More information on Stable Diffusion from the Stable Diffusion github page:
Stable Diffusion is a latent text-to-image diffusion model. Thanks to a generous compute donation from Stability AI and support from LAION, we were able to train a Latent Diffusion Model on 512×512 images from a subset of the LAION-5B database. Similar to Google’s Imagen, this model uses a frozen CLIP ViT-L/14 text encoder to condition the model on text prompts. With its 860M UNet and 123M text encoder, the model is relatively lightweight and runs on a GPU with at least 10GB VRAM. See this section below and the model card.

The weights are available via the CompVis organization at Hugging Face under a license which contains specific use-based restrictions to prevent misuse and harm as informed by the model card, but otherwise remains permissive. While commercial use is permitted under the terms of the license, we do not recommend using the provided weights for services or products without additional safety mechanisms and considerations, since there are known limitations and biases of the weights, and research on safe and ethical deployment of general text-to-image models is an ongoing effort. The weights are research artifacts and should be treated as such.

The CreativeML OpenRAIL M license is an Open RAIL M license, adapted from the work that BigScience and the RAIL Initiative are jointly carrying in the area of responsible AI licensing. See also the article about the BLOOM Open RAIL license on which our license is based.

You can use the built-in file up/download functionality of the remote desktop DCV software to e.g. download rendered images. The button is located to the upper left in the DCV window.

Deforum Stable Diffusion Feature Guide with Examples

The Stable Diffusion Deforum Google Doc has a detailed overview of Deforum features.

Examples of videos created with Deforum SD: https://www.youtube.com/channel/UCKhP12shDrUkFtuGUMmqCAw.

Prompt Weights

Currently, prompts weights are not implemented yet in deforum, however following a template should yield fair results:

            [Medium]          [Subject]         [Artist]              [Details]                 [Repository]

Ex. “A Sculpture of a Purple Fox by Alex Grey, with tiny ornaments, popular on CGSociety”

6. Example prompts for Stable Diffusion

A highly realistic, true to life portrait of a young woman looking similar to grace kelly, by karol bak, james jean, tom bagshaw, rococo, sharp focus, trending on artstation, cinematic lighting, hyper realism, octane render, 8 k, hyper detailed
A highly realistic, true to life portrait of a young woman looking similar to grace kelly, by karol bak, james jean, tom bagshaw, rococo, sharp focus, trending on artstation, cinematic lighting, hyper realism, octane render, 8 k, hyper detailed (upscaled to 2048×2048, face restoration with GFPGAN, seed: 1735754603)
  • A well-preserved library among hidden ruins, matte painting, trending on artstation
  • A highly realistic, true to life portrait of a young woman, by karol bak, james jean, tom bagshaw, rococo, sharp focus, trending on artstation, cinematic lighting, hyper realism, octane render, 8 k, hyper detailed
  • Tiny room, isometric diorama cutaway render of Fallout 2, by Curzio Ferrara, octane render, 100mm, 3D blender render, 8k, vibrant, incredibly detailed, high contrast, grungy, colorful, contest winner
  • A distant futuristic city full of tall buildings inside a huge transparent glass dome, In the middle of a barren desert full of large dunes, Sun rays, Artstation, Dark sky full of stars with a shiny sun, Massive scale, Fog, Highly detailed, Cinematic, Colorful
  • haunted house, trees in the distance, tree leaves on the ground, mischievous and gorgeous, vibrantcolors, awardwinning, intricate, insanely detailed, digitalpainting, conceptart, horrorvibes
  • Ultra realistic photo, red hair girl working in a bar, beautiful face, intricate, highly detailed, smooth, sharp focus, art by artgerm and greg rutkowski and alphonse mucha — (try replacing red hair with ginger, fire hair, …. )
  • A place in wales, tucked out of view magic happens, only seen by a few. for just one day, for only just one hour. The last summer’s day break at Gelli aur. there you must follow a winding trout stream. search all the oaks with a tiny light beam, inspired by ( greg rutkowski ) and charlie bowater
  • Asian Brad Pitt, high quality, trending on Artstation
  • A beautiful view of hogwarts school of witchcraft and wizardry and the great lake, concept art, by Thomas Kinkade, architecture, atmospheric, sense of awe and scale, artstation HQ
  • Ice goddess with beautiful face with a glowing blue crystal on her forehead, frosty white eyes, winter mist around her, white plated armor, pale skin, white smoke:: photorealism, octane render, frostbite, 8k, cinematic, 35mm
  • A levitating and floating beautiful detailed haussmannian palace villa by Neil Blevins and Gilles Beloeil, M C Escher and Lee Madgwick over a lake by Cyril Rolando, colorful, geometric, Vray, illuminated windows, Transcended Beyond Physics, Gravitational Anomaly, Detailed Realistic, Detailed Digital Painting, vibrantcolors, 5-Dimensional, Assassin’s Creed, Color Grading, Ektachrome
  • Blonde-haired beautiful Warrior Queen, in fantasy armor, with Iron crown, cross symbolism, with a fit body, dark forest background, hopeful light, photorealistic, painted by artgerm, Akihiko yoshida, sakimichan, krenz cushart, low angle shot, digital painting
haunted house, trees in the distance, tree leaves on the ground, mischievous and gorgeous, vibrantcolors, awardwinning, intricate, insanelydetailed, digitalpainting, conceptart, horrorvibes
Haunted house, trees in the distance, tree leaves on the ground, mischievous and gorgeous, vibrantcolors, awardwinning, intricate, insanely detailed, digitalpainting, conceptart, horrorvibes
  • Ultra realistic photo, princess peach in the mushroom kingdom, beautiful face, intricate, highly detailed, smooth, sharp focus, art by artgerm and greg rutkowski and alphonse mucha
  • portrait of a woman made of cracked marble. high contrast. macro photography
  • A beautiful neon cyberpunk city street at night, apartment, skyscrapers, by alphonse mucha caravaggio monet ,4K resolution, 8K resolution, a lot of Decoration and embellishments, sci-fi, photorealistic, highly detailed, sharp focus, clean 8k, volumetric lighting, octane render, ceramic
  • A modern teenage girl’s bedroom, indirect sunlight, high detail, lush decor, realistic, photorealistic, 8k
  • Ana de Armas as Red Sonja, portrait, detailed features, intricate, highly detailed, sharp focus
  • (painting of girl from behind looking a fleet of imperial ships in the sky, in a meadow of flowers. ) by donato giancola and Eddie Mendoza, elegant, dynamic lighting, beautiful, poster, trending on artstation, poster, anato finnstark, wallpaper, 4 k, award winning, digital art, imperial colors, fascinate view
  • A painting of a beautiful woman in the middle of a city street by Daniel F Gerhartz, William Adolphe Bouguereau, John William Waterhouse, and Thomas Kinkade. City skyline. Chiaroscuro. Volumetric Lighting

Other resources for prompts: