Generating videos from text prompts using Stable Diffusion

 In this article, methods for generating videos from text prompts using Stable Diffusion AI.

Stable Diffusion lets you create amazing images from text prompts using AI.

Let’s take this concept a bit further and generate videos from text prompts using Stable Diffusion.

Use whichever way you are comfortable with and is suitable for your use case.

1. Generating Stable Diffusion AI Videos from browser GUI using Google Colab Notebook
2. Generating Stable Diffusion AI Videos via command line

Let’s go through them one by one.

Generating Stable Diffusion AI Videos from browser GUI using Google Colab Notebook

1. Open Google Colab Notebook in your browser.
2. From the top menu bar, select Runtime -> Run All
3. The notebook will prompt you to login to Hugging Face. Follow the instructions there.

Generating Stable Diffusion AI Videos via command line

To use this method, you need to have Python3 or above installed and should know your way around command line programs.

1. Download the code from this Github repo
   • There are two ways to do this. You can use either:
     1. Click on the green code button and press Download Zip, or
     2. open your command line application(called Terminal on Mac/Linux or cmd/Powershell on Win) and use the command: git clone git@github.com:nateraw/stable-diffusion-videos.git
   • Both of the above methods will download the code on your local machine. If you used the download zip method, you would need to extract the archive before proceeding further.
2. From the command line app, change the directory to the extracted code above (cd <folder path>)
3. Install required Python modules by running the command: pip install -r requirements.txt.If this command gives an error, try pip3 install -r requirements.txt. If you are still seeing errors then you are either not in the correct folder/directory or need to install IP or pip3 on your system.
4. Run below command to generate a video from a text prompt

python stable_diffusion_walk.py \
--prompts "['earth', 'mars', 'jupiter']" \
--seeds 903,123,42 \
--output_dir dreams \
--name animals_test \
--guidance_scale 8.5 \
--num_steps 5 \
--height 512 \
--width 512 \
--num_inference_steps 50 \
--scheduler klms \
--disable_tqdm \
--make_video \
--use_lerp_for_text \
--do_loop

replace the given prompts with your own. Play around withe settings to see what you can geenrate.