Dreambooth trained on 5 images of a dog

source

​
What is Dreambooth?

  • Dreambooth is a method of personalizing text-to-image models, such as Stable Diffusion.
  • Dreambooth personalizes Stable Diffusion by fine tuning on a small set of reference images (around 3-5 images) that you provide.
  • Current models such as Stable Diffusion produce high-quality images, but it struggles to make new versions of your data in different settings or poses.

​
How Does Dreambooth Work?

  • During fine-tuning, Dreambooth learns to recognize unique features of your data.
  • The model associates an identifier (called sks) to your unique features.
  • The sks identifier allows the model to embed your unique features into the output domain of the model.
  • The sks identifier will be used to synthesize photorealistic images of your features in different scenes or poses.
  • The sks identifier was chosen randomly, because it is a rare word in the English dictionary.

​
Training with a sample dataset

Training data of five dog photos

  • Please download the images of the dog found here to the --dataset-dir folder.
  • --class-word describes what images in your dataset contain (in this example, the dataset contains dog photos).
  • --regularization-prompt describes the type of images you want to generate.
  • Make a directory with mkdir logs and pass in the directory as --log-dir to get logs in that directory.
  • We set sensible defaults for the steps, batch size, and learning rate. Refer to How do I change default settings? for more information. 
rubbrband train dreambooth --dataset-dir="./data" \
--class-word="dog" \
--regularization-prompt="a photo of a dog"
--log-dir logs

​
Inference on sample prompts

Output of the prompt 'sks dog on the beach'

  • After doing a training run on Dreambooth, run the rubbrband eval command.
  • In the --input-prompt, make sure the class-word (e.g. dog) is in the prompt.
  • In the --input-prompt, make sure the sks identifier is in the prompt. 
rubbrband eval dreambooth --input-prompt "sks dog on the beach"

​
Where are my outputs?

  • To save the output image on your local machine
rubbrband copy-from dreambooth /home/engineering/samples/ ./data
  • The checkpoint file is already on your local machine inside the --log-dir

​
How do I change default settings?

  • Dreambooth fine tuning is very sensitive and may require changing the default settings to get good outputs.
  • This blog post covers some of the settings that have been found to work well.