Dreambooth

Dreambooth trained on 5 images of a dog

What is Dreambooth?

Dreambooth is a method of personalizing text-to-image models, such as Stable Diffusion.
Dreambooth personalizes Stable Diffusion by fine tuning on a small set of reference images (around 3-5 images) that you provide.
Current models such as Stable Diffusion produce high-quality images, but it struggles to make new versions of your data in different settings or poses.

During fine-tuning, Dreambooth learns to recognize unique features of your data.
The model associates an identifier (called sks) to your unique features.
The sks identifier allows the model to embed your unique features into the output domain of the model.
The sks identifier will be used to synthesize photorealistic images of your features in different scenes or poses.
The sks identifier was chosen randomly, because it is a rare word in the English dictionary.

Training data of five dog photos

Please download the images of the dog found here to the --dataset-dir folder.
--class-word describes what images in your dataset contain (in this example, the dataset contains dog photos).
--regularization-prompt describes the type of images you want to generate.
Make a directory with mkdir logs and pass in the directory as --log-dir to get logs in that directory.
We set sensible defaults for the steps, batch size, and learning rate. Refer to How do I change default settings? for more information.

rubbrband train dreambooth --dataset-dir="./data" \
--class-word="dog" \
--regularization-prompt="a photo of a dog"
--log-dir logs

Output of the prompt 'sks dog on the beach'

After doing a training run on Dreambooth, run the rubbrband eval command.
In the --input-prompt, make sure the class-word (e.g. dog) is in the prompt.
In the --input-prompt, make sure the sks identifier is in the prompt.
- For more information on what sks means, refer to How Does Dreambooth Work?

rubbrband eval dreambooth --input-prompt "sks dog on the beach"

rubbrband copy-from dreambooth /home/engineering/samples/ ./data

Dreambooth fine tuning is very sensitive and may require changing the default settings to get good outputs.
This blog post covers some of the settings that have been found to work well.