Handwritten Text Generation from Visual Archetypes

🤗 HuggingFace implementation!

This repository contains the reference code and dataset for the paper Handwritten Text Generation from Visual Archetypes. If you find it useful, please cite it as:

@inproceedings{pippi2023handwritten,
  title={{Handwritten Text Generation from Visual Archetypes}},
  author={Pippi, Vittorio and Cascianelli, Silvia and Cucchiara, Rita},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year={2023}
}

Installation

conda create --name vatr python=3.9
conda activate vatr
conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.7 -c pytorch -c nvidia
git clone https://github.com/aimagelab/VATr.git && cd VATr
pip install -r requirements.txt

From this folder you have to download the files IAM-32.pickle and resnet_18_pretrained.pth and place them into the files folder.

gdown --folder "https://drive.google.com/drive/u/2/folders/1FGJe2uCuK8T9HrFzY_Zc-KMIo0oPJGGY"

Training

python train.py

Useful arguments:

python train.py
        --feat_model_path PATH  # path to the pretrained resnet 18 checkpoint. If none, the resnet will be trained from scratch
        --dataset DATASET       # dataset to use. Default IAM
        --resume                # resume training from the last checkpoint with the same name
        --wandb                 # use wandb for logging

Pretraining dataset

The model resnet_18_pretrained.pth was pretrained by using this dataset: download link

Generate styled Handwtitten Text Images

🤗 HuggingFace implementation!

To generate all samples for FID evaluation you can use the following script:

python generate_fakes.py --checkpoint files/vatr.pth

To generate a specific text with a given input style folder containing images of handwritten single words you can use the following script:

python generator.py --style-folder "files/style_samples/00" --checkpoint "files/vatr.pth" --output "files/output_00.png" --text "That's one small step for man, one giant leap for mankind ΑαΒβΓγΔδ"

Output for That's one small step for man, one giant leap for mankind ΑαΒβΓγΔδ:

Implementation details

This work is partially based on the code released for Handwriting-Transformers