Out of memory!

[primepake]

I used the same config with your training but got OOM! Are you sure it's correct?

[LorenzoAgnolucci]

Hi! What GPU are you using?

[primepake]

I'm using H100 with 80GB

[LorenzoAgnolucci]

I see. Are you training the model with the default resolution?

[primepake]

yes 512x512? have you tried with accelerator? I trained with it and got OOM

[LorenzoAgnolucci]

By default, the --train-patch-size parameter is set to 128. I've never tried using accelerator, sorry.

[primepake]

I just started the training but I'm checking your code, it only input with the shape
imgs_lq.shape, imgs_gt.shape, imgs_ref.shape: torch.Size([2, 5, 3, 128, 128]) torch.Size([2, 5, 3, 128, 128]) torch.Size([2, 5, 3, 128, 128])
so why in the inference you run with 512 image?

[primepake]

can you share the chart of you loss values?

[LorenzoAgnolucci]

In image and video restoration, training with a patch size smaller than the test one is quite a common way to save memory during training. On which dataset are you training? Is it the one of the paper or a custom one?

[primepake]

I'm training with car dataset

[LorenzoAgnolucci]

Can I see a video example somewhere? Is it public?

[primepake]

yes! it recorded around the car
https://www.kaggle.com/datasets/tapakah68/videos-around-cars

[LorenzoAgnolucci]

What's the end goal of your work? Because the videos don't seem degraded with artifacts similar to those considered in our paper.

[primepake]

I want to increase the quality of the image by degradation ground truth image while training.

[LorenzoAgnolucci]

What kind of degradation? Our model is designed for analog video restoration, I'm not sure it's suitable for your purpose.

[primepake]

it's simple degradations (gaussian noise, blur gaussian, blur part of car, downsample_upsample) in video

[LorenzoAgnolucci]

How did you determine the reference frames? Did you use the standard textual prompts or did you change them to fit your needs?

By the way, I'm closing the issue since the out of memory problem is now solved.

[primepake]

I just select random from dataset. Do you if we select different views it will support that?

[LorenzoAgnolucci]

Why should random frames represent reference frames for the restoration? In the paper, we devised a specific methodology for analog videos. You should try to adjust that for your needs if you want to use our model for your purpose.

[primepake]

I'm just thinking while training we can select random then the model can learn better after that, in the inference stage we can choose reference based on CLIP score.

[LorenzoAgnolucci]

I've never tried it, so I'm not sure if it will work (or not)

[primepake]

Thanks! Can you provide your training chart?

[LorenzoAgnolucci]

I don't have the charts anymore, but I have the values of the losses at the end of the training: about 3.2 for the pixel_loss, and about 4.6 for the perceptual_loss. However, since you are using a different training dataset, I don't think these values are useful for you. But from what I remember, the profiles of the losses were similar to your charts.

[primepake]

Thanks a lot!