Support distributed evaluation

[mehdidc]

Currently, evaluation is done on rank zero. This PR provides support for distributed evaluation (using an optional --distributed-evaluation argument) to make evaluation faster (supports both zero-shot and retrieval).

[rom1504]

this LGTM

could you please double check @mitchellnw or @rwightman ?

[mitchellnw]

In the pytorch imagenet example for distributed imagenet eval they have an aux_val_loader to handle the case where the test set size is not divisible by num_gpus - do we need to have this? https://github.com/pytorch/examples/blob/main/imagenet/main.py#L396-L402

[usuyama]

nit pick but maybe better to pass a more specific argument e.g. gather_tensor=args.distributed_evaluation?

[mehdidc]

nit pick but maybe better to pass a more specific argument e.g. gather_tensor=args.distributed_evaluation?

Thanks I agree, I think it's better

[usuyama]

maybe I'm misunderstanding, but does it skip the last few samples due to the last partial batch?

[mehdidc]

maybe I'm misunderstanding, but does it skip the last few samples due to the last partial batch?

in evaluation, num_samples is only used for logging :

open_clip/src/training/train.py

Line 172 in 03839c5

samples_per_val = dataloader.num_samples

,

open_clip/src/training/train.py

Line 217 in 03839c5

f"Eval Epoch: {epoch} [{num_samples} / {samples_per_val}]\t"

it is not affecting the dataloader which depends only on the wds pipeline. But it would be good to get it correct anyway, have to check how many examples each worker exactly receive.

[mehdidc]

In the pytorch imagenet example for distributed imagenet eval they have an aux_val_loader to handle the case where the test set size is not divisible by num_gpus - do we need to have this? https://github.com/pytorch/examples/blob/main/imagenet/main.py#L396-L402

Thanks for the link, not sure why they do drop_last=True on val_loader (not used here), probably to avoid having a GPU worker with much fewer examples than the others? so rather, they seem to do drop_last, and compute the last few examples validation performance in all GPU workers.

[mitchellnw]

@mehdidc I think this is actually necessary or else you can get different val perf when different numbers of gpus are used, e.g., see this comment: https://github.com/facebookresearch/deit/blob/main/main.py#L221-L223

[mehdidc]

@mehdidc I think this is actually necessary or else you can get different val perf when different numbers of gpus are used, e.g., see this comment: https://github.com/facebookresearch/deit/blob/main/main.py#L221-L223

I see, thanks @mitchellnw! OK so I need to fix this. I really thought that DistributedSampler with drop_last=False would do the the "right" thing (although wouldn't be ideal for disributed setting if it would be case) in the sense of seing each example exactly once (even when dataset size is not divisible by nb of workers, e.g. the remaining examples can be distributed to a subset of workers)

[dmlpt]

Is this argument "--distributed_evaluation" not available in the current version?

[mehdidc]

@dmlpt Not yet, I still need to fix the val dataloader like @mitchellnw mentions and rebase on master