How to improve the test performance on ScanNet?

[zymhnu]

Thanks for sharing. I had some problems when I submitted my testing results to ScanNet. I had a competitive performance on the validation set, such as 74.0% mIoU, but only had a low performance on the test set, like 71.0%mIoU. Moreover, I reproduced ptv2 and got 74.5% mIoU on the validation set, and I only got 71.1%mIoU when submitted to ScanNet.

[haibo-qiu]

Hi, I share the same concern as I achieved 74.3% on the ScanNet validation set but only managed to obtain 70.6% on the ScanNet test set.

[Gofinge]

I never encounter the same situation. I submitted the benchmark twice, one for 73.x and another is 75.x. We also discussed the phenomenon, and some senior researchers told me that the test result on the ScanNet benchmark is not stable, and the only solution is to try more time.

[zymhnu]

Thank you for your reply. But are there any tricks on the ScanNet test? Or just modify the value of "split"?