AUPIMO: Redefining Anomaly Localization Benchmarks with High Speed and Low Tolerance


João P. C. Bertoldo (PSL University), Dick Ameln (Intel), Ashwin Vaidya (Intel), Samet Akcay (Intel)
The 35th British Machine Vision Conference
PDFPosterVideo (Right click to download)Supplementary

Abstract

Recent advances in anomaly localization research have seen AUROC and AUPRO scores on public benchmark datasets like MVTec and VisA converge towards perfect recall. However, high AUROC and AUPRO scores do not always reflect qualitative performance, which limits the validity of these metrics. We argue that the lack of an adequate and domain-specific metric restrains progression of the field, and we revisit the evaluation procedure in anomaly localization. In response, we propose the Area Under the Per-IMage Overlap (AUPIMO) as a recall metric that introduces two major distinctions. First, it employs a validation scheme based solely on normal images, which avoids biasing the evaluation towards known anomalies. Second, recall scores are assigned per image, which is fast to compute and enables more comprehensive analyses (eg. cross-image performance variance and statistical tests). Our experiments (27 datasets, 8 models) show that the stricter task imposed by AUPIMO redefines anomaly localization benchmarks: current algorithms are not suitable for all datasets, problem-specific model choice is advisable, and MVTec AD and VisA have not been near-solved. Available on GitHub https://github.com/jpcbertoldo/aupimo.

Citation

@inproceedings{Bertoldo_2024_BMVC,
author    = {João P. C. Bertoldo and Dick Ameln and Ashwin Vaidya and Samet Akcay},
title     = {AUPIMO: Redefining Anomaly Localization Benchmarks with High Speed and Low Tolerance},
booktitle = {35th British Machine Vision Conference 2024, {BMVC} 2024, Glasgow, UK, November 25-28, 2024},
publisher = {BMVA},
year      = {2024},
url       = {https://papers.bmvc2024.org/0223.pdf}
}


Copyright © 2024 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection