Our companion paper, On Challenging Aspects of Reproducibility in Deep Anomaly Detection, has been accepted for presentation at the Fourth Workshop on Reproducible Research in Pattern Recognition (satellite event of ICPR 2022).
In it, we discuss aspects of reproducibility for our anomaly detection algorithm MCHAD, as well as anomaly detection with deep neural networks in general. In particular, we discussed the following challenges for the reproducibility:
- Nondeterminism: conducting the same experiment with different random seeds might lead to significantly different outcomes.
- Sensitivity to hyper-parameters: slight changes in hyper-parameters can drastically alter the outcomes.
- Complexity: the more complex an algorithm, the more likely an implementation contains errors.
- Dataset Selection: The performance of a method is going to depend on the dataset on which you evaluate it.
- Resource Limitations: resource requirements can limit the number of individuals or institutions that are able to reproduce the training.
- Dependencies: dependencies, in the form of data, pre-trained weights, or software libraries, might get taken down at some point.
The large number of dependencies in our experiments may harm the reproducibility of our exact numerical results. However, we argue that the reproducibility of conclusions should be prioritized over the reproducibility of exact numerical results since the former contributes to the advancement of scientific knowledge.
