Benchmarking in Biomedical Image Analysis

Biologists and physicians have to be able to rely on the correctness of results obtained by automatic analysis of biomedical images. This, in turn, requires paying proper attention to quality control of the developed algorithms and software for this task. Both the medical image analysis and bioimage analysis communities are becoming increasingly aware of the strong need for benchmarking various image analysis methods in order to compare their performance and assess their suitability for specific applications. Benchmark datasets with reference annotations (both simulated and real data annotated by experts) have become publicly available and challenges are being organized in association with well-known conferences, such as ISBI and MICCAI.

We follow the recent developments in this respect and common ways of measuring algorithm performance and try to contribute to forming guidelines for best practices for designing biomedical image analysis benchmarks and challenges, including proper dataset selection (training versus test sets, simulated versus real data), task description and defining corresponding evaluation measures that can be used to rank performance. We closely collaborate in this respect with challenge chairs of ISBI and MICCAI conferences and participate in the activities of MICCAI Challenge Working Group.

Proper benchmarking of image analysis algorithms and software makes life easier not only for developers (who can learn the strengths and weaknesses of existing methods) but also for users (who can select methods that best suit their particular needs). Also reviewers can better assess the usefulness of a newly developed analysis method if it is compared to the best performing methods for a particular task on the same type of data using standard measures.

Our participation in this effort is threefold:

  1. We attend to the benchmarking of cell segmentation and tracking methods. We have established an international benchmarking framework in this field called Cell Tracking Challenge hosted by ISBI 2013, 2014, 2015, 2019 and 2020 and running also in open submission mode. In October 2018, a new “spin-off” Time-lapse Cell Segmentation Benchmark was launched to address numerous requests for benchmarking only cell segmentation methods. We have also attended to the benchmarking of algorithms for small object (dot, spot, particle) segmentation and tracking. For more details, see:

  2. We try to spread awareness of the importance of this topic by publishing papers, giving lectures and organizing tutorials on different occasions, for example:

  3. We participate in international infrastructural projects and initiatives that include benchmarking in biomedical image analysis, namely:

Measuring Error
Copyright notice: Banner image ("measure a thousand times, cut once" by Avrene) is released under the creative common license.