Benchmarking in Biomedical Image Analysis

Biologists and physicians have to be able to rely on the correctness of results obtained by automatic analysis of biomedical images. This, in turn, requires paying proper attention to quality control of the developed algorithms and software for this task. Both the medical image analysis and bioimage analysis communities are becoming increasingly aware of the strong need for benchmarking various image analysis methods in order to compare their performance and assess their suitability for specific applications. Benchmark datasets with reference annotations (both simulated and real data annotated by experts) have become publicly available and challenges are being organized in association with well-known conferences, such as ISBI and MICCAI.

We follow the recent developments in this respect and common ways of measuring algorithm performance and try to contribute to forming guidelines for best practices for designing biomedical image analysis benchmarks and challenges, including proper dataset selection (training versus test sets, simulated versus real data), task description and defining corresponding evaluation measures that can be used to rank performance. We closely collaborate in this respect with challenge chairs of ISBI and MICCAI conferences and participate in the activities of MICCAI Challenge Working Group.

Proper benchmarking of image analysis algorithms and software makes life easier not only for developers (who can learn the strengths and weaknesses of existing methods) but also for users (who can select methods that best suit their particular needs). Also reviewers can better assess the usefulness of a newly developed analysis method if it is compared to the best performing methods for a particular task on the same type of data using standard measures.

Our participation in this effort is threefold:

1. We attend to the benchmarking of cell segmentation and tracking methods. We have established an international benchmarking framework in this field called Cell Tracking Challenge hosted by ISBI 2013, 2014, 2015, 2019 and 2020 and running also in open submission mode. In October 2018, a new “spin-off” Time-lapse Cell Segmentation Benchmark was launched to address numerous requests for benchmarking only cell segmentation methods. We have also attended to the benchmarking of algorithms for small object (dot, spot, particle) segmentation and tracking. For more details, see:

   • MAIN PAPER on An Objective Comparison of Cell Tracking Algorithms (Nature Methods, Dec 2017)
   • Web resource for benchmarking: Cell Tracking Challenge (online submission mode since Feb 2017)
   • Web resource for benchmarking: Masaryk University Cell Image Collection (in one place since Sep 2016)
   • Paper on Cell tracking accuracy measurement (PLoS ONE, Dec 2015)
   • Paper on Evaluation of 3-D Spot Detection Methods in Confocal Microscopy (Cytometry, May 2015)
   • Paper on A Benchmark for Comparison of Cell Tracking Algorithms (Bioinformatics, Jun 2014)
   • Paper on Particle Tracking Challenge (Nature Methods, Mar 2014)

2. We try to spread awareness of the importance of this topic by publishing papers, giving lectures and organizing tutorials on different occasions, for example:

   • NEW! Paper on Transparent Reporting of Biomedical Image Analysis Challenges (Medical Image Analysis, Dec 2020)
   • Paper on Why Rankings of Biomedical Image Analysis Competitions Should Be Interpreted with Care (Nature Communications, Dec 2018)
   • Lecture on Benchmarking at the Course on Advanced Methods in Biomedical Image Analysis (CBIA Brno, Sep 2017 and Dec 2020)
   • Invited talk on Benchmarking in Biomedical Image Analysis (IUPESM, Jun 2018)
   • Invited talk on Rigorous Bioimage Analysis using Benchmarking (Bioimage Informatics, Sep 2017)
   • Invited talk on Synthetic Cell Image Datasets for Benchmarking (NEUBIAS Symposium, Feb 2017)
   • Invited talk (record available) on Analysis and Synthesis of Cell Images (Prague Computer Science Seminar, Oct 2016)
   • Tutorial on Benchmarking in Biomedical Image Analysis (MICCAI, Sep 2017)
   • Tutorial on Benchmarking in Bioimage Analysis (NEUBIAS Training School, Feb 2017)
   • Tutorial on Benchmarking in Biomedical Image Analysis (ISBI, Apr 2016)
   • Review on Simulations in Cell Imaging used for Benchmarking (Cytometry, Dec 2016)
   • Review on Benchmarking in Bioimage Analysis (Springer, May 2016)

3. We participate in international infrastructural projects and initiatives that include benchmarking in biomedical image analysis, namely:

   • MAIN ACTIVITY: MICCAI Challenge Working Group (MICCAI, since Jul 2018)
   • Networking project Network of European BioImage Analysts (NEUBIAS) (EU COST, since May 2016)
   • Networking project Czech-BioImaging (part of Euro-BioImaging, CZ/EU Research Infrastructure, since Jan 2016)

Copyright notice: Banner image ("measure a thousand times, cut once" by Avrene) is released under the creative common license.