Evaluation of disparity maps

Abstract

A  disparity  map  is  the  output  of  a  stereo  correspondence  algorithm.  It  is  estimated  in  an  intermediate  step  of  a  3D information recovery process, from two or more images. A performance assessment of stereo correspondence algorithms may be addressed by a quantitative comparison of estimated disparity maps against ground-truth data. This assessment requires of the use of a methodology, which involves several evaluation elements and methods. Some elements and methods have been discussed with more attention than others in the literature. In the one hand, the quantity of used images and their relation to the application domain are topics rising large debate. On the other hand, there exist few publications on evaluation measures and error criteria. In practice, contradictory evaluation results may be obtained if different error measures are used, even on a same test-bed. In this paper, an evaluation methodology for stereo correspondence algorithms is presented. In contrast to conventional methodologies, it allows an interactive selection of multiple evaluation elements and methods. Moreover, it is based on a formal definition of error criteria based on set partitions. Experimental evaluation results showed that the proposed methodology allows a  better  understanding  and  analysis  of  algorithms  performance  than  the  Middlebury  methodology.  Final  remarks  highlights the relevance of discussing on the different elements and methods involved in an evaluation process.