RbA: Segmenting Unknown Regions Rejected by All

1KUIS AI Center and Department of Computer Engineering, Koç University 2University of Oxford
ICCV 2023
TL;DR Object queries in mask classification behave like one vs. all classifiers. We experimentally confirm this behavior and reinforce it to find unknown objects as pixels rejected by all known classes.


Standard semantic segmentation models owe their success to curated datasets with a fixed set of semantic categories,without contemplating the possibility of identifying unknown objects from novel categories. Existing methods in outlier detection suffer from a lack of smoothness and objectness in their predictions, due to limitations of the per-pixel classification paradigm. Furthermore, additional training for detecting outliers harms the performance of known classes. In this paper, we explore another paradigm with region-level classification to better segment unknown objects. We show that the object queries in mask classification tend to behave like one vs. all classifiers. Based on this finding, we propose a novel outlier scoring function called RbA by defining the event of being an outlier as being rejected by all known classes. Our extensive experiments show that mask classification improves the performance of the existing outlier detection methods, and the best results are achieved with the proposed RbA. We also propose an objective to optimize RbA using minimal outlier supervision. Further fine-tuning with outliers improves the unknown performance, and unlike previous methods, it does not degrade the inlier performance.

Method Overview


Object queries (colored ) in mask classifiers each specialize in detecting a different semantic class, and the performance of each query is independent of the other queries. Hence, we model each object query as an independent one vs. all classifier and consider unknown regions as those rejected by all (RbA) queries. RbA can be regularized with outlier data to supress the overconfidence of known class scores for unknown pixels.

Applying RbA to Real Life Videos

credit: https://www.youtube.com/watch?v=EOosn78WsMg

Qualitative Results

We show the qualitative results compared to previous SOTA on SMIYC Anomaly and Obstacle tracks:


Quantitative Results


          author    = {Nazir Nayal and Mısra Yavuz and João F. Henriques and Fatma Güney},
          title     = {RbA: Segmenting Unknown Regions Rejected by All},
          booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
          year      = {2023},