• [2024/01/02] 🔥 Seg-TTO is out! We release the paper and code for inference and deployment. Seg-TTO achieves SOTA results on supervised and unsupervised open-vocabulary semantic segmentation.

We present Seg-TTO, a plug-in-play module focusing on segmentation-specific test time prompt tuning and visual attributes to address the lack of knowledge of vision language models in domain-specific datasets.

For further details and visualization results, please check out our [paper TODO] and our [project page TODO].

Please follow this to setup environment for CAT-Seg version and this to setup environment for CLIP-DINOiser version of Seg-TTO.

Please follow dataset preparation.

If you want to try your own images locally, please try cat seg demo and clip dinoiser demo.

Please follow this to evaluate CAT-Seg version and this to evaluate CLIP-DINOiser version of Seg-TTO.

Figure 2: Zero-Shot Semantic Segmentation on Out-of-Domain Datasets

Figure 3: Zero-Shot Unsupervised Semantic Segmentation on Out-of-Domain Datasets

We provide pre-trained weights for our models reported in the paper. All of the models were evaluated with two 24GB NVIDIA RTX A5000 or 16GB NVIDIA Quadro RTX 5000 and can be reproduced with the evaluation script above. Please note that the results may vary slightly with installed module versions.

We would like to acknowledge the contributions of public projects CAT-Seg, MESS and CLIP-Dinoiser, whose code has been utilized in this repository.

@misc{desilva2025SegTTO,
      title={Test-Time Optimization for Domain Adaptive Open Vocabulary Segmentation}, 
      author={Ulindu De Silva and Didula Samaraweera and Sasini Wanigathunga and Kavindu Kariyawasam and Kanchana Ranasinghe and Muzammal Naseer and Ranga Rodrigo},
      year={2025},
}