We propose Decoupled Matching Network (DMNet) which decouples what an object is from how it's posed, then fuses multiple supports with cyclic, reliability-weighted refinement - improving few-shot segmentation most where prior methods break down: large viewpoint and scale mismatch.
Department of Computer Science, Stony Brook University, Stony Brook, NY, USA
Abstract
Few-shot semantic segmentation (FSS) often degrades when support and query dmnet-images exhibit large pose or scale differences, since conventional prototype matching operates in a geometry-entangled feature space. We propose a lightweight encoder–decoder framework that disentangles object representations into a geometry-invariant semantic code and a low-dimensional modulation code capturing instance-specific geometric variation.
The reconstruction decoder is trained with equivariance constraints so geometric information is absorbed by the modulation pathway while semantic consistency is preserved in a separate branch - improving robustness to pose-scale discrepancies before segmentation, without explicit cross-image alignment. To strengthen supervision from limited supports, we introduce a refinement module that synthesizes lightweight augmented views of each support and performs cyclic ensemble refinement for more stable predictions. Predictions from multiple supports are then fused with spatially adaptive reliability weighting, producing cleaner, better-aligned query predictions. Across standard FSS benchmarks, DMNet consistently improves performance - particularly under large viewpoint change - and ablations confirm both the disentanglement and the cyclic, spatially weighted refinement are critical to the gains.
Method
Rather than estimating an explicit geometric transform to align support and query - brittle when object categories and instances are unseen at test time - DMNet takes the opposite approach: geometric variation should never be allowed to disrupt semantic matching in the first place.
The object-centric features of support and query are passed through a shared encoder and decomposed into two compact latent codes: a semantic code capturing geometry-invariant class identity, and a modulation code - visualized as a 2D geometric pointer - encoding instance-specific pose and scale. A decoder recombines both codes into reconstructed features under an equivariance constraint, so similarity transforms applied in the modulation subspace produce matching transforms in the reconstruction. A lightweight segmentation head Dseg then predicts the query mask from the geometry-decoupled features.
The modulation code stores the instance-specific geometric information of the object, such as pose, scale, rotation, and spatial layout. DMNet projects this high-dimensional modulation code into a compact 2D space using a learnable matrix. This 2D representation becomes the geometric pointer p, which allows the model to control geometry separately from semantic identity. By transforming this pointer and forcing the reconstructed feature map to transform consistently, DMNet learns a modulation space where geometry changes are predictable and do not corrupt semantic matching.
After semantic and geometric information are decoupled, DMNet further improves prediction quality using two refinement strategies: cyclic ensembling for more stable matching and spatially adaptive fusion for better boundary-aware segmentation.
Cyclic ensembling improves robustness by repeatedly exchanging support and query roles during matching. Instead of relying on a single one-way prediction, DMNet aggregates multiple cyclic predictions so that the final mask becomes more stable and less sensitive to accidental feature misalignment.
Spatially adaptive fusion combines information from different feature streams based on local image content. This helps the model preserve fine object boundaries while still using high-level semantic cues for category-level matching.
Results
DMNet consistently improves few-shot segmentation performance by reducing geometric mismatch between support and query features. Instead of only reporting numbers in a table, the results below highlight the main performance trends visually.
75.6
mIoU on PASCAL-5i, 1-shot
+2.1
improvement over the strongest baseline
✓
semantic matching is less disrupted by object geometry
Citation
@inproceedings{biswas2026dmnet,
title = {Mitigating Pose-Scale Discrepancy Bias and Reforming
Multi-Support Reasoning for Few-Shot Semantic Segmentation},
author = {Biswas, Shreya and Yin, Zhaozheng},
booktitle = {European Conference on Computer Vision (ECCV)},
year = {2026}
}