← Back to Projects

Hint++: Human-Guided 3D Adaptation

My Hint++ work covers frozen-teacher evaluation, an entropy-only adaptation baseline, and adaptive safety signals for 3D semantic segmentation. It investigates how human corrections can guide model updates while protecting regions that should remain stable.

Technologies used

Python
PyTorch
Point Transformer V3
SONATA
Pointcept

When an overall score hides the failure

A segmentation model can perform well overall and still miss a class that matters. My work on Hint++ starts with that problem: measure where a pretrained 3D model fails, then investigate how human feedback can guide adaptation without needlessly disturbing its other predictions.

Frozen SONATA teacher prediction for S3DIS office_39, showing segmented objects and the reported beam-class failure — Frozen-teacher prediction for S3DIS office_39. This is the baseline, not an adapted Hint++ result.

Establishing a fixed teacher

The implementation uses Point Transformer V3 with SONATA features through Pointcept. A frozen teacher supplies a fixed reference for the research. Keeping it unchanged makes later student updates comparable against a stable set of predictions.

The repository’s S3DIS Area 5 baseline reports 75.41% mean intersection-over-union, but only 0.19% IoU for beams. These are frozen-teacher results, not an improvement delivered by Hint++. The per-class breakdown is the reason to look beyond a single aggregate metric.

Per-class S3DIS Area 5 baseline IoU chart with 75.41 percent mean IoU and 0.19 percent beam IoU — The frozen-teacher baseline reports 75.41% mean IoU but 0.19% beam IoU. Aggregate performance hides a severe rare-class failure.

Testing the entropy-only baseline

I also investigated TENT-style adaptation, which reduces prediction entropy without human labels. In the PTv3 implementation, adaptation targets LayerNorm scale and shift parameters while leaving the rest of the model frozen. Online and episodic experiments distinguish accumulated updates from a fresh model for each room.

The limitation is that confidently sharper predictions are not necessarily more accurate. Common classes dominate the point-level objective, so rare classes require separate inspection. The TENT experiments and the frozen-teacher baseline are distinct runs and should not be combined into one reported result.

Turning correction history into a safety signal

The adaptive-moment component tracks first and second moments of a per-class correction signal. A teacher-confidence prior scales the resulting weights. The implementation stores this state as module buffers so it can be saved, restored, and reset between experiments.

m_hat = m_k / (1.0 - beta1 ** t)
v_hat = v_k / (1.0 - beta2 ** t)
weight = eta * eta_k * m_hat / (sqrt(v_hat) + eps)

The denominator uses the history of squared corrections, while the first moment reflects their direction and consistency. This is a per-class control signal for further research. It is not, by itself, proof that an update preserves segmentation accuracy.

What is implemented and what remains a research plan

The inspected source contains frozen-teacher evaluation, TENT baseline experiments, initialization outputs, and the adaptive-moment module with numerical checks and unit tests. A permission field, a protected-region KL gate, exemplar memory, and the complete adaptation loop remain parts of the wider research plan.

A divergence check against a teacher would constrain disagreement with that teacher. It cannot guarantee correct predictions when the teacher is wrong. I would evaluate accepted and rejected updates, rare-class accuracy, and collateral changes before claiming a complete safety guarantee.