Abstract

Spurious correlations are a major source of errors for machine learning models, in particular
when aiming for group-level fairness. It has been recently shown that a powerful approach to
combat spurious correlations is to re-train the last layer on a balanced validation dataset, isolating
robust features for the predictor. However, key attributes can sometimes be discarded by neural
networks towards the last layer. In this work, we thus consider retraining a classifier on a set of
features derived from all layers. We utilize a recently proposed feature selection strategy to select
unbiased features from all the layers. We observe this approach gives significant improvements
in worst-group accuracy on several standard benchmarks. Another pain point in transfer learning
is with out-of-distribution tasks having large distribution shifts relative to the source task. Full
finetuning suffers in performance as it disturbs backbone parameter weights during the starting
few optimization steps and is forced to make drastic adaptations to correct for large losses initially
observed in training. Linear tuning is another approach shown to improve model generalization
capabilities and is especially effective for transfer learning on out-of-distribution downstream tasks.
We further evaluate the usefulness of intermediate layer information by incorporating it with a linear
tuning approach. Results over datasets from a common visual task adaptation benchmark show that
the empirical benefits from simply leveraging intermediate layers are similar to the proposed method
and there is no noticeable gain in accuracy from incorporating a linear tuning step.