Deep Classifiers Ignore Almost Everything They See (and how we may be able to fix it)

Jörn Jacobsen
Mar 25 · 6 min read
Excessive Invariance: All images shown cause a competitive ImageNet-trained network to output the exact same probabilities over all 1000 classes (logits shown above each image).

Exploring Invariances of Learned Classifiers

We split the output of the invertible network into two subspaces: Zs represents the class scores and Zn everything not seen by the classifier.
Analytically Analyzing Logit Pre-images: Compute hidden representation for one image (left), throw away Zn, but keep logits Zs. Compute hidden representation for an arbitrary image from other class (right), throw away Zs, but keep Zn. Concatenate resulting Zs and Zn, invert the network and look at result!
Top row: images from which logit vectors Zs are taken. Bottom row: images from which nuisance vectors Zn are taken. Middle row: resulting inverted images with identical logit configurations as top row images. We have analytically computed adversarial examples!

We have stumbled upon an analytic adversarial attack.


How is this Related to Adversarial Examples?

The classical viewpoint (short orange arrow): perturbation-based adversarial examples x* apply changes to an input x such that x* stays in the same ground truth class as x, while crossing the decision-boundary (dashed line) of the model. Our alternative viewpoint (long pink arrow): invariance-based adversarial examples x* apply changes to an input x that change the ground truth class of x*, without crossing the learned decision-boundary.

Why are Deep Classifiers so Invariant?

Left: Cross-entropy trained networks are easily attacked with our analytic invariance-based attack. Right: Independence cross-entropy trained model. Our attack is not successful anymore, it is only able to change the style of the digit, not its semantic content.

Main Reference:

Thanks to Richard Zemel

Jörn Jacobsen

Written by

Postdoctoral Fellow at Vector Institute

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade