CatLIP: more efficient image model pre-training is worth web-scale text
Executive Summary for Managers/Leaders
What is it?: a new, (potentially) more efficient way to train deep learning computer vision models using both images and text, but that doesn’t rely on manually-labeled pairs of images and text.
Why should you care?: With computer vision, you’re trying to get an algorithm to find things in images better, faster, and more reliably than a human can. That starts by having to train the algorithm to find those things, and that typically means you need a person to “label” them.
Labeling images is costly; the alternative is long (and costly) compute as a model learns to represent things from the image in its own way. In other words, you have potentially long timelines to a practical, computer vision application regardless of the route you choose.
CatLIP[1] offers a faster way to train vision models that draws on the strengths of both common approaches. It does this while still learning good representations of what is in the images, meaning it has learned something useful.
Which questions should you be asking your DS/ML/AI folks?:
- Do we already have labeled images? If so, how many? If we don’t, how long do we have until we need to deliver something (and how much budget do we have for compute)?
- Does the thing we’re trying to find in the images we care about look enough like objects that are captured in common benchmark datasets (like ImageNet)? If not, how do you propose to account for these differences?
- Have we tried using a pre-trained vision model or API? If not, why not?
Summary for Data Scientists/ML Engineers/The-Technically-Curious
What is it?: a proposed alternative to the CLIP approach for the pretraining of vision transformer architectures for computer vision. CatLIP accelerates pretraining through using synsets of extracted nouns from image captions (rather than contrasting against negative image-text pairs using a text encoder) and subsequently reframing the pretraining problem such that binary cross-entropy loss can be used. This approach works in part due to web-scale image-text datasets.
What is cool about it?: reframing the pretraining problem to classification, instead of contrastive pairwise similarity. This subtle shift in thinking led to efficiency, without (the authors claim) a sacrifice in performance or representation robustness. A focus on efficiency rather than aggregate performance is a secondary (and important for the planet) cool thing.
Questions I have:
- Are there other factors that could explain the efficiency gains other than the workflow changes and problem reframing?
- Are the benefits only realized at very large scale?
- What is the relative cost of generating the synset vocabulary compared to other methods?
Post your thoughts in the comments below!
[1] Mehta, S., et al., CatLIP: CLIP-level Visual Recognition Accuracy with 2.7x Faster Pre-training on Web-scale Image-Text Data. https://arxiv.org/abs/2404.15653
