Category Theory, Dragons and Self-Love

This is a real-time reading review series of Applied Category Theory Papers. This one is covering Compositionality for Recursive Neural Networks by Martha Lewis, U. Amsterdam<- the person who wrote and actually understands this paper, and to whom I must give all apologies for my limited understanding.

1. Vector space semantics — the meaning of words as vectors is in the most basic sense just something like: sentence = function (word 1, word 2, word 3 …) and that’s stupid and useless for most organic language if you actually want to do effective prediction. You could do something like co-occurrence stats but for (better/worse) those sentence structures often have antonyms in the same place so, that’s a total dud.
2. So then, if it’s a hard problem we can’t solve linearly let’s throw a neural network at it because (not even sarcasm here) what the hell else can we do? BUT WAIT, HOW DOES THE NEURAL NETWORK ACTUALLY WORK, IF IT DOES WORK?
3. Enter Category Theory, of course. Well, specifically enter Mehrnoosh Sadrzadeh el al. application of it to semantics in 2013, and the well-described general back-prop stuff is in Bolt et al., 2017.

Alright here’s the brief:

So this is an interesting combination of categorical state representation with tensors (informed by a TreeRNN) and demonstrated with ‘diagrammatic calculus (thanks Statebox).

So when “Dragons Breath Fire”:

A huge issue with using categorical composition to model semantics is that the dimensionality of the tensors needed is too damn high — and training gets too expensive. Although, in a world where training NLP has the carbon cost greater than the whole life of a cow, how bad can it be?

The valid clap back to traditional TreeRNN modeling provided in this paper is that it can’t tell the difference between stuff like “dog” and “brown” — basically, it can’t help us know if a word is routing more information through it, whereas categorical representation gives us that like WHOOP, THERE IT IS:

Now Let’s Learn to Love Ourselves

Reflexive pronouns like “himself” play an information routing role and are traditionally stupid hard to figure out in a traditional NLP vectoring method. But if you make it categorical and represent that information transfer, it just points to both the subject and the object of the verb, creating this totally awesome categorical display that may or may not be my next way too obscure reference tattoo idea:

Honestly, how dope is that categorical representation.

So the moral of the whole story is that “information-routing words can be understood as part of the structure of the tree, rather than as vectors”. So, two major cool things from this paper:

1. Using CT for a linear version of TreeRNNs is totally doable and simplifies the training for it
2. The categorical model is actually more flexible in the way that it represents tricky semantic concepts like reflexive-pronoun-routed self-love.