Finnegans Wake — A Study in Information Entropy
Entropy is a measure of disorder in a system. A porcelain plate may be considered to have low entropy, it is a well organized structure. Smash that plate on the floor and the entropy of the system dramatically increases, there is less order. We may also look at entropy as a measure of randomness (tantamount to a measure of predictability). And from this perspective we can formulate the analogous notion of Information Entropy.
Wikipedia:
Information entropy is defined as the average amount of information produced by a stochastic source of data.The measure of information entropy associated with each possible data value is the negative logarithm of the probability mass function for the value. Thus, when the data source has a lower-probability value (i.e., when a low-probability event occurs), the event carries more “information” (“surprisal”) than when the source data has a higher-probability value. The amount of information conveyed by each event defined in this way becomes a random variable whose expected value is the information entropy.
We may apply this concept to written works (expanding on the bold text above) by creating a measure of the probability value for each word, treating it as an event. We must assume that the author (or source of data) in our model is stochastic, which we of course know is false, but this assumption yields useful insight when comparing different works and authors. We can then formulate an average level of information for the text as a whole, and hence define an information entropy. It would be interesting to see what such a measure would yield when applied to different written works.
We may measure the probability value of each word by counting its occurrence and comparing it to the length of the text as a whole. We then compute the average information entropy for each word (or event) and obtain the expected information entropy.
In our first foray into this study we attempt to measure the ‘surprisal’, which is directly related to the information entropy. Here we compare the number of distinct words the total amount of words in the text. If we treat each distinct word (abstractly) as different idea, we can get a sense of the redundancy of the language used in each of the following texts. In other words, we attempt to measure how much linguistic scaffolding is used to support each idea, and perhaps infer the depth of meaning. In the following computations, we use the first 20,000 words of each text:
- Harry Potter and the Order of Phoenix ratio of distinct words to total: 0.061
- Crime and Punishment ratio of distinct words to total: 0.052
- Finnegans Wake ratio of distinct words to total: 0.268 (That means every 3 or 4 words James Joyce uses a word which has never occurred before in the text!)
From this we may compute the average information entropy (average negative log of the surprisal of each word) for each text:
- Harry Potter and the Order of Phoenix average information entropy: 11.264
- Crime and Punishment average information entropy: 11.212
- Finnegans Wake average information entropy: 11.934
I will refrain from making any definitive conclusions here, however the results are notable and fit our intuitions. A deeper study is to follow which may allow us to make more confident inferences about the power of the language used by each author, and perhaps formulate a measure of the depth of meaning per sentence.
Edits:
09–08–2023: Meaning is NOT the same thing as ‘Information Entropy’ as defined my Shannon. Meaning is something which the entity (man or machine) viewing the information gives it.
