Hacking Type Erasure with Super Type Token

How do you differentiate a List of Integer from a List of String at run time?

If you’re writing application code, the need for Reified Generics usually signal code smell, you should probably refactor your way out of it. Code-generation-heavy libraries however, depends their lives on it. Good news is that there is a hack to achieve it.

The application of Reified Generics is most visible to us when dealing with object serialisation, e.g. How do you parse a JSON array into a list of User, vice versa. You have to somehow capture the type of List<User> and give it to the parser.

Most of the popular libraries provide a way to capture this complex type, but this leads to leaky abstraction and forces you to adopt their implementation of the hack, as we shall see by exploring the API of 3 major libraries. Then, we will learn the underlying working principle of the hack by building a simple version of Super Type Token.

Photo by eberhard grossgasteiger on Unsplash

Retrieving an Object using HTTP Client

Given a sample JSON

And a User object

Imagine that there’s a server that lets you retrieve a specific User. If you’re using Spring, you’re probably using the build-in RestTemplate or WebClient.

RestTemplate (Blocking, scheduled for deprecation)

WebClient (Non-blocking, reactive)

In both example we are instructing Spring to deserialise the JSON response into a User object by passing in User.class, or Class<User>.

User.class, or Class<User> is known as Type Token. A Type Token allows you to capture simple reifiable type.

Retrieving a List of Object using HTTP Client

It is also common to retrieve a list of objects over the network.

While the server should probably send a top level object instead of array to avoid JSON Hijacking, many time you have to deal with API that is out of your control. Usually that means deserialising the API response into a list of object, in this case, List<User>. This is where the complication arise, how do we capture a non-reifiable type? You can't do List<User>.class because the Type Erasure mechanism doesn't allow that. List.class won't work either because you’re not being specific.

We need a way to exactly represent a complex non-reifiable type like List<User>.

Super Type Token

Remember the time I mentioned that there is some obscure black magic that allows you to capture the run time type information (RTTI) of non-reifiable generic type? This is it.

A Super Type Token allows you to capture some non-reifiable type statically at compile time.

To solve our JSON serialisation problem, Spring introduced a Super Type Token called ParameterizedTypeReference. This is how it looks like.

Pretty standard declaration, except that extra pair of braces at the end stood out like a cancerous tumour— an anonymous class. Don’t worry, we will explore the Why later.

Like any good API, Spring provides method overload for ParameterizedTypeReference, you could simply swap User.class with a ParameterizedTypeReference that captures List<User> type, and you will be able to retrieve List<User> from the JSON response.

RestTemplate

WebClient

Side note for WebClient, a more idiomatic way of retrieving a list of objects would be using .bodyToFlux(User.class) and store it as Flux<User>, this way you’re making full use of the Publisher's capabilities and let the framework manages the gory serialisation.

Baader-Meinhof Phenomenon

Or Frequency Illusion, once you have learnt something new, the weird thing that you’ve never seen before starts appearing more frequently in your life. Super Type Token pattern can be found in most serialisation library like Gson and Jackson. Of course they are named differently, but the underlying principle remains the same. Let’s take a look at Gson and Jackson’s implementation of Super Type Token, take note of the anonymous class pattern used in both example.

Gson

It’s kinda weird that fromJson takes a java.lang.reflect.Type instead of providing an overload for Guava's own TypeToken. My guess is that because TypeToken is still beta at the time of this writing.

Jackson

As we have seen with ParameterizedTypeReference, TypeReference, and TypeToken, they work the same way — wrapping the type that you want to capture in a custom class as parameterised type.

Spring stays true to their nature and prefer longer and explicit name, though I personally find TypeToken or even TypeReference more succinct. I’m sure there are other flavour of Super Type Token out there, but these 3 should cover the majority of the use cases.

Super Type Token Hacks

This hack exploits the fact that,

1. Anonymous class is not subjected to Type Erasure
2. Type parameter of the generic superclass can be retrieved with Reflection

Let’s try and build our own version of Super Type Token, and see how this 2 points interact to achieve the hack. Trust me, you can build a simple version in 2 steps, with just 8 lines of code.

Step 1 — Preserving T at Runtime

A Super Type Token is nothing more an abstract class that is parameterised by a single argument T — the type that we want to capture. It has a single method getType that allows you to get the RTTI of T at run time. Let's focus on the contract, we will implement the method in Step 2.

If you look at source code of Spring, Guava, and Jackson’s implementation, they are all abstract classes. By making it abstract, we are forcing the client to instantiate it through an anonymous class. Why? Because we want to exploit point 1. If we instantiate SuperTypeToken through an anonymous class, then we can preserve the type of T. Let’s see how.

Line 1 is of particular interest, when we create an anonymous class of SuperTypeToken, the compiler actually generate a concrete subclass that extends SuperTypeToken, the generated class looks like this.

Notice that the generated code has List<String>, it is not erased! One important consequence of using anonymous class is that the generated class is NOT generic and thus is not subjected to Type Erasure, so the type List<String> is preserved in the generated byte code. Now, all we have to do is to figure out a way to retrieve it through the getType method.

Witnessing Anonymous Class in the Wild

Instead of using anonymous class, if we try to simulate the compiler-generated class by creating a subclass of SuperTypeToken, then invoke getType, we will get exactly the same output.

But it would be kinda fucked up to have to create an additional class just for the purpose of capturing a type. Nice thing about anonymous class is that, at the time of instantiation, functionally you are creating a subclass with T locked in as List<String>, as demonstrated by CompilerGeneratedSubClass.

Step 2: Using Reflection to Get the RTTI of T

Now that we know why SuperTypeToken needs to be instantiated in an oddly specific manner, let’s move on and see how we can retrieve T at run time. In other word, what is that forbidden sorcery inside thegetType method?!

Not bad. Not bad at all. Only 4 additional lines of code. Well technically we could do it in 1 line but let’s not.

Breaking it down

1. getClass returns the class of the generated subclass, recall CompilerGeneratedSubClass, because it is the one calling this method.
2. getGenericSuperclass returns the class of SuperTypeToken.
3. The generic super class returned is of type java.lang.reflect.Type, not very useful, so we cast it to ParameterizedType which offer methods to retrieve RTTI of T. The cast is guaranteed to succeed because we know that SuperTypeToken is parameterised.
4. getActualTypeArguments returns an array of parameterised type. Invoking this call on a Map<K, V> will return an array of [K, V], invoking on SuperTypeToken<List<String>> will return an array of [List<String>], so just get the first element from the array and it’s done!

Does It Work with All Non-Reifiable Types?

Super Type Token has some limitation, I define it earlier as something that allows you to capture some non-reifiable type statically at compile time.

Not all non-reifiable types can be captured. As a rule of thumb, whatever type that you want a Super Type Token to capture, it must be known statically at compile time, like List<String> or HashMap<String, Supplier<User>>. You cannot use it to capture a T, List<T>, List<? extends Number>, or anything that is not fully known at compile time.

To understand why, take a look at the generated subclass of SuperTypeToken when we try to capture a T.

Now when we invoke getType, it will just return T as it should, because that’s what it was given at compile time, regardless of what T might be at run time.

Conclusion

That’s all folks, explain to your colleagues what those{} means next time you encounter a Super Type Token!

1. User.class, or Class<User> is known as Type Token. A Type Token allows you to capture reifiable type.
2. A Super Type Token allows you to capture some non-reifiable type statically at compile time.
3. It is a hack based on Anonymous Class and Reflection.
4. It doesn’t work with unknown type like T, List<T>, List<? extends Number.
5. Spring’s ParameterizedTypeReference, Jackson’s TypeReference, Gson’s TypeToken are different implementations of Super Type Token, same working principle.

Follow me @ https://twitter.com/darrenbkl