MonkeyType: type inference for transpiling Python to Rust

In my previous post about Python to Rust transpiler I’ve said, that one of the biggest problems is absence of types in python and inability to infer them for functions. But it turns out there is a solution to this problem. Guys from Instagram have developed an incredible tool for runtime type inference called MonkeyType. It monitors what types functions accept and return while running a program. This approach works quite nicely so I want to share with you how it’s done.

Let’s take a look at the following totally practical example for determining if there is an even number in a list:

After transpiling to Rust you get:

This code can’t be compiled by rustc because only we know that argument is a vector, but not the compiler:

2 |     for num in numbers {
  |                ^^^^^^^ `T0` is not an iterator

And return type RT is not bool, for sure

4 |     return true;
  |            ^^^^ expected type parameter, found bool

But now let’s run our python code using monkeytype run __init__.py and then apply collected results by monkeytype apply main

And whoosh! We have List[int] type automatically derived for numbers argument and bool for a return type

Now the transpiled result will successfully compile by rustc because enough type information is known

Note how List[int] has become Vec<i32>

This MonkeyType utility is quite powerful, so I ran it through our consensus prototype project and implemented translation for most frequently encountered types:

• List[T]will become Vec<T>
• Dict[T,S] will be HashMap<T,S>
• Optional[T] translates to Option<T>
• Tuple[T,S] is going to be (T,S)

But there are also types like Any and Union which cannot be easily translated to Rust type system due to their dynamic nature. And they suggest architectural problems in the code which have to be addressed.

In addition I’ve also implemented support for variable type annotations in translator, so you can write x:int = 1 in python and it will translate to let x: i32 = 1; Though it must be noted MonkeyType does not generate such annotations at all.

Porting guide

So, If you are like me and have found yourself in need to port a Python project to Rust I highly recommend you using MonkeyType before converting any code. First thing to do is to make sure you have Python 3.7, because typing support has much improved since its introduction in version 3.5. Then install MonkeyType using: pip install MonkeyType

Now launch your main file like this monkeytype run main.py. Since this utility infers types as it runs, you want to make sure you execute as much code as possible. If you have user interaction run all possible branching paths. If you have high test coverage you could run tests using monkeytype run `which pytest`. As you finished collecting symbols you can put all modules into file with monkeytype list-modules > module_list.txt. Edit that file and remove all external modules. Now use while read p; do monkeytype apply $p; done <module_list.txt to apply type annotations to all project modules from that file. In the process you may encounter python errors. Here are my errors and suggested solutions:

• Cyclic dependencies, which I resolved by creating forward declaration or simply removing both import and types referenced by them
• Methods returning Self. Python doesn’t support this in 3.7, so you may want to add from __future__ import annotations to the very beginning of file where function returns its class instance.

Because process of adding types stops on errors, you should fix them, remove corresponding file from module_list.txt and then reapply everything again. Repeat this steps until you got no errors. After that you can invoke pyrs on your folder to convert whole project to Rust. Viola! Both your Python and Rust methods now have types. I also worked a bit on struct definition generation lately, so your class fields will guess their types based on that as well.

Conclusion

Big props to guys at Instagram for creating and open sourcing such an amazing tool. It adds some performance burden and requires code to be evaluated to infer types, but works well and makes idea of converting Python to Rust a bit more practical. It may save some time when migrating languages, but don’t expect your code to compile out-of-the-box, because Rust’s borrow checker exists for a reason.

Check out the monkeytype in examples folder of pyrs repository to try it for yourself.