Google Summer of Code 2021: Implementing a Java Loader for MetaCall
Java is a high-level programming language currently being used in various parts of the tech industry with more than 3 billion devices using it. Java has a wide variety of applications like mobile, desktop, web, and many more.
MetaCall is a polyglot runtime environment that lets developers call functions and methods between different programming languages. For example, we can call a Python function from a Node.js file. This is done by converting the function to a C/C++ method using an embedding API.
MetaCall supported languages like C, C++, Python, Node.js, etc. but it lacked Java as an option. So, as a part of Google Summer of Code 2021, I set out to change this by implementing the Java loader for MetaCall, which would allow MetaCall to call functions from Java.
Work Done
My journey with MetaCall started by working on a Proof Of Concept (POC) using the Java Native Interface (JNI), the reflection API, and the compilation API in Java to build a small working example that would be required to make the Java loader work in MetaCall.
After implementing the PoC successfully, the next task was to start working on the MetaCall core repo and start mapping the functions from the PoC to the loader.
In the first 4 weeks of GSoC, I worked on implementing the basic functions to initialize the loader, add the default classPaths and execution paths, the function to discover the data present in the Java file which consisted of data types, methods, and constructors. In the next 3 weeks, I worked on implementing the getters, setters, and invoke methods for static members/attributes, which are used to get and set values of static attributes and invoke the static methods present. In the last 3 weeks of GSoC, I worked on implementing the same functionality but for non-static attributes and objects.
The major problem and issues I faced during these 10 weeks was the fact that Java is not easy to embed as it is a strongly typed language compared to JavaScript or Python. All these things made the project even more complex than anticipated. For instance, supporting all the primitive data types and arrays of different data types, each had to be converted individually. Due to this, it became even more difficult to make the project dynamic. Nevertheless, all the problems were resolved and I was able to implement the whole loader.
In the end, I was also working on a whole test suite using Google Test. This was testing all the common functionality with most of the data types and edge cases.
Showcase: Calling Java from Python
Consider a simple Java file as follows:
This is a simple Java file that contains a function Add which takes two integers and returns their sum. Even though this is a simple example, MetaCall supports complex functions, data types, and constructors too.
By using the Python Port for MetaCall, we can now import Java classes directly from Python:
This script can then be called using the MetaCall command-line tool:
Design and implementation
We planned to use a mix of Java and C++ instead of using only the JNI for the proper working of the loader. This means, instead of doing more complex tasks in C++ like compiling a file, extracting the data, or converting using JNI, we used a bootstrap.java file to accomplish these tasks. Bootstrapping is the usage of the runtime/loader language and code execution for the tasks required for MetaCall’s introspection.
This approach made it easy for us to work with the JVM and the JRE and accomplish these tasks:
- Compiling a .java file in Java classes.
- Loading all the compiled classes to the JVM.
- Getting a list of classes that are loaded into JVM.
- Extracting the attributes, methods, and constructor details present in a Java class.
- Reading details such as name, signature, parameters, and return types.
We directly used JNI in C++ to accomplish all the tasks like calling the methods, getting or setting attribute values.
The loader implements the following list of methods:
- Initialization: It is the first function that MetaCall calls in the loader to initialize a runtime environment. In our case, to start the JVM and the JRE.
- Load From File: Load from file is the function that takes the absolute path of the Java files and compiles them to Java classes.
- Load From Package: Load from package is the function that is used to load files that are already compiled, like classes or JARs.
- Load From Memory: Load from Memory function takes a binary string representing a Java class and compiles and loads it to JVM.
- Discover: This function performs the most important task to “inspect” the data present in the Java class. It performs all the data extraction related to attributes, methods, and constructors and registers them to the MetaCall Core so that MetaCall can identify the type and call it in the Java loader. Discover handles the attributes, with their names, data types and Java signature, the constructors present in a class, including name, list of parameters and their types, and finally the methods, registering in all of them the visibility (public, private or protected).
- Destroy: it cleans all the resources and stops the JVM.
- Getters and Setters: These functions are used to get or set the value of an attribute, no matter if it is static or non-static. If the attribute is static, then the interface referring to class must be used, otherwise the object must be used instead.
- Invoke: These functions are used to invoke methods present in a class by passing the required parameters, if the method is static, then the class interface is used, otherwise the object is used.
For more detailed information on the development process, refer to the source code in the MetaCall Core repository and the full list of commits and PRs related to the project.
Conclusion
The addition of a Java loader to MetaCall will now allow it to support a large and expanding range of programming languages through the implementation of a single loader. While Java and consequently the loader still have a long way to go, this project provides a good base for future development.
Ketan Gupta.
Google Summer of Code 2021 Student from MetaCall.
