GSoC `23- Weekly Reports

Processing Android Weekly Reports

Community Bonding Period: Before the commencement of the coding period(May 28th, ’23) we had our community bonding period where I familiarised myself with the code base and made a streamlined communication channel with my mentor, Aditya Rana.

Problems faced while setting up Processing Android:

The major issue was that some Gradle dependencies were not supported by the JAVA-17 as discussed in issue #732. Though, this issue was majorly solved after Andres included the jar files in the new commits, some dependencies were imported externally.
Overall, I was able to successfully setup the PDE with Android.

Weekly Updates:

Week 1 (28th May-4th June):

As per my proposal, the task was to identify the classes which needed to be migrated and understand their functionality.

After reading through the codebase and the latest drafts of Processing for Android (provided by Andres Colubri), I came to understand that I need to work on the following Java classes: VRActivity.java, VRCamera.java, VRGraphics.java, VRSurface.java, VRGraphicsMono.java, VRGraphicsStereo.java

After thoroughly inspecting the files, I came to understand that:

1. VRActivity acts as the Main Activity while using VR-Processing and uses other classes(such as VRGraphics) for it’s functioning.
2. VRGraphics was handling important methods ( eyeTransform, headTransformand setVRCamera) and updating them as the user moves around.
3. VRCamera, as it sounds is used to define the camera view.
4. The VRGraphicsMono and VRGraphicsStereo were using the VRGraphics class to render the scene in the chosen mode.
   Both classes are calling the same createSurface method with boolean reset being the only difference. When reset is true, we get a Stereo render and on false it gives a Monoscopic View.

After identifying all these classes, the next task was to include the CardBoard SDK dependencies using gradle and migrate these classes.

Week 2 (5th June- 11th June):

This week, I needed to import all the necessary Cardboard SDK dependencies in the project and start migrating the classes.

I thought this to be a simple task but I faced multiple issues here:

1. I couldn’t find the relevant gradle dependencies to include in the project. So, I decided to download and add them manually from the sample Cardboard project provided by Google.
2. The Sample Cardboard project didn’t seem to have any gradle dependencies either, so I decided to look for any Jar files but no luck there either.
3. Instead they had used C++ header files for all the project.

After a thorough discussion with my mentor, I came to understand that my understanding of the project was a bit wrong. The Cardboard SDK was built on Android Native(NDK). After a brief discussion with Aditya, we came to the conclusion, that our next step should be:

1. To get familiar with NDK development.
2. Add the Cardboard SDK C/C++ files to the project and develop a JNI Bridge and make these files compatible with Processing.

I’m currently getting familiar with Android NDK and the next task is to develop the JNI Bridge and import the C/C++ files to the project.

Week 3 (12th June -18th June)

The main task was to develop a JNI Bridge and import the C/C++ files in the project.

After thorough research, analysis of the Cardboard SDK, and insightful discussions with my mentor, I realised the task is more complex and intricate than we anticipated. The Cardboard SDK relies heavily on NDK, so we are figuring out how to integrate it into processing. I am still going through Processing-VR ,what all capabilities it provides, and trying to frame a generalised approach on how to integrate Cardboard-NDK into Processing.

Cardboard NDK is relatively new, hence there isn’t much sample project and supporting documentation to aid our learning. Thus, we have to go by the traditional way of hit and trial, to understand it’s functionality.

After reading the code base I realised that Processing-Android uses templates to build the apps and then push them into the device. Cardboard NDK relies on dynamic C++ header files, necessitating the creation of a new template for seamless integration with Processing.

To ensure a successful integration process, my mentor and I have decided to take a step-by-step approach. Firstly, I will focus on gaining hands-on experience by developing simple VR apps using the Cardboard SDK with Android NDK. This practical experience will provide a solid foundation for integrating the Cardboard SDK into Processing.

While, we are devising a way to tackle the VR integration issue and I have decided to start with adding the ARCore Image Markers functionality.

For the upcoming week, my main priority will be to:

1. Develop a sample ARCore app with Augmented image functionality to get comfortable with ARCore Development.
2. Add ARCore dependency to Processing.

Throughout this process, I am dedicated to continuous learning, particularly in the area of JNI. By improving my knowledge and skills in JNI, I will be well-equipped to overcome any challenges that arise and deliver exceptional results.

Week 4 (19th June- 25th June):

This week I reorganised my project to start with AR-Markers. I started with learning more about the AR-Core and how AR Image Markers worked. After that I made a comprehensive guide on how we can add it to Processing-AR and wrote an article on it.(here)

Week 5 (26th June- 2nd July):

An new Discussion(here) was opened by my mentor to let everyone in on the progress and brainstorm on how to add this functionality to Processing.

From 2nd July — 13th September (Week 6 -16th), the discussions were continued here. I added a progress update blog for the month of August(here). My GSOC timeline was extended till November 7th and the PR for AR-Markers was successfully merged(here).

Now, the task at hand is to pick up migrating the VR Libraries which was dropped earlier. For the next week, my main focus will be on changing the VR templates to make PAndroid compatible with C/C++ classes and then create a JNI bridge.

Week 17 ( 17th Sept- 24th Sept):

We brainstormed on how to migrate the VR library to Cardboard SDK. By now, I have learned how the JNI works and how it can be used in Android. The sources I used to educate myself were Android NDK Documentation and the lectures from Java for Engineers.
After a thorough discussion with @ranaaditya , we created a rough idea on how we can achieve this:

• Add C++ support to Android Mode by editing the build gradle and VR templates.
• Import the Cardboard SDK, NDK lib and .so files(eg. arm64 , armeabi, x86, x86_64)
• Create a JNI bridge, CMake list and other c++ dependencies.
• Change the existing VR templates to add external native builds
• Map the new methods( from Cardboard-SDK) to their predecessors( from GVR).
• Migrate the methods to the new SDK.
• Test the new library with sample apps.

With this, I’ve started on the second part of my GSoC delieverables.

Week 18 ( 25th Sept- 1st October):

I created a Github Discussion to keep everyone updated about the progress. Subsequently, I changed the VR templates for Processing VR to add JNI support to the VR Sketches. This brought to light another issue. PDE was not compatible with Native development. Android Studio took care of this by adding NDK and C++ support to the project, but to do the same in processing we need to include NDK in separate directory which should be installed when the Sketch app is created. This issue needs to be addressed first, before we move on to add the CMakeLists.txt and Cardboard SDK.

Week 19 ( 2nd Oct- 8th October):

I found that the issue with NDK dependency can be resolved by editing the Gradle files and modules. I edited the VR-Gradle templates to install NDK when the Sketch App is build. This created issues with gradle-tooling as the versions mismatched.

The issue with tooling is not complicated but will require much hit-and-try. The PDE build works on older versions of tooling while Cardboard is build on newer ones. I’ve to find that sweet spot where they are both compatible. apart from this, I need to switch to NDK 21 as that’s the one seems to be the most compatible with PDE + Cardboard build.

Week 20 (9th Oct- 15th October):

Updates on Current progress:

• I’ve successfully changed the VR templates to change the sketch directory to support JNI. The updated template includes Cardboard SDK copied to app folder, Jni library copied to a newly created JNI folder inside Main folder, CMakeLists.txt being copied to app folder.
• Since we need the project to support NDK, we had to make changes for the app to install it separately inside user’s SDK path, in case none is found. I made the changes so that P-Android-VR checks whether any NDK is already present or not. In case not, It installs NDK version 21, as that’s the Version most compatible with the Current PDE+Cardboard build.
• I’ve also added the misc. files that facilitate the support of cardboard such as arm64 etc. Though some were already present, I made sure to check for missing ones and then copy them.

Currently, I’m trying to add the JNI bridge but facing some issues about how I should go about it. Usually, the JNI targets the MainActivity, and these activities use the native functions. But processing works very differently. The Java files such as VRActivity.java, VRGraphics.java etc are solely responsible for constructing the VR app in sketch. I’ve updated the progress on the discussion as well and hope to complete the project soon, as most of the work has been finished.

Week 21 (16th Oct- 22nd October):

It seems that I’ve hit a roadblock. The JNI bridge works during build time when the Java files are compiled. This poses an issue as the Java files in processing android are not compiled. Instead, a jar is created using them which is sent to the Sketch App. Without compiling these files, a JNI bridge can’t be created. I’ve informed my mentor about this issue and hope to resolve it soon.

Week 22 (23rd Oct- 29th October):

I informed the community about the issues I’ve been facing via the discussion and to resolve them, we decided to do a peer programming session with my mentors @codeanticode and @ranaaditya. Also, other members of the community(@SableRaf) came forward to help me out. After a long discussion, we came to the conclusion to build wrappers for the Cardboard SDK.
The outcome for VR Implementation was changed slightly after discussing with Andres. We discussed everything of the Project with Andres in detail, then Aditya and Andres came to a conclusion that we will create a wrapper library around Cardboard SDK so anyone including Processing and outside the org can use it in very less complexity.

Why this — with current project structure its not possible to integrate Cardboard SDK as of now, it needs rewrite many things which is out of project scope.

Why we have come to this conclusion:

• We didn’t know the complexity of the Cardboard SDK earlier, its was much more than anticipated
• Cardboard has completely different set of renderers than what we are using internally in Processing, writing all renderers was out of project scope
• As per the current structure of processing-android code, integrating JNI bridge seems to be impossible at this point of time. This is because Cardboard SDk is focused on integration with the apps not with the multi module android libraries (we are here).

Why wrappers for Cardboard SDK ?

Cardboard is designed to integrate with android apps as it needs 2 levels of Gradle build files (Project + app level) which generally normal android apps have. Current processing-android project structure is not that of an app, its multi module library with complex Gradle builds with just one top level of Gradle build file for whole project. If we need to integrate JNI into it, we need 2 levels — project and app level gradle which we don’t have. And without JNI you cannot communicate with native C code in android and Cardboard is just full native C code.

We generally need a wrapper with JNI bridges to use C/C++ code in JAVA/KOTLIN. With current processing-android structure we can use wrappers very easily (but still need to reimplement renderers according to cardboard functionality) if we have prebuilt JNIs in wrappers.

You can think of this wrapper library as a layer over the Cardboard SDK where you don’t have to deal with native C code directly, the wrapper will do everything for you and you can use all native code in java itself.

This wrapper library will serve 2 purposes:

1. Processing will be able to use it for cardboard integration in future not just in android-mode but in any java/kotlin project.

2. Other devs outside Org can also use it.

After Andres’ suggestion we have come to this decision to make a wrapper library for cardboard which processing-android / processing org can use in future.

Week 23 (30th Oct- 5th November):

The main aim for this week was to create a Wrapper class for the existing Cardboard SDK. The aim was to create a wrapper class that could be used in Activities and Java classes to create a simple VR app without the hassle of creating a separate JNI Bridge for the native library. The CardboardWrapper helps us achieve this. To understand how it works, I recommend going the documentation and a sample app was created to better understand how this can be used to create simple VR apps.

This work has been integrated in Processing under Cardboard Wrapper SDK. https://github.com/processing/cardboard-wrapper-sdk

With this final update, my Google Summer of Code came to an end. I learned many things over the past 6 months and grew along with the project. It was my first time working on such a large code base. Contributing to such an extensive project did seem overwhelming at first, but as I learned and gained more experience, I felt comfortable doing it.

This was a real journey for me. And I would like to offer my sincere gratitude to Aditya Rana for being my guide, mentor and friend over the span of this program. I can’t fathom completing the project without his immeasurable help and guidance. I just love the processing community for being so welcoming to me and would love to be an integral part of the community in the coming years.

Thanks for Everything so far.

Signing out,

Gaurav Puniya.