Building an API
One of the fundamental goals for this project was for our team to develop not just a game using the Neato robots, but to further contribute to the community of robotics and game enthusiasts by developing a platform through which people can be creative. As a result, our team developed an API that handles the definition, mapping and projection of objects in 3D space to one’s camera view.
Our team started off by developing a class that would initially handle the definition of Sprites in the world coordinate system. In order to signal the creation of such a Sprite, one would have to specify the following:
- Image Path
Specify the directory and, therefore, the absolute path under which the png being used is.
The API expects the provision of 4 corners that define the bounding box around the image that is being used.
- Rotation Speed
Initially that parameter is assumed to be zero, however, the user can choose to alter that.
The API provides the user with several methods that might appear insightful for the specific use case.
This method expects as inputs the coordinates of a given Sprite in the world coordinate system (aSpriteCordinates). This is an array of arrays, where each subarray contains information about the X,Y, and Z coordinate of the given corner.
Furthermore, the method also expects a data structure input (once again an array) that describes the location of the camera in 3D space (cameraLocation), as well as the angle at which the robot is located with respect to the ceiling markers (cameraAngle).
The method returns a two-dimensional array with information on the Sprite’s location in the pixel frame.
Pixelate is a method in our API called within the transformWorldToPixels method. It performs several functions. First and foremost, it shifts the axes between the camera’s coordinate system and our pixelated camera’s view. It checks a certain component of the coordinates of the Sprite and raises a flag if the Sprite should not be visible in the camera’s view. Finally, the method makes use of camera characteristics, such as the K matrix that defines the degrees of freedom one is using information on the focal length of the lens, to calculate the resulting pixel coordinate.
If one is willing to render a Sprite that is rotating in the camera view around a given axis, i.e. a rotating coin, one can do so very easily be calling this method. The method expects an array of points that it is expected to rotate, as well as an angle theta, according to which the rotation should take place. It is important to note that in the current code version, we have hardcoded the midline about which the object rotates. One can easily change the position of that midline from the init method of the class.
One of the most important methods in our API is warpAndCombine. When we initially started testing the projection of simple points on the camera view, it was relatively easy to around for skewing and perspective, as there was none.
However, given the fact that our goal was to project images that behave like real-world (virtual) objects, we had to take into account the development of a function that would handle the warping and mapping effect. warpAndCombine expects a specific set of inputs to run; a sourceImage, destinationPoints, and a destinationImage.
The above method depends heavily on overlayImage. This method handles the creation of a mask around our image, as well as an inverse mask that given a specific ROI, will be able to take only the region of the “logo” from the image. Once this is done, the “logo” is taken back and positioned in the ROI that then returns the final image.
- renderSprite and addSpriteToView
These are the last two methods in the API that handle the actual projection and rendering of the image onto the camera view. They check at several stages if pixels exist to be rendered once they append the sprite coordinates to the view, addSpriteToView handles the final incrementation of the theta value, if any rotation exists.
This API will allow users to easily render images and Sprites on their camera feeds that adjust based on the location their robot is in. Scaling is also handled, as well as sorting of positioning of object projected in the camera view. Images that are defined to be further away will appear as smaller and, furthermore, will appear to be behind the ones that are defined closer to robot’s view.
The resulting projection of certain coins can be seen below, as well as a video that displays rotating coins in the camera view, streamed onto a web server can be found here.