My internship experience at Sturfee
Hello, I am Jinmeng Rao. Aug 28 was my last day as an intern at Sturfee. In the past summer, I worked as a Geospatial Vision Intern here to design and implement some cool computer vision algorithms for solving real-world problems. I am very grateful that I had the opportunity to work with this super cool and talented team at Silicon Valley, and I want to share my wonderful experience and what I learned from here with you.
Why do I choose Sturfee
I am a fan of geospatial technologies such as high-accuracy positioning, city-scale Augmented Reality (AR), and spatial intelligence. Sturfee is the pioneer that successfully combines them together and brings them to real-world application scenarios in practice. Sturfee’s Visual Positioning Service (VPS) localizes users fast and accurately in the city and turns the entire city into a canvas for drawing enriched AR contents. This is very attractive to me and also the main reason why I joined Sturfee. I believe this is what an intelligent urban life in the future looks like: you take a smartphone, scan your surroundings, and the information you want is right in your sight — you care about your city, and your city cares about you too.
What did I do during my internship
During my 3-month internship, I mainly worked with my supervisor Dr. Jiangye Yuan, Senior AI Engineer at Sturfee, on designing computer vision algorithms for improving camera pose accuracy based on ground features. This work is very meaningful since higher accuracy camera poses will bring more seamless, accurate, and smoother AR experiences.
Our team developed a cross-view PnP aligner tool for estimating and refining camera pose. This tool shows both ground view images and aerial view images (satellite images) and allows users to select matching points from these two views. In the backend, it formulates the estimation as a PnP problem, and computes accurate camera poses. One of the main difficulties for users is to identify matching points between ground view and aerial view due to the significant difference in image appearance. My first task is to synthesize aerial views from ground view images so that we can recognize and match ground features more easily from both the synthesized view and the satellite view, and find matching points from them. To achieve this, we utilized the ray-tracing algorithm to project pixels from each frame of ground view images to the ground plane in the camera coordinate system, converted them to the world coordinate system, and filled the blank locations with interpolation. After processing, we created the synthesized aerial view that shows ground features from a top-down perspective, which are much more similar to the satellite view.
My next task is to integrate this synthesized view into our multi-frame PnP aligner tool and evaluate the camera pose estimation accuracy. We integrated all the ray tracing and interpolation code into the tool, and we designed and implemented the user interaction and view synchronization part. After the integration, we evaluated the camera pose estimation accuracy of the modified tool, and we found that the camera pose accuracy has significant improvement and it is much easier for users to select matching points.
Since a synthesized view looks similar to a satellite view, we started to think about, could we write an algorithm to automatically find matching points between these two views and infer the best camera pose accordingly? This is a cool idea and definitely worth a try. We did lots of experiments for selecting, testing, and fine-tuning keypoint detection algorithms. We also developed a new algorithm combining a grid-based keypoint matching algorithm and camera pose search. We ran our algorithm on benchmarking datasets and found when the ground features are presented in both views, our algorithm is able to find matching keypoints across both views and computes accurate camera poses.
Work is vacation at Sturfee
Yep, that is right. This sentence is the answer to a cryptogram puzzle posted by our CEO Anil at Sturfee’s Friday Challenge. Although due to the COVID-19 pandemic, like everyone else, I have to work from home for the whole summer, I can easily feel the energetic and fun work environment at Sturfee.
The company focuses on innovation and technology. Every week we have a company sync meeting, followed by a brown-bag presentation, where everyone makes a presentation to share their awesome work in their field. I think this is an excellent opportunity for interns like me to learn about state-of-the-art knowledge, solution, and experience in industry. The scope of the presentation covers system architecture design, algorithm design, SDK development, commercialization progress, etc. As a graduate student, I think it is not easy for me to access this comprehensive knowledge from school. Every presentation I have a new harvest, and I cherish the knowledge I learned from Sturfee.
Behold! Here comes my favorite activity: Friday Challenges! Every Friday, Anil will post an interesting puzzle, and the first one that solved this puzzle will get an Amazon gift card! The scope of puzzles covers pattern recognition, cryptography, image recognition, voice recognition, geometry, etc. The puzzle is not very difficult but usually, it takes some time to figure it out. I was lucky enough (and worked hard enough!) to solve three puzzles and get three Amazon gift cards so far!
Last but not least…
I would thank everyone at Sturfee for giving me an awesome and fruitful internship experience. I also want to especially thank my supervisor Jiangye, who is nice, patient, and reliable. Each time I encounter problems in my work, he can always give me valuable suggestions and put me on the right track. The most important thing I learned from him is how to think and solve problems from both a researcher’s and an engineer’s perspective.
My summer internship at Sturfee ends today, but my ideal for creating a better life with spatial intelligence never ends, and my experiences at Sturfee will definitely help me do better in the future.
About me
I am currently a second-year Ph.D. student in Geographic Information System in Geospatial Data Science Lab at the University of Wisconsin–Madison, supervised by Prof. Song Gao. My research interests include geoAI, computer vision, and location-based services.
Last summer I was a Geospatial Vision Intern at Sturfee Inc., supervised by Dr. Jiangye Yuan. I worked on designing computer vision algorithms for camera pose estimation to improve city-scale Augmented Reality experience.
Contact me:
Home Page: www.jinmengrao.com
Email: jinmeng.rao@wisc.edu
LinkedIn: https://www.linkedin.com/in/jinmeng-rao-5b4438121/
