Google Summer of Code 2023 with CloudCV

Name: Ayush Rathore

Project Title: Evaluation Infrastructure Optimization

Mentors: Gunjan Chhablani, Rahul Singh, Gautam Jajoo

Tech-Stack: DevOps, Backend

Project Repo Link: EvalAI

Presentation Link: Presentation

Introduction

As the summer comes to a close, I am thrilled to present the culmination of my efforts during the Google Summer of Code (GSoC) with EvalAI. This journey has been an exceptional experience, allowing me to immerse myself in the realm of open-source development and make meaningful contributions to the EvalAI platform. Throughout the GSoC period, I actively engaged in enhancing various aspects of EvalAI, ranging from backend improvements through infrastructure optimizations to implementing front-end features. In this blog post, I’ll guide you through the contributions I’ve made, shedding light on project details and the pull requests I’ve submitted.

Project Details

The primary objective of this project was to enhance EvalAI’s functionalities by automating the deployment of large worker instances in AWS, incorporating relevant features for efficient challenge management, and building a robust and efficient test suite. The main focuses of the project included:

1. Automating large worker deployment processes on EvalAI using AWS EC2 instances to reduce infrastructure costs.
2. Introducing features that make challenge management more seamless, decreasing the reliance of challenge hosts on EvalAI administrators.
3. Enhancing the reliability and stability of EvalAI by implementing comprehensive tests for both frontend and backend components.

Snapshot of Merged Pull Requests

Here’s a selection of the pull requests (PRs) I’ve contributed to and that have been successfully merged:

• [Backend] Added Tests for Code-Upload Worker. PR Link
• [Feature] Introduced Admin Actions for EC2 Start and Stop Workers. PR Link
• [Feature] Added Support for Creating EC2 Instances via API. PR Link
• [BugFix] Fixed EC2 Auto Scaling During Instance Initialization. PR Link
• [Backend] Implemented API to Start and Stop EC2 Instances. PR Link
• [BugFix] Resolved Error in EC2 Deployment Script. PR Link
• [Monitoring] Added EC2 Auto Scaling Script. PR Link
• Fixed: #4096 [Scripts] Added Crontab and Comments to Deploy EC2 Worker Script. PR Link
• [Backend] Added Custom Worker Image for Challenges. PR Link
• [Feature] Introduced Automatic EC2 Creation for Challenges. PR Link
• [Monitoring] Integrated Django Error Logging to Console. PR Link
• Introduced Submission Count by Status API for All Challenges. PR Link
• [BugFix] Resolved Challenge Validation for Missing/Extra YAML Files. PR Link
• [Monitoring] Utilized Latest Images for Alertmanager. PR Link
• [Frontend] Added Logging for Evaluation Module Failures. PR Link
• [Backend] Added Tests for Code-Upload Worker. PR Link
• [Backend] Implemented Tests for Validate and Create-Update Challenge. PR Link
• [Frontend] Added Challenge Approval Request Button. PR Link
• [BugFix] Resolved Manage and All Submissions Page Visibility. PR Link
• Fixed #3923: Display User Timezone on Submission Worker Logs. PR Link
• Fixed #3884: Enhanced Validate Challenge Config for Handling Corner Cases. PR Link
• [Monitoring] Added Auto Cancel Script. PR Link
• Supported Multiple Challenge Versions with Different Branches. PR Link

Conclusion

Participating in GSoC with EvalAI has been an enriching journey, filled with both challenges and achievements. The opportunity to collaborate with a dynamic community of developers and contribute to a versatile platform like EvalAI has been nothing short of remarkable. My experience in working on this GSoC’23 project with CloudCV has been truly awesome, standing out as one of the most valuable experiences I’ve encountered. Not only did I have the privilege to contribute to a significant real-world project, but I also underwent significant learning and growth throughout the program.

I extend my heartfelt gratitude to my mentors, Gunjan Chhablani, Rahul Singh, and Gautam Jajoo, for their unwavering guidance, support, and continuous feedback throughout the program. Thank you immensely for patiently addressing all my queries and aiding me in overcoming obstacles.

Lastly, I express my appreciation to Google for organizing GSoC and offering this fantastic opportunity to collaborate within a thriving open-source community on such a substantial real-world project.