GSoC 2021 Final Report — JBOSS: Create and Enhance APIs related to Education Platform
Introduction
The project aims to Create and Enhance APIs and to produce a well-built backend mechanism related to an Open Source educational platform that strives to enable 100% education and will help in upskilling developing countries and rural areas.
The project focused on these areas of interest:-
1). Creating APIs for User, Course, Course Reviews, Doubts, Assignments, and Mentors.
2). Making sure that the above endpoints are well integrated with the client-side.
3). Add Authentication and implement a role-based workflow so that the stakeholders have appropriate control over the web application.
4). Documenting the required body and the expected response body of all the endpoints.
5). Writing unit tests for the APIs created.
Project Details
Organisation:- JBOSS
Source Code:- https://github.com/codeforcauseorg/edu-serverProject Url:- Here
Student:- Devesh Kumar
Work Done:-
All the merged PRs: Click here
Tech Stack used:-
1). Nestjs ( NestJS is a framework for building efficient, scalable Node.js web applications. It uses modern JavaScript, is built with TypeScript )
2). MongoDB
3). Docker
4). Firebase (for authentication)
I will divide the work done into 3 parts:-
1). Concepts and APIs
2). Features
3). Documentation and Testing
Concepts and APIs:-
1). Before the creation of APIs, the schemas had to be created for each collection. The Database used was MongoDB. A NoSql database was used even after instances of relational data because NoSQL is better for unstructured data like documents or JSON and it provided the opportunity for horizontal scaling.
2). There were a lot of fields in each mongo collection that required relationships.
Course Schema had one to many relationships with Doubts, Assignments, Reviews and Schedule Schemas.
It had a many-to-many relationship with Mentor Schema.
Each Schedule had a one-to-many relationship with lecture Schema.
Each Doubt Schema had a one-to-many relationship with Doubt Answer Schema. (Course-based Doubt Forum was also implemented and the same schema was used to implement an open Doubt Forum to view all doubts and doubt answers in one place).
There was an Enrolled Course Schema as well which had a relationship with user schema to store the userId for each enrolled course with the courseId and some extra features to make it personalized for each user. This was done using Mongoose virtuals.
3). Initial Architecture (click here)
This is how the initial application architecture looked like.
After a lot of changes and brainstorming and the addition of new features and schemas, the new architecture was formulated and worked on.
4). Providers are a fundamental concept in Nest. The main idea of a provider is that it can be injected as dependency; this means objects can create various relationships with each other, and the function of “wiring up” instances of objects can largely be delegated to the Nest runtime system.
Services serve as the business logic of the web application and all the database queries related to each function in the controller was handled in the service.
The services for each controller were injected into them as consumed by the functions.
A controller’s purpose is to receive specific requests for the application. The routing mechanism controls which controller receives which requests. Frequently, each controller has more than one route, and different routes can perform different actions.
Using decorators to describe each type of CRUD the APIS and with the conglomeration of all the above steps, the endpoints were created.
5). For all request body-based requests, we need to determine the DTO (Data Transfer Object) schema. A DTO is an object that defines how the data will be sent over the network. Create and Update endpoints both had their DTOs defined.
I used the class validator feature of NestJs to set further conditions on how the DTO should be like.
Using decorators like @IsArray) @IsString() and suitable Enums wherever required helped a lot in debugging and data type validation.
6). Apart from the authentication middleware (discussed in the Features section below) cors handling middleware was also set to handle cross-origin requests.
7). I had made a rough doc (click here) to note down my daily progress and all the Pull requests made accordingly as a record of my work.
Features:-
1). Firebase authentication was created using the firebase admin and suitable middleware were created for the same.
2). Role-based authentication was implemented using the @Useguards method of NestJs. There are two roles, student(default) and the Admin role.
The middleware is configured in such a way that when the firebase token is decoded, the user information is retrieved from it and it checks whether the user exists or not.
If the user does not exist then a new user is created, and a default role of STUDENT is assigned.
Only the database owners can set admin roles to themselves. This ensures complete Role-based security for those endpoints which have
@Roles(Role.ADMIN) on their controller.
3). Wishlist and Cart List-> The user of the product can wishlist courses or add them to or remove them from his/her cart. The server-side implementation of this has been done in a meticulous way and it has been ensured that no duplicacy of courses occurs while adding/deleting them.
4). Pagination-> Adding pagination with skipNum parameter of MongoDB in course , cards and doubts.
As we see below, the query parameter skipNum will tell us how many documents to skip and the limit for the number of documents returned via each pipeline is set to 10 documents
5). Tags-> The website has a lot of endpoints that have a tag attached to it. This helps ease searching relevant information and makes the UI/UX more appealing. ENUMS have been used wherever required and class validators have been set up accordingly.
6). Announcements-> The users with the admin role can post announcements on the web app that will be visible to all.
7). Doubts-> There is a doubt forum implementation as well with doubts that can be answered. The doubts will have the course id parameter as well so the doubts can be populated in the respective course section as well.
8). Gamification Board-> A Gamification Board was implemented to show the lists of users sorted according to their score on the web app.
Documentation and Testing:-
1). Testing was done using the jest library.
Tests were written to test CRUD endpoints of user and courses. Each of the controllers and services were tested with ideal and non-ideal test cases and npm run test was also configured in the github actions to ensure that there are no changes that break the tests before the code is merged.
Testing of controllers had the CRUD operations tested along with the equality of the expected response. Mock functions were used and proper calling of service functions was also tested using the .toHaveBeenCalled method.
In the services file, the CRUD operation business logic was tested too.
2). A module is a class annotated with a @Module() decorator. The @Module() decorator provides metadata that Nest makes use of to organize the application structure. For almost each set of collection of APis, there was a dedicated module. In the folder structure, each of the modules had a dedicated folder for documentation of each of the endpoints
Swagger was configured for the documentation and with the help decorators like @ApiOperations, @APiTags(To attach a controller to a specific tag), @ApiParam(to determine whether a parameter was been used by the function of the controller) and the NestJS CLI features, correct documentation was configured along with the request body, response body and example values for the same.
A set of examples for each one of them can be seen below:-
A total of 41+ PRs merged, a lot of code written, a lot of bugs fixed and a lot of brainstorming along with a great deal of learning is what describes the work done in brief.
Challenges:-
The most challenging part of GSOC was definitely learning NestJS which is a progressive Node.js framework for building efficient, reliable, and scalable server-side applications.
It required reading docs thoroughly and it did not have a lot of YouTube video tutorials as compared to any other framework like React. It required code to be written in a very structured format, which helped in debugging but required great precision. Learning it and working on such a huge project while making scalable endpoints for it was really challenging. But, the toil has really been beneficial and I hope that I make the best out of them.
There were several build errors caused by the use of absolute imports and had to be fixed by the use of relative imports.
Scoping was used to inject request object into the services and the tests started failing because of this. Therefore, module.get had to be replaced with module.resolved because of this.
While using an array of enums for tags, it had to be validated using the @IsEnum() class-validator. Since it was an array of tags, and the class validator validated only a single field, the desired validation could not be done. For this problem to be solved { each: true } had to be passed in the decorator.
Several tiny errors kept popping up as the progress gained momentum and sometimes the code had to be completely refactored/rewritten in a different approach altogether to suit the workflow.
Experience:-
I have learned a lot during the GSOC period and the code speaks for itself. With several PRs merged and several approaches discussed, I think that I have become a better developer and a team worker.
I communicated with my mentors and met amazing people along the way and learned amazing stuff from them.
NesjtJs and MongoDBhad excellent documentation and all the features that I implemented taught me a new concept. From role guards to class validators to pre-hooks in mongoose, everything was just enlightening.
This GSoC with JBOSS was an enjoyable experience. The mentors I have been assigned with were very friendly, helpful, and supportive. The community as a whole is fabulous and enthusiastic.
The coding experience was enlightening. I have learned a lot of best practices in coding and Open Source. The work was always enjoyable and insightful. Mentors were very generous in explaining why I should be doing things in a particular way. The Overall experience was just awesome. The mentors were very knowledgeable. I don’t have words to portray how fantastic they were.
Acknowledgment:-
I am very thankful to my mentor Abhishek Kumar. Without him, the work never would have been this joyful and rewarding. He was always available to answer my questions on time and also actively ensured that my Pull Requests were reviewed and that the coding standards were up to the mark. He also provided appropriate links to resources and made sure that the deadlines were met through regular meets. I love the JBOSS Community for its welcoming and enthusiastic nature.
I thank Adarsh, another GSOC student at JBOSS who was working on the client-side of the project. Without him, I would never have known the shortcomings of my work. You can check out how the integration came out on the client side on his blog here.
Finally, thanks to the GSoC program, without which, I wouldn’t be a part of this awesome project and gain this memorable experience. I am thankful to them for creating this kind of opportunity for students
Future Work:-
1). A new feature of conducting contests can be added to the web application.
2). Graphql can be used to further improve the workflow of the web application.
3). A new feature of chatting between the users of the web application can be implemented as an enhancement.