[Sparta Coding Club India] My Learning Journey: From Python to Full-Stack Development

1. New Learnings

Web Scraping Basics and Saving Data to MongoDB

Recently, I dived into web scraping, a technique used to extract data from websites. This skill is crucial for gathering large amounts of data from the web, which can then be analyzed or stored for future use. I used Python’s requests and seleniumlibrary to fetch web pages and BeautifulSoup to parse the HTML content. The data collected was then stored in a MongoDB Atlas database, a NoSQL database that allows for flexible and scalable storage of data.

Explanation:

Web scraping involves sending a GET request to a website, fetching the HTML content, and then parsing that content to extract the desired information. MongoDB Atlas, a cloud-based database service, was used to store the extracted data, providing an easy way to manage and query the data later.

Example Code Snippet:

from selenium import webdriver
from selenium.webdriver.common.by import By
from bs4 import BeautifulSoup
import time
from pymongo import MongoClient
client = MongoClient('mongodb+srv://bijayeeni:sparta@cluster0.sik7a.mongodb.net/?retryWrites=true&w=majority&appName=Cluster0')
db = client.dbsparta

# Set up Selenium WebDriver
driver = webdriver.Chrome()  # Uses the default Chrome driver

# Open the IMDb Top 250 page
driver.get('https://www.imdb.com/chart/top/?ref_=nv_mv_250')

# Wait for the page to load fully
time.sleep(5)  # Adjust the time as needed to ensure the content is fully loaded


soup = BeautifulSoup(driver.page_source, 'html.parser')

# Close the browser
driver.quit()

# Using select
movies = soup.select('ul.ipc-metadata-list > li.ipc-metadata-list-summary-item')

for movie in movies:
    movie_title = movie.select_one('h3.ipc-title__text').text
    year = movie.select_one('span.sc-b189961a-8').text
    rating = movie.select_one('span.ipc-rating-star--rating').text
        
    doc = {
        'movie': movie_title,
        'year': year,
        'rating': rating
    }
    db.movies.insert_one(doc)

Creating a Server with Flask

Another significant milestone was learning how to create a server using Flask, a micro web framework in Python. Flask allows you to build web applications quickly by providing the essentials, like routing and request handling, without the complexity of larger frameworks.

Explanation:

Using Flask, I learned how to create routes that handle GET and POST requests, linking HTML forms to backend Python code. This was particularly useful in my mini-projects, where I created simple web applications that interacted with a MongoDB database.

Example Code Snippet:

from flask import Flask, render_template, request, jsonify
from pymongo import MongoClient

client = MongoClient('mongodb+srv://bijayeeni:sparta@cluster0.sik7a.mongodb.net/?retryWrites=true&w=majority&appName=Cluster0')

db = client.dbsparta
app = Flask(__name__)

@app.route('/')
def home():
   return render_template('index.html')

@app.route("/bucket", methods=["POST"])
def bucket_post():
    # sample_receive = request.form['sample_give']
    bucket_receive = request.form['bucket_give']

    count = db.bucket.count_documents({})
    num = count + 1

    doc = {
        'num': num,
        'bucket': bucket_receive,
        'done': 0,
    }
    db.bucket.insert_one(doc)
    return jsonify({'msg': 'data saved!'})

@app.route("/bucket/done", methods=["POST"])
def bucket_done():
    num_receive = request.form['num_give']
    db.bucket.update_one(
        {'num': int(num_receive)},
        {'$set': {'done': 1}}
    )
    return jsonify({'msg': 'update done!'})

@app.route("/delete", methods=["POST"])
def delete_bucket():
    num_receive = request.form['num_give']
    db.bucket.delete_one({'num': int(num_receive)})
    return jsonify({'msg': 'delete done!'})

@app.route("/bucket", methods=["GET"])
def bucket_get():
    buckets_list = list(db.bucket.find({}, {'_id': False}))
    return jsonify({'buckets': buckets_list})

if __name__ == '__main__':
   app.run('0.0.0.0', port=5000, debug=True)

2. Challenges Faced

Interacting with MongoDB Using PyMongo

One of the challenges I faced was understanding how to interact with MongoDB using PyMongo, especially when it came to complex queries and updates. MongoDB’s flexible schema allows for a lot of freedom, but it also means you need to be careful with how you structure and query your data.

Solution:

I overcame this by practicing with different types of queries, such as filtering, updating, and deleting documents. I also learned to use MongoDB’s aggregation framework for more advanced data processing.

Example Code Snippet:

from pymongo import MongoClient
client = MongoClient('mongodb+srv://bijayeeni:sparta@cluster0.sik7a.mongodb.net/?retryWrites=true&w=majority&appName=Cluster0')
db = client.dbsparta

db.books.insert_one({
    'title': 'Harry Potter',
    'author': 'J.K. Rowling',
    'rating': 90
})

db.books.insert_one({
    'title': 'The Fisherman and the Fish',
    'author': 'Joseph Choi',
    'rating': 10
})

db.books.insert_one({
    'title': 'Fire in the Water',
    'author': 'Some Dude',
    'rating': 57
})

db.books.update_one(
    {'title': 'The Fisherman and the Fish'},
    {'$set': {'author': 'Jimmy Kim'}}
)

db.books.delete_one({'rating': 90})

3. Reflection on My Journey So Far

Improvement and Growth

Looking back, I’ve made significant progress in my web development journey. Starting with basic Python syntax, I gradually moved on to more complex topics like web scraping, database management, and server creation using Flask. Each new topic built on the previous one, reinforcing my understanding and giving me a strong foundation in full-stack development.

Differences from When I Started

When I first began, the idea of creating a web application that interacted with a database seemed daunting. However, through consistent practice and the hands-on projects provided by Sparta Coding Club India, I’ve gained confidence in my abilities. I’m now comfortable working with both frontend and backend technologies, and I understand how to integrate them to create functional web applications.

Future Learning Goals

Moving forward, I want to dive deeper into backend development, particularly focusing on creating more complex APIs and integrating them with frontend frameworks like React. I also aim to improve my understanding of deployment practices, so I can bring my applications from the development environment to production.

Conclusion

My journey with Sparta Coding Club India has been incredibly rewarding. I’ve learned so much in a short amount of time, and I can’t wait to continue growing as a developer. If you’re considering a career in web development or just want to learn new skills, I highly recommend giving it a try!