Open in app

Sign in

Write

Sign in

Let’s Build A Web App! Week 27 of my 30 week web dev journey

Rona Chong
8 min readFeb 5, 2018

--

In which I set up my prod and stage databases and worked on some CI/CD.

Time is running short! With this week over, there’s only 2 weeks left to work on my project.

Previous posts in this series:

(Might have lost proper count here)

Now let’s get back to update on what I’ve been up to in the 27th week of this journey.

What did I set out to do?

I’ve been working on deploying my app for a few weeks now, so my goal for this week was (again) to finish with that and restart my work on the reader module of moshimoji.xyz. Up till now I’ve been slogging through a series of errors mostly caused by mis( or lack of )configuration for prod but what’s left remaining for finishing with my deployment is setting up the database part of the stack in prod. Last week I realized I needed something that would persist my user data, so I settled on using Amazon RDS instead of a db instance in my Docker swarm.

What did I get done?

As I mentioned above, I set up my prod and stage databases and worked on some CI/CD tooling for my workflow. Sadly, I didn’t get around to verifying that my database setups worked properly with my app. To make myself feel better and feel like I got more done, here’s a breakdown of what I did each day.

Monday: I read a little more Amazon RDS documentation and initiated my RDS instance. I realized that using one instance for both prod and stage would be problematic (I don’t want any CRUD operations that I do on my stage version of the site to impact the data that users get on the actual site), so I decided that for stage, I would use the same local database that I used for dev. From there I knew that my Django application would have to differentiate between prod and stage to run with the correct database endpoint. I figured that checking the env (via environmental variables) would be best, and verified that setting up environmental variables for my prod and stage swarms via Docker compose file would work.

Tuesday: I added the logic to my wsgi.py for my Django application to check the env for stage or prod and run with the appropriate config (which includes the database endpoint). I ran my app and verified with a print statement that the correct config was being used on the stage swarm. However, I also got an ‘ImproperlyConfigured’ message, which, combined with some introspection, let me know that my Django app was running with no database config — despite the config file it was supposedly using.

This puzzled me for an hour or two until I re-checked my print statement and realized I was using the same print in both files. My app was actually running with the wrong config (the prod config, which didn’t yet have any db config). Finally I went back to the logic I added and fixed my mistake. Checking the right places earlier would’ve saved me some grief, but a unit test for this part would’ve helped me identify my mistake even earlier, probably.

After that, I found that my app was having trouble connecting to the database on my localhost(504 timeout error) so I checked some firewalls and tried some telnets and curls to start debugging.

Wednesday: I banged my head against the 504 Gateway Error for quite a lot of time, trying to identify the nature of the issue with curls from lots of sources to first my private ip address, and then ‘my’ public ip address, when I realized that the private ip would be inaccessible to hosts outside of the network my localhost was on. I checked my ip tables and my security rules and added some rules to explicitly accept connections from my swarm, even though I saw nothing that should prevent it. I checked with some peers and together we looked at access logs to confirm that the requests weren’t even reaching my services/processes.

A table comparing my expectations regarding connection success with reality

Having checked everything I could think of, I finally escalated to asking one of the Holberton staff and within minutes, Jul, our local security/sysadmin, cleared the matter up to me: the public ip I thought was serving my laptop specifically belonged to the router in front of it (Holberton’s), and that router was configured to drop all incoming traffic to its specific ip. So unless I got my laptop its own public ip address, I’d be unable to use my database on my localhost for my remote swarm.

From there I realized I needed another solution for my staging database, and I decided that using a database within my Docker swarm should hopefully suffice if data persistence wasn’t crucial for the stage version of the site.

Thursday: I worked with the Jinja templating engine to set up a script that would generate both versions of the compose file I’d be using to spin up my Docker swarms (one version for the stage swarm, one version for the prod swarm). Spent a few hours reading about secret management with Docker swarm and deciding on how to deploy my apps with secrets (for security reasons and whatnot). After that I finalized my script to generate the compose files and verified that it did what I wanted.

The next steps were to push the secrets and generated compose files to my swarms, redeploy the swarms with the compose files and check that the swarms worked correctly with the secrets and database configs. Here I decided I was done with issuing commands build, push, pull and deploy in manual ssh sessions, and I should set up a process to sync my newest versions of secrets and compose files to my swarm whenever I needed to. Since I’d used ansible not too long ago and knew it didn’t take a lot to set up, I went with that.

After fiddling with my ansible configuration a bit to get it to run with the right python interpreter, I wrote a playbook to provision my swarm nodes with python (necessary to run most ansible modules on them) and emacs (finally!).

Friday: I proceeded to work with ansible to push the necessary files to my swarms and deploy. Writing a playbook to backup the most recent copies of my secrets/compose on the remote manager nodes, sync new versions over, and output comparison diffs went fine. From there I wanted to create a deploy playbook with a build option that would let me specify which components to rebuild/push after development (new compose? new docker image for backend? new docker image for frontend? new nginx image?) but ran into some roadblocks with parsing the options and using conditionals. In particular, my build roles would run no matter what ‘when’ conditionals I specified for them, which is a pretty frustrating and bizarre behavior I still haven’t gotten to the bottom of.

What didn’t I get done?

Here’s what I haven’t achieved relative to my big picture goals:

  • “Finish deploying my app on a stage and prod cloud, respectively”
  • “Restart work on the reader module of moshimoji.xyz”

The biggest thing that I didn’t get done this week (as in the most vital thing) was to verify that my database setups worked properly with my app. This was probably something I could have and should have prioritized over making nice ansible playbooks to do what I wanted to.

What went well?

  • Getting unblocked quickly once I asked a more knowledgable person about why my attempts to connect to my laptop from outside of the network might fail despite the things I checked.
  • Checking with an informal mentor of sorts about an approach I had in mind, and quickly discarding it for a better option after discussing with him.
  • Working more consistently this week by making sure not to fragment my blocks of time with external engagements and whatnot.
  • Working with Jinja and Ansible — at FIRST.

What was difficult?

  • Lack of context/proper knowledge about network setups led me to have an unclear picture with regards to ip addresses and what types of connections would be possible and what would not with regards to my localhost. Only when I consulted someone with that knowledge did I get a proper move-on.
  • Same old issue of knowing that I spend a lot of time slowly resolving what I’m doing wrong when software and tooling don’t work as expected, when I could resolve such questions a lot faster if I were on a team with colleagues or mentors to consult readily. I do have terrific people at my disposal who are willing to help, but these are people who are already working day jobs and don’t necessarily use the same solutions as me.
  • Judging whether to work a little bit more each day to get more done, or to wrap my day up at a good time in the evening so I keep a fresh mind (and stop working before the point of diminishing returns).
  • I have a habit of documenting what I do each day because it helps me feel organized, but it adds overhead every morning, so that even if I get to school relatively early (9, 9:30 am), I’m liable to start working 2 hours later.

What would I do differently next time?

This coming week I’ll reprioritize to verify my db setup first (over setting up the Ansible playbooks) since it’s more important for me to make sure my stack works correctly than to have a nice CI/CD setup. Since it’s not absolutely necessary for me to have the nice deploy playbook to do everything, I’ll focus on setting up more atomic plays that I can use for now.

A few cool things I learned about

  • the locate command, which I can use over the find command if I’m just looking for something by string in path
  • Django Configurations (to use classes to write your config instead of a bunch of variables) and the DJANGO_SETTINGS_MODULE
  • public ip addresses, private ip addresses, and routers
  • docker secrets
  • ansible playbooks and roles, yaml anchors, and the template module

Misc links for your browsing pleasure:

Docker
Time Management
Project Management
DevOps
Ansible

--

--

Rona Chong

Written by Rona Chong

89 Followers

(✿◕‿◕)

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams