Star Wars API — Part 2: Active Record, Migration, Rake, and Associations

In Part 1 we built the simplest of CLI applications to return some information that is given to us from the Star Wars API. If you want to checkout that project, the link to the post is here. And here is the link to the Github Repository, each folder is for a different part of this blog.

In this section we look into persisting the information from the API in order to access is via a web-app in the future. (look for Rails and Localhost in Part 3) Thankfully there are several gems available to us that allow us to easily create a database and insert records via Ruby saving us the time it would take to manually enter all of these several hundred rows and associations manually.

In this blog post we will go into the process of setting up ActiveRecord and Rake, and then filling our database with the info from the API.

Also note the database will be stored in the home directory of the repository and not in the part2 folder so it will be easily accessible for future parts.

Here is a preview of the steps:

Interpreting the Data

Set-up: Gemfile, config/environment.rb

  1. Setting up ActiveRecord Models, Rake, and Migration Files
  2. Writing methods for inserting “base” information
  3. Creating Associations and Migration Files for Association Tables
  4. Writing methods for creating associations
  5. Testing integrity of the data via Rake

Interpreting the Data

This take a little bit of time. When we look at each piece of data we need to identify the attributes which belong to each model and which pieces of information are an association. When we create the migration files a little later we only want to create necessary columns. When 2 models/objects both have a has_many relationship to each other then we need to create a relational table for this association and their columns do not need to be included in the base tables.

Below is the list of each model and it’s attributes and whether or not we want to insert them on the first round of iterating over the data, or the second time when we create the associations.

FILMS
1st Iteration:
title, episode_id, opening_crawl, director, producer, realease_date, created, edited, url
2nd Iteration:
characters, planets, starships, vehicles, species
CHARACTERS
1st Iteration:
name, height, mass, hair_color, skin_color, eye_color, birth_year, gender created, edited, url
2nd Iteration:
homeworld(=planet), films, species, vehicles, starships
SPECIES
1st Iteration:
name, classification, designation, average_height, skin_colors, hair_colors, eye_colors, average_lifespan, language, created, edited, url
2nd Iteration:
homeworld(=planet), people(=characters), films
PLANETS
1st Iteration:
name, rotation_period, orbital_period, diameter, climate, gravity, terrain, surface_Water, population, created, edited, url
2nd Iteration:
residents(=characters), films
STARSHIPS
1st Iteration:
name, model, manufacturer, cost_in_credits, length, max_atmosphering_speed, crew, passengers, cargo_capacity, consumables, hyperdrive_rating, MGLT, starship_class, created, edit, url
2nd Iteration:
pilots(=characters), films
VEHICLES
1st Iteration:
name, model, manufacturer, cost_in_credits, length, max_atmosphering_speed, crew, passengers, cargo_capacity, consumables, vehicle_class, created, edit, url
2nd Iteration:
pilots(=characters), films

Below is the list of the associations:

FILMS
has_many [planets, species, vehciles, starships, characters]
CHARACTER
has_many [films, starships, vehicles, species]
belongs_to [planet]
SPECIES
has_many [characters, films]
has_many [planets] through [characters]
PLANETS
has_many [films, characters]
has_many [species] through [characters]
STARSHIPS
has_many [characters, films]
VEHICLES
has_many [characters, films]

Here is the list of relational tables needed (the two words in the table name need to be in alphabetical order):

characters_films            #films_characters would not work!!!
characters_species
characters_starships
characters_vehicles
films_planets
films_species
films_starhips
films_vehicles

Later we will go into detail on how to set up these tables.

set-up

Gemfile

For this lab we need to add the following gems:

rake, activerecord, sqlite3, nokogiri, rest-client

config/environment.rb

In this file we need to require all other files, require our gems, and set-up the connection to the database.

File Tree

├── Gemfile
├── Gemfile.lock
├── Rakefile
├── app
│ └── models
│ ├── apicommunicator.rb
│ ├── characters.rb
│ ├── films.rb
│ ├── planets.rb
│ ├── species.rb
│ ├── starships.rb
│ └── vehicles.rb
├── bin
│ └── run.rb
├── config
│ └── environment.rb
├── config.ru
└── db
├── migrations
│ ├── 001_create_characters_table.rb
│ ├── 002_create_films_table.rb
│ ├── 003_create_species_table.rb
│ ├── 004_create_starships_table.rb
│ ├── 005_create_vehicles_table.rb
│ ├── 006_create_planets_table.rb
│ └── 007_create_all_relational_tables.rb
└── seeds.rb

step 1

Active Record Models

If we look at the homepage for the SWAPI we can easily tell we will need 6 different models, one for each table in our database corresponding to each of the 6 branches of the API. We will need to create these 6 tables via migration.

Let’s take a loot at what an ActiveRecord model would look like:

class Model < ActiveRecord::Base
end

This is the bare-minimum needed to set-up the “Model” class to work with Active Record. The class just needs to be initiated and then inherit from ActiveRecord::Base. When we add associations later on this is where we include them, but we’ll come back to that in a few steps. This is all we need for now. You should have 6 classes in app/models that contain the singular version of the file name. [characters.rb => class Character, films.rb => class Film, species.rb => class Specie, etc.]

Keep in mind when we need to manually set the Primary Keys later we will need to add the code inside each class.

Rakefile

In order to use rake we need to set-up a Rakefile. This file sits in the home directory and has no extension. In this file we define what we want rake to do. Let’s breakdown the 6 tasks in our Rakefile.

task :environment do
require_relative ‘./config/environment.rb’
end

This allows us to include our config/environment.rb in future rake tasks.

esc ‘drop into the Pry console’
task :console => :environment do
Pry.start
end

This allows us to type “rake console” in our terminal and drop into our program with access to all the information in our database. This is extremely helpful for testing the integrity of our data and we will use this extensively in Step 5.

desc 'database migration functions'
namespace :db do
  task :migrate => :environment do
ActiveRecord::Migrator.migrate("db/migrations/")
end
  task :drop => :environment do
File.delete("../db/starwars.db") if File.exist ("../db/starwars.db")
end
  task :reset => [:drop, :migrate] do
end
  task :seed do
require_relative 'db/seeds.rb'
end
end

As the description denotes, we are describing the tasks which interact with our database (starwars.db). The first one will run all our migrate files which will create all our tables. This is not the default migration file location. For my program I named the folder within /db/ “migrations” when by default it is named “migrate.” The drop command deletes the database file. The reset command runs the drop command followed by the migrate command. This is good when we are building our methods to insert data into the database. We will without a doubt make a mistake and need to totally reset the database. Reset allows us to do it in one command. Seed allows us to fill in the database. In this lab we aren’t using this command, but this was included to show how to set it up. You would fill the seed.rb with the commands to insert values into your database.

Migrations

The next step is to setup files to migrate in the first 6 database tables. I chose to do each in their own file so if there was a failure I could figure out which table my error was in significantly quicker. Let’s break down the template for creating these files by inspecting the first one from this repository:

## FILE LOCATION ./db/migrations/001_create_characters_table.rb ##
class CreateCharactersTable < ActiveRecord::Migration
def change
create_table(:characters, {id: false}) do |t|
t.integer :id
t.string :name
t.string :height
t.string :mass
t.string :hair_color
t.string :skin_color
t.string :eye_color
t.string :birth_year
t.string :gender
t.string :created
t.string :edited
t.string :url
t.belongs_to :planet, index: true
end
end
end

First, the file name: They need to be made in sequential order so the first file we migrate will start with 001, the second with 002, and so on. Also the words in the filename should match the name of the class within in it. The only difference is the class name is CamelCased and does not contain any spaces. [001_create_characters_table => class CreateCharactersTable]

Second, we need to inherit from ActiveRecord::Migration. This is done the same way the models inherit from ActiveRecord::Base.

Next is the methods. Here we have a single method called “change.” This method is 2 other Migration methods combined, “up” which would create the database table, and “down” which would drop it. CHANGE allows us to only have to write the code for the UP part while DOWN is implied.

Next is the method called within change. Here we are creating a table, but you can also add_column, drop_column, change_table, and many others. You can find all your options when looking at the ActiveRecord::Migration Ruby on Rails Documentation. In this example we are creating a table with our own defined PRIMARY KEY. To define your own primary key you need 3 things:

  1. {id: false} | This needs to be included in the options of the create_table method. This tells active record that it does not need to create an id primary key for us as we plan on creating it ourself.
  2. t.integer :id | This created a column in the database with the name “id” and a value of integer. This will be our primary key, but it is not auto-incrementing and we are allowed to set the values ourselves.
  3. This happens in the model file. For our fake class ‘model’ from above we would set a primary key of ‘id’ like this:
class Model < ActiveRecord::Base
self.primary_key = ‘id’
end

Ok, great! We now have a template for our migration file and we can repeat this for all 6 files.

Now that all 6 models are made, all 6 table migration files have been created, and the Rakefile is setup to execute these migrations we can move onto Step 2!

step 2

Writing the Methods to Pull Information on the 1st Iteration

As stated in the “Interpreting the Data” we already know what information we want to add to the database the first time we iterate over the arrays we pull. Let’s create a model for accessing the API, making an array of all objects from a class, and then assigning the columns to the different pieces of data.

First, creating the model. In ./app/models/ we add the file “apicommunicator.rb” and add the only attr_accessor we will need, a page url that will be passed around through several methods:

class ApiCommunicator
  attr_accessor :page_url
end

Second, accessing the API:

def import_page_data
page_json = RestClient.get(@page_url)
hash = JSON.parse(page_json)
hash
end

Here, we use a page url to create a hash of all of the data presented on each page. It will be looking for a class variable @page_url each time it is called.

Third, creating an array of all data for one Model/Object:

def pull_all_pages_data
output = []
until @page_url == nil
page_data = import_page_data
page_data[“results”].each do |x|
output << x
end
@page_url = page_data[“next”]
end
output
end

Here, when we are fed the initial page of we iterate through all pages until the next page is ‘nil’ and add every object to the array ‘output.’ We then return the array which we can iterate over for data.

Next, creating a method for each top-level link:

def pull_characters
  #this pulls the characters and saved them to the DB.
#this does not include relational data
#since we need everthing in the DB before we can
#start relating them all to each other
#this pattern repeats for the next 5 classes
  @page_url = “http://swapi.co/api/people/"
@chars_array = pull_all_pages_data
@chars_array.each do |x|
new_character = Character.new(
id: x[“url”].split(‘/’).last.to_i,
name: x[“name”],
height: x[“height”],
mass: x[“mass”],
hair_color: x[“hair_color”],
skin_color: x[“skin_color”],
eye_color: x[“eye_color”],
birth_year: x[“birth_year”],
gender: x[“gender”],
created: x[“created”],
edited: x[“edited”],
url: x[“url”])
new_character.save
end
end

Here is where we set the instance variable @page_url and we set it to the first page of characters. Then pull all pages data will handle getting all pages for us by calling import_page_data on each page. Once we have the array, set to the instance variable @chars_array (we will call this later when we set relations to avoid making extra API calls) and then iterate over this array and creating an object out of each element.

The only item that is different from all the others is “id”. If we inspect the API then we can tell they have created an ID in the form of a url for each instance of an object. We can use their ID in our own system to make associations much easier. Let’s break down how we set the id:

id: x[“url”].split(‘/’).last.to_i
x["url"] for the first character is: http://swapi.co/api/people/1/
If we split by '/' and take the last element of the array we are returned "1". We need to turn this into an integer to match the datatype expected by the database.

We then need to create a method following this template for the 5 other top level links. When you are done you should have a method to pull data, parse it and save it to individual instances for all 6 objects. I chose to name my methods pull_characters, pull_films, pull_planets, pull_species, pull_vehicles, and pull_starships. Check out the details in the repository in part 2 to see the specifics for all 6 methods.

Good Time to Test!

Here would be a good time to create a bin/run.rb and try pulling the data for the 6 objects. Here is what my run.rb looked like when I was testing this. After testing I would open the database in sqlite3browser and inspect the data to see if it resembles what I am looking for.

require_relative '../config/environment.rb'
x = ApiCommunicator.new
x.pull_characters
x.pull_films
x.pull_species
x.pull_planets
x.pull_starships
x.pull_vehicles
### This will take a minute or 2 to run ###
### Every page of the API needs to accessed and scraped ###

If this runs successfully you can either check the database via a browser or type “rake console” in your terminal followed by Character.find(1) and it should return Luke Skywalker:

step 3

Migrations and Associations

As pointed out in the “Interpreting The Data” section we need to add associations or relations. For this post, for 2 objects that both have has_many relationships with each other, will be represented with has_and_belong_to_many.

Here is the code snippet of every class and all of their code:

class Character < ActiveRecord::Base
has_and_belongs_to_many :films
has_and_belongs_to_many :starships
has_and_belongs_to_many :vehicles
has_and_belongs_to_many :specie
belongs_to :planet
self.primary_key = ‘id’
end
class Film < ActiveRecord::Base
has_and_belongs_to_many :planets
has_and_belongs_to_many :specie
has_and_belongs_to_many :vehicles
has_and_belongs_to_many :starships
has_and_belongs_to_many :characters
self.primary_key = ‘id'
end
class Planet < ActiveRecord::Base
has_and_belongs_to_many :films
has_many :characters
has_many :specie, through: :characters
self.primary_key = ‘id’
end
class Specie < ActiveRecord::Base
has_and_belongs_to_many :characters
has_and_belongs_to_many :films
has_many :planets, through: :characters
self.primary_key = ‘id’
end
class Starship < ActiveRecord::Base    #Vehicle is exactly the same.
has_and_belongs_to_many :characters
has_and_belongs_to_many :films
self.primary_key = ‘id’
end

This means we need to create 8 tables. I created a migration file ‘007_create_all_relational_tables.rb’ containing the code:

class CreateAllRelationalTables < ActiveRecord::Migration
def change
  create_table :characters_films do |t|
t.belongs_to :character, index: true
t.belongs_to :film, index: true
end
  create_table :films_planets do |t|
t.belongs_to :film, index: true
t.belongs_to :planet, index: true
end
  create_table :films_species do |t|
t.belongs_to :film, index: true
t.belongs_to :specie, index: true
end
  create_table :films_starships do |t|
t.belongs_to :film, index: true
t.belongs_to :starship, index: true
end
  create_table :films_vehicles do |t|
t.belongs_to :film, index: true
t.belongs_to :vehicle, index: true
end
  create_table :characters_starships do |t|
t.belongs_to :character, index: true
t.belongs_to :starship, index: true
end
  create_table :characters_vehicles do |t|
t.belongs_to :character, index: true
t.belongs_to :vehicle, index: true
end
  create_table :characters_species do |t|
t.belongs_to :character, index: true
t.belongs_to :specie, index: true
end
end
end

Now if you run ‘rake db:reset’ in console you should see 7 passing migrations instead of 6. This means that these 8 tables have been successfully created.

Now we can also check if our relations are set up properly. Re-import your data, now that you have reset the database, via bin/run.rb. You should get an empty array on all calls of a has_and_belongs_to_many relationships.

Here you can see :

char = Character.find(1) which returns Luke Skywalker. He has the attributes of planet, species, films, vehicles and starships but we have not added them in yet so they are blank array in the has_many relationships and a nil value for the belongs_to relationship.

Here is:

film = Film.find(1) which returns episode#4. It also has no returns for those 6 attributes, but the fact they return blanks and not errors means we have set-up the relationships correctly.

The error here is because of the way we set up those relationships. Film has_and_belongs_to_many “specie” not “species” so that affects the name of the method we call on it

step 4

Methods for the 2nd Iteration which Assigns the Relationships via Active Record

Lets re-inspect the relationships between objects. If we closely examine these relationships we see that we can create ALL of our associations by only iterating over 2 of the objects. One side of all of the relationships are contained in the Character and Films objects. So let’s write the method for character first since it is a little bit more complicated. Here is the code snippet for the method and below the snippet we will break it down:

def pull_characters_relations
@chars_array.each do |x|
current_char = Character.find(x[“url”].split(‘/’).last.to_i)
x[“films”].each do |film|
current_char.films << Film.find(film.split(‘/’).last.to_i)
end
if x[“species”].empty?
else
x[“species”].each do |specie|
current_char.specie <<
Specie.find(specie.split(‘/’).last.to_i)
end
end
if x[“vehicles”].empty?
else
x[“vehicles”].each do |vehicle|
current_char.vehicles <<
Vehicle.find(vehicle.split(‘/’).last.to_i)
end
end
if x[“starships”].empty?
else
x[“starships”].each do |starship|
current_char.starships <<
Starship.find(starship.split(‘/’).last.to_i)
end
end
current_char.planet =
Planet.find(x[“homeworld”].split(‘/’).last.to_i)
current_char.save
end
end

First, it pulls @chars_array from the instance method we used before called pull_characters. This array is all 87 characters in the database in 1 array. We then iterate over this array to start to look into the 5 categories we passed over before. For 4 of these 5 categories the return value of the x[“keyword”] is an array, which we iterate over to grab the id in the url. The same ID we used when we created these records. So then we can #find this instance of the Object and shovel it into the corresponding array of the Character object. The only different example is Planet since x[“homeworld”] returns a string instead of the array. This just takes the iteration step out. We still need to grab the ID as an integer and find the planet with that ID and set it equal to the planet for each character.

The version for films, #pull_all_films_relations , is an even more straightforward version of the method clipped above.

In the version on the repository all the necessary methods are run in the correct order in the bin/run.rb file. To execute all these methods, when in the “part 2” directory enter “ruby bin/run.rb” in console and wait a minute for all the API calls.

step 5

Testing Your Database with Rake

Let’s dive right in. Assuming you ran the code above and all your methods have completed without failing it is time to test. Type “rake console” in your terminal to enter a pry console with access to your database.

Let’s start by checking the length of the Class.all methods and see if they match the count on the SWAPI pages.

Solid start, this matches exactly to the counts in the API.

Next we should check to see if the first film has the right number of associated characters, vehicles, starships, planets and species.

This also matches our expected counts exactly. You can verify yourself by looking at this link for the first film of the API. And if you remember above these 5 methods returned empty arrays when called on any films before we made the associations.

Let’s do one last test to make sure our “has_many_through” relation is working as intended.

Here you can see that an instance of the Planet class can return information on the species that call the place home.

You are more than welcome to check all the details of the database with rake, but the fact we received no errors with mismatching ID’s during the relations methods, that our counts match up perfect, and that our “has_many_through” works as intended is a very strong indication that we have successfully persisted this data.

Congratulations! Next we’ll create an index page for each class and show pages for every instance of every class to display all the info in the database via Rails. (Link here)

Robert Hopkins | robert.hopkins@flatironschool.com