myfile: Using Active Storage

Photo by Cleyder Duque from Pexels

Introduction

This article builds upon the simple myfile sample application that we built in the first article. We’re going to switch to using Active Storage for handling the file uploads/downloads, storing all files in OCI Object Storage.

Background

This is part of a series of articles on using some popular Ruby gems with OCI. If you’ve not been following along, please see the first article, which provides the base app that we’ll be working off of in this tutorial.

We’ll be working with OCI Object Storage, so if you don’t have an OCI account yet, you should sign-up for one now! Have any questions, comments or want to chat with other like-minded techno geeks? Feel free to comment on this article (below) as well as join the fun on Slack.

What it is

There’s really no new functionality, per-se (at least on the front-end) that we’ll be introducing. Rather, how we store files is going to change. We’ll persist them to OCI Object Storage and create a full-featured interface for handling “attachments” using Rails Active Storage.

What it is not

To remind folks, this is not designed to be a “go out and do this today in production!” situation, but rather to show how to integrate (use) OCI Object Storage in a Rails application. This particular implementation uses Active Storage to handle the management of attachments. Security best practices are not highlighted — it’s more of a “show-and-tell” example.

Getting started

Start with a “clean” copy of the myfile sample Rails application that we built in the first article of this series. You can either copy-and-paste the myfile application to another directory (such as myfile_active_storage) or create a branch to work off of. Either way, we’ll be starting where we left off, so grab a cup of tea and let’s dive in!

Create an OCI Object Storage Bucket

First things first, we need a bucket created. I opted to do this using Terraform. Here’s an example Terraform code snippet (the provider, etc. is not shown — just the few relevant excerpts relating to the bucket):

data "oci_objectstorage_namespace" "this" {
  compartment_id = var.tenancy_ocid
}
resource "oci_objectstorage_bucket" "test_bucket" {
  compartment_id = var.compartment_ocid
  name = "myfile-development"
  namespace = data.oci_objectstorage_namespace.this.namespace
  access_type = "NoPublicAccess"
  storage_tier = "Standard"
  
  lifecycle {
    ignore_changes = [defined_tags["Oracle-Tags.CreatedBy"], defined_tags["Oracle-Tags.CreatedOn"]]
  }
}

The above snippet creates a bucket called myfile-development in a specific compartment. The bucket is made private (public access is not granted) and the Standard storage tier is used. Although some of these might be the default settings, I like to explicitly spell it out to make it painfully obvious what is being done. Someone who might not be familiar with the OCI Terraform provider and the oci_objectstorage_bucket resource can still infer some of the basics from looking at the above.

If you’d rather, you could use the OCI Console to create the bucket. If you’re unfamiliar with how to do this, please see the directions in the OCI documentation.

Before we’re totally done, there’s the matter of setting up credentials for the OCI S3 Compatibility API. Follow the directions for setting up a Customer Secret Key in the section titled Setting up access to Oracle Cloud Infrastructure found in the OCI S3 Compatibility API documentation. Store these credentials somewhere secure as you’ll be needing them shortly!

Mirroring data

Active Storage provides the ability to mirror data from one location to another. How might this be used? One of the easiest ways is to replicate from one OCI region to another. This functionality is also available natively within OCI, using the OCI Object Storage Replication functionality.

Which is best? It’s up to you. One red flag with the Active Storage method is that it’s not atomic. The Active Storage Mirror Service documentation is very clear about mirroring not being a “guaranteed” action. My preference is to go with the OCI-native functionality, which will deliver solid performance that’s not dependent on my application code. In my opinion, this guarantees a higher level of assurance that it’ll be done properly.

We’re getting ahead of ourselves, though — we haven’t even set up Active Storage in our project yet. Let’s do that!

Adding Active Storage

Active Storage uses several tables that need to be created (the Active Storage documentation tells you more about these). This is accomplished by the following commands:

$ rails active_storage:install

This will create the migration responsible for creating the three tables. Run the migration with:

$ rake db:migrate

Now we’ve got the basic infrastructure in place to use Active Storage in our application. We’ll need to tell Active Storage which storage locations are available: configure config/storage.yml with something like the following:

# config/storage.yml
oci:
  service: S3
  access_key_id: <%= ENV['OCI_KEY'] %>
  secret_access_key: <%= ENV['OCI_SECRET'] %>
  bucket: myfile-<%= Rails.env %>
  region: <%= ENV['OCI_REGION'] %>
  endpoint: <OCI S3 Compatibility API URL >
  force_path_style: true

I’m going to avoid the endless argument of which is better (more secure): config files (that are locked down and NOT committed to the repo, of course) or environment variables. For this example we’ll be using environment variables. Don’t worry about the environment names used here! Modify as you’d like… you’ll get the idea of how it’s done.

The endpoint parameter should be set to the proper URL. Refer to the documentation to see a list of the different OCI S3 Compatibility API endpoints available. You’ll need to also get your OCI Object Storage namespace to complete the API. The Terraform above gets the namespace, but it can also be obtained via the OCI Console, the OCI CLI or other methods. See the OCI Object Storage Namespaces documentation for more info on how to your OCI namespace.

Here’s an even more elegant solution for the endpoint (again, this would go in config/storage.yml):

# config/storage.ym
oci:
  service: S3
  access_key_id: <%= ENV['OCI_KEY'] %>
  secret_access_key: <%= ENV['OCI_SECRET'] %>
  bucket: myfile-<%= Rails.env %>
  region: <%= ENV['OCI_REGION'] %>
  endpoint: https://<%= ENV['OCI_NAMESPACE'] %>.compat.objectstorage.<%= ENV['OCI_REGION'] %>.oraclecloud.com
  force_path_style: true

By simply providing two more environment variables (or these could be configuration file settings), we’ve found a way to make it easy for us to use different OCI regions with the solution, without having to change the endpoint URL.

NOTE: The force_path_style parameter must be set to true, otherwise you’ll get an error like the following:

Aws::S3::Errors::MalformedXML (The XML you provided was not well-formed or did not validate against our published schema.):

Edit config/environments/development.rb and make sure that a line like the following is present:

# config/environments/development.rb
# use OCI Object Storage for file storage
config.active_storage.service = :oci

Note that there will likely already be a line present for the config.active_storage.service setting (mine was set to use :local).

Before we get ahead of ourselves, notice how we are using the S3 service for the oci storage service? We need to install the aws-sdk-s3 gem for this to work. Let’s do this now, by adding the following to Gemfile:

# Gemfile
gem "aws-sdk-s3", require: false

(this is directly copied from the Active Storage documentation)

Let’s run Bundler to make sure that all gems are ready for our use:

$ bundle

With the basic “infrastructure” (tables, storage location definitions, etc.) in place we’re ready to migrate our app to use Active Storage (instead of our own homebrew solution).

Revised model structure

If you take a look at the migration that Active Storage created for us (with the rails active_storage:install command), you’ll notice that the active_storage_blobs table has several fields that duplicate what we have in our own things table: filename and content_type. The active_storage_attachments table has the blob column, which stores the actual binary file data. It turns out we no longer need our own columns in the things table for these data points! Let’s generate a new migration to get rid of several unneeded columns:

$ rails g migration RemoveThingsColumns

You’ll find the file in db/migrate (it tells you the path in the create line as well). Go ahead and modify yours to look something like the following:

class RemoveThingsColumns < ActiveRecord::Migration[7.0]
  def change
    remove_column :things, :file_name, :string
    remove_column :things, :content_type, :string
    remove_column :things, :file_data, :binary
    add_column :things, :document, :attachment
  end
end

We’re removing three columns (file_name, content_type and file_data) from the things table and adding a new one (attachments) that will be our linkage to the Active Storage goodness! What this means is that we’ll have a thing record, which can have one document attached (and managed) via Active Storage (the :attachment type). How awesome is that?! I’m liking this. Apply the migration by running:

$ rake db:migrate

Now we’re ready to update the model/controller in our app. First, update the model to establish the relationship between a thing and a single document. Modify app/models/thing.rb to look like the following:

# app/models/thing.rb
class Thing < ApplicationRecord
  has_one_attached :document
end

Now let’s edit app/controllers/things_controller.rb. Scroll to the bottom, under private, updating the thing_params method from:

# app/controllers/things_controller.rb
    def thing_params
      ret = params.require(:thing).permit(:description)
      ret.merge( { file_data: params[:thing][:file_data].read, file_name: params[:thing][:file_data].original_filename, content_type: params[:thing][:file_data].content_type } )
    end

To this:

# app/controllers/things_controller.rb
    def thing_params
      params.require(:thing).permit(:description, :document)
    end

In the same controller, change this:

# app/controllers/things_controller.rb
class ThingsController < ApplicationController
  before_action :set_thing, only: %i[ show edit update destroy download ]

To this:

# app/controllers/things_controller.rb
class ThingsController < ApplicationController
  before_action :set_thing, only: %i[ show edit update destroy ]

And remove the download method altogether (it’s no longer needed as Active Storage has this taken care of for us!):

# app/controllers/things_controller.rb
  def download
    # credit to https://piotrmurach.com/articles/streaming-large-zip-files-in-rails/ for the header to set
    response.set_header('Content-Disposition', "attachment; filename=\"#{@thing.file_name}\"")
    response.content_type = @thing.content_type
    response.write(@thing.file_data)
  end

Remove the route for download, editing /config/routes.rb, to change this:

# config/routes.rb
Rails.application.routes.draw do
  resources :things do
    get 'download', on: :member
  end
  # Define your application routes per the DSL in https://guides.rubyonrails.org/routing.html
# Defines the root path route ("/")
  root "things#index"
end

To this:

# config/routes.rb
Rails.application.routes.draw do
  resources :things
  # Define your application routes per the DSL in https://guides.rubyonrails.org/routing.html
# Defines the root path route ("/")
  root "things#index"
end

Edit (or create, if it’s missing) config/initializers/active_storage.rb, making sure the following line exists:

# config/initializers/active_storage.rb
Rails.application.config.active_storage.resolve_model_to_route = :rails_storage_proxy

Let’s move on with updating the different views. Modify app/views/things/_form.html.erb, changing this:

# app/views/things/_form.html.erb
  <div>
    <%= form.label :file_data, style: "display: block" %>
    <%= form.file_field :file_data %>
  </div>

To this:

# app/views/things/_form.html.erb
  <div>
    <%= form.label "Document", style: "display: block" %>
    <%= form.file_field :document %>
  </div>

Go ahead and change app/views/things/_thing.html.erb from this:

# app/views/things/_thing.html.erb
<div id="<%= dom_id thing %>">
  <p>
    <strong>File name:</strong>
    <%= link_to thing.file_name, download_thing_path(thing) %>
  </p>
<p>
    <strong>Content type:</strong>
    <%= thing.content_type %>
  </p>
<p>
    <strong>File data length (bytes):</strong>
    <%= thing.file_data.length %>
  </p>
<p>
    <strong>Description:</strong>
    <%= thing.description %>
  </p>
</div>

To this:

# app/views/things/_thing.html.erb
<div id="<%= dom_id thing %>">
  <% if thing.document.attached? %>
  <p>
    <strong>File name:</strong>
    <%= link_to thing.document.filename, rails_storage_proxy_path(thing.document) %>
  </p>
<p>
    <strong>Content type:</strong>
    <%= thing.document.content_type %>
  </p>
<p>
    <strong>File data length (bytes):</strong>
    <%= thing.document.byte_size %>
  </p>
  <% else %>
  <p>No document is attached to this <i>thing</i>.</p>
  <% end %>
  
  <p>
    <strong>Description:</strong>
    <%= thing.description %>
  </p>
</div>

Remember, this isn’t about good Ruby/Rails code, but rather how to get Active Support working with OCI Object Storage. Toward that end I took some shortcuts and ended up embedding some logic in the view (bad, bad, bad). The resulting ERB template is pretty messy. We really should be using something like Mustache to separate template from logic. I get it. You get it. Again, we’re keeping things pretty simplistic: I didn’t want to add this additional “clutter” to the picture, instead keeping the focus on the task at hand: getting Active Storage to use OCI Object Storage.

We’re now ready to see this thing through! Fire it up with something like the following (this works for many *nix shells):

$ export OCI_KEY=<YOUR OCI KEY HERE>
$ export OCI_SECRET=<YOUR OCI SECRET HERE>
$ export OCI_NAMESPACE=<YOUR OCI NAMESPACE HERE>
$ export OCI_REGION=<YOUR OCI REGION HERE>

The OCI region needs to be its region identifier. Check out the OCI documentation for a list of regions and identifiers for more info. Then you run it:

$ rails s

For some reason, I had an error that arose at this point. In case you run into the same thing “link_tree argument must be a directory”, I went into app/assets/config/manifest.js and removed the offending line that it was complaining about. In my case, the following line was the problem (which I removed):

//= link_tree ../../../vendor/javascript .js

Things proceeded fine from that point.

What we have

Active Storage offers a really nice interface for handling files in our application. There are several different methods available on the document attachment. Follow along yourself in the rails console (run rails c):

t = Thing.last
# this will tell us whether or not a file is attached
t.document.attached?
# here's how to get the file size
t.document.byte_size
# self-explanatory
t.document.filename.to_s
# again, no explanation needed here
t.document.content_type

This is just a really small sampling of some of the attributes available. Take a look at here and here for more info on some of the great methods and attributes Active Storage provides.

Cleaning up

This has been a lot of fun, right?! Now it’s time to clean-up the mess we’ve left behind. In case you try to delete the myfile-development bucket (or whatever name you selected), you’ll likely encounter an error. This is because a bucket needs to be empty to be deleted. You could iterate through all of the different objects and manually delete them, but that’s a real pain. While we’re at it, rather than just deleting the objects in Object Storage, let’s go ahead and keep the local database clean as well, deleting the records (and objects) for each thing once we’re done. Here goes… open up rails console (rails c), then issue the following (don’t forget to make sure your environment variables are set first!):

Thing.all.each do |t|
  t.document.purge
  t.destroy
end

One might think that simply calling Thing.destroy_all would work, but I found this to be problematic. The above couple of lines cleanly destroys the objects (“purges” them, one by one) and destroys the record in the local SQLite3 database (destroying the actual “thing” resource).

At this point, it’s safe to go ahead and delete the OCI Object Storage bucket (myfile-development) that we’ve been working with, as it should be empty now.

Resources

There are some great references and resources that you might find useful (I did!):

• https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm
• https://docs.oracle.com/en-us/iaas/api/#/en/s3objectstorage/20160918/
• https://registry.terraform.io/providers/hashicorp/oci/latest/docs/resources/objectstorage_bucket
• https://www.oracle.com/cloud/storage/object-storage/faq/
• https://guides.rubyonrails.org/active_record_migrations.html
• https://guides.rubyonrails.org/active_storage_overview.html
• https://www.rubyguides.com/2019/01/ruby-environment-variables/
• https://www.rubydoc.info/gems/activestorage
• https://api.rubyonrails.org/classes/ActiveStorage.html

Wrapping it up

This has been a fun little journey. Active Storage is alive and well. It just so happens that it works really well with OCI Object Storage! Not that I had any doubts, but it was great to do a quick show-and-tell for you to show off just how easy this integration can be. Hopefully you’ve found this at least mildly amusing, if not helpful, reference. It’s been a good exercise for me, allowing me to document some of the specific integration points.

What’s next

What if you’re already using a particular gem (other than Active Storage) for handling file attachments? Wouldn’t it be nice to see how myfile might work with some other file handling gems (instead of Active Storage)? Not that there’s anything wrong with Active Storage… I think it’s actually really cool. But what if you already have a favorite gem for handling attachments that you’d like to continue using (along with OCI Object Storage)? That’s an interesting idea… stay tuned for some upcoming iterations. Until next time, keep the bits flowing.