File upload from Phoenix Channels to Amazon S3

Hello there, it looks like it’s my first article and so it is. I assume there’s much to learn for me in terms of writing good stories, so don’t be afraid to share your feedback with me. I will appreciate it, thanks in advance.

Few weeks ago I was looking for good tutorial how to upload file eg. pdf from channels, but I haven’t found one. Crazy thought: make it on your own. Ok!

This tutorial will show you how to upload files to Amazon S3 (or locally if you wish) using Phoenix Channels and JavaScript.

Prerequisites:

  • Elixir ≥ 1.3
  • Phoenix ≥ 1.2
  • Arc
  • Basic knowledge of JavaScript

Let’s code! We will do that in two simple steps.

  1. Generate the app and add dependencies
# generate new phoenix project without creating database
mix phoenix.new file_upload --no-ecto
# press "y" for fetching all dependencies or run mix deps.get
# move to project's folder
cd file_upload/

To use Arc, go to Hex- official package manager for Elixir and Erlang and download this package from here. To add it to our project, open in the root folder file mix.exs and add information about Arc (and ex_aws — it dependency, but we have to list it explicit) in two functions:

def application do
[mod: {FileUpload, []},
applications: [:phoenix, :phoenix_pubsub,:phoenix_html,:cowboy, :logger, :gettext, :arc]] # here add arc
end
defp deps do
[{:phoenix, "~> 1.2.1"},
{:phoenix_pubsub, "~> 1.0"},
{:phoenix_html, "~> 2.6"},
{:phoenix_live_reload, "~> 1.0", only: :dev},
{:gettext, "~> 0.11"},
{:arc, "~> 0.7.0"}, # add this line - get it from Hex's page
{:ex_aws, "~> 1.1"},
{:cowboy, "~> 1.0"}]
end

Now to start working with Arc, we have to update our dependencies using previously used command:

mix deps.get

We can check that Arc is usable in IEx console, so if you want to do that use:

iex -S mix
# check that Arc is in proper directory in /deps
iex(1)> Arc.module_info()[:compile][:source]
'c:/Users/patnowak/Desktop/file_upload/deps/arc/lib/arc.ex'

Now we want to use Arc’s mix task to generate a file_uploader for us

mix arc.g file_uploader

It will be placed in /web/uploader/file_uploader.ex file. If you would open it, it’s only imports some functions and contain a lot comments.

defmodule FileUpload.FileUploader do
use Arc.Definition
# Include ecto support (requires package arc_ecto installed):
# use Arc.Ecto.Definition
@versions [:original]
# a lot of comments
end

This one use does all the magic for you behind the curtains, because it uses another Arc modules and all they need to work is some default config from you.

If you choose to use local storage, just add in the uploader:

def __storage, do: Arc.Storage.Local

Otherwise, add in /config/config.exs complete informations about your S3 bucket:

config :arc,
bucket: “name of the bucket”
# arc uses ex_aws under the hood, which communicates with AWS 
# services including S3, which relies on hackney - HTTP client
# library in Erlang
config :ex_aws,
access_key_id: “your access key id”,
secret_access_key: “your secret access key”,
region: “your region”,
host: “your host”,
s3: [
scheme: “https://”,
host: “your host”,
region: “your region”
]

If we have done it, we can get to the step 2.

2. Phoenix channels in Elixir and handle the connection in JavaScript

I hope that you are familiar with Phoenix channels, but if you don’t, you’ll get what’s happening here very fast.

Short introduction

Usually almost everything we do in Internet is related with request /response cycle. You send a request, you’ll get and response and cycle is finished. There is no state between many cycles, so our browsers and applications try to figure out how to keep some important data in between.

However channels supports stateful communication. Imagine endless journey when you just add, modify or remove something from your inventory. In Phoenix this concept is called channel. In technical details: each channel is separated Erlang process (super fast and lightweight) and to communicate with channels usually websockets are used, but channels also supports other transport protocols.

There are plenty of web frameworks, which support websocket connections, including infamous Ruby on Rails with Action Cable, but nothing can compare to performance of Phoenix, thanks to Elixir and Erlang. Enough said. Move along to coding. You’ll be suprised that we don’t need code that much, because of Phoenix.

Coding time

Channels have simple interface — we can join a channel and handle the message we received eg. send a response to all users in the channel.

We could use generator for channel — mix phoenix.gen.channel <name>, but it’s better to code it from the scratch. We will name this channel upload.

If you’re using Mac or Linux you can create new file in /web/channels with:

touch upload_channel.ex

or simply adding new file (works on Windows as well).

For a start it should looks like this:

defmodule FileUpload.UploadChannel do
use FileUpload.Web, :channel
end

This single line with use will import for all of the content in /web/web.ex from channel function:

def channel do
quote do
use Phoenix.Channel
import FileUpload.Gettext
end
end

Line with use Phoenix.Channel basically injects functions we need to interact with socket and also imports from Phoenix.socket assign function, which allows us to add some information to socket in this infinite web journey.

For now there is no channel we can join, we don’t even use anyhow this new UploadChannel module.

First of all, we have to add an entry about channels we just want to open in socket handler — /web/channels/user_socket.ex file. Type there:

channel “upload:*”, FileUpload.UploadChannel

Then, all our changes will be in our channel’s file and in JS. It’s really not much. To allow any socket comes to our channel- we have to implement join/3 function. It expects three parameters:

  • name of the topic
  • auth message (map of parameters)
  • socket itself

And it should it returns one of three answers:

where map is the auth message — second argument.

We would like to skip this parameter, so our join function can looks like this:

def join("upload:new", _, socket) do
{:ok, socket}
end

It couldn’t be simpler.

Now it’s time to add some lines on JS side to check whether we can join to this upload:new topic. By default our static files eg. JS files, lies in /web/static/js directory. Here’s also socket.js, which import everything that Phoenix provides in that matter. We can create a new file — upload.js to handle our upload stuff, but copy this from the socket.js:

let channel = socket.channel("topic:subtopic", {})
channel.join()
.receive("ok", resp => { console.log("Joined successfully", resp) })
.receive("error", resp => { console.log("Unable to join", resp) })

And replace “topic:subtopic” with “upload:new”. Also we need to create a function, which accepts a socket and in that function connect with the socket and then paste the code from above. Also please comment the snippet in socket.js.

Your upload.js should looks like:

let Upload = {
init(socket) {
socket.connect()
let channel = socket.channel("upload:new", {})
channel.join()
.receive("ok", resp => { console.log("Joined successfully", resp) })
.receive("error", resp => { console.log("Unable to join", resp) })
}
}
export default Upload

For now it won’t do anything unless you use it. By default your app (if brunch is there) pack all your *.js files into one- app.js and it triggers brunch watch for each change of these files.

You have to import our upload, as well as socket, in app.js and use them explicitly:

import socket from “./socket”
import Upload from “./upload”
Upload.init(socket)

Now if you run the server with:

mix phoenix.server

And go to localhost:4000 you should see default Phoenix page and in your script console entry like this:

Joined successfully …

I won’t force you to write on your own simplest HTML tags so you can paste this into your /web/templates/page/index.html.eex:

<h1>Upload a file</h1>
<%= form_for @conn, “”, [multipart: true, as: :super], fn f -> %>
<%= file_input(f, :file) %>
<%= submit “Upload”, id: “button” %>
<% end %>

Don’t worry — there’s no magic out there. We simply use form_for function to generate form with file input for us and simple button to submit it.

Now, we’d like to script it a bit to handle what the submit button should do. Open again our upload.js and bellow joining to the channel, add:

let button = document.getElementById(“button”)
button.addEventListener(“click”, (e) => {
e.preventDefault()
this.onUpload(channel)
}, false)
// below the init method
onUpload(channel) {
let fileInput = document.getElementById("super_file")
let file = fileInput.files[0]
let reader = new FileReader()
reader.addEventListener("load", function(){
let payload = {binary: reader.result.split(",", 2)[1], filename: file.name}
channel.push("upload:file", payload)
.receive(
"ok", (reply) => {
console.log("got reply", reply)
}
)
}, false)
reader.readAsDataURL(file)
}

It’s a lot of code, but what you see is what you get — you wrap some HTML tags with variables and use FileReader to send content of the uploaded file as a data URL — think “binary” in Elixir. We want to send an object with tho fields: filename and binary to meet Arc’s requirements:

A map with a filename and binary keys (eg, %{filename: "image.png", binary: <<255,255,255,...>>})

This piece of code won’t do anything now. We don’t have handle_in yet to process in on the Phoenix side. Handle_in is responsible for receiving incoming messages and to process them and usually sending some informations to the channel subscribers.

Handle_in accepts three parameters — name of the message (action), parameters received and socket itself, so simplest handle_in would be:

handle_in(“upload:file”, params, socket) do
IO.puts params["filename"]
{:noreply, socket}

And when we choose a file and click the button, we’ll get an info about filename in the IEx console, that runs the server.

To adjust our parameters to Arc’s notation (atoms instead of strings) and decode binary that we received from JS use two private helper functions inside our upload_channel:

defp keys_to_atoms(params) do
Map.new(params, fn {k, v} -> {String.to_atom(k), v} end)
end
defp decode_binary(params) do
Map.put(params, :binary, Base.decode64!(params.binary))
end

Add also

alias FileUpload.FileUploader 

at the top of the file and then modify our handle_in callback:


def handle_in(“upload:file”, params, socket) do
params
|> keys_to_atoms()
|> decode_binary()
|> FileUploader.store()
{:noreply, socket}
end

And this should works :) If you need to customize what you want to get eg. url of uploaded file I recommend not to use Arc’s uploader.url function but create your own — I experienced that this provided by Arc doesn’t fit to my project — I used in url not only name of the bucket, but region and bucket name, and my url looks something like that:

s3.<region>.amazonaws.com/<bucket>/<path_to_file>

To solve this issue I created simple helper module, which generates url like this one.

I hope this tutorial wasn’t terrible experience for you. Even if so, please right a comment what can be improved. I’ll try to write shorter entries if I’ll find a topic worth to mention. Have a nice day!