Basecamp is not the baseline for a Rails company

Before talking about the numbers I’d like to point out that David has access to the crème de la crème of Ruby on Rails developers. Being the creator of Rails he and his company are a dream employer for the top notch people. Of course all A players want to play with other A players. All of my Rails clients have serious problems to find good and affordable developers because they don’t have the visibility or coolness factor of Basecamp. Yes, many of my clients are not A players them self. But who am I to blame them for it? Rails is not just for A players. It never should be!

Of course the developers of Basecamp are better than the average Rails company. And of course they max out the potential of Ruby on Rails. None of my clients could do that. Most companies don’t have access to heros or rockstars. And one of the reasons I love Rails is that they don’t need to. But having just normal developers is first reason why 15% for most companies is unfeasable. But let’s dive into the raw technical arguments because I don’t buy the 15% claim for Basecamp either.

A swarm of Redis servers

David lists the different kind of servers they run at AWS. He mentions “A swarm of Redis servers providing caching.” Unfortunately he didn't quantify how much a “swarm” is. Plus he didn’t say which services use the Redis servers. Knowing a bit about caching and performance optimization in Rails (watch my RailsConf “Cache = Cash!” talk on YouTube) I guess that a big chunk of the swarm does caching for Rails because it is a good solution for many Rails performance issues. In my opinion it is unfair to not include those costs in his post. They need those Redis servers because they are using Rails. Without concrete numbers I have to guesstimate: If you spend $ 450,000 on Rails servers … hmmm … let’s say you spent $ 200,000 on Redis which are just used by Rails (including job servers).

David, in case you are reading this: Please do write an other post with more details so that everybody can make up his/her mind about this. I’ll be happy to update this article once those numbers are out. Thank you in advance!

Using caching in Rails is a very solid performance booster. But it is costly. Of course you need Redis or any other caching server but you also need more developer time. And as David recognized it those are high costs for any company. The problem with writing code which uses a lot of caching is the complexity. It is very error prone. A lot of my clients hire me to find bugs where they shot themself in the foot by some complex caching system.

So we are dealing with a relative slow environment which we speed up by using caching but that itself adds complexity which means we need more manpower which is expensive.

The cost of deployment

The article doesn’t mention how Basecamp deploys it’s application. In my experience this is a tricky aspect of any Rails application. For an application this size and with that many active concurrend users you want a zero downtime deployment and the possiblity to roll back. Without having more knowledge about the used technology at Basecamp I can not argue if the 15% would have to be increased by a small swarm of additional servers to handle this or not.

Order of magnitude

David writes: “Let’s imagine that there was some amazing technology that would let us do everything we’re doing with Ruby on Rails, but it was TWICE AS FAST!” Later he writes: “Now imagine we found a true silver bullet. One where the compute spend could be reduced by an order of magnitude.”

Long story short: There is such a solution. It is the Phoenix Framework. But I don’t want to reheat the Rails vs. Phoenix discussion here. There are plenty of good reasons why to use Rails (see my other post about it).

The value of $ x00,000

David argues that using a silver bullet like Phoenix would reduce their costs by “just” $ 400,000/year which wouldn’t even pay for two developers. I have a couple of issues with this:

• Assuming Basecamp could reduce the cost by $ 400,000 it would mean that the 15% number would be so much lower on that alternative solution.
• As written above: I don’t buy the 15% in the first place. Even for Basecamp. I bet $ 20 that their real operational cost because they are using Rails is higher than 15% (see e.g. the Redis argument).
• $ x00,000 is a lot of money for most companies. Let’s be a little bit more realistic about those numbers.

Let’s talk about Heroku for a moment

Basecamp uses AWS but many Rails companies use Heroku. Heroku became kind of a default way of running Rails for a majority. They lure developers with free mini versions, super easy setup and painless deployment in the beginning. And once the applications run on Heroku companies find it hard to move to AWS or bare metal. Anybody who has ever seen a Heroku invoice for a big installation knows that running a Rails application in that way can be quite expensive.

Rails is bad?

For a first version of this post I got a lot of feedback about how great Rails is. I never said something different in my text. I just want to paint a more realistic picture for an average company about the costs. They will not just spend 15% for Ruby servers. They will spend more. Still it is a good choice. The benefits of Rails are tremendous (e.g. big eco system, relative easy onboarding of new team members, ActiveRecord, I could go on for ever).

I want people to use Ruby on Rails! I understand why Basecamp uses Rails. I just do not want newbies being lured into using Rails with the promise that only 15% of their operations budget is going to go into Ruby servers.

One favor

If you have read all the article I’d like to ask you for one favor: Have a look and create an account at my open-source social network https://www.vutuv.de

Think of it as a fast and free LinkedIn which will get some Medium features in the near future. It is not fancy yet but this is a chicken and egg problem. Therefor I need your help to get more people on board. Thank you!

To achieve this in nginx you have to create a list of bots you don’t want to log and put it as follows in your /etc/nginx/nginx.conf

map $http_user_agent $log_excluded_ua {
 ~Googlebot 0;
 ~bingbot 0;
 ~AhrefsBot 0;
 ~Yahoo! 0;
 ~proximic 0;
 ~GrapeshotCrawler 0;
 ~MJ12bot 0;
 ~Mediapartners-Google 0;
 ~SemrushBot 0;
 ~MegaIndex.ru 0;
 ~Mediatoolkitbot 0;
 ~DotBot 0;
 default 1;
}

After that you can tell nginx in your site configuration to only log an access when it’s not a crawler:

access_log /var/log/nginx/www.example.com-access.log combined if=$log_excluded_ua;

It took me a while to understand and appreciate features like Active Storage, Active Job, Action Cable or the new Action Mailbox. Many times my initial thought was “There is already at least one good solution (gem) for that problem. Why reinventing the wheel?” But during my work on so many different Rails projects I began to realize that it is so much better to give each team a good to go initial set of tools. Of course they can use a different gem (like so many do for testing) but they have a very good start with the vanilla Rails project. And the more projects use the same base the better the general documentation and code quality gets. It has never been easier to find help for a Rails problem on stackoverflow.com than today. No need for fancy gems to solve standard problems. BTW: Many times those fancy gems don’t survive the next Ruby upgrade because their maintainer lost interrest or just doesn’t have time for it.

Development of new Rails projects has never been easier and faster! But …

Deployment is still a nightmare

What good is a framework if it is not easy to deploy to a production server? In the pre-Rails days many of us just copied a directory of PHP files to the web server via scp or ftp and that was it. With Rails it became more complicated.

Of course you can use Heroku which takes most of the pain out of this. Many companies do use Heroku but it is expensive and not always super fast. Many companies want to deploy their Rails application to a normal out of the box linux distribution (e.g. Debian or Redhat).

Apache or Nginx don’t do the hard lifting for Rails like they do with PHP. But you still need them. So you have to make a choice of Puma, Unicorn or whatever to actually run your Rails application and then you have to put Apache or Nginx in front of that. Or you use Phusion Passenger (which involves some fiddling around with the Nginx or Apache modules). All complicated enough for a Rails newbie. Additionally you have to use RVM or rbenv to run the needed Ruby version.

But you still have to find a way to actually copy your source code to the server, run all needed migrations and restart the server. The bravest even go so far to attempt zero downtime deployments (a little secret: You want to have a look at Phoenix Framework). So you have to dive into Capistrano or some cool new kid like Mina.

To setup deployment for a simple Rails application on Heroku takes a couple of minutes. 30 minutes tops (if you have to get the credit card information of your boss first). But the setup of a deployment process on a vanilla Linux Server from the ground up takes hours if not days. You have to read too many different tutorials and have to make too many decisions. And you end with a snow flake. Beautiful but one of a kind and hard to support.

Docker! ?

I hear all those voices which shout: Why don’t you use Docker for this? Docker makes everything so easy!

No, it does not! I challenge you to find a single tutorial which describes how to setup a Docker deployment from start to finish for a Rails application for a Debian server. Most of the how-tos are half baked at best. They just show you how to run Docker for your development system.

But don’t get me wrong: I’m all for Docker. This posting is not pro or against Docker. This posting is about a standard deployment for Rails.

But we pros know how to do it!

That doesn’t help at all. If a normal developer who maybe works on 1 or 2 Rails projects can not create a simple non-Heroku deployment to production than that is a big problem. Rails became visible with @DHH ‘s 15 minute blog video. It became visible and popular because that was so easy to accomplish. But without an easy production environment it’s worth not much.

Active Deployment

To be honest I don’t care which technology we use on the server. I don’t care how we all deploy. What I care about is that it has to be an easy to use system which comes out of the box. No more deployment snow flakes. Convention over configuration! I want to dive into an exisiting Rails project knowing that it probably uses the standard tool for deployment.

I’m calling out for DHH or any body else in the Rails community to think about this. Take a step back and realize how hard this problem is for a normal Ruby team which does not have super hero DevOps power.

I don’t have the technical knowledge to find an optimal solution for this problem. I just see the pain and it hurts me that many potential new Rails projects never get a chance just because they don’t have an easy way to deploy it to a non-Heroku environment.

Please discuss and share this.

I needed a method which I can call with the existing URL and it returns the new URL or nil in case that URL doesn’t work any more. I spare you the details but this is not as easily done as said. Redirections have to be handled properly over multiple steps. I tried a lot of different tools and gems for this job. After testing a couple of stackoverflow solutions I stumbled upon the curb gem (https://github.com/taf2/curb) which uses libcurl.

With that I was able to solve the problem with just a few lines of code:

def checked_url(url)
  begin
    result = Curl::Easy.perform(url) do |curl|
      curl.head = true
      curl.follow_location = true
      curl.timeout = 3
    end
    result.last_effective_url
  rescue
    nil
  end
end

And here is the test. Let’s assume I want to check the url nyt.com

irb(main):042:0> checked_url(‘nyt.com’)
=> “https://www.nytimes.com/"
irb(main):043:0>

As always: If you need Ruby on Rails consulting => https://www.wintermeyer-consulting.de and follow me on https://twitter.com/wintermeyer

Background: Often I use Devise as the one-stop-shop solution. But lately I found myself (again) in a Rails application trying to find out how to customize Devise for the specific needs of that application. Just to realize that I would have saved so much time by just implementing the authentication from scratch by myself.

The groundwork for tutorial was done by Ryan Bates many years ago.

Important: This tutorial works under the assumption that the session encryption of Rails is secure. If you disagree with that assumption you are bright enough to add an additional random token.

Green Field

We start with a fresh Rails application:

$ rails new shop
$ cd shop

Later we are going to redirect to root. So we start with creating an empty root page:

$ rails g controller home index

Please add the following code to config/routes.rb :

Rails.application.routes.draw do
  root ‘home#index’
end

And some content for that page in the file app/views/home/index.html.erb :

<p id=”notice”><%= notice %></p>

<h1>Example</h1>
<p>Lorem ipsum …</p>

If you like this post I’d like to ask you for a favour:
Create an account at my open-source business network https://www.vutuv.de

Thank you and see you there!

Password Digest

Obviously we do not store the clear text password in the database but a digest of it. For that we need to activate the bcrypt gem in the file Gemfile:

# Use ActiveModel has_secure_password
gem ‘bcrypt’, ‘~> 3.1.7’

And run bundle afterwords:

$ bundle

User Model

Now we create a User scaffold. Feel free to add any additional fields you might need (e.g. first_name, last_name). I just use email:uniq to store the email address (and create an unique database index) and password:digest to create a password_digest field in the new users table.

$ rails g scaffold User email:uniq password:digest
$ rails db:migrate

The digest part puts some Rails magic into action. The Rails generator creates a password_digest field in the table and asks for an additional password_confirmation in the form and the controllers user_params without you having to do anything extra. has_secure_password in the model takes care of encrypting the password and provides theauthenticate method to authenticate with that password.

Before a first test we need to add some validations in app/models/user.rb to make sure that we have an email address and that it is unique:

class User < ApplicationRecord
  has_secure_password

  validates :email, presence: true, uniqueness: true
end

Now we can fire up Rails and create a new user in the browser:

$ rails s

Let’s just check how Rails stores the password digest in the table:

$ rails c
Running via Spring preloader in process 3204
Loading development environment (Rails 5.2.1)
>> User.first
User Load (0.2ms) SELECT “users”.* FROM “users” ORDER BY “users”.”id” ASC LIMIT ? [[“LIMIT”, 1]]
=> #<User id: 1, email: “sw@wintermeyer-consulting.de”, password_digest: “$2a$10$t6Q2R.N5fevFjhL/W1X.EulEJQ8TDWIzCvHpbDrAtQo…”, created_at: “2018–09–18 12:13:26”, updated_at: “2018–09–18 12:13:26”>

So only the digest is saved. Everything is secure. But we still need to create some sort of login to actually use it.

Sessions

When a user logs in he/she creates a new session. When the same user logs out that session gets destroyed. Therefor we create a sessions controller with three actions:

$ rails g controller sessions new create destroy

We put the following code into app/controllers/sessions_controller.rb:

class SessionsController < ApplicationController
  def new
  end

  def create
    user = User.find_by_email(params[:email])
    if user && user.authenticate(params[:password])
      session[:user_id] = user.id
      redirect_to root_url, notice: "Logged in!"
    else
      flash.now[:alert] = "Email or password is invalid"
      render "new"
    end
  end

  def destroy
    session[:user_id] = nil
    redirect_to root_url, notice: "Logged out!"
  end

end

As you can see we use session[:user_id] to store the logged in user id. In case you haven’t worked with sessions yet have a look at https://guides.rubyonrails.org/security.html#sessions

We need to put this code for the form in app/views/sessions/new.html.erb :

<p id=”alert”><%= alert %></p>

<h1>Login</h1>

<%= form_tag sessions_path do |form| %>
  <div class=”field”>
    <%= label_tag :email %>
    <%= text_field_tag :email %>
  </div>

  <div class=”field”>
    <%= label_tag :password %>
    <%= password_field_tag :password %>
  </div>

  <div class=”actions”>
    <%= submit_tag “Login” %>
  </div>
<% end %>

Routes

The routes are a bit cumbersome. But we can fix this with this code in config/routes.rb :

Rails.application.routes.draw do
  root ‘home#index’
  
  resources :users
  resources :sessions, only: [:new, :create, :destroy]

  get ‘signup’, to: ‘users#new’, as: ‘signup’
  get ‘login’, to: ‘sessions#new’, as: ‘login’
  get ‘logout’, to: ‘sessions#destroy’, as: ‘logout’
end

Now a user can use http://localhost:3000/login to login and http://localhost:3000/logout to logout. Much easier for everybody.

current_user

In most Rails applications the logged in user is available with a current_user helper. This come handy too if you want to use an authorization gem like cancancan. The most popular way to add this functionality is this code in app/controllers/application_controller.rb:

class ApplicationController < ActionController::Base
  helper_method :current_user

  def current_user
    if session[:user_id]
      @current_user ||= User.find(session[:user_id])
    else
      @current_user = nil
    end
  end
end

To put it into use we change the content of app/views/home/index.html.erb:

<% if current_user %>
  Logged in as <%= current_user.email %>.
  <%= link_to “Log Out”, logout_path %>
<% else %>
  <%= link_to “Sign Up”, signup_path %> or 
  <%= link_to “Log In”, login_path %>
<% end %>

<p id=”notice”><%= notice %></p>

<h1>Example</h1>
<p>Lorem ipsum …</p>

The End

And here is the screencast where I log in with my account and log out afterwards:

Shameless Plug

Do you want to learn more about Rails 5.2? Buy my book at https://www.amazon.com/Learn-Rails-5-2-Accelerated-Development/dp/148423488X

In case you need consulting or training:
https://www.wintermeyer-consulting.de

Contact Information

Email: sw@wintermeyer-consulting.de
Twitter: https://twitter.com/wintermeyer

Homepage: https://www.wintermeyer-consulting.de
Email: sw@wintermeyer-consulting.de
Twitter: @wintermeyer

Abstract

With the caching of web applications, most people tend to wait to implement it until they encounter performance problems. First the admin usually looks at the database and adds an index here and there. If that does not help, the admin then takes a look at the views and adds fragment caching. But this is not the best approach for working with caches.

The aim of this chapter is to help you understand how key-based cache expiration works. You can then use this approach to plan new applications already on the database structure level in such a way that you can cache optimally during development.

With the caching of web applications, most people tend to wait to implement it until they encounter performance problems. First the admin usually looks at the database and adds an index here and there.

If that does not help, the admin then takes a look at the views and adds fragment caching. But this is not the best approach for working with caches. The aim of this chapter is to help you understand how key-based cache expiration works. You can then use this approach to plan new applications already on the database structure level in such a way that you can cache optimally during development.

There are two main arguments for using caching:

• The application becomes faster for the user. A faster web page
  results in happier users, which results in a better conversion rate.
• You need less hardware for the web server because you require less
  CPU and RAM resources for processing the queries.

If these two arguments are irrelevant for you, then there’s no need to
read this chapter.

I will cover three caching methods:

• HTTP caching: This is the sledgehammer among the caching methods and
  the ultimate performance weapon. In particular, web pages that are
  intended for mobile devices should try to make the most of HTTP
  caching. If you use a combination of key-based cache expiration and
  HTTP caching, you save a huge amount of processing time on the server
  and also bandwidth.
• Page caching: This is the screwdriver among the caching methods. You
  can get a lot of performance out of the system, but it is not as good
  as HTTP caching.
• Fragment caching: This is the tweezers among the caching methods, so
  to speak. But do not underestimate it!

The aim is to optimally combine all three methods.

The Example Application

You will use a simple phone book with a company model and an employees
model.

Create the new Rails app, as shown here:

$ rails new phone_book
 […]
$ cd phone_book
$ rails generate scaffold company name
 […]
$ rails generate scaffold employee company:references \
 last_name first_name phone_number
 […]
$ rails db:migrate
 […]

Models

Listing [3]14–1 and Listing [4]14–2 show the setup for the two models.

class Company < ApplicationRecord
 validates :name,
 presence: true,
 uniqueness: true
 has_many :employees, dependent: :destroy
 def to_s
 name
 end
end

Listing 14–1 app/models/company.rb

class Employee < ApplicationRecord
 belongs_to :company, touch: true
 validates :first_name,
 presence: true
 validates :last_name,
 presence: true
 validates :company,
 presence: true
 def to_s
 “#{first_name} #{last_name}”
 end
end

Listing 14–2 app/models/employee.rb

Views

Go ahead and change the two company views, shown in Listing 14–3 and
Listing 14–4, to list the number of employees in the index view and
all the employees in the show view.

[…]
<table>
 <thead>
 <tr>
 <th>Name</th>
 <th>Number of employees</th>
 <th colspan=”3"></th>
 </tr>
 </thead>
 <tbody>
 <% @companies.each do |company| %>
 <tr>
 <td><%= company.name %></td>
 <td><%= company.employees.count %></td>
 […]
 </tr>
 <% end %>
 </tbody>
</table>
[…]

Listing 14–3 app/views/companies/index.html.erb

<p id=”notice”><%= notice %></p>
<p>
 <strong>Name:</strong>
 <%= @company.name %>
</p>
<% if @company.employees.any? %>
<h1>Employees</h1>
<table>
 <thead>
 <tr>
 <th>Last name</th>
 <th>First name</th>
 <th>Phone number</th>
 </tr>
 </thead>
 <tbody>
 <% @company.employees.each do |employee| %>
 <tr>
 <td><%= employee.last_name %></td>
 <td><%= employee.first_name %></td>
 <td><%= employee.phone_number %></td>
 </tr>
 <% end %>
 </tbody>
</table>
<% end %>
<%= link_to ‘Edit’, edit_company_path(@company) %> |
<%= link_to ‘Back’, companies_path %>

Listing 14–4 app/views/companies/show.html.erb

If you like this post I’d like to ask you for a favour:
Create an account at my open-source business network https://www.vutuv.de

Thank you and see you there!

Example Data

To easily populate the database, you can use the Faker gem (see
http://faker.rubyforge.org/). With Faker, you can generate random
names and phone numbers. Please add the line shown in Listing 14–5
in the Gemfile.

[…]
gem ‘faker’
[…]

Listing 14–5 Gemfile

Then start bundle, as shown here:

$ bundle

With db/seeds.rb, you can create 30 companies with a random number of
employees in each case, as shown in Listing 14–6.

30.times do
 company = Company.new(:name => Faker::Company.name)
 if company.save
 SecureRandom.random_number(100).times do
 company.employees.create(
 first_name: Faker::Name.first_name,
 last_name: Faker::Name.last_name,
 phone_number: Faker::PhoneNumber.phone_number
 )
 end
 end
end

Listing 14–6 db/seeds.rb

You can populate it via rails db:seed.

$ rails db:seed

You can start the application with rails server and retrieve the
example data with a web browser by going to the URL
http://localhost:3000/companies or http://localhost:3000/companies/1.

Normal Speed of the Pages to Optimize

In this chapter, you will optimize the example web pages. Start the
Rails application in development mode with rails server. (The relevant
time values, of course, depend on the hardware you are using.)

$ rails server

To access the web pages, use the command-line tool curl
(http://curl.haxx.se/). Of course, you can also access the web
pages with other web browsers. You can look at the time shown in the
Rails log for creating the page. In reality, you need to add the time
it takes for the page to be delivered to the web browser.

List of All Companies (Index View)

At the URL http://localhost:3000/companies, the user can see a list of
all the saved companies with the relevant number of employees.

Generating the page takes 89ms on my machine.

Completed 200 OK in 89ms (Views: 79.0ms | ActiveRecord: 9.6ms)

Detailed View of a Single Company (Show View)

At the URL http://localhost:3000/companies/1, the user can see the
details of the first company with all the employees.

Generating the page takes 51ms on my machine.

Completed 200 OK in 51ms (Views: 48.9ms | ActiveRecord: 0.9ms)

HTTP Caching

HTTP caching attempts to reuse already loaded web pages or files. For
example, if you visit a web page such as www.nytimes.com or
www.wired.com several times a day to read the latest news, then
certain elements of that page (for example, the logo at the top of the
page) will not be loaded again from the server on your second visit.
Your browser already has these files in the local cache, which saves
the loading time and bandwidth.

Within the Rails framework, your aim is to answer the question “Has a
page changed?” in the controller. Normally, most of the time is spent
on rendering the page in the view. I’d like to repeat that: most of the
time is spent on rendering the page in the view!

Last-Modified

The web browser knows when it has downloaded a resource (e.g., a web
page) and then placed it into its cache. On a second request, it can
pass this information to the web server in an If-Modified-Since:
header. The web server can then compare this information to the
corresponding file and either deliver a newer version or return an HTTP
304 Not Modified code as response. In the case of a 304, the web
browser delivers the locally cached version. Now you are going to say,
“That’s all very well for images, but it won’t help me at all for
dynamically generated web pages such as the index view of the
companies.” However, you are underestimating the power of Rails.
[Para, Type = Important, ID = Par41]

Please modify the times used in the examples in accordance with your own
circumstances.

Go ahead and edit the show method in the controller file
app/controllers/companies_controller.rb, as shown in Listing 14–7.

# GET /companies/1
# GET /companies/1.json
def show
 fresh_when last_modified: @company.updated_at
end

Listing 14–7 app/controllers/companies_controller.rb

After restarting the Rails application, take a look at the HTTP header
of http://localhost:3000/companies/1, as shown here:

$ curl -I http://localhost:3000/companies/1
HTTP/1.1 200 OK
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Last-Modified: Sat, 27 Jan 2018 18:38:05 GMT
[…]

The Last-Modified entry in the HTTP header was generated by fresh_when
in the controller. If you later go to the same web page and specify
this time as well, then you do not get the web page back; you get a 304
Not Modified message, as shown here:

$ curl -I http://localhost:3000/companies/1 — header
‘If-Modified-Since: Sat, 27 Jan 2018 18:38:05 GMT’
HTTP/1.1 304 Not Modified
[…]

In the Rails log, you will find this:

Started HEAD “/companies/1” for 127.0.0.1 at 2018–01–27 18:24:21 +0100
Processing by CompaniesController#show as */*
 Parameters: {“id”=>”1"}
 Company Load (0.1ms) SELECT “companies”.* FROM “companies” WHERE
“companies”.”id” = ? LIMIT ? [[“id”, 1], [“LIMIT”, 1]]
Completed 304 Not Modified in 2ms (ActiveRecord: 0.1ms)

It took Rails 2ms on my machine to answer this request, compared to the
51ms of the standard variation. This is much faster! So, you have used
fewer resources on the server and saved a massive amount of bandwidth.
The user will be able to see the page much more quickly.

etag

Sometimes the update_at field of a particular object is not meaningful
on its own. For example, if you have a web page where users can log in
and this page then generates web page contents based on a role model,
it can happen that user A as the admin is able to see an Edit link that is not displayed to user B as a normal user. In such a scenario, the Last-Modified header explained earlier does not help. Actually, it would do harm.

In these cases, you can use the etag header. The etag is generated by the web server and delivered when the web page is first visited. If the user visits the same URL again, the browser can then check whether the corresponding web page has changed by sending an If-None-Match: query to the web server.

Please edit the index and show methods in the controller file
app/controllers/companies_controller.rb, as shown in Listing 14–8.

# GET /companies
# GET /companies.json
def index
 @companies = Company.all
 fresh_when etag: @companies
end
# GET /companies/1
# GET /companies/1.json
def show
 fresh_when etag: @company
end

Listing 14–8 app/controllers/companies_controller.rb

A special Rails feature comes into play for the etag: Rails automatically sets a new CSRF token for each new visitor of the web site. This prevents cross-site request forgery attacks (see http://wikipedia.org/wiki/Cross_site_request_forgery). But it also means that each new user of a web page gets a new etag for the same page. To ensure that the same users also get identical CSRF tokens, these are stored in a cookie by the web browser and consequently sent back to the web server every time the web page is visited. You have to tell curl that you want to save all cookies in a file and transmit these cookies later if a request is received.

For saving, you use the -c cookies.txt parameter.

$ curl -I http://localhost:3000/companies -c cookies.txt
HTTP/1.1 200 OK
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
ETag: W/”53830a75ef520df8ad8e1894cf1e5003"
 […]

With the parameter -b cookies.txt, curl sends these cookies to the web server when a request arrives. Now you get the same etag for two subsequent requests.

$ curl -I http://localhost:3000/companies -b cookies.txt
HTTP/1.1 200 OK
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
ETag: W/”53830a75ef520df8ad8e1894cf1e5003"
[…]
$ curl -I http://localhost:3000/companies -b cookies.txt
HTTP/1.1 200 OK
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block
X-Content-Type-Options: nosniff
ETag: W/”53830a75ef520df8ad8e1894cf1e5003"
[…]

You now use this etag to find out in the request with If-None-Match if
the version you have cached is still up-to-date.

$ curl -I http://localhost:3000/companies -b cookies.txt — header
‘If-None-Match: W/”53830a75ef520df8ad8e1894cf1e5003"’
HTTP/1.1 304 Not Modified
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
ETag: W/”53830a75ef520df8ad8e1894cf1e5003"
[…]

You get a 304 Not Modified in response. Let’s look at the Rails log.

Started HEAD “/companies” for 127.0.0.1 at 2018–01–27 18:36:25 +0100
Processing by CompaniesController#index as */*
 (0.2ms) SELECT COUNT(*) AS “size”, MAX(“companies”.”updated_at”) AS
timestamp FROM “companies”
Completed 304 Not Modified in 24ms (ActiveRecord: 0.2ms)

Rails took only 24ms on my machine to process the request. Plus, you have saved bandwidth again. The user will be happy with the speedy web application.

Find more generic information about etag headers at
https://en.wikipedia.org/wiki/HTTP_ETag.

current_user and Other Potential Parameters

As the basis for generating an etag, you can pass not just an object but also an array of objects. This way, you can solve the problem with the logged-in user who might get different content than a non-logged-in user. Let’s assume that a logged-in user is output with the method current_user.

You have to add etag { current_user.try :id } in app/controllers/application_controller.rb to make sure that all etags
in the application include the current_user.id value, which is nil if nobody is logged in, as shown in Listing 14–9.

class ApplicationController < ActionController::Base
 etag { current_user.try :id }
end

Listing 14–9 app/controllers/application_controller.rb

You can chain other objects in this array too and use this approach to
define when a page has not changed.

The Magic of touch

What happens if an employee is edited or deleted? Then the show view
and potentially the index view would have to change as well. That is
the reason for the following line in the employee model:

belongs_to :company, touch: true

Every time an object of the class Employee is saved in edited form and
if touch: true is used, ActiveRecord updates the superordinate Company
element in the database. The updated_at field is set to the current
time. In other words, it is “touched.”

This approach ensures that the correct content is delivered.

stale?

Up to now, I was assuming that only HTML pages are being delivered. So,
I showed how to use fresh_when and then do without the respond_to do
|format| block. But HTTP caching is not limited to HTML pages. What if
you want to render JSON, for example, as well and want to deliver it
via HTTP caching? You need to use the method stale?. Using stale?
resembles using the method fresh_when. Here’s an example:

def show
 if stale? @company
 respond_to do |format|
 format.html
 format.json { render json: @company }
 end
 end
end

Using Proxies (public)

I have also been assuming you were using a cache on the web browser.
But on the Internet, there are many proxies that are often closer to
the user and can therefore be useful for caching in the case of
nonpersonalized pages. If the example is a publicly accessible phone
book, then you can activate the free services of the proxies with the
parameter public: true in fresh_when or with stale?.

Here’s an example:

# GET /companies/1
# GET /companies/1.json
def show
 fresh_when @company, public: true
end

You can go to the web page and get the output, as shown here:

$ curl -I http://localhost:3000/companies/1
HTTP/1.1 200 OK
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
ETag: W/”f37a06dbe0ee1b4a2aee85c1c326b737"
Last-Modified: Sat, 27 Jan 2018 17:16:53 GMT
Content-Type: text/html; charset=utf-8
Cache-Control: public
[…]

The header Cache-Control: public tells all proxies that they can also cache this web page.

Using proxies always has to be done with great caution. On the one hand, they are brilliantly suited for delivering your own web page quickly to more
users, but on the other hand, you have to be absolutely sure that no personalized pages are cached on public proxies. For example, CSRF tags
and flash messages should never end up in a public proxy. For CSRF tags, it is a good idea to make the output of csrf_meta_tag in the default app/views/layouts/application.html.erb layout dependent on the
question of whether the page may be cached publicly, as shown here:

<%= csrf_meta_tag unless response.cache_control[:public] %>

Cache-Control with Time Limit

When using etag and Last-Modified, you can assume that the web browser
definitely checks once more with the web server if the cached version of a web page is still current. This is a very safe approach.

But you can take the optimization one step further by predicting the future: if you am already sure when delivering the web page that this web page is not going to change in the next two minutes, hours, or days, then you can tell the web browser this directly. It then does not need to check back again within this specified period of time. This overhead savings has advantages, especially with mobile web browsers with relatively high latency. Plus, you save server load on the web server.

In the output of the HTTP header, you may already have noticed the corresponding line in the etag and Last-Modified examples, shown here:

Cache-Control: max-age=0, private, must-revalidate

The item must-revalidate tells the web browser that it should
definitely check back with the web server to see whether a web page has
changed in the meantime. The second parameter, private, means that only
the web browser is allowed to cache this page. Any proxies on the way are not permitted to cache this page.

If you decide for the phone book that the web page is going to stay unchanged for at least two minutes, then you can expand the code example by adding the method expires_in. The controller app/controllers/companies.rb will then contain the following code for the method show:

# GET /companies/1
# GET /companies/1.json
def show
 expires_in 2.minutes
 fresh_when @company, public: true
end

Now you get a different cache control information in response to a request.

$ curl -I http://localhost:3000/companies/1
HTTP/1.1 200 OK
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Date: Sat, 27 Jan 2018 17:58:56 GMT
ETag: W/”f37a06dbe0ee1b4a2aee85c1c326b737"
Last-Modified: Sat, 27 Jan 2018 17:16:53 GMT
Content-Type: text/html; charset=utf-8
Cache-Control: max-age=120, public
[…]

The two minutes are specified in seconds (max-age=120), and you no longer need must-revalidate. So, in the next 120 seconds, the web browser does not need to check back with the web server to see whether the content of this page has changed.

This mechanism is also used by the asset pipeline. Assets created there in the Production environment can be identified clearly by the checksum in the file name and can be cached for a long time both in the web browser and in public proxies. That’s why you have the following section in the Nginx configuration file:

location ^~ /assets/ {
 gzip_static on;
 expires max;
 add_header Cache-Control public;
}

Fragment Caching

With fragment caching, you can cache individual parts of a view. You can safely use it in combination with HTTP caching and page caching.
The advantages, once again, are a reduction of server load and faster web page generation, which means increased usability.

Please create a new example application (see “The Example Application”).

Enabling Fragment Caching in Development Mode

Fragment caching is by default disabled in the Development environment.
You can activate it with the command rails dev:cache, which touches the
file tmp/caching-dev.txt.

$ rails dev:cache
Development mode is now being cached.

To deactivate caching, run the same command again (this will delete the
file tmp/caching-dev.txt).

$ rails dev:cache
Development mode is no longer being cached.

In production mode, fragment caching is enabled by default.

Caching the Table of the Index View

On the page http://localhost:3000/companies, a computationally intensive table with all the companies is rendered. You can cache this table as a whole. To do so, you need to enclose the table in a <% cache(‘name_of_cache’) do %> … <% end %> block.

<% cache(‘name_of_cache’) do %>
[…]
<% end %>

Please edit the file app/views/companies/index.html.erb as shown in
Listing 14–10.

<h1>Companies</h1>
<% cache(‘table_of_all_companies’) do %>
<table>
 <thead>
 <tr>
 <th>Name</th>
 <th>Number of employees</th>
 <th colspan=”3"></th>
 </tr>
 </thead>
 <tbody>
 <% @companies.each do |company| %>
 <tr>
 <td><%= company.name %></td>
 <td><%= company.employees.count %></td>
 <td><%= link_to ‘Show’, company %></td>
 <td><%= link_to ‘Edit’, edit_company_path(company) %></td>
 <td><%= link_to ‘Destroy’, company, method: :delete, data: {
confirm:
 ‘Are you sure?’ } %></td>
 </tr>
 <% end %>
 </tbody>
</table>
<% end %>
<br />
<%= link_to ‘New Company’, new_company_path %>

Listing 14–10 app/views/companies/index.html.erb

Then you can start the Rails server with rails server and go to the URL
http://localhost:3000/companies.

The first time, a page that has a fragment cache is a little bit slower because the cache has to be written. The second time it is a lot of faster.

Deleting the Fragment Cache

With the method expire_fragment, you can clear specific fragment caches. Basically, you can build this idea into the model in the same way as shown in the section “Deleting Page Caches Automatically.”

The model file app/models/company.rb will look like Listing 14–11.

class Company < ActiveRecord::Base
 validates :name,
 presence: true,
 uniqueness: true
 has_many :employees, dependent: :destroy
 after_create :expire_cache
 after_update :expire_cache
 before_destroy :expire_cache
 def to_s
 name
 end
 def expire_cache
 ActionController::Base.new.expire_fragment(‘table_of_all_companies’
)
 end
end

Listing 14–11 app/models/company.rb

Because the number of employees also has an effect on this table, you also have to expand the file app/models/employees.rb accordingly, as shown in Listing 14–12.

class Employee < ActiveRecord::Base
 belongs_to :company, touch: true
 validates :first_name,
 presence: true
 validates :last_name,
 presence: true
 validates :company,
 presence: true
 after_create :expire_cache
 after_update :expire_cache
 before_destroy :expire_cache
 def to_s
 “#{first_name} #{last_name}”
 end
 def expire_cache
 ActionController::Base.new.expire_fragment(‘table_of_all_companies’
)
 end
end

Listing 14–12 app/models/employees.rb

Deleting specific fragment caches often involves a lot of effort in terms of programming. First, you often miss things; second, in big projects it’s not easy to keep track of all the different cache names. Often it is easier to automatically create names via the method cache_key. These then expire automatically in the cache.

Auto-expiring Caches

Managing fragment caching is rather complex with the naming convention
used in the section “Caching the Table of the Index View.” On the one hand, you can be sure that the cache does not have any superfluous ballast if you have programmed neatly, but on the other, it does not really matter. A cache is structured in such a way that it deletes old and no longer required elements on its own. If you use a mechanism that gives a fragment cache a unique name, as in the asset pipeline, then you do not need to go to the trouble of deleting fragment caches.

Rails has you covered. And it is pretty easy to do.

Let’s edit the index view in the file app/views/companies/index.html.erb, as shown in Listing 14–13.

<h1>Companies</h1>
<% cache(@companies) do %>
<table>
 <thead>
 <tr>
 <th>Name</th>
 <th>Number of employees</th>
 <th colspan=”3"></th>
 </tr>
 </thead>
 <tbody>
 <% @companies.each do |company| %>
 <tr>
 <td><%= company.name %></td>
 <td><%= company.employees.count %></td>
 <td><%= link_to ‘Show’, company %></td>
 <td><%= link_to ‘Edit’, edit_company_path(company) %></td>
 <td><%= link_to ‘Destroy’, company, method: :delete, data: {
confirm:
 ‘Are you sure?’ } %></td>
 </tr>
 <% end %>
 </tbody>
</table>
<% end %>
<br />
<%= link_to ‘New Company’, new_company_path %>

Listing 14–13 app/views/companies/index.html.erb

You ask Rails to generate a cache key for @companies and use it. If you want to see the name of that cache key in your log, you have to add config.action_controller.enable_fragment_cache_logging = true in the
file config/environments/development.rb.

There is no general answer to the question of how much detail you should use
fragment caching. Do some experimenting with it and then look in the log to see how long things take.

Russian Doll Caching

In the previous example, you created one fragment cache for the whole table of companies. If one company within that table changes, the whole table has to be re-rendered. Depending on the kind of data, that might take a lot of time.

The idea of Russian doll caching is that you cache not only the whole table but each row of the table too. So, when one row changes, just this row has to be rendered; all other rows can be fetched from the cache. When done well, this can save a lot of resources.

Please take a look at the updated example, as shown in Listing 14–14.

<h1>Companies</h1>
<% cache(@companies) do %>
<table>
 <thead>
 <tr>
 <th>Name</th>
 <th>Number of employees</th>
 <th colspan=”3"></th>
 </tr>
 </thead>
 <tbody>
 <% @companies.each do |company| %>
 <% cache(company) do %>
 <tr>
 <td><%= company.name %></td>
 <td><%= company.employees.count %></td>
 <td><%= link_to ‘Show’, company %></td>
 <td><%= link_to ‘Edit’, edit_company_path(company) %></td>
 <td><%= link_to ‘Destroy’, company, method: :delete, data: {
confirm: ‘Are you sure?’ } %></td>
 </tr>
 <% end %>
 <% end %>
 </tbody>
</table>
<% end %>
<br />
<%= link_to ‘New Company’, new_company_path %>

Listing 14–14 app/views/companies/index.html.erb

Change the Code in the View Results in an Expired Cache

Rails tracks an MD5 sum of the view you use. So if you change the file
(e.g., app/views/companies/index.html.erb), the MD5 changes, and all
the old caches will expire.

Cache Store

The cache store manages the stored fragment caches. If not configured
otherwise, this is the Rails MemoryStore. This cache store is good for developing but less suitable for a production system because it acts
independently for each Ruby on Rails process. So, if you have several
Ruby on Rails processes running in parallel in the production system,
each process holds its own MemoryStore.

MemCacheStore

Most production systems use memcached ([23]http://memcached.org/) as a
cache store. To enable memcached as a cache store in your production system, you need to add the line shown in Listing [24]14–15 in the file config/environments/production.rb.

config.cache_store = :mem_cache_store

Listing 14–15 config/environments/production.rb

The combination of appropriately used auto-expiring caches and memcached is an excellent recipe for a successful web page.

Other Cache Stores

In the official Rails documentation you will find a list of other cache
stores; see http://guides.rubyonrails.org/caching_with_rails.html#cache-stores.

Page Caching

Page caching was removed from the core of Rails 4.0, but it is still available as a gem, and it is powerful. To do page caching, you need a bit of knowledge to configure your web server (e.g., Nginx or Apache). Page caching is not for the faint-hearted.

With page caching, it’s all about placing a complete HTML page (in other words, the render result of a view) into a subdirectory of the public directory and having it delivered directly from there by the web server (for example, Nginx) whenever the web page is visited next. Additionally, you can also save a compressed .gz version of the HTML page there. A production web server will automatically deliver files under public itself and can also be configured so that any .gz files present are delivered directly.

In complex views, that may take 500ms or even more for rendering; the
amount of time you save is of course considerable. As a web page operator, you once more save valuable server resources and can service more visitors with the same hardware. The web page user profits from a faster delivery of the web page.

When programming your Rails application, please ensure that you also update this page or delete it! You will find a description of how to do this in the section “Deleting the Page Caches Automatically.” Otherwise, you will end up with an outdated cache later.

Please also ensure that page caching rejects all URL parameters by default. For example, if you try to go to: http://localhost:3000/companies?search=abc, this automatically becomes http://localhost:3000/companies. But that can easily be fixed with different route logic.

Please install a fresh example application (see the section “The Example Application”) and add the gem with the following line in Gemfile:

gem ‘actionpack-page_caching’

Now install it with the command bundle install.

$ bundle install
[…]

Lastly, you have to tell Rails where to store the cache files. Please add the line shown in Listing 14–16 in your config/application.rb file.

config.action_controller.page_cache_directory =
“#{Rails.root.to_s}/public/deploy”

Listing 14–16 config/application.rb

Activating Page Caching in Development Mode

First you need to go to the file config/environments/development.rb and
set the item config.action_controller.perform_caching to true, as shown in Listing 14–17.

config.action_controller.perform_caching = true

Listing 14–17 config/environments/development.rb

Otherwise, you cannot try page caching in development mode. In production mode, page caching is enabled by default.

Configure Your Web Server

Now you have to tell your web server (e.g., Nginx or Apache) that it should check the /public/deploy directory first before hitting the Rails application. You have to configure it so that it will deliver a .gz file if one is available.

There is no one perfect way of doing it. You have to find the best way of doing it in your environment on your own.

As a quick and dirty hack for development, you can set page_cache_directory
to public. Then your development system will deliver the cached page.

config.action_controller.page_cache_directory =
“#{Rails.root.to_s}/public”

Caching the Company Index and Show View

Enabling page caching happens in the controller. If you want to cache the show view for Company, you need to go to the controller
app/controllers/companies_controller.rb and enter the command caches_page :show at the top, as shown in Listing [28]14–18.

class CompaniesController < ApplicationController
 caches_page :show
[…]

Listing 14–18 app/controllers/companies_controller.rb

Before starting the application, the public directory looks like this:

public/
+ — 404.html
+ — 422.html
+ — 500.html
+ — apple-touch-icon-precomposed.png
+ — apple-touch-icon.png
+ — favicon.ico
+ — robots.txt

After starting the application with rails server and going to the URLs
http://localhost:3000/companies and http://localhost:3000/companies/1
via a web browser, it looks like this:

public
+ — 404.html
+ — 422.html
+ — 500.html
+ — apple-touch-icon-precomposed.png
+ — apple-touch-icon.png
+ — deploy
| + — companies
| + — 1.html
+ — favicon.ico
+ — robots.txt

The file public/deploy/companies/1.html has been created by page caching.

From now on, the web server will only deliver the cached versions when
these pages are accessed.

gz Versions

If you use page caching, you should also cache directly zipped .gz files. You can do this via the option :gzip => true or use a specific compression parameter as a symbol instead of true (for example,
:best_compression).

The controller app/controllers/companies_controller.rb will look like Listing 14–19 at the beginning.

class CompaniesController < ApplicationController
 caches_page :show, gzip: true
[…]

Listing 14–19 app/controllers/companies_controller.rb

This automatically saves a compressed version and an uncompressed
version of each page cache.

public
+ — 404.html
+ — 422.html
+ — 500.html
+ — apple-touch-icon-precomposed.png
+ — apple-touch-icon.png
+ — deploy
| + — companies
| + — 1.html
| + — 1.html.gz
+ — favicon.ico
+ — robots.txt

The File Extension .html

Rails saves the page accessed at http://localhost:3000/companies under the file name companies.html. So, the upstream web server will find and deliver this file if you go to http://localhost:3000/companies.html, but not if you try to go to http://localhost:3000/companies because the extension .html at the end of the URL is missing.

If you are using the Nginx server, the easiest way to do this is to adapt the try_files instruction in the Nginx configuration file as follows:

try_files $uri/index.html $uri $uri.html @unicorn;

Nginx then checks if a file with the extension .html of the currently
accessed URL exists.

Deleting Page Caches Automatically

As soon as the data used in the view changes, the saved cache files
have to be deleted. Otherwise, the cache would no longer be up-to-date.

According to the official Rails documentation, the solution for this problem is the class ActionController::Caching::Sweeper. But this approach, described at http://guides.rubyonrails.org/caching_with_rails.html#sweepers, has
a big disadvantage: it is limited to actions that happen within the controller. So, if an action is triggered via URL by the web browser, the corresponding cache is also changed or deleted. But if an object is deleted in the console, for example, the sweeper would not realize this. For that reason, I will show you an approach that does not use a sweeper but works directly in the model with ActiveRecord callbacks.

In the phone book application, you always need to delete the cache for http://localhost:3000/companies and http://localhost:3000/companies/company_id when editing a company. When editing an employee, you also have to delete the corresponding cache
for the relevant employee.

Models

You still need to fix the models so that the corresponding caches are deleted automatically as soon as an object is created, edited, or deleted, as shown in Listing 14–20 and Listing 14–21.

class Company < ActiveRecord::Base
 validates :name,
 presence: true,
 uniqueness: true
 has_many :employees, dependent: :destroy
 after_create :expire_cache
 after_update :expire_cache
 before_destroy :expire_cache
 def to_s
 name
 end
 def expire_cache
 ActionController::Base.expire_page(Rails.application.routes.url_hel
pers.company_path(self))
 ActionController::Base.expire_page(Rails.application.routes.url_hel
pers.companies_path)
 end
end

Listing 14–20 app/models/company.rb

class Employee < ActiveRecord::Base
 belongs_to :company, touch: true
 validates :first_name,
 presence: true
 validates :last_name,
 presence: true
 validates :company,
 presence: true
 after_create :expire_cache
 after_update :expire_cache
 before_destroy :expire_cache
 def to_s
 “#{first_name} #{last_name}”
 end
 def expire_cache
 ActionController::Base.expire_page(Rails.application.routes.url_hel
pers.employee_path(self))
 ActionController::Base.expire_page(Rails.application.routes.url_hel
pers.employees_path)
 self.company.expire_cache
 end
end

Listing 14–21 app/models/employee.rb

Preheating

Now that you have read your way through this chapter, here is a final tip: preheat your cache!

For example, if you have a web application in a company and you know that at 9 a.m. all employees are going to log in and then access this web application, then it’s a good idea to let your web server go through all those views a few hours in advance with a cron job. At night, your server is probably bored anyway.

Check out the behavior patterns of your users. With public web pages, this can be done, for example, via Google Analytics (www.google.com/analytics/). You will find that at certain times of the day, there is a lot more traffic going in. If you have a quiet phase prior to this, you can use it to warm up your cache.

The purpose of preheating is to save server resources and achieve better quality for the user because the web page is displayed more quickly.

Further Information

The best source of information on this topic is in the Rails
documentation at: http://guides.rubyonrails.org/caching_with_rails.html. There you can find additional information (e.g., low-level caching).

Caching in Ruby on Rails 5.2 was originally published in Ruby Inside on Medium, where people are continuing the conversation by highlighting and responding to this story.

Their argumentation is so plausible:
Every 404 loses incoming link juice (backlinks). Google will slap you hands. Obviously! Who in his right mind would use a 404 when a 301 would save all that beautiful link juice?!

For any non-technical and even for the many technical clients their argument is valid and makes sense. But let’s take a deeper look into this.

What is a 404 or 301?

Web server communicate with web browsers. One part of that communication is a status code. The most common one is 200 which says that the requested URL exists. If it doesn’t exist the web server answers with a 404 (Not Found Error Message) error message. So the web browser understands that this URL doesn’t exist. It can display an error message. But most times the web server will transfer an error HTML file itself. This can contain anything. It doesn’t even have to say “404” anywhere. It is for humans only!

Sometimes files move on the server. They get renamed. And because a URL is kind of a file name itself there is a way to tell a web browser that a URL has moved to a different URL. Maybe someone started a blog and used this format for a web page: /articles/2017–02–28-foobar.html. But after some time he/she decides to use a different URL: /2017/02/28/articles/foobar.html. This would be a perfect example to use a 301(Moved Permanently) which tells the web client that the URL has changed. So the web browser can use this information to fetch the content from the new URL. It’s like a forwarding request for your mail.

The SEO Anti 404 Argument

Let’s assume you sell apples and oranges in your online shop. A customer can get informations about apples and buy them at /products/apples. And a customer can get information about oranges and buy them at /products/oranges. One day you decide to not sell apples any more. Your SEO consultants says that countless webpages link to your /products/apples page. He can proof this to you with his fancy tools. He knows that Google calculates your search result position by that number of incoming links. So he wants you to not respond with a 404 but a 301 which redirects to /products/oranges. By that you can recycle the incoming apple link power for your oranges.

Great argumentation and logic! And thanks god nobody can argue against it. You can’t ask Google how they rate. Or can you? Actually you can and I did. I ask John Mueller who is Webmaster Trends Analyst at Google about exactly this problem:

Stefan Wintermeyer on Twitter

@JohnMu Assuming I sell apples and oranges online. Many people link to my /products/apples and /products/oranges. Now I stop selling apples. SEO consultants tell me to 301 on /products/apples but I feel like 404 or even 410. SEO c. tell me I'll lose energy/juice/whatever. Right?

He answered right away:

John ☆.o(≧▽≦)o.☆ on Twitter

@wintermeyer If people want apples, why would you show them oranges? If they're the same kind of product, then that (301) might make sense, but if people want something different, then I'd treat them as something different (404).

But a Antonio Sangio smelled deception and asked a follow up question:

Antonio Sangiao on Twitter

@JohnMu @wintermeyer But if @wintermeyer has 404 in his website, Google will penalize his website, right?

See! Antonio rubbed salt into the wound. Google will penalize this website. For sure. Obviously! But wait! Here’s John answer:

John ☆.o(≧▽≦)o.☆ on Twitter

@pasquinoweb @wintermeyer No, definitely not. Returning a 404 for a missing page is something a technically correct, high-quality website should do.

Wait a minute! So a 404 is correct and a misleading 301 is not?! Who would have thought of this!

John Mueller of Google says that you should return a correct 404 for /products/apples without the fear of being penalized! Google will not penalize correct behavior.

Did your SEO guy lie to you?

No, he probably didn’t lie. At least not by purpose. He just doesn’t understand his business. He follows the lead of all the other SEO consultants he knows. Like a lemming following his fellow lemmings.

SEO consultants love to use FUD (Fear, uncertainty and doubt). And the whole 301 vs. 404 argument is about your biggest fear: Losing Google search result ranking.

Let’s use our Brain

We know that Google counts the amount of incoming links to rank your webpage. So if e.g. 500 other webpages link you your /products/apples webpage it means that this page must be important. It must be an important resource for all things “apples”. By using a 301 to redirect them to /products/oranges you will water down the relevance of your orange page. Because people expect some sort of information about apples and not oranges. So by using a 301 you will not just not save any link juice but harm your existing ranking for oranges! Let that sink in for a moment.

By using a 404 you tell Google and any other search engine that this URL doesn’t exist any more. Google will still calculate the overall importance of your online shop high because of those 500 incoming links even when they result in a 404. But Google will send less “apple” requests to your page. Which makes sense since you stopped selling apples.

404 can still deliver a human readable web page!

So we established that you return a 404 for /products/apples because you stopped selling apples. But for those humans who click on that incoming link you can display a page which says “Sorry, we don’t sell apples any more but have a look at our great oranges! Just click on /products/oranges for a complete list.”

By that you don’t lose the human but you give Google the needed information.

What about 302?

In short: 301 means for ever and 302 means just for a while. But since we established the fact that you do not want to 301 any of your apple requests it doesn’t matter anyhow.

BTW: There is no “for ever” for a search engine. It will try less often but will never not try it at all.

Conclusion

It’s very easy: If you stop selling a product or offering a service of any sort than you want to deliver a 404 for that URL. Keep using 301 but only if a URL has changed. Not if it doesn’t exist any more.

And next time your SEO consultant aka Lemming brings up the “use 301 and not 404” fairy tale you’ll give him the URL of this article.

PS: Let me close with a link to my business homepage so that Google adds this to my ranking: Best SEO advise ever! SCNR. ;-)

Kudos to DHH and the core team for doing this. Most projects would have stick with the old way because it was kind of ok.

Once you installed Rails 5.2 each of your new Rails projects has an already good to go setup for using credentials. No more generating keys manually. The important master key is automatically generated and stored in the file config/master.key which can be shared with other developers in the team but which should never be checked into the Git repository. The default .gitignore has been updated accordingly:

# Ignore master key for decrypting credentials and more.
/config/master.key

Editing Credentials

All credentials are stored encrypted in the file config/credentials.yml.enc. Obviously you can not edit the file directly. You have to use the command rails credentials:editto edit them. For that to work you have to set the shell environment variable EDITOR first. Or you can do both with this one liner in your Bash shell:

$ EDITOR=vim rails credentials:edit

Now you can edit your credentials in yaml format. In this example I add a credential with the name foobar and the value test:

# aws:
# access_key_id: 123
# secret_access_key: 345

# Used as the base secret for all MessageVerifiers in Rails, including the one protecting cookies.
secret_key_base: 9846dad34a3168…68d634f
foobar: test

Accessing Credentials

You can access a credential anywhere in your application with AppName::Application.credentials.name_of_the_credential. An example from within the console:

$ rails console
Running via Spring preloader in process 19662
Loading production environment (Rails 5.2.0)
>> Shop::Application.credentials.foobar
=> “test”
>> exit

If you like this post I’d like to ask you for a favour:
Create an account at my open-source business network https://www.vutuv.de

Thank you and see you there!

Server

To use the credentials in production you have to copy the config/master.keyfile to your production environment or setting it up with an environment variable.

Screencast

I’m a big fan of screencasts too. So here it is:

More?

In case you need on site Ruby on Rails training:
Please send me an email to sw@wintermeyer-consulting.de

I’m currently working on my new Ruby on Rails 5.2 book. You can follow me on Twitter to get updates too: https://twitter.com/wintermeyer

In case you speak German: https://www.wintermeyer-consulting.de

I always use RVM to install Ruby. Why should you too?

• You simply do not have any root rights on the system. In that case,
  you have no other option.
• You want to run several Ruby or Rails versions that are separated cleanly. Maybe not today but for a future system upgrades it is always handy to have the old version still available in case you have to roll back. This can be easily done with RVM.

Ruby on Rails 5.2 on Debian 9.3 (Stretch)

This description assumes that you have a freshly installed Debian
GNU/Linux 9.3 (“Stretch”). You will find an ISO image for the
installation at http://www.debian.org. I recommend the approximately 250
MB net installation CD image. For instructions on how to install
Debian-GNU/Linux, please go to http://www.debian.org/distrib/netinst.

If you have root rights on the target system, you can use the following
commands to ensure that all required programs for a successful
installation of RVM are available. If you do not have root rights, you
have to either hope that your admin has already installed everything you
need, or send them a quick e-mail with the corresponding lines.

Login as root, update the package lists and upgrade the system:

root@debian:~# apt-get update
[..]
root@debian:~# apt-get upgrade

Installation of the packages required for the RVM installation:

root@debian:~# apt-get -y install curl gawk g++ \
make libreadline6-dev zlib1g-dev libssl-dev \
libyaml-dev libsqlite3-dev sqlite3 autoconf \
libgdbm-dev libncurses5-dev libtool bison nodejs \
pkg-config libffi-dev libgmp-dev libgmp-dev git \
dirmngr

Now is a good time to log out as root:

root@debian:~# exit
logout
xyz@debian:~$

Log in with your normal user account (in our case, it’s the user `xyz`) and run the following commands:

xyz@debian:~$ gpg — keyserver hkp://keys.gnupg.net --recv-keys \
409B6B1796C275462A1703113804BB82D39DC0E3 \
7D2BAF1CF37B13E2069D6956105BD0E739499BDB
[…]
xyz@debian:~$ curl -sSL https://get.rvm.io | bash
[…]

To be able to use RVM you need to run a script first.
rvm will tell you which script (it’s path depends on the user name):

xyz@debian:~$ source /home/xyz/.rvm/scripts/rvm

Now you can use RVM to install Ruby 2.5 and after that Rails 5.2 with gem:

xyz@debian:~$ rvm install 2.5
[…]
xyz@debian:~$ gem install rails
[…]

gem install rails installs the current stable Rails version. You can use the
format gem install rails -v 5.2.0 to install a specific version and gem install rails — pre to install a current beta version.

RVM, Ruby 2.5 and Rails 5.2 are now installed. You can check it
with the following commands:

xyz@debian:~$ ruby -v
ruby 2.5.0p0 (2017–12–25 revision 61468) [x86_64-linux]
xyz@debian:~$ rails -v
Rails 5.2.0
xyz@debian:~$

If you like this post I’d like to ask you for a favour:
Create an account at my open-source business network https://www.vutuv.de

Thank you and see you there!

Ruby on Rails 5.2 on macOS 10.13 (High Sierra)

macOS 10.13 includes Ruby by default. Not interesting for our purposes.
We want Ruby 2.5 and Rails 5.2. To avoid interfering with the existing
Ruby and Rails installation and therefore the packet management of Mac
OS X, we install Ruby 2.5 and Rails 5.2 with RVM (Ruby Version Manager).

Before you start installing Ruby on Rails, you must install the _latest_
Apple Xcode tools on your system. The easiest way is via the Mac App
Store (search for “xcode”) or via the website
https://developer.apple.com/xcode/

Please take care to install all the command line tools!

Log in with your normal user account (in our case, it’s the user `xyz`) and run the following commands:

$ gpg — keyserver hkp://keys.gnupg.net --recv-keys \
409B6B1796C275462A1703113804BB82D39DC0E3 \
7D2BAF1CF37B13E2069D6956105BD0E739499BDB
[…]
$ curl -sSL https://get.rvm.io | bash
[…]

rvm will tell you a source command which you can run to setup rvm
for your current shell/terminal. Most times it is just easier to close
the current shell and open a new terminal window. Than everything in
the new terminal is setup properly.

$ rvm install 2.5
[…]
$ gem install rails
[…]

gem install rails installs the current stable Rails version. You can use the
format gem install rails -v 5.2.0 to install a specific version and gem install rails — pre to install a current beta version.

RVM, Ruby 2.5 and Rails 5.2 are now fully installed. You can check it
with the following commands.

$ ruby -v
ruby 2.5.0p0 (2017–12–25 revision 61468) [x86_64-darwin17]
$ rails -v
Rails 5.2.0

Have fun with Rails 5.2!

The Build Environment

I assume you run a stable and up-to-date Debian Stretch (9.1) system. You must be root for this how-to. Probably you already have some of the following packages installed. But to be safe run the following commands:

apt-get install git
apt-get build-dep nginx

Install the Brotli packages

Stretch provides a stable Brotli package which we install first:

apt-get install brotli

But we need the development libraries too and those are only available in experimental Debian. That is still better than compiling them by ourself so we add this line to the /etc/apt/sources.list file:

deb http://deb.debian.org/debian experimental main

Now we can install it:

apt-get update
apt-get install libbrotli-dev/experimental

Once this package becomes stable we want to replace it with the stable version.

The ngx_brotli Modul

Next we have to compile the Nginx Brotli module:

cd /opt
git clone https://github.com/google/ngx_brotli
cd /opt/ngx_brotli
git submodule update —- init

Once there is an official Debian module for this we have to delete the directory /opt/ngx_brotli and install the official package.

Nginx Debian Package

We have to download the Nginx Debian package sources:

cd /usr/src
mkdir nginx
cd nginx
apt-get source nginx
cd nginx-1.10.3

The “1.10.3” part may vary depending on the time you are using this how-to. Please do an ls to get the right directory name.

Use your favourite editor to open the file debian/rules and search the part where extras are defined (nginx comes in a couple of different flavours in Debian we are going to use the extra version which has all the batteries included). Add this line there:

 —-add-module=/opt/ngx_brotli

It’s time to build all the Debian packages. That might take a moment:

cd /usr/src/nginx/nginx-1.10.3/
dpkg-buildpackage -b

Once the packages are build we can install them:

cd ..
dpkg -i nginx-common_*.deb libnginx-mod*_amd64.deb nginx-extras*_amd64.deb

Once Debian provides official Nginx packages which support Brotli you can easily replace those with them.

Configure Nginx

Open /etc/nginx/nginx.conf with you editor and add the following configuration in the http block of it:

brotli on;
brotli_comp_level 6;
brotli_static on;
brotli_types text/html text/plain text/css application/javascript application/x-javascript text/xml application/xml application/xml+rss text/javascript image/x-icon image/vnd.microsoft.icon image/bmp image/svg+xml;

Lastly you have to restart nginx:

service nginx restart

Now you are good to go. Your Nginx delivers pre-compressed files with the file extension .br (brotli_static on;) and compresses everything else on the fly.

Only with SSL

Chrome and Firefox only ask for Brotli compression for https:// connections. So be sure to configure SSL next. And while you are doing that add HTTP/2.

Shameless Plug

In case you need WebPerformance, Ruby on Rails or Phoenix Framework consulting or training please don’t hesitate to send me an email to sw@wintermeyer-consulting.de or a DM at https://twitter.com/wintermeyer