How to work with GitHub Actions in your rails App + Solr engine.

Some weeks ago, I worked on a new CI workflow in my current project, but I had a lot of issues with the Solr service when I tried to integrate all tests in the CI with GitHub Actions. In this post, I want to show you how to make a CI workflow in your Rails app with Solr engine search.

We will use the following versions:

• Rails 5.2.4
• Ruby 2.5.8
• Postgresql 12.3

Note: Use rbenv or rvm to manage your Ruby versions, and you can use Homebrew to install and run Postgresql.

We’re going to create our project:

$ rails _5.2.4_ new rails_solr_app -T -d postgresql
$ cd rails_solr_app
$ rails db:setup

Then we can run our server:

$ rails s

http://localhost:3000/ :

We’ll use the Rails scaffold to create a complete CRUD of users:

$ rails g scaffold User name:string
$ rails db:migrate

Add the following line to our routes ⇒ config/routes.rb:

root to: "users#index"

Refresh the browser:

We need to add some random data, so we can use random_name_generator gem(https://github.com/folkengine/random_name_generator), then add the following line into the Gemfile:

gem 'random_name_generator', '~> 1.2', '>= 1.2.1'

And run:

$ bundle install

Then we can add some validations on the user’s model:

class User < ApplicationRecord
  validates :name, presence: true, length: { minimum: 3 }
end

We can add some users, so we will use the seed generator of rails, we have to edit db/seeds.rb:

rng = RandomNameGenerator.new
50.times do |i|
  name = rng.compose(3)
  User.create(name: name)
end

Now run:

$ rails db:seed

If you open the rails console, you can see:

$ rails c

User.pluck(:name)

The next step is to create a search functionality, so we have to add some gems:

gem 'jquery-rails', '~> 4.4'
gem 'sunspot_rails', '~> 2.5'

In the development and test group:

gem 'sunspot_solr', '~> 2.5'

So, you can run:

$ bundle install
$ rails generate sunspot_rails:install

The above command generates a default configuration file(config/sunspot.yml):

production:
  solr:
    hostname: localhost
    port: 8983
    log_level: WARNING
    path: /solr/production
    # read_timeout: 2
    # open_timeout: 0.5
development:
  solr:
    hostname: localhost
    port: 8982
    log_level: INFO
    path: /solr/development
test:
  solr:
    hostname: localhost
    port: 8981
    log_level: WARNING
    path: /solr/test

Now, we are going to generate a Solr folder with default configuration files and indexes:

$ bundle exec rake sunspot:solr:start

And we will add the following line into .gitignore:

/solr/*

We will use jquery, so we have to add in app/assets/javascripts/application.js:

//= require jquery

Add a new route:

get "/search" => "users#search"

Modify views/users/index.html.erb:

<p id="notice"><%= notice %></p>
<h1> Search Users</h1>
<%= form_tag(search_path, :method => "get", :remote => true, class: 'search-users') do %>
  <%= label_tag(:name, "Search for User name:") %>
  <%= text_field_tag(:name) %>
  <%= submit_tag("Search") %>
<% end %>
<div class="users-results">
</div>
<h1>All Users</h1>
<table>
  <thead>
    <tr>
      <th>Name</th>
      <th colspan="3"></th>
    </tr>
  </thead>
<tbody>
    <% @users.each do |user| %>
      <tr>
        <td><%= user.name %></td>
        <td><%= link_to 'Show', user %></td>
        <td><%= link_to 'Edit', edit_user_path(user) %></td>
        <td><%= link_to 'Destroy', user, method: :delete, data: { confirm: 'Are you sure?' } %></td>
      </tr>
    <% end %>
  </tbody>
</table>
<br>
<%= link_to 'New User', new_user_path %>

Add a searchable block into User’s model:

searchable do
  # -- Solr search engine --
  text :name
end

Run:

$ bundle exec rake sunspot:solr:reindex

Note: Every time that we modify the searchable block, We have to run the above command.

Now, we will create a service to search users by name, create a new folder models/services and new file models/services/search_users.rb:

class Services::SearchUsers
  attr_reader   :params
  def initialize(params)
    @params = params
  end
  def search
    return [] unless params[:name].present?
    results = User.search do
      fulltext params[:name] do
        fields(:name)
      end
    end
    results.present? ? results.results : []
  end
end

And create a new partial to show all results(views/users/_search_results.json.erb):

<% if @users.present? %>
  <table>
    <thead>
      <tr>
        <th>Search Results</th>
        <th colspan="3"></th>
      </tr>
    </thead>
    <tbody>
      <% @users.each do |user| %>
        <tr>
          <td><%= user.name %></td>
          <td><%= link_to 'Show', user %></td>
        </tr>
      <% end %>
    </tbody>
  </table>
<% else %>
  <div>
    <p>No results</p>
  </div>
<% end %>

Then, we add search action in the user's controller:

def search
  @search_user = Services::SearchUsers.new(params)
  @users = @search_user.search
   respond_to do |format|
     format.json do
       render json: { data: render_to_string(partial: 'users/search_results', layout: false) }, status: 200
      end
    end
  end

Finally, we need to add a JS code to work with JSON response in the search action, so we have to add the following code in assets/javascripts/users.js.coffee (modify extension):

$(document).on "ajax:success", "form.search-users", (event) ->
  [data, status, xhr] = event.detail;
  $('.users-results').html(data.data);

Now, we are going to do some searches:

You can see the Solr query:

But, the default behavior on the Solr schema doesn’t allow us to get similar names, in fact, we have to write the exact name:

If we want to modify this behavior, we have to do some changes:

1. Modify the user’s model:

class User < ApplicationRecord
  validates :name, presence: true, length: { minimum: 3 }
  searchable do
    # -- Solr search engine --
    text :name
    text :name, :as => :name_ngram
  end
end

2. Add the following code into solr/configsets/sunspot/conf/schema.xml:

<fieldType name="text_ngram" class="solr.TextField" omitNorms="false">
  <analyzer>
    <tokenizer class="solr.WhitespaceTokenizerFactory"/>
      <filter class="solr.LowerCaseFilterFactory" />
      <filter class="solr.EdgeNGramFilterFactory" minGramSize="3" maxGramSize="6"/>
  </analyzer>
</fieldType>
<dynamicField name="*_ngram" stored="false" type="text_ngram" multiValued="true" indexed="true"/>

3. Run:

$ bundle exec rake sunspot:solr:restart
$ bundle exec rake sunspot:solr:reindex

Note: Every time we modify the schema file, we have to restart the Solr service

4. Now, we can check the correct behavior:

Tests with rspec

Now, we can create some tests for our application, especially the search service, so add in the Gemfile(Test group):

gem 'rspec-rails', '~> 4.0', '>= 4.0.1'
gem 'database_cleaner-active_record', '~> 1.8'

Run:

$ bundle install
$ rails generate rspec:install

The last command generates boilerplate configuration files:

create  .rspec
create  spec
create  spec/spec_helper.rb
create  spec/rails_helper.rb

Now, we will create some simple tests to validate the user’s model by creating a new folder spec/models and new file spec/models/user_spec.rb:

require 'rails_helper'
RSpec.describe 'User' do
  context 'User validations' do
    it 'Name can´t be blank' do
      expect{ User.create!(name: nil) }.to raise_error(ActiveRecord::RecordInvalid, "Validation failed: Name can't be blank, Name is too short (minimum is 3 characters)")
    end
    it 'Name must be more than 3 character' do
      expect{ User.create!(name: 'Ju') }.to raise_error(ActiveRecord::RecordInvalid, "Validation failed: Name is too short (minimum is 3 characters)")
    end
    it 'Name is correct' do
      expect{ User.create!(name: 'Juan') }.to_not raise_error
    end
  end
end

Then, run all spec files with:

$ bundle exec rspec

We have a failure because we didn’t start the solr service into test environment:

bundle exec rake sunspot:solr:start RAILS_ENV=test

http://localhost:8981/solr/#/:

And run again all spec files:

$ bundle exec rspec

The next step is to create some tests for our search service:

We need to use database_cleaner gem, by adding in spec/spec_helper.rb:

require 'database_cleaner/active_record'
RSpec.configure do |config|
  DatabaseCleaner.strategy = :truncation
  .
  .
  .
end

Create new folder spec/models/services and new file spec/models/services/search_users_spec.rb:

require 'spec_helper'
RSpec.describe "Services::SearchUsers" do
before(:all) do
    DatabaseCleaner.clean
    user_1 = User.create!(name: 'Juan David')
    user_2 = User.create!(name: 'Juan Antonio')
User.solr_reindex
  end
describe 'Search Services' do
it 'Should not return any user' do
      subject = Services::SearchUsers.new({ name: "Brad" }).search
      expect(subject.count).to  eq(0)
    end
it 'Should be return two user' do
      subject = Services::SearchUsers.new({ name: "jua" }).search
      expect(subject.count).to  eq(2)
    end
  end
end

Run:

$ bundle exec rspec

At this point, we have some tests to check into our CI, so we can generate our first commit.

Setting up Continuous Integration

Continuous Integration (CI) is the process of testing our code every time that we generate changes and these changes are pushed to version control(Github), Github actions is a feature that allows you to define some workflows to run tests.

We need to create our CI file, so we have some steps:

1. Define the name of our workflow => CI
2. Define which events we want to run => [push, pull_request]
3. Define a list of jobs => test
4. Choose operating system => ubuntu
5. Define services => postgres
6. Define steps that we want our workflow to run: Clone our repository(actions/checkout@v1), Set up Ruby, Solr container(5.3.1), Build and run all test

So, we have to create some folders and one file in the main root of our project => .github/workflows/ci.yml:

name: CI
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    services:
      db:
        image: postgres:11@sha256:85d79cba2d4942dad7c99f84ec389a5b9cc84fb07a3dcd3aff0fb06948cdc03b
        ports: ["5432:5432"]
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
    steps:
      - uses: actions/checkout@v1
      - name: Set up Ruby
        uses: actions/setup-ruby@v1
        with:
          ruby-version: '2.5.8'
 - name: Create Solr container
        run: |
          docker run -d --name collabra-solr -p 8981:8983 solr:5.3.1
 - name: Build and run tests
        env:
          DATABASE_URL: postgres://postgres:@localhost:5432/test
          POSTGRES_PASSWORD: postgres
          RAILS_ENV: test
        run: |
          sudo apt-get -yqq install libpq-dev
          gem install bundler:2.1.4
          bundle install --jobs 4 --retry 3
          curl -o - 'http://localhost:8981/solr/admin/cores?action=CREATE&name=test&configSet=basic_configs'
          bundle exec rake db:create
          bundle exec rake db:migrate
          bundle exec rspec spec

We can make a new commit and push our app into a GitHub repository, later go to actions tab:

Ups!!! , we have some errors in our workflow:

The main reason is our custom schema configuration, so we have to work on it, with our Solr image:

• Stop local Solr service:

bundle exec rake sunspot:solr:stop RAILS_ENV=test

• Create an account in Docker(https://hub.docker.com/signup) and install docker on your local computer.
• Create an empty folder in your local computer for example my-solr-image.
• Open a new terminal, and go to my-solr-image folder. We are going to copy the solr configuration(5.3.1) in our folder, with the followings commands:

$ docker create --rm --name copier solr:5.3.1 
$ docker cp copier:/opt/solr/server/solr/configsets/basic_configs custom-config 
$ docker rm copier

• Now, we have the default Solr(5.3.1) config:

• We have to create a new file called Dockerfile(root folder) with the following code(add your information):

FROM solr:5.3.1
MAINTAINER JUAN GAVIRIA XXX@XXX.com
USER root
RUN mkdir -p /opt/solr/server/solr/configsets/myconfig
WORKDIR /opt/solr/server/solr/configsets/myconfig
COPY ./custom-config .
WORKDIR /

With the above code, we create a new image based on solr:5.3.1 but with our configuration, then run:

$ docker build -t my-solr-image .

• Actually, we have my-solr-image image:

$ docker images

• We are going to create a container based on our image, in the port 8981(according to rails_solr_app/config/sunspot.yml):

$ docker run -d --name test-image -p 8981:8983 my-solr-image
$ docker ps

The second command is to check the container status:

If we open http://localhost:8981/solr/#/, we have our container running but without cores:

• The next step is to create a core called test, with our custom configuration in the folder myconfig, just open the following link:

http://localhost:8981/solr/admin/cores?action=CREATE&name=test&configSet=myconfig

You can see the custom field:

• Now, we can run all tests:

$ bundle exec rspec

• Later, we have to publish our image(mrjuangaviria ⇒ change for your user in Docker platform):

$ docker build -t mrjuangaviria/my-solr-image:1.0 . 
$ docker push mrjuangaviria/my-solr-image:1.0

• We can use this image into CI Github actions(ci.yml), so change these two lines:

docker run -d --name collabra-solr -p 8981:8983 mrjuangaviria/my-solr-image:1.0
curl -o - 'http://localhost:8981/solr/admin/cores?action=CREATE&name=test&configSet=myconfig'

• Finally, we are going to commit and push this change:

You can see the final code here !!!!

I hope this post has been useful, thank you for reading!!!

References

• https://lucene.apache.org/solr/
• https://github.com/sunspot/sunspot
• https://lucene.apache.org/solr/guide/6_6/documents-fields-and-schema-design.html
• https://github.com/rspec/rspec-rails
• https://github.com/DatabaseCleaner/database_cleaner