Using batch processing to improve performance when working with large datasets in Ruby on Rails.

Using batch processing to improve performance when working with large datasets in Ruby on Rails

As a Ruby on Rails developer, it is not uncommon to encounter situations where you need to export data from a database table to a CSV file, and then upload that file to Amazon S3. However, when dealing with large datasets, this process can become slow and resource-intensive, which can cause problems such as timeouts and memory issues.

In this article, we will discuss a solution to this problem that involves streaming data from the database and uploading the fetched data to Amazon S3 using upload_stream.

Whta is the Problem?

Let’s say you have a database table with a large number of rows, and you need to export that data to a CSV file and upload it to Amazon S3. Here is a sample code that exports data to a CSV file

require 'csv'

csv_file = CSV.generate do |csv|
  csv << ['id', 'name', 'email']
  User.all.each do |user|
    csv << [user.id, user.name, user.email]
  end
end

File.write('users.csv', csv_file)

This code fetches all the rows from the users table, generates a CSV file in memory, and then writes that file to disk. However, when dealing with a large number of rows, this approach can quickly become slow and resource-intensive.

Solution

To solve this problem, we can stream data from the database and upload the fetched data to Amazon S3 using upload_stream. Here is a sample code that implements this solution

require 'csv'
require 'aws-sdk-s3'

s3 = Aws::S3::Resource.new(region: 'us-east-1')
bucket = s3.bucket('bucket-name')

header = ['id', 'name', 'email']

CSV.open('users.csv', 'w', write_headers: true, headers: header) do |csv|
  User.find_each(batch_size: 1000) do |user|
    csv << [user.id, user.name, user.email]
  end
end

File.open('users.csv', 'rb') do |file|
  object = bucket.object('users.csv')
  object.upload_stream do |write_stream|
    write_stream.write(file.read)
  end
end

This code streams data from the users table using find_each with a batch_size of 1000. This approach fetches data in batches, which helps prevent memory issues and reduces the load on the database. It then generates a CSV file in memory using CSV.open with the write_headers and headers options to add a header row to the file.

After the CSV file is generated, it is written to disk and then uploaded to Amazon S3 using upload_stream. This method streams the file contents directly to S3, which helps prevent memory issues and reduces the load on the server.

Conclusion

In this article, we discussed a solution to a common problem faced by Ruby on Rails developers when exporting large datasets to a CSV file and uploading that file to Amazon S3. By streaming data from the database and uploading the fetched data to S3 using upload_stream, we can improve the performance and scalability of our applications.

If you are considering developing a Ruby on Rails project, you may want to consider working with Xprolabs(https://xprolabs.com). Xprolabs is a leading software development company with extensive experience in Ruby on Rails project development. Our team of experienced developers, designers, and project managers are dedicated to delivering high-quality solutions that meet the unique needs of our clients. you can reach us at tauqeer.ahmad@xprolabs.com.