Escaping CarrierWave versions
For one of the projects we have been working on, we had to be able to upload videos to the server, convert them to formats supported by browsers with <video> tag and serve them to the user. Since Ruby and Ruby on Rails are our tools of choice we used CarrierWave for file uploading and storage. Since Carrier Wave supports processing of input files into various versions, we decided to give it a try.
CarrierWave Gem
Since processing video files takes a lot of time, we had to do it in the background so we used carrierwave-backgrounder. When looking for a way to do video processing, we found the gem carrierwave-video that used streamio-ffmpeg for transcoding and it seemed fine for our task.
Our Gemfile looked like this:
gem 'carrierwave', '0.10.0'
gem 'carrierwave-backgrounder', '0.4.2'
gem 'carrierwave-video', '0.5.6'To encode videos to proper formats we used encode_video method from carrierwave-video:
version :mp4 do
process encode_video: [:mp4, { progress: :on_progress }]
endversion :webm do
process encode_video: [:webm, { progress: :on_progress }]
end
After some time there was a need in the project to be able to cut out small pieces of videos and serve them independently.
- At first we tried to do it the same way as previous processing:
version :mp4 do
process encode: [:mp4, { progress: :on_progress }]
endversion :webm do
process encode: [:webm, { progress: :on_progress }]
enddef encode(format, opts={})
encode_video(format, opts) do |movie, params|
params[:custom] = "-ss #{model.playback.start_in_hms} -t #{model.playback.duration_in_hms}"
end
end
Unfortunately this had a very serious drawback — since all of the versions were being processed from one file inputted to the uploader, they had to be transcoded every time. That was far from perfect as it took additional time and unnecessarily used resources so we hacked our way a bit: since the original video was already in the formats we needed, we could just pass those versions instead and copy the audio and video streams.
The Solution
To make the code less cluttered, we sprinkled it with a bit of DSL:
support_formats custom: proc { |model| "-ss #{model.start_time} -t #{model.duration}" },
source: proc { |model, version| model.parent.file.versions[version].file },
mode: :copydef support_formats(support_opts={})
FORMATS.each do |version_name, opts|
opts = opts.reverse_merge(support_opts) (conditional = opts.delete(:if)) && (conditional = conditional.to_sym)
uploader = version(version_name, if: conditional) { process encode_format: [opts] }
uploader[:uploader].class_eval <<-RUBY, __FILE__, __LINE__ + 1
def full_filename(file)
file + ".#{version_name}.#{opts[:extension]}" # forcing the extension, otherwise ffmpeg got confused
end
RUBY
end
enddef encode_format(opts={})
cache_stored_file! if !cached? if opts[:mode] == :copy
opts[:video_codec] = 'copy'
opts[:audio_codec] = 'copy'
end opts[:custom] = opts[:custom].call(model) if opts[:custom].respond_to?(:call) source = source.call(model, version_name) if source.respond_to?(:call)
source = file if source.nil?
source = source.path if source.respond_to?(:path) # etc
end
This setup kind of worked, but it posed a lot of problems: we didn’t have control over what versions were transcoded and we had to recreate each version if any of the transcoding failed. More than that, if any transcoding happened during deployment of new version, sidekiq had to be killed and restarted and it didn’t have a way of going back from where it started so the whole processing had to be either redone or ditched altogether and marked as crashed.
We have tried various solutions of mitigating this problem but unfortunately using carrierwave-backgrounder made everything more messy. That gem was great for simpler logic but unfortunately choked a bit when we tried extending it more. This also caused the logic of processing to be divided between non-logical parts, like processing code ending up partially in sidekiq worker (because it was easy to set custom worker when mounting an uploader), non-obvious or custom callbacks being thrown all over the place or processing starting non-asynchronously if we weren’t careful enough. The API got brittle and the whole codebase gradually became a mess.
class StreamUploadWorker < ::CarrierWave::Workers::StoreAsset def perform(*args)
set_args(*args) if args.present?
video = constantized_resource.find id # custom callbacks in model
run_callback video, :before_upload
super(*args)
video.reload run_callback video, :after_upload_success rescue
run_callback video, :after_upload_failure
video.broken! if video.respond_to?(:broken!) # logging
end def run_callback(video, callback)
video.send(callback) if video.respond_to?(callback)
end
end
The last straw…
The last straw, however, came with a new requirement. The project has matured enough to generate significant traffic and serving multimedia content from a dedicated server no longer seemed like a viable solution. We needed to have files on both a local server (for processing) and on some kind of cloud solution (for hosting). As CarrierWave versions are nothing more than differently named files in a directory, using them seemed like a bad idea considering the amount of patchwork needed. It was time to clean the house.
We solved the problem by ditching versions altogether. We created a separate model for storing files that could either have local or remote (fog) uploader attached. Then we wrote our own transcoding logic, top down, easy to understand, using streamio-ffmpeg directly with specified file as the source and putting it in the path CarrierWave expected.
class FileObject < ActiveRecord::Base
belongs_to :owner, polymorphic: true class Local < FileObject
mount_uploader :file, FileObjectUploader
end class Cloud < FileObject
mount_uploader :file, AwsUploader
end
endclass Processor
# partial, example code def recreate_versions!(video, file)
formats_with_options(video, file).each do |format, opts|
if video.parent.present?
original = video.parent.version(format)
file = original.file if original.respond_to?(:file)
end file_object = FileObject::Local.create!(owner: video, version: format)
filename = "#{SecureRandom.uuid}.#{opts[:extension]}" FileUtils.mkdir_p file_object.file.store_dir
destination = "#{file_object.file.store_dir}/#{filename}" transcode!(source, destination, format, opts)
file_object.update_column(:file, filename) S3UploadWorker.perform_async(file_object.id) unless Rails.env.development?
end
end
end
Summary
CarrierWave is a great solution for file uploading. It’s also a great solution for simple processing. The moral of the story is, you have to use the appropriate tools. Carrier Wave versions are not enough for complex processing or any processing that doesn’t use the original uploaded file as the source. It may seem obvious in retrospect but that’s what happens when your codebase gradually evolves. When this happens, always try to find some time to stop, look back and ask yourself: “Is this code doing what it was originally made for?”.
Happy coding!
