AppPerf Updated to Rails 5.2.2

I finally got around to updating my AppPerf project to the latest version of Rails and also removed a bunch of old files and code I didn’t need any longer. I will continue to clean up code as I come across it and work on adding some new features. Just wanted give a quick update regarding the upgrade.

Migrating from Carrierwave to ActiveStorage in Rails

For the Video Upload Platform series that I have been working on (Sorry I haven’t updated in a bit), I had been using Carrierwave to manage the file uploads. It also has a bunch of plugins/gems available to help with some tasks, however I wanted to move the series more into as “What is actually happening” set of posts. For this reason, I have decided to migrate that project from using Carrierwave to using ActiveStorage, since that has been released somewhat recently and is build into Rails.

If you have been following along in that series, then it should be pretty straight forward using the details I’m about to go into. If not, thats ok too, it should still be pretty familiar to you. At any rate lets get started!

The first thing we want to make sure is that we are update to date with Rails. If you have a specific version set in your Gemfile, remove that, or update it to the latest:

# Gemfile

- gem "rails", "~> 5.1.6
+ gem "rails"

One we have that updated, we just need to bundle the project to pull all the new fun in

bundle

Ok great. Now we want to add the ActiveStorage migrations and configure the set up to use local storage for our example. Do this by running the following command

bundle exec rails active_storage:install
bundle exec rake db:migrate

Great, now we should have our database migrated to contain the ActiveStorage tables. Let create the proper config file that ActiveStorage uses now. Create the file config/storage.yml and paste in the following YAML:

local:
  service: Disk
  root: <%= Rails.root.join("storage") %>

This tells ActiveStorage to store things locally. If you are interested in using some type of other storage methods, check out the ActiveStorage Overview page. We only have one more configuration step to go, and we should be good. Add the following line to your config/environments/development.rb file (test.rb and production.rb) as well if you want:

config.active_storage.service = :local

Now that we have ActiveStorage all set up, we can go ahead and update our model to migrate from Carrierwave to ActiveStorage. I will show our model and the line we want to remove and add:

class Video < ApplicationRecord
-  mount_uploader :file, VideoUploader
+  has_one_attached :file
end

As you can see, we no longer are going to use the CarrierwaveUploader. We should now be able to upload files using the same controller and form uploads that we had already created. The last thing we need to change however, is the view. We will pass the url for the upload using some built-in Rails helpers. In the app/views/videos/show.html.erb file, we need to make this change:

<p id="notice"><%= notice %></p>

<p>
  <strong>Title:</strong>
  <%= @video.title %>
</p>

<p>
  <strong>File:</strong>
-  <video controls width="640" height="480" src="<%= @video.file.url %>"></video>
+  <video controls width="640" height="480" src="<%= url_for(@video.file) %>"></video>
</p>

<%= link_to 'Edit', edit_video_path(@video) %> |
<%= link_to 'Back', videos_path %>

The last piece that we need to add in, which I will do in another post, is the ability to do the background Transcoding, that we were getting out of the box with the carrierwave-video gem, but this should get us starting with using ActiveStorage instead of Carrierwave.

Let me know if you have any thoughts in the comments below, or run into any issues. Thanks!

How to organize a routes.rb file in Rails

If you have a small Ruby on Rails application, you probably don’t have much need to organize your routes.rb file. However if your project is large, you probably have a somewhat complicated, and quite frankly, messy routes file. Sometimes its hard to really understand where your routes are, especially when you are working with hundreds of lines of routes. I had such an issue in a recent project I was working on.

The project has multiple contexts within it, and each on of them has their own set of routes. These were all smushed into the main routes.rb file along with the normal base routes. What I wanted to do was create multiple routes files under a config/routes folder and split them up there to make them more manageable. Here is an example of how to split them up:

# config/routes.rb
Rails.application.routes.draw do
  # Some base routes here
  resources :pages


  extend AdminRoutes
  extend UserRoutes
  extend OrganizationRoutes
end

As you can see, we have a basic routes file, however you will notice that there are a few extend Class directives in there. These will load in the corresponding class file, that I will have stored in the config/routes folder. Here is an example of what one of those files looks like.

# config/routes/admin_routes.rb

module AdminRoutes
  def self.extended(router)
    router.instance_exec do
      authenticate :user, lambda { |u| u.admin? } do
        mount RailsAdmin::Engine => '/admin', as: 'rails_admin'
        mount Sidekiq::Web => '/sidekiq'
      end
    end
  end
end

I will show you one more, since the AdminRoutes file is mostly just the entry point for some mounted Rails Engines, and not normal resources

# config/routes/user_routes.rb

module UserRoutes
  def self.extended(router)
    router.instance_exec do

      authenticated :user do
        root to: "dashboard#show", as: :authenticated_root
      end

      devise_for :users,
        :skip => [:registrations],
        :controllers => { :invitations => 'user/invitations' }
      as :user do
        get '/user/new' => 'user/registrations#new', as: 'new_user_registration'
        post '/user' => 'user/registrations#create', as: 'user_registration'

        get 'user/edit' => 'user/registrations#edit', :as => 'edit_user_registration'
        put ':id' => 'user/registrations#update', :as => 'registration'

        delete '' => 'user/registrations#destroy'
      end

    end
  end
end

The last thing we need to do, is make sure that the new config/routes folder gets picked up by rails. Add the following line to your config/application.rb file

config.autoload_paths += %W(#{config.root}/config/routes)

Hopefully these examples help you with splitting up a large routes.rb file into smaller, more manageable pieces. Let me know in the comments if this helps, or you have any issues getting this to work.

WordPress/Nginx response long pauses

I have been working more and more with WordPress lately, and since I am into technology, I would have to host it myself. Right? 🙂 Anyway, I recently ran into an issue that took me a while to figure out. Well, almost fully figured out. At least I have mitigated the main issue. Here is it…

I was noticing that my page would load at a decent speed, nothing to cause alarm. However, when I would access individual pages, they would sometime (usually) take 15 to 20, and sometimes longer, seconds to load. I would confirm the same by using curl to request the page. The odd thing I noticed doing this, was that only part of the document was returned, and then there was a long pause, and finally the rest of the document came.

When this first started happening, I had suspected maybe there was something wrong with my nginx proxy settings, since I was passing PHP requests off to PHP-FPM. I tweaked pretty much every setting I could find and nothing helped. Then I started looking at PHP settings, maybe there wasn’t enough memory and it was buffering to disk. Nothing helped there either. I had other WordPress sites as well. Some were exhibiting the issue and some were not. I figured the issue had to be somewhere in WordPress, but wasn’t sure where yet.

I started by deactivating a bunch of plugins on my site to see if there was an issue there. It didn’t seem to help, although I could have just been a fluke, since sometimes the pages would work fine. This led me to believe it might have been network related somehow, but since the pages would still hang for a bit during a curl request I had suspected that it might have been more of a backend thing. It took some digging, but I finally searched for the right terms in the big Google, and case across an article that helped out, https://deliciousbrains.com/wp-migrate-db-pro/doc/wp_http_block_external/.

After I implemented this change in my WordPress blog, pages were loading lightning fast again. The only word of caution I should use is that this blocks all external requests, including plugin calls and updates, so you will want to combine this change with the WP_ACCESSIBLE_HOSTS setting which is also mentioned in that page.

WordPress vs Everything Else – Why you should not build your own blog

Being a web developer, the idea that I would use a blogging platform always made me cringe a little bit. I mean, why not. I had the skills to create my own blog that I could customize exactly how I wanted. And thats where I started originally.

I wanted to learn a new language, Ruby on Rails, and what better way to start than follow one of many “How to build a blog with Ruby on Rails” posts that were littered across the internet at the time. So I did that for a while, and back then (circa 2006-ish) there wasn’t much in the way of deployment and server-ease like Heroku, so I had to manually do all that myself, with FastCGI and eventually Mongrel.

 

I quickly realized that all the customization that I wanted to build, I no longer wanted to build. I just wanted a place to share some ideas and not worry about the management of another web application. I was already doing that in my day job. At this point, I decided to use WordPress, but this wasn’t my final decision, I would come to find out.

I had started using WordPress for a while, but again, the developer bug crept back in. I wanted to find something else that wasn’t so “WordPress-y”. I tried some other things like Joomla, Typepad, Jekyll and other static blogging codebases. I eventually settled on using Bolt, which is another PHP based blogging platform. I have no idea why….

I used that for a while and finally got tried of the lack of community around it and some customizations that I missed, that were available with WordPress out of the box. After battling with that for a while, I decided to move back to WordPress and import all my data back into it. I think this is where I’ll live now and its just so flexible and easy to just start writing.

I am in no way saying, don’t write a blog if you want to learn a new language. It’s a super simple idea that I think anyone can grasp, and you don’t want to be overwhelmed while learning a new language. I would just recommend to not bother using it for your official, public blog that you want to write on all the time.

So at the end of the day, I am not a super fan of having a PHP based blog, when I mostly code and write about Ruby on Rails code, but the fact that I pretty much never have to dig into the internals of WordPress, makes it worth it in the end.

AppPerf Latency Bands with color schemes!

I haven’t posted lately about any updates I have been working on in regard to my open source application performance monitoring tool call AppPerf. I have been trying to get some time here and there to work on and improve it and add new features that would be useful. I am happy to say that I finally added a new chart on the overview page that should hopefully make it easier to identify when you are having an issue in regards to the performance of your application. I got the inspiration while reading about query bands on the VidixCortex site. Coincidentally, I have decided to call this new visualization, Latency Bands, as they are measuring the latency across different percentiles of data.

Previously, I was using two separate graphs to represent what I think I am able to show in one graph. I also think its much easier to identify when an issue has risen. First I will show what was previously being displayed on the page:

As you can see, I was showing a Latency and Latency distribution chart. The Latency chart was showing 50, 75, 90, and 95th percentiles. Likewise, the distribution graph was showing, well the distribution of requests across all 100 percentiles. At first glance, its hard to tell what is actually going on here, and with AppPerf, I want to make is extremely easy and quick to say, “Hey, I see a problem.”. Also, while these particular percentiles are helpful, what about the 95th and up percentiles that you are missing. Those are also important. We don’t know if the 100th is 5000ms,50,000ms, or even more. Not that helpful.

Now lets introduce the Latency Bands chart:

Quickly looking at this chart, you can see that something is going on at about 11:39 AM, as indicated by the red spike in there. The way this chart works is that it groups the latencies into 10 buckets based on their latencies. As the latency increases for that bucket, so does the color. The color range goes from blue to red, as more requests are grouped into each bucket. If you want to see the breakdown of the latencies, and how many requests there were, you can mouse over each data point and view the following:

Once you find a spike in your latency, you can then drag over that range to zoom in farther, and create a time scope that shows you what was going on with each of your components at that time. Hopefully this feature will allow you to identify more problems quickly, and determine how to fix them!

Shoot me some comments below if this was helpful!

Thanks

How to create a video upload platform using Ruby on Rails – Part 2

Welcome back to part 2 of How to create a video upload platform using Ruby on Rails. In Part 1, we worked on setting up the basic Ruby on Rails application that allowed us to upload, store, and playback video files from our app. In this next part, we are going to enhance that process some by adding some FFMPEG Transcoding. This way, we can get all videos that are uploaded converted into a common format that will make it easier to work with on the playback side.

So lets get started. The first thing we need to do is make sure we have ffmpeg installed. On mac this is pretty simple using brew. If you don’t have HomeBrew installed, please follow the directions at https://brew.sh/. If you are on another linux or windows system, there are many guides online to help set ffmpeg up for you. Once you have that installed, we can install the ffmpeg command:

brew install ffmpeg

I decided to install a lot of the extra ffmpeg options to make sure I had the best compatibility. Here is a more complete line

brew install ffmpeg --with-chromaprint --with-fdk-aac --with-libass --with-librsvg --with-libsoxr --with-libssh --with-tesseract --with-libvidstab --with-opencore-amr --with-openh264 --with-openjpeg --with-openssl --with-rtmpdump --with-rubberband --with-sdl2 --with-snappy --with-tools --with-webp --with-x265 --with-xz --with-zeromq --with-zimg

Once you have ffmpeg installed and running we need to add the carrierwave-video gem to our project:

# Gemfile
gem "carrierwave-video"

Next, we need to update our upload to include this new library. Open the app/uploaders/video_uploader.rb file and add the following include line to it:

class VideoUploader < CarrierWave::Uploader::Base
 include CarrierWave::Video # <== Add this line here
 
 ...

In order for Carrierwave to actually encode the video, we need to also add a line that instructs it to do so. Lets also adjust the resolution while we are at it.

class VideoUploader < CarrierWave::Uploader::Base
  include CarrierWave::Video

  process encode_video: [:mp4, resolution: "640x480"]

  ...

The last thing we need to do is change the file extension. If we upload an avi or mov file, we convert that to mp4 but with Carrierwave it will maintain the original extension. So add the following lines right after the process line added above

def full_filename(for_file)
  super.chomp(File.extname(super)) + '.mp4'
end

def filename
  original_filename.chomp(File.extname(original_filename)) + '.mp4'
end

Fire up your Rails app and upload a video just like we did in Part 1. You should see the following in your Rails console. Notice the line Running transcoding...

Started POST "/videos" for 127.0.0.1 at 2018-05-17 09:41:10 -0400
Processing by VideosController#create as HTML
  Parameters: {"utf8"=>"✓", "authenticity_token"=>"pCgWw3lxulQ9qiGxfck5oqFXgQo0tqO7gG371YEbFaiiRMdAc98uPGQCvTYzn0/Kub/pQtQI9n95I+grmGvQ6w==", "video"=>{"title"=>"adf", "file"=>#<ActionDispatch::Http::UploadedFile:0x007fbb5f814be8 @tempfile=#<Tempfile:/var/folders/__/7ns9m_817l71zfd0xw5618n00000gn/T/RackMultipart20180517-60568-c6cwal.mov>, @original_filename="step.mov", @content_type="video/quicktime", @headers="Content-Disposition: form-data; name=\"video[file]\"; filename=\"step.mov\"\r\nContent-Type: video/quicktime\r\n">}, "commit"=>"Create Video"}
I, [2018-05-17T09:41:10.439284 #60568]  INFO -- : Running transcoding...
["/usr/local/bin/ffmpeg", "-y", "-i", "/Sites/VideoSite/tmp/1526564470-60568-0006-7430/step.mov", "-vcodec", "libx264", "-acodec", "aac", "-s", "640x480", "-r", "30", "-strict", "-2", "-map_metadata", "-1", "-aspect", "1.3333333333333333", "/Sites/VideoSite/tmp/1526564470-60568-0006-7430/tmpfile.mp4"]

I, [2018-05-17T09:41:17.640387 #60568]  INFO -- : Transcoding of /Sites/VideoSite/tmp/1526564470-60568-0006-7430/step.mov to /Sites/VideoSite/tmp/1526564470-60568-0006-7430/tmpfile.mp4 succeeded

   (0.3ms)  begin transaction
  SQL (0.3ms)  INSERT INTO "videos" ("title", "file", "created_at", "updated_at") VALUES (?, ?, ?, ?)  [["title", "adf"], ["file", "step.mp4"], ["created_at", "2018-05-17 13:41:17.643006"], ["updated_at", "2018-05-17 13:41:17.643006"]]
   (1.0ms)  commit transaction
Redirected to http://localhost:3000/videos/1
Completed 302 Found in 7260ms (ActiveRecord: 1.6ms)

 

There we go. We are now successfully encoding the video file to a common format using ffmpeg. In the next part we will look at moving this process to the background so that we don’t force the user to wait on the transcoding, and add some status indications of how the progress is going. Until next time!

How to create a video upload platform using Ruby on Rails – Part 1

Lets create a video upload platform using the popular Ruby on Rails web framework. I have never done a multi-part series before so I thought I would take a stab at it using a small project I did in my spare time. This series might go slow since I have limited time to write these, but hopefully it will be informative.

The small project I was working on was attempting to create a video upload platform using Ruby on Rails. The app would allow you to upload videos and would play them back using Adaptive Bitrate and HLS techniques to help adjust to changing bandwidth requirements. One other thing I added for fun, was a small Node.js server that handles and transcoded videos On-The-Fly when it could, or would fallback to the normal post transcoding process. This allowed the app to receive the upload and transcode at the same time, reducing the amount of time the user had to wait. Since uploading videos has a lot of idle time, why not use that to do some more work.

This walk through will hopefully try and reattempt the work I did with that project and hopefully someone can find fun with it.

We will assume you have a basic rails environment set up and can create new Rails apps:

rails new VideoSite

Let it do its stuff and then lets go into the project.

cd VideoSite

At this point, we have a freshly baked Ruby on Rails application. We need a place where we can upload some videos. We will use the popular Carrierwave gem for handling our uploads. Lets add that to our project and bundle the project:

# Gemfile

gem 'carrierwave'
bundle install

Next we need to create the Uploader using the Carrierwave generator.

rails generate uploader Video

Great. Next lets go ahead and create our controller/model/views for our videos pages and then migrate our database.

rails generate scaffold Video title:string file:string
rake db:migrate

Before we can start uploading videos however, we need to attach the uploader to our video model. Open up app/models/video.rb and change it as follows.

class Video < ApplicationRecord
  mount_uploader :file, VideoUploader
end

Now we can fire up our rails application and upload some videos! Start your rails server with rails s and then navigate to http://localhost:3000/videos/new and you should see our amazing video page.

In order to actually select a video file, we need to update the view to render a file_field instead of a text_field as well as allow multipart uploads. Lets open up the app/views/videos/_form.html.erb file and change it to the following.

<%= form_with(model: video, local: true, html: { multipart: true }) do |form| %>
  <% if video.errors.any? %>
    <div id="error_explanation">
      <h2><%= pluralize(video.errors.count, "error") %> prohibited this video from being saved:</h2>

      <ul>
      <% video.errors.full_messages.each do |message| %>
        <li><%= message %></li>
      <% end %>
      </ul>
    </div>
  <% end %>

  <div class="field">
    <%= form.label :title %>
    <%= form.text_field :title, id: :video_title %>
  </div>

  <div class="field">
    <%= form.label :file %>
    <%= form.file_field :file, id: :video_file %>
  </div>

  <div class="actions">
    <%= form.submit %>
  </div>
<% end %>

We should now be able to choose files and upload them!

Go ahead and select a video and click the Create Video Button. You should then see something like this.

This isn’t really all that fun since we cannot see the video, just the path to it. Lets fix that by using the HTML video tag. Open up the app/views/video/show.html.erb file and change it to the following and then reload the page and you should see your video playing.

<p id="notice"><%= notice %></p>

<p>
  <strong>Title:</strong>
  <%= @video.title %>
</p>

<p>
  <strong>File:</strong>
  <video controls width="640" height="480" src="<%= @video.file %>"></video>
</p>

<%= link_to 'Edit', edit_video_path(@video) %> |
<%= link_to 'Back', videos_path %>

Congratulation! You have the beginning of your very own video uploading service. If you try to load this in other browsers, you may run into problems however, since not all browsers are capable of playing all formats all the time. Another issue is that some web servers will send the entire video file down to the client, which we do not want. It would be a lot nicer if only the part of the video we request is what we receive, continuously. In the next part, we will continue to expand on our simple video uploading by adding some fancy transcoding techniques and use some front end tools that will help us render on all browsers, including mobile and stream smaller chunks of video at a time.

Hope this was fun. Take a look at Part II next!

How to create a Sidekiq service application without Rails

Sometimes you want to create a small Sidekiq service to process requests, without all the extra bloat (and memory usage) from a Rails framework. It turns out that creating such service is not all that hard to do. Create your application using the folders and files below as your starting point.

Gemfile

# Gemfile
source 'https://rubygems.org'
ruby "2.5.0"
git_source(:github) do |repo_name|
  repo_name = "#{repo_name}/#{repo_name}" unless repo_name.include?("/")
  "https://github.com/#{repo_name}.git"
end

gem "sidekiq"

config/sidekiq.yml

---
:concurrency: <%= ENV["CONCURRENCY"] || 25 %>
:queues:
 - default

config/application.rb

# config/application.rb

ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile", __FILE__)

require "bundler/setup" if File.exist?(ENV["BUNDLE_GEMFILE"])
Bundler.require

ENV["RACK_ENV"] ||= "development"

module Example
 class Application
   def self.root
     @root ||= File.dirname(File.expand_path('..', __FILE__))
   end

   def self.env
     @env ||= ENV["RAILS_ENV"] || ENV["RACK_ENV"] || ENV["ENV"] || "development"
   end

   def self.logger
     @logger ||= Logger.new(STDOUT)
   end
 end
end

require_relative "initializers/sidekiq"

config/initializers/sidekiq.rb

# config/initializers/sidekiq.rb

Sidekiq.configure_server do |config|
  config.redis = { url: ENV["REDIS_URL"] }
end

app/workers/my_worker.rb

# app/workers/my_worker.rb

class MyWorker
  include Sidekiq::Worker

  def perform(*args)
    # Do work here
  end
end

Once you have created those files, then you can run the following on the command line:

bundle exec sidekiq -e ${RACK_ENV:-development} -r ./config/application.rb -C ./config/sidekiq.yml

If you want to enqueue jobs from another application using Sidekiq, its as easy as:

Sidekiq::Client.enqueue_to "default", MyWorker, "param_1" => "1", "param_2" => "2"

When you have all this working, then you can start adding things like your database connections and other gems and code to the worker. Let me know if this helps anyone or if you run into any problems!

Installing mysql2 Gem error targeting Mac OS X 10.5 or later

I recently tried to install the mysql2 gem and was presented with the following error. I was on El Capitan 10.11.6 and using ruby 1.9.3-p125.

Gem::Installer::ExtensionBuildError: ERROR: Failed to build gem native extension.
/Users/…/.rvm/rubies/ruby-1.9.3-p125/bin/ruby extconf.rb
checking for rb_thread_blocking_region()… yes
checking for rb_wait_for_single_fd()… yes
checking for mysql.h… yes
checking for errmsg.h… yes
checking for mysqld_error.h… yes
creating Makefile

make

compiling client.c
couldn’t understand kern.osversion `15.6.0’
compiling mysql2_ext.c
couldn’t understand kern.osversion `15.6.0’
compiling result.c
couldn’t understand kern.osversion `15.6.0’
linking shared-object mysql2/mysql2.bundle
couldn’t understand kern.osversion `15.6.0’
ld: -rpath can only be used when targeting Mac OS X 10.5 or later
collect2: ld returned 1 exit status
make: *** [mysql2.bundle] Error 1

Gem files will remain installed in /Users/…/vendor/bundle/ruby/1.9.1/gems/mysql2-0.3.11 for inspection.
Results logged to /Users/…/vendor/bundle/ruby/1.9.1/gems/mysql2-0.3.11/ext/mysql2/gem_make.out

An error occurred while installing mysql2 (0.3.11), and Bundler cannot continue.
Make sure that `gem install mysql2 -v ‘0.3.11’` succeeds before bundling.

After some searching I found out the solution was to install the gem by setting the Mac Deployment Target as so:

MACOSX_DEPLOYMENT_TARGET=10.9 gem install mysql2

Hope this helps someone else in the future!

1 2 3 6