Ruby on Rails website is slow; could it be the SQL queries? - sql

Question

I have been working on a web application made with Ruby on Rails. I've "completed" the site, but it runs very slow and pages sometimes take on the order of ten seconds to load. I've broken this post up into three sections:
Overview
Diagnosis
Possible Ideas
Overview
As a very rough overview, this website displays personal projects, each of which have their own page. Each project (which I call a “post” in the code) is stored in a table. I’m not sure if it’s a bad idea to store a whole post in a database as some posts have a decent amount of text and images in their body. Each entry in the posts table has the following attributes:
# == Schema Information
#
# Table name: posts
#
# id :integer not null, primary key
# title :string
# body :text
# description :text
# slug :string
# created_at :datetime not null
# updated_at :datetime not null
# image_file_name :string
# image_content_type :string
# image_file_size :integer
# image_updated_at :datetime
# thumbnail_file_name :string
# thumbnail_content_type :string
# thumbnail_file_size :integer
# thumbnail_updated_at :datetime
# published :boolean default("f")
# published_at :datetime
#
Diagnosis
I’m using Heroku to host the application. I’ve upgraded to Hobby level dynos, so I could have access to some of the metrics they provide, like throughput, memory usage, etc. I don’t think Heroku is what is making the website slow, but rather my code. In an effort to debug, I've added a third party addon, Scout, that tracks the bottlenecks in the application.
Scout offers a trace feature that quantizes which code paths are taking the most time in the application. You can see below (in the bottom half of the picture) that there are a significant amount of traces that take upwards of ten seconds. Not very good…
When I click on the second line (6/17 2:04 PM), it gives a breakdown of the response time:
Expanding the SQL statements shows that most of the time intensive queries are from acting on the posts database, and sometimes sorting/ordering the posts (see below).
Possible Ideas
Is there anything obvious I am doing wrong here? I am pretty stuck, and not sure how to speed things up. Given what Scout is saying, I have two ideas:
Controller/SQL queries the controller invokes are slow
Embedded Ruby code in HTML is slow.
Controller/SQL queries the controller invokes are slow:
The code below shows the code where I am assigning #posts in the PostsController. The home method is run when the user visits the home page, and the index method is run when the user goes to the posts page. Could these queries be slow because there is a fair amount of data in the database (5 posts worth of text and images)?
class PostsController < ApplicationController
#before_action :authenticate_user!
before_action :set_post, only: [:show, :edit, :update, :destroy, :publish, :unpublish]
def home
#posts = Post.all.published.order('published_at DESC')
end
# GET /posts
# GET /posts.json
def index
if user_signed_in? and current_user.admin
#posts = Post.all.order("id")
else
#posts = Post.all.published
end
end
Embedded Ruby Code in HTML is slow:
I am using Ruby in some of my HTML code to sort the posts by date and to determine the most recent post. For example, in the sidebar of the website (to the left of the home page), there is section that displays “Recent”, and the logic behind that is:
<h4 style="border-bottom: 1px solid #bbb">Recent</h4>
<% #post=Post.all.published.order("published_at").last %>
<% if #post == nil or #post.published_at == nil %>
<div class="temp_sidebar">Coming Soon!</div>
<% else %>
<%= render partial: "layouts/media", locals: {post: #post} %>
<% end %>
Similarly, in the “Archives” section of the sidebar, I’m sorting the posts by date, and am doing this logic:
<% if Post.published.length != 0 %>
<h4>Archives</h4>
<div id="sidebar" class="list-group">
<% #published_posts = Post.published %>
<% archives = Hash.new(0) %>
<% #published_posts.each do |post| %>
<% if archives[post.date] == 0 %>
<% archives[post.date] = 1%>
<% else %>
<% archives[post.date] += 1 %>
<% end %>
<% end %>
<% archives.each do |key, value| %>
<button class="accordion"><%= key %> <span class="badge"> <%= value %></span></button>
<div class="panel">
<% #published_posts.each do |post| %>
<% if post.date == key %>
<p><%= link_to post.title, post_path(#post) %></p>
<% end %>
<% end %>
</div>
<% end %>
</div>
<% end %>
My idea is that maybe iterating over posts is taking a very long time, but I’m not entirely sure. I feel like this is valid code that is ok to be used, but maybe something about it is very slow. Would you guys have any ideas here? It may also be good to note here that the app's memory usage is pretty high: around 500MB. Maybe these queries are slow because of all the data that is getting fetched, but that said, I'm am not quite sure what "a lot" of data is for a web app like this. And of course, my hypothesis as to why this site is slow could be totally wrong, so I'm very open to your thoughts. Lastly, if the SQL queries/code I am using is slow, are there ways I could speed it up/improve its performance? Thank you for any help in advance!

Answer 1

I see two problems: a lack of SQL indexes, and too many calls to Post.all.
Your slow queries involve WHERE published = ?. If posts.published is not indexed, this will have to scan the entire table, not just the published posts. You also tend to sort by posts.published_at and without an index this will also be slow.
To fix this, add an index on posts.published and posts.published_at in a migration.
add_index(:posts, :published)
add_index(:posts, :published_at)
You can read more about indexes in the answers to What is an index in SQL?
Using Post.all or Post.published means loading every post from the database into memory. If you're not using them all, it's a huge waste of time.
For example, it's unwieldy to display every post on your index and home pages. Instead you should use pagination to fetch and display only a page of posts at a time. There are gems for this such as kaminari and will_paginate as well as larger administrative solutions such as ActiveAdmin. And if you don't like page links, if you look around you can find examples of using them for "infinite scroll".
Finally, you can add caching. Since your site isn't going to update very frequently, you can cache at various levels. Have a read through Caching with Rails: An Overview.
But caching brings its own problems. Consider if you need it after you do basic performance optimizations.

Answer 2

Schwern,
Thank you for the pointers! With the help of your post, I was able to minimize the number of calls to Post.all. These were killing my response time big time. I did not realize that calling Post.all loaded the all of the posts and their attributes to memory (it's a learning process haha).
For places where I did not need to load every attribute of a post, I ended up doing Post.select("attribute"). For example, originally in the home method of the PostsController, I had:
def home
#posts = Post.all.published.order('published_at DESC')
end
Running Post.all.published.order('published_at DESC') in the rails console shows that this query takes approximately 4 seconds:
Updating the home method to use Post.select, as opposed to Post.all (as shown below) significantly improved the response time of the query.
def home
#SLOW: loads everything from a post
##posts = Post.all.published.order('published_at DESC')
#FAST: only loads the necessary elements from a post, and does not waste time loading body (body = lots of data)
#posts = Post.select("id", "title", "description", "slug", "created_at", "updated_at", "image_file_name", "thumbnail_file_name", "published", "published_at", "date").published.order('published_at DESC')
end
Running #posts = Post.select("id", "title", "description", "slug", "created_at", "updated_at", "image_file_name", "thumbnail_file_name", "published", "published_at", "date").published.order('published_at DESC') in the rails console shows that this query takes approximately 2ms! Huge improvement! This is because I am not loading the body attribute, among other things of the post, which contain a lot of data, and thus consume a lot of time to load into memory. No point of loading it into memory if you're not going to use it!
In a similar spirit, I was able to improve the performance of the other sections of code (see below for fixes) and ultimately make the website have significantly faster response times!
Controller fix:
class PostsController < ApplicationController
#before_action :authenticate_user!
before_action :set_post, only: [:show, :edit, :update, :destroy, :publish, :unpublish]
def home
#SLOW: loads everything from a post
##posts = Post.all.published.order('published_at DESC')
#FAST: only loads the necessary elements from a post, and does not waste time loading body (body = lots of data)
#posts = Post.select("id", "title", "description", "slug", "created_at", "updated_at", "image_file_name", "thumbnail_file_name", "published", "published_at", "date").published.order('published_at DESC')
end
# GET /posts
# GET /posts.json
def index
if user_signed_in? and current_user.admin
#SLOW loads everything from a post
##posts = Post.all.order("id")
#FAST: only loads the necessary elements from a post, and does not waste time loading body (body = lots of data)
#posts = Post.select("id", "title", "description", "slug", "created_at", "updated_at", "image_file_name", "thumbnail_file_name", "published", "published_at", "date").order('id')
else
#SLOW loads everything from a post
##posts = Post.all.published
#FAST: only loads the necessary elements from a post, and does not waste time loading body (body = lots of data)
#posts = Post.select("id", "title", "description", "slug", "created_at", "updated_at", "image_file_name", "thumbnail_file_name", "published", "published_at", "date").published
end
end
Embedded Ruby Code in HTML fix:
"Recent" section fix:
<h4 style="border-bottom: 1px solid #bbb">Recent</h4>
<!-- LINE BELOW IS SLOW!!! (to test, uncomment line, and embrace in "<% %>") -->
<!-- #post = Post.all.published.order(:published_at).last -->
<!-- FAST! Line below replaces line above and is much faster! -->
<% #post = Post.select("id", "title", "description", "slug", "created_at", "updated_at", "image_file_name", "thumbnail_file_name", "published", "published_at", "date").published.order('published_at DESC').first %>
<% if #post == nil or #post.published_at == nil %>
<div class="temp_sidebar">Coming Soon!</div>
<% else %>
<%= render partial: "layouts/media", locals: {post: #post} %>
<% end %>
"Archives" section fix:
<!-- LINE BELOW IS SLOW!!! (to test, uncomment line, and embrace in "<% %>") -->
<!-- if Post.published.length != 0 -->
<!-- FAST! Line below replaces line above and is much faster! -->
<% if Post.select("id").published.count("id") != 0 %>
<h4>Archives</h4>
<div id="sidebar" class="list-group">
<!-- LINE BELOW IS SLOW!!! (to test, uncomment line, and embrace in "<% %>") -->
<!-- #published_posts = Post.published -->
<!-- FAST! Line below replaces line above and is much faster! -->
<% #published_posts = Post.select("id", "title", "date").published %>
<% archives = Hash.new(0) %>
<% #published_posts.each do |post| %>
<% if archives[post.date] == 0 %>
<% archives[post.date] = 1%>
<% else %>
<% archives[post.date] += 1 %>
<% end %>
<% end %>
<% archives.each do |key, value| %>
<button class="accordion"><%= key %> <span class="badge"> <%= value %></span></button>
<div class="panel">
<% #published_posts.each do |post| %>
<% if post.date == key %>
<p><%= link_to post.title, post_path(post.id) %></p>
<% end %>
<% end %>
</div>
<% end %>
</div>
<% end %>