I have a rails data migration (postgres db) where I have to use pure sql to convert the data due to some model restrictions. The data is stored as json as a string, but I need it to be a usable hash for other purposes.
My migration works to convert it to the hash. However, my down method ends up just deleting the data or leaving it as an empty {}. Btw to clear up any confusion, my column name is actually saved as data in table Games
Based on my up method, how would i properly reverse the migration using sql only?
class ConvertGamesDataToJson < ActiveRecord::Migration[6.0]
def up
statement = <<~SQL
update games set data = regexp_replace(trim(both '"' from data::text), '\\\\"', '"', 'g')::jsonb;
SQL
ActiveRecord::Base.connection.execute(statement)
# this part works!
end
def down
statement = <<~SQL
update games set data = to_json(data::text)::jsonb;
SQL
ActiveRecord::Base.connection.execute(statement)
end
end
Here is how the it looks after properly converting it
data: {
"id"=>"d092a-f2323",
"recent"=>'yes',
"note"=>"some text",
"order"=>1
}
how it is before the migration and what it needs to rollback to:
data:
"{
\"id\":\"d092a-f2323\",
\"recent\":\"yes\",
\"note\":\"some text\",
\"order\":1,
}"
If you're displaying a data structure in the rails console, those \" aren't really there. They're just formatting because the console has wrapped the string in ". For example...
[2] pry(main)> %{"up": "down"}
=> "\"up\": \"down\""
But if we print it...
[3] pry(main)> puts %{"up": "down"}
"up": "down"
Given that is a JSON string, you can simply change the type of the column to jsonb and be done with it.
-- up
alter table games alter column data type jsonb USING data::jsonb;
-- down
alter table games alter column data type text;
Postgres doesn't know how to automatically cast text to jsonb, so we need to tell it. using data::jsonb does a simple cast of the text to jsonb. It can cast jsonb to text just fine.
You can do this in a migration with change_column.
def up
change_column :users, :data, :jsonb, using: 'data::jsonb'
end
def down
change_column :users, :data, :text
end
Related
I have a database column of type text that stores a hash in my database.
column_a = {sample_list: [{email: test#gmail.com},{email: test2#gmail.com}]}
I want to generate another column in the database that contains contains a string of email_address eg:
column_B = "test#gmail.com,test2#gmail.com"
I know how to get generate the string from the hash, my questions are what model changes do I have to make so that whenever column_A is populated, column_B should be updated?
How do I handle migrations of historic data in this case?
class ClassName < ApplicationRecord
before_save :maybe_update_column_b
def maybe_update_column_b
update_column_b if column_a_changed?
end
def update_column_b
#your code to update column b
end
end
then (at a time you determine to be safe to do an update) run this in a console. Find each will batch the data.
ClassName.find_each do |cn|
cn.update_column_b
cn.save
end
This has been really difficult to find information on. The crux of it all is that I've got a Rails 3.2 app that accesses a MySQL database table with a column of type POINT. Without non-native code, rails doesn't know how to interpret this, which is fine because I only use it in internal DB queries.
The problem, however, is that it gets cast as an integer, and forced to null if blank. MySQL doesn't allow null for this field because there's an index on it, and integers are invalid, so this effectively means that I can't create new records through rails.
I've been searching for a way to change the value just before insertion into the db, but I'm just not up enough on my rails lit to pull it off. So far I've tried the following:
...
after_validation :set_geopoint_blank
def set_geopoint_blank
raw_write_attribute(:geopoint, '') if geopoint.blank?
#this results in NULL value in INSERT statement
end
---------------------------
#thing_controller.rb
...
def create
#thing = Thing.new
#thing.geopoint = 'GeomFromText("POINT(' + lat + ' ' + lng + ')")'
#thing.save
# This also results in NULL and an error
end
---------------------------
#thing_controller.rb
...
def create
#thing = Thing.new
#thing.geopoint = '1'
#thing.save
# This results in `1` being inserted, but fails because that's invalid spatial data.
end
To me, the ideal would be to be able to force rails to put the string 'GeomFromText(...)' into the insert statement that it creates, but I don't know how to do that.
Awaiting the thoughts and opinions of the all-knowing community....
Ok, I ended up using the first link in steve klein's comment to just insert raw sql. Here's what my code looks like in the end:
def create
# Create a Thing instance and assign it the POSTed values
#thing = Thing.new
#thing.assign_attributes(params[:thing], :as => :admin)
# Check to see if all the passed values are valid
if #thing.valid?
# If so, start a DB transaction
ActiveRecord::Base.transaction do
# Insert the minimum data, plus the geopoint
sql = 'INSERT INTO `things`
(`thing_name`,`thing_location`,`geopoint`)
values (
"tmp_insert",
"tmp_location",
GeomFromText("POINT(' + params[:thing][:lat].to_f.to_s + ' ' + params[:thing][:lng].to_f.to_s + ')")
)'
id = ActiveRecord::Base.connection.insert(sql)
# Then load in the newly-created Thing instance and update it's values with the passed values
#real_thing = Thing.find(id)
#real_thing.update_attributes(b, :as => :admin)
end
# Notify the user of success
flash[:message] = { :header => 'Thing successfully created!' }
redirect_to edit_admin_thing_path(#real_thing)
else
# If passed values not valid, alert and re-render form
flash[:error] = { :header => 'Oops! You\'ve got some errors:', :body => #thing.errors.full_messages.join("</p><p>").html_safe }
render 'admin/things/new'
end
end
Not beautiful, but it works.
I currently have a .find method in one of my rails controller actions - the relevant part is:
.find(:all, :select => 'last_name as id, last_name as name')
I am getting some odd behaviour trying to alias the last_name column as id - if I alias it as anything else, it works fine (i can do last_name as xyz and it outputs the last name in a column called xyz, but as I am using this to populate a drop-down where I need to have the name in the id column, i need it to be called 'id').
I should point out that it does output an id column, but it is always "id":0.
Could anyone shed any light on what I need to do to get this column aliased as 'id'?
Thanks!
I'm not sure of how you can do this in a Rails query statement. Rails is going to try and take over the id column, casting the value returned by the database as id with the type of column that id is (presumably integer). That's why your id column keeps getting set to 0, because "string".to_i #=> 0
However, there is a way to do it, once you have the results back.
Since you have the question tagged as Rails 3, it is preferable to use the new ActiveRelation syntax. You can do the following:
# First, get the results from the query, then loop through all of them.
Customer.select("last_name as 'ln', last_name as 'name'").all.collect do |c|
# The first step of the loop is to get the attributes into a hash form
h = c.attributes
# The next step is to create an "id" key in the hash.
# The Hash#delete method deletes the key/value pair at the key specified and returns the value.
# We'll take that returned value and assign it to the just created "id" key.
h["id"] = h.delete("ln")
# And we have to call out the hash to ensure that it's the returned value from the collect.
h
end
That will get you a hash with the id value as the text string value last_name and a name value as the same.
Hope that helps!
You shouldn't need to setup aliases in the finder SQL just to populate a drop-down. Instead simply use the last_name value for the value attribute (as well as the display text).
Eg if you're using the collection_select helper:
<%= f.collection_select :attribute_id, #collection, :last_name, :last_name %>
With a simple model like that
class Model < ActiveRecord::Base
# ...
end
we can do queries like that
Model.where(["name = :name and updated_at >= :D", \
{ :D => (Date.today - 1.day).to_datetime, :name => "O'Connor" }])
Where the values in the hash will be substituted into the final SQL statement with proper escaping depending on the underlying database engine.
I would like to know a similar feature for SQL execution like:
ActiveRecord::Base.connection.execute( \
["update models set name = :name, hired_at = :D where id = :id;"], \
{ :id => 73465, :D => DateTime.now, :name => "O'My God" }] \
) # THIS CODE IS A FANTASY. NOT WORKING.
(Please do not solve the example with loading a Model object, modifying and then saving! The example is only an illustration for the feature I would like to have / know. Concentrate on the subject!)
The original problem is that I want to insert large amount (many thousand lines) of data into the database. I want to use some features of the SQL abstraction of the ActiveRecord framework but I don't want to use model objects based on ActiveRecord::Base because they are damn slow! (8 queries per second for my current problem.)
query = ActiveRecord::Base.connection.raw_connection.prepare("INSERT INTO users (name) VALUES(:name)")
query.execute(:name => 'test_name')
query.close
Extending the #peufeu solution with concrete code example for bulk insert:
users_places = []
users_values = []
timestamp = Time.now.strftime('%Y-%m-%d %H:%M:%S')
params[:users].each do |user|
users_places << "(?,?,?,?)"
users_values << user[:name] << user[:punch_line] << timestamp << timestamp
end
bulk_insert_users_sql_arr = ["INSERT INTO users (name, punch_line, created_at, updated_at) VALUES #{users_places.join(", ")}"] + users_values
begin
sql = ActiveRecord::Base.send(:sanitize_sql_array, bulk_insert_users_sql_arr)
ActiveRecord::Base.connection.execute(sql)
rescue
"something went wrong with the bulk insert sql query"
end
Here is the reference to sanitize_sql_array method in ActiveRecord::Base, it generates the proper query string by escaping the single quotes in the strings. For example the punch_line "Don't let them get you down" will become "Don\'t let them get you down".
Yes you could do raw SQL, but checkout the ar-extensions gem that helps with batch inserts:
https://github.com/zdennis/ar-extensions
Here's a post on it, and various other techniques:
http://www.coffeepowered.net/2009/01/23/mass-inserting-data-in-rails-without-killing-your-performance/
For INSERTs, batching them using a long VALUES clause (as shown by Simon's link) is the fastest way (unless you want to generate a text file and load it in your database with MySQL's LOAD DATA INFILE). But you have to be very careful about escaping your text values (which is not done in the example).
I was asking "what database are you using" because it does matter for mass UPDATEs.
For instance, you can do this on postgres (and I believe SQL Server changing "columnX" to "colX" ):
UPDATE foo
JOIN (VALUES (1,2),(3,4),... long list) v ON (foo.id=v.column1)
SET foo.bar = v.column2
And you can update a load of rows using a single statement, very fast.
If you don't need Ruby to perform some Ruby-specific magic on your data, the fastest way to transfer data from one DB to a different one is to export as a text file (CSV or tab separated), load it on the other DB (LOAD DATA INFILE on MySQL), perhaps in a temporary table, and bulk process using SQL.
EDIT : Here's how I do this in Python :
sql = [ "INSERT INTO foo (column list) VALUES " ]
values = []
for tuple in tuple_list:
append "(?,?,?,?)" to sql
extend values list with tuple
Then join sql into a string, you get "INSERT INTO foo (column list) VALUES (?,?,?,?),(?,?,?,?),(?,?,?,?)" with the "(?,?,?,?)" repeated as many times as you have lines to insert.
Then "values" contains a list of (a1,b1,c1,d1,a2,b2,c2,d2,a3,b3,c3,d3) with an,bn,cn,dn being the tuples you want to insert for line n. Each one corresponds to a placeholder in the sql string.
Then pass this to the usual "execute query with parameters" function which will handle quoting and escaping as usual.
I encountered a similar issue recently when tying to insert 100K+ records into a MySQL database for a Rails 4 app using mysql2 gem. The data included characters that had to be sanitized prior to insert.
The solution I ended going with was a slightly modified version of Option 3 described at https://www.coffeepowered.net/2009/01/23/mass-inserting-data-in-rails-without-killing-your-performance/
Here's the relevant code block from the above link:
TIMES = 10000
inserts = []
TIMES.times do
inserts.push "(3.0, '2009-01-23 20:21:13', 2, 1)"
end
sql = "INSERT INTO user_node_scores (`score`, `updated_at`, `node_id`, `user_id`) VALUES #{inserts.join(", ")}"
The modification I made was using the public method ActiveRecord::Base.sanitize() on values that required it.
inserts = []
created = Time.now.strftime "%Y-%m-%d %H:%M:%S"
params[:audits].each do |audit|
inserts.push "(#{audit.user_id), #{created}," + ActiveRecord::Base.sanitize(audit.comment) + ", #{audit.status})"
end
sql = "INSERT INTO user_audits (`user_id`, `created_at`, `comment`, `status`) VALUES #{inserts.join(", ")}"
I know that you can ask ActiveRecord to list tables in console using:
ActiveRecord::Base.connection.tables
Is there a command that would list the columns in a given table?
This will list the column_names from a table
Model.column_names
e.g. User.column_names
This gets the columns, not just the column names and uses ActiveRecord::Base::Connection, so no models are necessary. Handy for quickly outputting the structure of a db.
ActiveRecord::Base.connection.tables.each do |table_name|
puts table_name
ActiveRecord::Base.connection.columns(table_name).each do |c|
puts "- #{c.name}: #{c.type} #{c.limit}"
end
end
Sample output: http://screencast.com/t/EsNlvJEqM
Using rails three you can just type the model name:
> User
gives:
User(id: integer, name: string, email: string, etc...)
In rails four, you need to establish a connection first:
irb(main):001:0> User
=> User (call 'User.connection' to establish a connection)
irb(main):002:0> User.connection; nil #call nil to stop repl spitting out the connection object (long)
=> nil
irb(main):003:0> User
User(id: integer, name: string, email: string, etc...)
If you are comfortable with SQL commands, you can enter your app's folder and run rails db, which is a brief form of rails dbconsole. It will enter the shell of your database, whether it is sqlite or mysql.
Then, you can query the table columns using sql command like:
pragma table_info(your_table);
complementing this useful information, for example using rails console o rails dbconsole:
Student is my Model, using rails console:
$ rails console
> Student.column_names
=> ["id", "name", "surname", "created_at", "updated_at"]
> Student
=> Student(id: integer, name: string, surname: string, created_at: datetime, updated_at: datetime)
Other option using SQLite through Rails:
$ rails dbconsole
sqlite> .help
sqlite> .table
ar_internal_metadata relatives schools
relationships schema_migrations students
sqlite> .schema students
CREATE TABLE "students" ("id" integer PRIMARY KEY AUTOINCREMENT NOT NULL, "name" varchar, "surname" varchar, "created_at" datetime NOT NULL, "updated_at" datetime NOT NULL);
Finally for more information.
sqlite> .help
Hope this helps!
You can run rails dbconsole in you command line tool to open sqlite console. Then type in .tables to list all the tables and .fullschema to get a list of all tables with column names and types.
To list the columns in a table I usually go with this:
Model.column_names.sort.
i.e. Orders.column_names.sort
Sorting the column names makes it easy to find what you are looking for.
For more information on each of the columns use this:
Model.columns.map{|column| [column.name, column.sql_type]}.to_h.
This will provide a nice hash.
for example:
{
id => int(4),
created_at => datetime
}
For a more compact format, and less typing just:
Portfolio.column_types
I am using rails 6.1 and have built a simple rake task for this.
You can invoke this from the cli using rails db:list[users] if you want a simple output with field names. If you want all the details then do rails db:list[users,1].
I constructed this from this question How to pass command line arguments to a rake task about passing command line arguments to rake tasks. I also built on #aaron-henderson's answer above.
# run like `rails db:list[users]`, `rails db:list[users,1]`, `RAILS_ENV=development rails db:list[users]` etc
namespace :db do
desc "list fields/details on a model"
task :list, [:model, :details] => [:environment] do |task, args|
model = args[:model]
if !args[:details].present?
model.camelize.constantize.column_names.each do |column_name|
puts column_name
end
else
ActiveRecord::Base.connection.tables.each do |table_name|
next if table_name != model.underscore.pluralize
ActiveRecord::Base.connection.columns(table_name).each do |c|
puts "Name: #{c.name} | Type: #{c.type} | Default: #{c.default} | Limit: #{c.limit} | Precision: #{c.precision} | Scale: #{c.scale} | Nullable: #{c.null} "
end
end
end
end
end