I've got a .sql file that I'd like to load into my Rails database using a Rake task. How can I do this?
The easiest way:
bundle exec rails db < $SQL_FILE
example:
bundle exec rails db < my_db.sql
The Easy Way
This works for simple cases.
ActiveRecord::Base.connection.execute(IO.read("path/to/file"))
Solution found on the Ruby On Rails mailing list from 2006 (but still works in 2011 on Rails 3.1).
Footnotes
This related question implied this solution, but rejected it for big imports. I wanted to show it explicitly, since it works for smaller ones.
The file I was trying to import contained a LOCK TABLES followed by an insert. The data was for a MySQL database. Mysql2 said it had an invalid SQL syntax error until I removed the lock and unlock statements.
On MySQL this gave me a syntax error. Splitting the sql into statements made it work.
sql = File.read(sql_file)
statements = sql.split(/;$/)
statements.pop # remove empty line
ActiveRecord::Base.transaction do
statements.each do |statement|
connection.execute(statement)
end
end
Related
In the database, I have multiple materialized views with big definitions. I also have multiple migrations that change the definitions of some of these materialized views using DROP and CREATE statements. Thus, we often are dropping / recreating the same views over and over, with small changes. These (rather bulky) statements are now stored inside strings:
class MyMigrationName < ActiveRecord::Migration[5.2]
def up
sql = <<~SQL
...
create materialized view if not exists foo_1 as ... ;
create materialized view if not exists foo_2 as ... ;
...
SQL
execute sql
end
def down
...
end
I am considering switching from this current approach to a different one, where the SQL code is stored inside separate SQL files, for example in db/migrate/concerns/create_foo_matviews.sql. The code is read from the file and executed from inside the migrations, like so:
class MyMigrationName < ActiveRecord::Migration[5.2]
def up
execute File.read(File.expand_path('./concerns/create_foo_matviews.rb', __FILE__))
end
def down
...
end
The pros of this approach are:
It is easier to see the differences between the old and the new SQL code using git diff (especially important given that materialized views' definitions are big, but the actual changes in migrations are relatively small).
The SQL file adds syntax highlighting to the SQL code.
There is less copy/pasted code if I only change the relevant parts in the SQL file.
Are there any problems associated with this proposed approach? If yes, what would be an alternative solution to maximize maintainability?
See also
Is it possible to use an external SQL file in a Rails migration?
Running sql file using rails migration file
Execute SQL-Statement from File with ActiveRecord
I'd leave it in the Migration.
Mainly because the migration then contains everything that actually makes up the DB change.
You would need to have two external SQL files (up and down) that I need to search/find first before I understand what the migration does.
Depending on the Editor you are using, you will get (limited) syntax highlighting
The migrations that execute custom SQL would all look the same, just the name of the external file would be different.
What problem are you trying to solve? Just the "bulky" strings? I don't think that this is problem (to be honest once the migration is run, you not go back to it anyhow) that is worth spennding a lot of time on. Just to the simplest thing: SQL in heredoc string.
There are also gems that allow you to create (materialized) views with normal migration code (by adding support for create_view or similar) but i'd not add an additional dependency for something this simple.
Also consider changing from schema.rb to structure.sql, if not yet done.
Sound like you want to create your own helpers to create materialized views, something like add_index or add_column.
You could make a module named like MaterializedMigrations in your lib directory. then you can required it in a initializer and for last you include it in your migration code, like this:
class MyMigrationName < ActiveRecord::Migration[5.2]
include MaterializedMigrations
def up
create_materialized_view("name_of_view")
end
end
The helper API is only a suggestion, you could design better API for your use cases.
I have a very strange 'reload.sql' file that I need to use to build a database.
It references about 200 XXX.dat files with straight-up readable data (although useless without explanations regarding the meaning of the fields).
I have tried msssql server, mysql workbench (on a server local-hosted on wamp), and directly accessing it through DBeaver and IBConsole, but I cannot manage to execute/build it.
It uses a weird syntax. There are elements like
begin
...
end
go
that hinted me towards T-SQL, but using sqlcmd on it gave me thousands upon thousands of errors regarding keywords.
Specifically, the very first batch of executable lines says
SET OPTION date_order = 'YMD'
go
SET OPTION PUBLIC.preserve_source_format = 'OFF'
go
SET TEMPORARY OPTION tsql_outer_joins = 'ON'
go
SET TEMPORARY OPTION st_geometry_describe_type = 'binary'
go
SET TEMPORARY OPTION st_geometry_on_invalid = 'Ignore'
go
SET TEMPORARY OPTION non_keywords = 'attach,compressed,detach,kerberos,nchar,nvarchar,refresh,varbit'
go
which generates about 150 errors 'Incorrect syntax near OPTION keyword' on its own, and according to google is part of a 'rexx' procedure but 'date_order' should then be 'DATFMT', right?
Another track is that of SyBase, but I cannot for the life of me get it to work (through my trials I did manage to build a .db file, that, well, is useless to me since I can't build it either..).
I've tried accessing it through ODBC pilots as well but none worked (the paradox ODBC did not crash, but said there was an error with a FROM clause, which are generated automatically...).
I need to know a way to build a database from this file or directly access the data it references, which I can't really post since it contains private medical data.
Also what madman came up with this.
The very first google link (for me anyway) against 'st-geometry-describe-option' shows this is a SAP SQL Anywhere database i.e. http://dcx.sybase.com/1200/en/dbadmin/st-geometry-describe-option.html
So I would suggest starting from the SQL Anywhere documentation and you will need to install the database software beforehand.
I would think that there would be more information on this particular topic, but here goes -- I need to be able to see what my SQLite tables in my Rails app look like. I've tried a lot of the suggestions I've seen online, but they all give me errors, or I'm not using the right commands.
Here's what I've been trying
rails db
(goes into sqlite db)
sqlite>.table
(shows relevant table)
sqlite>select * from table1;
Receive error:
Error: unknown command or invalid arguments: "select". Enter ".help" for help
I've been trying to figure out why this won't work, but I'm a bit stumped.
You can use DB browser for SQLite https://sqlitebrowser.org/ to view and manage SQLite database tables.
I have a Rails 4 application that I use in conjunction with sidekiq to run asynchronous jobs. One of the jobs I normally run outside of my Rails application is a large set of complex SQL queries that cannot really be modeled by ActiveRecord. The connection this set of SQL queries has with my Rails app is that it should be executed anytime one of my controller actions is invoked.
Ideally, I'd queue a job from my Rails application within the controller for Sidekiq to go ahead and run the queries. Right now they're stored in an external file, and I'm not entirely sure what the best way is to have Rails run the said SQL.
Any solutions are appreciated.
I agree with Sharagoz, if you just need to run a specific query, the best way is to send the query string directly into the connection, like:
ActiveRecord::Base.connection.execute(File.read("myquery.sql"))
If the query is not static and you have to compose it, I would use Arel, it's already present in Rails 4.x:
https://github.com/rails/arel
You didn't say what database you are using, so I'm going to assume MySQL.
You could shell out to the mysql binary to do the work:
result = `mysql -u #{user} --password #{password} #{database} < #{huge_sql_filename}`
Or use ActiveRecord::Base.connection.execute(File.read("huge.sql")), but it won't work out of the box if you have multiple SQL statements in your SQL file.
In order to run multiple statements you will need to create an initializer that monkey patches the ActiveRecord::Base.mysql2_connection to allow setting MySQL's CLIENT_MULTI_STATEMENTS and CLIENT_MULTI_RESULTS flags.
Create a new initializer config/initializers/mysql2.rb
module ActiveRecord
class Base
# Overriding ActiveRecord::Base.mysql2_connection
# method to allow passing options from database.yml
#
# Example of database.yml
#
# login: &login
# socket: /tmp/mysql.sock
# adapter: mysql2
# host: localhost
# encoding: utf8
# flags: 131072
#
# #param [Hash] config hash that you define in your
# database.yml
# #return [Mysql2Adapter] new MySQL adapter object
#
def self.mysql2_connection(config)
config[:username] = 'root' if config[:username].nil?
if Mysql2::Client.const_defined? :FOUND_ROWS
config[:flags] = config[:flags] ? config[:flags] | Mysql2::Client::FOUND_ROWS : Mysql2::Client::FOUND_ROWS
end
client = Mysql2::Client.new(config.symbolize_keys)
options = [config[:host], config[:username], config[:password], config[:database], config[:port], config[:socket], 0]
ConnectionAdapters::Mysql2Adapter.new(client, logger, options, config)
end
end
end
Then update config/database.yml to add flags:
development:
adapter: mysql2
database: app_development
username: user
password: password
flags: <%= 65536 | 131072 %>
I just tested this on Rails 4.1 and it works great.
Source: http://www.spectator.in/2011/03/12/rails2-mysql2-and-stored-procedures/
Executing one query is - as outlined by other people - quite simply done through
ActiveRecord::Base.connection.execute("SELECT COUNT(*) FROM users")
You are talking about a 20.000 line sql script of multiple queries. Assuming you have the file somewhat under control, you can extract the individual queries from it.
script = Rails.root.join("lib").join("script.sql").read # ah, Pathnames
# this needs to match the delimiter of your queries
STATEMENT_SEPARATOR = ";\n\n"
ActiveRecord::Base.transaction do
script.split(STATEMENT_SEPARATOR).each do |stmt|
ActiveRecord::Base.connection.execute(stmt)
end
end
If you're lucky, then the query delimiter could be ";\n\n", but this depends of course on your script. We had in another example "\x0" as delimiter. The point is that you split the script into queries to send them to the database. I wrapped it in a transaction, to let the database know that there is coming more than one statement. The block commits when no exception is raised while sending the script-queries.
If you do not have the script-file under control, start talking to those who control it to get a reliable delimiter. If it's not under your control and you cannot talk to the one who controls it, you wouldn't execute it, I guess :-).
UPDATE
This is a generic way to solve this. For PostgreSQL, you don't need to split the statements manually. You can just send them all at once via execute. For MySQL, there seem to be solutions to get the adapter into a CLIENT_MULTI_STATEMENTS mode.
If you want to execute raw SQL through active record you can use this API:
ActiveRecord::Base.connection.execute("SELECT COUNT(*) FROM users")
If you are running big SQL every time, i suggest you to create a sql view for it. It be boost the execution time. The other thing is, if possible try to split all those SQL query in such a way that it will be executed parallely instead of sequentially and then push it to sidekiq queue.
You have to use ActiveRecord::Base.connection.execute or ModelClass.find_by_sql to run custom SQL.
Also, put an eye on ROLLBACK transactions, you will find many places where you dont need such ROLLBACK feature. If you avoid that, the query will run faster but it is dangerous.
Thanks all i can suggest.
use available database tools to handle the complex queries, such as views, stored procedures etc and call them as other people already suggested (ActiveRecord::Base.connection.execute and ModelClass.find_by_sql for example)- it might very well cut down significantly on query preparation time in the DB and make your code easier to handle
http://dev.mysql.com/doc/refman/5.0/en/create-view.html
http://dev.mysql.com/doc/connector-cpp/en/connector-cpp-tutorials-stored-routines-statements.html
abstract your query input parameters into a hash so you can pass it on to sidekiq, don't send SQL strings as this will probably degrade performance (due to query preparation time) and make your life more complicated due to funny SQL driver parsing bugs
run your complex queries in a dedicated named queue and set concurrency to such a value that will prevent your database of getting overwhelmed by the queries as they smell like they could be pretty db heavy
https://github.com/mperham/sidekiq/wiki/API
https://github.com/mperham/sidekiq/wiki/Advanced-Options
have a look at Squeel, its a great addition to AR, it might be able to pull some of the things you are doing
https://github.com/activerecord-hackery/squeel
http://railscasts.com/episodes/354-squeel
I'll assume you use MySQL for now, but your mileage will vary depending on the DB type that you use. For example, Oracle has some good gems for handling stored procedures, views etc, for example https://github.com/rsim/ruby-plsql
Let me know if some of this stuff doesn't fit your use case and I'll expand
I see this post is kind of old. But I would like to add my solution to it. I was in a similar situation; I also needed a way to force feed "PRAGMA foreign_keys = on;" into my sqlite connection (I could not find a previous post that spelled it out how to do it.) Anywho, this worked like a charm for me. It allowed me to write "pretty" sql and still get it executed. Blank lines are ignored by the if statement.
conn = ActiveRecord::Base.establish_connection(adapter:'sqlite3',database:DB_NAME)
sqls = File.read(DDL_NAME).split(';')
sqls.each {|sql| conn.connection.execute(sql<<';') unless sql.strip.size == 0 }
conn.connection.execute('PRAGMA foreign_keys = on;')
I had the same problem with a set of sql statements that I needed to execute all in one call to the server. What worked for me was to set up an initializer for Mysql2 adapter (as explained in infused answer) but also do some extra work to process multiple results. A direct call to ActiveRecord::Base.connection.executewould only retrieve the first result and issue an Internal Error.
My solution was to get the Mysql2 adapter and work directly with it:
client = ActiveRecord::Base.connection.raw_connection
Then, as explained here, execute the query and loop through the results:
client.query(multiple_stms_query)
while client.next_result
result = client.store_result
# do something with it ...
end
I have been researching a way to get the SQL statements that are built by a generated Migration file. These extend Doctrine_Migration_Base. Essentially I would like to save the SQL as change scripts.
The execution path leads me to Doctrine_Export which has methods that build the SQL statement and executes them. I have found no way of asking for just them. The export methods found in Doctrine_Export only operate on Doctrine_Record models and not Migration scripts.
From the command line './doctrine migrate version#' the path goes:
Doctrine_Cli::run(cmd)
Doctrine_Task_Migrate::setArguments(args)
Doctrine_Task_Migrate::execute()
Doctrine_Migration::migrate(to)
Doctrine_Migration_Process::Doctrine_Export::various
create, drop, alter methods with sql
equivalents.
Has anyone tackled this before? I really would not like to change Doctrine base files. Any help is greatly appreciated.
Could you make a dev server, and do the migration on that, storing a SQL Trace as you go?you don't have to keep the results, but you would get a list of every command.
Taking into account Rob Farley's suggestion, I modified:
Doctrine_Core::migrate
Doctrine_Task_Migrate::execute
When the execute method is called the optional argument 'dryRun' is checked. If true
then a 'Doctrine_Connection_Profiler' instance is created. The 'dryRun' value is then passed onto
the 'Doctrine_Core::migrate' method. The 'dryRun' value of true allows the changes to rollback when done executing the SQL statements. When the method returns, the profiler is parsed and non-empty SQL statements
not containing 'migration_version' are saved and displayed to the terminal.