Import SQL file to Rails 4 ActiveRecord db? - sql

I've looked over several other questions on here, and they're vaguely similar, but not exactly what I'm looking for.
What I'm trying to do is import/"convert" a *.sql file which contains 8 tables, each of which contain roughly 24 columns. This file is actually fairly flat file, seeing as though the only queries that worked previous had to do with associating a shared :id between tables (so, SELECT * FROM table1, table2 WHERE id = '1' would pull all results, which was fine at the time).
I've searched around, but can't find a clever way to do this, so I'm asking you Rails pros for help now.

I'm assuming what you want is basically to convert your SQL file into a Rails database schema file without having to go through and do this yourself manually.
One quick way to do this would be to manually execute the SQL file, perhaps by logging into your database and loading the file that way, or by doing something like what was done in this question:
ActiveRecord::Base.connection.execute(IO.read("path/to/file"))
Once you have the schema that was defined in your .sql file actually loaded into your database, you will want to follow the steps outlined in this question:
First run rake db:schema:dump which will generate a db/schema.rb database file based on the current state of the database.
From here, you can create a db/migrate/001_original_schema.rb migration that references the schema.rb file as follows:
class OriginalDatabaseMigration < ActiveRecord::Migration
def self.up
# contents of schema.rb here
end
def self.down
# drop all the tables
end
end

If I understood your question right you need to populate your db from .sql file. I am doing it this way:
connection = ActiveRecord::Base.connection
ql = File.read('db/some_sql_file.sql')
statements = sql.split(/;$/)
statements.pop
ActiveRecord::Base.transaction do
statements.each do |statement|
connection.execute(statement)
end
end
Put your sql file to db folder.

One way I was able to do this - using rails dbconsole
.import FILE TABLE Import data from FILE into TABLE
And essentially .import ./path/to/file TABLE_NAME
Works like a champ.

I have faced the same problem, i just created a script and parsed
all the SQL sentences adding 'execute("' at the begining and '")' at the end of each line.
Then i created a new migartion as usual, and pasted all the output on the migration up script. That works for me.
Be aware of avoid any comments on the SQL file so the parsing will be easier.

Related

Trying to create a table and load data into same table using Databricks and SQL

I Googled for a solution to create a table, using Databticks and Azure SQL Server, and load data into this same table. I found some sample code online, which seems pretty straightforward, but apparently there is an issue somewhere. Here is my code.
CREATE TABLE MyTable
USING org.apache.spark.sql.jdbc
OPTIONS (
url "jdbc:sqlserver://server_name_here.database.windows.net:1433;database = db_name_here",
user "u_name",
password "p_wd",
dbtable "MyTable"
);
Now, here is my error.
Error in SQL statement: SQLServerException: Invalid object name 'MyTable'.
My password, unfortunately, has spaces in it. That could be the problem, perhaps, but I don't think so.
Basically, I would like to get this to recursively loop through files in a folder and sub-folders, and load data from files with a string pattern, like 'ABC*', and load recursively all these files into a table. The blocker, here, is that I need the file name loaded into a field as well. So, I want to load data from MANY files, into 4 fields of actual data, and 1 field that captures the file name. The only way I can distinguish the different data sets is with the file name. Is this possible? Or, is this an exercise in futility?
my suggestion is to use the Azure SQL Spark library, as also mentioned in documentation:
https://docs.databricks.com/spark/latest/data-sources/sql-databases-azure.html#connect-to-spark-using-this-library
The 'Bulk Copy' is what you want to use to have good performances. Just load your file into a DataFrame and bulk copy it to Azure SQL
https://docs.databricks.com/data/data-sources/sql-databases-azure.html#bulk-copy-to-azure-sql-database-or-sql-server
To read files from subfolders, answer is here:
How to import multiple csv files in a single load?
I finally, finally, finally got this working.
val myDFCsv = spark.read.format("csv")
.option("sep","|")
.option("inferSchema","true")
.option("header","false")
.load("mnt/rawdata/2019/01/01/client/ABC*.gz")
myDFCsv.show()
myDFCsv.count()
Thanks for a point in the right direction mauridb!!

Saving an sqllite3 database as csv file in Ruby on Rails 5

I am currently working on a project and I want to know how to save an sqllite database in rails as a csv file. I want it when you click the button, the current database on the system download. Can anybody help me? Thanks!
Your problem isn't really specific to Rails. Instead, you're mostly dealing with an administrative issue. You should write a script to export your database as csv, something like this:
#!/bin/bash
./bin/sqlite3 ./my_app/db/my_database.db <<!
.headers on
.mode csv
.output my_output_file.csv
select * from my_table;
!
This script exports a single table. If you have additional tables, you'll want to add them to your script.
The only Rails related issue is the matter of calling that script. Save the script within your application structure; I'd suggest my_app/assets or some similar location.
Now you can run that script using system(command) where command is the absolute path for your script, within a set of double-quotes.

Rails 3 Generate Seed file from single Database Table

I have tried scouring the mighty google, but can't seem to find what I am looking for (probably because what I am wanting to do is stupid)
But, is there a way to create a seed file from just a single database table instead of dumping the entire database contents?
Essentially, I have a Products table and I am just wanting to generate a seed file from this table.
Found what I was looking for using this Rails Gem https://github.com/rhalff/seed_dump
Use this Code:
rake db:seed:dump MODELS=User
Obviously replace User with the model name you are wanting to work with, or a comma separated list of models.

Seeding thousands of records in Rails 3

I have several tables that need to be populated when I move my project to production, each of these tables has several thousand rows. I have the data stored in a CSV file now, but using the seed.rb file seems like it would be cumbersome because the data from my CSV file would have to be formatted to meet the seed.rb format. If this were only a handful of rows, it wouldn't such a problem. What would be the best/easiest way to get this data loaded?
I would probably use a little custom script and the faster_csv gem which has good tools to parse .csv files quickly. Then you can map the fields to model attributes.
I would implement this via TDD as model methods and use ActiveRecords's create method to instantiate instances. While this is slower than writing SQL straight out, it's safer in that your data will run through all the model validations and you have a better confidence in the data integrity.
Flushing out data integrity issues from legacy data import up front will save you a lot of trouble later.
If I were doing this using MySQL, I'd use MySQL's load data function, like
load data infile '/my/rails/project/data.csv' replace into table table_name fields terminated by 'field_terminator' lines terminated by 'line_terminator';
If the tables' designs do not change frequently, you could put a such a statement into a perl, ruby, or shell script.
Like others have mentioned many db's have bulk load support. But if your looking for a Rails style solution; ar-extensions has bulk insert
http://rorstuff.blogspot.com/2010/05/activerecord-bulk-insertion-of-data-in.html
https://github.com/zdennis/ar-extensions
You also can checkout ActiveWarehouse
https://github.com/zdennis/ar-extensions
fast_seeder gem will help you. It populates database from CSV files using multiple inserts and supports different DB adapters
As most of the answers are outdated, this gem will help you: https://github.com/zdennis/activerecord-import.
E.g. if you have a collection of books, you can use:
Book.import(books)
It will execute only one SQL statement.
The gem also works with associations.

Is it possible to use an external SQL file in a Rails migration?

I have to create a Rails migration which creates many triggers and stored procedures.
Normally one would do that using the execute method, but because of the size of the statements, I'd rather keep them in an external file and reference it from the migration.
How can I do that? Is it even possible?
You can just store them in a text file and read them in via a File object.
sql = ""
source = File.new("./sql/procedures.sql", "r")
while (line = source.gets)
sql << line
end
source.close
execute sql
It's ugly, but works. I'd strongly recommend keeping the stored procedures/triggers inside migrations for easy rollbacks.
If you do the "external file" method, you'll need to maintain two additional files per migration, one for adding all the stuff, and one for dropping in in case of a:
rake db:rollback
I did the following where we needed:
class RawSqlMigration < ActiveRecord::Migration
def up
execute File.read(File.expand_path('../../sql_migrations/file.sql', __FILE__))
end
end
Mike’s answer works without problems if you have only one statement in the file, but if there is more statements (multiple inserts and updates, for example), ActiveRecord will fail as it doesn’t support multiple statements with one call by default.
One solution would be to modify ActiveRecord to support multiple statements, as instructed here.
Other solution would be to ensure that your SQL file contains only one statement per row and use a loop like
source = File.open "db/foo.sql", "r"
source.readlines.each do |line|
line.strip!
next if line.empty? # ensure that rows that contains newlines and nothing else does not get processed
execute line
end
source.close