I have tried scouring the mighty google, but can't seem to find what I am looking for (probably because what I am wanting to do is stupid)
But, is there a way to create a seed file from just a single database table instead of dumping the entire database contents?
Essentially, I have a Products table and I am just wanting to generate a seed file from this table.
Found what I was looking for using this Rails Gem https://github.com/rhalff/seed_dump
Use this Code:
rake db:seed:dump MODELS=User
Obviously replace User with the model name you are wanting to work with, or a comma separated list of models.
Related
I Googled for a solution to create a table, using Databticks and Azure SQL Server, and load data into this same table. I found some sample code online, which seems pretty straightforward, but apparently there is an issue somewhere. Here is my code.
CREATE TABLE MyTable
USING org.apache.spark.sql.jdbc
OPTIONS (
url "jdbc:sqlserver://server_name_here.database.windows.net:1433;database = db_name_here",
user "u_name",
password "p_wd",
dbtable "MyTable"
);
Now, here is my error.
Error in SQL statement: SQLServerException: Invalid object name 'MyTable'.
My password, unfortunately, has spaces in it. That could be the problem, perhaps, but I don't think so.
Basically, I would like to get this to recursively loop through files in a folder and sub-folders, and load data from files with a string pattern, like 'ABC*', and load recursively all these files into a table. The blocker, here, is that I need the file name loaded into a field as well. So, I want to load data from MANY files, into 4 fields of actual data, and 1 field that captures the file name. The only way I can distinguish the different data sets is with the file name. Is this possible? Or, is this an exercise in futility?
my suggestion is to use the Azure SQL Spark library, as also mentioned in documentation:
https://docs.databricks.com/spark/latest/data-sources/sql-databases-azure.html#connect-to-spark-using-this-library
The 'Bulk Copy' is what you want to use to have good performances. Just load your file into a DataFrame and bulk copy it to Azure SQL
https://docs.databricks.com/data/data-sources/sql-databases-azure.html#bulk-copy-to-azure-sql-database-or-sql-server
To read files from subfolders, answer is here:
How to import multiple csv files in a single load?
I finally, finally, finally got this working.
val myDFCsv = spark.read.format("csv")
.option("sep","|")
.option("inferSchema","true")
.option("header","false")
.load("mnt/rawdata/2019/01/01/client/ABC*.gz")
myDFCsv.show()
myDFCsv.count()
Thanks for a point in the right direction mauridb!!
I've looked over several other questions on here, and they're vaguely similar, but not exactly what I'm looking for.
What I'm trying to do is import/"convert" a *.sql file which contains 8 tables, each of which contain roughly 24 columns. This file is actually fairly flat file, seeing as though the only queries that worked previous had to do with associating a shared :id between tables (so, SELECT * FROM table1, table2 WHERE id = '1' would pull all results, which was fine at the time).
I've searched around, but can't find a clever way to do this, so I'm asking you Rails pros for help now.
I'm assuming what you want is basically to convert your SQL file into a Rails database schema file without having to go through and do this yourself manually.
One quick way to do this would be to manually execute the SQL file, perhaps by logging into your database and loading the file that way, or by doing something like what was done in this question:
ActiveRecord::Base.connection.execute(IO.read("path/to/file"))
Once you have the schema that was defined in your .sql file actually loaded into your database, you will want to follow the steps outlined in this question:
First run rake db:schema:dump which will generate a db/schema.rb database file based on the current state of the database.
From here, you can create a db/migrate/001_original_schema.rb migration that references the schema.rb file as follows:
class OriginalDatabaseMigration < ActiveRecord::Migration
def self.up
# contents of schema.rb here
end
def self.down
# drop all the tables
end
end
If I understood your question right you need to populate your db from .sql file. I am doing it this way:
connection = ActiveRecord::Base.connection
ql = File.read('db/some_sql_file.sql')
statements = sql.split(/;$/)
statements.pop
ActiveRecord::Base.transaction do
statements.each do |statement|
connection.execute(statement)
end
end
Put your sql file to db folder.
One way I was able to do this - using rails dbconsole
.import FILE TABLE Import data from FILE into TABLE
And essentially .import ./path/to/file TABLE_NAME
Works like a champ.
I have faced the same problem, i just created a script and parsed
all the SQL sentences adding 'execute("' at the begining and '")' at the end of each line.
Then i created a new migartion as usual, and pasted all the output on the migration up script. That works for me.
Be aware of avoid any comments on the SQL file so the parsing will be easier.
I have used the seed file to generate data that needs to go into the DB, but now I need to generate some countries, provinces and cities into three different tables and I haven't been able to find a good way to put that into the seed.
However I have found a sql file that has all that information and that I should dump into the DB. My question is:
Where should I store that in my Rails folder as a convention?
Thanks
I would create a directory at db/seeds, and place that seed file, and any others you may have in there.
I have a SQL script that has four columns and about 100 records. I only need two columns. I want to transfer the two columns into my seeds.rb file so that I can have these records in my db when I deploy my app. What would be the easiest way possible to do this? How would it look in my seeds.rb file?
The first thing to do is get the database in the format you want, and then create a database dump of some sort. MySQL makes this easier than Sqlite. Put the INSERT statements into your file like this:
ActiveRecord::Base.connection.execute("INSERT INTO `example` (`abbreviation`,`name`)
VALUES
('ABC', 'Alphabet Broadcasting Company'),
('DEF', 'Denver Echo Factory'),
('GHI', 'Gimbal Helper Industries')
")
Although seeds.rb is a handy way of pre-populating certain critical things, like essential administrators or lookup tables for countries, it does become difficult to maintain over time as seeds.rb must always conform to the latest schema.
It may be easier to simply deploy a seed Sqlite file and migrate that instead. With MySQL you typically deploy and load in a seed database dump to get things started, then migrate and enhance as required.
I have several tables that need to be populated when I move my project to production, each of these tables has several thousand rows. I have the data stored in a CSV file now, but using the seed.rb file seems like it would be cumbersome because the data from my CSV file would have to be formatted to meet the seed.rb format. If this were only a handful of rows, it wouldn't such a problem. What would be the best/easiest way to get this data loaded?
I would probably use a little custom script and the faster_csv gem which has good tools to parse .csv files quickly. Then you can map the fields to model attributes.
I would implement this via TDD as model methods and use ActiveRecords's create method to instantiate instances. While this is slower than writing SQL straight out, it's safer in that your data will run through all the model validations and you have a better confidence in the data integrity.
Flushing out data integrity issues from legacy data import up front will save you a lot of trouble later.
If I were doing this using MySQL, I'd use MySQL's load data function, like
load data infile '/my/rails/project/data.csv' replace into table table_name fields terminated by 'field_terminator' lines terminated by 'line_terminator';
If the tables' designs do not change frequently, you could put a such a statement into a perl, ruby, or shell script.
Like others have mentioned many db's have bulk load support. But if your looking for a Rails style solution; ar-extensions has bulk insert
http://rorstuff.blogspot.com/2010/05/activerecord-bulk-insertion-of-data-in.html
https://github.com/zdennis/ar-extensions
You also can checkout ActiveWarehouse
https://github.com/zdennis/ar-extensions
fast_seeder gem will help you. It populates database from CSV files using multiple inserts and supports different DB adapters
As most of the answers are outdated, this gem will help you: https://github.com/zdennis/activerecord-import.
E.g. if you have a collection of books, you can use:
Book.import(books)
It will execute only one SQL statement.
The gem also works with associations.