I have the following Ruby script: It creates a database, reads a csv file and inserts each row into the database.
require "sqlite3"
require "csv"
require "pp"
begin
db = SQLite::Database.new("myDB.db")
db.execute("CREATE TABLE IF NOT EXISTS MYTABLE(Id INTEGER PRIMARY KEY AUTOINCREMENT,
stations TEXT, dayparts TEXT, age TEXT, rtg DOUBLE, reach DOUBLE")
myData = {}
CSV.foreach("test_file.csv", :headers=>true, :header_converters => :symbol, :converters => :all)
do |row|
row.to_hash.each do |key, value|
mydata[key.to_sym] = value
end
db.execute("INSERT INTO myDB(stations, dayparts, age, rtg, reach) VALUES(?,?,?,?,?)",
myDATA[:stations], myData[:dayparts], myData[:age], myData[:rtg], mydata[:dlyrch000])
end
rescue SQLite3::Exception => e
puts "Exception occured"
puts e
ensure
db.close if db
end
When I run this script with static data. that is this line:
db.execute("INSERT INTO myDB(stations, dayparts, age, rtg, reach) VALUES(?,?,?,?,?)",
myDATA[:stations], myData[:dayparts], myData[:age], myData[:rtg], mydata[:dlyrch000])
is replaced by this:
db.execute("INSERT INTO myDB(stations, dayparts, age, rtg, reach) VALUES(?,?,?,?,?)",
test, test, 33, .8989, .23434)
A database is created with this data.
But when I try this script as above it throws an exception:
syntax error, unexpected ",", expecting ')' ... rtg, reach) VALUES (?,?,?,?,?)",
---> myData [:stations], myData[....
etc.
I have tried different options but cannot seem to get around this. Can someone please help me with this
There are three syntax errors I can see.
The CREATE TABLE SQL statement as a missing close parenthesis before the closing quote. It should look like
db.execute("CREATE TABLE IF NOT EXISTS MYTABLE(Id INTEGER PRIMARY KEY AUTOINCREMENT,
stations TEXT, dayparts TEXT, age TEXT, rtg DOUBLE, reach DOUBLE)")
The block for CSV.foreach has to start on the same line as the closing parenthesis for the method call, like this
CSV.foreach("test_file.csv", :headers => true, :header_converters => :symbol, :converters => :all) do |row|
...
enc
The database constructor uses SQLite when it should be SQLite3. Like this
db = SQLite3::Database.new("myDB.db")
I can't see anything wrong with the part of your code that is raising an error, but I presume you aren't showing the current version of your program as it is a long way from getting as far as that.
Related
I have a rails data migration (postgres db) where I have to use pure sql to convert the data due to some model restrictions. The data is stored as json as a string, but I need it to be a usable hash for other purposes.
My migration works to convert it to the hash. However, my down method ends up just deleting the data or leaving it as an empty {}. Btw to clear up any confusion, my column name is actually saved as data in table Games
Based on my up method, how would i properly reverse the migration using sql only?
class ConvertGamesDataToJson < ActiveRecord::Migration[6.0]
def up
statement = <<~SQL
update games set data = regexp_replace(trim(both '"' from data::text), '\\\\"', '"', 'g')::jsonb;
SQL
ActiveRecord::Base.connection.execute(statement)
# this part works!
end
def down
statement = <<~SQL
update games set data = to_json(data::text)::jsonb;
SQL
ActiveRecord::Base.connection.execute(statement)
end
end
Here is how the it looks after properly converting it
data: {
"id"=>"d092a-f2323",
"recent"=>'yes',
"note"=>"some text",
"order"=>1
}
how it is before the migration and what it needs to rollback to:
data:
"{
\"id\":\"d092a-f2323\",
\"recent\":\"yes\",
\"note\":\"some text\",
\"order\":1,
}"
If you're displaying a data structure in the rails console, those \" aren't really there. They're just formatting because the console has wrapped the string in ". For example...
[2] pry(main)> %{"up": "down"}
=> "\"up\": \"down\""
But if we print it...
[3] pry(main)> puts %{"up": "down"}
"up": "down"
Given that is a JSON string, you can simply change the type of the column to jsonb and be done with it.
-- up
alter table games alter column data type jsonb USING data::jsonb;
-- down
alter table games alter column data type text;
Postgres doesn't know how to automatically cast text to jsonb, so we need to tell it. using data::jsonb does a simple cast of the text to jsonb. It can cast jsonb to text just fine.
You can do this in a migration with change_column.
def up
change_column :users, :data, :jsonb, using: 'data::jsonb'
end
def down
change_column :users, :data, :text
end
I built a scraper that saves to a .csv file and am now attempting to save rows from that .csv file to an sqlite3 database with an IF statement, but it's not working. I've tried formatting the values in a dozen different ways and am getting nowhere.
"Match" prints every time the IF statement is True, but the row doesn't get added to the sqlite database. Calling cur.fetchall()/one()/etc results in 'None' being returned.
db = sqlite3.connect(':memory:')
cur = db.cursor()
cur.execute("DROP TABLE IF EXISTS jobs_table")
cur.execute('''CREATE TABLE IF NOT EXISTS
jobs_table(id TEXT,
date TEXT,
company TEXT,
position TEXT,
tags TEXT,
description TEXT,
url TEXT)''')
skills = ('python')
for row in csv_data:
if skills in row.get('description').lower():
print('')
print('Match!')
cur.execute("INSERT INTO jobs_table(id,
date,
company,
position,
tags,
description,
url) VALUES(:id,
:epoch,
:date,
:company,
:position,
:tags,
:description,
:url)", row)
I assume the problem is in my cur.execute() function, but I can't figure out how else it should be run. Any takers?
If you are calling cur.fetchone() right after the cur.execute() is normal to get None (or [] for cur.fetchall()). You need to execute a query first to get the results, for example cur.execute("SELECT * FROM jobs_table").
Select decrypt(PRODUCT_NUMBER,'123456789') as PRODUCT_NUMBER FROM Test
PRODUCT_NUMBER is a column in Test Table and contains Encrypted data Decrypt() is a function created and working fine.
When i run this Sql on Oracle SQL Developer it gives correct Result but when i run the same on JSP it gives me error on Function.
In JSP i call this by:
String sql = "Select decrypt(PRODUCT_NUMBER,'123456789') as PRODUCT_NUMBER FROM Test";
rs = conn.executeQuery(sql);
I think it takes PRODUCT_NUMBER as String ('PRODUCT_NUMBER') and Not as Column Name so it gives error.
java.sql.SQLException: ORA-01465: invalid hex number
This is the Decrypt Function
create or replace FUNCTION decrypt(p_raw IN RAW, p_key IN VARCHAR2) RETURN VARCHAR2 IS
v_retval RAW(255);
p_key2 RAW(255);
BEGIN
p_key2 := utl_raw.cast_to_raw(p_key);
dbms_obfuscation_toolkit.DES3Decrypt
(
input => p_raw,
key => p_key2,
which => 1,
decrypted_data => v_retval
);
RETURN RTRIM(utl_raw.cast_to_varchar2(v_retval), CHR(0));
END decrypt;
Solved!!
Query and Function both work perfect.
Actually My Co Developer had pointed the Connection to another replica instance of the DB that contained -1 in some columns so that's why it was giving error.
I reverted that and query worked like a charm.
Thanks for your time Everyone:)
This has been really difficult to find information on. The crux of it all is that I've got a Rails 3.2 app that accesses a MySQL database table with a column of type POINT. Without non-native code, rails doesn't know how to interpret this, which is fine because I only use it in internal DB queries.
The problem, however, is that it gets cast as an integer, and forced to null if blank. MySQL doesn't allow null for this field because there's an index on it, and integers are invalid, so this effectively means that I can't create new records through rails.
I've been searching for a way to change the value just before insertion into the db, but I'm just not up enough on my rails lit to pull it off. So far I've tried the following:
...
after_validation :set_geopoint_blank
def set_geopoint_blank
raw_write_attribute(:geopoint, '') if geopoint.blank?
#this results in NULL value in INSERT statement
end
---------------------------
#thing_controller.rb
...
def create
#thing = Thing.new
#thing.geopoint = 'GeomFromText("POINT(' + lat + ' ' + lng + ')")'
#thing.save
# This also results in NULL and an error
end
---------------------------
#thing_controller.rb
...
def create
#thing = Thing.new
#thing.geopoint = '1'
#thing.save
# This results in `1` being inserted, but fails because that's invalid spatial data.
end
To me, the ideal would be to be able to force rails to put the string 'GeomFromText(...)' into the insert statement that it creates, but I don't know how to do that.
Awaiting the thoughts and opinions of the all-knowing community....
Ok, I ended up using the first link in steve klein's comment to just insert raw sql. Here's what my code looks like in the end:
def create
# Create a Thing instance and assign it the POSTed values
#thing = Thing.new
#thing.assign_attributes(params[:thing], :as => :admin)
# Check to see if all the passed values are valid
if #thing.valid?
# If so, start a DB transaction
ActiveRecord::Base.transaction do
# Insert the minimum data, plus the geopoint
sql = 'INSERT INTO `things`
(`thing_name`,`thing_location`,`geopoint`)
values (
"tmp_insert",
"tmp_location",
GeomFromText("POINT(' + params[:thing][:lat].to_f.to_s + ' ' + params[:thing][:lng].to_f.to_s + ')")
)'
id = ActiveRecord::Base.connection.insert(sql)
# Then load in the newly-created Thing instance and update it's values with the passed values
#real_thing = Thing.find(id)
#real_thing.update_attributes(b, :as => :admin)
end
# Notify the user of success
flash[:message] = { :header => 'Thing successfully created!' }
redirect_to edit_admin_thing_path(#real_thing)
else
# If passed values not valid, alert and re-render form
flash[:error] = { :header => 'Oops! You\'ve got some errors:', :body => #thing.errors.full_messages.join("</p><p>").html_safe }
render 'admin/things/new'
end
end
Not beautiful, but it works.
I know that you can ask ActiveRecord to list tables in console using:
ActiveRecord::Base.connection.tables
Is there a command that would list the columns in a given table?
This will list the column_names from a table
Model.column_names
e.g. User.column_names
This gets the columns, not just the column names and uses ActiveRecord::Base::Connection, so no models are necessary. Handy for quickly outputting the structure of a db.
ActiveRecord::Base.connection.tables.each do |table_name|
puts table_name
ActiveRecord::Base.connection.columns(table_name).each do |c|
puts "- #{c.name}: #{c.type} #{c.limit}"
end
end
Sample output: http://screencast.com/t/EsNlvJEqM
Using rails three you can just type the model name:
> User
gives:
User(id: integer, name: string, email: string, etc...)
In rails four, you need to establish a connection first:
irb(main):001:0> User
=> User (call 'User.connection' to establish a connection)
irb(main):002:0> User.connection; nil #call nil to stop repl spitting out the connection object (long)
=> nil
irb(main):003:0> User
User(id: integer, name: string, email: string, etc...)
If you are comfortable with SQL commands, you can enter your app's folder and run rails db, which is a brief form of rails dbconsole. It will enter the shell of your database, whether it is sqlite or mysql.
Then, you can query the table columns using sql command like:
pragma table_info(your_table);
complementing this useful information, for example using rails console o rails dbconsole:
Student is my Model, using rails console:
$ rails console
> Student.column_names
=> ["id", "name", "surname", "created_at", "updated_at"]
> Student
=> Student(id: integer, name: string, surname: string, created_at: datetime, updated_at: datetime)
Other option using SQLite through Rails:
$ rails dbconsole
sqlite> .help
sqlite> .table
ar_internal_metadata relatives schools
relationships schema_migrations students
sqlite> .schema students
CREATE TABLE "students" ("id" integer PRIMARY KEY AUTOINCREMENT NOT NULL, "name" varchar, "surname" varchar, "created_at" datetime NOT NULL, "updated_at" datetime NOT NULL);
Finally for more information.
sqlite> .help
Hope this helps!
You can run rails dbconsole in you command line tool to open sqlite console. Then type in .tables to list all the tables and .fullschema to get a list of all tables with column names and types.
To list the columns in a table I usually go with this:
Model.column_names.sort.
i.e. Orders.column_names.sort
Sorting the column names makes it easy to find what you are looking for.
For more information on each of the columns use this:
Model.columns.map{|column| [column.name, column.sql_type]}.to_h.
This will provide a nice hash.
for example:
{
id => int(4),
created_at => datetime
}
For a more compact format, and less typing just:
Portfolio.column_types
I am using rails 6.1 and have built a simple rake task for this.
You can invoke this from the cli using rails db:list[users] if you want a simple output with field names. If you want all the details then do rails db:list[users,1].
I constructed this from this question How to pass command line arguments to a rake task about passing command line arguments to rake tasks. I also built on #aaron-henderson's answer above.
# run like `rails db:list[users]`, `rails db:list[users,1]`, `RAILS_ENV=development rails db:list[users]` etc
namespace :db do
desc "list fields/details on a model"
task :list, [:model, :details] => [:environment] do |task, args|
model = args[:model]
if !args[:details].present?
model.camelize.constantize.column_names.each do |column_name|
puts column_name
end
else
ActiveRecord::Base.connection.tables.each do |table_name|
next if table_name != model.underscore.pluralize
ActiveRecord::Base.connection.columns(table_name).each do |c|
puts "Name: #{c.name} | Type: #{c.type} | Default: #{c.default} | Limit: #{c.limit} | Precision: #{c.precision} | Scale: #{c.scale} | Nullable: #{c.null} "
end
end
end
end
end