I am trying to use Redis for my realtime search queries. After reading the article mentionedb below I tried the python scripts and it worked fine. But in database, I'm having two fields:
i) country_name
ii) country_id
How can I insert the country_id and make it searchable as discussed in the article.
The elements in a Redis Sorted Set are just strings. You can concatenate the name and id for storage purposes, and after querying you split the string back into fields.
Pseudo example:
# Store it
redis.zadd('countries', 0, 'Italy:6379', 0, 'foobar:42', ...
...
# Query
q = redis.zrangebylex('countries', ...
f = q.split(':')
print 'name: {}, id: {}'.format(f[0], f[1])
Related
I have a JSON API payload containing tablename, columnlist - how to build a SELECT query from it using pypika?
So far I have been able to use a string columnlist, but not able to do advanced querying using functions, analytics etc.
from pypika import Table, Query, functions as fn
def generate_sql (tablename, collist):
table = Table(tablename)
columns = [str(table)+'.'+each for each in collist]
q = Query.from_(table).select(*columns)
return q.get_sql(quote_char=None)
tablename = 'customers'
collist = ['id', 'fname', 'fn.Sum(revenue)']
print (generate_sql(tablename, collist)) #1
table = Table(tablename)
q = Query.from_(table).select(table.id, table.fname, fn.Sum(table.revenue))
print (q.get_sql(quote_char=None)) #2
#1 outputs
SELECT "customers".id,"customers".fname,"customers".fn.Sum(revenue) FROM customers
#2 outputs correctly
SELECT id,fname,SUM(revenue) FROM customers
You should not be trying to assemble the query in a string by yourself, that defeats the whole purpose of pypika.
What you can do in your case, that you have the name of the table and the columns coming as texts in a json object, you can use * to unpack those values from the collist and use the syntax obj[key] to get the table attribute with by name with a string.
q = Query.from_(table).select(*(table[col] for col in collist))
# SELECT id,fname,fn.Sum(revenue) FROM customers
Hmm... that doesn't quite work for the fn.Sum(revenue). The goal is to get SUM(revenue).
This can get much more complicated from this point. If you are only sending column names that you know to belong to that table, the above solution is enough.
But if you have complex sql expressions, making reference to sql functions or even different tables, I suggest you to rethink your decision of sending that as json. You might end up with something as complex as pypika itself, like a custom parser or wathever. than your better option here would be to change the format of your json response object.
If you know you only need to support a very limited set of capabilities, it could be feasible. For example, you can assume the following constraints:
all column names refer to only one table, no joins or alias
all functions will be prefixed by fn.
no fancy stuff like window functions, distinct, count(*)...
Then you can do something like:
from pypika import Table, Query, functions as fn
import re
tablename = 'customers'
collist = ['id', 'fname', 'fn.Sum(revenue / 2)', 'revenue % fn.Count(id)']
def parsed(cols):
pattern = r'(?:\bfn\.[a-zA-Z]\w*)|([a-zA-Z]\w*)'
subst = lambda m: f"{'' if m.group().startswith('fn.') else 'table.'}{m.group()}"
yield from (re.sub(pattern, subst, col) for col in cols)
table = Table(tablename)
env = dict(table=table, fn=fn)
q = Query.from_(table).select(*(eval(col, env) for col in parsed(collist)))
print (q.get_sql(quote_char=None)) #2
Output:
SELECT id,fname,SUM(revenue/2),MOD(revenue,COUNT(id)) FROM customers
I'm working with Redisgraph.
I have a node Person with three properties: name (string), age (number), isAlive (boolean).
If I store the age as number, without the quotes, it correctly store it as a number. So, if I query:
MATCH (p:Person) RETURN p
what I have is:
{ name: 'John', age: 30, isAlive: 'true' }
but there's a way to query and get real booleans?
What I want is:
{ name: 'John', age: 30, isAlive: true }
Thank you!
It sounds like you're querying RedisGraph using redis-cli. The RESP protocol that processes module replies only allows strings and integers as primitive data types that can be passed, so your request can't be accomplished through redis-cli.
All of the client libraries, however, will decode replies to their correct type. I'd recommend using one as an intermediary to interact with RedisGraph - https://oss.redis.com/redisgraph/clients/.
Redisgraph can return a compact format where the type of the values are included. In order to use this you need to pass the --compact flag (which also works in redis-cli):
GRAPH.QUERY demo "MATCH (a) RETURN a" --compact
Some client libraries takes advantage of this compact format in order to return the correct type. The type of value is returned as an integer:
typedef enum {
PROPERTY_UNKNOWN = 0,
PROPERTY_NULL = 1,
PROPERTY_STRING = 2,
PROPERTY_INTEGER = 3,
PROPERTY_BOOLEAN = 4,
PROPERTY_DOUBLE = 5,
} PropertyTypeUser;
You can read more about the compact format here.
I have a key which has fields and values. All the fields have string values.
One of these fields I want it to be a table or set or list (meaning holding multiple values). This field is called zonetable
I only know how to use hset but as far as I know it cannot do what I want. I would like to do something like that
hmset L0001:ad65ed38-66b0-46b4-955c-9ff4304e5c1a field1 blabla field2 blibli zonetable [1,2,3,4]
Key : L0001:ad65ed38-66b0-46b4-955c-9ff4304e5c1a
field1: "string value"
field2: "string value"
zonetable: [1,2,3,4] ---- the table
Maybe you can make use of Json. use json serialize your table (list or something) into a json string, then use hset to save it into your redis.
When you want to retrieve it, first get it from redis and then deserialize it from json to list.
If you use python, you can do it like this:
table = json.dumps(zonetable)
redis.hset(Key, 'zonetable', table)
when you want to retrieve it :
table = redis.hget(Key, 'zonetable')
zonetable = json.loads(table)
As you say, you use the native command, you can also do this.
first, convert your zonetable to json string using python interpreter
>>> import json
>>> table = [1,2,3,4]
>>> json.dumps(table)
'[1, 2, 3, 4]'
then use this in redis-cli
hmset L0001:ad65ed38-66b0-46b4-955c-9ff4304e5c1a field1 blabla field2 blibli zonetable '[1,2,3,4]'
Yes, one more thing I want to say, if you know the rule of how to convert object to json, you could do it by yourself.
I'm having trouble using ruby with dbi for some reason, I'm trying to do a select and put the results in an array but no luck.
require 'dbi'
db = DBI.connect('DBI:OCI8:database', XXXX, XXXX)
#Gets Consumer Id Number you want to create accounts for
numberOfAccounts = []
puts("Please enter a CID")
NewCID = gets.chomp()
numberOfAccounts << db.execute("select T_NBR from T_CBA where C_ID='#{NewCID}'").fetch
My array ends up like this:
[[<#BigDecimal:fc115f8,'0.8000169202 2E11',12(16)>]]
where I would like to have several different numbers like [222, 3232, 2323] etc.
I've searched online but to no avail.
DBI has probably determined that the underlying column can contain integers too large to fit in a regular int type, based on the data field. Or it may just use BigDecimal for all integer types to avoid worrying about it.
If you know that your values are all small enough to fit into a regular integer, you can convert the array to integers after you've populated it, like so:
1.9.3-p194 :014 > numberOfAccounts
=> [[#<BigDecimal:119cd90,'0.123E3',9(36)>], [#<BigDecimal:119cd18,'0.456E3',9(36)>]]
1.9.3-p194 :015 > numberOfAccounts.flatten!.collect!(&:to_i)
=> [123, 456]
1.9.3-p194 :016 > numberOfAccounts
=> [123, 456]
With a simple model like that
class Model < ActiveRecord::Base
# ...
end
we can do queries like that
Model.where(["name = :name and updated_at >= :D", \
{ :D => (Date.today - 1.day).to_datetime, :name => "O'Connor" }])
Where the values in the hash will be substituted into the final SQL statement with proper escaping depending on the underlying database engine.
I would like to know a similar feature for SQL execution like:
ActiveRecord::Base.connection.execute( \
["update models set name = :name, hired_at = :D where id = :id;"], \
{ :id => 73465, :D => DateTime.now, :name => "O'My God" }] \
) # THIS CODE IS A FANTASY. NOT WORKING.
(Please do not solve the example with loading a Model object, modifying and then saving! The example is only an illustration for the feature I would like to have / know. Concentrate on the subject!)
The original problem is that I want to insert large amount (many thousand lines) of data into the database. I want to use some features of the SQL abstraction of the ActiveRecord framework but I don't want to use model objects based on ActiveRecord::Base because they are damn slow! (8 queries per second for my current problem.)
query = ActiveRecord::Base.connection.raw_connection.prepare("INSERT INTO users (name) VALUES(:name)")
query.execute(:name => 'test_name')
query.close
Extending the #peufeu solution with concrete code example for bulk insert:
users_places = []
users_values = []
timestamp = Time.now.strftime('%Y-%m-%d %H:%M:%S')
params[:users].each do |user|
users_places << "(?,?,?,?)"
users_values << user[:name] << user[:punch_line] << timestamp << timestamp
end
bulk_insert_users_sql_arr = ["INSERT INTO users (name, punch_line, created_at, updated_at) VALUES #{users_places.join(", ")}"] + users_values
begin
sql = ActiveRecord::Base.send(:sanitize_sql_array, bulk_insert_users_sql_arr)
ActiveRecord::Base.connection.execute(sql)
rescue
"something went wrong with the bulk insert sql query"
end
Here is the reference to sanitize_sql_array method in ActiveRecord::Base, it generates the proper query string by escaping the single quotes in the strings. For example the punch_line "Don't let them get you down" will become "Don\'t let them get you down".
Yes you could do raw SQL, but checkout the ar-extensions gem that helps with batch inserts:
https://github.com/zdennis/ar-extensions
Here's a post on it, and various other techniques:
http://www.coffeepowered.net/2009/01/23/mass-inserting-data-in-rails-without-killing-your-performance/
For INSERTs, batching them using a long VALUES clause (as shown by Simon's link) is the fastest way (unless you want to generate a text file and load it in your database with MySQL's LOAD DATA INFILE). But you have to be very careful about escaping your text values (which is not done in the example).
I was asking "what database are you using" because it does matter for mass UPDATEs.
For instance, you can do this on postgres (and I believe SQL Server changing "columnX" to "colX" ):
UPDATE foo
JOIN (VALUES (1,2),(3,4),... long list) v ON (foo.id=v.column1)
SET foo.bar = v.column2
And you can update a load of rows using a single statement, very fast.
If you don't need Ruby to perform some Ruby-specific magic on your data, the fastest way to transfer data from one DB to a different one is to export as a text file (CSV or tab separated), load it on the other DB (LOAD DATA INFILE on MySQL), perhaps in a temporary table, and bulk process using SQL.
EDIT : Here's how I do this in Python :
sql = [ "INSERT INTO foo (column list) VALUES " ]
values = []
for tuple in tuple_list:
append "(?,?,?,?)" to sql
extend values list with tuple
Then join sql into a string, you get "INSERT INTO foo (column list) VALUES (?,?,?,?),(?,?,?,?),(?,?,?,?)" with the "(?,?,?,?)" repeated as many times as you have lines to insert.
Then "values" contains a list of (a1,b1,c1,d1,a2,b2,c2,d2,a3,b3,c3,d3) with an,bn,cn,dn being the tuples you want to insert for line n. Each one corresponds to a placeholder in the sql string.
Then pass this to the usual "execute query with parameters" function which will handle quoting and escaping as usual.
I encountered a similar issue recently when tying to insert 100K+ records into a MySQL database for a Rails 4 app using mysql2 gem. The data included characters that had to be sanitized prior to insert.
The solution I ended going with was a slightly modified version of Option 3 described at https://www.coffeepowered.net/2009/01/23/mass-inserting-data-in-rails-without-killing-your-performance/
Here's the relevant code block from the above link:
TIMES = 10000
inserts = []
TIMES.times do
inserts.push "(3.0, '2009-01-23 20:21:13', 2, 1)"
end
sql = "INSERT INTO user_node_scores (`score`, `updated_at`, `node_id`, `user_id`) VALUES #{inserts.join(", ")}"
The modification I made was using the public method ActiveRecord::Base.sanitize() on values that required it.
inserts = []
created = Time.now.strftime "%Y-%m-%d %H:%M:%S"
params[:audits].each do |audit|
inserts.push "(#{audit.user_id), #{created}," + ActiveRecord::Base.sanitize(audit.comment) + ", #{audit.status})"
end
sql = "INSERT INTO user_audits (`user_id`, `created_at`, `comment`, `status`) VALUES #{inserts.join(", ")}"