I want to store intermediate results to avoid multiple calculations for one thing. What I'm looking for is something like this:
h1_activ = sigmoid(self.bias_visiblie + T.dot(D, self.W))
h1_sample = h1_activ > rnds.uniform((n_samples, self.n_hidden ))
f_h1_sample = theano.function(
inputs=[D],
outputs=h1_sample,
# I'd like to take the result from 'h1_sample' and store it into 'H1_sample'
updates=[(self.H1_sample, ??? )]
)
The code above does not run of course but is there a way to do something like this? Storing an intermediate value into a shared variable?
You can write the final results, which use the same intermediate results, in the same theano.function.
For example:
h1_activ = sigmoid(self.bias_visiblie + T.dot(D, self.W))
h1_sample = h1_activ > rnds.uniform((n_samples, self.n_hidden ))
# h2_sample use the intermediate result h1_sample.
h2_sample = h1_sample * 2
f_h1_sample = theano.function(
inputs=[D],
outputs=[h1_sample, h2_sample],
)
h2_smaple is a final result which uses h1_sample.
Also you can save the intermediate results and use them as inputs in another theano.function.
Different theano.functions correspond to different computation graphs. I think no calculation can be shared between different computation graphs.
Related
I have a JSON API payload containing tablename, columnlist - how to build a SELECT query from it using pypika?
So far I have been able to use a string columnlist, but not able to do advanced querying using functions, analytics etc.
from pypika import Table, Query, functions as fn
def generate_sql (tablename, collist):
table = Table(tablename)
columns = [str(table)+'.'+each for each in collist]
q = Query.from_(table).select(*columns)
return q.get_sql(quote_char=None)
tablename = 'customers'
collist = ['id', 'fname', 'fn.Sum(revenue)']
print (generate_sql(tablename, collist)) #1
table = Table(tablename)
q = Query.from_(table).select(table.id, table.fname, fn.Sum(table.revenue))
print (q.get_sql(quote_char=None)) #2
#1 outputs
SELECT "customers".id,"customers".fname,"customers".fn.Sum(revenue) FROM customers
#2 outputs correctly
SELECT id,fname,SUM(revenue) FROM customers
You should not be trying to assemble the query in a string by yourself, that defeats the whole purpose of pypika.
What you can do in your case, that you have the name of the table and the columns coming as texts in a json object, you can use * to unpack those values from the collist and use the syntax obj[key] to get the table attribute with by name with a string.
q = Query.from_(table).select(*(table[col] for col in collist))
# SELECT id,fname,fn.Sum(revenue) FROM customers
Hmm... that doesn't quite work for the fn.Sum(revenue). The goal is to get SUM(revenue).
This can get much more complicated from this point. If you are only sending column names that you know to belong to that table, the above solution is enough.
But if you have complex sql expressions, making reference to sql functions or even different tables, I suggest you to rethink your decision of sending that as json. You might end up with something as complex as pypika itself, like a custom parser or wathever. than your better option here would be to change the format of your json response object.
If you know you only need to support a very limited set of capabilities, it could be feasible. For example, you can assume the following constraints:
all column names refer to only one table, no joins or alias
all functions will be prefixed by fn.
no fancy stuff like window functions, distinct, count(*)...
Then you can do something like:
from pypika import Table, Query, functions as fn
import re
tablename = 'customers'
collist = ['id', 'fname', 'fn.Sum(revenue / 2)', 'revenue % fn.Count(id)']
def parsed(cols):
pattern = r'(?:\bfn\.[a-zA-Z]\w*)|([a-zA-Z]\w*)'
subst = lambda m: f"{'' if m.group().startswith('fn.') else 'table.'}{m.group()}"
yield from (re.sub(pattern, subst, col) for col in cols)
table = Table(tablename)
env = dict(table=table, fn=fn)
q = Query.from_(table).select(*(eval(col, env) for col in parsed(collist)))
print (q.get_sql(quote_char=None)) #2
Output:
SELECT id,fname,SUM(revenue/2),MOD(revenue,COUNT(id)) FROM customers
I'm working with a query that is used by multiple services but the number of results returned are different based on filtering.
To avoid copying and pasting the query, I was wondering if it was possible to pass in piece of sql into a sql parameter and it would work? I'm also open to alternative solutions.
EXAMPLE:
MapSqlParameterSource parameters = new MapSqlParameterSource();
parameters.addValue("filter", "and color = blue");
namedParameterJdbcTemplate.query(“select * from foo where name = 'Joe' :filter”, parameters, new urobjRowMapper());
It is very dangerous and fragile to let callers pass SQL to your program, because it opens you up to SQL injection - the very problem the parameters are there to prevent.
A better approach is to pre-code the filters in your query, and protect them by a special "selector" parameter:
SELECT *
FROM foo
WHERE name='Joe' AND
(
(:qselect = 1 AND color='blue')
OR (:qselect = 2 AND startYear = 2021)
OR (:qselect = 3 AND ...)
)
I am writing a Lua script to execute in redis. I am working with Sorted Sets that look like this:
Member: 96954_1_1557705600
Score: 1557705600
The score is a unix epoch time stamp.
I would like to first get the results that are between two time stamps, then filter those based of a glob pattern in the member. Something like MATCH *_1_*.
My script looks like this, but it's failing when I try and pass the Lua table to zscan:
local start_date = KEYS[1]
local end_date = KEYS[2]
local limited_by_date = redis.call('zrangebyscore','rooms', start_date, end_date)
return redis.call('zscan', unpack(limited_by_date), 'match *_1_*')
limited_by_date correct contains the values I expect, but how can I search through them with zscan now?
When you get limited_by_date, you can iterate the array yourself, and output items that match the pattern.
local result = {}
for i, mem in ipairs(limited_by_date) do
if string.match(mem, ".+_1_.+") then result[#result + 1] = mem end
end
return result
There's no need to use the ZSCAN command. In fact, if you use ZSCAN, you have to intersect ZSCAN result and ZRANGEBYSCORE result to get the final answer.
I have a search form with several fields and it currently works. I have written the search action in the following way:
conditions = {}
conditions[:x] = params[:x] unless params[:x].blank?
conditions[:y] = params[:y] unless params[:y].blank?
conditions[:z] = params[:z] unless params[:z].blank?
etc.
#results = Material.where(conditions)
And this is fine. But now I want to add a condition that says essentially "where the level is less than or equal to params[:level], which would look like this in my head:
conditions[:level] <= params[:level] ...
But this doesn't work because it seems you can only add hashes using this syntax. So my question is how I would add such a condition to the query.
You can't add this to your existing conditions hash. The hash key: value arguments to where can only produce where key = value, not key <= value.
Use a parameterized string and an additional where:
#results = Material.where(conditions).where("level <= ?", params[:level])
It seems that the PyTable columns are alphabetically ordered when using both dictionary or class for schema definition for the call to createTable(). My need is to establish a specific order and then use numpy.genfromtxt() to read and store my data from text. My text file does not have the variable names included alphabetically as they are for the PyTable.
For example, assuming text file is named mydata.txt and is organized as follows:
time(row1) bVar(row1) dVar(row1) aVar(row1) cVar(row1)
time(row2) bVar(row2) dVar(row2) aVar(row2) cVar(row2)
...
time(rowN) bVar(rowN) dVar(rowN) aVar(rowN) cVar(rowN)
So, the desire is to create a table that is ordered with these columns
and then use the numpy.genfromtxt command to populate the table.
# Column and Table definition with desired order
class parmDev(tables.IsDescription):
time = tables.Float64Col()
bVar = tables.Float64Col()
dVar = tables.Float64Col()
aVar = tables.Float64Col()
cVar = tables.Float64Col()
#...
mytab = tables.createTable( group, tabName, paramDev )
data = numpy.genfromtxt(mydata.txt)
mytab.append(data)
This is desired because it is straightforward code and is very fast. But, the PyTable columns are always ordered alphabetically and the appended data is ordered according to the desired order. Am I missing something basic here? Is there a way to have the order of the table columns follow the class definition order instead of being alphabetical?
Yes, you can define an order in tables in several different ways. The easiest one is to use the pos parameter for each column. See the docs for the Col class:
http://pytables.github.io/usersguide/libref/declarative_classes.html#the-col-class-and-its-descendants
For your example, it will look like:
class parmDev(tables.IsDescription):
time = tables.Float64Col(pos=0)
bVar = tables.Float64Col(pos=1)
dVar = tables.Float64Col(pos=2)
aVar = tables.Float64Col(pos=3)
cVar = tables.Float64Col(pos=4)
Hope this helps