This may be slightly confusing so I'll try to be detailed. I need to query several months of daily records based on a certain variable. In this case a file name. There were several failed files that were rerun, and the rerun will include the original job run number in the new file name. For instance:
run 4145 failed. File name of "ORIGINALFILENAME"
run 4150 file name will look like "YYYYMMDDHHMMSS_4145_ORIGINALFILENAME"
There are 50+ failed files over the course of several months. Is there a way to build my where statement such that it will return both the failed runs (easily done via status_code = 'FAILURE') as well as the reruns? I imagine something like where file_name like '%_(select run_number where status_code = 'FAILURE')_ORIGINALFILENAME
The % character will work for any or no characters. So
where file_name like '%ORIGINALFILENAME'
will work for the failures and the reruns
This script return all the tuples which contain status_code = 'FAILURE' and where the file_name contains the sub-string "_ORIGINALFILENAME".
SELECT file_name FROM tblYourTable
WHERE status_code = 'FAILURE'
AND file_name LIKE '%[_ORIGINALFILENAME]%'
but... on a second look the requirement seems to be: "...statement such that it will return both the failed runs (easily done via status_code = 'FAILURE') as well as the reruns..." which if I understand correctly implies to adjust the predicate to return either scenario i.e.:
SELECT file_name FROM tblYourTable
WHERE status_code = 'FAILURE'
OR file_name LIKE '%[_ORIGINALFILENAME]%'
I want to store intermediate results to avoid multiple calculations for one thing. What I'm looking for is something like this:
h1_activ = sigmoid(self.bias_visiblie + T.dot(D, self.W))
h1_sample = h1_activ > rnds.uniform((n_samples, self.n_hidden ))
f_h1_sample = theano.function(
inputs=[D],
outputs=h1_sample,
# I'd like to take the result from 'h1_sample' and store it into 'H1_sample'
updates=[(self.H1_sample, ??? )]
)
The code above does not run of course but is there a way to do something like this? Storing an intermediate value into a shared variable?
You can write the final results, which use the same intermediate results, in the same theano.function.
For example:
h1_activ = sigmoid(self.bias_visiblie + T.dot(D, self.W))
h1_sample = h1_activ > rnds.uniform((n_samples, self.n_hidden ))
# h2_sample use the intermediate result h1_sample.
h2_sample = h1_sample * 2
f_h1_sample = theano.function(
inputs=[D],
outputs=[h1_sample, h2_sample],
)
h2_smaple is a final result which uses h1_sample.
Also you can save the intermediate results and use them as inputs in another theano.function.
Different theano.functions correspond to different computation graphs. I think no calculation can be shared between different computation graphs.
I need to perform data smoothing using averaging, with a non-standard group_by variable that is created on-the-fly. My model consists of two tables:
class WthrStn(models.Model):
name=models.CharField(max_length=64, error_messages=MOD_ERR_MSGS)
owner_email=models.EmailField('Contact email')
location_city=models.CharField(max_length=32, blank=True)
location_state=models.CharField(max_length=32, blank=True)
...
class WthrData(models.Model):
stn=models.ForeignKey(WthrStn)
date=models.DateField()
time=models.TimeField()
temptr_out=models.DecimalField(max_digits=5, decimal_places=2)
temptr_in=models.DecimalField(max_digits=5, decimal_places=2)
class Meta:
ordering = ['-date','-time']
unique_together = (("date", "time", "stn"),)
The data in WthrData table are entered from an xml file in variable time increments, currently 15 or 30 minutes, but that could vary and change over time. There are >20000 records in that table. I want to provide an option to display the data smoothed to variable time units, e.g. 30 minutes, 1, 2 or N hours (60, 120, 180, etc minutes)
I am using SQLIte3 as the DB engine. I tested the following sql, which proved quite adequate to perform the smoothing in 'bins' of N-minutes duration:
select id, date, time, 24*60*julianday(datetime(date || time))/N jsec, avg(temptr_out)
as temptr_out, avg(temptr_in) as temptr_in, avg(barom_mmhg) as barom_mmhg,
avg(wind_mph) as wind_mph, avg(wind_dir) as wind_dir, avg(humid_pct) as humid_pct,
avg(rain_in) as rain_in, avg(rain_rate) as rain_rate,
datetime(avg(julianday(datetime(date || time)))) as avg_date from wthr_wthrdata where
stn_id=19 group by round(jsec,0) order by stn_id,date,time;
Note I create an output variable 'jsec' using the SQLite3 function 'julianday', which returns number of days in the integer part and fraction of day in the decimal part. So, multiplying by 24*60 gives me number of minutes. Dividing by N-minute resolution gives me a nice 'group by' variable, compensating for varying time increments of the raw data.
How can I implement this in Django? I have tried the objects.raw(), but that returns a RawQuerySet, not a QuerySet to the view, so I get error messages from the html template:
</p>
Number of data entries: {{ valid_form|length }}
</p>
I have tried using a standard Query, with code like this:
wthrdta=WthrData.objects.all()
wthrdta.extra(select={'jsec':'24*60*julianday(datetime(date || time))/{}'.format(n)})
wthrdta.extra(select = {'temptr_out':'avg(temptr_out)',
'temptr_in':'avg(temptr_in)',
'barom_mmhg':'avg(barom_mmhg)',
'wind_mph':'avg(wind_mph)',
'wind_dir':'avg(wind_dir)',
'humid_pct':'avg(humid_pct)',
'rain_in':'avg(rain_in)',
'rain_sum_in':'sum(rain_in)',
'rain_rate':'avg(rain_rate)',
'avg_date':'datetime(avg(julianday(datetime(date || time))))'})
Note that here I use the sql-avg functions instead of using the django aggregate() or annotate(). This seems to generate correct sql code, but I cant seem to get the group_by set properly to my jsec data that is created at the top.
Any suggestions for how to approach this? All I really need is to have the QuerySet.raw() method return a QuerySet, or something that can be converted to a QuerySet instead of RawQuerySet. I can not find an easy way to do that.
The answer to this turns out to be really simple, using a hint I found from
[https://gist.github.com/carymrobbins/8477219][1]
though I modified his code slightly. To return a QuerySet from a RawQuerySet, all I did was add to my models.py file, right above the WthrData class definition:
class MyManager(models.Manager):
def raw_as_qs(self, raw_query, params=()):
"""Execute a raw query and return a QuerySet. The first column in the
result set must be the id field for the model.
:type raw_query: str | unicode
:type params: tuple[T] | dict[str | unicode, T]
:rtype: django.db.models.query.QuerySet
"""
cursor = connection.cursor()
try:
cursor.execute(raw_query, params)
return self.filter(id__in=(x[0] for x in cursor))
finally:
cursor.close()
Then in my class definition for WthrData:
class WthrData(models.Model):
objects=MyManager()
......
and later in the WthrData class:
def get_smoothWthrData(stn_id,n):
sqlcode='select id, date, time, 24*60*julianday(datetime(date || time))/%s jsec, avg(temptr_out) as temptr_out, avg(temptr_in) as temptr_in, avg(barom_mmhg) as barom_mmhg, avg(wind_mph) as wind_mph, avg(wind_dir) as wind_dir, avg(humid_pct) as humid_pct, avg(rain_in) as rain_in, avg(rain_rate) as rain_rate, datetime(avg(julianday(datetime(date || time)))) as avg_date from wthr_wthrdata where stn_id=%s group by round(jsec,0) order by stn_id,date,time;'
return WthrData.objects.raw_as_qs(sqlcode,[n,stn_id]);
This allows me to grab results from the highly populated WthrData table smoothed over time increments, and the results come back as a QuerySet instead of RawQuerySet
Currently I have a perl script that accesses our database, performs certain queries and prints output to the terminal. Instead, I would like to output the results into a template latex file before generating a pdf. For most of my queries I pull out numbers and store these as scalar variables (eg how often a particular operator carries out a given task). eg.
foreach $op (#operator) {
$query = "SELECT count(task_name) FROM table WHERE date <= '$date_stop' and
date >= '$date_start' and task=\'$operator[$index]\';";
#execute query
$result=$conn->exec($query);
$conres = $conn->errorMessage;
if ($result->resultStatus eq PGRES_TUPLES_OK) {
if($result->ntuples > 0) {
($task[$index]) = $result->fetchrow;
}
printf("$operator[$index] carried out task: %d\n", $task[$index]);
} else {
die "Failed.\n$conres\n\n";
exit -1;
}
$index++;
}
printf("**********************************\n\n");
In the final report I will summarise how many times each operator completed each task in a table. In addition to this there will also be some incidents which must be reported. I can print these easily to the terminal using a command such as
$query = "SELECT operator, incident_type from table_name WHERE incident_type = 'Y'
and date <= '$date_stop' and date >= '$date_start';";
$result=$conn->exec($query);
$conres = $conn->errorMessage;
if ($result->resultStatus eq PGRES_TUPLES_OK) {
if($result->ntuples > 0) {
$result->print(STDOUT, 1, 1, 0, 0, 0, 1, "\t", "", "");
}
} else {
die "Failed.\n$conres\n\n";
exit -1;
}
An example of the output of this command is
operator | incident_type
-----------------------------
AB | Incomplete due to staff shortages
-------------------------------
CD | Closed due to weather
-----------------------------
How can I make my perl script pass the operator names and incidents into a string array rather than just sending the results to the terminal?
You should consider updating your script to use DBI. This is the standard for database connectivity in Perl.
DBI has a built in facility for inserting parameters into a query string. It is safer and faster than manually creating the string yourself. Before the loop, do this once:
#dbh is a database handle that you have already opened.
my $query = $dbh->prepare(
"SELECT count(task_name) FROM table WHERE date<=? and date>=? and task=?"
);
Then within the loop, you only have to do this each time:
$query->execute($date_stop,$date_start,$op);
Note that the parameters you pass to execute automatically get inserted in place of the ?'s in your statement. It handles the quoting for you.
Also in the loop, after you execute the statement, you can get the results like this:
my $array_ref = $query->fetchall_array_ref;
Now all of the rows are stored in a two-dimensional array structure. $array_ref->[0][0] would get the first column of the first row returned.
See the DBI documentation for more information.
As others have mentioned, there are quite a few other mistakes in your code. Make sure you start with use strict; use warnings;, and ask more questions if you need further help!
Lots of good feedback to your script, but nothing about your actual question.
How can I make my perl script pass the operator names and incidents into a string array rather than just sending the results to the terminal?
Have your tried creating an array and pushing items to it?
my #array;
push (#array, "foo");
Or using nested arrays:
push (#array, ["operator", "incident"]);
I wanted to know how we could store the result of my select query on a variable.
#ppt2 = Ppt.select('slide_name').where('id=?',4)
#ppt1 = Ppt.update_all({:time2=>#ppt2},['id like ?','1'])
Here, slide_name and time2 are both text attributes of the same table ppt.
What happens on the above execution is that the time2 field in id=1 gets the value "#ppt2" whereas I want it to get the value of slide_name from id=4 which does not get stored in #ppt1.
In other words, how do I store the value of the select query in #ppt2 so that it can be used in the next line?
Any help is appreciated.
Call the slide_name method on your first result.
#ppt2 = Ppt.select('slide_name').find(4).slide_name