Key-value pairs in SQL table - sql

I am building a Django app and I would like to show some statistics on the main page, like total number of transactions, percentage of successful transactions, daily number of active users etc.
I don't want to calculate these values in the view every time a user requests the main page for performance reasons. I thought of 2 possible solutions.
(1) Create a number of one-record tables
Create a table for each of the statistics, e.g.:
from django.db import models
class LastSuccessfulTransactionDate(models.Model):
date = models.DateTimeField()
class TotalTransactionAmount(models.Model):
total_amount = models.DecimalField(max_digits=8, decimal_places=2)
# ...
and make sure that only one record exists in each table.
(2) Create a table with key-value data
class Statistics(models.Model):
key = models.CharField(max_length=100)
value = models.TextField()
and save the data by doing:
from datetime import datetime
from decimal import Decimal
import pickle
statistics = {
'last_successful_transaction_date': datetime(2010, 2, 3),
'total_transaction_amount': Decimal('1234.56'),
}
for k, v in statistics.items():
try:
s = Statistics.objects.get(key=k)
except Statistics.DoesNotExist:
s = Statistics(key=k)
s.value = base64.b64encode(pickle.dumps(v, pickle.HIGHEST_PROTOCOL)).decode()
s.save()
and retrieve by:
for s in Statistics.objects.all():
k = s.key
v = pickle.loads(base64.b64decode(s.value.encode()))
print(k, v)
In both cases the data would be updated every now and then by a cron job (they don't have to be very acurate).
To me solution (2) looks better, because to display the main page I would need to get data from the Statistics table only, not from a number of one-record tables. Is there any recommended solution to this problem? Thanks

There is even better solution if you are using Postgresql, which is JSONField. You can directly store the key value pair in the JSONField. Try like this:
# model
class Statistics(models.Model):
records = JSONField()
# usage
statistics = {
'last_successful_transaction_date': datetime(2010, 2, 3),
'total_transaction_amount': Decimal('1234.56'),
}
Statistics.objects.create(records=statistics)
Statistics.objects.filter(records__last_successful_transaction_date= datetime(2010, 2, 3)) # Running query
s = Statistics.objects.filter(records__last_successful_transaction_date= datetime(2010, 2, 3)).first()
new_statistics = {
'last_successful_transaction_date': datetime(2012, 2, 3),
'total_transaction_amount': Decimal('1234.50'),
}
records = s.records
records.update(new_statistics)
s.records = records
s.save()

Related

Arcpy Script to loop through field and run Union Analysis

I have a polygon file in form of a fishnet. Also another feature class with polygons named Trawl_Buffers. There is a unique field within Trawl_Buffers based on YEAR. I'd like to create a script to run a selection on YEAR, and then perform a union analysis with the fishnet polygon for each YEAR. So the desired output would be "Trawl_Buffers_union2003", "Trawl_Buffers_union2004" etc. I have a function that will get me the unique list of the years, and puts them in a list which i called vals.
Then seems I need to run a for loop over this list of unique years, create a temporary selection, then use that as input for the union, but I am having trouble implementing the query process.
Here is where I started, but seriously tripping
import arcpy
#Set the data environment
arcpy.env.overwriteOutput = True
arcpy.env.workspace = r'C:\Data\working\AK_Fishing_VMS\2021_Delivery\ArcPro_proj\ArcPro_proj.gdb'
trawlBuffs = r'C:\Data\working\AK_Fishing_VMS\2021_Delivery\ArcPro_proj\ArcPro_proj.gdb\buffers\buffers_testing'
fishnet = r'C:\Data\working\AK_Fishing_VMS\2021_Delivery\ArcPro_proj\ArcPro_proj.gdb\fishnets\vms_net1k'
unionOut = r'C:\Data\working\AK_Fishing_VMS\2021_Delivery\ArcPro_proj\ArcPro_proj.gdb\unions\union'
# function to get unique values for the YEAR field found within the trawlBuffs fc
def unique_values(table, field):
with arcpy.da.SearchCursor(table, [field]) as cursor:
return sorted({row[0] for row in cursor})
# Get the unique values for the field 'YEAR' found within the 'trawl_buffs' featureclass table
vals = unique_values(trawlBuffs, "YEAR")
# Create a query string for the selected country
yearSelectionClause = '"YEAR" = ' + "'" + vals + "'"
#loop through the years, create selection, union, make permanent
for year in vals:
year_layer = str(year) + "_union"
arcpy.MakeFeatureLayer_management(trawlBuffs, year_layer)
arcpy.SelectLayerByAttribute_management(year_layer, "NEW_SELECTION", "\"YEAR"\" = %d" % (year))
arcpy.Union_analysis(fishnet, year_layer , unionOut)
arcpy.CopyFeatures_management(year_layer, "union_" + str(year))

Loading multiple GTFS.zip format files in r5r

I am trying to analyze trips in a long time period in r5r that requires more than one GTFS files. I am using a for loop since I want to study trips in various depature dates in the Excel file. Right now, I have placed all three GTFS.zip files with different names in the data path together, but I could only receive mode information by public transportation within one date range, while trips in the other two dates produced walk time only. Is there a way to let r5r include all of them?
options(java.parameters = "-Xmx16G")
library(r5r)
library(sf)
library(data.table)
File_Path = file.path("C:","Research", "Data Sep", fsep = .Platform$file.sep)
list.files(File_Path)
poi <- fread(file.path(File_Path, "OriginsDestinationsPugetSound.csv"))
r5r_core <- setup_r5(data_path = File_Path, verbose = TRUE)
mode <- c("WALK","TRANSIT")
max_walk_dist <- 1000 # in meters
max_trip_duration <- 300 # in minutes
LengthOfFile = length(poi[[1]])
ListOfDetailedItineries = (matrix(ncol = 15,nrow = 0))
start = 1
end = 25
for (i in start:end) {
OriginPoint = poi[i,2:4]
DestinationPoint = poi[i,5:7]
Time_of_Trip = poi[i,9]
departure_datetime = as.POSIXct(Time_of_Trip[[1]], format = "%m/%d/%Y %H:%M")
dit <- detailed_itineraries(r5r_core = r5r_core,
origins = OriginPoint,
destinations = DestinationPoint,
mode = mode,
departure_datetime = departure_datetime,
max_walk_dist = max_walk_dist,
max_trip_duration = max_trip_duration,
shortest_path = TRUE,
verbose = TRUE)
ListOfDetailedItineries = rbind(ListOfDetailedItineries, as.matrix(dit))
cat('On iteration ',i,'\n',dit[[9]],"\n")
flush.console()
}
dit1 = as.data.frame(ListOfDetailedItineries)
As far as I understood, you have 3 feeds inside your data directory and you tried to generate travel time estimates for 3 different departure times, but only one of them return transit public transport trips. Is that correct?
If that's the case, you have to make sure that the public transit services listed in your GTFS feeds run on your specified departure times. This information is usually listed in the calendar table, but can also be listed in the calendar_dates table in some feeds.
The best practice here would be to choose dates that fall inside the service intervals of all 3 of your feeds. Alternatively, you can edit the start_date/end_date columns of their calendar table to include the days you have already chosen.

Append all stock.move ID's to list from all chain of related records

I have records in the stock.move table that has One2many relation to stock.move.reconcile with column move_to_id which is the ID of another record in stock.move table. So this chain can be thousands of records.
As you can see in my example I loop trough all records and go down tier by tier, but as I said before there could be thousands of linked records so my approach will not work here.
I do know that probably I need to use a while loop here, something like while there is move_to_ids, then I should loop through records and keep adding IDs to list, but I just can't figure it out how to do it.
move1(stock.move ID = 10) record that has One2many relation with 2 records inside: move_to_ids (stock.move.reconcile)
each of move_to_ids has move_to_id(many2one, 'stock.move' ID = 11)
each of this move_to_id(stock.move, ID=11) records have again any number of move_to_ids (stock.move.reconcile) and each of thismove_to_idsrecords havemove_to_id('stock.move', ID=12)` and so on.
So basically I want to add to list all move_to_id IDs 10, 11, 12, etc. to list for all related move_to_ids.
moves_to_recalculate = [10,11,12] and so on until when there is 0 move_to_ids to get move_to_id from.
class StockMove(models.Model):
_name = 'stock.move'
move_to_ids = fields.One2many(
'stock.move.reconcile', 'move_from_id', string='Move to')
move_from_ids = fields.One2many(
'stock.move.reconcile', 'move_to_id', string='Move From'
)
class StockMoveReconcile(models.Model):
_name = 'stock.move.reconcile'
_description = 'Stock Move Reconcile'
move_to_id = fields.Many2one('stock.move', string='Move To')
move_from_id = fields.Many2one('stock.move', string='Move From')
def recalculate(self):
moves = self.browse(('active_ids'))
moves_to_recalculate = []
for move1 in moves:
#I add my first move in chain to list
moves_to_recalculate.append(move1.id)
#First move have 2 moves_to_ids so i make another loop to add it ID to list
for second_tier_move in move.move_to_ids:
moves_to_recalculate.appen(second_tier_move.move_to_id.id)
# secont tier move has 1 move_to_ids so i do another loop, and add it's ID to list.
for third_tier_move in second_tier_move.move_to_ids:
moves_to_recalculate.appen(third_tier_move.move_to_id.id)
#third_tier_move has another move_to_ids , and so on.
I got answer from other resource but I think i need to post it here because the answer is excellent.
def recalculate_fifo(self):
moves = self.browse(self._context.get('active_ids'))
moves_to_recalculate = moves
current_moves = moves
while current_moves:
current_moves = current_moves.mapped('move_to_ids.move_to_id')
current_moves -= moves_to_recalculate
moves_to_recalculate |= current_moves

Web2py - SQLFORM.smartgrid: how to retrieve field name and value before submission

I use smartgrid with 'in the grid' edition of values. To do this, I use the .represent function to add the row_id to the variable name, so that I can retrieve what to update in request.post_vars, where I can retrieve the list of filed name, ids, with the submitted value.
But I'd like to identify what has been changed by the user in the smartgrid without making an additional I/O in the DB. Is there a global variable where the form fields and initial values are recorded before user change ? Or a function to extract form fields and values before sending it to the view?
db.define_table('cny',Field('f_cny'))
def tst():
tb = db.cny
if len(request.post_vars) >0:
d = {}
for k,v in request.post_vars.iteritems():
(f,sep,r) = k.partition('_r_')
if r:
if not r in d:
d[r] = {}
d[r][f] = v #should only be done if d[r][f] was changed!!!
for r in d:
if len(d[r]):
db(tb.id==r).update(**d[r])
tb.f_cny.represent=lambda v,r:string_widget(tb.f_cny, v, **{'_name':'f_cny_r_%s' % r.id})
f = SQLFORM.smartgrid(tb,
linked_tables=[],
selectable=(lambda ids:redirect(URL(request.controller,request.function,args=request.args))))
return dict(f=f)

Error Handling with Live Data Matlab

I am using a live data API that is returning the next arriving trains. I plan on giving the user the next 5 trains arriving. If there are less than 5 trains arriving, how you handle that? I am having trouble thinking of a way, I was thinking a way with if statements but don't know how I would set them up.
time1Depart = dataReturnedFromLiveAPI{1,1}.orig_departure_time;
time2Depart = dataReturnedFromLiveAPI{1,2}.orig_departure_time;
time3Depart = dataReturnedFromLiveAPI{1,3}.orig_departure_time;
time4Depart = dataReturnedFromLiveAPI{1,4}.orig_departure_time;
time5Depart = dataReturnedFromLiveAPI{1,5}.orig_departure_time;
time1Arrival = dataReturnedFromLiveAPI{1,1}.orig_arrival_time;
time2Arrival = dataReturnedFromLiveAPI{1,2}.orig_arrival_time;
time3Arrival = dataReturnedFromLiveAPI{1,3}.orig_arrival_time;
time4Arrival = dataReturnedFromLiveAPI{1,4}.orig_arrival_time;
time5Arrival = dataReturnedFromLiveAPI{1,5}.orig_arrival_time;
The code right now uses a matrix that goes from 1:numoftrains but I am using just the first five.
It's a bad practice to assign individual value to a separate variable. Better if you pass all related values to a vector or cell array depending on class of orig_departure_time and orig_arrival_time.
It looks like dataReturnedFromLiveAPI is a cell array of structures. Then you can do:
timeDepart = cellfun(#(x), x.orig_departure_time, ...
dataReturnedFromLiveAPI(1,1:min(5,size(dataReturnedFromLiveAPI,2))), ...
'UniformOutput',0 );
timeArrival = cellfun(#(x), x.orig_arrival_time, ...
dataReturnedFromLiveAPI(1,1:min(5,size(dataReturnedFromLiveAPI,2))), ...
'UniformOutput',0 );
Then you how to access the values one by one as
time1Depart = timeDepart{1};
If orig_departure_time and orig_arrival_time are numeric scalars, you can use ...'UniformOutput',1.... You will get output as a vector and can get single values with timeDepart(1).