I'm hoping to build a Django query to my model that lets my filter change as the query progresses.
I have a model Activity that I'm querying for. Each object has a postal_code field and I'm querying for multiple zip codes stored in an array postal_codes_to_query across a date range. I'd like to ensure that I get an even spread of objects across each of the zip codes. My database has millions of Activities, so when I query with a limit, I only receive activities that match zip codes early on in postal_codes_to_query. My current query is below:
Activity.objects.filter(postal_code__in=postal_codes_to_query).filter(start_time_local__gte=startTime).filter(start_time_local__lte=endTime).order_by('start_time_local')[:10000]
If I'm searching for say 20 zip codes, Ideally I'd like to receive 10000 activities, with 500 activities for each zip code that I queried on.
Is this possible in Django? If not, is there some custom SQL I could write to achieve this? I'm using a Heroku Postgres database in case that matters.
You can't do this in a single query, either in Django nor (as far as I know) in SQL.
The best bet is simply to iterate through the list of zips, querying for max 500 in each one:
activities_by_zip = {}
for code in postal_codes_to_query:
activities = Activity.objects.filter(postal_code=code).filter(
start_time_local__gte=startTime).filter(
start_time_local__lte=endTime).order_by('start_time_local')[:500]
activities_by_zip[code] = activities
Of course, this is one query per zip, but I think that's the best you're going to do.
Related
I'm new to MS Access and am trying to speed up a data gathering process that is taking forever in Powershell. In Powershell I have 10 or so web API calls to get data and each comes back as an object with multiple properties (fields.) Each set of data has related fields to 1 or more of the other sets of data. Getting the data is very quick but piping an array of objects to where-object to select-object takes over an hour and there's really not that much data. Each object contains 500-1500 "records" and 5 to 10 "fields" so I thought why not export that data and use something that's intended to search through data to do the job? I exported each object as a separate .CSV file. So enter MS Access..
I imported each of the CSV's as a separate table (easy enough.) I'm going to simplify this down for this example to the following 3 tables:
[Tables]https://i.stack.imgur.com/UCH1F.jpg
Every table has fields that relate it over to other tables. Pretty much there's some sort of Id field in every table that is related to another Id field in a different table that I need to pull a field called "name" from. I'm trying to follow the bread crumbs from the Player name to it's Network name to it's Application name, to it's Layout name, etc... I want to build a query that I would eventually just be able to export as an Excel file. I also would prefer to just write out the SQL unless it's really easier to to understand the visual query builder. I'm looking to build a sheet with the following information:
Player's Name would include all names from the Players table and getting just that data makes sense to me. SELECT Name AS PlayerName FROM Players Everything else, not so much. I feel like this will end up being some mega query as I get deeper into related table after related table. In Excel, it would be straightforward using Vlookups across tabs but that doesn't seem to be the best approach. Given the info above, I'm trying to achieve the following output:
Result table
Any help with strategy and syntax greatly appreciated!
You're looking for the JOIN clause.
SELECT
Players.Name PlayerName, Networks.Name PlayerNetwork, Applications.Name ApplicationName
FROM
Players
LEFT OUTER JOIN
Networks
ON
Networks.ID = Players.NetworkId
LEFT OUTER JOIN
Applications
ON
Applications.Id = Players.ApplicationID
I'm not a programmer, but trying to learn. I'm a nurse, and need to pull data for medical referral tracking from a database. I have a piece of GUI software which builds JOIN queries for me to pull things from the database. One of the operators I can use in the drop-down is "IN." The referral documentation is stored in the table as codes made up of one to three letters. For example, the code for a completed dental referral is CDF, and the code for a dental referral is D.
I want to build a report to allow other nurses to pull all their outstanding referrals, so I'll want to pull "D" but not "CDF"
If I use IN as the operator, and set my parameters to 'S','D','BP' {etc} will that also pull the records which have the other, longer codes which contain those same letters? (like CDF, CSR, CBP)
I don't want to test it because I only have access to the production database, and I don't want to hose up actual patient records. Thanks in advance for any help!
Assuming that the column that holds the referral code holds one and only one code per record (which is what it sounds like) the query should function as you want and will not attempt to match substrings.
In any event, there's no danger that a query in the form IN ('S', 'D', 'BP') will match substrings. To perform substring matches in SQL you have to use the LIKE operator.
The situation in which this will not work is if the referral code column holds multiple codes separated by commas. This is an all-too-common mistake in designing databases but if the product you're using is commercial rather than home-grown, I think it's very unlikely to be the case. If it is, searching it is much more difficult.
(I tried posting this to the CFWheels Google Group (twice), but for some reason my message never appears. Is that list moderated?)
Here's my problem: I'm working on a social networking app in CF on Wheels, not too dissimilar from the one we're all familiar with in Chris Peters's awesome tutorials. In mine, though, I'm required to display the most recent status message in the user directory. I've got a User model with hasMany("statuses") and a Status model with belongsTo("user"). So here's the code I started with:
users = model("user").findAll(include="userprofile, statuses");
This of course returns one record for every status message in the statuses table. Massive overkill. So next I try:
users = model("user").findAll(include="userprofile, statuses", group="users.id");
Getting closer, but now we're getting the first status record for each user (the lowest status.id), when I want to select for the most recent status. I think in straight SQL I would use a subquery to reorder the statuses first, but that's not available to me in the Wheels ORM. So is there another clean way to achieve this, or will I have to drag a huge query result or object the statuses into my CFML and then filter them out while I loop?
You can grab the most recent status using a calculated property:
// models/User.cfc
function init() {
property(
name="mostRecentStatusMessage",
sql="SELECT message FROM statuses WHERE userid = users.id ORDER BY createdat DESC LIMIT 1,1"
);
}
Of course, the syntax of the SELECT statement will depend on your RDBMS, but that should get you started.
The downside is that you'll need to create a calculated property for each column that you need available in your query.
The other option is to create a method in your model and write custom SQL in <cfquery> tags. That way is perfectly valid as well.
I don't know your exact DB schema, but shouldn't your findAll() look more like something such as this:
statuses = model("status").findAll(include="userprofile(user)", where="userid = users.id");
That should get all statuses from a specific user...or is it that you need it for all users? I'm finding your question a little tricky to work out. What is it you're exactly trying to get returned?
I'm building a chatroom application, and I want to keep track of which users are currently in the chatroom. However, I can't just store this array of users (or maybe a list would be better) in a field in one of my records in the Chatroom table.
Obviously one of the SQL data types is not an array, which leads me to this issue: what is the best way to fake/mock array functionality in a SQL database?
It seems there are 3 options:
1: Store the list/array of users as a string separated by commas, and just do some parsing when I want to get it back to an array
2: Since the max amount of users is allowed to be 10, just have 10 extra fields on each Chatroom record representing the users who are currently there
3: Have a new table Userchats, which has two fields, a reference to the chatroom, and a user name
I dunno, which is the best? I'm also open to other options. I'm also using Rails, which seems irrelevant here, but may be of interest.
Option 3 is the best. This is how you do it, in a relational schema. It is also the most flexible and future-proof option.
It can grow easier in width (extra columns say, a date joined, a channel status, a timestamp last talked) and length (extra rows when you decide there now can be 15 users in a room instead of 10).
The proper way to do this is to add an extra table representing an instance of a user being in a chatroom. In most cases, this is probably what you will want to do, since it gives you more flexibility in the types of queries you can do (for instance: list all chatrooms a particular user is in, find the average number of people in each chatroom, etc.) You would just need to add a new table - something like chat_room_users, with a chat_room_id, and a user_id.
If you're deadset on not adding an extra table, then Rails (or more specifically ActiveRecord), does have some functionality to store data structures like arrays in a SQL column. Just set up your column as a string or text type in a Rails migration, and add:
serialize :users
You can then use this column as a normal Ruby array / object, and ActiveRecord will automatically serialize / deserialize this object as you work with it. Keep in mind that's there are a lot of tradeoffs with this approach - you will never be able to query what users are in a particular room using SQL and will instead need to pull all data down to Ruby before working with it.
I want to know the best way of testing data access functionality.
I know that it is possible to create mocks to change data layer objects that is using to test business logic.
However is it possible to test if sql queries to database are correct.
Scenario: A method must return employees which were applied for work last month.
I can return list of object and check if each employee's startDate property is correct (last month).
So if it returns 3 employees and they have correct startDate value but there are more two employee in database which aren't returned. How to write test for that case? :)
Thanks in advance.
You set up the test DB so you shold know what data is in it. If you expect 5 employees to be returned from the query and you get only 3, you know there is an error.
You can test the query with different setups: empty table, only new employees, only old employees, a mix of the two (with special care to the borderline cases), etc.
I don't think you need to check the two other employees in the database which aren't returned.
The key is, when setting up your test data, you would want to ensure you have enough records that don't match the criteria (in addition to the records that do match), then you run the fetch and make sure you get back the correct number of records that do match the criteria.
Preparing the test data in this manner ensures that your method is returning the expected results.