I have the following model, where the relation tables create a many to many relationship, as usual.
|Table car | |relation car worker|
|Id car |1----*|id car |
|car things| |id worker |*-----
|
|
|table worker | /
|id worker |1----------/
|worker things | \
\_______________
\
|table building | |relation worker building| |
|id building |1----*|id building | |
|building things| |id worker |*---------
When I load this model in Power Bi, it can build a table visualization (and others) containing one of the following:
Option 1:
Car things
Worker things
Option 2:
Worker things
Building things
But it totally fails when I try to put in the table visualization something from the edges of the model:
Car things
Building things
This is the error:
What's going on here? Why the error?
Basically I need to see which cars visit which buildings (and do one or two summarizations)
From what i can understand from your model.. you need to set your cross filter direction to both rather than single so it can filter back up the tables
Here is an example which may help you to understand
For example, for 1 from tableA, it can relate 1 or 2 from tableB, since 1 from Table B can also relate 1 or 2 from tableC, so for 1 from tableA, it can’t determine whether 1 or 2 from tableC should be refer to
*----------------------------*
|Table A Table B Table C |
*----------------------------*
| 1 1 1,2 |
| 1 2 1,2 |
*----------------------------*
You can create a new table with the formula
tableName = CROSSJOIN(tableA,tableB,tableC....tableZ)
I had to create a new table as if I were doing an inner join in SQL (which works just as expected).
Click "modeling", "create table".
For the formula:
NewTable = NATURALINNERJOIN(tb_cars,
NATURALINNERJOIN(tb_rel_car_worker,
NATURALINNERJOIN(tb_workers,
NATURALINNERJOIN(tb_rel_worker_building, tb_buildings))))
Then I start using everything from this table instead of the others.
In order to create metrics that include zero counts grouped by one of the edge tables, I had to leave this table separate:
Partial Table= NATURALINNERJOIN(tb_rel_car_worker,
NATURALINNERJOIN(tb_workers,
NATURALINNERJOIN(tb_rel_worker_building, tb_buildings))))
Then in the relationship editor, I added a relation between Partial Table[id_car] and tb_cars[id_car].
Now the metrics grouped by car can use the technique of adding +0 to the DAX formulas to show also cars whose metric sum zero in charts.
Related
Good day, everyone. Hope you're doing well. I'm a Django newbie, trying to learn the basics of RESTful development while helping in a small app project. We currently want some of our models to update accordingly based on the data we submit to them, by using the Django ORM and the fields that some of them share wih OneToMany relationsips. Currently, there's a really difficult query that I must do for one of my fields to update automatically given that filter. First, let me explain the models. This are not real, but a doppleganger that should work the same:
First we have a Report model that is a teacher's report of a student:
class Report(models.Model):
status = models.CharField(max_length=32, choices=Statuses.choices, default=Statuses.created,)
student = models.ForeignKey(Student, on_delete=models.CASCADE,)
headroom_teacher = models.ForeignKey(TeacherStaff, on_delete=models.CASCADE,)
# Various dates
results_date = models.DateTimeField(null=True, blank=True)
report_created = models.DateTimeField(null=True, blank=True)
.
#Other fields that don't matter
Here we have two related models, which are student and headroom_teacher. It's not necessary to show their models, but their relationship with the next two models is really important. We also have an Exams model:
class Exams(models.Model):
student = models.ForeignKey(student, on_delete=models.CASCADE,)
headroom_teacher = models.ForeignKey(TeacherStaff, on_delete=models.CASCADE,)
# Various dates
results_date = models.DateTimeField(null=True, blank=True)
initial_exam_date = models.DateTimeField(null=True, blank=True)
.
#Other fields that don't matter
As you can see, the purpose of this app is akin to reporting on the performance of students after completing some exams, and every Report is made by a teacher for specific student on how he did on those exams. Finally we have a final model called StudentMood that aims to show how should an student be feeling depending on the status of their exams:
class StudentMood(models.Model):
report = models.ForeignKey(Report, on_delete=models.CASCADE,)
student_status = models.CharField(
max_length=32, choices=Status.choices,
default=None, null=True, blank=False)
headroom_teacher = models.ForeignKey(TeacherStaff, on_delete=models.CASCADE,)
And with these three models is that we arrive to the crux of the issue. One of our possible student_status options is called Anxious for results, which we believe a student will feel during the time when he already has done an exam and is waiting for the results.
I want to automatically set my student_status to that, using a custom manager that takes into account the date that the report has been done or the day the data has been entered. I believe this can be done by making a query taking into account initial_exam_date.
I already have my custom manager set up, and the only thing missing is this query. I have no choice but to do it with Django's ORM. However, I've come up with an approximate raw SQL query, that I'm not sure if it's ok:
SELECT student_mood.id AS student_mood_id FROM
school_student_mood LEFT JOIN
school_reports report
ON student_mood.report_id = report.id AND student_mood.headroom_teacher_id = report.headroom_teacher_id
JOIN school_exams exams
ON report.headroom_teacher_id = exams.headroom_teacher_id
AND report.student_id = exams.student_id
AND exams.results_date > date where the student_mood or report data is entered, I guess
And that's what I've come to ask for help. Could someone shed some light into how to transfer this into a single query?
Without having an environment setup or really knowing exactly what you want out of the data. This is a good start.
Generally speaking, the Django ORM is not great for these types of queries, and trying to use select_related or prefetches results in really complex and inefficient queries.
I've found the best way to achieve these types of queries in Django is to break each piece of your puzzle down into a query that returns a "list" of ids that you can then use in a subquery.
Then you keep working down until you have your final output
from django.db.models import Subquery
# Grab the students of any exams where the result_date is greater than a specific date.
student_exam_subquery = Exam.objects.filter(
results_date__gt=timezone.now()
).values_list('student__id', flat=True)
# Grab all the student moods related to reports that relate to our "exams" where the student is anxious
student_mood_subquery = StudentMood.objects.filter(
student_status='anxious',
reports__student__in=Subquery(student_exam_subquery)
).values_list('report__id', flat=True)
# Get the final list of students
Student.objects.values_list('id', flat=True).filter(
reports__id__in=Subquery(student_mood_subquery)
)
Now I doubt this will work out of the box, but it's really to give you an understanding of how you might go about solving this in a way that is readable to future devs and the most efficient (db wise).
So, the issue I was running into, is that the school has exam cycles each period, and it was difficult to retrieve only the students' report for this cycle. Let's assume we have the following database:
+-----------+-----------+----------------+-------------------+-------------------+------------------+
| Student | Report ID | StudentMood ID | Exam Cycle Status | Initial Exam Date | Report created a |
+-----------+-----------+----------------+-------------------+-------------------+------------------+
| Student 1 | 1 | 1 | Done | 01/01/2020 | 02/01/2020 |
| Student 2 | 2 | 2 | Done | 01/01/2020 | 02/01/2020 |
| Student 1 | 3 | 3 | On Going | 02/06/2020 | 01/01/2020 |
| Student 2 | 4 | 4 | On Going | 02/06/2020 | 01/01/2020 |
+-----------+-----------+----------------+-------------------+-------------------+------------------+
And Obviously, I wanted to limit my query to just this cycle, like this:
+-----------+-----------+----------------+-------------------+-------------------+------------------+
| Student | Report ID | StudentMood ID | Exam Cycle Status | Initial Exam Date | Report created a |
+-----------+-----------+----------------+-------------------+-------------------+------------------+
| Student 1 | 3 | 3 | On Going | 02/06/2020 | 01/01/2020 |
| Student 2 | 4 | 4 | On Going | 02/06/2020 | 01/01/2020 |
+-----------+-----------+----------------+-------------------+-------------------+------------------+
Now, your answer, trent, was really useful, but I'm still having issues retrieving in the shape of the above:
qs_exams = Exams.objects.filter(initial_exam_date__gt=now()).values_list('student__id', flat=True)
qs_report = Report.objects.filter(student__id__in=qs_exams).values_list('id', flat=True)
qs_mood = StudentMood.objects.select_related('report') \
.filter(report__id__in=qs_report).order_by('report__student_id', '-created').distinct()
But this query is still giving me all the StudentMoods throughout the school year. Sooooo, any ideas?
I take a Database course in which we have listings of AirBnBs and need to be able to do some SQL queries in the Relationship-Model we made from the data, but I struggle with one in particular :
I have two tables that we are interested in, Billing and Amenities. The first one have the id and price of listings, the second have id and wifi (let's say, to simplify, that it equals 1 if there is Wifi, 0 otherwise). Both have other attributes that we don't really care about here.
So the query is, "What is the difference in the average price of listings with and without Wifi ?"
My idea was to build to JOIN-tables, one with listings that have wifi, the other without, and compare them easily :
SELECT avg(B.price - A.price) as averagePrice
FROM (
SELECT Billing.price, Billing.id
FROM Billing
INNER JOIN Amenities
ON Billing.id = Amenities.id
WHERE Amenities.wifi = 0
) A, (
SELECT Billing.price, Billing.id
FROM Billing
INNER JOIN Amenities
ON Billing.id = Amenities.id
WHERE Amenities.wifi = 1) B
WHERE A.id = B.id;
Obviously this doesn't work... I am pretty sure that there is a far easier solution to it tho, what do I miss ?
(And by the way, is there a way to compute the absolute between the difference of price ?)
I hope that I was clear enough, thank you for your time !
Edit : As mentionned in the comments, forgot to say that, but both tables have idas their primary key, so that there is one row per listing.
Just use conditional aggregation:
SELECT AVG(CASE WHEN a.wifi = 0 THEN b.price END) as avg_no_wifi,
AVG(CASE WHEN a.wifi = 1 THEN b.price END) as avg_wifi
FROM Billing b JOIN
Amenities a
ON b.id = a.id
WHERE a.wifi IN (0, 1);
You can use a - if you want the difference instead of the specific values.
Let's assume we're working with data like the following (problems with your data model are noted below):
Billing
+------------+---------+
| listing_id | price |
+------------+---------+
| 1 | 1500.00 |
| 2 | 1700.00 |
| 3 | 1800.00 |
| 4 | 1900.00 |
+------------+---------+
Amenities
+------------+------+
| listing_id | wifi |
+------------+------+
| 1 | 1 |
| 2 | 1 |
| 3 | 0 |
+------------+------+
Notice that I changed "id" to "listing_id" to make it clear what it was (using "id" as an attribute name is problematic anyways). Also, note that one listing doesn't have an entry in the Amenities table. Depending on your data, that may or may not be a concern (again, refer to the bottom for a discussion of your data model).
Based on this data, your averages should be as follows:
Listings with wifi average $1600 (Listings 1 and 2)
Listings without wifi (just 3) average 1800).
So the difference would be $200.
To achieve this result in SQL, it may be helpful to first get the average cost per amenity (whether wifi is offered). This would be obtained with the following query:
SELECT
Amenities.wifi AS has_wifi,
AVG(Billing.price) AS avg_cost
FROM Billing
INNER JOIN Amenities ON
Amenities.listing_id = Billing.listing_id
GROUP BY Amenities.wifi
which gives you the following results:
+----------+-----------------------+
| has_wifi | avg_cost |
+----------+-----------------------+
| 0 | 1800.0000000000000000 |
| 1 | 1600.0000000000000000 |
+----------+-----------------------+
So far so good. So now we need to calculate the difference between these 2 rows. There are a number of different ways to do this, but one is to use a CASE expression to make one of the values negative, and then simply take the SUM of the result (note that I'm using a CTE, but you can also use a sub-query):
WITH
avg_by_wifi(has_wifi, avg_cost) AS
(
SELECT Amenities.wifi, AVG(Billing.price)
FROM Billing
INNER JOIN Amenities ON
Amenities.listing_id = Billing.listing_id
GROUP BY Amenities.wifi
)
SELECT
ABS(SUM
(
CASE
WHEN has_wifi = 1 THEN avg_cost
ELSE -1 * avg_cost
END
))
FROM avg_by_wifi
which gives us the expected value of 200.
Now regarding your data model:
If both your Billing and Amenities table only have 1 row for each listing, it makes sense to combine them into 1 table. For example: Listings(listing_id, price, wifi)
However, this is still problematic, because you probably have a bunch of other amenities you want to model (pool, sauna, etc.) So you might want to model a many-to-many relationship between listings and amenities using an intermediate table:
Listings(listing_id, price)
Amenities(amenity_id, amenity_name)
ListingsAmenities(listing_id, amenity_id)
This way, you could list multiple amenities for a given listing without having to add additional columns. It also becomes easy to store additional information about an amenity: What's the wifi password? How deep is the pool? etc.
Of course, using this model makes your original query (difference in average cost of listings by wifi) a bit tricker, but definitely still doable.
How would I go about joining results from multiple SQL queries so that they are side by side (but unrelated)?
The reason I am thinking of this is so that I can run 1 query in Google Big Query and it will return 1 single table which I can import into Excel and do some charts.
e.g. Query 1 looks at dataset TableA and returns:
**Metric:** Sales
**Value:** 3,402
And then Query 2 looks at dataset TableB and returns:
**Name:** John
**DOB:** 13 March
They would both use different tables and different filters, etc.
What would I do to make it look like:
---Sales----------John----
---3,402-------13 March----
Or alternatively:
-----Sales--------3,402-----
-----John-------13 March----
Or is there a totally different way to do this?
I can see the use case for the above, I've used something similar to create a single table from multiple tables with different metrics to query in Data Studio so that filters apply to all data in the dataset for example. However in that case, the data did share some dimensions that made it worthwhile doing.
If you are going to put those together with no relationship between the tables, I'd have 4 columns with TYPE describing the data in that row to make for easier filtering.
Type | Sales | Name | DOB
Use UNION ALL to put the rows together so you have something like
"Sales" | 3402 | null | null
"Customer Details" | null | John | 13 March
However, like the others said, make sure you have a good reason to do that otherwise you're just creating a bigger table to query for no reason.
I have a SQL Server database with an Access front end where I need users to be able to allocate resources to multiple projects at once. For example, we have a stock of a particular part that is used in a number of mechanical assemblies and we need to allocate these parts to the particular assemblies for production.
How do people normally represent data with these requirements?
Currently my data is stored as follows:
Resource | A | B | C (etc.)
---------+-------+-------+------
a | 10 | 20 | NULL
b | 11 | NULL | 31
c | 12 | NULL | NULL
d | NULL | 40 | NULL
Where A, B, C are different projects.
Advantages:
Easy visualisation and updating of Resources across all projects
Disadvantages:
Database structure changes every time a project is added or finished..
.. therefore many queries need to be rewritten/dynamic
Difficult to get aggregate resource allocation summaries
If old projects are retained, the table could easily exceed the column count limit
Alternatives
It seems to me that a more 'standard' representation would be a table as below. However, I have found it more or less impossible to present this to the user in a way that will allow easy visualisation and resource allocation over multiple projects.
ID | Project | Resource | Quantity
-----+---------+----------+----------
1 | A | a | 10
2 | A | b | 11
3 | A | c | 12
4 | B | a | 20
5 | B | d | 40
6 | C | b | 31
Advantages:
No structural changes when adding/removing projects
Easy resource summaries
Easy archiving of old projects
Disadvantages:
Views that recreate the interface of the top example using JOINs will only allow editing of one column at a time and will not allow insertion or deletion by updating from/to NULL:
e.g.
-- 'Resources' table has resource ID as primary key (& other info about resource),
-- 'ProjectResources' is the 'standard' table above
SELECT ResourceTable.ID, ProjA.Quantity AS A, ProjB.Quantity AS B, ProjC.Quantity AS C
FROM Resources
LEFT JOIN (SELECT ProjectResources.Quantity, ProjectResources.Resource
FROM ProjectResources
WHERE ProjectResources.Project = 'A') AS ProjA
ON Resources.ID = ProjA.Resource
LEFT JOIN (SELECT ProjectResources.Quantity, ProjectResources.Resource
FROM ProjectResources
WHERE ProjectResources.Project = 'B') AS ProjB
ON Resources.ID = ProjB.Resource
LEFT JOIN (SELECT ProjectResources.Quantity, ProjectResources.Resource
FROM ProjectResources
WHERE ProjectResources.Project = 'C') AS ProjC
ON Resources.ID = ProjC.Resource
Using an INSTEAD OF UPDATE trigger on the above view can make it fully editable, but using this view in any further queries to add information (e.g. the 'stock' of the resource that we're allocating) make these fields read only (error along lines of 'field cannot be updated because it it participates in a JOIN and has an INSTEAD OF UPDATE trigger')
Both of the above options require that the front end can cope with a varying number of table columns, which is a bit awkward, though in the second case this requirement can be limited only to specific circumstances.
Are there any other options on how to represent this data and allow easy editing that I have missed?
Your current database schema is not desirable for the reasons you listed. You are probably better off making the schema sensible and then dealing with the view in your UI rather than producing a bad schema to suit a particular view (if you want other views too, you'll be back in the same bucket with a hard-to-use schema for the new view).
While it's possible to produce the view as a crosstab query in Access, you will not be able to edit it (same as you found for your query; you are essentially building a crosstab query manually there).
A potential solution is to generate a temporary table in the front-end with the desired projects as columns (perhaps a subset of all projects), populate it with the resource data from the real table and display it as a datasheet. It will be editable and you can write changes back to the real table. It requires coding but it resolves the disadvantages.
I am working through a group by problem and could use some direction at this point. I want to summarize a number of variables by a grouping level which is different (but the same domain of values) for each of the variables to be summed. In pseudo-pseudo code, this is my issue: For each empYEAR variable (there are 20 or so employment-by-year variables in wide format), I want to sum it by the county in which the business was located in that particular year.
The data is a bunch of tables representing business establishments over a 20-year period from Dun & Bradstreet/NETS.
More details on the database, which is a number of flat files, all with the same primary key.
The primary key is DUNSNUMBER, which is present in several tables. There are tables detailing, for each year:
employment
county
sales
credit rating (and others)
all organized as follows (this table shows employment, but the other variables are similarly structured, with a year postfix).
dunsnumber|emp1990 |emp1991|emp1992|... |emp2011|
a | 12 |32 |31 |... | 35 |
b | |2 |3 |... | 5 |
c | 1 |1 | |... | |
d | 40 |86 |104 |... | 350 |
...
I would ultimately like to have a table that is structured like this:
county |emp1990|emp1991|emp1992|...|emp2011|sales1990|sales1991|sales1992|sales2011|...
A
B
C
...
My main challenge right now is this: How can I sum employment (or sales) by county by year as in the example table above, given that county as a grouping variable changes sometimes by the year and specified in another table?
It seems like something that would be fairly straightforward to do in, say, R with a long data format, but there are millions of records, so I prefer to keep the initial processing in postgres.
As I understand your question this sounds relatively straight forward. While I normally prefer normalized data to work with, I don't see that normalizing things beforehand will buy you anything specific here.
It seems to me you want something relatively simple like:
SELECT sum(emp1990), sum(emp1991), ....
FROM county c
JOIN emp e ON c.dunsnumber = e.dunsnumber
JOIN sales s ON c.dunsnumber = s.dunsnumber
JOIN ....
GROUP BY c.name, c.state;
I don't see a simpler way of doing this. Very likely you could query the system catalogs or information schema to generate a list of columns to sum up. the rest is a straight group by and join process as far as I can tell.
if the variable changes by name, the best thing to do in my experience is to put together a location view based on that union and join against it. This lets you hide the complexity from your main queries and as long as you don't also join the underlying tables should perform quite well.