I have some Data which i want to PIVOT, and i am able to PIVOT it using a Single Column Scenario like below -
Dim FY2019 FY2020 FY2021
NA 20000 30000 25000
EUROPE 10000 15000 20000
ASIA 30000 10000 20000
spark.table("default.sample").groupBy("Dim").pivot("qtr").sum("amount")
Now the challenge am facing is when i am trying to Pivot using 2 or more columns if i want to.
Like i want the data in the below format where the Pivoting should be of 2 Columns.
Dim Country FY2019 FY2020 FY2021
NA US 15000 20000 20000
NA CANADA 5000 10000 5000
EUROPE UK 8000 9000 15000
EUROPE FRANCE 2000 6000 5000
ASIA INDIA 10000 5000 10000
ASIA CHINA 10000 5000 10000
Please help and provide guidance how i can i achieve it using Spark?
Thanks in Advance
On a form, I have a chart that shows the Increase History of each employee (What rating they were given each year, and how much of an increase they got)
I can't post an image since I have less than 10 reputation, but I hope it's clear.
What I need is a similar graph, but for the history of each employee's salary. I have a StartingSalary field in IncreaseEmployeesQ, and an Increase field. It's complicated, all I've managed to do is this:
SELECT IncreaseEmployeesQ.LocalID, Sum([Increase]+[StartingSalary]) AS CurrentSalary
FROM IncreaseEmployeesQ
GROUP BY IncreaseEmployeesQ.LocalID;
But what this does is add the StartingSalary each time, since it's repeated in each year. and it gives me one value for each employee, instead of one value for each year so I can have a chart that tracks the progress of the employee's salary.
I tried going to the original IncreaesT instead of the query that has it and EmployeesT (IncreaseEmployeesQ), thinking maybe I can have a calculated field if I add a StartingSalary field in IncreaseT (It's originally in EmployeesT) and then link it in relationships and enforce referential integrity, but I kept getting an error message. After some research I gathered that the reason is because the two tables have different Primary keys, so I resorted to the Query.
Is the chart I'm aiming to get to even possible? a chart that shows how each employee's salary has been progressing since 2010? (that's as far back as my data goes)
-assuming that a query is the right way to get this done- The query I'm working on looks like this:
LocalID Increase Years StartingSalary
1 1000 2013 7000
1 500 2014 7000
1 0 2015 7000
1 500 2016 7000
2 0 2013 5000
2 500 2014 5000
2 500 2015 5000
2 0 2016 5000
What I want it to look like (so I make a chart later) is this:
LocalID Increase Years StartingSalary CurrentSalary
1 1000 2013 7000 8000
1 500 2014 7000 8500
1 0 2015 7000 8500
1 500 2016 7000 9000
2 0 2013 5000 5000
2 500 2014 5000 5500
2 500 2015 5000 6000
2 0 2016 5000 6000
If it turns out like this, I can make a chart that has the Years and the CurrentSalary for each employee.
But all I've managed to do is the code above, which gives me this result
LocalID Increase Years StartingSalary CurrentSalary
1 1000 2013 7000 30000
1 500 2014 7000 30000
1 0 2015 7000 30000
1 500 2016 7000 30000
2 0 2013 5000 21000
2 500 2014 5000 21000
2 500 2015 5000 21000
2 0 2016 5000 21000
I hope everything is clear now
You want a cumulative sum. One way to do this in MS Access uses a correlated subquery:
SELECT ieq.*,
(ieq.StartingSalary +
(SELECT SUM(increase)
FROM IncreaseEmployeesQ as ieq2
WHERE ieq2.LocalID = ieq.LocalId AND ieq2.Years <= ieq.Years
)
) as CurrentSalary
FROM IncreaseEmployeesQ as ieq
There's my sql data:
code name total
---------------
3 Sprite 2400
17 Coke 1500
6 Dew 1000
17 Coke 3000
6 Dew 2000
But code and name has duplicated values and I want to sum total from each duplicated field.
Something like this:
code name total
---------------
3 Sprite 2400
17 Coke 4500
6 Dew 3000
How could I do that in sql?
SELECT code, name, sum(total) AS total FROM table GROUP BY code, name
eg. i have records like this in my table.
Name Date Salary
Raja 01/10/2000 5000
Raja 02/10/2000 5000
Raja 03/10/2000 5000
Anu 01/10/2000 3000
Anu 02/10/2000 3000
Anu 03/10/2000 3000
King 01/10/2000 4000
King 02/10/2000 4000
King 03/10/2000 4000
i want to take print like the below eg using crystal report. (sum all the salary and display in single row).
Salary from 01/10/1981 To 03/10/1981
NAME Salary
Raja 15,000
Anu 9,000
King 12,000
use a simple sql statement, with group-by clause and sum on the salary column.
select Name, Sum(salary)
from table
group by Name
I'm analyzing some code that utilizes empty OVER clauses in the contest of Count().
Example:
SELECT
ROW_NUMBER() OVER (ORDER BY Priority DESC) AS RowID,
CAST((COUNT(*) OVER() / #pagesize) AS Int) AS TotalPages,
I'm trying to understand why the empty OVER clause is being used here.
There are other standard select elements below those two lines I listed above, and when I remove the empty OVER clause from the second the TotalPages line, I get errors like this:
Column 'TableA.Priority' is invalid in the select list because it is not contained in either an aggregate function or the GROUP BY clause.
As soon as I put the OVER() back, the error is gone.
My understanding of the OVER clause is very limited... I feel Like I understand what's going on in the RowID line... but the TotalPages line just baffles me.
OVER() is part of analytic function and define partitions in your recordset. OVER() is just one partition and applied to the whole dataset
i.e. COUNT(*) OVER() will return in each row how many records in your dataset.
look to this http://msdn.microsoft.com/en-us/library/ms189461.aspx
Say our table is employees:
+-----------+-------+---------+
| badge_num | name | surname |
+-----------+-------+---------+
| 1 | John | Smith |
| 2 | Mark | Pence |
| 3 | Steve | Smith |
| 4 | Bob | Smith |
+-----------+-------+---------+
Running
SELECT surname, COUNT(*)
FROM employees
GROUP BY surname;
we'll get:
+---------+----------+
| surname | COUNT(*) |
+---------+----------+
| Smith | 3 |
| Pence | 1 |
+---------+----------+
While running
SELECT surname, COUNT(*) OVER()
FROM employees
GROUP BY surname;
we'll get:
+---------+-----------------+
| surname | COUNT(*) OVER() |
+---------+-----------------+
| Smith | 2 |
| Pence | 2 |
+---------+-----------------+
In the second case, in each row we are just counting the number of rows of the whole select (not the single partition).
To summarize things, the OVER clause can be used with Ranking Functions(Rank, Row_Number, Dense_Rank..), Aggregate Functions like (AVG, Max, Min, SUM...etc) and Analytics Functions like (First_Value, Last_Value, and few others).
Let's See basic syntax of OVER clause
OVER (
[ <PARTITION BY clause> ]
[ <ORDER BY clause> ]
[ <ROW or RANGE clause> ]
)
PARTITION BY:
It is used to partition data and perform operations on groups with the same data.
ORDER BY:
It is used to define the logical order of data in Partitions. When we don't specify Partition, entire resultset is considered as a single partition
:
This can be used to specify what rows are supposed to be considered in a partition when performing the operation.
Let's take an example:
Here is my dataset:
Id Name Gender Salary
----------- -------------------------------------------------- ---------- -----------
1 Mark Male 5000
2 John Male 4500
3 Pavan Male 5000
4 Pam Female 5500
5 Sara Female 4000
6 Aradhya Female 3500
7 Tom Male 5500
8 Mary Female 5000
9 Ben Male 6500
10 Jodi Female 7000
11 Tom Male 5500
12 Ron Male 5000
So let me execute different scenarios and see how data is impacted and I'll come from difficult syntax to simple one
Select *,SUM(salary) Over(order by salary RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) as sum_sal from employees
Id Name Gender Salary sum_sal
----------- -------------------------------------------------- ---------- ----------- -----------
6 Aradhya Female 3500 3500
5 Sara Female 4000 7500
2 John Male 4500 12000
3 Pavan Male 5000 32000
1 Mark Male 5000 32000
8 Mary Female 5000 32000
12 Ron Male 5000 32000
11 Tom Male 5500 48500
7 Tom Male 5500 48500
4 Pam Female 5500 48500
9 Ben Male 6500 55000
10 Jodi Female 7000 62000
Just observe the sum_sal part. Here I am using order by Salary and using "RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW".
In this case, we are not using partition so entire data will be treated as one partition and we are ordering on salary.
And the important thing here is UNBOUNDED PRECEDING AND CURRENT ROW. This means when we are calculating the sum, from starting row to the current row for each row.
But if we see rows with salary 5000 and name="Pavan", ideally it should be 17000 and for salary=5000 and name=Mark, it should be 22000. But as we are using RANGE and in this case, if it finds any similar elements then it considers them as the same logical group and performs an operation on them and assigns value to each item in that group. That is the reason why we have the same value for salary=5000. The engine went up to salary=5000 and Name=Ron and calculated sum and then assigned it to all salary=5000.
Select *,SUM(salary) Over(order by salary ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) as sum_sal from employees
Id Name Gender Salary sum_sal
----------- -------------------------------------------------- ---------- ----------- -----------
6 Aradhya Female 3500 3500
5 Sara Female 4000 7500
2 John Male 4500 12000
3 Pavan Male 5000 17000
1 Mark Male 5000 22000
8 Mary Female 5000 27000
12 Ron Male 5000 32000
11 Tom Male 5500 37500
7 Tom Male 5500 43000
4 Pam Female 5500 48500
9 Ben Male 6500 55000
10 Jodi Female 7000 62000
So with ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW The difference is for same value items instead of grouping them together, It calculates SUM from starting row to current row and it doesn't treat items with same value differently like RANGE
Select *,SUM(salary) Over(order by salary) as sum_sal from employees
Id Name Gender Salary sum_sal
----------- -------------------------------------------------- ---------- ----------- -----------
6 Aradhya Female 3500 3500
5 Sara Female 4000 7500
2 John Male 4500 12000
3 Pavan Male 5000 32000
1 Mark Male 5000 32000
8 Mary Female 5000 32000
12 Ron Male 5000 32000
11 Tom Male 5500 48500
7 Tom Male 5500 48500
4 Pam Female 5500 48500
9 Ben Male 6500 55000
10 Jodi Female 7000 62000
These results are the same as
Select *, SUM(salary) Over(order by salary RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) as sum_sal from employees
That is because Over(order by salary) is just a short cut of Over(order by salary RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)
So wherever we simply specify Order by without ROWS or RANGE it is taking RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW as default.
Note: This is applicable only to Functions that actually accept RANGE/ROW. For example, ROW_NUMBER and few others don't accept RANGE/ROW and in that case, this doesn't come into the picture.
Till now we saw that Over clause with an order by is taking Range/ROWS and syntax looks something like this RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
And it is actually calculating up to the current row from the first row. But what If it wants to calculate values for the entire partition of data and have it for each column (that is from 1st row to last row). Here is the query for that
Select *,sum(salary) Over(order by salary ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) as sum_sal from employees
Id Name Gender Salary sum_sal
----------- -------------------------------------------------- ---------- ----------- -----------
1 Mark Male 5000 62000
2 John Male 4500 62000
3 Pavan Male 5000 62000
4 Pam Female 5500 62000
5 Sara Female 4000 62000
6 Aradhya Female 3500 62000
7 Tom Male 5500 62000
8 Mary Female 5000 62000
9 Ben Male 6500 62000
10 Jodi Female 7000 62000
11 Tom Male 5500 62000
12 Ron Male 5000 62000
Instead of CURRENT ROW, I am specifying UNBOUNDED FOLLOWING which instructs the engine to calculate till the last record of partition for each row.
Now coming to your point on what is OVER() with empty braces?
It is just a short cut for Over(order by salary ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING)
Here we are indirectly specifying to treat all my resultset as a single partition and then perform calculations from the first record to the last record of each partition.
Select *,Sum(salary) Over() as sum_sal from employees
Id Name Gender Salary sum_sal
----------- -------------------------------------------------- ---------- ----------- -----------
1 Mark Male 5000 62000
2 John Male 4500 62000
3 Pavan Male 5000 62000
4 Pam Female 5500 62000
5 Sara Female 4000 62000
6 Aradhya Female 3500 62000
7 Tom Male 5500 62000
8 Mary Female 5000 62000
9 Ben Male 6500 62000
10 Jodi Female 7000 62000
11 Tom Male 5500 62000
12 Ron Male 5000 62000
I did create a video on this and if you are interested you can visit it.
https://www.youtube.com/watch?v=CvVenuVUqto&t=1177s
Thanks,
Pavan Kumar Aryasomayajulu
HTTP://xyzcoder.github.io