Input Data
SNO | Name | Salary | HireDate
------------------------------------------
1 | A | 10 | 01-13-2014
2 | B | 20 | 11-15-2014
3 | C | 3 | 05-03-2015
4 | D | 4 | 07-03-2015
5 | E | 5 | 12-03-2015
6 | F | 60 | 25-03-2015
7 | G | 70 | 30-03-2015
Final Output Data
I want to get only current month data using hive query
SNO | Name | Salary | HireDate
----------------------------------------
3 | C | 3 | 05-03-2015
4 | D | 4 | 07-03-2015
5 | E | 5 | 12-03-2015
6 | F | 60 | 25-03-2015
7 | G | 70 | 30-03-2015
Do this in shell script:
curmon=`date +%m-%Y`
cusdate="01-$curmon";
$HIVE_HOME/bin/hive -e "select * from tablename where HireDate>$cusdate;"
curmon will store current month and year.
cusdate will store 1st day of this month.
Hive query will display all the results greater than 1st day of this month. (Change tablename and column as per your requirements)
Just use current_date and the date time functions in Hive. This is probably the easiest way:
select id.*
from inputdata id
where year(hiredate) = year(current_date()) and
month(hiredate) = month(current_date());
EDIT:
Having just tried this out, current_date() is not in at least one implementation of Hive 0.14, despite the documentation. So, you can try:
select id.*
from inputdata id
where year(hiredate) = year(from_unixtime(unix_timestamp())) and
month(hiredate) = month(from_unixtime(unix_timestamp()));
Related
Hi I have following Table with the current month 'Month':
+---------------+
| current_Month |
+---------------+
| 12 |
+---------------+
And I have another Table with workers 'Workers'
+--------+--------------------------+
| Name | Month_joined_the_company |
+--------+--------------------------+
| Peter | 12 |
| Paul | 9 |
| Sarah | 5 |
| Donald | 12 |
+--------+--------------------------+
I now want, based on my Month table, Display all workers which joined the company untill the previous month the current month is 10 I would like to have this output
+--------+--------------------------+
| Name | Month_joined_the_company |
+--------+--------------------------+
| Paul | 9 |
| Sarah | 5 |
+--------+--------------------------+
But on the end of the year, I would like to include all workers even thos which month is equal with the current month
+--------+--------------------------+
| Name | Month_joined_the_company |
+--------+--------------------------+
| Peter | 12 |
| Paul | 9 |
| Sarah | 5 |
| Donald | 12 |
+--------+--------------------------+
I now have this Statement, but it does not work...
SELECT *
FROM workers
WHERE
CASE
WHEN (SELECT TOP (1) Current_Month FROM Month) = 12
THEN (Month_joined_the_company <= (SELECT TOP (1) Current_Month FROM Month))
ELSE (Month_joined_the_company < (SELECT TOP (1) Current_Month FROM Month))
END
But this does not work and I get an error. Can someone help me, how I can use CASE in a WHERE-Clause
Is this what you want?
select w.*
from workers w
inner join month m
on m.current_month = 12
or w.month_joined_the_company < m.current_month
This phrases as: if current_month = 12 then return all workers, else just return those whose month_joined_the_company is stricly smaller than current_month.
NB: you should probably consider use date datatypes to store these values, otherwise what happens when a new year begins?
here is my table called "Employee"
eID | name |
==============
1 | Mike |
2 | Josh |
3 | Mike |
And table called "Sells"
sID | eID | | price |
=========================
1 | 1 | | 8 |
2 | 3 | | 9 |
3 | 3 | | 5 |
4 | 1 | | 4 |
5 | 1 | | 3 |
This should be my expected result: returns the total income per employee
name | Income |
==================
Mike | 15 |
Josh | 0 |
Mike | 14 |
Actually, I know use the query "SUM...GROUP BY..." to get the incomes of 15 and 14, but I don't know how to get the income of 0 which is not shown on the "Sells" table.
Could someone give me some help? Thanks a lot.
You just need to use a left outer join, so you can get the sum for missing values too. You could use case expression to deal with null values
SELECT e.name,
COALESCE(SUM(price), 0) as Income
FROM employees e
LEFT OUTER JOIN sells s
ON e.eid = s.eid
GROUP BY e.eid, e.name
Edited: case expression is not needed. I put coalesce on the return of sum fuction, in order to deal with missing values (SUM over an empty set returns NULL)
I am new to sparksql and i was trying to experiment certain queries with that.
This is the query i am trying to execute
sqlContext.sql(SELECT id , category ,AVG(mark) FROM data GROUP BY id, category)
I am not getting proper output when i run the query.
instead of actual value of category i am getting some value as 1,2,3.
I am stuck at this weird error for long time
but when i do simple select statement and one group by its working perfectly
sqlContext.sql(SELECT id , category FROM data)
sqlContext.sql(SELECT id ,AVG(mark) FROM data GROUP BY id)
What is wrong? Does SPARKSQL has something to do with multiple group by.
right now i am running this complex query
sqlContext.sql(SELECT data.id , data.category, AVG(id_avg.met_avg) FROM (SELECT id, AVG(mark) AS met_avg FROM data GROUP BY id) AS id_avg, data GROUP BY data.category, data.id)
This works, but taking a longer time to execute.
Please Help
Sample data:
|id | category | marks
| 1 | a | 40
| 2 | b | 44
| 3 | a | 50
| 4 | b | 40
| 1 | a | 30
The output should be:
|id | category | avg
| 1 | a | 35
| 2 | b | 44
| 3 | a | 50
| 4 | b | 40
Please try this query:
SELECT
data.id
, data.category
, AVG(mark)
FROM data
GROUP BY
data.id
, data.category
Based on this sample data:
|id | category | marks
| 1 | a | 40
| 2 | b | 44
| 3 | a | 50
| 4 | b | 40
| 1 | a | 30
The output WILL be this:
|id | category | avg
| 1 | a | 35
| 2 | b | 44
| 3 | a | 50
| 4 | b | 40
and, the following expected row cannot be produced using group by:
| 5 | a | 30
That is a bug in sparksql.
Try using the next version. Its fixed.
i got the proper output by using spark-1.0.2
it worked with pure scala code also. Try either of them :)
This should be a simple one, but say I have a table with data like this:
| ID | Date | Value |
| 1 | 01/01/2013 | 40 |
| 2 | 03/01/2013 | 20 |
| 3 | 10/01/2013 | 30 |
| 4 | 14/02/2013 | 60 |
| 5 | 15/03/2013 | 10 |
| 6 | 27/03/2013 | 70 |
| 7 | 01/04/2013 | 60 |
| 8 | 01/06/2013 | 20 |
What I want is the sum of values per week of the year, showing ALL weeks.. (for use in an excel graph)
What my query gives me, is only the weeks that are actually in the database.
With SQL you cannot return rows that don't exist in some table. To get the effect you want you could create a table called WeeksInYear with only one field WeekNumber that is an Int. Populate the table with all the week numbers. Then JOIN that table to this one.
The query would then look something like the following:
SELECT w.WeekNumber, SUM(m.Value)
FROM MyTable as m
RIGHT OUTER JOIN WeeksInYear AS w
ON DATEPART(wk, m.date) = w.WeekNumber
GROUP BY w.WeekNumber
The missing weeks will not have any data in MyTable and show a 0.
I need to create a form to summarize wage of each employees according to Date_start and Date_end that I selected.
I have 3 related tables.
tbl_labor
lb_id | lb_name | lb_OT ($/day) | If_social_sec
1 | John | 10 | yes
2 | Mary | 10 | no
tbl_production
pdtn_date | lb_id | pdtn_qty(pcs) | pd_making_id
5/9/12 | 1 | 200 | 12
5/9/12 | 1 | 40 | 13
5/9/12 | 2 | 300 | 12
7/9/12 | 1 | 48 | 13
13/9/12 | 2 | 220 | 14
15/9/12 | 1 | 20 | 12
20/9/12 | 1 | 33 | 14
21/9/12 | 2 | 55 | 14
21/9/12 | 1 | 20 | 12
tbl_pdWk_process
pd_making_id | pd_cost($/dozen) | pd_id
12 | 2 | 001
13 | 5 | 001
14 | 6 | 002
The outcome will look like this:
lb_name | no.working days | Total($)| OT payment | Social_sec($)| Net Wage |
John | 4 | 98.83 | 20 (2x10) | 15 | 103.83 (98.83+20-15)|
Mary | 2 | 160 | 10 (1x10) | 0 | 170 (160+10-0) |
My conditions are:
Wage are calculate between 2 dates that i specify eg. 5/9/12 - 20/9/12
Wage must be calculated from (pd_cost * pdtn_qty). However, pdtn_qty was kept in 'pieces' whereas pd_cost was kept in 'dozen'. Therefore the formula should be (pdtn_qty * pd_cost)/12
Add OT * no. of OT days that each worker did (eg. John had 2 OT days, Mary 1 OT day)
Total wages must be deducted eg. 15$ if If_social_sec are "TRUE"
Count no. of working days that each employees had worked.
I've tried but i couldn't merge all this condition together in one SQL statement, so could you please help me. Thank you.
OK this is really messy. Mainly because Access has no COUNT(DISTINCT ) option. So counting the working days is a mess. If you can skip that, then you can drop all the pdn1,pdn2,pdn3 stuff. But id does work. Couple of notes
1. I think your maths is not quite right in the example given, I make it like this:
2 I've used the IIF clause to simulate 2 OT for john, 1 for mary. You'll need to change that. Good luck.
select
lab.lb_name,
max(days),
sum(prod.pdtn_qty * pdWk.pd_cost / 12) as Total ,
max(lab.lb_OT * iif(lab.lb_id=1,2,1)) as OTPayment,
max(iif(lab.if_social_sec='yes' , 15,0 ) ) as Social_Sec,
sum(prod.pdtn_qty * pdWk.pd_cost / 12.00) +
max(lab.lb_OT * iif(lab.lb_id=1,2,1)) -
max(iif(lab.if_social_sec='yes' , 15,0 ) ) as NetWage
from
tbl_labor as lab,
tbl_production as prod,
tbl_pdWk_process as pdwk,
(select pdn2.lb_id, count(pdn2.lb_id) as days from
(select lb_id
from tbl_production pdn1
where pdn1.pdtn_date >= #9/5/2012#
and pdn1.pdtn_date <= #2012-09-20#
group by lb_id, pdtn_date ) as pdn2
group by pdn2.lb_id) as pdn3
where prod.pdtn_date >= #9/5/2012#
and prod.pdtn_date <= #2012-09-20#
and prod.lb_id = lab.lb_id
and prod.pd_making_id = pdwk.pd_making_id
and lab.lb_id = pdn3.lb_id
group by lab.lb_name
OK to add the items not in production table, you'll need to append something like this:
Union
select lab.lb_name,
0,
0,
max(lab.lb_OT * iif(lab.lb_id=1,2,1)) ,
max(iif(lab.if_social_sec='yes' , 15,0 ) ),0
from tbl_labor lab
where lb_id not in ( select lb_id from tbl_production where pdtn_date >= #2012-09-05# and pdtn_date <= #2012-09-20# )
group by lab.lb_name
Hope this helps.