How to do date difference with previous row date using hiveql? - hive

I have a date column I need to do date difference with previous row date using hive query?

Below would be the query -
select date, datediff(date, prev_date) from (select date, lag(date) over (partition by <colName> order by <Name>) as prev_date from tableName) t1;
If you can provide the sample data with tableName Can help you with the exact query

Related

Find the average lowest item in a collection grouped by date in SQL

My SQL isn't the best - I can get this working in C# but it seems more efficient to get it in my data layer - I've got a table Prices:
ID
Price
DateTime
Each row is exactly 1 hour from the next, so I have a snapshot of a price every hour.
I'm trying to work out which hour in a day over the entire dataset has the lowest price (on average).
So ideally I'm after a list of each hour in the day ranked by how cheap on average that hour is over the entire dataset - so a maximum of 24 rows (one for each hour in the day).
Any help would be greatly appreciated!
Thanks :D
Which database are you on?
Different DBs have different ways to extract date from a timestamp column.
Postgres has date(timestamp), In Oracle, you can use trunc(timestamp). Or most DBs have to_char/to_date. So you can try that.
Once you have extracted the date, you can try something like this -
select ID,
Price,
DateTime,
trunc(DateTime) as day,
rank() over (partition by trunc(DateTime) order by Price asc) as least_for_day
from Prices
Now you can use the "least_for_day" ranked column and select by day.
Again, depending on the DB, you can either directly qualify on the ranked column in the same SQL or use the above as a sub-query and filter for the rank.
You can use a query like below
select
hour,
avg(daily_rank) avg_rank
from
(
select *, hour= format((datetime as datetime),'HH'), daily_rank= dense_rank() over (partition by cast(datetime as date) order by price asc)
) t
group by hour
Thank you very much to #Many Manjunath and #DhruvJoshi. Final solution below;
WITH prices AS
(
SELECT
[Price],
[DateTime],
CAST([DateTime] AS TIME) 'Time',
CAST([DateTime] as date) 'Date',
rank() over (partition by cast([DateTime] as date) order by [Price] asc) as least_for_day
FROM [dbo].[Prices]
)
SELECT [Time], count(*) 'Qty Cheapest' FROM prices
WHERE least_for_day = 1
GROUP BY [Time]
ORDER BY 2 DESC
That returns 24 rows:

Select Max difference between two dates in the same column

I have a table of accident date I want to calculate the maximum between the difference of date i and date i + 1 which are in the same column. when we declare an accident date, I want to find the record of days without accidents.
You can use lag(). Assuming a table structure like mytable(dt), where dt is of a date-like datatype, you would do:
select max(diff)
from (select dt - lag(dt) over(order by dt) diff from mytable) t

SQL Server data search with date range

I have a table with the following columns:
Date
Skills,
Customer ID
I want to find out Date(x), Customers, Count of Customers in between Date(x) and Date(x)+6
Can somebody guide me how to make this query, or can I create this function in SQL Server?
If I understand you correctly, you want something like this:
(take care, can be bad syntax, because i "work" only with oracle. But I think that it should work)
select date, customer_id, COUNT(*)
from your_table --add your table
where date between getdate() and DATEADD(day, 6, getdate())
-- between current database system date and +6 day
group by date, customer id
order by COUNT (*) desc -- if you want, you can order your result - ASC||DESC
If you have data on each date, then perhaps this is what you want:
select date, count(*),
sum(count(*)) over (order by date rows between 6 preceding and current row) as week_count
from t
group by date;

Latest date from date, month, year

I have an SQLite DB with date, month, year fields in integers (I believe they should have used a date field but the choice wasn't mine). I would like to select the row whose date value is the latest. What is the best query to do that?
select * from your_table
order by year, month, `date` desc
limit 1

Last day of the month with a twist in SQLPLUS

I would appreciate a little expert help please.
in an SQL SELECT statement I am trying to get the last day with data per month for the last year.
Example, I am easily able to get the last day of each month and join that to my data table, but the problem is, if the last day of the month does not have data, then there is no returned data. What I need is for the SELECT to return the last day with data for the month.
This is probably easy to do, but to be honest, my brain fart is starting to hurt.
I've attached the select below that works for returning the data for only the last day of the month for the last 12 months.
Thanks in advance for your help!
SELECT fd.cust_id,fd.server_name,fd.instance_name,
TRUNC(fd.coll_date) AS coll_date,fd.column_name
FROM super_table fd,
(SELECT TRUNC(daterange,'MM')-1 first_of_month
FROM (
select TRUNC(sysdate-365,'MM') + level as DateRange
from dual
connect by level<=365)
GROUP BY TRUNC(daterange,'MM')) fom
WHERE fd.cust_id = :CUST_ID
AND fd.coll_date > SYSDATE-400
AND TRUNC(fd.coll_date) = fom.first_of_month
GROUP BY fd.cust_id,fd.server_name,fd.instance_name,
TRUNC(fd.coll_date),fd.column_name
ORDER BY fd.server_name,fd.instance_name,TRUNC(fd.coll_date)
You probably need to group your data so that each month's data is in the group, and then within the group select the maximum date present. The sub-query might be:
SELECT MAX(coll_date) AS last_day_of_month
FROM Super_Table AS fd
GROUP BY YEAR(coll_date) * 100 + MONTH(coll_date);
This presumes that the functions YEAR() and MONTH() exist to extract the year and month from a date as an integer value. Clearly, this doesn't constrain the range of dates - you can do that, too. If you don't have the functions in Oracle, then you do some sort of manipulation to get the equivalent result.
Using information from Rhose (thanks):
SELECT MAX(coll_date) AS last_day_of_month
FROM Super_Table AS fd
GROUP BY TO_CHAR(coll_date, 'YYYYMM');
This achieves the same net result, putting all dates from the same calendar month into a group and then determining the maximum value present within that group.
Here's another approach, if ANSI row_number() is supported:
with RevDayRanked(itemDate,rn) as (
select
cast(coll_date as date),
row_number() over (
partition by datediff(month,coll_date,'2000-01-01') -- rewrite datediff as needed for your platform
order by coll_date desc
)
from super_table
)
select itemDate
from RevDayRanked
where rn = 1;
Rows numbered 1 will be nondeterministically chosen among rows on the last active date of the month, so you don't need distinct. If you want information out of the table for all rows on these dates, use rank() over days instead of row_number() over coll_date values, so a value of 1 appears for any row on the last active date of the month, and select the additional columns you need:
with RevDayRanked(cust_id, server_name, coll_date, rk) as (
select
cust_id, server_name, coll_date,
rank() over (
partition by datediff(month,coll_date,'2000-01-01')
order by cast(coll_date as date) desc
)
from super_table
)
select cust_id, server_name, coll_date
from RevDayRanked
where rk = 1;
If row_number() and rank() aren't supported, another approach is this (for the second query above). Select all rows from your table for which there's no row in the table from a later day in the same month.
select
cust_id, server_name, coll_date
from super_table as ST1
where not exists (
select *
from super_table as ST2
where datediff(month,ST1.coll_date,ST2.coll_date) = 0
and cast(ST2.coll_date as date) > cast(ST1.coll_date as date)
)
If you have to do this kind of thing a lot, see if you can create an index over computed columns that hold cast(coll_date as date) and a month indicator like datediff(month,'2001-01-01',coll_date). That'll make more of the predicates SARGs.
Putting the above pieces together, would something like this work for you?
SELECT fd.cust_id,
fd.server_name,
fd.instance_name,
TRUNC(fd.coll_date) AS coll_date,
fd.column_name
FROM super_table fd,
WHERE fd.cust_id = :CUST_ID
AND TRUNC(fd.coll_date) IN (
SELECT MAX(TRUNC(coll_date))
FROM super_table
WHERE coll_date > SYSDATE - 400
AND cust_id = :CUST_ID
GROUP BY TO_CHAR(coll_date,'YYYYMM')
)
GROUP BY fd.cust_id,fd.server_name,fd.instance_name,TRUNC(fd.coll_date),fd.column_name
ORDER BY fd.server_name,fd.instance_name,TRUNC(fd.coll_date)