Exclude values from same column depending on date, using Hive SQL - sql

I need to extract a set of IDs from a table using Hive. The table from which I am to extract the data is partitioned by date. What I need are distinct IDs that appear in the table eight days ago but are not in the table for dates that represent the last seven days. I have tried using a subquery:
SELECT DISTINCT id
FROM my_table
WHERE date = '2016-07-14'
AND id NOT IN (
SELECT DISTINCT id
FROM my_table
WHERE date BETWEEN '2016-07-15' AND '2016-07-21'
);
However, I am getting an error message containing Unsupported language features in query (entire error message is too long to post here). Since I cannot use this approach in Hive SQL, what are my options here?

The same functionality can be done using LEFT JOIN:
SELECT a.ID
FROM
(
SELECT DISTINCT ID
FROM my_table
WHERE date = '2016-07-14'
)a
LEFT JOIN (
SELECT DISTINCT ID
FROM my_table
WHERE date BETWEEN '2016-07-15' AND '2016-07-21'
) s on a.ID=s.ID
WHERE s.ID IS NULL;

Related

Finding Max(Date) BEFORE specified date in Redshift SQL

I have a table (Table A) in SQL (AWS Redshift) where I've isolated my beginning population that contains account id's and dates. I'd like to take the output from that table and LEFT join back to the "accounts" table to ONLY return the start date that precedes or comes directly before the date stored in the table from my output.
Table A (Beg Pop)
-------
select account_id,
min(start_date),
min(end_date)
from accounts
group by 1;
I want to return ONLY the date that precedes the date in my current table where account_id match. I'm looking for something like...
Table B
-------
select a.account_id,
a.start_date,
a.end_date,
b.start_date_prev,
b.end_date_prev
from accounts as a
left join accounts as b on a.account_id = b.account_id
where max(b.start_date) less than a.start_date;
Ultimately, I want to return everything from table a and only the dates where max(start_date) is less than the start_date from table A. I know aggregation is not allowed in the WHERE clause and I guess I can do a subquery but I only want the Max date BEFORE the dates in my output. Any suggestions are greatly appreciated.
I want to return ONLY the date that precedes the date in my current table where account_id match
If you want the previous date for a given row, use lag():
select a.*,
lag(start_date) over (partition by account_id order by start_date) as prev_start_date
from accounts a;
As I understand from the requirement is to display all rows from a base table with the preceeding data sorted based on a column and with some conditions
Please check following example which I took from article Select Next and Previous Rows with Current Row using SQL CTE Expression
WITH CTE as (
SELECT
ROW_NUMBER() OVER (PARTITION BY account_id ORDER BY start_date) as RN,
*
FROM accounts
)
SELECT
PreviousRow.*,
CurrentRow.*,
NextRow.*
FROM CTE as CurrentRow
LEFT JOIN CTE as PreviousRow ON
PreviousRow.RN = CurrentRow.RN - 1 and PreviousRow.account_id = CurrentRow.account_id
LEFT JOIN CTE as NextRow ON
NextRow.RN = CurrentRow.RN + 1 and NextRow.account_id = CurrentRow.account_id
ORDER BY CurrentRow.account_id, CurrentRow.start_date;
I tested with following sample data and it seems to be working
create table accounts(account_id int, start_date date, end_date date);
insert into accounts values (1,'20201001','20201003');
insert into accounts values (1,'20201002','20201005');
insert into accounts values (1,'20201007','20201008');
insert into accounts values (1,'20201011','20201013');
insert into accounts values (2,'20201001','20201002');
insert into accounts values (2,'20201015','20201016');
Output is as follows

Need to find a difference of data from the same table in hive

I have a history table with loaded timestamp column. I need to fetch the subtracted data using the timestamp column.
Logic:To get the email address by subtracting data from (loaded_timestamp -1)and current_timestamp.Only the subtracted data should be the output.
Select query :
select t1.email_addr
from (select *
from table t1
where loaded_timestamp = current_timestamp
) left outer join
(select *
from table t2
where loaded_timestamp = date_sub(current_timestamp,1)
)
where t1.email!=t2.email;
Table has following columns
Email address, First name , last name, loaded_timestamp.
xxx#gmail.com,xxx,aaa,2020-03-08.
yyy#gmail.com,yyy,bbb,2020-03-08.
zzz#gmail.com,zzz,ccc,2020-03-08.
xxx#gmail.com,xxx,aaa,2020-03-09.
yyy#gmail.com,yyy,bbb,2020-03-09.
Desired Result
zzz#gmail.com
So if subtract the two dates from the same table i.e (2020-03-09 - 2020-03-08 ). I should get only the record which is not matching . Matching records should be discarded and unmatched record should be the output.
The best I can figure out is that you want emails that appear only once. If that is the case, use window functions:
select t.*
from (select t.*, count(*) over (partition by email) as cnt
from t
) t
where cnt = 1;
If you want emails in the data but not loaded on the current date, then:
select t.email
from t
group by t.email
having max(timestamp) <> current_date;

SQL query to sum a column prior to date and show all entries after that date

I have a table where limits were sanctioned to the customer
I am trying to get the output as below picture i.e. total amount sanctioned till particular date
I am trying below code but this sums the total sanction amount
select gam.id, sum(SANCTION_AMOUNT) from gam
join (select ID,ACCOUNT_OPEN_DATE from gam where ACCOUNT_OPEN_DATE between'01-04-2019' and '30-04-2019' AND SCHEME_CODE IN ('SB','CCKLY')) ) action
on( gam.ACCOUNT_OPEN_DATE <=action.ACCOUNT_OPEN_DATE and gam.id=action.cust_id) group by gam.id;
In Oracle, this can be a way:
select id, sanction_amount, scheme_code, account_open_date,
sum(sanction_amount) over (partition BY ID order by account_open_date) as total_sanction_amount
from gam
order by account_open_date
Not sure your database is MySQL or Oracle, But this below script is workable in most of the database. Just adjust the table and column names accordingly.
You can check MySQL DEMO HERE
SELECT *,
(
SELECT SUM(sanction_Amount)
FROM Your_Table B
WHERE B.ID = A.ID
AND B.acc_open_date <= A.acc_open_date
) Total_sanction_Amount
FROM Your_Table A

Select last item for each unique column value

I have a table containing message logs. Each conversation has a conversation ID.
I want to select distinct conversation IDs, and for each of them, find the latest message with that conversation ID and join it into the row.
This is what I tried but it doesn't add any data into the table except the two columns (conversationId and id). I want to get all columns from that table for each row with the latest
SELECT
logs.conversationId,
-- latest message id
MAX(logs.id) AS id
FROM [dbo].[Logs] AS logs
-- trying to get the remaining columns for the last message with that conversation ID
LEFT JOIN [dbo].[Logs] AS logs2 ON logs.id = logs2.id
WHERE
-- only conversations for last month
logs.timestamp >= DATEADD(month, -1, GETDATE())
GROUP BY logs.conversationId
When I try to add another column into SELECT, I get the error saying I need to add that column into the GROUP BY clause. But that causes the statement to run for an extremely long time, over 20 seconds for just a few dozen rows in the result.
use row_number() function
select *
from (
select *,
row_number() over(partition by conversationId order by id desc) as rn
from logs
) as t where t.rn=1
First get max log id per conversion from logs and then apply left join:
select * from
(SELECT
logs.conversationId,
MAX(logs.id) AS id
FROM [dbo].[Logs] AS logs group by logs.conversationId)a
left join [dbo].[Logs] AS logs2 ON a.id = logs2.id and a.conversationid=logs.conversationid
I would use a subquery in where to make it.
select *
from logs t
where t.id = (
SELECT MAX(tt.id)
from logs tt
WHERE tt.conversationId = t.conversationId
GROUP BY tt.conversationId
)
Note
if you make index in id might be faster than row_number version

How to select multiple rows in SQL Server while filling one column with the first value

Each of my rows have a date. I want the database to keep the good date. But I am in a situation where I want only the first date. But I still want all the other rows. So I would like to fill the date column with all the same date in my result.
For an example (Because I don't think I expressed myself well)
I have this:
name value date
a 10 5/13
b 14 2/13
c 20 1/13
a 11 7/13
a 5 8/13
b 8 9/13
I want it to become like this in the result:
name value date
a 26 5/13
b 22 5/13
c 20 5/13
I searched for this information but I only find the way to select the first row.
for now I'm doing
SELECT name, SUM(value), date FROM table
ORDER BY name
And I'm kind of clueless for what to do next.
Thanks :)
Databases don't have a concept of "first". Here is an attempt, but no guarantees unless you have a way of ordering to determine first:
select name, sum(value), const.date
from table cross join
(select top 1 date from table) const
group by name, const.date
If you only want to do this for a query, to provide this aggregated data for some specific client requirement, then #freshPrince's answer is appropriate. But if want to actually modify the data in the table itself, and prevent the issue from arising again, then you need to change the schema.
Create Table newTable(
name varChar(30) not null,
date datetime not null,
value decimal(10,2) not null default(0),
primary key (name, date) )
Insert newTable (name, date, value)
Select name, SUM(value), Min(date)
FROM currentTable
Group By Name
and delete the old table... then rename the new table to whatever...
You will also have to modify the process used to insert new rows so that instread of always inserting a new row, it updates the existing row for a specified name and date if it already exists...
Your question is slightly confusing since your desired result is showing a date that does not exists with either b or c but if that is the result that you want want you could use something similar to the following:
select name, sum(value) value, d.date
from yt
cross join
(
select min(date) date
from yt
where name = (select min(name)
from yt)
) d
group by name, d.date;
See SQL Fiddle with Demo
But it seems like you actually would want the min(date) for each name:
select name, sum(value) value, min(date)
from yt
group by name;
See SQL Fiddle with Demo.
If the order of the date should be the determined by the name then you could use:
select t.name, sum(value) value, d.date
from yt t
cross join
(
select top 1 name, date
from yt
order by name, date
) d
group by t.name, d.date;
See Demo