Suppose you have a table in a database that tracks ticket appearances across 9 office locations for between 4 and 5 times periods every day. In each of these time periods, people might appear for one ticket (I guess we can use a transaction model), or they can appear for many (I've seen some with 15, heard of others with up to 50).
A query was created using Access 2013 to capture the data from this table. This is done for archiving (the current date table records are purged after 24 hours), and general analysis.
The current query logic is as follows:
SELECT Format([T1]![APP_DTE],"Short Date") AS APPEAR_DTE,
T1.OFFICE, OFFCDES.OFFICE_NAME, OFFCDES.LOC_CDE, OFFCDES.LOC_NAME,
T1.APP_TIME_PER, T1.ROOM_NUM, T1.INCL_CDE, T1.ID_NUMB,
T1.ADJ_APPEAR_NUM, T2.APPEAR_CDE,
IIf([T1]![APPEAR_LOGIN_TIME] Is Null,"",[T1]![APPEAR_LOGIN_TIME]) AS LOGIN_TIME1,
IIf([T1]![OFFICE_LOGIN] Is Null,"",[T1]![OFFICE_LOGIN]) AS OFFICE_LOGIN,
IIf([T1]![DECISION_TIME] Is Null,"",[T1]![DECISION_TIME]) AS DECISION_TIME,
[T1]![ID_NUMB] & "-" & CStr([T2]![OFFICE_REP_ID]) & "-" & CStr([T1]![APP_DTE]) AS CASEKEY,
Weekday([APP_DTE]) AS NUM_DAY,
[T1]![OFFICE] & "-" & Weekday([T1]![APP_DTE]) & "-" & [T1]![APP_TIME_PER] AS OFFICE_TP_KEY, "" AS RecordKey
FROM (T1 INNER JOIN OFFCDES ON T1.OFFICE = OFFCDES.OFFICE)
INNER JOIN T2 ON T1.APPEAR_NUM = T2.APPEAR_NUM
WHERE (((Format([T1]![APP_DTE],"Short Date"))=Format(Date(),"Short Date")));
I realize that the above code will need to be cleaned up for readability, and that it's probably not as efficient as it could be. That part I can't take credit for, I inherited all of this from a coworker and have just started to wrap my head around it.
The query result includes a ton of information, some (most) of which is irrelevant to what I'm trying to accomplish. It's important to capture all of this information since the information, once gone, is gone. With that stated:
What I'm trying to do:
I'm trying to incorporate a key/sequentially numbered column that would tell me how many times a particular ID_NUMB has occurred at a particular Office/Time Period on the date pulled. I'm also not looking for a total, rather something along the lines of the SEQ column here:
ID_NUMB | APPEAR_NUM | OFFICE | TP | DATE | SEQ
1 | 1XZ2 | 01 | 01 | 9/7/2016 | 1
1 | 1XZ3 | 01 | 01 | 9/7/2016 | 2
2 | 3MN1 | 03 | 01 | 9/7/2016 | 1
3 | J915 | 02 | 01 | 9/7/2016 | 1
1 | 1TY1 | 01 | 01 | 9/7/2016 | 1
3 | P91D | 05 | 01 | 9/7/2016 | 1
2 | U11E | 11 | 05 | 9/7/2016 | 1
While I have looked around and found many fine numbering solutions, most aren't quite what I'm looking to accomplish due to how fine grained the count seems to be.
I've thought about writing the data to a temp table, but using some of the information that I've found didn't lead to any kind of solution. Similarly,
I tried writing a separate query that could retrieve this information and add the sequential number in, kind of an adaption of the solution found on the MS Answers site. I put that aside as it seemed like the wrong approach.
I even toyed with writing the query to a temp table and then having a piece of VBA handle the sequential numbering, but that approach too, didn't get too far.
What would the right approach to generate a sequential number in a column as I outlined be? I'm at a loss.
I think this SQL might be close to what you want (based on the limited example) It does an INNER JOIN on the table T1 - and does a simple sort before it builds the SEQ on the fly
SELECT
a.ID_NUMB,
a.ADJ_APPEAR_NUM,
a.OFFICE,
a.APP_TIME_PER AS TP,
Format(a.[APP_DTE],"Short Date") AS APP_DATE,
(SELECT Count(ID_NUMB)
FROM T1 b
WHERE (b.ADJ_APPEAR_NUM <= a.ADJ_APPEAR_NUM
AND b.ID_NUMB = a.ID_NUMB
AND b.OFFICE = a.OFFICE
AND b.APP_TIME_PER = a.APP_TIME_PER
AND b.APP_DTE = a.APP_DTE)) AS SEQ
FROM T1 AS a
ORDER BY a.ID_NUMB, a.OFFICE, a.ADJ_APPEAR_NUM;
Query Results (using your data for T1)
For this kind of tasks, the most straight-forward method is usually to open a recordset using DAO with the records ordered as to the sequence you need.
Then loop through the recordset, for each record keeping track of the conditions that will force a new sequence number or just add one to the previous sequence number. Now, build the sequence number, update the record, and move on until no more records.
Edit: Example looping records:
Dim rst As DAO.Recordset
'Me.RecordsetClone
Set rst = Me.RecordsetClone
rst.Sort = "Dato, ID asc"
Set rst = rst.OpenRecordset()
rst.MoveFirst
While rst.EOF = False
Debug.Print rst!Dato.Value, rst!Id.Value
' If SomeCondition = True Then
rst.Edit
rst!Dec1.Value = rst!Id.Value
rst.Update
' End If
rst.MoveNext
Wend
Related
I have a table that has date information and is being copied to another table and trying to perform an incremental load.
date = date format
hour = int
person
date
hour
bob
2023-01-01
1
bill
2023-01-02
2
select * into test.person_copy from
(select * from original.person)
My thought process of performing the incremental load is to check on the max(date) & max(hour) from the original table against the copied table to identify what is the gap between the max values from both tables. However, I'm not entirely sure how to implement the logic as it doesn't seem straight forward with the where clause. Having clause might make more sense, but also doesn't seem correct?
select * into test.person_copy from
(select * from original.person org
Having max(org.date, org.hour) > (select max(copy.date,copy.hour) from test.person_copy copy)
)
The other variation I had in mind was to use HAVING NOT IN
Having max(org.date, org.hour) NOT IN (select max(copy.date,copy.hour) from test.person_copy copy)
Wasn't sure if logic is correct. Hour field will be of importance's, but can live with just the date fields.
Expected output would be that the logic would check for existing max(date) and only insert if it doesn't exist. Example below, 2023-01-03
| person | date | hour |
|--------|------------|------|
| bob | 2023-01-01 | 1 |
| bill | 2023-01-02 | 2 |
| test | 2023-01-03 | 2 |
Don't have access to a RedShift environment but the following query should work:
select *
into test.person_copy
from original.person org
where dateadd(hrs, org.hour, org.date) >
(select max(dateadd(hrs, cpy.hour, cpy.date))
from test.person_copy cpy
)
This assumes that when the previous hour's copy was made entire set of source rows for that date&hour was copied (the new incremental load would have all rows for the dates&hours not already copied). This means that you need additional criteria in the select to make sure that you include only completed date-hours (i.e. make sure that you don't include the rows with hour=10 while the time is still 10:30).
My table is stud.
+-----+------+-------+
| no | name | grade |
+-----+------+-------+
| 101 | naga | A |
| 102 | raj | A |
| 103 | john | A |
+-----+------+-------+
The query I'm using is:
SELECT * FROM stud WHERE no = 101 AND grade = 'A'.
If am using single record buffering, how much data is being stored in the buffer area?
This query doesn't do anything. There is no "into" clause. meaning it wont store anything selected.
You are probably looking to do something like this....
SELECT * FROM stud into wa_stud WHERE no = 101 AND grade = 'A'.
"processing of each single row is performed here
endselect.
or perhaps something like this, where only 1 row (the first rows ordered by primary key) is selected...
select single * from stud into wa_stud where no = 101 and grade = 'A' .
or perhaps you want everything brought in to a table, meaning number and grade does not include the full primary key.
select * from stud into table it_stud where no = 101 and grade = 'A'.
this is from ABAP Keyword documentation in SE38:
SAP Buffer - Single Record Buffering
Only those rows in the table are buffered that are actually accessed.
This requires less space in the buffer than when using generic or full
buffering. On the other hand, more administration work is required and
significantly more direct database accesses.
So since your query returns a single record (based on the data you displayed) it should just get one row and hold in the buffer.
I'd suggest looking at SAP help and Google - also have a look at SELECT SINGLE and incompletely specified keys - there used to be a problem with the buffer being bypassed in some situations - have a read for reference.
I'm trying to get the MOST recent date that comes before tom_temp.Begin_Time out of tbl_Trim_history.Comp. The SQL I'm using is:
SELECT
Tom_Temp2.feeder,
Tom_Temp.CauseType,
Tom_Temp.RootCause,
Tom_Temp.Storm_Name_Thunder,
Tom_Temp.DeviceGroup,
tbl_Trim_History.[COMP],
Tom_Temp.[Begin_Time]
FROM Tom_Temp2
LEFT JOIN (Tom_Temp
LEFT JOIN tbl_Trim_History
ON Tom_Temp.feeder = tbl_Trim_History.CIRCUIT_ID)
ON Tom_Temp2.feeder = Tom_Temp.feeder
WHERE (((tbl_Trim_History.[COMP]) < [Tom_Temp].[Begin_Time]));
I'm having a hard time figuring out where I need to put my max() function in this statement in order to make sure I don't get back every single tbl_Trim_history.[COMP] that occurs prior to the tom_temp.Begin_Time date. I only want the most recent date from tbl_Trim_history.[COMP] that occurs BEFORE the tom_temp.begin_Time .... NOT every historical date record.
Any help you guys could give me would be awesome because I keep getting back sets that I can tell are not what I'm looking for / expecting.
Thanks everyone. I appreciate the feedback.
Edit in regard to the responses below:
Due to the character limits, I just edited the master post for you guys.
I can't really post the data as it is somewhat confidential, so the best I can do is give you an example. Also, this is access, but my background is MySQL. Sorry for the tags, I wasn't sure what was similar since the access tag just didn't seem to fit the question.
The Data being received are about 168 records. Someone pointed out that there is an inner join occurring here, but I wanted to indicate I'm actually using 3 different tables.
1 table contains my feeders,
Another contains a list of all outages that I am joining to using all my feeders contained in the first table
Then I have another table that contains all the trim history for each feeder. The outage table is joined to the trim table.
When I run the query above, I get data like this
feeder | comp | Begin_time
___________________________________________
123456 | 10/4/2012 | 3/3/2016 11:26:00AM
123456 | 10/17/2015 | 3/3/2016 11:26:00AM
456789 | 6/28/2008 | 9/20/2013 10:05AM
456789 | 12/1/2012 | 9/20/2013 10:05AM
456789 | 7/3/2013 | 9/20/2013 10:05AM
what I want is data like this:
feeder | comp | Begin_time
___________________________________________
123456 | 10/17/2015 | 3/3/2016 11:26:00AM
456789 | 7/3/2013 | 9/20/2013 10:05AM
where the comp date is the closest to date / time occuring BEFORE Begin_time date.
I tried this query:
SELECT Tom_Temp2.feeder, Tom_Temp.CauseType, Tom_Temp.RootCause, Tom_Temp.Storm_Name_Thunder, Tom_Temp.DeviceGroup, Max(tbl_Trim_History.COMP) AS MaxOfCOMP, Tom_Temp.Begin_Time
FROM Tom_Temp2
LEFT JOIN (Tom_Temp LEFT JOIN tbl_Trim_History ON Tom_Temp.feeder = tbl_Trim_History.CIRCUIT_ID) ON Tom_Temp2.feeder = Tom_Temp.feeder
GROUP BY Tom_Temp2.feeder, Tom_Temp.CauseType, Tom_Temp.RootCause, Tom_Temp.Storm_Name_Thunder, Tom_Temp.DeviceGroup, Tom_Temp.Begin_Time
HAVING (((Max(tbl_Trim_History.COMP))<[Tom_Temp].[Begin_Time]));
But of the 168 records I get back in my first query, I'm only getting back 20 records with this query.
The reason I know this is wrong is because some records are missing between the set of 168 and the set of 20. For example, I'd be missing any records for feeder 456789. However, I know this record should be returned because it's in my table of feeders that should be returned (Tom_Temp2).
After manually deleting unwanted rows of data, I know that I should get a record count of 85. So my most recent attempt to use the Max query is way off.
Can data in Hive be transposed? As in, the rows become columns and columns are the rows? If there is no function straight up, is there a way to do it in a couple of steps?
I have a table like this:
| ID | Names | Proc1 | Proc2 | Proc3 |
| 1 | A1 | x | b | f |
| 2 | B1 | y | c | g |
| 3 | C1 | z | d | h |
| 4 | D1 | a | e | i |
I want it to be like this:
| A1 | B1 | C1 | D1 |
| x | y | z | a |
| b | c | d | e |
| f | g | h | i |
I have been looking up other related questions and they all mention using lateral views and explode, but is there a way to selectively choose columns for lateral(ly) view(ing) and explod(ing)?
Also, what might be the rough process to achieve what I would like to do? Please help me out. Thanks!
Edit: I have been reading this link: https://cwiki.apache.org/Hive/languagemanual-lateralview.html and it shows me half of what I want to achieve. The first example in the link is basically what I'd like except that I don't want the rows to repeat and want them as column names. Any ideas on how to get the data to a form such that if I do an explode, it would result in my desired output, or the other way, ie, explode first to lead to another step that would then lead to my desired output table. Thanks again!
I don't know of a way out of the box in hive to do this, sorry. You get close with explode etc. but I don't think it can get the job done.
Overall, conceptually, I think it's hard to a transpose without knowing what the columns of the destination table are going to be in advance. This is true, in particular for hive, because the metadata related to how many columns, their types, their names, etc. in a database - the metastore. And, it's true in general, because not knowing the columns beforehand, would require some sort of in-memory holding of data (ok, sure with spills) and users may need to be careful about not overflowing the memory and such (just like dynamic partitioning in hive).
In any case, long story short, if you know the columns of the destination table beforehand, life is good. There isn't a set command in hive per se, to the best of my knowledge, but you could use a bunch of if clauses and case statements (ugly I know, but that's how I have done the same in the past) in the select clause to transpose the data. Something along the lines of SQL - How to transpose?
Do let me know how it goes!
As Mark pointed out there's no easy way to do this in Hive since PIVOT doesn't present in Hive and you may also encounter issues when trying to use the case/when 'trick' since you have multiple values (proc1,proc2,proc3).
As for testing purposes, you may try a different approach:
select v, o1, o2, o3 from (
select k,
v,
LEAD(v,3) OVER() as o1,
LEAD(v,6) OVER() as o2,
LEAD(v,9) OVER() as o3
from (select transform(name,proc1,proc2,proc3) using 'python strm.py' AS (k, v)
from input_table) q1
) q2 where k = 'A1';
where strm.py:
import sys
for line in sys.stdin:
line = line.strip()
name, proc1, proc2, proc3 = line.split('\t')
print '%s\t%s' % (name, proc1)
print '%s\t%s' % (name, proc2)
print '%s\t%s' % (name, proc3)
The trick here is to use a python script in the map phase which emits each column of a row as distinct rows. Then every third (since we have 3 proc columns) row will form the resulting row which we get by peeking forward (lead).
However, this query does the job, it has the drawback that as the input grows, you need to peek the next 3rd element in the query which may lead to performance hit. Anyway you may evaluate it for testing purposes.
I have a query I need to perform to show search results for a project. What needs to happen, I need to sort the results by the "horsesActiveDate" and this applies to all of them except for any ad with the adtypesID=7. Those results are sorted by date but they must always result after all other ads.
So I will have all my ads in the result set be ordered by the Active Date AND adtypesID != 7. After that, I need all adtypesID=7 to be sorted by Active Date and appended at the bottom of all the results.
I'm hoping to put this in one query instead of two and appending them together in PHP. The way the code is written, I have to find a way to get it all in one query.
So here is my original query which has worked great until I had to ad the adtypesID=7 which has different sorting requirements.
This is the query that exists now that doesn't take into account the adtypesID for sorting.
SELECT
horses.horsesID,
horsesDescription,
horsesActiveDate,
adtypesID,
states.statesName,
horses_images.himagesPath
FROM horses
LEFT JOIN states ON horses.statesID = states.statesID
LEFT JOIN horses_images ON horses_images.himagesDefault = 1 AND horses_images.horsesID = horses.horsesID AND horses_images.himagesPath != ''
WHERE
horses.horsesStud = 0
AND horses.horsesSold = 0
AND horses.horsesID IN
(
SELECT DISTINCT horses.horsesID
FROM horses
LEFT JOIN horses_featured ON horses_featured.horsesID = horses.horsesID
WHERE horses.horsesActive = 1
)
ORDER BY adtypesID, horses.horsesActiveDate DESC
My first thought was to do two queries where one looked for all the ads that did not contain adtypesID=7 and sort those as the query does, then run a second query to find only those ads with adtypesID=7 and sort those by date. Then take those two results and append them to each other. Since I need to get this all into one query, I can't use a php function to do that.
Is there a way to merge the two query results one after the other in mysql? Is there a better way to run this query that will accomplish this sorting?
The Ideal Results would be as below (I modified the column names so they would be shorter):
ID | Description | ActiveDate | adtypesID | statesName | himagesPath
___________________________________________________________________________
3 | Ad Text | 06-01-2010 | 3 | OK | image.jpg
2 | Ad Text | 05-31-2010 | 2 | LA | image1.jpg
9 | Ad Text | 03-01-2010 | 4 | OK | image3.jpg
6 | Ad Text | 06-01-2010 | 7 | OK | image5.jpg
6 | Ad Text | 05-01-2010 | 7 | OK | image5.jpg
6 | Ad Text | 04-01-2010 | 7 | OK | image5.jpg
Any help that can be provided will be greatly appreciated!
I am not sure about the exact syntax in MySQL, but something like
ORDER BY case when adtypesID = 7 then 2 else 1 end ASC, horses.horsesActiveDate DESC
would work in many other SQL dielects.
Note that most SQL dialects allow the order by to not only be a column, but an expression.
This should work:
ORDER BY (adtypesID = 7) ASC, horses.horsesActiveDate DESC
Use a Union to append two queries together, like this:
SELECT whatever FROM wherever ORDER BY something AND adtypesID!=7
UNION
SELECT another FROM somewhere ORDER BY whocares AND adtypesID=7
http://dev.mysql.com/doc/refman/5.0/en/union.html
I re-wrote your query as:
SELECT h.horsesID,
h.horsesDescription,
h.horsesActiveDate,
adtypesID,
s.statesName,
hi.himagesPath
FROM HORSES h
LEFT JOIN STATES s ON s.stateid = h.statesID
LEFT JOIN HORSES_IMAGES hi ON hi.horsesID = h.horsesID
AND hi.himagesDefault = 1
AND hi.himagesPath != ''
LEFT JOIN HORSES_FEATURED hf ON hf.horsesID = h.horsesID
WHERE h.horsesStud = 0
AND h.horsesSold = 0
AND h.horsesActive = 1
ORDER BY (adtypesID = 7) ASC, h.horsesActiveDate DESC
The IN subquery, using a LEFT JOIN and such, will mean that any horse record whose horsesActive value is 1 will be returned - regardless if they have an associated HORSES_FEATURED record. I leave it to you for checking your data to decide if it should really be an INNER JOIN. Likewise for the STATES table relationship...