Select All Columns Except the Ones I Transformed? - dbt

Apologies,
I am still a beginner at DBT. Is there a way to select all the columns that I didn't explicitly put in my select statement? Something like this:
{{ config(materialized='view') }}
with my_view as (
select
TO_DATE(SOME_DATE_FIELD, 'mm/dd/yyyy HH:mi:ss AM') AS SOME_DATE_FIELD,
{{ -- do something that gets the other columns from my_db.my_schema.my_table }}
from my_db.my_schema.my_table
),
final as (
select * from my_view
)
select * from final
Thanks!

If you use BigQuery or DuckDB, you can use * except (BQ) or * exclude (DuckDB). That would look like this:
...
with my_view as (
select
TO_DATE(SOME_DATE_FIELD, 'mm/dd/yyyy HH:mi:ss AM') AS SOME_DATE_FIELD,
* except (SOME_DATE_FIELD)
from my_db.my_schema.my_table
),
...
If you're on a different dialect, you can use the star macro in dbt_utils, which would look like this:
with my_view as (
select
TO_DATE(SOME_DATE_FIELD, 'mm/dd/yyyy HH:mi:ss AM') AS SOME_DATE_FIELD,
{{ dbt_utils.star(from=ref('my_model'), except=["SOME_DATE_FIELD"]) }}
from {{ ref('my_model') }}
),
...
Note:
to use dbt_utils.star() you first need to install dbt_utils as a package in your project. See the docs
in dbt you really shouldn't select from my_db.my_schema.my_table directly. That table should either be a model or a source, and you should use either the ref or source macro to select from it, so dbt can build a DAG:
from {{ ref("my_table") }}
or
from {{ source("my_source", "my_table") }}

Related

How to order a VARCHAR2 column representing date and time with AM and PM Oracle SQL?

Hello I have the following column and I want to order it by date from earliest to latest. the format looks like this.
08/16/2019 08:09:51 AM
I came up with this but its giving me an error.
ORDER BY date_format(STR_TO_DATE(`END_TIME`,'%m-%d-%y %h:%i:%s %p'),'%Y-%m-%d %H:%i:%s')
I couldn't strictly order it by date because 2 PM came before 8 AM which is wrong so I tried to put it in the correct format, I feel like im close. Any ideas?
Maybe I am wrong but i do not see nothing wrong here:
select to_char(to_date(Date_c,'mm/dd/yyyy hh:mi:ss AM'),'mm/dd/yyyy hh:mi:ss AM')
from myTable
order by to_date(Date_c,'mm/dd/yyyy hh:mi:ss AM');
That is whit this:
select *
from myTable
order by to_date(Date_c,'mm/dd/yyyy hh:mi:ss AM');
Here is the DEMO

Oracle SQL - Using TO_DATE to add a second to a time pulled from a string

This is building on the question I asked yesterday Link
I'm pulling time from the 'Flight' column whose data looks like this:
Dayton 01:23:59
Which gives me this:
01:23:59
I then want to add 1 second to this. I'm using the following TO_DATE function:
to_date(substr(Flight,length(Flight)-8,8), HH:MI:SS') + interval '1' second
This works but the format includes the date:
2018-07-01T01:24:00.000+00:00
I need it to look like this:
01:24:00
I've tried using the SUBSTR to extract only the time to no avail.
Any ideas on how I can add 1 second to the above and preserve the HH24:MI:SS format?
The direct way is using to_char string conversion function with HH24:MI:SS pattern as
with t(myTime) as
(
select to_date(substr('Dayton 01:23:59',length('Dayton 01:23:59')-8,9), 'HH:MI:SS')
+ interval '1' second
from dual
)
select to_char(myTime, 'hh24:mi:ss') as myTime_Chr from t;
MYDATE_CHR
01:24:00
or alternatively regexp_replace function maybe used as (provided that the city name doesn't contain a digit):
select regexp_replace('Dayton 01:24:00','[^0-9:]') as myTime
from dual;
MYTIME
01:24:00
P.S. dual maybe replaced by your real table name.
But seperating this column into two seperate columns as city and time is better.
SQL Fiddle Demo

Hadoop Data- Convert Sting timestamp into Hadoop date in SQL Assistant

I have a fields Name| ID | Timestamp
Timestamp is string like '06/29/2000 00:00:00'
Now I have to filter the table based on date- let say
Select Name
,ID
,Timestamp
From Table Where **Function**(Timestamp)= '2000-06-29' (or 2000/06/29 or 06/29/2000)
I am using SQL assistant as UI tool with Hadoop HI
I tried TO_DATE and couple of other functions.
Please advise
You can first change your timestamp format and apply the to_date function to trim the time from timestamp.
You can convert the timestamp format as below.
select from_unixtime(unix_timestamp('06/29/2000 00:00:00' ,'dd/MM/yyyy HH:mm:SS'), 'yyyy-MM-dd HH:mm:SS') from table;
Apply to_date function to the above sql.
Select Name
,ID
,Timestamp
From Table Where to_date(from_unixtime(unix_timestamp('06/29/2000 00:00:00' ,'dd/MM/yyyy HH:mm:SS'), 'yyyy-MM-dd HH:mm:SS'))= '2000-06-29'
I haven't tried the above solution, as i don't have environment with me right now. Let me know if you ran into any errors.
You say Timestamp is as string? Have you tried to compare strings?
Select Name
,ID
,Timestamp
From Table
where SUBSTR(Timestamp, 1, 10) = '06/29/2000'
This helped me and we can still play around:
Select Name
,ID
,Timestamp
From Table Where TO_DATE(from_unixtime(unix_timestamp(Timestamp,'MM/dd/yyyy
HH:m:ss'),'yyyy-MM-dd'))>= '2018-07-10'

Get String as Date in HSQL

select *
from employee
having to_date(date, 'DD/MM/YYYY HH24:MI:SS') = to_date('01/01/2012 10:00:00', 'DD/MM/YYYY HH24:MI:SS')
Column 'date' is a VARCHAR/STRING
as to_date doesnt work in HSQL, how can make the above query work in HSQL ??
TO_DATE does work with latest versions HSQLDB. You can use version 2.2.9 or future release versions.
But your query is wrong as indicated in comment by a_horse_with_no_name and needs WHERE instead of HAVING.
select *
from employee
where to_date(date, 'DD/MM/YYYY HH24:MI:SS') = to_date('01/01/2012 10:00:00', 'DD/MM/YYYY HH24:MI:SS')
In fact you can also simplify the query
select *
from employee
where to_date(date, 'DD/MM/YYYY HH24:MI:SS') = timestamp'2012-01-01 10:00:00'
I know it's an old thread, but it is the top google result so it is probably better to update it than start a new one - especially I have solved my problem. I had a massive problem with this query (hsqldb version 2.3.2) where my date is a string created in java and stored as a varchar in the database.
The to_date call doesn't work without a cast for me and the documentation was a bit vague on how to make the cast correctly (I thought the varchar cast needs to be done specifying the actual length while it doesn't seem to be the case.
select * from table2 where to_date(cast(timestamp as VARCHAR(254)), 'YYYY-MM-DD') > to_date('2015-09-02', 'YYYY-MM-DD')
notice the cast(xxx as VARCHAR(254)) there!

Oracle: not a valid month

I have a table with the following fields:
Reports (table name)
Rep_Date (date)
Rep_Time (date)
The Rep_Time field has values like '01/01/1753 07:30:00' i.e. the time part is relevant. I have written the following query:
select Reports.pid, MaxDate from Reports
INNER JOIN (
select pid, max(TO_DATE(TO_CHAR(REP_DATE, 'DD/MM/YYYY')
|| TO_CHAR(REP_TIME, 'HH24:MI:SS'), 'DD/MM/YYYY HH24:MI:SS')) As MaxDate
from reports
group by pid
) ReportMaxDate
on Reports.PID = ReportMaxDate.PID
AND To_Date(To_Char(MaxDate, 'DD/MM/YYYY')) = REP_DATE
WHERE REPORTS.PID=61
The derived table part of the query runs, but when I run the entire query I get an error: "not a valid month". Why is this?
In order to help debug this; if I run the following query:
select rep_date, rep_time from reports where pid=61 and rownum=1
I get:
Rep_Date = 01/04/2009
Rep_Time = 01/01/1753 13:00:00
UPDATE 15:58
I am now able to execute the following query:
select Reports.pid, MaxDate from Reports
INNER JOIN (
select pid, max(TO_DATE(TO_CHAR(REP_DATE, 'DD/MM/YYYY')
|| TO_CHAR(REP_TIME, 'HH24:MI:SS'), 'DD/MM/YYYY HH24:MI:SS')) As MaxDate
from reports group by pid
) ReportMaxDate
on Reports.PID = ReportMaxDate.PID
AND to_date(to_char(maxdate,'MM/DD/YYYY'),'MM/DD/YYYY') = REP_DATE
WHERE REPORTS.PID=61
However, I need to add one more statement to the WHERE clause comparing the time part of MaxDate to rep_time: to_date(to_char(maxdate,'MM/DD/YYYY'),'MM/DD/YYYY') = REP_DATE does not work.
1.
To_Date(To_Char(MaxDate, 'DD/MM/YYYY')) = REP_DATE
is causing the issue. when you use to_date without the time format, oracle will use the current sessions NLS format to convert, which in your case might not be "DD/MM/YYYY". Check this...
SQL> select sysdate from dual;
SYSDATE
---------
26-SEP-12
Which means my session's setting is DD-Mon-YY
SQL> select to_char(sysdate,'MM/DD/YYYY') from dual;
TO_CHAR(SY
----------
09/26/2012
SQL> select to_date(to_char(sysdate,'MM/DD/YYYY')) from dual;
select to_date(to_char(sysdate,'MM/DD/YYYY')) from dual
*
ERROR at line 1:
ORA-01843: not a valid month
SQL> select to_date(to_char(sysdate,'MM/DD/YYYY'),'MM/DD/YYYY') from dual;
TO_DATE(T
---------
26-SEP-12
2.
More importantly, Why are you converting to char and then to date, instead of directly comparing
MaxDate = REP_DATE
If you want to ignore the time component in MaxDate before comparision, you should use..
trunc(MaxDate ) = rep_date
instead.
==Update : based on updated question.
Rep_Date = 01/04/2009 Rep_Time = 01/01/1753 13:00:00
I think the problem is more complex. if rep_time is intended to be only time, then you cannot store it in the database as a date. It would have to be a string or date to time interval or numerically as seconds (thanks to Alex, see this) . If possible, I would suggest using one column rep_date that has both the date and time and compare it to the max date column directly.
If it is a running system and you have no control over repdate, you could try this.
trunc(rep_date) = trunc(maxdate) and
to_char(rep_date,'HH24:MI:SS') = to_char(maxdate,'HH24:MI:SS')
Either way, the time is being stored incorrectly (as you can tell from the year 1753) and there could be other issues going forward.
To know the actual date format, insert a record by using sysdate. That way you can find the actual date format. for example
insert into emp values(7936, 'Mac', 'clerk', 7782, sysdate, 1300, 300, 10);
now, select the inserted record.
select ename, hiredate from emp where ename='Mac';
the result is
ENAME HIREDATE
Mac 06-JAN-13
voila, now your actual date format is found.
You can also change the value of this database parameter for your session by using the ALTER SESSION command and use it as you wanted
ALTER SESSION SET NLS_DATE_FORMAT = 'DD-MM-YYYY';
SELECT TO_DATE('05-12-2015') FROM dual;
05/12/2015
You can customize as you wish the format inside the ''
SELECT TO_CHAR(column_name, 'MONTH-DAY-yy') FROM table_name;