Invalid data error in Redshift - sql

Question

I have a query I am running in redshift that produces an error when I try to compare two dates. I have determined this is due to a data problem where the dates are VARCHAR and some are empty strings. The best solution is clearly to fix this at the source, but while trying to build a work around, I stumbled upon some very odd behavior.
To get around, I preselect the dates that are not empty strings, and cast as dates, then convert to integer date format (YYYYMMDD) and convert to INT. This runs fine. However, if I try to compare this with an integer in a WHERE clause, the query crashes with a data type error.
Here is a toy version of the working query
SELECT
date_id,
COUNT(*)
FROM
(
SELECT
CONVERT(int, date_id) AS date_id
FROM
(
SELECT
DATE_PART('year', start_dttm)*10000+DATE_PART('month', start_dttm)*10+DATE_PART('day', start_dttm) AS date_id
FROM
(
SELECT
CAST(start_dttm AS DATETIME) AS start_dttm
FROM
sfe.calendar_detail
WHERE
start_dttm <> ''
) cda
) cdb
) cd
GROUP BY
date_id
;
And here is the failed query
SELECT
date_id,
COUNT(*)
FROM
(
SELECT
CONVERT(int, date_id) AS date_id
FROM
(
SELECT
DATE_PART('year', start_dttm)*10000+DATE_PART('month', start_dttm)*10+DATE_PART('day', start_dttm) AS date_id
FROM
(
SELECT
CAST(start_dttm AS DATETIME) AS start_dttm
FROM
sfe.calendar_detail
WHERE
start_dttm <> ''
) cda
) cdb
) cd
WHERE
date_id >= 20170920
GROUP BY
date_id
;
As I mentioned above, the correct solution is to fix the data type and count empty dates as Nulls not empty strings, but I am very curious as to why the second query crashes on an invalid data type error.
Many Thanks!
Edit:
Here is the error
ERROR: Invalid digit, Value '1', Pos 0, Type: Integer
DETAIL:
-----------------------------------------------
error: Invalid digit, Value '1', Pos 0, Type: Integer
code: 1207
context:
query: 2006739
location: :0
process: query0_39 [pid=0]
-----------------------------------------------

Answer 1

Rather than converting dates to the human-readable YYYYMMDD format, it is always better to keep them as DATE or TIMESTAMP format. This way, date operations can be easily performed (eg adding 5 days to a date). You can still do easy comparison operators by using 'YYYYMMDD'::DATE.
Given that you are converting from a String, and casting to a Date seems to work, and that you have some empty strings, use this to convert it to a date:
SELECT
NULLIF(start_dttm, '')::DATE AS dt
FROM sfe.calendar_detail
WHERE dt > '20170920'::DATE
This will return a NULL if the string is empty, and a Date if it contains a date that could be converted.