Storing extremely small values in Amazon Redshift - sql

Question

I am creating a table in Amazon Redshift using the following command:
CREATE TABLE asmt.incorrect_question_pairs_unique
AS
SELECT question1,
question2,
occurrences,
occurrences / (SUM(occurrences)::FLOAT) OVER () AS prob_q1_q2
FROM (SELECT question1,
question2,
SUM(occurrences) AS occurrences
FROM asmt.incorrect_question_pairs
GROUP BY question1,
question2
HAVING SUM(occurrences) >= 50)
I also tried an alternate:
CREATE TABLE asmt.incorrect_question_pairs_unique
AS
SELECT question1,
question2,
occurrences,
occurrences::float / SUM(occurrences) OVER () AS prob_q1_q2
FROM (SELECT question1,
question2,
SUM(occurrences) AS occurrences
FROM asmt.incorrect_question_pairs
GROUP BY question1,
question2
HAVING SUM(occurrences) >= 50)
I'd like the column prob_q1_q2 to be a float column, which is why I am converting the denominator/numerator to float. But in the resulting table, I get all zeros in that column.
I would like to point out that the SUM(occurrences) would amount to about 10 Billion, so the column prob_q1_q2 will contain extremely small values. Is there a way to store such small values in Amazon Redshift?
How do I make sure that all the values in the column are non-zero float?
Any help would be appreciated.

Answer 1

METHOD 1 - I have had the same problem! In my case it was million of rows so I Multiplied the result by 10000. whenever I wanted to select values from that column I would divide by 10000 in the select statement to make it even. I know its not the perfect solution but works for me.
METHOD 2 - I created a sample table with Numeric(12,6) datatype and when I imported the result set similar to yours, I can see the float values upto 6 decimal precision.
I guess, the conversion does not work when you use create table AS command, you need to create the table specifying the datatype which enforces the result set to be stored to a certain precision level. Its odd! how the same select returns 0.00 but when inserted into table with enforced column, it returns 0.00333.
If I’ve made a bad assumption please comment and I’ll refocus my answer.

Answer 2

Patthebug,
You might be getting a way too low number which cannot be stored in the FLOAT type of Amazon Redshift. Try using DECIMAL instead, there is no way it cannot store your value it's a 128 bit variable.
The way it works is the following, if the value if too big or in your case too small and it exceeds the max/min value of your type the last digits are trimmed and then the new (trimmed) value is stored in the variable/column of your type.
When it is trimming a big value you lose almost nothing lets say you are trimming 20 cents out of 20 billion dollars, you wont be hurt much. But in your case when the number is too small you can loose everything when it trims the last digits to fit in the type
(f.e. A type can store up to 5 digits and you want to store a value of 0.000009 in a variable/column of this type. Your value cannot be fit in the type so its trimmed from the last 2 digits so it can be fit and you receive a new value of 0.0000 )
So if you followed my thought just changing the ::float to ::decimal should fix your issue.
P.S. decimal might require specifying it's size f.e. decimal(127,100)

Answer 3

Try:
select cast(num1 as float) / cast(num2 as float);
This will give you results upto 2 decimal places (by default), but takes up some of your processing time. Doing anything else will round-off the decimal part.

Answer 4

You can have up to 38 digits in a DECIMAL/NUMERIC column with of 37 digits of scale.
CREATE TEMP TABLE precision_test (test NUMERIC(38,37)) DISTSTYLE ALL
;
INSERT INTO precision_test
SELECT CAST( 0.0000000000000000000000000000000000001 AS NUMERIC(38,37)) test
;
SELECT * FROM precision_test
;
--Returns 0.0000000000000000000000000000000000001