PostgreSQL csv import not working for only integer - sql

I have the following problem using PostgreSQL 14
On Windows 10 with latest updates.
I need to insert values into the following table.
CREATE TABLE StateList (
ID int GENERATED ALWAYS AS IDENTITY,
State_Number int NOT NULL,
ElectionGroup_ID INT NOT NULL,
Election_Number int NOT NULL,
UNIQUE (State_Number, ElectionGroup_ID, Election_Number),
PRIMARY KEY (ID)
);
I want to do the following command:
COPY StateList(Election_Number, State_Number, ElectionGroup_ID )
FROM '...\csvFileStateLists19.csv'
WITH (
FORMAT CSV,
DELIMITER ','
);
the "csvFileStateLists19" being
"19","9","4"
"19","5","238"
"19","5","21"
"19","15","1"
"19","5","10"
It worked fine for another table that used strings and integer.
But here I always get:
ERROR: FEHLER: ungültige Eingabesyntax für Typ integer: »19«
CONTEXT: COPY statelist, Zeile 1, Spalte election_number: »19«
SQL state: 22P02
Which is usually the sign that the number is an empty string or really not a number. but its not! It's a 19, why doesn't it work?
I generated the file in java,
its utf8 encoded,
database is "German_Germany.1252"
show client_encoding; => UNICODE
show server_encoding; => UTF8
SELECT pg_encoding_to_char(encoding) FROM pg_database WHERE datname = 'database1'; => UTF8
select pg_encoding_to_char(encoding), datcollate, datctype from pg_database where datname = 'database1';
Returns
"UTF8" "German_Germany.1252" "German_Germany.1252"
Thank you for your help!

Well, with your input, I get the same error message - just in English, not German - I did it in Vertica, Stonebraker's successor of PosgreSQL, whose CSV parser works very much the same:
COPY statelist FROM LOCAL 'st.csv' DELIMITER ',' EXCEPTIONS 'st.log';
-- error messages in "st.log"
-- COPY: Input record 1 has been rejected (Invalid integer format '"19"' for column 1 (State_Number)).
-- COPY: Input record 2 has been rejected (Invalid integer format '"19"' for column 1 (State_Number)).
-- COPY: Input record 3 has been rejected (Invalid integer format '"19"' for column 1 (State_Number)).
-- COPY: Input record 4 has been rejected (Invalid integer format '"19"' for column 1 (State_Number)).
-- COPY: Input record 5 has been rejected (Invalid integer format '"19"' for column 1 (State_Number)).
Well, that's no wonder really. "9" is a string literal, not an INTEGER literal. It's a VARCHAR(1) consisting of the numeric letter "9", not an INTEGER.
Try adding the ENCLOSED BY '"' clause. It worked for me:
COPY statelist FROM LOCAL 'st.csv' DELIMITER ',' ENCLOSED BY '"' EXCEPTIONS 'st.log';
-- out Rows Loaded
-- out -------------
-- out 5
SELECT * FROM statelist;
-- out State_Number | ElectionGroup_ID | Election_Number
-- out --------------+------------------+-----------------
-- out 19 | 5 | 10
-- out 19 | 5 | 21
-- out 19 | 5 | 238
-- out 19 | 9 | 4
-- out 19 | 15 | 1

Not an answer just proof that double quoted numeric values in a CSV are not the problem:
cat csv_test.csv
"19","9"
"19","5"
"19","5"
"19","15"
"19","5"
test(5432)=# \d csv_test
Table "public.csv_test"
Column | Type | Collation | Nullable | Default
--------+---------+-----------+----------+---------
col1 | integer | | |
col2 | integer | | |
select * from csv_test;
col1 | col2
------+------
(0 rows)
\copy csv_test from 'csv_test.csv' with csv;
COPY 5
select * from csv_test;
col1 | col2
------+------
19 | 9
19 | 5
19 | 5
19 | 15
19 | 5
So now maybe we can get on with answers that solve the issue.

Related

How to trim the large number(130 digit) to 40 digit in oracle

I want to trim the number to 40 digit, but getting below error:
Query:
select 1123123211231231231231231231231231231123123123123123123123213213123213123213123123213123123123123213123123123126666666355555899 from dual;
Error:
ORA-01426: numeric overflow
01426. 00000 - "numeric overflow"
*Cause: Evaluation of an value expression causes an overflow/underflow.
*Action: Reduce the operands.
Error at Line: 14 Column: 8
Like this, perhaps? Enclose it into single quotes (so that it becomes a string) and apply SUBSTR to it:
SQL> select substr('1123123211231231231231231231231231231123123123123123123123213213123213123213123123213123123123123213123123123126666666355555899', 1, 40) result from dual;
RESULT
----------------------------------------
1123123211231231231231231231231231231123
SQL>
Use a string and TO_NUMBER( value DEFAULT NULL ON CONVERSION ERROR ):
SELECT TO_NUMBER( value DEFAULT NULL ON CONVERSION ERROR )
FROM long_numbers
Which, for the sample data:
CREATE TABLE long_numbers ( value ) AS
SELECT '1123123211231231231231231231231231231123123123123123123123213213123213123213123123213123123123123213123123123126666666355555899' FROM DUAL UNION ALL
SELECT '112312321123123123123123123123123123112312312312312312312321321312321312321312312321312312312312321312312312312666666635555589' FROM DUAL UNION ALL
SELECT '0.000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000012345678901234567890123456789012345678901234567890' FROM DUAL;
Outputs:
| TO_NUMBER(VALUEDEFAULTNULLONCONVERSIONERROR) |
| ----------------------------------------------------------------------------------------------------------------------------------------------------: |
| null |
| 112312321123123123123123123123123123112300000000000000000000000000000000000000000000000000000000000000000000000000000000000000 |
| .0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000123456789012345678901234567890123456789 |
Why doesn't the first value work?
From the Oracle data types documentation:
The following numbers can be stored in a NUMBER column:
Positive numbers in the range 1 x 10-130 to 9.99...9 x 10125 with up to 38 significant digits
Negative numbers from -1 x 10-130 to 9.99...99 x 10125 with up to 38 significant digits
Zero
So your first example will not work as it is outside the limits that the NUMBER data type can accept. There is nothing that can be done to display it as a NUMBER as it is too big.
What can I use instead of a NUMBER data type?
If you require the exact value then you will need to store it as a string.
If you want an approximate numeric value then you can store it as a BINARY_DOUBLE:
SELECT TO_BINARY_DOUBLE( value )
FROM long_numbers
Outputs:
| TO_BINARY_DOUBLE(VALUE) |
| :---------------------- |
| 1.1231232112312312E+126 |
| 1.1231232112312312E+125 |
| 1.2345678901234568E-110 |
db<>fiddle here

Concatenate columns and adds digits postgresql

Given the table table1 in PostgreSQL:
number1 | number2 | min_length | max_length
40 | 1801 | 8 | 8
40 | 182 | 8 | 8
42 | 32 | 6 | 8
42 | 4 | 6 | 6
43 | 691 | 9 | 9
I want to create new table table2 like:
start | stop
4018010000 | 4018019999
4018200000 | 4018299999
42320000 | 42329999
423200000 | 423299999
4232000000 | 4232999999
42400000 | 42499999
43691000000 | 43691999999
So the new table will be consisting of:
column_1 = a concatenation of old_column_1 + old_column_2 + a number of "0" equal to (old_column_3 - length of the old_column_2)
column_2 = a concatenation of old_column_1 + old_column_2 + a number of "9" equal to (old_column_3 - length of the old_column_2)
And when min_length is not equal to max_length, I need to take into account all the possible lengths. So for the line "42";"32";6;8 , all the lengths are: 6,7 and 8.
I tried to create the new table2 AS table1, then to create new columns start and stop, then to concatenate the columns 1 and 2 like that:
create table table2 as select * from table1;
alter table table2 add column start text,
add column stop text;
update table2 set start = number1 || number2
For the concatenation of the first 2 columns. But I can't figure out how to do the all concatenation, to add the "0"s and the "9"s.
Assuming all columns are NOT NULL, and max_length is always greater than min_length this does the job:
CREATE TABLE table2 AS
SELECT t.number1::text || rpad(t.number2::text, len, '0') AS start
, t.number1::text || rpad(t.number2::text, len, '9') AS stop
FROM table1 t, generate_series(min_length, max_length) len
db<>fiddle here
The manual for generate_series() and rpad().
If number1 or number2 can be NULL, throw in COALESCE:
SELECT COALESCE(t.number1::text, '') || rpad(COALESCE(t.number2::text,''), len, '0') AS start
, COALESCE(t.number1::text, '') || rpad(COALESCE(t.number2::text,''), len, '9') AS stop
FROM table1 t, generate_series(min_length, max_length) len;
db<>fiddle here
If min_length or max_length can be NULL, you'll have to define what's supposed to happen.

PostgreSQL and PHP. Fetching from query adds space to char string

I have a table with a few fields. One field is char[128]. Now i store there a string 'hello'.
Now. In PHP i call: arr = pg_fetch_array(pg_query('select * from table')) but when I get value from this column i get 'hello '. When I execute 'select char_length(this_field) from table' using pgAdmin then I get value 5 not 6. Do you know why there is an extra space in PHP there?
Using VARCHAR instead of CHAR solves this problem.
padding to the length is documented:
https://www.postgresql.org/docs/current/static/datatype-character.html
character(n), char(n) fixed-length, blank padded
example:
t=# with c(t) as (values('abc'::char(3)),('a'::char(3)))
select t,concat(t,'.') from c;
t | concat
-----+--------
abc | abc.
a | a .
(2 rows)
regarding length:
t=# with c(t) as (values('abc'::char(3)),('a'::char(3)))
select t,concat(t,'.'),octet_length(t),char_length(t) from c;
t | concat | octet_length | char_length
-----+--------+--------------+-------------
abc | abc. | 3 | 3
a | a . | 3 | 1
(2 rows)
using character varying or text indeed changes this behaviour.

How to substring and join with another table with the substring result

I have 2 tables: errorlookup and errors.
errorlookup has 2 columns: codes and description.
The codes are of length 2.
errors has 2 columns id and errorcodes.
The errorcodes are of length 40 meaning they code store 20 error codes for each id.
I need to display all the description associated with the id by substring the errorcodes and matching with code in errorlookup table.
Sample data for errorlookup:
codes:description
12:Invalid
22:Inactive
21:active
Sample data for errors:
id:errorcodes
1:1221
2:2112
3:1222
I cant use LIKE as it would result in too many errors. I want the errorcodes column to be broken down into strings of length 2 and then joined with the errorlookup.
How can it be done?
If you really cannot alter the tables structure, here's another approach:
Create an auxilary numbers table:
CREATE TABLE numbers
( i INT NOT NULL
, PRIMARY KEY (i)
) ;
INSERT INTO numbers VALUES
( 1 ) ;
INSERT INTO numbers VALUES
( 2 ) ;
--- ...
INSERT INTO numbers VALUES
( 100 ) ;
Then you could use this:
SELECT err.id
, err.errorcodes
, num.i
, look.codes
, look.descriptionid
FROM
( SELECT i, 2*i-1 AS pos --- odd numbers
FROM numbers
WHERE i <= 20 --- 20 pairs
) num
CROSS JOIN
errors err
JOIN
errorlookup look
ON look.codes = SUBSTR(err.errorcodes, pos, 2)
ORDER BY
err.errorcodes
, num.i ;
Test at: SQL-Fiddle
ID ERRORCODES I CODES DESCRIPTIONID
1 1221 1 12 Invalid
1 1221 2 21 Active
3 1222 1 12 Invalid
3 1222 2 22 Inactive
2 2112 1 21 Active
2 2112 2 12 Invalid
I think the cleanest solution is to "normalize" your errocodes table using a PL/SQL function. That way you can keep the current (broken) table design, but still access its content as if it was properly normlized.
create type error_code_type as object (id integer, code varchar(2))
/
create or replace type error_table as table of error_code_type
/
create or replace function unnest_errors
return error_table pipelined
is
codes_l integer;
i integer;
one_row error_code_type := error_code_type(null, null);
begin
for err_rec in (select id, errorcodes from errors) loop
codes_l := length(err_rec.errorcodes);
i := 1;
while i < codes_l loop
one_row.id := err_rec.id;
one_row.code := substr(err_rec.errorcodes, i, 2);
pipe row (one_row);
i := i + 2;
end loop;
end loop;
end;
/
Now with this function you can do something like this:
select er.id, er.code, el.description
from table(unnest_errors) er
join errorlookup el on el.codes = er.code;
You can also create a view based on the function to make the statements a bit easier to read:
create or replace view normalized_errorcodes
as
select *
from table(unnest_errors);
Then you can simply reference the view in the real statement.
(I tested this on 11.2 but I believe it should work on 10.x as well)
I think you're on the right track with LIKE. MySQL has an RLIKE function that allows matching by regular expression (I don't know if it's present in Oracle.) You could use errorlookup.code as a pattern to match against errors.errorcodes. The (..)* pattern is used to prevent things like "1213" from matching, for example, "21".
SELECT *
FROM error
JOIN errorlookup
WHERE errorcodes RLIKE CONCAT('^(..)*',code)
ORDER BY id;
+------+----------+------+
| id | errorcode| code |
+------+----------+------+
| 1 | 11 | 11 |
| 2 | 1121 | 11 |
| 2 | 1121 | 21 |
| 3 | 21313245 | 21 |
| 3 | 21313245 | 31 |
| 3 | 21313245 | 32 |
| 4 | 21 | 21 |
+------+----------+------+

Oracle - Integer datatype precision max? I'm able to enter more than 38 numbers into a integer field

I have an integer column, and according to others an integer is supposed to have a precision of 38 and is basically an alias for the type delcaration of Number(38)
I'm sure I'm missing something, but how am I able to enter 128 digits into an INTEGER column?
CREATE TABLE TEST
(
ID_INT INTEGER NOT NULL
);
insert into test( id_int)
values ( '0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890129010');
Version: Oracle 11
You can insert the row. But the data itself is then truncated. For example, note that the last 2 digits are lost when I query the data
SQL> insert into test( id_int )
2 values( 123456789012345678901234567890123456789012 );
1 row created.
SQL> select id_int from test;
ID_INT
------------------------------------------------------
123456789012345678901234567890123456789000
INTEGER is actually an alias for NUMBER (which can also be written as NUMBER(*)) and not NUMBER(38). NUMBER on its own means no precision and you can store any value. The 38 is a guarantee of 38 digits of precision to allow portability between different systems running Oracle though it will happily allow numbers that are a lot higher - just don't expect it to always port correctly if you ever have to. I created a test table:
create table TESTNUM
(
ID_INT integer
,ID_NUM38 number(38)
,ID_NUM number(*)
);
And here is a query to show the precisions stored:
select CNAME, COLTYPE, WIDTH, SCALE, PRECISION
from COL
where TNAME = 'TESTNUM';
I get back:
+----------+---------+-------+-------+-----------+
| CNAME | COLTYPE | WIDTH | SCALE | PRECISION |
+----------+---------+-------+-------+-----------+
| ID_INT | NUMBER | 22 | 0 | |
| ID_NUM38 | NUMBER | 22 | 0 | 38 |
| ID_NUM | NUMBER | 22 | | |
+----------+---------+-------+-------+-----------+
I believe precision refers to maximum number of significant digits
Not the same thing as only allowing 38 length number.
See here for explanation of significant digits.