I created a hive table which has numeric columns such as double and string columns.My file contains some NULL values for both numeric and string columns. When I try to load a file into this table, NULL values for the numeric columns is replaced by '\N' in file.I know this is hive property that handle null values for numeric type columns but i want to prevent it or Is there any way that i can change NULL into something else instead of '\N'.
By default NULL values are written in the data files as \N and \Nin the data files are being interpreted as NULL when querying the data.
This can be overridden by using TBLPROPERTIES('serialization.null.format'=...)
E.g.
TBLPROPERTIES('serialization.null.format'='') means the following:
An empty field in the data files will be treated as NULL when you query the table
When inserting rows to the table, NULL values will be written to the data files as empty fields
This property can be declared as part of the table creation
create table mytable (...)
tblproperties('serialization.null.format'='')
;
and can be changed later on
alter table mytable set tblproperties('serialization.null.format'='')
;
Related
I have an issue later in my process when I want the append tables with a different Datatypes.
I am creating a new table out of an existing table. One column is the Calenderweek(KW) which was originally a STRING. In order to append my tables later on I need the same datatype for the column.
Is there a way to change the datatype for a column while creating the new table?
CREATE TABLE IF NOT EXISTS
MyNewTable
AS(
SELECT
Column_1 AS
Column_1_alias,
**KW_ AS KW,**
FROM
SourceTable);
What this Query does is that it only grabs the value of the column KW that contains a number, then checks if the STRING value contains a character and removes it from the STRING. Finally it CAST to the desired value type of the column, so it ends as an INT.
CREATE TABLE IF NOT EXISTS
dataset.MyNewTable
AS(
SELECT
Column1 AS
Column1_alias,
CAST(REGEXP_REPLACE(KW,'[^0-9^]','') as INT64) as KW_Alias
FROM
`project.dataset.source`
WHERE REGEXP_CONTAINS(KW,'[0-9]')
);
Another possible solution is to use the function REPLACE instead of REGEXP_REPLACE, to replace the string to a number.
In the source file I have null values for a column X. But the datatype of the column X in the target table is INTEGER.I using fastload to load from the source file to the target table. Since the default value of integer is '0', its populating 0 in the target for the source null values. But I want to populate null in the target table for the null values in source file. Could you please help me with this?
I have a sql table which data type is int and it doesn’t accept null value.
So what I want to do is add “null” using derived column.
What I did is used drived transformation and add a new column and use expression (DT_WSTR,10) “null”
And then used data conversion and changed the data type into DT_14 but the data conversion fails upon execution.
Is there any other way to do this?
You can't do what you're trying to do. An INTEGER NOT NULL column will throw an error if you try to insert a text value into it, as you've seen.
There are really only two options.
Insert a zero for any NULL values that come through.
Insert a dummy value that's out of the range of values for the column, such as 999999 or the minimum or maximum values for an integer data type.
Or, of course, as Gordon suggested in the comments, drop the NOT NULL constraint on the column and insert the NULL values.
I've created a table schema and specified that for some attributes, values cannot be null. For one column of this table, values are to be imported from a column of some another table but the problem i am facing is that when i use insert statement to copy values from that column of another table to the column of this newly created table, the attributes of this new column start screaming because they kind of have a constraint on them that while insertion their values cannot be NULL!
How do i cope with this?
One solution is that for other attributes, just for time being, i can state that null values can be accommodated so that i can successfully import values from column of other table and then later on put condition on the remaining attributes that values are not be NULL. But how do i do do this?
You need to convert NULL to some DEFAULT values while importing.
I am not sure which DB engine you are using, in mysql:
Use something like IFNULL(column_name, "").
Reference
You may simply be looking for the default clause. When you define a column, you can specify;
intcol int not null default 0
If the column is not specified for an insert, then it will default to 0. In some databases, if a NULL value is supplied, it will also get the default value.
I am working in SQL Server 2008. I have a table with many columns that will not have values (at least, for the given situation). So, they will have a NULL value when I query each of them. I would like to instead make these NULL values be empty strings (i.e., ''). What is the best way to achieve this? My current idea is to set a DEFAULT value of '' on each them at the time that the table is created. However, since there are so many of them, this will be very tedious.
You have 2 options:
As you said, give it a default value of empty string for columns you don't want to be null when you create table/add new columns.
When you select nullable columns from the table you can use IsNull(ColumnName,'') which means if ColumnName is null it'll return empty string ('').
Create a table with the same structure as your current table, with a different name, and the default value as ''.
Insert into that table from your original table.
Delete the original table.
Change the name of the new table to the original table name.