I'm working on data analysis using BigQuery. I have 12 CSV files for 12 months that need to be put together into one table.
I did that using INSERT INTO which worked well for some months but I got the error
Query column 6 has type STRING which cannot be inserted into column start_station_id, which has type INT64 at [2:1]
for some months.
The dataset is called "Dataset", the table I have put the data together is called Main, and the table I can't INSERT INTO Main is Dec2020
I have tried to change datatype in Dec2020 to int64 using the query below. It says the query has run successfully but when I try to INSERT INTO again, it gives me the same error: Query column 6 has type STRING which cannot be inserted into column start_station_id, which has type INT64 at [2:1]
Query:
WITH SELECT CAST (start_station_id AS int64) AS start_station_id
FROM `Dataset.Dec2020`,
Then
INSERT INTO `Dataset.Main`
SELECT *
FROM `Dataset.Dec2020`
Kindly help me out.
a couple of suggestions
Never use *, always explicitly list your columns in both the INSERT and SELECT statements so that you know which source column is being written to which target column. It also ensure that if additional columns are ever added to the source/target it won't break your code
Put the CAST in your SELECT statement - I'm not sure what you're doing with that WITH statement but as it is unconnected to the subsequent INSERT it is going to have no effect
Related
I have a table with detailed rows (for 1 id many rows). For this reason I have created a table with struct types in order to reduce the rows and make them 1 id 1 row.
How can I insert data into an impala table with struct types? Also, how can I aggregate after values from a struct type?
Instead of using struct to store multiple values, you can use group_concat(col, separator)
For example, if a customer has 3 account numbers and you want to store them in 1 row separated by comma, you can use below code -
select cust_id, name, group_concat(cust_acc,',') as concat_account
from cust_details
group by 1,2
You can use pipe if your data has comma in it.
Another benifit of above solution is, you can use split_part(concat_account,',',1) to get first account number.
Now, if your data is very complex and you cant use group concat, you can use struct. Its little tricky because you have to prepare data first in the struct format and then load them. Pls refer to below link - How to insert Array<Struct> values in Impala?
I have tables (of different schema) that consist of numerous rows (millions) with a unique id and at least 100-200 columns of various data types (INT64, String, Datetime, Float...etc). I need to unpivot the columns to rows dynamically and display pertaining values (including null values) in the next column. I need this only for data related to a selected id.
Here is an example of what I need.
An idea of how tables look and final result:
I wrote this code but I am getting the following error:
"Query error: The datatype of column does not match with other datatypes in the IN clause. Expected STRING, Found INT64 at [4:74]"
code I wrote:
declare myup string;
set myup=(
select concat('(',string_agg(column_name,','),')'),
from (select distinct column_name from `abc-def-
bigqueryghi.dataset_info.INFORMATION_SCHEMA.COLUMNS`
where table_name='table_1'
and column_name not in ("id")
)
);
execute immediate format("""
select*from `abc-def-bigquery-ghi.dataset_info.table_1`
unpivot
(values for column_name in %s)""",myup);
It is not possible to explicitly cast each column by name into string since some tables have up to 200 columns.
Null values also need to be displayed in final result since this needs to then be visualized on Google Data Studio.
Any ideas on how to solve this is highly appreciated.
I want to execute MERGE query in BigQuery tables which contain numeric column.When I used below query to execute I am getting error as Query error: Value of type FLOAT64 cannot be assigned to Numeric, which has type NUMERIC.
In my case, Its not possible to recognize datatype of number fields otherwise I can cast as per datatype of column.
Query =
MERGE Target T USING
(SELECT * FROM (SELECT 874 as ID,1012.0 as numericVal) S ON T.numericVal= S .numericVal WHEN MATCHED THEN UPDATE SET ID= S .ID,numericVal = S .numericVal
WHEN NOT MATCHED THEN INSERT ROW;
If you are not able to change the source, try casting as numeric and it'll be rounded accordingly
SELECT 874 as ID, CAST(1012.0 as NUMERIC) as numericVal
I have two a table and a view . The table if of two rows of datatypes nvarchar and money. I have being updating the table by selecting from the view like below.
Insert into MyTable
Select * from MyView
Recently, this update fails due to an error "String or binary data would be truncated." However, when i modified by select statement to something like.
Select * from Myview WHERE Column is not null
OR
Select * from Myview WHERE Column > 0
The above work with a warning saying Warning: Null value is eliminated by an aggregate or other SET operation. . It occurred to me that may may be one of the null value records contain something that's not null. My table column is of money type and accept null. I presumed the error may be due to something that's not of money data type. The record is huge. Is there any way i can filter and return those aliens records?
I also i learnt that i can eliminate the error by turning ANSI WARNING SETTION ON & OFF Here . My concern is wouldn't that result in loss of data. Please any help would be appreciated.
String or binary data would be truncated happened because the data coming from the MyView is larger than the column size in MyTable
Use
Select Max(Len(FieldName)) From MyTable
to check the maximum length of the nvarchar field in the MyTable
Or you can use Left when inserting data something Llike this
Insert into MyTable
Select Left(FieldName,50), Column1 from MyView
Note the 50 should be the size of the nvarchar field in MyTable
String or binary data would be truncated is a very common error. It usually happens when we try to insert any data in string (varchar,nvarchar,char,nchar) data type column which is more than size of the column. So you need to check the data size with respect to the column width and identify which column is creating problem and fix it.
Here is another thread of the same problem as yours in stackoverflow.
string or binary data would be truncated
Hope this will help.
Regards
looks like the data in some column in table MyView exceeds the limit of the corresponding one in table MyTable
I tried to search for previous articles related to this, but I can't find one specific to my situation. And because I'm brand new to StackOverflow, I can't post pictures so I'll try to describe it.
I have two datasets. One is 34 rows, 1 column of all NULLs. The other 13 rows, 1 column of varchars.
When I try to UNION ALL these two together, i get the following error:
Conversion failed when converting the varchar value to data type int.
I don't understand why I'm getting this error. I've UNIONed many NULL columns and varchar columns before, among many other types and I don't get this conversion error.
Can anyone offer suggestions why this error occurs?
The error occurs because you have corresponding columns in the two of the subqueries where the type of one is an integer and the type of the other is a character. Then, the character value has -- in at least one row -- a value that cannot be automatically converted to an integer.
This is easy to replicate:
select t.*
from (select 'A' as col union all
select 1
) t;
Here is the corresponding SQL Fiddle.
SQL Server uses pretty sophisticated type precedence rules for determining the destination type in a union. In practice, though, it is best to avoid using implicit type conversions. Instead, explicitly cast the columns to the type you intend.
EDIT:
The situation with NULL values is complicated. By itself, the NULL value has no type. So, the following works fine:
select NULL as col
union all
select 'A';
If you type the NULL, then the query will fail:
select cast(NULL as int) as col
union all
select 'A';
Also, if you put SQL Server in a position where it has to assign a type, then SQL Server will make the NULL an integer. Every column in a table or result set needs a type, so this will also fail:
select (select NULL) as col
union all
select 'A';
Perhaps your queries are doing something like this.
I have also encountered this error when I accidentally had the fields out of sequence in the 2 SELECT queries that I was unioning. Adjusting the fields' sequence fixed the problem.