I have tables (of different schema) that consist of numerous rows (millions) with a unique id and at least 100-200 columns of various data types (INT64, String, Datetime, Float...etc). I need to unpivot the columns to rows dynamically and display pertaining values (including null values) in the next column. I need this only for data related to a selected id.
Here is an example of what I need.
An idea of how tables look and final result:
I wrote this code but I am getting the following error:
"Query error: The datatype of column does not match with other datatypes in the IN clause. Expected STRING, Found INT64 at [4:74]"
code I wrote:
declare myup string;
set myup=(
select concat('(',string_agg(column_name,','),')'),
from (select distinct column_name from `abc-def-
bigqueryghi.dataset_info.INFORMATION_SCHEMA.COLUMNS`
where table_name='table_1'
and column_name not in ("id")
)
);
execute immediate format("""
select*from `abc-def-bigquery-ghi.dataset_info.table_1`
unpivot
(values for column_name in %s)""",myup);
It is not possible to explicitly cast each column by name into string since some tables have up to 200 columns.
Null values also need to be displayed in final result since this needs to then be visualized on Google Data Studio.
Any ideas on how to solve this is highly appreciated.
Related
I'm working on data analysis using BigQuery. I have 12 CSV files for 12 months that need to be put together into one table.
I did that using INSERT INTO which worked well for some months but I got the error
Query column 6 has type STRING which cannot be inserted into column start_station_id, which has type INT64 at [2:1]
for some months.
The dataset is called "Dataset", the table I have put the data together is called Main, and the table I can't INSERT INTO Main is Dec2020
I have tried to change datatype in Dec2020 to int64 using the query below. It says the query has run successfully but when I try to INSERT INTO again, it gives me the same error: Query column 6 has type STRING which cannot be inserted into column start_station_id, which has type INT64 at [2:1]
Query:
WITH SELECT CAST (start_station_id AS int64) AS start_station_id
FROM `Dataset.Dec2020`,
Then
INSERT INTO `Dataset.Main`
SELECT *
FROM `Dataset.Dec2020`
Kindly help me out.
a couple of suggestions
Never use *, always explicitly list your columns in both the INSERT and SELECT statements so that you know which source column is being written to which target column. It also ensure that if additional columns are ever added to the source/target it won't break your code
Put the CAST in your SELECT statement - I'm not sure what you're doing with that WITH statement but as it is unconnected to the subsequent INSERT it is going to have no effect
In Hue/Hive,
Describe mytablename;
gives the list of columns, their types and comments. Is there any way to query in Hive, treating result from describe as a table ?
For example I want to count the number of numeric/character/specific type columns, filter column names, total number of columns (currently requires scrolling down per 100 each, which is a hassle with 1000+ columns), etc
Queries such as
select count(*) from (Describe mytablename);
select count(*) from (select * from describe mytablename);
are of course invalid
Any ideas ?
You can create a sql file --> hive.sql containing "describe dbname.tablename"
hive -f hive.sql > /path/file.txt
create table dbname.desc
(
name String,
type String,
desc String
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
then, load data from path '/path/file.txt' into table dbname.desc.
So I have several VARCHAR(4000) columns that are filled to the MAX in my Oracle SQL Table. I am trying to concatenate/combine them into a single column (LONG or CLOB). How do I do this? I've tried concatentate and obviously that doesn't work... I receive 'a result of string concatenation is too long' error.
I've tried the following:
select id, (comment_1 || comment_2)
from table
group by id;
Thanks in advance!
You can use select id, to_clob(comment1) || comment2 from table group by id. But generally better way is modify the database design and add clob column into the table.
I have two a table and a view . The table if of two rows of datatypes nvarchar and money. I have being updating the table by selecting from the view like below.
Insert into MyTable
Select * from MyView
Recently, this update fails due to an error "String or binary data would be truncated." However, when i modified by select statement to something like.
Select * from Myview WHERE Column is not null
OR
Select * from Myview WHERE Column > 0
The above work with a warning saying Warning: Null value is eliminated by an aggregate or other SET operation. . It occurred to me that may may be one of the null value records contain something that's not null. My table column is of money type and accept null. I presumed the error may be due to something that's not of money data type. The record is huge. Is there any way i can filter and return those aliens records?
I also i learnt that i can eliminate the error by turning ANSI WARNING SETTION ON & OFF Here . My concern is wouldn't that result in loss of data. Please any help would be appreciated.
String or binary data would be truncated happened because the data coming from the MyView is larger than the column size in MyTable
Use
Select Max(Len(FieldName)) From MyTable
to check the maximum length of the nvarchar field in the MyTable
Or you can use Left when inserting data something Llike this
Insert into MyTable
Select Left(FieldName,50), Column1 from MyView
Note the 50 should be the size of the nvarchar field in MyTable
String or binary data would be truncated is a very common error. It usually happens when we try to insert any data in string (varchar,nvarchar,char,nchar) data type column which is more than size of the column. So you need to check the data size with respect to the column width and identify which column is creating problem and fix it.
Here is another thread of the same problem as yours in stackoverflow.
string or binary data would be truncated
Hope this will help.
Regards
looks like the data in some column in table MyView exceeds the limit of the corresponding one in table MyTable
I have table categories (c) and an another table (x) with a column which can contain cat IDs separated with comma as varchar data type. I want to Select related categories but I'm having error "Conversion failed when converting the varchar value '5,' to data type int." when trying to select:
SELECT ID, Title FROM c WHERE ID IN (SELECT catIDs FROM x WHERE ID=any);
The subquery returns data like "1,3,4"
You need to split the 1,3,4 string returned by the subquery into separate int values. SQL Server does not have a built-in function to do it, but you can use this user-defined function.
Create the function dbo.Split in your database and then re-write your query as follows:
SELECT ID, Title
FROM c
WHERE ID IN
(
SELECT s
FROM dbo.Split(',', '1,3,4')
)
I replaced the subquery with example results 1,3,4 to shorten the query and make it easier to understand.
If I get it right, you actually have values like "1,3,4" in your column catIDs. So you should extract a substring in the select of your subquery.
By the way, I'm not an MS SQL Server expert, but it's probably a bad database design to do so. I'm not sure the RDBMS engine will use indexes in such a case...