Is it possible to add data from a new BigQuery table to existing Tableau extract?
For example, there are BigQuery tables partitioned by date like access_20160101, access_20160102, ... and data from 2016/01/01 to 2016/01/24 is already in Tableau server extract. Now a new table for 2016/01/25, access_20160125 has been created and I want to add the data to the existing extract, but don't want to read the old tables because there is no change in them but loading them will be charged by Google.
If I understand correctly: you created an extract for a table in BigQuery and now you want to append data in a different table to that extract.
As long as both tables have exactly the same column names and data types in those columns you can do this:
Create an extract from the new table.
Append that extract to the old one. (see: add data from a file)
Now you have one extract with the data of both tables.
Related
I am trying to clean and reconcile some datasets.
I have my original tables in raw format (all strings) and I am copying the information from the source, to another table I have created with the right data types.
However, I would like the new data in the new table to reflect the original source table in the form of (i.e.) 1_blablabla. So that, in the new table, I would have:
1_blablabla
2_blablabla
3_blablabla
and so on...
So far I am creating the new table as follows:
CREATE TABLE *table_name*(
*column1_name* DATATYPE(XX),
*column2_name* DATATYPE(XX),
*column3_name* DATATYPE(XX),
*column4_name* DATATYPE(XX)
);
And then inserting the data from the old table with the raw data, to the new table with the "correct" format, like this:
INSERT INTO *new_table* (*column1_name*, *column2_name*, *column3_name*, *column4_name*)
SELECT (*sourcecolumn1*, *sourcecolumn2*, *sourcecolumn3*, *sourcecolumn4*)
FROM sourcetable1;
However, this loads the data as like for like, exactly as it is in the source table,
and instead, I would like it to reflect, at least in the rows of one of the columns, the prefix "1_" for example, so that it is intuitive to see where it came from.
Using bigQuery UI I've created new table free_schem_table and haven't set any schema, then I tried to execute:
insert into my_dataset.free_schema_table (chatSessionId, chatRequestId,senderType,senderFriendlyName)
values ("123", "1234", "CUSTOMER", "Player")
But BigQuery UI demonsrtrated me the popup where written:
Column chatSessionId is not present in table my_dataset.free_schema_table at [1:43]
I expected that BiqQuery is a NoSql storage and I should be able to insert rows with different columns.
How could I achieve it ?
P.S.
schema:
BigQuery requires a schema with strong type.
If you need free schema, similar thing in BigQuery is to define a single column in STRING type and store JSON inside.
JSON functions will help you extract field from JSON string later, but you don't benefit from BigQuery's optimization if you predefine your schema and save data in different columns.
I have a variant table where raw json data is stored in a column called "raw" as shown here.
Each row of this table is a full data pull from an API and ingested via snowpipe. Within the json there is a 'pxQueryTimestamp' key and value pair. The latest value for this field should have the most up to date data. How would I go about only normalizing this row?
Usually my way around this, is to only pipe over the latest data from "s3" so that this table has only one row, then I normalize that.
I'd like to have a historic table of all data pulls as show below but when normalizing we only care about the most relevant up to date data.
Any help is appreciated!
If you are saying that you want to flatten and retain everything in the most current variant record, then I'd suggest leveraging a STREAM object in Snowflake, which would then only have the latest variant record. You could then TRUNCATE your flattened table and run an insert from the STREAM object to your flattened table, which would then move the offset forward and your STREAM would then be empty.
Take a look at the documentation here:
https://docs.snowflake.net/manuals/user-guide/streams.html
I am creating table using google drive as a source and google sheet as a format.
I have selected "Drive" as a value for create table from. For file Format, I selected Google Sheet.
Also I selected the Auto Detect Schema and input parameters.
Its creating the table but the first row of the sheet is also loaded as a data instead of table fields.
Kindly tell me what I need to do to get the first row of the sheet as a table column name not as a data.
It would have been helpful if you could include a screenshot of the top few rows of the file you're trying to upload at least to see the data types you have in there. BigQuery, at least as of when this response was composed, cannot differentiate between column names and data rows if both have similar datatypes while schema auto detection is used. For instance, if your data looks like this:
headerA, headerB
row1a, row1b
row2a, row2b
row3a, row3b
BigQuery would not be able to detect the column names (at least automatically using the UI options alone) since all the headers and row data are Strings. The "Header rows to skip" option would not help with this.
Schema auto detection should be able to detect and differentiate column names from data rows when you have different data types for different columns though.
You have an option to skip header row in Advanced options. Simply put 1 as the number of rows to skip (your first row is where your header is). It will skip the first row and use it as the values for your header.
I wanted to get rid off one data source in Tableau, that's why instead of using 2 different data source for one dashboard, I wanted to copy all relevant fields from one data source to other. Is there any way in Tableau, by which I can copy-paste tos field from one to other data source?
In the attached screenshot, I wanted to copy the advisor sales field in data source biadvisorSalesmonth24 to bitransactionPartnerDay365:
You cannot make schema or structure changes to a table / datasource from within Tableau. If advisor sales is not in the bitransactionPartnerDay365 data source, then you will have to keep both data sources in the workbook and join them together.
Now, if you are familiar with the datasets and know the necessary table layout, you could write a custom SQL command and use that SQL command to retrieve the desired data as a single data source.