I am getting the BigQuery table name at runtime and I pass that name to the BigQueryIO.write operation at the end of my pipeline to write to that table.
The code that I've written for it is:
rows.apply("write to BigQuery", BigQueryIO
.writeTableRows()
.withSchema(schema)
.to("projectID:DatasetID."+tablename)
.withWriteDisposition(WriteDisposition.WRITE_TRUNCATE)
.withCreateDisposition(CreateDisposition.CREATE_IF_NEEDED));
With this syntax I always get an error,
Exception in thread "main" java.lang.IllegalArgumentException: Table reference is not in [project_id]:[dataset_id].[table_id] format
How to pass the table name with the correct format when I don't know before hand which table it should put the data in? Any suggestions?
Thank You
Very late to the party on this however.
I suspect the issue is you were passing in a string not a table reference.
If you created a table reference I suspect you'd have no issues with the above code.
com.google.api.services.bigquery.model.TableReference table = new TableReference()
.setProjectId(projectID)
.setDatasetId(DatasetID)
.setTableId(tablename);
rows.apply("write to BigQuery", BigQueryIO
.writeTableRows()
.withSchema(schema)
.to(table)
.withWriteDisposition(WriteDisposition.WRITE_TRUNCATE)
.withCreateDisposition(CreateDisposition.CREATE_IF_NEEDED));
Related
I want get the lookup object data for that I have one object called Agent and another object called pincode. added lookup data field in pincode.
In Query all connector i am applying this query: select firstName__c, lastname__c, experience__c, primarycontactnumber__c, role__c, active__c, pincodeId__c, CreatedBy.FirstName, pincode.Name FROM Agent__c where active__c= true
But here i am getting error SALESFORCE:INVALID_REQUEST_DATA for pincode.Name this field.
What is the correct query to get the pincode lookup data in the query?
Got the solution. By using select pincode__r I can get the lookup object data.
firstName__c, lastname__c, experience__c, primarycontactnumber__c, role__c, active__c, pincodeId__c, CreatedBy.FirstName, pincode__r.Name FROM Agent__c where active__c= true
I have tried using the JsonTimePartitioning class in apache beam JAVA sdk to write data to dynamic tables in bigquery but i get "cannot find symbol" for the class JsonTimePartitioning.
this is how i try to import the class
import com.google.api.services.bigquery.model.JsonTimePartitioning;
and this is how i try to use it in my pipeline
.withWriteDisposition(WriteDisposition.WRITE_APPEND)
.withJsonTimePartitioningTo(new JsonTimePartitioning().setType("DAY")));
I can't seem to find the JsonTimePartitioning anywhere. Can you point to an example that you are trying to follow? The existing methods on BigQueryIO either accept an instance of TimePartiotioning, or a value-provider for a String that is actually a JSON-serialized instance of the same TimePartitioning. And in fact, when calling the TimePartitioning version of the method, you still end up just serializing it into string internally:. You can find an example of how it's used here:
Loading historical data into time-partitioned BigQuery tables To load
historical data into a time-partitioned BigQuery table, specify
BigQueryIO.Write.withTimePartitioning(com.google.api.services.bigquery.model.TimePartitioning)
with a field used for column-based partitioning. For example:
PCollection<Quote> quotes = ...;
quotes.apply(BigQueryIO.write()
.withSchema(schema)
.withFormatFunction(quote -> new TableRow()
.set("timestamp", quote.getTimestamp())
.set(..other columns..))
.to("my-project:my_dataset.my_table")
.withTimePartitioning(new TimePartitioning().setField("time"))); ```
I'm seeing an issue when creating a spark streaming table using kafka from the snappy shell.
'The exception 'Invalid input 'C', expected dmlOperation, insert, withIdentifier, select or put (line 1, column 1):'
Reference: http://snappydatainc.github.io/snappydata/streamingWithSQL/#spark-streaming-overview
Here is my sql:
CREATE STREAM TABLE if not exists sensor_data_stream
(sensor_id string, metric string)
using kafka_stream
options (
storagelevel 'MEMORY_AND_DISK_SER_2',
rowConverter 'io.snappydata.app.streaming.KafkaStreamToRowsConverter',
zkQuorum 'localhost:2181',
groupId 'streamConsumer',
topics 'test:01');
The shell seems to not like the script at the first character 'C'. I'm attempting to execute the script using the following command:
snappy> run '/scripts/my_test_sensor_script.sql';
any help appreciated!
There is some inconsistency in documentation and actual syntax.The correct syntax is:
CREATE STREAM TABLE sensor_data_stream if not exists (sensor_id string,
metric string) using kafka_stream
options (storagelevel 'MEMORY_AND_DISK_SER_2',
rowConverter 'io.snappydata.app.streaming.KafkaStreamToRowsConverter',
zkQuorum 'localhost:2181',
groupId 'streamConsumer', topics 'test:01');
One more thing you need to do is to write row converter for your data
Mike, You need to create your own rowConverter class by implementing following trait -
trait StreamToRowsConverter extends Serializable {
def toRows(message: Any): Seq[Row]
}
and then specify that rowConverter fully qualified class name in the DDL.
The rowConverter is specific to a schema.
'io.snappydata.app.streaming.KafkaStreamToRowsConverter' is just an placeholder class name, which should be replaced by your own rowConverter class.
There were three similar questions in StackOverFlow but none gave an answer..
If have found why this error in occurring but don't know the fix.
I am using Strongly Typed Dataset for my project which is created as a dll for DAL.
I have added the Sql Server Table into this dataset using the designer and has created a DataAdapter
It works fine when i insert using DataTableAdapter
daLabTest.Insert(txtLabTestId.Text, cmbLabTestType.Text, cmbTestName.Text, txtLabFees.Text, dtpLabEffDate.Value)
but when i want to show the data from the table in a combobox or gridview i get this error.
i told that i found out what the problem is, I just previewed the data using DataSet designer and found out that the Function returns data like this...
The query i wrote to view this in dataset is
Select distinct(TestType) from LabTestTypes
so this should return only one column but the dataset is returning 5 columns but others as null, and the TestName column is a primary which should not be null when returned, so the problem exists..
To resolve this i tried to change the NullValue & AllowDBNull property to [Empty] and true respectively but that didn't worked for me.
Please help me in this...
That overly general constraint exception is nasty, where's the InnerException after so many complaints?!
This template may help identify the problem row and column but a "Fill" version of the query function is needed. E.g. GetDistinct*() --> Fill*(). Then a table can be created and interrogated for the row's error text.
SomeTable tTable = new SomeTable()
try {
// sorry, if you have a GetData, change to the fill version
someTable.FillByActiveLogin(tTable, loginName);
} catch (System.Data.ConstraintException constrExc) {
System.Data.DataRow[] rowsErr = tTable.GetErrors();
for (int i = 0; i < rowsErr.Count(); i++)
if (rowsErr[i].HasErrors)
Dbg.WriteLine(rowsErr[i].RowError);
}
(Thanks Michael S for this hint whoever/wherever you are!)
I got this error in a function from a DLL that uses a stored procedure. The procedure did not return all the fields in the table. One of the fields excluded was one that cannot be null. That apparently caused the constraint exception. When I changed the procedure and the DLL to include that field, the exception went away.
After spending ages on this problem myself, I have resolved it modifying the query in the dataset to return a dummy value (that can be ignored) for each key field that is not required in the output.
So your query would become...
Select distinct TestType, 1 as ID, "Dummy" as TestName, "Dummy" as TestFees, "Dummy" as TestDate
from LabTestTypes
I have set up the code as described in this question.
Creating an alias works, as well as dropping it.
For members that I have created myself, this is working correctly, but for existing members I get the following error when selecting from the alias:
SQL State: 42704
Vendor Code: -204
Message: [SQL0204] MyMemberName in MyLib type *FILE not found.
Cause . . . . . : MyMemberName in
TPLWHS type *FILE was not found. If the member name is *ALL, the table
is not partitioned. If this is an ALTER TABLE statement and the type
is *N, a constraint or partition was not found. If this is not an
ALTER TABLE statement and the type is *N, a function, procedure,
trigger or sequence object was not found. If a function was not found,
MyMemberName is the service program that contains the function. The
function will not be found unless the external name and usage name
match exactly. Examine the job log for a message that gives more
details on which function name is being searched for and the name that
did not match.
Recovery . . . : Change the name and try the request
again. If the object is a node group, ensure that the DB2 Multisystem
product is installed on your system and create a nodegroup with the
CRTNODGRP CL command. If an external function was not found, be sure
that the case of the EXTERNAL NAME on the CREATE FUNCTION statement
exactly matches the case of the name exported by the service program.
Any help you can offer is much appreciated. Thanks!
EDIT: Here is my code:
create alias MyLib.MyAlias for MyLib.MyLogicalFile(MyMember);
select * from MyLib.MyAlias;
drop alias MyLib.MyAlias;
The format of Lib.Alias has worked for me when I directly created the phyiscal and logical members. Perhaps the logical file is missing? I'll double check...
This error message can indicate that the file/logical file/member does not exist.