Knows someone where I can find a sample of adding primavera UDF through API?
My code is below but nothing happens when I run it.
I can connect to the database, read the UDF, delete the udf, but not to create
Thank you
package apitest;
import com.primavera.bo.base.u;
import com.primavera.common.value.ObjectId;
import com.primavera.integration.client.*;
import com.primavera.integration.client.Session;
import com.primavera.integration.client.EnterpriseLoadManager;
import com.primavera.integration.client.RMIURL;
import com.primavera.integration.common.DatabaseInstance;
import com.primavera.integration.client.bo.BOIterator;
import com.primavera.integration.client.bo.object.Project;
import com.primavera.integration.client.bo.object.UDFCode;
import com.primavera.integration.client.bo.InternalBOHelper;
import com.primavera.integration.client.bo.helper.UDFCodeHelper;
public class API
{
public static void main( String[] args )
{
System.setProperty("primavera.bootstrap.home","C:\\P6IntegrationAPI_1");
Session session = null;
try
{
DatabaseInstance[] dbInstances = Session.getDatabaseInstances(
RMIURL.getRmiUrl( RMIURL.LOCAL_SERVICE ) );
// Assume only one database instance for now, and hardcode the username and
// password for this sample code
session = Session.login( RMIURL.getRmiUrl( RMIURL.LOCAL_SERVICE ),
dbInstances[0].getDatabaseId(), "admin", "admin" );
//u.delete(session, (UDFCode)(new String ("High")));
UDFCode u = new UDFCode(session);
u.setCodeValue("cdc");
u.setDescription("cdcds");
u.setObjectId(ObjectId.USESSION_OVERRIDE_ID);
u.setCodeTypeObjectId(ObjectId.USESSION_OVERRIDE_ID);
u.setSequenceNumber(0);
u.create();
System.out.println("cdcx");
}
catch ( Exception e )
{
System.out.println(e.getCause());
}
finally
{
if ( session != null )
session.logout();
}
}
}
A UDF Field in P6 is in fact 3 Types of Tables/Classes.
UdfType - is the type of UDF (Name, Datatype etc...)
UdfValue - this is the actual value u want to set i guess ?
UdfCodes - Thats UserDefined Codes so its a bit complex and is in fact a Code and no Field!
So if u want a simple UDF, just create a UdfType, this is part of the globaldata and must exists befor u add values. You can also add it over the client/web and fill it with values later.
Next try to insert some UdfValues on Activities. Thats the way i go every time ;)
Use the following template code to connect to the DB . Make sure you change bootstrap,username password values before using it.
import java.util.Date;
import java.util.Iterator;
import com.primavera.PrimaveraException;
import com.primavera.ServerException;
import com.primavera.bo.events.enm.SpreadPeriodType;
import com.primavera.common.value.Duration;
import com.primavera.common.value.ObjectId;
import com.primavera.common.value.spread.ActivitySpread;
import com.primavera.common.value.spread.ActivitySpreadPeriod;
import com.primavera.integration.client.GlobalObjectManager;
import com.primavera.integration.client.Session;
import com.primavera.integration.client.bo.BOIterator;
import com.primavera.integration.client.bo.BusinessObjectException;
import com.primavera.integration.client.bo.enm.ActivityStatus;
import com.primavera.integration.client.bo.enm.ProjectStatus;
import com.primavera.integration.client.bo.object.Activity;
import com.primavera.integration.client.bo.object.ActivityNote;
import com.primavera.integration.client.bo.object.EPS;
import com.primavera.integration.client.bo.object.NotebookTopic;
import com.primavera.integration.client.bo.object.Project;
import com.primavera.integration.client.bo.object.BaselineProject;
import com.primavera.integration.common.CopyActivityOptions;
import com.primavera.integration.common.CopyProjectOptions;
import com.primavera.integration.common.CopyWBSOptions;
import com.primavera.integration.common.DatabaseInstance;
import com.primavera.integration.client.bo.object.*;
import com.primavera.integration.client.*;
import com.primavera.integration.network.NetworkException;
import com.primavera.integration.*;
public class EPSFilter {
static Session session = null;
//static Session session = null;
private static final String PRIMAVERA_BOOTSTRAP_HOME = "primavera.bootstrap.home";
static final String PRIMAVERA_BOOTSTRAP_HOME_VALUE = "C:\\P6EPPM_832\\p6";
static final String PRIMAVERA_USERNAME = "admin";
static final String PRIMAVERA_PASSWORD = "admin";
static final String PRIMAVERA_SYSTEM = "pmdb832_n1";
static void openSession() throws PrimaveraException {
for (DatabaseInstance dbi : Session.getDatabaseInstances(null)) {
if (PRIMAVERA_SYSTEM.equalsIgnoreCase(dbi.getDatabaseName())) {
session = Session.login(null, dbi.getDatabaseId(), PRIMAVERA_USERNAME, PRIMAVERA_PASSWORD);
//session = Session.login (RMIURL.getRmiUrl( RMIURL.LOCAL_SERVICE ), "1", "admin", "admin");
System.out.println("Connected to Primavera instance " + PRIMAVERA_SYSTEM);
return;
}
}
throw new RuntimeException("Error, Primavera instance " + PRIMAVERA_SYSTEM + " not found");
}
public static void main(String[] args) throws BusinessObjectException, ServerException, NetworkException {
try {
System.setProperty(PRIMAVERA_BOOTSTRAP_HOME, PRIMAVERA_BOOTSTRAP_HOME_VALUE);
openSession();
} catch (PrimaveraException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
Your code should follow to create the UDF's. If you still face issues. Please post the error you are getting
Related
When following Cubicoder's modding tutorial for Forge 1.12.2, and creating my first item, the texture for the item will not load. I have double checked all of my code against his code. I have my latest log here. I have my registration handler RegistrationHandler.java down below.
package notacyborg.tutorialmod;
import net.minecraft.creativetab.CreativeTabs;
import net.minecraft.item.Item;
import net.minecraftforge.event.RegistryEvent.Register;
import net.minecraftforge.fml.common.Mod.EventBusSubscriber;
import net.minecraftforge.fml.common.eventhandler.SubscribeEvent;
import notacyborg.tutorialmod.util.RegistryUtil;
#EventBusSubscriber(modid = TutorialMod.MODID)
public class RegistrationHandler
{
#SubscribeEvent
public static void registerItems(Register<Item> event)
{
final Item[] items = {
RegistryUtil.setItemName(new Item(), "first_item").setCreativeTab(CreativeTabs.MISC)
};
event.getRegistry().registerAll(items);
}
}
ModelRegistrationHandler.java
package notacyborg.tutorialmod.client;
import net.minecraft.client.renderer.block.model.ModelResourceLocation;
import net.minecraft.item.Item;
import net.minecraftforge.client.event.ModelRegistryEvent;
import net.minecraftforge.client.model.ModelLoader;
import net.minecraftforge.fml.common.Mod.EventBusSubscriber;
import net.minecraftforge.fml.common.eventhandler.SubscribeEvent;
import net.minecraftforge.fml.relauncher.Side;
import notacyborg.tutorialmod.TutorialMod;
import notacyborg.tutorialmod.init.ModItems;
#EventBusSubscriber(value = Side.CLIENT, modid = TutorialMod.MODID)
public class ModelRegistrationHandler
{
#SubscribeEvent
public static void registerModels(ModelRegistryEvent event)
{
registerModel(ModItems.FIRST_ITEM, 0);
}
private static void registerModel(Item item, int meta)
{
ModelLoader.setCustomModelResourceLocation(item, meta,
new ModelResourceLocation(item.getRegistryName(), "inventory"));
}
}
And my first_item.json model file.
{
"parent": "item/generated",
"textures": {
"layer0": "tutorialmod:textures/items/first_item"
}
}
Any help is appreciated!
Your error log says that it was not able to find the model file of your first_item. Make sure that you have put your first_item.json (model file) in assets/Your-Mod-ID/models/item/first_item.json
In your first_item.json file, line 4 should be:
"textures": {
"layer0": "tutorialmod:item/first_item"
}
Try it out and post an error log too if you encounter any further errors.
I tried out all the different method selectors as seen on this page: https://junit.org/junit5/docs/current/api/org/junit/platform/launcher/core/LauncherDiscoveryRequestBuilder.html
For example tried to do it like so:
selectMethod("org.example.order.OrderTests#test3"),
like so:
selectMethod("org.example.order.OrderTests#test3(TestInfo)"),
or like so: selectMethod("org.example.order.OrderTests#test3(org.junit.jupiter.engine.extension.TestInfoParameterResolver$DefaultTestInfo)")
Each time, no tests are found.
When I only select the class the method resides in, it works: selectClass("org.example.order.OrderTests")
(but I'm looking to call the method explicitly)
I am assuming the behavior is the same for other parameter types that are resolved at runtime by a ParameterResolver.
Your assumption is wrong. You can select one and only one test method.
As you mentioned on this page Discovery Selectors there are a lot of examples.
DiscoverySelectors.selectMethod provide three way to select desired method(s)
public static MethodSelector selectMethod(String className, String methodName, String methodParameterTypes) {
...
}
public static MethodSelector selectMethod(String className, String methodName) {
...
}
and
public static MethodSelector selectMethod(String fullyQualifiedMethodName) throws PreconditionViolationException {
...
}
You've tried to use the last method but the fullyQualifiedMethodName was wrong a little bit. If you take a look on javadoc it will turn up.
Parameter type list must exactly match and every non-primitive types must be fully qualified as well.
In your example the package is missing. Try it like: selectMethod("org.example.order.OrderTests#test3(org.junit.jupiter.api.TestInfo)")
Here is a short test.
package io.github.zforgo.stackoverflow;
import org.junit.jupiter.api.RepeatedTest;
import org.junit.jupiter.api.RepetitionInfo;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestInfo;
public class ClassWithTestInfo {
#Test
void foo() {
}
#Test
void foo(TestInfo info) {
}
#RepeatedTest(3)
void foo(RepetitionInfo info) {
}
}
package io.github.zforgo.stackoverflow;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.engine.descriptor.MethodBasedTestDescriptor;
import org.junit.platform.engine.DiscoverySelector;
import org.junit.platform.engine.FilterResult;
import org.junit.platform.engine.discovery.DiscoverySelectors;
import org.junit.platform.launcher.LauncherDiscoveryRequest;
import org.junit.platform.launcher.PostDiscoveryFilter;
import org.junit.platform.launcher.core.LauncherDiscoveryRequestBuilder;
import org.junit.platform.launcher.core.LauncherFactory;
import java.lang.reflect.Method;
import java.util.ArrayList;
import java.util.List;
class DiscoveryTest {
#Test
#DisplayName("Should select only the desired method(s)")
void doTEst() {
Assertions.assertAll(
() -> {
var methods = discover(DiscoverySelectors.selectClass(ClassWithTestInfo.class));
Assertions.assertEquals(3, methods.size());
},
() -> {
// your way
var fqmn = "io.github.zforgo.stackoverflow.ClassWithTestInfo#foo(TestInfo)";
var methods = discover(DiscoverySelectors.selectMethod(fqmn));
Assertions.assertEquals(0, methods.size());
},
() -> {
// good way
var fqmn = "io.github.zforgo.stackoverflow.ClassWithTestInfo#foo(org.junit.jupiter.api.TestInfo)";
var methods = discover(DiscoverySelectors.selectMethod(fqmn));
Assertions.assertEquals(1, methods.size());
}
);
}
private List<Method> discover(DiscoverySelector... selectors) {
final List<Method> methodCollector = new ArrayList<>();
LauncherDiscoveryRequest request = LauncherDiscoveryRequestBuilder.request()
.selectors(selectors)
.filters((PostDiscoveryFilter) object -> {
Method m = ((MethodBasedTestDescriptor) object).getTestMethod();
methodCollector.add(m);
return FilterResult.included("Matched");
})
.build();
LauncherFactory.create().discover(request);
return methodCollector;
}
}
I'm new to data flow and trying to get schema of table in big query dynamically.
Also i need to get the name of destination table dynamically for which i'm using dynamic destination class in BigQueryIO.write.to(). It works if the schema is provided for the destination table before executing the pipeline. But to get the schema dynamically i'm using BigQuery Snippets which takes datasetId and tableId as input and returns schema for a given table. It gives errors mentioned below when tried to run the pipeline with Snippets.
Any help is appreciated.
Thanks in advance.
Exception in thread "main" java.lang.NoSuchMethodError: com.google.api.client.googleapis.services.json.AbstractGoogleJsonClient$Builder.setBatchPath(Ljava/lang/String;)Lcom/google/api/client/googleapis/services/AbstractGoogleClient$Builder;
at com.google.api.services.bigquery.Bigquery$Builder.setBatchPath(Bigquery.java:3519)
at com.google.api.services.bigquery.Bigquery$Builder.<init>(Bigquery.java:3498)
at org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.newBigQueryClient(BigQueryServicesImpl.java:881)
at org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.access$200(BigQueryServicesImpl.java:79)
at org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.<init>(BigQueryServicesImpl.java:388)
at org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.<init>(BigQueryServicesImpl.java:345)
at org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.getDatasetService(BigQueryServicesImpl.java:105)
at org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$TypedRead.validate(BigQueryIO.java:676)
at org.apache.beam.sdk.Pipeline$ValidateVisitor.enterCompositeTransform(Pipeline.java:640)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:656)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:660)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:311)
at org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:245)
at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:458)
at org.apache.beam.sdk.Pipeline.validate(Pipeline.java:575)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:310)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:297)
at project2.configTable.main(configTable.java:146)
Code:
package project2;
import java.io.File;
import java.util.ArrayList;
import java.util.List;
import org.apache.avro.Schema;
import org.apache.beam.runners.dataflow.DataflowRunner;
import org.apache.beam.runners.dataflow.options.DataflowPipelineOptions;
import org.apache.beam.sdk.io.TextIO;
import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO;
import org.apache.beam.sdk.io.gcp.bigquery.DynamicDestinations;
import org.apache.beam.sdk.io.gcp.bigquery.TableDestination;
import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.Write.CreateDisposition;
import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.Write.WriteDisposition;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.options.ValueProvider.NestedValueProvider;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.transforms.View;
import org.apache.beam.sdk.transforms.DoFn.ProcessContext;
import org.apache.beam.sdk.transforms.DoFn.ProcessElement;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.ValueInSingleWindow;
import com.google.api.services.bigquery.model.Table;
import com.google.api.services.bigquery.model.TableFieldSchema;
import com.google.api.services.bigquery.model.TableRow;
import com.google.api.services.bigquery.model.TableSchema;
import com.google.cloud.bigquery.BigQuery;
import com.google.cloud.bigquery.BigQueryOptions;
import com.google.cloud.bigquery.Field;
import com.google.cloud.bigquery.FieldList;
import com.google.cloud.bigquery.BigQuery;
import com.google.cloud.bigquery.BigQueryOptions;
import com.google.cloud.bigquery.DatasetInfo;
import com.google.cloud.bigquery.Field;
import com.google.cloud.bigquery.FieldValueList;
import com.google.cloud.bigquery.InsertAllRequest;
import com.google.cloud.bigquery.InsertAllResponse;
import com.google.cloud.bigquery.LegacySQLTypeName;
import com.google.cloud.bigquery.QueryJobConfiguration;
import com.google.cloud.bigquery.StandardTableDefinition;
import com.google.cloud.bigquery.TableId;
import com.google.cloud.bigquery.TableInfo;
import java.util.HashMap;
import java.util.Map;
import avro.shaded.com.google.common.collect.ImmutableList;
public class configTable {
public static void main(String[] args) {
// TODO Auto-generated method stub
customInt op=PipelineOptionsFactory.as(customInt.class);
op.setProject("my-new-project");
op.setTempLocation("gs://train-10/projects");
op.setWorkerMachineType("n1-standard-1");
op.setTemplateLocation("gs://train-10/main-template-with-snippets");
op.setRunner(DataflowRunner.class);
org.apache.beam.sdk.Pipeline p=org.apache.beam.sdk.Pipeline.create(op);
PCollection<TableRow> indata=p.apply("Taking side input",BigQueryIO.readTableRows().from("my-new-project:training.config"));
PCollectionView<String> view=indata.apply("Convert to view",ParDo.of(new DoFn<TableRow, String>() {
#ProcessElement
public void processElement(ProcessContext c) {
TableRow row=c.element();
c.output(row.get("file").toString());
}
})).apply(View.asSingleton());
PCollection<TableRow> mainop = p.apply("Taking input",TextIO.read().from(NestedValueProvider.of(op.getInputFile(), new SerializableFunction<String, String>() {
public String apply(String input) {
// TODO Auto-generated method stub
return "gs://train-10/projects/"+input;
}
} ))).apply("Transform",ParDo.of(new DoFn<String, TableRow>() {
#ProcessElement
public void processElement(ProcessContext c ) {
c.output(new TableRow().set("data", c.element()));
}
}));
mainop.apply("Write data",BigQueryIO.writeTableRows().to(new DynamicDestinations<TableRow, String>() {
#Override
public String getDestination(ValueInSingleWindow<TableRow> element) {
// TODO Auto-generated method stub
String d=sideInput(view);
String tablespec="my-new-project:training."+d;
return tablespec;
}
#Override
public List<PCollectionView<?>> getSideInputs() {
return ImmutableList.of(view);
}
#Override
public TableDestination getTable(String destination) {
// TODO Auto-generated method stub
//String dest=String.format("%s:%s.%s","my-new-project","training", destination);
String dest=destination;
return new TableDestination(dest, dest);
}
#Override
public TableSchema getSchema(String destination) {
BigQuery bigquery = BigQueryOptions.getDefaultInstance().getService();
com.google.cloud.bigquery.Table table=bigquery.getTable("training", destination);
com.google.cloud.bigquery.Schema tbschema=table.getDefinition().getSchema();
FieldList tfld=tbschema.getFields();
List<TableFieldSchema> flds=new ArrayList<>();
for (Field each : tfld) {
flds.add(new TableFieldSchema().setName(each.getName()).setType(each.getType().toString()));
}
return new TableSchema().setFields(flds);
}
}).withCreateDisposition(CreateDisposition.CREATE_IF_NEEDED).withWriteDisposition(WriteDisposition.WRITE_TRUNCATE));
p.run();
}
}
I don't think you can do both WRITE_TRUNCATE
.withCreateDisposition(CreateDisposition.CREATE_IF_NEEDED).withWriteDisposition(WriteDisposition.WRITE_TRUNCATE))
and get the table's definition
com.google.cloud.bigquery.Table table=bigquery.getTable("training", destination);
com.google.cloud.bigquery.Schema tbschema=table.getDefinition().getSchema();
Because even if the table exists, it may be recreated when paired with a BigQueryIO.Write.WriteDisposition.WRITE_TRUNCATE and at that point, the getTable call will fail. In other words, WRITE_TRUNCATE is not an atomic operation.
I suggest that you have the table (with right schema) created before hand (CREATE_NEVER) or append to the table if it exists (WRITE_EMPTY or WRITE_APPEND) or store the schema outside of the dataflow pipeline and read it in.
I am trying to
1.Read JSON events from Cloud Pub/Sub
2.Load the events from Cloud Pub/Sub to BigQuery every 15 minutes using file loads to save cost on streaming inserts.
3.The destination will differ based on "user_id" and "campaign_id" field in the JSON event, "user_id" will be dataset name and "campaign_id" will be the table name. The partition name comes from the event timestamp.
4.The schema for all tables stays same.
I am new to Java and Beam. I think my code mostly does what I am trying to do and I just a need little help here.
But I unable to access "campaign_id" and "user_id" field in the JSON message.
So, my events are not routing to the correct table.
package ...;
import com.google.api.services.bigquery.model.TableSchema;
import javafx.scene.control.TableRow;
import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO;
import org.apache.beam.sdk.io.gcp.bigquery.DynamicDestinations;
import org.apache.beam.sdk.io.gcp.bigquery.TableDestination;
import org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder;
import org.apache.beam.sdk.io.gcp.pubsub.PubsubIO;
import org.apache.beam.sdk.transforms.MapElements;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.transforms.SimpleFunction;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.ValueInSingleWindow;
import org.joda.time.Duration;
import org.joda.time.Instant;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.nio.charset.StandardCharsets;
import java.text.SimpleDateFormat;
import static org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED;
import static org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.Write.Method.FILE_LOADS;
import static org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.Write.WriteDisposition.WRITE_APPEND;
public class ClickLogConsumer {
private static final int BATCH_INTERVAL_SECS = 15 * 60;
private static final String PROJECT = "pure-app";
public static PTransform<PCollection<String>, PCollection<com.google.api.services.bigquery.model.TableRow>> jsonToTableRow() {
return new JsonToTableRow();
}
private static class JsonToTableRow
extends PTransform<PCollection<String>, PCollection<com.google.api.services.bigquery.model.TableRow>> {
#Override
public PCollection<com.google.api.services.bigquery.model.TableRow> expand(PCollection<String> stringPCollection) {
return stringPCollection.apply("JsonToTableRow", MapElements.<String, com.google.api.services.bigquery.model.TableRow>via(
new SimpleFunction<String, com.google.api.services.bigquery.model.TableRow>() {
#Override
public com.google.api.services.bigquery.model.TableRow apply(String json) {
try {
InputStream inputStream = new ByteArrayInputStream(
json.getBytes(StandardCharsets.UTF_8.name()));
//OUTER is used here to prevent EOF exception
return TableRowJsonCoder.of().decode(inputStream, Coder.Context.OUTER);
} catch (IOException e) {
throw new RuntimeException("Unable to parse input", e);
}
}
}));
}
}
public static void main(String[] args) throws Exception {
Pipeline pipeline = Pipeline.create(options);
pipeline
.apply(PubsubIO.readStrings().withTimestampAttribute("timestamp").fromTopic("projects/pureapp-199410/topics/clicks"))
.apply(jsonToTableRow())
.apply("WriteToBQ",
BigQueryIO.writeTableRows()
.withMethod(FILE_LOADS)
.withWriteDisposition(WRITE_APPEND)
.withCreateDisposition(CREATE_IF_NEEDED)
.withTriggeringFrequency(Duration.standardSeconds(BATCH_INTERVAL_SECS))
.withoutValidation()
.to(new DynamicDestinations<TableRow, String>() {
#Override
public String getDestination(ValueInSingleWindow<TableRow> element) {
String tableName = "campaign_id"; // JSON message in Pub/Sub has "campaign_id" field, how do I access it here?
String datasetName = "user_id"; // JSON message in Pub/Sub has "user_id" field, how do I access it here?
Instant eventTimestamp = element.getTimestamp();
String partition = new SimpleDateFormat("yyyyMMdd").format(eventTimestamp);
return String.format("%s:%s.%s$%s", PROJECT, datasetName, tableName, partition);
}
#Override
public TableDestination getTable(String table) {
return new TableDestination(table, null);
}
#Override
public TableSchema getSchema(String destination) {
return getTableSchema();
}
}));
pipeline.run();
}
}
I arrived at the above code based on reading:
1.https://medium.com/myheritage-engineering/kafka-to-bigquery-load-a-guide-for-streaming-billions-of-daily-events-cbbf31f4b737
2.https://shinesolutions.com/2017/12/05/fun-with-serializable-functions-and-dynamic-destinations-in-cloud-dataflow/
3.https://beam.apache.org/documentation/sdks/javadoc/2.0.0/org/apache/beam/sdk/io/gcp/bigquery/DynamicDestinations.html
4.BigQueryIO - Write performance with streaming and FILE_LOADS
5.Inserting into BigQuery via load jobs (not streaming)
Update
import com.google.api.services.bigquery.model.TableFieldSchema;
import com.google.api.services.bigquery.model.TableRow;
import com.google.api.services.bigquery.model.TableSchema;
import com.google.api.services.bigquery.model.TimePartitioning;
import com.google.common.collect.ImmutableList;
import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO;
import org.apache.beam.sdk.io.gcp.bigquery.TableDestination;
import org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder;
import org.apache.beam.sdk.io.gcp.pubsub.PubsubIO;
import org.apache.beam.sdk.transforms.MapElements;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.transforms.SimpleFunction;
import org.apache.beam.sdk.values.PCollection;
import org.joda.time.Duration;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.nio.charset.StandardCharsets;
import static org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED;
import static org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.Write.Method.FILE_LOADS;
import static org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.Write.WriteDisposition.WRITE_APPEND;
public class ClickLogConsumer {
private static final int BATCH_INTERVAL_SECS = 15 * 60;
private static final String PROJECT = "pure-app";
public static PTransform<PCollection<String>, PCollection<TableRow>> jsonToTableRow() {
return new JsonToTableRow();
}
private static class JsonToTableRow
extends PTransform<PCollection<String>, PCollection<TableRow>> {
#Override
public PCollection<TableRow> expand(PCollection<String> stringPCollection) {
return stringPCollection.apply("JsonToTableRow", MapElements.<String, com.google.api.services.bigquery.model.TableRow>via(
new SimpleFunction<String, TableRow>() {
#Override
public TableRow apply(String json) {
try {
InputStream inputStream = new ByteArrayInputStream(
json.getBytes(StandardCharsets.UTF_8.name()));
//OUTER is used here to prevent EOF exception
return TableRowJsonCoder.of().decode(inputStream, Coder.Context.OUTER);
} catch (IOException e) {
throw new RuntimeException("Unable to parse input", e);
}
}
}));
}
}
public static void main(String[] args) throws Exception {
Pipeline pipeline = Pipeline.create(options);
pipeline
.apply(PubsubIO.readStrings().withTimestampAttribute("timestamp").fromTopic("projects/pureapp-199410/topics/clicks"))
.apply(jsonToTableRow())
.apply(BigQueryIO.write()
.withTriggeringFrequency(Duration.standardSeconds(BATCH_INTERVAL_SECS))
.withMethod(FILE_LOADS)
.withWriteDisposition(WRITE_APPEND)
.withCreateDisposition(CREATE_IF_NEEDED)
.withSchema(new TableSchema().setFields(
ImmutableList.of(
new TableFieldSchema().setName("timestamp").setType("TIMESTAMP"),
new TableFieldSchema().setName("exchange").setType("STRING"))))
.to((row) -> {
String datasetName = row.getValue().get("user_id").toString();
String tableName = row.getValue().get("campaign_id").toString();
return new TableDestination(String.format("%s:%s.%s", PROJECT, datasetName, tableName), "Some destination");
})
.withTimePartitioning(new TimePartitioning().setField("timestamp")));
pipeline.run();
}
}
How about: String tableName = element.getValue().get("campaign_id").toString() and likewise for the dataset.
Besides, for inserting into time-partitioned tables, I strongly recommend using BigQuery's Column-Based Partitioning, instead of using a partition decorator in the table name. Please see "Loading historical data into time-partitioned BigQuery tables" in the javadoc - you'll need a timestamp column. (note that the javadoc has a typo: "time" vs "timestamp")
I'm using HSQLDB version 2.2.9 for testing purposes.
When I create named in memory database, files aren't deleted after calling shutdown function. On my filesystem I have folder DBname.tmp and files DBname.lck, DBname.log, DBname.properties and DBname.script. As I understand documentation (http://hsqldb.org/doc/2.0/guide/dbproperties-chapt.html#dpc_connection_url) it shouldn't happened.
For testing I'm using the following code:
import java.io.IOException;
import org.hsqldb.Server;
import org.hsqldb.persist.HsqlProperties;
import org.hsqldb.server.ServerAcl.AclFormatException;
import org.junit.Test;
public class HSQLDBInMemTest {
#Test
public void test() throws IOException, AclFormatException {
HsqlProperties props = new HsqlProperties();
props.setProperty("server.database.0", "test1");
props.setProperty("server.dbname.0", "test1");
props.setProperty("server.database.1", "test2");
props.setProperty("server.dbname.1", "test2");
Server hsqlServer = new Server();
hsqlServer.setRestartOnShutdown(false);
hsqlServer.setNoSystemExit(true);
hsqlServer.setProperties(props);
hsqlServer.start();
hsqlServer.shutdown();
}
}
Answered here: http://sourceforge.net/mailarchive/message.php?msg_id=30881908 by fredt
The code should look like:
import java.io.IOException;
import org.hsqldb.Server;
import org.hsqldb.persist.HsqlProperties;
import org.hsqldb.server.ServerAcl.AclFormatException;
import org.junit.Test;
public class HSQLDBInMemTest {
#Test
public void test() throws IOException, AclFormatException {
HsqlProperties props = new HsqlProperties();
props.setProperty("server.database.0", "mem:test1");
props.setProperty("server.database.1", "mem:test2");
Server hsqlServer = new Server();
hsqlServer.setRestartOnShutdown(false);
hsqlServer.setNoSystemExit(true);
hsqlServer.setProperties(props);
hsqlServer.start();
hsqlServer.shutdown();
}
}
The path for a memory database looks like props.setProperty("server.database.0", "mem:test1");