Google dataflow write to mutiple tables based on input - google-bigquery

I have logs which I am trying to push to Google BigQuery. I am trying to build the entire pipeline using google dataflow. The log structure is different and can be classified into four different type. In my pipeline I read logs from PubSub parse it and write to BigQuery table. The table to which the logs need to written is depending on one parameter in logs. The problem is I am stuck on a point where how to change TableName for BigQueryIO.Write at runtime.

You can use side outputs.
https://cloud.google.com/dataflow/model/par-do#emitting-to-side-outputs-in-your-dofn
The following sample code, reads a BigQuery table and splits it in 3 different PCollections. Each PCollections ends up sent to a different Pub/Sub topic (which could be different BigQuery tables instead).
Pipeline p = Pipeline.create(PipelineOptionsFactory.fromArgs(args).withValidation().create());
PCollection<TableRow> weatherData = p.apply(
BigQueryIO.Read.named("ReadWeatherStations").from("clouddataflow-readonly:samples.weather_stations"));
final TupleTag<String> readings2010 = new TupleTag<String>() {
};
final TupleTag<String> readings2000plus = new TupleTag<String>() {
};
final TupleTag<String> readingsOld = new TupleTag<String>() {
};
PCollectionTuple collectionTuple = weatherData.apply(ParDo.named("tablerow2string")
.withOutputTags(readings2010, TupleTagList.of(readings2000plus).and(readingsOld))
.of(new DoFn<TableRow, String>() {
#Override
public void processElement(DoFn<TableRow, String>.ProcessContext c) throws Exception {
if (c.element().getF().get(2).getV().equals("2010")) {
c.output(c.element().toString());
} else if (Integer.parseInt(c.element().getF().get(2).getV().toString()) > 2000) {
c.sideOutput(readings2000plus, c.element().toString());
} else {
c.sideOutput(readingsOld, c.element().toString());
}
}
}));
collectionTuple.get(readings2010)
.apply(PubsubIO.Write.named("WriteToPubsub1").topic("projects/fh-dataflow/topics/bq2pubsub-topic1"));
collectionTuple.get(readings2000plus)
.apply(PubsubIO.Write.named("WriteToPubsub2").topic("projects/fh-dataflow/topics/bq2pubsub-topic2"));
collectionTuple.get(readingsOld)
.apply(PubsubIO.Write.named("WriteToPubsub3").topic("projects/fh-dataflow/topics/bq2pubsub-topic3"));
p.run();

Related

BigQuery: Split table based on a column

Short question: I would like to split a BQ table into multiple small tables, based on the distinct values of a column. So, if column country has 10 distinct values, it should split the table into 10 individual tables, with each having respective country data. Best, if done from within a BQ query (using INSERT, MERGE, etc.).
What I am doing right now is importing data to gstorage -> local storage -> doing splits locally and then pushing into tables (which is kind of a very time consuming process).
Thanks.
If the data has the same schema, just leave it in one table and use the clustering feature: https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#creating_a_clustered_table
#standardSQL
CREATE TABLE mydataset.myclusteredtable
PARTITION BY dateCol
CLUSTER BY country
OPTIONS (
description="a table clustered by country"
) AS (
SELECT ....
)
https://cloud.google.com/bigquery/docs/clustered-tables
The feature is in beta though.
You can use Dataflow for this. This answer gives an example of a pipeline that queries a BigQuery table, splits the rows based on a column and then outputs them to different PubSub topics (which could be different BigQuery tables instead):
Pipeline p = Pipeline.create(PipelineOptionsFactory.fromArgs(args).withValidation().create());
PCollection<TableRow> weatherData = p.apply(
BigQueryIO.Read.named("ReadWeatherStations").from("clouddataflow-readonly:samples.weather_stations"));
final TupleTag<String> readings2010 = new TupleTag<String>() {
};
final TupleTag<String> readings2000plus = new TupleTag<String>() {
};
final TupleTag<String> readingsOld = new TupleTag<String>() {
};
PCollectionTuple collectionTuple = weatherData.apply(ParDo.named("tablerow2string")
.withOutputTags(readings2010, TupleTagList.of(readings2000plus).and(readingsOld))
.of(new DoFn<TableRow, String>() {
#Override
public void processElement(DoFn<TableRow, String>.ProcessContext c) throws Exception {
if (c.element().getF().get(2).getV().equals("2010")) {
c.output(c.element().toString());
} else if (Integer.parseInt(c.element().getF().get(2).getV().toString()) > 2000) {
c.sideOutput(readings2000plus, c.element().toString());
} else {
c.sideOutput(readingsOld, c.element().toString());
}
}
}));
collectionTuple.get(readings2010)
.apply(PubsubIO.Write.named("WriteToPubsub1").topic("projects/fh-dataflow/topics/bq2pubsub-topic1"));
collectionTuple.get(readings2000plus)
.apply(PubsubIO.Write.named("WriteToPubsub2").topic("projects/fh-dataflow/topics/bq2pubsub-topic2"));
collectionTuple.get(readingsOld)
.apply(PubsubIO.Write.named("WriteToPubsub3").topic("projects/fh-dataflow/topics/bq2pubsub-topic3"));
p.run();

Creating/Writing to Parititoned BigQuery table via Google Cloud Dataflow

I wanted to take advantage of the new BigQuery functionality of time partitioned tables, but am unsure this is currently possible in the 1.6 version of the Dataflow SDK.
Looking at the BigQuery JSON API, to create a day partitioned table one needs to pass in a
"timePartitioning": { "type": "DAY" }
option, but the com.google.cloud.dataflow.sdk.io.BigQueryIO interface only allows specifying a TableReference.
I thought that maybe I could pre-create the table, and sneak in a partition decorator via a BigQueryIO.Write.toTableReference lambda..? Is anyone else having success with creating/writing partitioned tables via Dataflow?
This seems like a similar issue to setting the table expiration time which isn't currently available either.
As Pavan says, it is definitely possible to write to partition tables with Dataflow. Are you using the DataflowPipelineRunner operating in streaming mode or batch mode?
The solution you proposed should work. Specifically, if you pre-create a table with date partitioning set up, then you can use a BigQueryIO.Write.toTableReference lambda to write to a date partition. For example:
/**
* A Joda-time formatter that prints a date in format like {#code "20160101"}.
* Threadsafe.
*/
private static final DateTimeFormatter FORMATTER =
DateTimeFormat.forPattern("yyyyMMdd").withZone(DateTimeZone.UTC);
// This code generates a valid BigQuery partition name:
Instant instant = Instant.now(); // any Joda instant in a reasonable time range
String baseTableName = "project:dataset.table"; // a valid BigQuery table name
String partitionName =
String.format("%s$%s", baseTableName, FORMATTER.print(instant));
The approach I took (works in the streaming mode, too):
Define a custom window for the incoming record
Convert the window into the table/partition name
p.apply(PubsubIO.Read
.subscription(subscription)
.withCoder(TableRowJsonCoder.of())
)
.apply(Window.into(new TablePartitionWindowFn()) )
.apply(BigQueryIO.Write
.to(new DayPartitionFunc(dataset, table))
.withSchema(schema)
.withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_APPEND)
);
Setting the window based on the incoming data, the End Instant can be ignored, as the start value is used for setting the partition:
public class TablePartitionWindowFn extends NonMergingWindowFn<Object, IntervalWindow> {
private IntervalWindow assignWindow(AssignContext context) {
TableRow source = (TableRow) context.element();
String dttm_str = (String) source.get("DTTM");
DateTimeFormatter formatter = DateTimeFormat.forPattern("yyyy-MM-dd").withZoneUTC();
Instant start_point = Instant.parse(dttm_str,formatter);
Instant end_point = start_point.withDurationAdded(1000, 1);
return new IntervalWindow(start_point, end_point);
};
#Override
public Coder<IntervalWindow> windowCoder() {
return IntervalWindow.getCoder();
}
#Override
public Collection<IntervalWindow> assignWindows(AssignContext c) throws Exception {
return Arrays.asList(assignWindow(c));
}
#Override
public boolean isCompatible(WindowFn<?, ?> other) {
return false;
}
#Override
public IntervalWindow getSideInputWindow(BoundedWindow window) {
if (window instanceof GlobalWindow) {
throw new IllegalArgumentException(
"Attempted to get side input window for GlobalWindow from non-global WindowFn");
}
return null;
}
Setting the table partition dynamically:
public class DayPartitionFunc implements SerializableFunction<BoundedWindow, String> {
String destination = "";
public DayPartitionFunc(String dataset, String table) {
this.destination = dataset + "." + table+ "$";
}
#Override
public String apply(BoundedWindow boundedWindow) {
// The cast below is safe because CalendarWindows.days(1) produces IntervalWindows.
String dayString = DateTimeFormat.forPattern("yyyyMMdd")
.withZone(DateTimeZone.UTC)
.print(((IntervalWindow) boundedWindow).start());
return destination + dayString;
}}
Is there a better way of achieving the same outcome?
I believe it should be possible to use the partition decorator when you are not using streaming. We are actively working on supporting partition decorators through streaming. Please let us know if you are seeing any errors today with non-streaming mode.
Apache Beam version 2.0 supports sharding BigQuery output tables out of the box.
I have written data into bigquery partitioned tables through dataflow. These writings are dynamic as-in if the data in that partition already exists then I can either append to it or overwrite it.
I have written the code in Python. It is a batch mode write operation into bigquery.
client = bigquery.Client(project=projectName)
dataset_ref = client.dataset(datasetName)
table_ref = dataset_ref.table(bqTableName)
job_config = bigquery.LoadJobConfig()
job_config.skip_leading_rows = skipLeadingRows
job_config.source_format = bigquery.SourceFormat.CSV
if tableExists(client, table_ref):
job_config.autodetect = autoDetect
previous_rows = client.get_table(table_ref).num_rows
#assert previous_rows > 0
if allowJaggedRows is True:
job_config.allowJaggedRows = True
if allowFieldAddition is True:
job_config._properties['load']['schemaUpdateOptions'] = ['ALLOW_FIELD_ADDITION']
if isPartitioned is True:
job_config._properties['load']['timePartitioning'] = {"type": "DAY"}
if schemaList is not None:
job_config.schema = schemaList
job_config.write_disposition = bigquery.WriteDisposition.WRITE_TRUNCATE
else:
job_config.autodetect = autoDetect
job_config._properties['createDisposition'] = 'CREATE_IF_NEEDED'
job_config.schema = schemaList
if isPartitioned is True:
job_config._properties['load']['timePartitioning'] = {"type": "DAY"}
if schemaList is not None:
table = bigquery.Table(table_ref, schema=schemaList)
load_job = client.load_table_from_uri(gcsFileName, table_ref, job_config=job_config)
assert load_job.job_type == 'load'
load_job.result()
assert load_job.state == 'DONE'
It works fine.
If you pass the table name in table_name_YYYYMMDD format, then BigQuery will treat it as a sharded table, which can simulate partition table features.
Refer the documentation: https://cloud.google.com/bigquery/docs/partitioned-tables

dataflow bigquery partition [duplicate]

I wanted to take advantage of the new BigQuery functionality of time partitioned tables, but am unsure this is currently possible in the 1.6 version of the Dataflow SDK.
Looking at the BigQuery JSON API, to create a day partitioned table one needs to pass in a
"timePartitioning": { "type": "DAY" }
option, but the com.google.cloud.dataflow.sdk.io.BigQueryIO interface only allows specifying a TableReference.
I thought that maybe I could pre-create the table, and sneak in a partition decorator via a BigQueryIO.Write.toTableReference lambda..? Is anyone else having success with creating/writing partitioned tables via Dataflow?
This seems like a similar issue to setting the table expiration time which isn't currently available either.
As Pavan says, it is definitely possible to write to partition tables with Dataflow. Are you using the DataflowPipelineRunner operating in streaming mode or batch mode?
The solution you proposed should work. Specifically, if you pre-create a table with date partitioning set up, then you can use a BigQueryIO.Write.toTableReference lambda to write to a date partition. For example:
/**
* A Joda-time formatter that prints a date in format like {#code "20160101"}.
* Threadsafe.
*/
private static final DateTimeFormatter FORMATTER =
DateTimeFormat.forPattern("yyyyMMdd").withZone(DateTimeZone.UTC);
// This code generates a valid BigQuery partition name:
Instant instant = Instant.now(); // any Joda instant in a reasonable time range
String baseTableName = "project:dataset.table"; // a valid BigQuery table name
String partitionName =
String.format("%s$%s", baseTableName, FORMATTER.print(instant));
The approach I took (works in the streaming mode, too):
Define a custom window for the incoming record
Convert the window into the table/partition name
p.apply(PubsubIO.Read
.subscription(subscription)
.withCoder(TableRowJsonCoder.of())
)
.apply(Window.into(new TablePartitionWindowFn()) )
.apply(BigQueryIO.Write
.to(new DayPartitionFunc(dataset, table))
.withSchema(schema)
.withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_APPEND)
);
Setting the window based on the incoming data, the End Instant can be ignored, as the start value is used for setting the partition:
public class TablePartitionWindowFn extends NonMergingWindowFn<Object, IntervalWindow> {
private IntervalWindow assignWindow(AssignContext context) {
TableRow source = (TableRow) context.element();
String dttm_str = (String) source.get("DTTM");
DateTimeFormatter formatter = DateTimeFormat.forPattern("yyyy-MM-dd").withZoneUTC();
Instant start_point = Instant.parse(dttm_str,formatter);
Instant end_point = start_point.withDurationAdded(1000, 1);
return new IntervalWindow(start_point, end_point);
};
#Override
public Coder<IntervalWindow> windowCoder() {
return IntervalWindow.getCoder();
}
#Override
public Collection<IntervalWindow> assignWindows(AssignContext c) throws Exception {
return Arrays.asList(assignWindow(c));
}
#Override
public boolean isCompatible(WindowFn<?, ?> other) {
return false;
}
#Override
public IntervalWindow getSideInputWindow(BoundedWindow window) {
if (window instanceof GlobalWindow) {
throw new IllegalArgumentException(
"Attempted to get side input window for GlobalWindow from non-global WindowFn");
}
return null;
}
Setting the table partition dynamically:
public class DayPartitionFunc implements SerializableFunction<BoundedWindow, String> {
String destination = "";
public DayPartitionFunc(String dataset, String table) {
this.destination = dataset + "." + table+ "$";
}
#Override
public String apply(BoundedWindow boundedWindow) {
// The cast below is safe because CalendarWindows.days(1) produces IntervalWindows.
String dayString = DateTimeFormat.forPattern("yyyyMMdd")
.withZone(DateTimeZone.UTC)
.print(((IntervalWindow) boundedWindow).start());
return destination + dayString;
}}
Is there a better way of achieving the same outcome?
I believe it should be possible to use the partition decorator when you are not using streaming. We are actively working on supporting partition decorators through streaming. Please let us know if you are seeing any errors today with non-streaming mode.
Apache Beam version 2.0 supports sharding BigQuery output tables out of the box.
I have written data into bigquery partitioned tables through dataflow. These writings are dynamic as-in if the data in that partition already exists then I can either append to it or overwrite it.
I have written the code in Python. It is a batch mode write operation into bigquery.
client = bigquery.Client(project=projectName)
dataset_ref = client.dataset(datasetName)
table_ref = dataset_ref.table(bqTableName)
job_config = bigquery.LoadJobConfig()
job_config.skip_leading_rows = skipLeadingRows
job_config.source_format = bigquery.SourceFormat.CSV
if tableExists(client, table_ref):
job_config.autodetect = autoDetect
previous_rows = client.get_table(table_ref).num_rows
#assert previous_rows > 0
if allowJaggedRows is True:
job_config.allowJaggedRows = True
if allowFieldAddition is True:
job_config._properties['load']['schemaUpdateOptions'] = ['ALLOW_FIELD_ADDITION']
if isPartitioned is True:
job_config._properties['load']['timePartitioning'] = {"type": "DAY"}
if schemaList is not None:
job_config.schema = schemaList
job_config.write_disposition = bigquery.WriteDisposition.WRITE_TRUNCATE
else:
job_config.autodetect = autoDetect
job_config._properties['createDisposition'] = 'CREATE_IF_NEEDED'
job_config.schema = schemaList
if isPartitioned is True:
job_config._properties['load']['timePartitioning'] = {"type": "DAY"}
if schemaList is not None:
table = bigquery.Table(table_ref, schema=schemaList)
load_job = client.load_table_from_uri(gcsFileName, table_ref, job_config=job_config)
assert load_job.job_type == 'load'
load_job.result()
assert load_job.state == 'DONE'
It works fine.
If you pass the table name in table_name_YYYYMMDD format, then BigQuery will treat it as a sharded table, which can simulate partition table features.
Refer the documentation: https://cloud.google.com/bigquery/docs/partitioned-tables

Spark Streaming + Spark SQL

I am trying to process logs via Spark Streaming and Spark SQL. The main idea is to have a "compacted" dataset with Parquet format for "old" data converted to DataFrame as needed for queries, the compacted dataset loading is done with:
SQLContext sqlContext = JavaSQLContextSingleton.getInstance(sc.sc());
DataFrame compact = null;
compact = sqlContext.parquetFile("hdfs://auto-ha/tmp/data/logs");
As the uncompacted dataset (I compact the dataset daily) is composed of many files, I would like to have the data in the current day within a DStream in order to get those queries fast.
I have tried the DataFrame approach without results....
DataFrame df = JavaSQLContextSingleton.getInstance(sc.sc()).createDataFrame(lastData, schema);
df.registerTempTable("lastData");
JavaDStream SumStream = inputStream.transform(new Function<JavaRDD<Row>, JavaRDD<Object>>() {
#Override
public JavaRDD<Object> call(JavaRDD<Row> v1) throws Exception {
DataFrame df = JavaSQLContextSingleton.getInstance(v1.context()).createDataFrame(v1, schema);
......drop old data from lastData table
df.insertInto("lastData");
}
});
Using this approach I do not get any results if I query the temp table in a different thread for example.
I have also tried to use the RDD transform method, more specifically I tried to follow the Spark Example where I create a empty RDD and then I union the DSStream RDD contents with the empty RDD:
JavaRDD<Row> lastData = sc.emptyRDD();
JavaDStream SumStream = inputStream.transform(new Function<JavaRDD<Row>, JavaRDD<Object>>() {
#Override
public JavaRDD<Object> call(JavaRDD<Row> v1) throws Exception {
lastData.union(v1).filter(let only recent data....);
}
});
This approach does not work too as I do not get any contents in the lastData
Could I use for this purpose Windowed computations or updateStateBy key?
Any suggestions?
Thanks for your help!
Well I finally got it.
I use updateState function and return 0 if the timestamp is older than 24 hour like this.
final static Function2<List<Long>, Optional<Long>, Optional<Long>> RETAIN_RECENT_DATA
= (List<Long> values, Optional<Long> state) -> {
Long newSum = state.or(0L);
for (Long value : values) {
newSum += value;
}
//current milis uses UTC
if (System.currentTimeMillis() - newSum > 86400000L) {
return Optional.absent();
} else {
return Optional.of(newSum);
}
};
Then on each batch I register the DataFrame as temp table:
finalsum.foreachRDD((JavaRDD<Row> rdd, Time time) -> {
if (!rdd.isEmpty()) {
HiveContext sqlContext1 = JavaSQLContextSingleton.getInstance(rdd.context());
if (sqlContext1.cacheManager().isCached("alarm_recent")) {
sqlContext1.uncacheTable("alarm_recent");
}
DataFrame wordsDataFrame = sqlContext1.createDataFrame(rdd, schema);
wordsDataFrame.registerTempTable("alarm_recent");
wordsDataFrame.cache();//
wordsDataFrame.first();
}
return null;
});
You could use mapwithState with Spark1.6.
The mapwithState function is much more efficient and easy to implement.
Take a look at this link.
mapwithState supports cool functionality like State time out and initialRDD which comes handy while maintaining a Stateful Dstream.
Thanks
Manas

Example Program to insert a row using BAPI with JCO3

I am trying to "insert" (or) "add a row" to Purchase Requisition using standard BAPI (PurchaseRequisition.CreateFromData).
I am using JCo3. The example in JCo3 indicates that we should use table.appendRow() OR table.insertRow() methods. I am trying with table.appendRow() & table.appendRows(1). When i try to insert a row, i dont get any error and the row is not inserted.
Below is the program i am trying to execute.
/** Below are the inputs required for this program to run /
/ Step 1 **/
String BAPI_NAME = "BAPI_REQUISITION_CREATE";
/** Step 2 **/
String query_input_column1 = "DOCUMENTY_TYPE";
String query_input_column1_value = "NB";
String query_input_column2 = "PREQ_NAME";
String query_input_column2_value = "Name";
String query_input_column3 = "ACCTASSCAT";
String query_input_column3_value = "U";
String query_input_column4 = "DELIV_DATE";
String query_input_column4_value = "20131101";
String query_input_column5 = "MATERIAL";
String query_input_column5_value = "DELL-RQ2013";
String query_input_column6 = "QUANITY";
int query_input_column6_value = 10100;
/** Step 3 **/
String targetTableUnderBAPI = "REQUISITION_ITEMS";
/** Step 4 **/
/** For the confirmation read the value from export parameter after insertion execution **/
String result_column1 = "NUMBER";
JCoDestination destination = null;
try {
destination = JCoDestinationManager.getDestination(DestinationManager.DESTINATION_NAME1);
JCoRepository repository = destination.getRepository();
JCoContext.begin(destination);
JCoFunction function = repository.getFunction(BAPI_NAME);
if(function == null)
throw new RuntimeException(BAPI_NAME + " not found in SAP.");
System.out.println("BAPI Name from function object: " + function.getName());
//function.getImportParameterList().setValue(query_input_column1, query_input_column1_value);
JCoTable table = function.getTableParameterList().getTable(targetTableUnderBAPI); //it is taken from the response value of metadata
//System.out.println("No of Columns: "+ table.getNumColumns());
System.out.println("Trying to execute append row");
table.appendRow();
table.setValue(query_input_column1,query_input_column1_value);
table.setValue(query_input_column2,query_input_column2_value);
table.setValue(query_input_column3,query_input_column3_value);
//table.setValue(query_input_column4,new java.util.Date(query_input_column4_value));
//skipped Other columns related code
try{
function.execute(destination);
}
catch(AbapException e){
System.out.println(e.toString());
return;
}
System.out.println("Let us check the result from export parameter");
String exportParamStructure = (String)function.getExportParameterList().getValue(result_column1); //getStructure(result_column1); // getValue(result_column1);
System.out.println("Resulting PR#: "+exportParamStructure);
} catch (JCoException e) {
e.printStackTrace();
}
finally
{
try {
JCoContext.end(destination);
} catch (JCoException e) {
e.printStackTrace();
}
}
I did not understand how to read the response and am trying to fetch it from exportParameters!!
Can anybody share a piece of code to insert and
getting confirmation response (do we get the PREQ_NO in response?)
I am adding date field value as "20131101", but not sure if the format and approach is right?
when i try to add Quantity column value, i get an error message complaining this column is not part of BAPIEBANC. But the column is visible in BAPIEBANC type.
any configuration on SAP side to be checked?
should i activate any fields in JCo side? if so, how
Please note that my knowledge on SAP is very limited.
Waiting for an expert's response.
Thanks.
First, you should take a look at SAP JCo documentation, e.g.
http://help.sap.com/saphelp_nw04/helpdata/en/6f/1bd5c6a85b11d6b28500508b5d5211/content.htm
Regarding your code:
Adding (one) row to the table looks right on first sight.
Your code says QUANITY instead of QUANTITY.
You should add date values as java.util.Date; if creating a Date from a String format, you should use java.text.DateFormat.parse(). See http://docs.oracle.com/javase/6/docs/api/java/util/Date.html (this is however Java specific and has nothing to do with JCo).
If changing anything in SAP, never forget to call BAPI_TRANSACTION_COMMIT in the end to finish the logical unit of work (aka transaction) or nothing will actually be changed.
If you don't like to fiddle with the more or less complicated and verbose JCo API, try using Hibersap which gives you a much nicer programming model when calling functions in SAP ERP: http://hibersap.org.
However, you will still need a basic understanding on how SAP function modules work technically (such as parameter types or data types) as well as on the domain specific model which lies behind them (in your case, creating a requisition). I.e. you may need to communicate with your SAP experts.
Here I added 2 types of insertion :
insertval() function for user defined module resides in sap with the help of abap programmer
Its an standard module for insert a ticket using jco to SOLMAN system. First you have to analyse import, export, table & structure parameters, and according to that you have to pass values and retrieve response. In second function it will return ticket n° after successfull insertion of ticket in solman.
I hope this sample code will help you, it worked for me.
public class jco
{
static String DESTINATION_NAME1 = "ABAP_AS_WITHOUT_POOL";
static String DESTINATION_NAME2 = "ABAP_AS_WITH_POOL";
static
{
Properties connectProperties = new Properties();
connectProperties.setProperty(DestinationDataProvider.JCO_ASHOST, "192.1.1.1");
connectProperties.setProperty(DestinationDataProvider.JCO_SYSNR, "01");
connectProperties.setProperty(DestinationDataProvider.JCO_CLIENT, "500");
connectProperties.setProperty(DestinationDataProvider.JCO_USER, "uname");
connectProperties.setProperty(DestinationDataProvider.JCO_PASSWD, "pwd");
connectProperties.setProperty(DestinationDataProvider.JCO_LANG, "en");
createDestinationDataFile(DESTINATION_NAME1, connectProperties);
connectProperties.setProperty(DestinationDataProvider.JCO_POOL_CAPACITY, "3");
connectProperties.setProperty(DestinationDataProvider.JCO_PEAK_LIMIT, "10");
createDestinationDataFile(DESTINATION_NAME2, connectProperties);
System.err.println("hai");
}
static void createDestinationDataFile(String destinationName, Properties connectProperties)
{
File destCfg = new File(destinationName+".jcoDestination");
try
{
try (FileOutputStream fos = new FileOutputStream(destCfg, false)) {
connectProperties.store(fos, "for tests only !");
}
}
catch (IOException e)
{
throw new RuntimeException("Unable to create the destination files", e);
}
}
public void insertval() throws JCoException
{
JCoDestination destination = JCoDestinationManager.getDestination(DESTINATION_NAME1);
JCoFunction jf=destination.getRepository().getFunction("ZUSER_DET");
jf.getImportParameterList().setValue("FIRST_NAME","member");
jf.getImportParameterList().setValue("LAST_NAME","c");
jf.getImportParameterList().setValue("USER_NO","1000");
jf.execute(destination);
System.out.println(jf);
}
public void insertticket() throws JCoException
{
JCoDestination destination = JCoDestinationManager.getDestination(DESTINATION_NAME2);
System.out.println("test"+"\n");
JCoFunction jf=destination.getRepository().getFunction("BAPI_NOTIFICATION_CREATE");
JCoTable jt1=jf.getTableParameterList().getTable("APPX_HEADERS");
JCoTable jt2=jf.getTableParameterList().getTable("APPX_LINES");
JCoTable jt3=jf.getTableParameterList().getTable("APPX_LINES_BIN");
JCoTable jt4=jf.getTableParameterList().getTable("NOTIF_NOTES");
JCoTable jt5=jf.getTableParameterList().getTable("NOTIF_PARTNERS");
JCoTable jt6=jf.getTableParameterList().getTable("NOTIF_SAP_DATA");
JCoTable jt7=jf.getTableParameterList().getTable("NOTIF_TEXT_HEADERS");
JCoTable jt8=jf.getTableParameterList().getTable("NOTIF_TEXT_LINES");
JCoStructure jfn1=jf.getImportParameterList().getStructure("NOTIF_EXT");
JCoStructure jfn2=jf.getImportParameterList().getStructure("NOTIF_CRM");
JCoStructure jfn3=jf.getImportParameterList().getStructure("IBASE_DATA");
jfn1.setValue("NUMB","1234");
jfn1.setValue("REFNUM","123");
jfn1.setValue("TYPE_NOTIF","SLFN");
jfn1.setValue("SUBJECT","tl");
jfn1.setValue("PRIORITY","2");
jfn1.setValue("LANGUAGE","EN");
jfn1.setValue("CATEGORY","Z01");
jfn2.setValue("CODE","0011");
jfn2.setValue("CODEGROUP","0011");
jfn2.setValue("CATEGORY","Z01");
jfn3.setValue("INSTANCE","489");
jfn3.setValue("IBASE","500");
jt1.appendRow();
jt1.setValue("DESCR","practise");
jt2.appendRow();
jt2.setValue("LINE","CVXCVXCV");
jt3.appendRow();
jt3.setValue("LINE","second text line");
jt4.appendRow();
jt4.setValue("TYPE_NOTE","my");
jt4.setValue("IDENT","hoe twwrtgw");
jt4.setValue("DESCRIPTION","its description ");
jt5.appendRow();
jt5.setValue("PARNR","new ");
jt5.setValue("TYPE_PAR","FN");
jt5.setValue("FUNC_PAR","EN");
jt5.setValue("PAR_ACTIVE","1");
jt6.appendRow();
jt6.setValue("INSTN","0020214076");
jt6.setValue("COMP","FI-AA");
jt6.setValue("SYSTYPE","P");
jt6.setValue("SYSID","PRD");
jt6.setValue("MANDT","900");
jt8.appendRow();
jt8.setValue("TXT_NUM","1");
jt8.setValue("TDFORMAT",">X");
jt8.setValue("TDLINE","/(performing all test)");
jf.execute(destination);
String jfex=jf.getExportParameterList().getString("REFNUM");
System.out.println("hi "+jfex);
}