When I try to find the size of all the tables using listTables API of BigQuery java client, it returns null. But if I use getTable individually, I get the proper data. Is this a known issue, or am I doing something wrong. Following is the code that returns null value for numBytes:
Page<Dataset> datasetPage = getAllDatasets("projectId");
if(datasetPage!=null) {
for (Dataset dataset : datasetPage.iterateAll()) {
for(Table table : dataset.list().iterateAll()) {
System.out.println(table.getNumBytes()); // Returns Null. **
}
}
}
In this Public Issue Tracker thread, it has been discussed that getting null value for numBytes and numRows using listTables is the expected behaviour. The BigQuery API considers retrieving numBytes and numRows to be an expensive operation and thus returns null. So, the listTables only returns partial information on a table.
As a workaround, use getTable() to retrieve the information of the table individually in a loop. I tested the below code snippet and was able to get the table size in bytes for all the tables.
public static void getAllTableSize(String projectId) {
try {
BigQuery bigquery = BigQueryOptions.getDefaultInstance().getService();
Page<Dataset> datasetPage = bigquery.listDatasets(projectId);
if(datasetPage!=null) {
for (Dataset datasetTemp : datasetPage.iterateAll()) {
for(Table table : datasetTemp.list().iterateAll()) {
Table tableTemp = bigquery.getTable(table.getTableId());
String tableName = tableTemp.getTableId().getTable();
Long tableSize = tableTemp.getNumBytes();
System.out.println("Table Name: " + tableName + " " + "Table Size: " + tableSize);
}
}
}
} catch (BigQueryException e) {
System.out.println("Error occurred: " + e.toString());
}
}
To answer my own question listTables api is designed to return only the partial information. This is mentioned in the code document
https://github.com/googleapis/java-bigquery/blob/dfa15e5ca08a3227f015a389c4c08732178a73e7/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/spi/v2/BigQueryRpc.java#L155
Related
I'm trying to do a select count query in Sql Server using Dapper. The expected response should be 0 when a profile does not exist. When I do the query in SSMS it returns correctly, but in the API using Dapper it returns 1. Any idea why this is happening?
public IActionResult GetProfileCount(string profileId)
{
int profileCount = 0;
using (IDbConnection db = new SqlConnection(connectionString))
{
try
{
profileCount = db.Query($"select count(*) from Profile where Id='{profileId}'").Count();
}
catch(Exception ex)
{
Console.WriteLine($"Error retrieving count for ProfileId: {profileId}", ex.Message);
}
}
return Ok(profileCount);
}
I see you added your own answer but could I recommend not doing it that way. When you do
profileCount = db.Query($"select * from Profile where Id='{profileId}'").Count();
What you are actually doing is selecting every field from the database, pulling it into your C# application, and then counting how many results you got back. Then you are binning all that data you got back, very inefficient!
Change it to this :
profileCount = db.QueryFirst<int>($"select count(*) from Profile where Id = #profileId", new { profileId })");
Instead you are selecting an "int" from the result set, which just so happens to be your count(*). Perfect!
More on querying in Dapper here : https://dotnetcoretutorials.com/2019/08/05/dapper-in-net-core-part-2-dapper-query-basics/
Also notice that (similar to the other answer), I am using parameterized queries. I also heavily recommend this as it protects you from SQL Injection. Your initial example is very vulnerable!
You can read a little more about SQL Injection in C#/MSSQL here https://dotnetcoretutorials.com/2017/10/11/owasp-top-10-asp-net-core-sql-injection/ But just know that Dapper protects you from it as long as you use the inbuilt helpers to add parameters to your queries.
Another option is use the method ExecuteScalar for "select count" queries:
profileCount = db.ExecuteScalar<int>("select count(*) from Profile where Id=#profileId", new { profileId });
Ref.: https://www.learndapper.com/selecting-scalar-values
Try and change your query to the following:
db.Query($"select count(*) from Profile where Id = #ProfileId", new { ProfileId = profileId }).Count()
I figured it out. The .Count() is counting the rows of the result, which is going to be 1 because the result is one row displaying the number 0. I switched my code to this and it works now.
public IActionResult GetProfileCount(string profileId)
{
int profileCount = 0;
using (IDbConnection db = new SqlConnection(connectionString))
{
try
{
profileCount = db.Query($"select * from Profile where Id='{profileId}'").Count();
}
catch(Exception ex)
{
Console.WriteLine($"Error retrieving count for ProfileId: {profileId}", ex.Message);
}
}
return Ok(profileCount);
}
Using jdbc I am querying my database of ambulance response times. My goal is to take the output and process it into statistics using Jakarta Commons Math library. So far I am successful in querying my database and outputting the response times to the console. My next step is to process this output statistically, such as mean, medians, mode, etc. This is where I am stuck. Shown below is my code.
package javaDatabase;
import java.sql.*;
import org.apache.commons.math3.stat.StatUtils;
public class javaConnect3
{
public static void main(String[] args)
{
Connection conn = null;
Statement stmt = null;
try
{
conn = DriverManager
.getConnection("jdbc:sqlserver://myServerAddress;database=myDatabase;integratedsecurity=false;user=myUser;password=myPassword");
stmt = conn.createStatement();
String strSelect = "SELECT M_SecondsAtStatus FROM MManpower WHERE M_tTime > 'august 25, 2014' AND M_Code = 'USAR'";
ResultSet rset = stmt.executeQuery(strSelect);
while (rset.next())
{
int values = rset.getInt("M_SecondsAtStatus");
System.out.println(values);
}
// I am hoping to derive useful statistics from my database, such as the following.
// this uses Jakarta Commons Math
// System.out.println("min: " + StatUtils.min(values));
// System.out.println("max: " + StatUtils.max(values));
// System.out.println("mean: " + StatUtils.mean(values));
// System.out.println("product: " + StatUtils.product(values));
// System.out.println("sum: " + StatUtils.sum(values));
// System.out.println("variance: " + StatUtils.variance(values));
} catch (SQLException ex)
{
ex.printStackTrace();
} finally
{
try
{
if (stmt != null)
stmt.close();
if (conn != null)
conn.close();
} catch (SQLException ex)
{
ex.printStackTrace();
}
}
}
}
An error message pops up in Eclipse and the variable "values" is red underlined; "values cannot be resolved to a variable".
I am not sure how to get this to work. I don't understand how to output my ambulance response times from the database into something Apache Commons math will understand.
How can I get Apache Commons math to take the output from my database and generate a statistical result?
You need something like this:
List<Double> values = new ArrayList<Double>();
while (rset.next()) {
values.add(rset.getInt("M_SecondsAtStatus"));
}
double mean = StatsUtils.mean(values.toArray(new Double [](values.size()));
It's likely that you could query the database for statistics. The name of the method depends on your database:
SELECT avg(M_SecondsAtStatus)
FROM MManpower
WHERE M_tTime > 'august 25, 2014'
AND M_Code = 'USAR'
I'd say the second one is more efficient, because you don't have to transfer all those values to the Java JVM to do the same calculation that the database can do for you.
Another alternative is to use a DescriptiveStatistics object to collect all the data and then use it to compute the summary statistics you need. Using it alleviates the need for doing all explicit type casting and conversion:
DescriptiveStatistics ds = new DescriptiveStatistics();
while(rset.next()) {
int observation = rset.getInt("M_SecondsAtStatus");
ds.addValue(observation);
}
System.out.println("min: " + ds.getMin());
System.out.println("max: " + ds.getMax());
...
Is there any way to programmatically understand what properties there are for a custom model in Alfresco?
For example, data dictionary in oracle helps you to find what tables are defined by which column names and column data types.
My goal is a sample code in java which extracts all custom defined content models, their attributes, their attributes data types, etc. For example my sample code should return to me that there is a custom content Model which has an integer attribute with name 'No' and a string attribute with name 'Description'. I know this is possible with DictionaryComponent, But I don't know how should I use it.
Take a look at org.alfresco.service.cmr.dictionary.DictionaryService.
will your Java-code running in-context with alfresco (as part of the /alfresco webapp) or outside?
If your running in-context, then Alfreso Java Foundation API is your choice where DictionaryService is part of. just inject the bean into your java code.
If your code is running outside of alfresco, then Alfresco REST API or CMIS is your choice.
If you're still lost, then train yourself first. Jeff wrote some good articles http://ecmarchitect.com/alfresco-developer-series & also a book.
Also a good starting point - Alfresco Developer Docs: http://docs.alfresco.com/4.2/topic/com.alfresco.enterprise.doc/concepts/dev-for-developers.html
thanks a lot of alfresian responses, My code sample is something like below:
I found the sample code from : this URL
public void GetAllAvailableDataTypes() throws IOException
{
Session session = getSession();
boolean includePropertyDefintions = true;
for (Tree t : session.getTypeDescendants(
null, // start at the top of the tree
-1, // infinite depth recursion
includePropertyDefintions // include prop defs
)) {
printTypes(t, "");
}
}
public void printTypes(Tree tree, String tab) {
ObjectType objType = (ObjectType) tree.getItem();
String type = objType.getId();
if(true)//type.endsWith("hstcase"))
{
System.out.println(tab + "TYPE:" + objType.getDisplayName() +
" (" + objType.getDescription() + ")");
//Print some of the common attributes for this type
System.out.print(tab + " Id:" + "-----"+ objType.getId() + "-----");
System.out.print(" Fileable:" + objType.isFileable());
System.out.print(" Queryable:" + objType.isQueryable());
if (objType instanceof DocumentType) {
System.out.print(" [DOC Attrs->] Versionable:" +
((DocumentType)objType).isVersionable());
System.out.println(" Content:" +
((DocumentType)objType).getContentStreamAllowed());
Map<String, org.apache.chemistry.opencmis.commons.definitions.PropertyDefinition<?>> props = objType.getPropertyDefinitions();
Set<String> keys = props.keySet();
Collection<org.apache.chemistry.opencmis.commons.definitions.PropertyDefinition<?>> porpsAsCollection = props.values();
for(org.apache.chemistry.opencmis.commons.definitions.PropertyDefinition<?> p: porpsAsCollection)
{
//System.out.println(p.getDescription());
PropertyType pt = p.getPropertyType();
System.out.println("Display Name: "+ p.getDisplayName());
System.out.println("Local Name: "+ p.getLocalName());
System.out.println("Attribute Type: "+ pt.value());
}
}
System.out.println(""); // end the line
}
List<Tree> childs = tree.getChildren();
for (Tree t : childs) {
// there are more - call self for next level
printTypes(t, tab + " ");
}
}
I am trying to "insert" (or) "add a row" to Purchase Requisition using standard BAPI (PurchaseRequisition.CreateFromData).
I am using JCo3. The example in JCo3 indicates that we should use table.appendRow() OR table.insertRow() methods. I am trying with table.appendRow() & table.appendRows(1). When i try to insert a row, i dont get any error and the row is not inserted.
Below is the program i am trying to execute.
/** Below are the inputs required for this program to run /
/ Step 1 **/
String BAPI_NAME = "BAPI_REQUISITION_CREATE";
/** Step 2 **/
String query_input_column1 = "DOCUMENTY_TYPE";
String query_input_column1_value = "NB";
String query_input_column2 = "PREQ_NAME";
String query_input_column2_value = "Name";
String query_input_column3 = "ACCTASSCAT";
String query_input_column3_value = "U";
String query_input_column4 = "DELIV_DATE";
String query_input_column4_value = "20131101";
String query_input_column5 = "MATERIAL";
String query_input_column5_value = "DELL-RQ2013";
String query_input_column6 = "QUANITY";
int query_input_column6_value = 10100;
/** Step 3 **/
String targetTableUnderBAPI = "REQUISITION_ITEMS";
/** Step 4 **/
/** For the confirmation read the value from export parameter after insertion execution **/
String result_column1 = "NUMBER";
JCoDestination destination = null;
try {
destination = JCoDestinationManager.getDestination(DestinationManager.DESTINATION_NAME1);
JCoRepository repository = destination.getRepository();
JCoContext.begin(destination);
JCoFunction function = repository.getFunction(BAPI_NAME);
if(function == null)
throw new RuntimeException(BAPI_NAME + " not found in SAP.");
System.out.println("BAPI Name from function object: " + function.getName());
//function.getImportParameterList().setValue(query_input_column1, query_input_column1_value);
JCoTable table = function.getTableParameterList().getTable(targetTableUnderBAPI); //it is taken from the response value of metadata
//System.out.println("No of Columns: "+ table.getNumColumns());
System.out.println("Trying to execute append row");
table.appendRow();
table.setValue(query_input_column1,query_input_column1_value);
table.setValue(query_input_column2,query_input_column2_value);
table.setValue(query_input_column3,query_input_column3_value);
//table.setValue(query_input_column4,new java.util.Date(query_input_column4_value));
//skipped Other columns related code
try{
function.execute(destination);
}
catch(AbapException e){
System.out.println(e.toString());
return;
}
System.out.println("Let us check the result from export parameter");
String exportParamStructure = (String)function.getExportParameterList().getValue(result_column1); //getStructure(result_column1); // getValue(result_column1);
System.out.println("Resulting PR#: "+exportParamStructure);
} catch (JCoException e) {
e.printStackTrace();
}
finally
{
try {
JCoContext.end(destination);
} catch (JCoException e) {
e.printStackTrace();
}
}
I did not understand how to read the response and am trying to fetch it from exportParameters!!
Can anybody share a piece of code to insert and
getting confirmation response (do we get the PREQ_NO in response?)
I am adding date field value as "20131101", but not sure if the format and approach is right?
when i try to add Quantity column value, i get an error message complaining this column is not part of BAPIEBANC. But the column is visible in BAPIEBANC type.
any configuration on SAP side to be checked?
should i activate any fields in JCo side? if so, how
Please note that my knowledge on SAP is very limited.
Waiting for an expert's response.
Thanks.
First, you should take a look at SAP JCo documentation, e.g.
http://help.sap.com/saphelp_nw04/helpdata/en/6f/1bd5c6a85b11d6b28500508b5d5211/content.htm
Regarding your code:
Adding (one) row to the table looks right on first sight.
Your code says QUANITY instead of QUANTITY.
You should add date values as java.util.Date; if creating a Date from a String format, you should use java.text.DateFormat.parse(). See http://docs.oracle.com/javase/6/docs/api/java/util/Date.html (this is however Java specific and has nothing to do with JCo).
If changing anything in SAP, never forget to call BAPI_TRANSACTION_COMMIT in the end to finish the logical unit of work (aka transaction) or nothing will actually be changed.
If you don't like to fiddle with the more or less complicated and verbose JCo API, try using Hibersap which gives you a much nicer programming model when calling functions in SAP ERP: http://hibersap.org.
However, you will still need a basic understanding on how SAP function modules work technically (such as parameter types or data types) as well as on the domain specific model which lies behind them (in your case, creating a requisition). I.e. you may need to communicate with your SAP experts.
Here I added 2 types of insertion :
insertval() function for user defined module resides in sap with the help of abap programmer
Its an standard module for insert a ticket using jco to SOLMAN system. First you have to analyse import, export, table & structure parameters, and according to that you have to pass values and retrieve response. In second function it will return ticket n° after successfull insertion of ticket in solman.
I hope this sample code will help you, it worked for me.
public class jco
{
static String DESTINATION_NAME1 = "ABAP_AS_WITHOUT_POOL";
static String DESTINATION_NAME2 = "ABAP_AS_WITH_POOL";
static
{
Properties connectProperties = new Properties();
connectProperties.setProperty(DestinationDataProvider.JCO_ASHOST, "192.1.1.1");
connectProperties.setProperty(DestinationDataProvider.JCO_SYSNR, "01");
connectProperties.setProperty(DestinationDataProvider.JCO_CLIENT, "500");
connectProperties.setProperty(DestinationDataProvider.JCO_USER, "uname");
connectProperties.setProperty(DestinationDataProvider.JCO_PASSWD, "pwd");
connectProperties.setProperty(DestinationDataProvider.JCO_LANG, "en");
createDestinationDataFile(DESTINATION_NAME1, connectProperties);
connectProperties.setProperty(DestinationDataProvider.JCO_POOL_CAPACITY, "3");
connectProperties.setProperty(DestinationDataProvider.JCO_PEAK_LIMIT, "10");
createDestinationDataFile(DESTINATION_NAME2, connectProperties);
System.err.println("hai");
}
static void createDestinationDataFile(String destinationName, Properties connectProperties)
{
File destCfg = new File(destinationName+".jcoDestination");
try
{
try (FileOutputStream fos = new FileOutputStream(destCfg, false)) {
connectProperties.store(fos, "for tests only !");
}
}
catch (IOException e)
{
throw new RuntimeException("Unable to create the destination files", e);
}
}
public void insertval() throws JCoException
{
JCoDestination destination = JCoDestinationManager.getDestination(DESTINATION_NAME1);
JCoFunction jf=destination.getRepository().getFunction("ZUSER_DET");
jf.getImportParameterList().setValue("FIRST_NAME","member");
jf.getImportParameterList().setValue("LAST_NAME","c");
jf.getImportParameterList().setValue("USER_NO","1000");
jf.execute(destination);
System.out.println(jf);
}
public void insertticket() throws JCoException
{
JCoDestination destination = JCoDestinationManager.getDestination(DESTINATION_NAME2);
System.out.println("test"+"\n");
JCoFunction jf=destination.getRepository().getFunction("BAPI_NOTIFICATION_CREATE");
JCoTable jt1=jf.getTableParameterList().getTable("APPX_HEADERS");
JCoTable jt2=jf.getTableParameterList().getTable("APPX_LINES");
JCoTable jt3=jf.getTableParameterList().getTable("APPX_LINES_BIN");
JCoTable jt4=jf.getTableParameterList().getTable("NOTIF_NOTES");
JCoTable jt5=jf.getTableParameterList().getTable("NOTIF_PARTNERS");
JCoTable jt6=jf.getTableParameterList().getTable("NOTIF_SAP_DATA");
JCoTable jt7=jf.getTableParameterList().getTable("NOTIF_TEXT_HEADERS");
JCoTable jt8=jf.getTableParameterList().getTable("NOTIF_TEXT_LINES");
JCoStructure jfn1=jf.getImportParameterList().getStructure("NOTIF_EXT");
JCoStructure jfn2=jf.getImportParameterList().getStructure("NOTIF_CRM");
JCoStructure jfn3=jf.getImportParameterList().getStructure("IBASE_DATA");
jfn1.setValue("NUMB","1234");
jfn1.setValue("REFNUM","123");
jfn1.setValue("TYPE_NOTIF","SLFN");
jfn1.setValue("SUBJECT","tl");
jfn1.setValue("PRIORITY","2");
jfn1.setValue("LANGUAGE","EN");
jfn1.setValue("CATEGORY","Z01");
jfn2.setValue("CODE","0011");
jfn2.setValue("CODEGROUP","0011");
jfn2.setValue("CATEGORY","Z01");
jfn3.setValue("INSTANCE","489");
jfn3.setValue("IBASE","500");
jt1.appendRow();
jt1.setValue("DESCR","practise");
jt2.appendRow();
jt2.setValue("LINE","CVXCVXCV");
jt3.appendRow();
jt3.setValue("LINE","second text line");
jt4.appendRow();
jt4.setValue("TYPE_NOTE","my");
jt4.setValue("IDENT","hoe twwrtgw");
jt4.setValue("DESCRIPTION","its description ");
jt5.appendRow();
jt5.setValue("PARNR","new ");
jt5.setValue("TYPE_PAR","FN");
jt5.setValue("FUNC_PAR","EN");
jt5.setValue("PAR_ACTIVE","1");
jt6.appendRow();
jt6.setValue("INSTN","0020214076");
jt6.setValue("COMP","FI-AA");
jt6.setValue("SYSTYPE","P");
jt6.setValue("SYSID","PRD");
jt6.setValue("MANDT","900");
jt8.appendRow();
jt8.setValue("TXT_NUM","1");
jt8.setValue("TDFORMAT",">X");
jt8.setValue("TDLINE","/(performing all test)");
jf.execute(destination);
String jfex=jf.getExportParameterList().getString("REFNUM");
System.out.println("hi "+jfex);
}
I am using the starschema JDBC driver for Google BigQuery in Pentaho PDI:
http://code.google.com/p/starschema-bigquery-jdbc/
My query through the BigQuery Web Console returns 129,993 rows, but when I execute the same query through the JDBC driver it only returns 100,000 rows. Is there some kind of option or limit that I am not aware of?
The StarSchema code looks like it is only returning the first page of results.
The code here here should be updated to get the rest of the results. It should look something like:
public static GetQueryResultsResponse getQueryResults(Bigquery bigquery,
String projectId, Job completedJob) throws IOException {
GetQueryResultsResponse queryResult = bigquery.jobs()
.getQueryResults(projectId,
completedJob.getJobReference().getJobId()).execute();
while(queryResult.getTotalRows() > queryResult.getRows().size()) {
queryResult.getRows().addAll(
bigquery.jobs()
.getQueryResults(projectId,
completedJob.getJobReference().getJobId())
.setStartIndex(queryResult.getRows().size())
.execute()
.getRows());
}
return queryResult;
}
Modified the code based on Jordan's answer, the solution looks like this:
public static GetQueryResultsResponse getQueryResults(Bigquery bigquery,
String projectId, Job completedJob) throws IOException {
GetQueryResultsResponse queryResult = bigquery.jobs()
.getQueryResults(projectId,
completedJob.getJobReference().getJobId()).execute();
long totalRows = queryResult.getTotalRows().longValue();
if(totalRows == 0){
//if we don't have results we'll get a nullPointerException on the queryResult.getRows().size()
return queryResult;
}
while( totalRows > (long)queryResult.getRows().size() ) {
queryResult.getRows().addAll(
bigquery.jobs()
.getQueryResults(projectId,
completedJob.getJobReference().getJobId())
.setStartIndex(BigInteger.valueOf((long)queryResult.getRows().size()) )
.execute()
.getRows());
}
return queryResult;
}
This should solve the problem.
Also uploaded the new version to google code, named bqjdbc-1.3.1.jar