Create table out of a query result? - google-bigquery

I'm using Bigquery's Java API. I'm running a select query and want the result saved to a destination table.
I've set the loadConfig.setDestinationTable() but I am getting "Load configuration must specify at least one source URI".
Could you please explain what am I doing wrong?

You don't want to set the loadConfig destination table, but the queryConfig.setDestinationTable() instead (since this isn't a load job -- it is a query job). As Fh said, if you share the code you're using we can give more detailed help.

this is the code i am using to do this:
public static String copyTable(String project, String dataSet, String table) {
String newTableName = table + "_copy_"+System.currentTimeMillis();;
try {
Job copyJob = new Job();
TableReference source = new TableReference();
source.setProjectId(project);
source.setDatasetId(dataSet);
source.setTableId(table);
TableReference destination = new TableReference();
destination.setProjectId(project);
destination.setDatasetId(dataSet);
destination.setTableId(newTableName);
JobConfiguration configuration = new JobConfiguration();
JobConfigurationTableCopy copyConf = new JobConfigurationTableCopy();
copyConf.setSourceTable(source);
copyConf.setDestinationTable(destination);
configuration.setCopy(copyConf);
copyJob.setConfiguration(configuration);
bigquery.jobs().insert(project, copyJob).execute();
return newTableName;
} catch (Exception e) {
e.printStackTrace();
logger.warn("unable to copy table :" + project + "."
+ dataSet + "." + table, e);
throw new RuntimeException(e);
}
}
please contact me if you have any more questions

Assuming you are running an interactive asynchronous query, you essentially want to pass the query, destination projectId, destination dataSetId and destination tableId in one request body. Refer to the Java API example here: https://developers.google.com/bigquery/querying-data#asyncqueries

Related

How do I create a BigQuery view with standard SQL using the Java API?

I can create views using the Java API, but the query needs to be legacy sql:
public void createView(String dataSet, String viewName, String query) throws Exception {
Table content = new Table();
TableReference tableReference = new TableReference();
tableReference.setTableId(viewName);
tableReference.setDatasetId(dataSet);
tableReference.setProjectId(projectId);
content.setTableReference(tableReference);
ViewDefinition view = new ViewDefinition();
view.setQuery(query);
content.setView(view);
LOG.debug("View to create: " + content);
try {
if (tableExists(dataSet, viewName)) {
bigquery.tables().delete(projectId, dataSet, viewName).execute();
}
} catch (Exception e) {
LOG.error("Could not delete table", e);
}
bigquery.tables().insert(projectId, dataSet, content).setProjectId(projectId).execute();
}
Is there a way to create a BQ view with standard sql using the API?
You need to set setUseLegacySQL(false) on the ViewDefinition object.
[..]
ViewDefinition view = new ViewDefinition();
view.setQuery(query);
view.setUseLegacySql(false); //<-- you're missing this
content.setView(view);
[..]
See the API docs here.
With the new API, you can use the "#standardSQL" notation to avoid the default LegacySQL setting (the setUseLegacySql() method no longer exists in the new API).
Here is an example:
ViewDefinition tableDefinition = ViewDefinition.newBuilder("#standardSQL\n WITH A AS (select 1 as foo) SELECT * from A").build();

Example Program to insert a row using BAPI with JCO3

I am trying to "insert" (or) "add a row" to Purchase Requisition using standard BAPI (PurchaseRequisition.CreateFromData).
I am using JCo3. The example in JCo3 indicates that we should use table.appendRow() OR table.insertRow() methods. I am trying with table.appendRow() & table.appendRows(1). When i try to insert a row, i dont get any error and the row is not inserted.
Below is the program i am trying to execute.
/** Below are the inputs required for this program to run /
/ Step 1 **/
String BAPI_NAME = "BAPI_REQUISITION_CREATE";
/** Step 2 **/
String query_input_column1 = "DOCUMENTY_TYPE";
String query_input_column1_value = "NB";
String query_input_column2 = "PREQ_NAME";
String query_input_column2_value = "Name";
String query_input_column3 = "ACCTASSCAT";
String query_input_column3_value = "U";
String query_input_column4 = "DELIV_DATE";
String query_input_column4_value = "20131101";
String query_input_column5 = "MATERIAL";
String query_input_column5_value = "DELL-RQ2013";
String query_input_column6 = "QUANITY";
int query_input_column6_value = 10100;
/** Step 3 **/
String targetTableUnderBAPI = "REQUISITION_ITEMS";
/** Step 4 **/
/** For the confirmation read the value from export parameter after insertion execution **/
String result_column1 = "NUMBER";
JCoDestination destination = null;
try {
destination = JCoDestinationManager.getDestination(DestinationManager.DESTINATION_NAME1);
JCoRepository repository = destination.getRepository();
JCoContext.begin(destination);
JCoFunction function = repository.getFunction(BAPI_NAME);
if(function == null)
throw new RuntimeException(BAPI_NAME + " not found in SAP.");
System.out.println("BAPI Name from function object: " + function.getName());
//function.getImportParameterList().setValue(query_input_column1, query_input_column1_value);
JCoTable table = function.getTableParameterList().getTable(targetTableUnderBAPI); //it is taken from the response value of metadata
//System.out.println("No of Columns: "+ table.getNumColumns());
System.out.println("Trying to execute append row");
table.appendRow();
table.setValue(query_input_column1,query_input_column1_value);
table.setValue(query_input_column2,query_input_column2_value);
table.setValue(query_input_column3,query_input_column3_value);
//table.setValue(query_input_column4,new java.util.Date(query_input_column4_value));
//skipped Other columns related code
try{
function.execute(destination);
}
catch(AbapException e){
System.out.println(e.toString());
return;
}
System.out.println("Let us check the result from export parameter");
String exportParamStructure = (String)function.getExportParameterList().getValue(result_column1); //getStructure(result_column1); // getValue(result_column1);
System.out.println("Resulting PR#: "+exportParamStructure);
} catch (JCoException e) {
e.printStackTrace();
}
finally
{
try {
JCoContext.end(destination);
} catch (JCoException e) {
e.printStackTrace();
}
}
I did not understand how to read the response and am trying to fetch it from exportParameters!!
Can anybody share a piece of code to insert and
getting confirmation response (do we get the PREQ_NO in response?)
I am adding date field value as "20131101", but not sure if the format and approach is right?
when i try to add Quantity column value, i get an error message complaining this column is not part of BAPIEBANC. But the column is visible in BAPIEBANC type.
any configuration on SAP side to be checked?
should i activate any fields in JCo side? if so, how
Please note that my knowledge on SAP is very limited.
Waiting for an expert's response.
Thanks.
First, you should take a look at SAP JCo documentation, e.g.
http://help.sap.com/saphelp_nw04/helpdata/en/6f/1bd5c6a85b11d6b28500508b5d5211/content.htm
Regarding your code:
Adding (one) row to the table looks right on first sight.
Your code says QUANITY instead of QUANTITY.
You should add date values as java.util.Date; if creating a Date from a String format, you should use java.text.DateFormat.parse(). See http://docs.oracle.com/javase/6/docs/api/java/util/Date.html (this is however Java specific and has nothing to do with JCo).
If changing anything in SAP, never forget to call BAPI_TRANSACTION_COMMIT in the end to finish the logical unit of work (aka transaction) or nothing will actually be changed.
If you don't like to fiddle with the more or less complicated and verbose JCo API, try using Hibersap which gives you a much nicer programming model when calling functions in SAP ERP: http://hibersap.org.
However, you will still need a basic understanding on how SAP function modules work technically (such as parameter types or data types) as well as on the domain specific model which lies behind them (in your case, creating a requisition). I.e. you may need to communicate with your SAP experts.
Here I added 2 types of insertion :
insertval() function for user defined module resides in sap with the help of abap programmer
Its an standard module for insert a ticket using jco to SOLMAN system. First you have to analyse import, export, table & structure parameters, and according to that you have to pass values and retrieve response. In second function it will return ticket n° after successfull insertion of ticket in solman.
I hope this sample code will help you, it worked for me.
public class jco
{
static String DESTINATION_NAME1 = "ABAP_AS_WITHOUT_POOL";
static String DESTINATION_NAME2 = "ABAP_AS_WITH_POOL";
static
{
Properties connectProperties = new Properties();
connectProperties.setProperty(DestinationDataProvider.JCO_ASHOST, "192.1.1.1");
connectProperties.setProperty(DestinationDataProvider.JCO_SYSNR, "01");
connectProperties.setProperty(DestinationDataProvider.JCO_CLIENT, "500");
connectProperties.setProperty(DestinationDataProvider.JCO_USER, "uname");
connectProperties.setProperty(DestinationDataProvider.JCO_PASSWD, "pwd");
connectProperties.setProperty(DestinationDataProvider.JCO_LANG, "en");
createDestinationDataFile(DESTINATION_NAME1, connectProperties);
connectProperties.setProperty(DestinationDataProvider.JCO_POOL_CAPACITY, "3");
connectProperties.setProperty(DestinationDataProvider.JCO_PEAK_LIMIT, "10");
createDestinationDataFile(DESTINATION_NAME2, connectProperties);
System.err.println("hai");
}
static void createDestinationDataFile(String destinationName, Properties connectProperties)
{
File destCfg = new File(destinationName+".jcoDestination");
try
{
try (FileOutputStream fos = new FileOutputStream(destCfg, false)) {
connectProperties.store(fos, "for tests only !");
}
}
catch (IOException e)
{
throw new RuntimeException("Unable to create the destination files", e);
}
}
public void insertval() throws JCoException
{
JCoDestination destination = JCoDestinationManager.getDestination(DESTINATION_NAME1);
JCoFunction jf=destination.getRepository().getFunction("ZUSER_DET");
jf.getImportParameterList().setValue("FIRST_NAME","member");
jf.getImportParameterList().setValue("LAST_NAME","c");
jf.getImportParameterList().setValue("USER_NO","1000");
jf.execute(destination);
System.out.println(jf);
}
public void insertticket() throws JCoException
{
JCoDestination destination = JCoDestinationManager.getDestination(DESTINATION_NAME2);
System.out.println("test"+"\n");
JCoFunction jf=destination.getRepository().getFunction("BAPI_NOTIFICATION_CREATE");
JCoTable jt1=jf.getTableParameterList().getTable("APPX_HEADERS");
JCoTable jt2=jf.getTableParameterList().getTable("APPX_LINES");
JCoTable jt3=jf.getTableParameterList().getTable("APPX_LINES_BIN");
JCoTable jt4=jf.getTableParameterList().getTable("NOTIF_NOTES");
JCoTable jt5=jf.getTableParameterList().getTable("NOTIF_PARTNERS");
JCoTable jt6=jf.getTableParameterList().getTable("NOTIF_SAP_DATA");
JCoTable jt7=jf.getTableParameterList().getTable("NOTIF_TEXT_HEADERS");
JCoTable jt8=jf.getTableParameterList().getTable("NOTIF_TEXT_LINES");
JCoStructure jfn1=jf.getImportParameterList().getStructure("NOTIF_EXT");
JCoStructure jfn2=jf.getImportParameterList().getStructure("NOTIF_CRM");
JCoStructure jfn3=jf.getImportParameterList().getStructure("IBASE_DATA");
jfn1.setValue("NUMB","1234");
jfn1.setValue("REFNUM","123");
jfn1.setValue("TYPE_NOTIF","SLFN");
jfn1.setValue("SUBJECT","tl");
jfn1.setValue("PRIORITY","2");
jfn1.setValue("LANGUAGE","EN");
jfn1.setValue("CATEGORY","Z01");
jfn2.setValue("CODE","0011");
jfn2.setValue("CODEGROUP","0011");
jfn2.setValue("CATEGORY","Z01");
jfn3.setValue("INSTANCE","489");
jfn3.setValue("IBASE","500");
jt1.appendRow();
jt1.setValue("DESCR","practise");
jt2.appendRow();
jt2.setValue("LINE","CVXCVXCV");
jt3.appendRow();
jt3.setValue("LINE","second text line");
jt4.appendRow();
jt4.setValue("TYPE_NOTE","my");
jt4.setValue("IDENT","hoe twwrtgw");
jt4.setValue("DESCRIPTION","its description ");
jt5.appendRow();
jt5.setValue("PARNR","new ");
jt5.setValue("TYPE_PAR","FN");
jt5.setValue("FUNC_PAR","EN");
jt5.setValue("PAR_ACTIVE","1");
jt6.appendRow();
jt6.setValue("INSTN","0020214076");
jt6.setValue("COMP","FI-AA");
jt6.setValue("SYSTYPE","P");
jt6.setValue("SYSID","PRD");
jt6.setValue("MANDT","900");
jt8.appendRow();
jt8.setValue("TXT_NUM","1");
jt8.setValue("TDFORMAT",">X");
jt8.setValue("TDLINE","/(performing all test)");
jf.execute(destination);
String jfex=jf.getExportParameterList().getString("REFNUM");
System.out.println("hi "+jfex);
}

C# Sql Data not saving

I have a few tables in a c# application I'm currently working on and for 4/5 of the tables everything saves perfectly fine no issues. For the 5th table everything seems good until I reload the program again (without modifying the code or working with a seperate install so that the data doesn't go away) The 4/5 tables are fine but the 5th doesn't have any records in it after it has been restarted (but it did the last time it was running). Below is some code excerpts. I have tried a few different solutions online including creating a string to run the sql commands on the database manually and creating the row directly as opposed to the below implementation which uses a generic data row.
//From main window
private void newInvoice_Click(object sender, EventArgs e)
{
PosDatabaseDataSet.InvoicesRow newInvoice = posDatabaseDataSet1.Invoices.NewInvoicesRow();
Invoices iForm = new Invoices(newInvoice, posDatabaseDataSet1, true);
}
//Invoices Table save [Works] (from Invoices.cs)
private void saveInvoice_Click(object sender, EventArgs e)
{
iRecord.Date = Convert.ToDateTime(this.dateField.Text);
iRecord.InvoiceNo = Convert.ToInt32(this.invoiceNumField.Text);
iRecord.Subtotal = (float) Convert.ToDouble(this.subtotalField.Text);
iRecord.Tax1 = (float)Convert.ToDouble(this.hstField.Text);
iRecord.Total = (float)Convert.ToDouble(this.totalField.Text);
iRecord.BillTo = this.billToField.Text;
invoicesBindingSource.EndEdit();
if (newRecord)
{
dSet.Invoices.Rows.Add(iRecord);
invoicesTableAdapter.Adapter.Update(dSet.Invoices);
}
else
{
string connString = Properties.Settings.Default.PosDatabaseConnectionString;
string queryString = "UPDATE dbo.Invoices set ";
queryString += "Date='" + iRecord.Date+"'";
queryString += ", Subtotal=" + iRecord.Subtotal;
queryString += ", Tax1=" + iRecord.Tax1.ToString("N2");
queryString += ", Total=" + iRecord.Total;
queryString += " WHERE InvoiceNo=" + iRecord.InvoiceNo;
using (SqlConnection dbConn = new SqlConnection(connString))
{
SqlCommand command = new SqlCommand(queryString, dbConn);
dbConn.Open();
SqlDataReader r = command.ExecuteReader();
dbConn.Close();
}
}
dSet.Invoices.AcceptChanges();
}
//Invoice Items save [works until restart] (also from Invoices.cs)
private void addLine_Click(object sender, EventArgs e)
{
DataRow iRow = dSet.Tables["InvoiceItems"].NewRow();
iRow["Cost"] = (float)Convert.ToDouble(this.costField.Text);
iRow["Description"] = this.descriptionField.Text;
iRow["InvoiceNo"] = Convert.ToInt32(this.invoiceNumField.Text);
iRow["JobId"] = Convert.ToInt32(this.jobIdField.Text);
iRow["Qty"] = Convert.ToInt32(this.quantityField.Text);
iRow["SalesPerson"] = Convert.ToInt32(this.salesPersonField.Text);
iRow["SKU"] = Convert.ToInt32(this.skuField.Text);
dSet.Tables["InvoiceItems"].Rows.Add(iRow);
invoiceItemsTableAdapter.Adapter.Update(dSet,"InvoiceItems");
PosDatabaseDataSet.InvoiceItemsDataTable dTable = (PosDatabaseDataSet.InvoiceItemsDataTable)dSet.InvoiceItems.Copy();
DataRow[] d = dTable.Select("InvoiceNo=" + invNo.ToString());
invoiceItemsView.DataSource = d;
}
Thanks in advance for any insight.
UPDATE: October 17, 2011. I am still unable to get this working is there any more ideas out there?
you must execute your Sql Command in order to persis the changes you made.
using (SqlConnection dbConn = new SqlConnection(connString))
{
dbConn.Open();
SqlCommand command = new SqlCommand(queryString, dbConn);
command.ExecuteNonQuery();
dbConn.Close();
}
The ExecuteReader method is intended (as the name says) to read the data from a SQL table. You need to use a different method as you can see above.
We need some more info first, you haven't shown the case where your code fails.
Common mistakes on this kind of code is calling DataSet.AcceptChanges() before actually committing the changes to the database.
Second is a conflict between databound data through the binding source vs edits to the dataset directly.
Lets see the appropriate code and we can try and help.
Set a breakpoint after teh call to invoiceItemsTableAdapter and check the InvoiceItems table for the row you have added. Release the breakpoint and then close your app. Check the database again. I would say that another table may be forcibly overwriting the invoice item table.

Partial replace on SQL image data column

This question is related to another one I posted earlier.
To recap, I need to fix an issue with an ancient legacy app where people messed up data storage by re-installing the software the wrong way.
The application stores data by saving a record in an SQL DB. Each record holds a reference to a file on disk of which the filename auto-increments.
By re-installing the app the filename auto-increment was re-set so the DB now holds multiple unrelated records which reference the same filename and I have to directories with files which I obviously cannot merge because of these identical filenames. The files hold no reference to the DB data so the only course of action that remains is to filter the DB records on date created and try to rename "EXED" to "IXED" or something like that.
The DB is relatively simple with one table containing a column that holds data of type "Image".
An example content of this image data is as follows:
0x3200001000000000000000200B0000000EFF00000300000031340000000070EC0100002C50000004000000C90000005D010000040000007955B63F4D01000004000000F879883E4F01000004000000BC95563E98010000040000009A99993F4A01000004000000000000004B01000004000000000000009101000004000000000000004E01000004000000721C83425101000004000000D841493F5E01000004000000898828414101000004000000F2D2BD3F4201000004000000FCA9B13F40010000040000007574204244010000040000000000204345010000040000007DD950414601000004000000000000004701000004000000000000009201000004000000000000008701000004000000D2DF13426A0100000400000000005C42740100000400000046B68F40500100000400000018E97A3F7901000004000000FB50CF3C7A01000004000000E645703F99010000040000000000E0404C010000040000008716593F8601000004000000000006439A0100000400000000008040700100000400000063D887449E01000004000000493CBA3E9C0100000400000069699D429B01000004000000DD60CA3F9D0100000400000035DE3C44B4010000040000008B5C744433000000040000003D0ABB4134000000040000000AFF7C44350000000400000093CB3942750400000400000054A69F41BA010000040000002635C64173040000040000008367C24100000080690100002B5000003101000032000010000000000000002009000000000000000100000000000000F00000000000000080080100000100000010000000540100000100000021F0AA42270000000200000010000000540100000200000021F0AA42280000000300000010000000540100000300000059C9E6432900000004000000100000005401000004000000637888442A00000005000000100000005401000005000000DFEF87442B00000006000000100000005401000006000000000000002C00000007000000100000005401000007000000000000002D00000008000000100000005401000008000000000000002D000000090000001000000054010000090000002F353D442D0000000A00000010000000540100000A00000035DE3C44340000000B00000010000000540100000B0000008B5C7444240000009D50000010000000CDCCCC3E2C513B41F65D5F3F2C51BB419E50000010000000CCBA2C3FE17C8C411553B13F83F32142000000403700000000FE0000090000004558454434386262002D50000008000000447973706E6F65008E5000000E00000056454C442052414D502033363000000000F000000000
The data is apparently Hex which mostly encodes meaningless crap but also holds the name of physical files (towards the end of the data field) in the filesystem that is linked to the SQL records:
??#7???????????EXED48bb?-P??????Dyspnoe??P??????VELD RAMP 360
I'm interested in the EXED part.
There is no clear regularity in the offset at which the filename appears and the filename is of variable length (so I do not know beforehand how long the substring will be).
I can call up all records with SQL like this:
SELECT COUNT(*) as "Number of EXED Files after critical date"
FROM [ZAN].[dbo].[zanu]
WHERE udata is not null
and SUBSTRING(udata, 1 , 2147483647) like '%EXED%'
and [udatum] > 0
and CONVERT(date,[udatum]) > CONVERT(date,'20100629')
What I would like to do now is know how to replace this EXED substring by something else (e.g. IXID).
I'm unfamiliar with SQL and Googling so far has yielded very little information on my options here.
I also have no other info on the original code that generated this data/the data format/encoding/whatever...
It's a mess really.
Any help is welcome!
An update on this:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Data.Linq;
using System.Text;
using System.Data.SqlClient;
using System.Threading;
namespace ZANLinq
{
class Program
{
static void Main(string[] args)
{
try
{
DataContext zanDB = new DataContext(#"Data Source=.\,1433;database=ZAN;Integrated Security=true");
string strSQL = #"SELECT
Idnr,
Udatum,
Uzeit,
Unr,
Uart,
Ubediener,
Uzugriff,
Ugr,
Uflags,
Usize,
Udata
FROM Zanu
WHERE (Udata IS NOT null and SubString(Udata, 1 , 2147483647) LIKE '%EXED%')
AND (Idnr = ' 2')";
var zanQuery = zanDB.ExecuteQuery<Zanu>(strSQL);
List<Zanu> list = zanQuery.ToList<Zanu>();
foreach (Zanu zanTofix in list)
{
string strOriginal = ASCIIEncoding.ASCII.GetString(zanTofix.Udata);
string strFixed = strOriginal.Replace("EXED", "IXED");
zanTofix.Udata = ASCIIEncoding.ASCII.GetBytes(strFixed);
}
zanDB.SubmitChanges();
//Console.WriteLine(zanResults.Count<Zanu>().ToString());
}
catch (SqlException e)
{
Console.WriteLine(e.Message);
}
}
}
}
It finds the records I'm interested in, I can easily manipulate the data but the commit doesnt work. I'm stumped, there are no exceptions, no indication the code is wrong.
Anybody have ideas?
UPDATE:
I think the above does not work because my table appears to have a composite PK (I cannot change this):
Since I could not debug this (no info anywhere, no exceptions, just a silent fail of the submitchanges()) I decided to use another approach and abandon Linq2SQL altogether:
try
{
SqlConnection thisConnection = new SqlConnection(#"Network Library=DBMSSOCN;Data Source=.\,1433;database=ZAN;Integrated Security=SSPI");
DataSet zanDataSet = new DataSet();
SqlDataAdapter zanDa;
SqlCommandBuilder zanCmdBuilder;
thisConnection.Open();
//Initialize the SqlDataAdapter object by specifying a Select command
//that retrieves data from the sample table.
zanDa = new SqlDataAdapter(#"SELECT
Idnr,
Udatum,
Uzeit,
Unr,
Uart,
Ubediener,
Uzugriff,
Ugr,
Uflags,
Usize,
Udata
FROM Zanu
WHERE (Udata IS NOT null and SubString(Udata, 1 , 2147483647) LIKE '%IXED%')
AND (Idnr = ' 2')
AND (Uzeit = '13:21')", thisConnection);
//Initialize the SqlCommandBuilder object to automatically generate and initialize
//the UpdateCommand, InsertCommand, and DeleteCommand properties of the SqlDataAdapter.
zanCmdBuilder = new SqlCommandBuilder(zanDa);
//Populate the DataSet by running the Fill method of the SqlDataAdapter.
zanDa.Fill(zanDataSet, "Zanu");
Console.WriteLine("Records that will be affected: " + zanDataSet.Tables["Zanu"].Rows.Count.ToString());
foreach (DataRow record in zanDataSet.Tables["Zanu"].Rows)
{
string strOriginal = ASCIIEncoding.ASCII.GetString((byte[])record["Udata"]);
string strFixed = strOriginal.Replace("IXED", "EXED");
record["Udata"] = ASCIIEncoding.ASCII.GetBytes(strFixed);
//string strPostMod = ASCIIEncoding.ASCII.GetString((byte[])record["Udata"]);
}
zanDa.Update(zanDataSet, "Zanu");
thisConnection.Close();
Console.ReadLine();
}
catch (SqlException e)
{
Console.WriteLine(e.Message);
}
This seems to work but any input on why the Linq does not work and whether or not my second solution is efficient/optimal or not is still very much appreciated.

error, string or binary data would be truncated when trying to insert

I am running data.bat file with the following lines:
Rem Tis batch file will populate tables
cd\program files\Microsoft SQL Server\MSSQL
osql -U sa -P Password -d MyBusiness -i c:\data.sql
The contents of the data.sql file is:
insert Customers
(CustomerID, CompanyName, Phone)
Values('101','Southwinds','19126602729')
There are 8 more similar lines for adding records.
When I run this with start > run > cmd > c:\data.bat, I get this error message:
1>2>3>4>5>....<1 row affected>
Msg 8152, Level 16, State 4, Server SP1001, Line 1
string or binary data would be truncated.
<1 row affected>
<1 row affected>
<1 row affected>
<1 row affected>
<1 row affected>
<1 row affected>
Also, I am a newbie obviously, but what do Level #, and state # mean, and how do I look up error messages such as the one above: 8152?
From #gmmastros's answer
Whenever you see the message....
string or binary data would be truncated
Think to yourself... The field is NOT big enough to hold my data.
Check the table structure for the customers table. I think you'll find that the length of one or more fields is NOT big enough to hold the data you are trying to insert. For example, if the Phone field is a varchar(8) field, and you try to put 11 characters in to it, you will get this error.
I had this issue although data length was shorter than the field length.
It turned out that the problem was having another log table (for audit trail), filled by a trigger on the main table, where the column size also had to be changed.
In one of the INSERT statements you are attempting to insert a too long string into a string (varchar or nvarchar) column.
If it's not obvious which INSERT is the offender by a mere look at the script, you could count the <1 row affected> lines that occur before the error message. The obtained number plus one gives you the statement number. In your case it seems to be the second INSERT that produces the error.
Just want to contribute with additional information: I had the same issue and it was because of the field wasn't big enough for the incoming data and this thread helped me to solve it (the top answer clarifies it all).
BUT it is very important to know what are the possible reasons that may cause it.
In my case i was creating the table with a field like this:
Select '' as Period, * From Transactions Into #NewTable
Therefore the field "Period" had a length of Zero and causing the Insert operations to fail. I changed it to "XXXXXX" that is the length of the incoming data and it now worked properly (because field now had a lentgh of 6).
I hope this help anyone with same issue :)
Some of your data cannot fit into your database column (small). It is not easy to find what is wrong. If you use C# and Linq2Sql, you can list the field which would be truncated:
First create helper class:
public class SqlTruncationExceptionWithDetails : ArgumentOutOfRangeException
{
public SqlTruncationExceptionWithDetails(System.Data.SqlClient.SqlException inner, DataContext context)
: base(inner.Message + " " + GetSqlTruncationExceptionWithDetailsString(context))
{
}
/// <summary>
/// PArt of code from following link
/// http://stackoverflow.com/questions/3666954/string-or-binary-data-would-be-truncated-linq-exception-cant-find-which-fiel
/// </summary>
/// <param name="context"></param>
/// <returns></returns>
static string GetSqlTruncationExceptionWithDetailsString(DataContext context)
{
StringBuilder sb = new StringBuilder();
foreach (object update in context.GetChangeSet().Updates)
{
FindLongStrings(update, sb);
}
foreach (object insert in context.GetChangeSet().Inserts)
{
FindLongStrings(insert, sb);
}
return sb.ToString();
}
public static void FindLongStrings(object testObject, StringBuilder sb)
{
foreach (var propInfo in testObject.GetType().GetProperties())
{
foreach (System.Data.Linq.Mapping.ColumnAttribute attribute in propInfo.GetCustomAttributes(typeof(System.Data.Linq.Mapping.ColumnAttribute), true))
{
if (attribute.DbType.ToLower().Contains("varchar"))
{
string dbType = attribute.DbType.ToLower();
int numberStartIndex = dbType.IndexOf("varchar(") + 8;
int numberEndIndex = dbType.IndexOf(")", numberStartIndex);
string lengthString = dbType.Substring(numberStartIndex, (numberEndIndex - numberStartIndex));
int maxLength = 0;
int.TryParse(lengthString, out maxLength);
string currentValue = (string)propInfo.GetValue(testObject, null);
if (!string.IsNullOrEmpty(currentValue) && maxLength != 0 && currentValue.Length > maxLength)
{
//string is too long
sb.AppendLine(testObject.GetType().Name + "." + propInfo.Name + " " + currentValue + " Max: " + maxLength);
}
}
}
}
}
}
Then prepare the wrapper for SubmitChanges:
public static class DataContextExtensions
{
public static void SubmitChangesWithDetailException(this DataContext dataContext)
{
//http://stackoverflow.com/questions/3666954/string-or-binary-data-would-be-truncated-linq-exception-cant-find-which-fiel
try
{
//this can failed on data truncation
dataContext.SubmitChanges();
}
catch (SqlException sqlException) //when (sqlException.Message == "String or binary data would be truncated.")
{
if (sqlException.Message == "String or binary data would be truncated.") //only for EN windows - if you are running different window language, invoke the sqlException.getMessage on thread with EN culture
throw new SqlTruncationExceptionWithDetails(sqlException, dataContext);
else
throw;
}
}
}
Prepare global exception handler and log truncation details:
protected void Application_Error(object sender, EventArgs e)
{
Exception ex = Server.GetLastError();
string message = ex.Message;
//TODO - log to file
}
Finally use the code:
Datamodel.SubmitChangesWithDetailException();
Another situation in which you can get this error is the following:
I had the same error and the reason was that in an INSERT statement that received data from an UNION, the order of the columns was different from the original table. If you change the order in #table3 to a, b, c, you will fix the error.
select a, b, c into #table1
from #table0
insert into #table1
select a, b, c from #table2
union
select a, c, b from #table3
on sql server you can use SET ANSI_WARNINGS OFF like this:
using (SqlConnection conn = new SqlConnection("Data Source=XRAYGOAT\\SQLEXPRESS;Initial Catalog='Healthy Care';Integrated Security=True"))
{
conn.Open();
using (var trans = conn.BeginTransaction())
{
try
{
using cmd = new SqlCommand("", conn, trans))
{
cmd.CommandText = "SET ANSI_WARNINGS OFF";
cmd.ExecuteNonQuery();
cmd.CommandText = "YOUR INSERT HERE";
cmd.ExecuteNonQuery();
cmd.Parameters.Clear();
cmd.CommandText = "SET ANSI_WARNINGS ON";
cmd.ExecuteNonQuery();
trans.Commit();
}
}
catch (Exception)
{
trans.Rollback();
}
}
conn.Close();
}
I had the same issue. The length of my column was too short.
What you can do is either increase the length or shorten the text you want to put in the database.
Also had this problem occurring on the web application surface.
Eventually found out that the same error message comes from the SQL update statement in the specific table.
Finally then figured out that the column definition in the relating history table(s) did not map the original table column length of nvarchar types in some specific cases.
I had the same problem, even after increasing the size of the problematic columns in the table.
tl;dr: The length of the matching columns in corresponding Table Types may also need to be increased.
In my case, the error was coming from the Data Export service in Microsoft Dynamics CRM, which allows CRM data to be synced to an SQL Server DB or Azure SQL DB.
After a lengthy investigation, I concluded that the Data Export service must be using Table-Valued Parameters:
You can use table-valued parameters to send multiple rows of data to a Transact-SQL statement or a routine, such as a stored procedure or function, without creating a temporary table or many parameters.
As you can see in the documentation above, Table Types are used to create the data ingestion procedure:
CREATE TYPE LocationTableType AS TABLE (...);
CREATE PROCEDURE dbo.usp_InsertProductionLocation
#TVP LocationTableType READONLY
Unfortunately, there is no way to alter a Table Type, so it has to be dropped & recreated entirely. Since my table has over 300 fields (😱), I created a query to facilitate the creation of the corresponding Table Type based on the table's columns definition (just replace [table_name] with your table's name):
SELECT 'CREATE TYPE [table_name]Type AS TABLE (' + STRING_AGG(CAST(field AS VARCHAR(max)), ',' + CHAR(10)) + ');' AS create_type
FROM (
SELECT TOP 5000 COLUMN_NAME + ' ' + DATA_TYPE
+ IIF(CHARACTER_MAXIMUM_LENGTH IS NULL, '', CONCAT('(', IIF(CHARACTER_MAXIMUM_LENGTH = -1, 'max', CONCAT(CHARACTER_MAXIMUM_LENGTH,'')), ')'))
+ IIF(DATA_TYPE = 'decimal', CONCAT('(', NUMERIC_PRECISION, ',', NUMERIC_SCALE, ')'), '')
AS field
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME = '[table_name]'
ORDER BY ORDINAL_POSITION) AS T;
After updating the Table Type, the Data Export service started functioning properly once again! :)
When I tried to execute my stored procedure I had the same problem because the size of the column that I need to add some data is shorter than the data I want to add.
You can increase the size of the column data type or reduce the length of your data.
A 2016/2017 update will show you the bad value and column.
A new trace flag will swap the old error for a new 2628 error and will print out the column and offending value. Traceflag 460 is available in the latest cumulative update for 2016 and 2017:
https://support.microsoft.com/en-sg/help/4468101/optional-replacement-for-string-or-binary-data-would-be-truncated
Just make sure that after you've installed the CU that you enable the trace flag, either globally/permanently on the server:
...or with DBCC TRACEON:
https://learn.microsoft.com/en-us/sql/t-sql/database-console-commands/dbcc-traceon-trace-flags-transact-sql?view=sql-server-ver15
Another situation, in which this error may occur is in
SQL Server Management Studio. If you have "text" or "ntext" fields in your table,
no matter what kind of field you are updating (for example bit or integer).
Seems that the Studio does not load entire "ntext" fields and also updates ALL fields instead of the modified one.
To solve the problem, exclude "text" or "ntext" fields from the query in Management Studio
This Error Comes only When any of your field length is greater than the field length specified in sql server database table structure.
To overcome this issue you have to reduce the length of the field Value .
Or to increase the length of database table field .
If someone is encountering this error in a C# application, I have created a simple way of finding offending fields by:
Getting the column width of all the columns of a table where we're trying to make this insert/ update. (I'm getting this info directly from the database.)
Comparing the column widths to the width of the values we're trying to insert/ update.
Assumptions/ Limitations:
The column names of the table in the database match with the C# entity fields. For eg: If you have a column like this in database:
You need to have your Entity with the same column name:
public class SomeTable
{
// Other fields
public string SourceData { get; set; }
}
You're inserting/ updating 1 entity at a time. It'll be clearer in the demo code below. (If you're doing bulk inserts/ updates, you might want to either modify it or use some other solution.)
Step 1:
Get the column width of all the columns directly from the database:
// For this, I took help from Microsoft docs website:
// https://learn.microsoft.com/en-us/dotnet/api/system.data.sqlclient.sqlconnection.getschema?view=netframework-4.7.2#System_Data_SqlClient_SqlConnection_GetSchema_System_String_System_String___
private static Dictionary<string, int> GetColumnSizesOfTableFromDatabase(string tableName, string connectionString)
{
var columnSizes = new Dictionary<string, int>();
using (var connection = new SqlConnection(connectionString))
{
// Connect to the database then retrieve the schema information.
connection.Open();
// You can specify the Catalog, Schema, Table Name, Column Name to get the specified column(s).
// You can use four restrictions for Column, so you should create a 4 members array.
String[] columnRestrictions = new String[4];
// For the array, 0-member represents Catalog; 1-member represents Schema;
// 2-member represents Table Name; 3-member represents Column Name.
// Now we specify the Table_Name and Column_Name of the columns what we want to get schema information.
columnRestrictions[2] = tableName;
DataTable allColumnsSchemaTable = connection.GetSchema("Columns", columnRestrictions);
foreach (DataRow row in allColumnsSchemaTable.Rows)
{
var columnName = row.Field<string>("COLUMN_NAME");
//var dataType = row.Field<string>("DATA_TYPE");
var characterMaxLength = row.Field<int?>("CHARACTER_MAXIMUM_LENGTH");
// I'm only capturing columns whose Datatype is "varchar" or "char", i.e. their CHARACTER_MAXIMUM_LENGTH won't be null.
if(characterMaxLength != null)
{
columnSizes.Add(columnName, characterMaxLength.Value);
}
}
connection.Close();
}
return columnSizes;
}
Step 2:
Compare the column widths with the width of the values we're trying to insert/ update:
public static Dictionary<string, string> FindLongBinaryOrStringFields<T>(T entity, string connectionString)
{
var tableName = typeof(T).Name;
Dictionary<string, string> longFields = new Dictionary<string, string>();
var objectProperties = GetProperties(entity);
//var fieldNames = objectProperties.Select(p => p.Name).ToList();
var actualDatabaseColumnSizes = GetColumnSizesOfTableFromDatabase(tableName, connectionString);
foreach (var dbColumn in actualDatabaseColumnSizes)
{
var maxLengthOfThisColumn = dbColumn.Value;
var currentValueOfThisField = objectProperties.Where(f => f.Name == dbColumn.Key).First()?.GetValue(entity, null)?.ToString();
if (!string.IsNullOrEmpty(currentValueOfThisField) && currentValueOfThisField.Length > maxLengthOfThisColumn)
{
longFields.Add(dbColumn.Key, $"'{dbColumn.Key}' column cannot take the value of '{currentValueOfThisField}' because the max length it can take is {maxLengthOfThisColumn}.");
}
}
return longFields;
}
public static List<PropertyInfo> GetProperties<T>(T entity)
{
//The DeclaredOnly flag makes sure you only get properties of the object, not from the classes it derives from.
var properties = entity.GetType()
.GetProperties(System.Reflection.BindingFlags.Public
| System.Reflection.BindingFlags.Instance
| System.Reflection.BindingFlags.DeclaredOnly)
.ToList();
return properties;
}
Demo:
Let's say we're trying to insert someTableEntity of SomeTable class that is modeled in our app like so:
public class SomeTable
{
[Key]
public long TicketID { get; set; }
public string SourceData { get; set; }
}
And it's inside our SomeDbContext like so:
public class SomeDbContext : DbContext
{
public DbSet<SomeTable> SomeTables { get; set; }
}
This table in Db has SourceData field as varchar(16) like so:
Now we'll try to insert value that is longer than 16 characters into this field and capture this information:
public void SaveSomeTableEntity()
{
var connectionString = "server=SERVER_NAME;database=DB_NAME;User ID=SOME_ID;Password=SOME_PASSWORD;Connection Timeout=200";
using (var context = new SomeDbContext(connectionString))
{
var someTableEntity = new SomeTable()
{
SourceData = "Blah-Blah-Blah-Blah-Blah-Blah"
};
context.SomeTables.Add(someTableEntity);
try
{
context.SaveChanges();
}
catch (Exception ex)
{
if (ex.GetBaseException().Message == "String or binary data would be truncated.\r\nThe statement has been terminated.")
{
var badFieldsReport = "";
List<string> badFields = new List<string>();
// YOU GOT YOUR FIELDS RIGHT HERE:
var longFields = FindLongBinaryOrStringFields(someTableEntity, connectionString);
foreach (var longField in longFields)
{
badFields.Add(longField.Key);
badFieldsReport += longField.Value + "\n";
}
}
else
throw;
}
}
}
The badFieldsReport will have this value:
'SourceData' column cannot take the value of
'Blah-Blah-Blah-Blah-Blah-Blah' because the max length it can take is
16.
Kevin Pope's comment under the accepted answer was what I needed.
The problem, in my case, was that I had triggers defined on my table that would insert update/insert transactions into an audit table, but the audit table had a data type mismatch where a column with VARCHAR(MAX) in the original table was stored as VARCHAR(1) in the audit table, so my triggers were failing when I would insert anything greater than VARCHAR(1) in the original table column and I would get this error message.
I used a different tactic, fields that are allocated 8K in some places. Here only about 50/100 are used.
declare #NVPN_list as table
nvpn varchar(50)
,nvpn_revision varchar(5)
,nvpn_iteration INT
,mpn_lifecycle varchar(30)
,mfr varchar(100)
,mpn varchar(50)
,mpn_revision varchar(5)
,mpn_iteration INT
-- ...
) INSERT INTO #NVPN_LIST
SELECT left(nvpn ,50) as nvpn
,left(nvpn_revision ,10) as nvpn_revision
,nvpn_iteration
,left(mpn_lifecycle ,30)
,left(mfr ,100)
,left(mpn ,50)
,left(mpn_revision ,5)
,mpn_iteration
,left(mfr_order_num ,50)
FROM [DASHBOARD].[dbo].[mpnAttributes] (NOLOCK) mpna
I wanted speed, since I have 1M total records, and load 28K of them.
This error may be due to less field size than your entered data.
For e.g. if you have data type nvarchar(7) and if your value is 'aaaaddddf' then error is shown as:
string or binary data would be truncated
You simply can't beat SQL Server on this.
You can insert into a new table like this:
select foo, bar
into tmp_new_table_to_dispose_later
from my_table
and compare the table definition with the real table you want to insert the data into.
Sometime it's helpful sometimes it's not.
If you try inserting in the final/real table from that temporary table it may just work (due to data conversion working differently than SSMS for example).
Another alternative is to insert the data in chunks, instead of inserting everything immediately you insert with top 1000 and you repeat the process, till you find a chunk with an error. At least you have better visibility on what's not fitting into the table.