Loading a flat file into SQL Server 2008 R2 via BizTalk 2010 - sql

I am trying to find my way around BizTalk 2010. The scenario is simple: I need to take a flat file (delimited, positional, segmented etc...) and load it into a table in SQL Server 2008 R2.
I am trying to do this with a WCF-SQL adapter.
At this point I do not care for speed (i.e., it does not matter if there will be multiple inserts or one bulk insert, but in the long run I would like to be able to do a bulk insert), I just want to get something working.
I have the following done so far:
TestTable1 in table in SQL Server 2008 R2
InsertIntoTestTable stored procedure in SQL Server 2008 R2
Visual Studio Project with (a) a schema for the incoming Flat File, (b) Generated Schema for the SQL table, (c) a Map from the incoming flat file schema to the SQL file schema and (d) a pipeline to disassemble the flat file.
I have deployed this project to BizTalk and created a receive port for the flat file specifying the receive folder and inbound map. When I drop a test file into the receive folder, it disappears but TestTable1 is not loaded with any values.
What am I missing? Where does BizTalk place the test file? How can I set this up?
I have looked over a number of tutorials (https://msdn.microsoft.com/en-us/biztalk/gg153515, https://sandroaspbiztalkblog.wordpress.com/2012/08/20/biztalk-server-transform-text-files-flat-files-into-xml-a-practical-example-part-3/, ) but just don't understand what I am missing.
Any help will be greatly appreciated.
So I got it to work. Not sure how yet, but I tweaked the adapter settings and changed the value of Quantity in the input file from 1.000 to 1. I also now able to write to an xml file. The error that I was getting is as follows looks like it was having problems with "Quantity" data type:
The adapter failed to transmit message going to send port "ToSQL" with URL "mssql://.//TestDB?". It will be retransmitted after the retry interval specified for this Send Port. Details:"Microsoft.ServiceModel.Channels.Common.XmlReaderParsingException: The input data for the field/parameter "Quantity" is invalid according to the expected SqlDbType Int. ---> System.FormatException: Input string was not in a correct format.
at System.Number.StringToNumber(String str, NumberStyles options, NumberBuffer& number, NumberFormatInfo info, Boolean parseDecimal)
at System.Number.ParseInt32(String s, NumberStyles style, NumberFormatInfo info)
at Microsoft.Adapters.Sql.MetadataHelper.ConvertXmlValueToDotNetObject(String xmlString, String fieldParameterName, SqlDbType sqlDbType, Int32 maxLength, Int32 precision)
--- End of inner exception stack trace ---
Server stack trace:
at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)
at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.End(SendAsyncResult result)
at System.ServiceModel.Channels.ServiceChannel.EndCall(String action, Object[] outs, IAsyncResult result)
at System.ServiceModel.Channels.ServiceChannel.EndRequest(IAsyncResult result)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at System.ServiceModel.Channels.IRequestChannel.EndRequest(IAsyncResult result)
at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfClient`2.RequestCallback(IAsyncResult result)".

Based on your error:
Quantity" is invalid according to the expected SqlDbType Int.
'1.000' is not a valid int type.
you should convert it to a valid format such as "1"

Related

BizTalk Server 2016: WCF-SAP adapter

We have updated our environment to BizTalk 2016 and we are using WCF-SAP Send Port to invoke a SAP RFC.
We are receiving this error with NCo connection type. In BizTalk 2009 the same RFC with a Send Port WCF-SAP works perfect:
Error Description: Microsoft.ServiceModel.Channels.Common.MetadataException: RFC could not be resolved against SAP system because its metadata could not be obtained. ---> SAP.Middleware.Connector.RfcInvalidStateException: Lookup of function metadata failed for destination NCoConnection ---> SAP.Middleware.Connector.RfcCommunicationException: The only destination NCoConnection62.87.2.23P47PRUEBA002ESono_bztk available failed when retrieving metadata from -- see log for details
at SAP.Middleware.Connector.RfcRepository.findAlternateDestination(Int32& destIndex, Int32& alternativeDestinationsCount, Exception ex)
at SAP.Middleware.Connector.RfcRepository.Execute(RfcFunction function)
at SAP.Middleware.Connector.RfcRepository.LookupFunctionMetadataClassic(String name)
--- End of inner exception stack trace ---
at SAP.Middleware.Connector.RfcRepository.LookupFunctionMetadataClassic(String name)
at SAP.Middleware.Connector.RfcRepository.LookupFunctionMetadata(String name)
at SAP.Middleware.Connector.RfcRepository.GetFunctionMetadata(String name)
at Microsoft.Adapters.SAP.NCo.RfcClientConnection.GetRfcFunctionInterface(String rfcName)
at Microsoft.Adapters.SAP.InternalRfcMetadata.InitializeRfcFieldsNCo(String originalRfcName, SAPConnection sapConnection)
at Microsoft.Adapters.SAP.SAPMetadataContract.ResolveOperationMetadata(String operationId, TimeSpan timeout, TypeMetadataCollection& extraTypeMetadataResolved)
at Microsoft.ServiceModel.Channels.Common.Design.MetadataCache.GetOperationMetadata(String uniqueId, Guid clientId, TimeSpan timeout)
at Microsoft.Adapters.SAP.SapFunctionMetadataNCo.ResolveOperationMetadataUsingSdkNCo(String absoluteName, String displayName, String funcName, String operationNamespace, SAPConnection sapConnection, Boolean isTrfc, MetadataLookup metadataLookup, TimeoutHelper timeoutHelper)
--- End of inner exception stack trace ---
Any idea?
Verify that the SAP system is set to unicode for this RFC, as stated here, where the similar problem is solved.
It also doesn't hurt to generate the SAP schema again in Visual Studio, maybe there are some changes.

data type argument cannot be null in data profiling viewer

I am new to Data Profile Viewer. I just created a package in SSIS which contains a Data Profiling Task Component. I am using the Sample database of Adventureworks2012 and I am creating an XML file through it using Employee table. But when I am trying to view the XML in Data Profile Viewer and selecting the Candidate profile Key it is showing me the below error:
Drill down query completed with error: The error message is:
System.ArgumentNullException: 'dataType' argument cannot be null.
Parameter name: dataType
at System.Data.DataColumn..ctor(String columnName, Type dataType, String expr, MappingType type)
at Microsoft.SqlServer.DataProfileViewer.QueryDrillDownWorker.DoWork(DoWorkEventArgs e)
at Microsoft.SqlServer.DataProfileViewer.DrillDownControlContainer.backgroundWorker_DoWork(Object sender, DoWorkEventArgs e)
at System.ComponentModel.BackgroundWorker.OnDoWork(DoWorkEventArgs e)
at System.ComponentModel.BackgroundWorker.WorkerThreadStart(Object argument)
Anyone having a knowledge on this how to resolve it. I am not able to find on Google.
Thanks for your help.
Try to add Polybase feature to your instance.

Importing data from multi-value D3 database into SQL issues

Trying to use the mv.NET by bluefinity tools. Made some integration packages with it for importing data from a d3 multi-value database into MS SQL 2012 but seem to be having some trouble with the mapping.
For the VOYAGES table have some commentX fields in the D3 application that are acting quite unwieldy and the INSERT fails after a certain number of rows with the following message
>Error: 0xC0047062 at INSERT, mvNET Source[354]: System.Exception: Error #8: dataReader[0] = LTPAC002 ci.BufferColumnIndex = 52, ci.ColumnName = COMMGROUP(Error #8: dataReader[0] = LTPAC002 ci.BufferColumnIndex = 52, ci.ColumnName = COMMGROUP(The value is too large to fit in the column data area of the buffer.))
at mvNETDataSource.mvNETSource.PrimeOutput(Int32 outputs, Int32[] outputIDs, PipelineBuffer[] buffers)
at Microsoft.SqlServer.Dts.Pipeline.ManagedComponentHost.HostPrimeOutput(IDTSManagedComponentWrapper100 wrapper, Int32 outputs, Int32[] outputIDs, IDTSBuffer100[] buffers, IntPtr ppBufferWirePacket)
Error: 0xC0047038 at INSERT, SSIS.Pipeline: SSIS Error Code DTS_E_PRIMEOUTPUTFAILED.The PrimeOutput method on mvNET Source returned error code 0x80131500.The component returned a failure code when the pipeline engine called PrimeOutput().The meaning of the failure code is defined by the component, but the error is fatal and the pipeline stopped executing.There may be error messages posted before this with more information about the failure.
The value is too large to fit in the column data area of the buffer. -> tried changing the input / outputs types but can't seem to get it right.
In the SQL table the columns are of type ntext.
In the .dtsx job the data type for the columns are of type Unicode String [DT_WSTR] with length 4000 , I guess these are auto-detected.
The import worked for other D3 files like this not sure why it fails for these comment fields.
Running the query on the mv.NET Data Manager ( on the d3 server) times out after 240 seconds so maybe this is the underlying issue?
Any ideas how to proceed? Thank you ~
Most like reason is column COMMGROUP does not have correct data type or some record in source do not fit in output type
To find error record (causing) you have to use on redirect row (property of component failing component ) and get the result set in some txt.csv /or tsv file .
then check data
The exception is being thrown from mv.NET so I suggest you call (or ask your reseller) to call Bluefinity support and ask them about this. You're paying for support, might as well use it. Those programs shouldn't be allowed to throw exceptions like that.
D3 doesn't export Unicode, that might be one issue. But if the Data Manager times-out then I suspect something is wrong in the connectivity into D3. Open a Connection Monitor from the Session Monitor and watch the connection when you make the request. I'm guessing it's either hanging or more probably it's falling into BASIC Debug.
Make sure all D3-side programs related to this are either all Flash-compiled, or all Not Flashed. Your app code will fall into Debug if it's not Flashed but MVNET.BP is.
If it's your program that's in Debug, fix it. If you're not sure which program it is, LIST-RUNTIME-ERRORS in DM.
If it's a MVNET.BP program, again work with Bluefinity. If you are using MVSP for connectivity then the Connection Monitor may be useless, you'll need to change that to an IP (Telnet) connection to see the raw data exchange.

Unable to deploy .dacpac to SQL'16. Method Not found: set_Encryption

I have a dacpac file that is used from SQL 2005 to SQL 2016 databases. Our developers have been using SQL 2016 for the past few months, so we are now moving to production with it.
When attempting to upgrade a database (no previous dacpackage) using our .dacpac file, it will give this error below
===================================
Method not found: 'Void Microsoft.SqlServer.TransactSql.ScriptDom.AlterTableAlterColumnStatement.set_Encryption(Microsoft.SqlServer.TransactSql.ScriptDom.ColumnEncryptionDefinition)'. (Microsoft.Data.Tools.Schema.Sql)
------------------------------
Program Location:
at Microsoft.Data.Tools.Schema.Sql.Deployment.SqlDeploymentPlanGenerator.AlterStatementGenerator.GenerateForSimpleColumn(SqlSimpleColumn oldColumn, SqlSimpleColumn newColumn)
at Microsoft.Data.Tools.Schema.Sql.Deployment.SqlDeploymentPlanGenerator.AlterStatementGenerator.GenerateAlterTableForColumn(IModelElement oldElement, IModelElement newElement)
at Microsoft.Data.Tools.Schema.Sql.Deployment.SqlDeploymentPlanGenerator.AlterStatementGenerator.GenerateStatement(IModelElement oldElement, IModelElement newElement)
at Microsoft.Data.Tools.Schema.Sql.Deployment.SqlDeploymentPlanGenerator.Sql90AlterStatementGenerator.GenerateScriptDom(IModelElement oldElement, IModelElement newElement)
at Microsoft.Data.Tools.Schema.Sql.Deployment.SqlDeploymentPlanGenerator.DeploymentScriptDomGenerator.GenerateTableAlter(SqlTable sourceTable, SqlTable targetTable)
at Microsoft.Data.Tools.Schema.Sql.Deployment.SqlDeploymentPlanGenerator.DeploymentScriptDomGenerator.GenerateFragment(Int32 operation, IModelElement element)
at Microsoft.Data.Tools.Schema.Sql.Deployment.SqlDeploymentPlanGenerator.DeploymentScriptDomGenerator.GenerateSteps(Int32 operation, IModelElement element)
at Microsoft.Data.Tools.Schema.Sql.Deployment.OrderedStepGenerator.Add(DeploymentScriptDomGenerator scriptDom, IModelElement element, Int32 operation)
at Microsoft.Data.Tools.Schema.Sql.Deployment.Analyzers.PlanMediator.BuildDependencyOrderedSteps(Int32 operation, List`1 classOrder, List`1 operationOrder, Dictionary`2 changes, Boolean preserveGraphs, Dictionary`2& relating, Dictionary`2& related)
at Microsoft.Data.Tools.Schema.Sql.Deployment.Analyzers.PlanMediator.BuildOperations()
at Microsoft.Data.Tools.Schema.Sql.Deployment.SqlDeploymentPlanGenerator.OnGeneratePlan()
at Microsoft.Data.Tools.Schema.Sql.Deployment.SqlDeploymentPlanGenerator.GeneratePlan(List`1 drops)
at Microsoft.Data.Tools.Schema.Sql.Deployment.SqlDeployment.BuildPlan()
at Microsoft.Data.Tools.Schema.Sql.Deployment.SqlDeployment.Controller.CreatePlan()
at Microsoft.SqlServer.Dac.DacServices.<>c__DisplayClass25.<CreateDeploymentArtifactGenerationOperation>b__23(Object operation, CancellationToken token)
at Microsoft.SqlServer.Dac.Operation.Microsoft.SqlServer.Dac.IOperation.Run(OperationContext context)
at Microsoft.SqlServer.Dac.ReportMessageOperation.Microsoft.SqlServer.Dac.IOperation.Run(OperationContext context)
at Microsoft.SqlServer.Dac.OperationExtension.Execute(IOperation operation, DacLoggingContext loggingContext, CancellationToken cancellationToken)
at Microsoft.SqlServer.Dac.DacServices.GenerateDeployReport(DacPackage package, String targetDatabaseName, DacDeployOptions options, Nullable`1 cancellationToken)
at Microsoft.SqlServer.Management.Dac.DacWizard.ReviewPlanPage.ActionStepCalculator_DoWork(Object sender, DoWorkEventArgs e)
at System.ComponentModel.BackgroundWorker.OnDoWork(DoWorkEventArgs e)
at System.ComponentModel.BackgroundWorker.WorkerThreadStart(Object argument)
As far as I can tell, we don't have Encryption enabled at all on any of the tables. This is a fresh database. We've even tried recreating it from scratch, but we get the same error once we try to apply the dacpac file.
Does anyone know what I can do to get the server to recognize the set_Encryption method that it seems to be missing? Any suggestions are appreciated.
Edit: I've further determined that it is not a SQL 2016 issue, but rather a SQL Server Management Studio 2016 issue.
I was able to get around this error, by first running the dacpac through SqlPackage.exe. After that- I was able to publish through management studio.

Access Denied Error for a method in web service

I have a webservice that works perfectly from localhost but when I hosted it on the testing server I get an error on one of the methods. I am using wsHttp Binding. everything else works fine on the client side except for this method. This is the error that I am getting
client error:
System.ServiceModel.Security.SecurityAccessDeniedException: Access is denied.
Server stack trace:
at System.ServiceModel.Channels.ServiceChannel.ThrowIfFaultUnderstood(Message reply, MessageFault fault, String action, MessageVersion version, FaultConverter faultConverter)
at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
Server error:
namespace.Service Error: 10001 : Error occurred in methodname().
System.Security.SecurityException: The source was not found, but some or all event logs could not be searched. Inaccessible logs: Security.
at System.Diagnostics.EventLog.FindSourceRegistration(String source, String machineName, Boolean readOnly)
at System.Diagnostics.EventLog.SourceExists(String source, String machineName)
at System.Diagnostics.EventLog.VerifyAndCreateSource(String sourceName, String currentMachineName)
at System.Diagnostics.EventLog.WriteEvent(EventInstance instance, Byte[] data, Object[] values)
at System.Diagnostics.EventLog.WriteEvent(EventInstance instance, Object[] values)
at System.Diagnostics.EventLogTraceListener.TraceEvent(TraceEventCache eventCache, String source, TraceEventType severity, Int32 id, String format, Object[] args)
at System.Diagnostics.TraceSource.TraceEvent(TraceEventType eventType, Int32 id, String format, Object[] args)
at AutoWatch.Entity.WcfService.TrackingService.UpdateIncidentStatusHistory(Int64 incidentId, String status, String username, String comment, Boolean SuspectFaultyUnit) in C:\..servicename.cs:line 566
at AutoWatch.Entity.WcfService.TrackingService.GetNewIncidentMessage(String username) in C:\..servicename.cs:line 444
The Zone of the assembly that failed was:
MyComputer
I added in the error I am getting on the server.
Is it possible I am getting this error because the service cannot write to the event log?
Please help.
Ensure that account running your process where the service is hosted has access rights to the database. For example in case of IIS the account running the application pool where the service is hosted must have login to database server and it must have permissions to do all necessary operations in your database.
Edit:
The server stack trace looks quite straightforward. You have a problem with writing to Windows Event Log! It cannot find source you requested and it doesn't have permission to create it.
I got the same error when I tried to use the following statement in a Web Method of a WCF Service:
string myTypeName = typeof(ErrorHandlerBehavior).AssemblyQualifiedName;
where ErrorHandlerBehavior derives from BehaviorExtensionElement.
The line works well when I test the service on my localhost. On the web, the call of the containing method raises the Exception "Access is denied".
In this case, I guess that the cause was that my IP does not allow the creation of a BehaviourExtensionElement in a Partially Trusted environment (my service is in a shared hosting environment).
Finally, I succeded following the first mechanism described at https://learn.microsoft.com/en-us/dotnet/framework/wcf/extending/configuring-and-extending-the-runtime-with-behaviors about Service Behaviors:
"Using an attribute on the service class. When a ServiceHost is constructed, the ServiceHost implementation uses reflection to discover the set of attributes on the type of the service. If any of those attributes are implementations of IServiceBehavior, they are added to the behaviors collection on ServiceDescription. This allows those behaviors to participate in the construction of the service run time."
I just modified the derivation (no other modification required):
public class ErrorServiceBehavior : Attribute, IServiceBehavior
{ ... }
and use the class as Attribute of my service:
[ErrorServiceBehavior()]
public partial class MyService : IMyService
{...}
No other modification required. Check the original sample at How do I create a global exception handler for a WCF Services?.