In my Azure data factory pipeline, I'm using a Copy data activity inside a ForEach activity to copy files from an input container to an archive container before processing the files in the input container. This normally works, but today I made the archive container immutable by adding a legal hold policy to it, and the next time the copy data activity ran, it failed with an error (see below). Is there any way around this, since you should be able to add new files to an immutable container?
Error code: 2200
Failure type: User configuration issue
Details:
Failure happened on 'Sink' side. ErrorCode=AdlsGen2OperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=ADLS Gen2 operation failed for: Operation returned an invalid status code 'Conflict'. Account: 'mydatalake'. FileSystem: 'raw'. Path: 'Source/ABC/File_2021_03_24.csv'. ErrorCode: 'PathImmutableDueToLegalHold'. Message: 'This operation is not permitted as the path is immutable due to one or more legal holds.'. RequestId: '37f75e88-501a-0026-2fa1-20d52e000000'. TimeStamp: 'Wed, 24 Mar 2021 11:30:54 GMT'..,Source=Microsoft.DataTransfer.ClientLibrary,''Type=Microsoft.Azure.Storage.Data.Models.ErrorSchemaException,Message=Operation returned an invalid status code 'Conflict',Source=Microsoft.DataTransfer.ClientLibrary,'
Source: Pipeline LoadMyData
Related
Hello everyone and thanks for reading this problem.
I have a solution in C# using the zkemkeeper dll to get the records from some access control devices. When I "ping" them, there isn't any problem, but when I try to connect to them (Using my solution or the standalone demo to get attendance) I get the "-307" error with the "Unable to connect message". That's not very clear and I would really aprecciate if someone can explain what this error is (please!!!!). I would really like to understand these errors myself, so, where can I find all the definition of these errors?
In short:
1.- What is the problem regarding the "-307" error?
2.- Is there any place where all these errors are documented?
Thanks in advance!!
enter image description here
maybe you should check your device, you can ping them
Attention
The dwErrorCode parameter specifies the error code. The values are described as follows:
During connection, the following error codes may be returned:
0 Connected successfully
-1 Failed to invoke the interface
-2 Failed to initialize
-3 Failed to initialize parameters
-5 Data mode read error
-6 Wrong password
-7 Reply error
-8 Receive timeout
-307 Connection timeout
In invoking other interfaces, the following error codes may be returned:
-201 Device is busy
-199 New Mode
-103 device send back error of face version error
-102 face template version error, like 8.0 face template send to 7.0 device
-101 malloc memory failed
-100 Not supported or the data does not exist
-10 The length of transmitted data is incorrect
-5 Data already exists
-4 Insufficient space
-3 Wrong size
-2 File read/write error
-1 The SDK is not initialized and needs to be reconnected
0 Data not found or duplicate data
1 Correct operation
4 Parameter error
101 Buffer allocation error
102 repeat invoking
Underlying error codes:
-12001 Socket creation timeout (connection timeout)
-12002 Insufficient memory
-12003 Wrong Socket version
-12004 Not TCP protocol
-12005 Waiting timeout
-12006 Data transmission timeout
-12007 Data reading timeout
-12008 Failed to read Socket
-13009 Waiting event error
-13010 Exceeded retry attempts
-13011 Wrong reply ID
-13012 Checksum error
-13013 Waiting event timeout
-13014 DIRTY_DATA
-13015 Buffer size too small
-13016 Wrong data length
-13017 Invalid data read1
-13018 Invalid data read2
-13019 Invalid data read3
-13020 Data loss
-13021 Memory initialization error
-15001 Invoking return value of status key issued by SetShortkey interface repeatedly
-15002 Invoking return value of description issued by SetShortkey interface repeatedly
-15003 The two level menu is not opened in the device, and the data need not be issued
getdevicedata and setdevicedata invocation error codes
-15100 Error occurs in obtaining table structure
-15101 The condition field does not exist in the table structure
-15102 Inconsistency in the total number of fields
-15103 Inconsistency in sorting fields
-15104 Memory allocation error
-15105 Data parsing error
-15106 Data overflow as the transmitted data exceeds 4M
-15108 Invalid options
-15113 Data parsing error: table ID not found
-15114 A data exception is returned as the number of fields is smaller than or equal to 0
-15115 A data exception is returned as the total number of table fields is inconsistent with the
total number of fields of the data
Firmware error codes:
2000 Return OK to execute
-2001 Return Fail to execute command
-2002 Return Data
-2003 Regstered event occorred
-2004 Return REPEAT Command
-2005 Return UNAUTH Command
0xffff Return Unknown Command
-4999 Device parameter read error
-4998 Device parameter write error
-4997 The length of the data sent by the software to the device is incorrect
228
229
-4996 A parameter error exists in the data sent by the software to the device
-4995 Failed to add data to the database
-4994 Failed to update the database
-4993 Failed to read data from the database
-4992 Failed to delete data in the database
-4991 Data not found in the database
-4990 The data amount in the database reaches the limit
-4989 Failed to allocate memory to a session
-4988 Insufficient space in the memory allocated to a session
-4987 The memory allocated to a session overflows
-4986 File does not exist
-4985 File read failure
-4984 File write failure
-4983 Failed to calculate the hash value
-4982 Failed to allocate memory
Note
This interface is applicable to the new architecture firmware.
I am currently working on error handling in ADF.
I know how to get error from a particular activity by using # activity('').error.message and then pass it to database.
Do you guys know a generic way that able us to catch any error on any activity on a pipeline?
Best regards!
You can leverage the flow path dependency aspect within Azure data factory to manage logging of error based on single activity rather than duplicating same activities :
The below blog : https://datasharkx.wordpress.com/2021/08/19/error-logging-and-the-art-of-avoiding-redundant-activities-in-azure-data-factory/ explains all the same in details
Below are the basic principles that need to be followed :
Multiple dependencies with the same source are OR’ed together.
Multiple dependencies with different sources are AND’ed together.
Together this looks like (Act_3fails OR Act_3Skipped) AND (Act_2Completes OR Act_2skipped)
When you have certain number of activities in pipeline1, and you want to capture any error message when one of the activities fail, then using execute pipeline activity is the correct approach.
When you use execute pipeline activity to trigger pipeline1, when any activity inside pipeline1 fails, it throws an error message which includes the name of the activity which failed.
Look at the following demonstration. I have a pipeline named p1 which has 3 activities get metadata, for each and Script.
I am triggering p1 pipeline in p2 pipeline using execute pipeline activity and storing the error message in a variable using set variable activity. For demonstration, I made sure each activity throws an error each time (3 times).
When the get metadata activity fails, the respective error message captured in p2 pipeline will be as follows:
Operation on target Get Metadata1 failed: The required Blob is missing. ContainerName: data2, path: data2/files/.
When only for each activity in p1 pipeline fails, the respective error message will be captured in p2 pipeline.
Operation on target ForEach1 failed: Activity failed because an inner activity failed
When only script activity in p1 pipeline fails, the respective error message will be captured in p2 pipeline.
Operation on target Script1 failed: Invalid object name 'mydemotable'
So, using Execute pipeline activity (to execute pipeline1) would help you to capture the error message from any different activities that fail inside the required pipeline.
I am getting the following error in a pipeline that has Copy activity with Rest API as source and Azure Data Lake Storage Gen 2 as Sink.
"message": "Failure happened on 'Sink' side. ErrorCode=AdlsGen2OperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=ADLS Gen2 operation failed for: Operation returned an invalid status code 'Conflict'. Account: '{Storage Account Name}'. FileSystem: '{Container Name}'. Path: 'foodics_v2/Burgerizzr/transactional/_567a2g7a/2018-02-09/raw/inventory-transactions.json'. ErrorCode: 'LeaseAlreadyPresent'. Message: 'There is already a lease present.'. RequestId: 'd27f1a3d-d01f-0003-28fb-400303000000'..,Source=Microsoft.DataTransfer.ClientLibrary,''Type=Microsoft.Azure.Storage.Data.Models.ErrorSchemaException,Message=Operation returned an invalid status code 'Conflict',Source=Microsoft.DataTransfer.ClientLibrary,'",
The pipeline runs in a for loop with Batch size = 5. When I make it sequential, the error goes away, but I need to run it in parallel.
This is known issue with adf limitation variable thread parallel running.
You probably trying to rename filename using variable.
Your option is to run another child looping after each variable execution.
i.e. variable -> Execute Pipeline
enter image description here
or
remove those variable, hard coded those variable expression in azure activity.
enter image description here
Hope this helps
I start Geode with a disk store using the example config. Then if I stop the server and start it again I get:
Exception in thread "main" java.lang.IllegalStateException: Detected multiple disk store initialization files named "BACKUPDeal.if". This disk store directories must only contain one initialization file.
If I remove the if file, then the error is: Exception in thread "main" java.lang.IllegalStateException: The init file "D:\deal\BACKUPdeal.if" does not exist. If it no longer exists then delete the following files to be able to create this disk store. Existing oplogs are: [D:\deal\BACKUPdeal_1.drf, D:\deal\BACKUPdeal_1.crf]
What am I missing here?
The example config uses 2 disk stores and I had renamed these to the same store, therefore it couldn't initialize 2 disk stores out of 1. To solve it I removed the 2nd disk store from the config like so:
<disk-store name="deal"
compaction-threshold="40"
auto-compact="false"
allow-force-compaction="true"
max-oplog-size="512"
queue-size="10000"
time-interval="15"
write-buffer-size="65536"
disk-usage-warning-percentage="80"
disk-usage-critical-percentage="98">
<disk-dirs>
<disk-dir>d:\deal</disk-dir>
</disk-dirs>
</disk-store>
I have an SSIS package that is scheduled to run every weekday morning at 8:15. It copies data to and from Active Directory and SQL. About two weeks ago, it started failing, with no changes having been made to the server (beyond MS updates).
The funny thing is that if I then immediately run the package again, it succeeds. Here is the error text from when it fails:
Date 7/14/2011 8:15:00 AM
Log Job History (Reference: Active Directory)
Step ID 1
Server MMCI-GD1SQL2
Job Name Reference: Active Directory
Step Name Run Package
Duration 00:00:32
Sql Severity 0
Sql Message ID 0
Operator Emailed
Operator Net sent
Operator Paged
Retries Attempted 0
Message
Executed as user: MMCI\service-sql. Microsoft (R) SQL Server Execute Package Utility Version 10.0.1600.22 for 32-bit Copyright (C) Microsoft Corp 1984-2005. All rights reserved.
Started: 8:15:00 AM Error: 2011-07-14 08:15:31.88
Code: 0xC0047062
Source: Synchronize Permissions Active Directory Permissions [133]
Description: System.DirectoryServices.AccountManagement.PrincipalOperationException: There is no such object on the server. ---> System.DirectoryServices.DirectoryServicesCOMException (0x80072030): There is no such object on the server.
at System.DirectoryServices.DirectoryEntry.Bind(Boolean throwIfFail)
at System.DirectoryServices.DirectoryEntry.Bind()
at System.DirectoryServices.DirectoryEntry.RefreshCache()
at System.DirectoryServices.AccountManagement.ADStoreCtx.LoadDirectoryEntryAttributes(DirectoryEntry de)
--- End of inner exception stack trace ---
at Microsoft.SqlServer.Dts.Pipeline.ScriptComponentHost.HandleUserException(Exception e)
at Microsoft.SqlServer.Dts.Pipeline.ScriptComponentHost.PrimeOutput(Int32 outputs, Int32[] outputIDs, PipelineBuffer[] buffers)
at Microsoft.SqlServer.Dts.Pipeline.ManagedComponentHost.HostPrimeOutput(IDTSManagedComponentWrapper100 wrapper, Int32 outputs, Int32[] outputIDs, IDTSBuffer100[] buffers, IntPtr ppBufferWirePacket) End Error Error: 2011-07-14 08:15:31.90
Code: 0xC0047038
Source: Synchronize Permissions SSIS.Pipeline
Description: SSIS Error Code DTS_E_PRIMEOUTPUTFAILED. The PrimeOutput method on component "Active Directory Permissions" (133) returned error code 0x80131501. The component returned a failure code when the pipeline engine called PrimeOutput(). The meaning of the failure code is defined by the component, but the error is fatal and the pipeline stopped executing. There may be error messages posted before this with more information about the failure. End Error DTExec: The package execution returned DTSER_FAILURE (1). Started: 8:15:00 AM Finished: 8:15:31 AM Elapsed: 31.343 seconds. The package execution failed. The step failed.
Any thoughts?
Has some new Group Policy been applied that changed the permissions for the account your automated run uses, but which doesn't apply to your user id? I'm assuming when you say "I then ... run the package", you mean your logged-in user id.
Based on the error message that you had provided, the issue seems to be that the task within your package is trying to query an object in Active Directory that might no longer exist.
System.DirectoryServices.AccountManagement.PrincipalOperationException:
There is no such object on the server. --->
System.DirectoryServices.DirectoryServicesCOMException (0x80072030):
There is no such object on the server.
I could be wrong on the below part. I am just speculating what your package might be doing based on the description provided.
Since your package synchronizes data between SQL Server and Active Directory, I assume that the task named Synchronize Permissions Active Directory Permissions selects some form of data stored in SQL Server and updates the content in Active Directory or vice versa. If my assumption is correct, this task is probably Script Task or Script Component. I believe that the code inside this component is failing to select an object (group/user) in Active Directory.
I would check whether a group/user was deleted in Active Directory on the days prior to when the package failed to run.
Hope this helps.