Migrating WorkItems that have no longer existing 'Team Project' references in their history fails - azure-devops-migration-tools

While migrating a project across two distinct Azure DevOps Organizations with the following config for the processor:
{
"ObjectType": "VstsSyncMigrator.Engine.Configuration.Processing.WorkItemMigrationConfig",
"Enabled": true,
"PrefixProjectToNodes": false,
"UpdateCreatedDate": true,
"UpdateCreatedBy": true,
"AppendMigrationToolSignatureFooter": false,
"LinkMigration": true,
"AttachmentMigration": true,
"AttachmentMazSize": 480000000,
"AttachmentWorkingPath": "c:\\temp\\WorkItemAttachmentWorkingFolder\\",
"FixHtmlAttachmentLinks": true,
"SkipToFinalRevisedWorkItemType": false,
"WorkItemCreateRetryLimit": 5,
"PauseAfterEachWorkItem": false,
"FilterWorkItemsThatAlreadyExistInTarget": false,
"BuildFieldTable": false,
"QueryBit": "AND [System.WorkItemType] NOT IN ('Test Suite', 'Test Plan') ",
"OrderBit": "[System.ChangedDate] desc",
"ReplayRevisions": true,
"CollapseRevisions": false
}
The process fails in the items that, in any of their revisions, reference Team Projects that no longer exist - those items were moved across projects within the same Organization in the past, and the originating project was in the meanwhile deleted.
[ User Story][Complete: 18/3284][sid:1471 |Rev:64 ][tid:null | Microsoft.TeamFoundation.WorkItemTracking.Client.DeniedOrNotExistException: TF26192: The team project specified by the ID 129 does not exist. Check the team project ID and try again.
at Microsoft.TeamFoundation.WorkItemTracking.Client.ProjectCollection.GetById(Int32 projectId)
at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItem.CheckUpdateCachedData(Boolean projectChanged, Boolean typeChanged)
at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItem.get_Type()
at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItemFieldData.Microsoft.TeamFoundation.WorkItemTracking.Internals.IWorkItemOpenFieldDataHelper.SetLatestData(Dictionary`2 latestData)
at Microsoft.TeamFoundation.WorkItemTracking.Internals.WorkItemHelper.LoadWorkItemFieldData(IRowSetCollectionHelper tables, IWorkItemOpenFieldDataHelper helper)
at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItem.LoadWorkItemFromRowSetInternal(Int32 rev, Nullable`1 asof, IWorkItemRowSets witem)
at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItem..ctor(WorkItemStore store, Int32 id, Int32 revision)
at VstsSyncMigrator.Engine.WorkItemStoreContext.GetRevision(WorkItem workItem, Int32 revision) in D:\a\1\s\src\VstsSyncMigrator.Core\Execution\ComponentContext\WorkItemStoreContext.cs:line 202
at VstsSyncMigrator.Engine.WorkItemMigrationContext.ReplayRevisions(List`1 revisionsToMigrate, WorkItem sourceWorkItem, WorkItem targetWorkItem, Project destProject, WorkItemStoreContext sourceStore, Int32 current, WorkItemStoreContext targetStore) in D:\a\1\s\src\VstsSyncMigrator.Core\Execution\MigrationContext\WorkItemMigrationContext.cs:line 361
Is there any way to overcome this while maintaining the full revision history?
The issue is overcome if I migrate only the tip revisions.

Related

Unable to successfully migrate a project: TF201077: The work item type cannot be found

I've successfully migrated both stock and bespoke projects but I'm having trouble with one project I can't move based on the same config file (so it's probably not the config).
It looks like it's unable to save the work item. I've tried project to project in the same org using the same process, org to org, and removing any iterations. I just get the same error.
The work items I'm migrating are extremely basic tasks/stories with no bespoke fields populated.
I'm getting the following:
[11:33:24 ERR] Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItemTypeDeniedOrNotExistException: TF201077: The work item type cannot be found. It may have been renamed or destroyed.
at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItem.get_Type()
at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItemFieldData.Microsoft.TeamFoundation.WorkItemTracking.Internals.IWorkItemOpenFieldDataHelper.SetLatestData(Dictionary`2 latestData)
at Microsoft.TeamFoundation.WorkItemTracking.Internals.WorkItemHelper.LoadWorkItemFieldData(IRowSetCollectionHelper tables, IWorkItemOpenFieldDataHelper helper)
at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItem.LoadWorkItemFromRowSetInternal(Int32 rev, Nullable`1 asof, IWorkItemRowSets witem)
at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItem..ctor(WorkItemStore store, Int32 id, Int32 revision)
at MigrationTools.TfsExtensions.GetRevision(WorkItemData context, Int32 rev) in D:\a\1\s\src\MigrationTools.Clients.AzureDevops.ObjectModel\TfsExtensions.cs:line 35
at VstsSyncMigrator.Engine.WorkItemMigrationContext.ReplayRevisions(List`1 revisionsToMigrate, WorkItemData sourceWorkItem, WorkItemData targetWorkItem) in D:\a\1\s\src\VstsSyncMigrator.Core\Execution\MigrationContext\WorkItemMigrationContext.cs:line 600
and at the end:
[11:33:25 ERR] Microsoft.TeamFoundation.WorkItemTracking.Client.ValidationException: TF237124: Work Item is not ready to save
at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItem.Save(SaveFlags saveFlags)
at MigrationTools.TfsExtensions.SaveToAzureDevOps(WorkItemData context) in D:\a\1\s\src\MigrationTools.Clients.AzureDevops.ObjectModel\TfsExtensions.cs:line 76
at VstsSyncMigrator.Engine.WorkItemMigrationContext.ReplayRevisions(List`1 revisionsToMigrate, WorkItemData sourceWorkItem, WorkItemData targetWorkItem) in D:\a\1\s\src\VstsSyncMigrator.Core\Execution\MigrationContext\WorkItemMigrationContext.cs:line 643
[11:33:25 INF] ===============================================================
[11:33:25 WRN] Work Item is not ready to save as it has some invalid fields. This may not result in an error. Enable LogLevel as 'Debug' in the config to see more.
[11:33:25 ERR] Microsoft.TeamFoundation.WorkItemTracking.Client.ValidationException: TF237124: Work Item is not ready to save
at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItem.Save(SaveFlags saveFlags)
at MigrationTools.TfsExtensions.SaveToAzureDevOps(WorkItemData context) in D:\a\1\s\src\MigrationTools.Clients.AzureDevops.ObjectModel\TfsExtensions.cs:line 76
at VstsSyncMigrator.Engine.WorkItemMigrationContext.<ProcessWorkItemAsync>d__35.MoveNext() in D:\a\1\s\src\VstsSyncMigrator.Core\Execution\MigrationContext\WorkItemMigrationContext.cs:line 486
Microsoft.TeamFoundation.WorkItemTracking.Client.ValidationException: TF237124: Work Item is not ready to save
at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItem.Save(SaveFlags saveFlags)
at MigrationTools.TfsExtensions.SaveToAzureDevOps(WorkItemData context) in D:\a\1\s\src\MigrationTools.Clients.AzureDevops.ObjectModel\TfsExtensions.cs:line 76
at VstsSyncMigrator.Engine.WorkItemMigrationContext.<ProcessWorkItemAsync>d__35.MoveNext() in D:\a\1\s\src\VstsSyncMigrator.Core\Execution\MigrationContext\WorkItemMigrationContext.cs:line 486
[11:33:25 ERR] Could not save migrated work item 35, an exception occurred.
System.AggregateException: One or more errors occurred. ---> Microsoft.TeamFoundation.WorkItemTracking.Client.ValidationException: TF237124: Work Item is not ready to save
at VstsSyncMigrator.Engine.WorkItemMigrationContext.<ProcessWorkItemAsync>d__35.MoveNext() in D:\a\1\s\src\VstsSyncMigrator.Core\Execution\MigrationContext\WorkItemMigrationContext.cs:line 531
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at VstsSyncMigrator.Engine.WorkItemMigrationContext.InternalExecute() in D:\a\1\s\src\VstsSyncMigrator.Core\Execution\MigrationContext\WorkItemMigrationContext.cs:line 205
---> (Inner Exception #0) Microsoft.TeamFoundation.WorkItemTracking.Client.ValidationException: TF237124: Work Item is not ready to save
at VstsSyncMigrator.Engine.WorkItemMigrationContext.<ProcessWorkItemAsync>d__35.MoveNext() in D:\a\1\s\src\VstsSyncMigrator.Core\Execution\MigrationContext\WorkItemMigrationContext.cs:line 531<---
[11:33:25 WRN] The following items could not be migrated: 35
[11:33:25 INF] DONE in 00:00:05.6093139
[11:33:25 FTL] Error while running WorkItemMigration
System.AggregateException: One or more errors occurred. ---> Microsoft.TeamFoundation.WorkItemTracking.Client.ValidationException: TF237124: Work Item is not ready to save
at VstsSyncMigrator.Engine.WorkItemMigrationContext.<ProcessWorkItemAsync>d__35.MoveNext() in D:\a\1\s\src\VstsSyncMigrator.Core\Execution\MigrationContext\WorkItemMigrationContext.cs:line 531
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at VstsSyncMigrator.Engine.WorkItemMigrationContext.InternalExecute() in D:\a\1\s\src\VstsSyncMigrator.Core\Execution\MigrationContext\WorkItemMigrationContext.cs:line 225
at MigrationTools._EngineV1.Processors.MigrationProcessorBase.Execute() in D:\a\1\s\src\MigrationTools\_EngineV1\Processors\MigrationProcessorBase.cs:line 47
---> (Inner Exception #0) Microsoft.TeamFoundation.WorkItemTracking.Client.ValidationException: TF237124: Work Item is not ready to save
at VstsSyncMigrator.Engine.WorkItemMigrationContext.<ProcessWorkItemAsync>d__35.MoveNext() in D:\a\1\s\src\VstsSyncMigrator.Core\Execution\MigrationContext\WorkItemMigrationContext.cs:line 531<---
[11:33:25 ERR] WorkItemMigration The Processor MigrationEngine entered the failed state...stopping run
[11:33:25 INF] Application is shutting down...
[11:33:25 INF] Terminating: Application forcebly closed.
[11:33:25 INF] Application Ending
[11:33:25 INF] The application ran in 00:00:10.7989077 and finished at 07/27/2022 11:33:25
The config is:
{
"ChangeSetMappingFile": null,
"Source": {
"$type": "TfsTeamProjectConfig",
"Collection": "https://dev.azure.com/XXXX/",
"Project": "XXXX",
"ReflectedWorkItemIDFieldName": "Custom.ReflectedWorkItemId",
"AllowCrossProjectLinking": false,
"AuthenticationMode": "AccessToken",
"PersonalAccessToken": "XXXXX",
"PersonalAccessTokenVariableName": "",
"LanguageMaps": {
"AreaPath": "Area",
"IterationPath": "Iteration"
}
},
"Target": {
"$type": "TfsTeamProjectConfig",
"Collection": "https://dev.azure.com/XXXX/",
"Project": "XXXX",
"ReflectedWorkItemIDFieldName": "Custom.ReflectedWorkItemId",
"AllowCrossProjectLinking": false,
"AuthenticationMode": "AccessToken",
"PersonalAccessToken": "XXXX",
"PersonalAccessTokenVariableName": "",
"LanguageMaps": {
"AreaPath": "Area",
"IterationPath": "Iteration"
}
},
"FieldMaps": [
],
"GitRepoMapping": null,
"LogLevel": "Information",
"CommonEnrichersConfig": null,
"Processors": [
{
"$type": "WorkItemMigrationConfig",
"Enabled": true,
"ReplayRevisions": true,
"PrefixProjectToNodes": false,
"UpdateCreatedDate": true,
"UpdateCreatedBy": true,
"WIQLQueryBit": "AND [Microsoft.VSTS.Common.ClosedDate] = '' AND [System.WorkItemType] NOT IN ('Test Suite', 'Test Plan','Shared Steps','Shared Parameter','Feedback Request')",
"WIQLOrderBit": "[System.ChangedDate] desc",
"LinkMigration": true,
"AttachmentMigration": true,
"AttachmentWorkingPath": "c:\\temp\\WorkItemAttachmentWorkingFolder\\",
"FixHtmlAttachmentLinks": false,
"SkipToFinalRevisedWorkItemType": true,
"WorkItemCreateRetryLimit": 5,
"FilterWorkItemsThatAlreadyExistInTarget": true,
"PauseAfterEachWorkItem": false,
"AttachmentMaxSize": 480000000,
"AttachRevisionHistory": false,
"LinkMigrationSaveEachAsAdded": false,
"GenerateMigrationComment": true,
"WorkItemIDs": null,
"MaxRevisions": 0,
"NodeStructureEnricherEnabled": null,
"UseCommonNodeStructureEnricherConfig": false,
"NodeBasePaths": [
"Product\\Area\\Path1",
"Product\\Area\\Path2"
],
"AreaMaps": {},
"IterationMaps": {},
"MaxGracefulFailures": 0
}
],
"Version": "12.0",
"workaroundForQuerySOAPBugEnabled": false,
"WorkItemTypeDefinition": {
"sourceWorkItemTypeName": "targetWorkItemTypeName"
},
"Endpoints": {
"InMemoryWorkItemEndpoints": [
{
"Name": "Source",
"EndpointEnrichers": null
},
{
"Name": "Target",
"EndpointEnrichers": null
}
]
}
}
Posting an answer in case anyone else had the same problem.
My issue was in the config file after all. It had example data under the node base paths:
"NodeBasePaths": [
"Product\\Area\\Path1",
"Product\\Area\\Path2"
],
and that prevented the node mapping which caused the save to fail and produced the error. Simply removing all data in that section if I'm not using it solved the problem.

Unable to migrate teams and their members associated with a project in ADO using azure-devops-migration-tools

what configuration do i need to use to migrate the teams and members of the team associated with a project from one org to another.
Below is the configuration i tried but it didn't seem to work.
{
"$type": "TeamMigrationConfig",
"Enabled": true,
"PrefixProjectToNodes": false,
"EnableTeamSettingsMigration": true,
"FixTeamSettingsForExistingTeams": false
}

Azure DevOps Server: Migrating Test Cases - Expecting end of string error

I am attempting my first migration between two different collections on Azure DevOps Server 2019.
The new collection has a custom inheritance process model.
I am trying to migrate test cases only for now. I would like to migrate test cases, test suites, and test plans.
I have added the configuration as per my understanding, but the migration keeps on failing.
Error message:
migration.exe Warning: 0 : [EXCEPTION] Microsoft.TeamFoundation.WorkItemTracking.Client.ValidationException: Expecting end of string. The error is caused by «BY».
at Microsoft.TeamFoundation.WorkItemTracking.Client.Query.Initialize(WorkItemStore store, String wiql, IDictionary context, Int32[] ids, Int32[] revs, Boolean dayPrecision)
Redacted Processor from the config file below.
Thanks!
"Processors": [
{
"ObjectType": "VstsSyncMigrator.Engine.Configuration.Processing.NodeStructuresMigrationConfig",
"PrefixProjectToNodes": false,
"Enabled": false,
"BasePaths": [
"****\\Market Regulation\\Market Surveillance - Bonds",
"****\\Trading Value Stream"
]
},
{
"ObjectType": "VstsSyncMigrator.Engine.Configuration.Processing.WorkItemMigrationConfig",
"ReplayRevisions": true,
"PrefixProjectToNodes": false,
"UpdateCreatedDate": true,
"UpdateCreatedBy": true,
"UpdateSourceReflectedId": false,
"BuildFieldTable": false,
"AppendMigrationToolSignatureFooter": false,
"QueryBit": "AND [System.AreaPath] = '****\\Market Regulation\\Market Surveillance – Bonds' AND [System.WorkItemType] = 'Test Case' ",
"OrderBit": "ORDER BY [System.Id]",
"Enabled": true,
"LinkMigration": true,
"AttachmentMigration": true,
"AttachmentWorkingPath": "c:\\temp\\WorkItemAttachmentWorkingFolder\\",
"FixHtmlAttachmentLinks": false,
"SkipToFinalRevisedWorkItemType": false,
"WorkItemCreateRetryLimit": 5,
"FilterWorkItemsThatAlreadyExistInTarget": true,
"PauseAfterEachWorkItem": false,
"AttachmentMazSize": 480000000,
"CollapseRevisions": false
}
]
I believe your "OrderBit" syntax is wrong, it should look like this:
"OrderBit": "[System.ChangedDate] desc"
Just adapt it to your situation.

During migration commit links are "Fixed", "Removed" but not "Added"

When I run migration in azure-devops-migration-tools, there is Fixing, Removing of commit links, but no Adding.
As a result commit links do not get migrated.
However it doesn't reproduce in every project - in most projects this works fine and commits also get added. For example:
My WorkItemMigrationConfig:
{
"ObjectType": "VstsSyncMigrator.Engine.Configuration.Processing.WorkItemMigrationConfig",
"ReplayRevisions": true,
"PrefixProjectToNodes": false,
"UpdateCreatedDate": true,
"UpdateCreatedBy": true,
"UpdateSourceReflectedId": false,
"BuildFieldTable": false,
"AppendMigrationToolSignatureFooter": false,
"QueryBit": "AND [System.ID] = 41128",
"OrderBit": "[System.ChangedDate] desc",
"Enabled": true,
"LinkMigration": true,
"AttachmentMigration": true,
"AttachmentWorkingPath": "c:\\temp\\WorkItemAttachmentWorkingFolder\\",
"FixHtmlAttachmentLinks": false,
"SkipToFinalRevisedWorkItemType": false,
"WorkItemCreateRetryLimit": 5,
"FilterWorkItemsThatAlreadyExistInTarget": true,
"PauseAfterEachWorkItem": false,
"AttachmentMazSize": 480000000,
"CollapseRevisions": false
}
I suspect this may be project specific, but currently I have no idea what is causing the issue. What may be the reason for this?
//edit
After some research in the tool's source code I have found that commit links are only added if commit link URIs are different (https://github.com/nkdAgility/azure-devops-migration-tools/blob/9ef6ee4fd863de30d8a2179450bc86cb5cfafeb5/src/VstsSyncMigrator.Core/Execution/OMatics/RepoOMatic.cs#L137)
In my case these links are the same because TFS / AzureDevops project IDs are the same, as destination project is a result of cloning source project's collection.
In order for this to work the Git repository must be in the target first! If you have changes the name of the Git Repository you must add it to the mapping.
FixGitCommitLinks - Allows you to fix the migrated Git commit hooks (and
thus external links) to point to the new repository in the target
project. If the source and target repository names are the same, this
will work out of the box. If the target repository has a different
name, you can specify that name via the “TargetRepository” property
https://nkdagility.github.io/azure-devops-migration-tools/
This is slightly out of date, you need to use a "GitRepoMapping" similar as the "WorkItemTypeDefinition" element in the yaml that maps all of the old names to the new names. Again it is only needed when you use a different name from the source.

Storing account settings in a single row with complex data

I need to store account settings for each account profile. I decided to use SQL DB for this, but not sure should I go with complex data (json/xml).
I found answers
Using a Single Row configuration table in SQL Server database. Bad idea?
https://softwareengineering.stackexchange.com/questions/163606/configuration-data-single-row-table-vs-name-value-pair-table
but none of them discusses using single row approach containing complex data
Complex data would be stored inside a following DB table
AccountID int
AccountSettings nvarchar(max)
and would contain AccountSettings data such as
"settings": {
"branding": {
"header_color": "1A00C3",
"page_background_color": "333333",
"tab_background_color": "3915A2",
"text_color": "FFFFFF",
"header_logo_url": "/path/to/header_logo.png",
"favicon_url": "/path/to/favicon.png",
},
"apps": {
"use": true,
"create_private": false,
"create_public": true
},
"tickets": {
"comments_public_by_default": true,
"list_newest_comments_first": true,
"collaboration": true,
"private_attachments": true,
"agent_collision": true
"list_empty_views": true,
"maximum_personal_views_to_list": 12,
"tagging": true,
"markdown_ticket_comments": false
},
"chat": {
"maximum_request_count": 5,
"welcome_message": "Hello, how may I help you?",
"enabled": true
},
"voice": {
"enabled": true,
"maintenance": false,
"logging": true
},
"twitter": {
"shorten_url": "optional"
},
"users": {
"tagging": true,
"time_zone_selection": true,
"language_selection": true
},
"billing": {
"backend": 'internal'
},
"brands": {
"default_brand_id": 47
},
"active_features": {
"on_hold_status": true,
"user_tagging": true,
"ticket_tagging": true,
"topic_suggestion": true,
"voice": true,
"business_hours": true,
"facebook_login": true,
"google_login": true,
"twitter_login": true,
"forum_analytics": true,
"agent_forwarding": true,
"chat": true,
"chat_about_my_ticket": true,
"customer_satisfaction": true,
"csat_reason_code": true,
"screencasts": true,
"markdown": true,
"language_detection": true,
"bcc_archiving": true,
"allow_ccs": true,
"advanced_analytics": true,
"sandbox": true,
"suspended_ticket_notification": true,
"twitter": true,
"facebook": true,
"feedback_tabs": true,
"dynamic_contents": true,
"light_agents": true
},
"ticket_sharing_partners": [
"foo#example.com"
]
}
Other solution is a widely used single row approach such as
AccountID int
SettingsName nvarchar(max)
SettingsValue nvarchar(max)
that could hold data such as
AccountID SettingsName SettingsValue
1 Branding.Header_Color 1A00C3
1 Branding.Page_Background_Color 333333
1 Apps.Use true
......
I assume both solutions are valid and would depend based on application needs, but would REALLY like to know is there an issue that i am not seeing when using complex data with single row approach to store application settings?
One concern is if you have a busy large database, you cannot do ONLINE re-indexing (if you have the Enterprise version) on XML, text, varchar(max) type fields. This causes grief for 2008 R2. The newer versions of MS SQL Server can re-index online varchar(max) fields, so it depends on which version you are running.
Also, you won't be able to query or index if you need to search for specific records, unless you go with the SettingsName, SettingsValue type of table, which I've used a lot. This solves the re-index online issue (if that applies to your situation), as well as indexing the fields for quick queries.