I am using the Azure DevOps Migration Tools and it hits an exception when I try to migrate a work items with 35977 revisions.
Here is my configuration:
Here is the SOAP error I am getting:
Wow, that's a lot of revisions. I'm not sure I have ever seen a work item with that many revisions. I assume that you have some sort of tool that auto updates the work items, which is what is causing this.
Since it take about 200ms per revision to save Work Items I would
expect it to 5000 seconds (thats 82h) to migrate just 1 work item with
that many!
Since the Azure DevOps Migration tools use the old SOAP API (Object Model) this one is out of our hands. There may be some way to page the revisions but I am unaware. If you do find a way to only load partial works items i'd be intersted... although thinking about it I think there is a wi.LoadPartial() method... never used that.
To move forward you could add AND [System.Rev] < 25000 to your query to not load those work items that have that many revisions.
This would allow you to continue with the CLosed items that are supported.
We have added a way to only migrate some of the revisions when there are more, but we never envisaged in our wildest defensive coding strategies that it would be above 1000 revisions. I can imagine memory and other issues cropping up.
Added to: https://github.com/nkdAgility/azure-devops-migration-tools/issues/1096
Related
Could anyone provide any best practices about multiple migration runs? Moving from TFS 2017.3.1 to Azure DevOps Service. Dealing with a fair number of work items (32k). Of course, TSTU throttling is making the run take a long time, so I was thinking of pushing what I could up front, then a second pass to pick up the new work items since the first big push. So...enabling UpdateSourceReflectedId would set the ReflectedWorkItemId on the source items that have already been migrated. But what happens if someone changes a work item that has already been pushed? Would the history delta get picked up? How is that typically resolved...I was thinking maybe a Querybit like: ReflectedWorkItemId <> '' and ChangedDate > (last run time), but is that necessary? Those already exist on target...would ReplayRevisions pick up only the missing changes? TIA...
I usually do the following for large runs:
Open work items edited in last 90 days
Closed work items edited in last 90 days
open out to more days in chunks
The important thing to note is that links are created only when both ends of the link exist.
After a long run you can then rerun "edited in last month" to bring any changes a cross.
Changes to avoid in the Source:
changing work item type
moving work item between team project
We handle these, but loosly.
I'm working on a couple of million records, as soon as I try to run an advanced find, and put as a criteria a linked entity, the advanced find goes in timeout.
Create custom views on this allows me to filter properly? Anyone knows the proper way of using the advanced find this way? Are there limitations on the out of the box CRM that i should be aware of?
In CRM 2013 - it is possible to add indexes for specific fields by adding the columns to the quick find view for the entity.
You will need to wait for the Indexing Management Job to run (which is run every 24 hours by default) - see http://blogs.msdn.com/b/darrenliu/archive/2014/04/02/crm-2013-maintenance-jobs.aspx.
In previous version of CRM, it was necessary to add the indexes directly to the database - this may be necessary for more complex queries.
was too early to post an answer. The problem that I encountered was related to the OOB advanced find. Looking for example for an account with some related contacts (a really plain search with a linked entity) i had a SQL timeout. Everything was OOB so I was a little bit clueless and I opened a case to Microsoft. They found a bug, if i was changing the sorting the advanced find started to work again. They are still investigating. So wasn't a setting problem but a crm bug.
The title is pretty clear but again: How do I disable "activities" for a custom module on SugarCRM 6.5+ 7+
I have a module containing millions of records and activities has been slowing it down to a breaking point. I managed to stop the activities through some hacking (deleting entries from the cache folder) but I would like to know how to do it the right way so on repair&rebuild + etc things will be normal/ok.
//edit1:
I'm happy to completely disable activities for a limited period of time while my script runs and then enable it again right after if that is possible.
Well, I figured out how to disable activities (activity stream, known in the past as sugar feed I think).
As my problem was running a script on 100k records etc disabling the whole activity stream temporarily in the beginning of the script and then turning it back on in the end was sufficient.
It's quite simple and it feels like an embarrassment I didn't look into the activity stream's source before since in order to disable it a simple:
Activity::disable();
does the job and to turn it back on:
Activity::enable();
There is also a "blacklist" array in the source etc but 1- It didn't solve the problem and 2- It's clearly not upgrade safe etc.
After using youtrack for quite a while my organization is considering a move to JIRA (because of many reasons). However JIRA doesn't seem to include a youtrack importer/migration out of the box (though there seems to be plenty of importers/migrations the other way around).
Has anyone migrated from youtrack to JIRA and have any experience in this?
Edit:
To anyone who might have this problem later, my final solution ended up something like this:
transfer all "basic" data by hand (user accounts, basic project setup etc)
write a small C# program using the atlassian sdk and the youtrack sdk that transfers from one to the other (creating empty placeholder issues if issues was missing due to someone deleting them in youtrack in order to keep numbering).
This approach worked good enough and I managed to transfer pretty much all data without any loss of any very important data (though of course all timestamps are messed up now, but we saw that as an acceptable loss).
Important to know is that youtrack handles issues moved from one project to another a bit counter-intuitive (they still show up in their first project even when they're moved away from there, but they have an issue id from their new project - a slight wtf when I ran into that the first time).
Also, while the atlassian sdk did allow me to "spoof" the creator of an issue (that is, being logged in as used A and creating an issue while telling the system that it's actually user B who is creating this issue) it does not allow you to do this with comments. So in order to transfer those properly I had to actually loop through the comments and log in with the corresponding new user and post the comments.
Also, attachments from youtrack was a bit annoying to download, so I ended up having to download those "by hand". :/
But all in all, it was relatively pain-free. Some assembly required, some final touch-ups required, but it was all done within a couple of days.
I had the same problem. After a discussion with JIM (Jira Importer) developer, I used YouTrack Rest API and Python script to make JSON files. Then I used JIM JSON import.
With this solution you can import almost all fields from YT - the standard one and files with description, links between issues and projects and so on...
I don't know if I can push it to GitHub, I have to ask my boss - I did it during my work hours.... But of course you can ask me if you want.
The easiest approach is probably to export the data from youtrack into CSV and use the JIRA CSV importer. You may have to modify some of the data to fit the expected format for the CSV importer
Has anyone had luck with removing large amount of issues from a jira database instead of using the frontend? Deleting 60000 issues with the bulktools is not really feasible.
Last time I tried it, the jira went nuts because of its own way of doing indexes.
How about doing a backup to xml, editing the xml, and reimporting?
We got gutsy and did a truncate on the jiraissues table and then use the rebuild index feature on the frontend. It looks like it's working!
This is old, but I see that this question was just edited recently, so to chime in:
Writing directly to the JIRA database is problematic. The reindex feature suggested in the Oct 14 08 answer just rebuilds the Lucene index, so it is unlikely to clean up everything that needs to be cleaned up from the database on a modern JIRA instance. Off the top of my head, this will probably leave data lying around in the following tables, among others:
custom field data (customfieldvalue table)
issue links (issuelink table)
versions and components (nodeassociation table, which contains other stuff too, so be careful!)
remote issue links or wiki mentions (remotelink table)
If one has already done such a manual delete on production, it's always a good idea to run the database integrity checker (YOURJIRAURL/secure/admin/IntegrityChecker!default.jspa) to make sure that nothing got seriously broken.
Fast forwarding to 2014, the best solution is to write a quick shell script that uses the REST API to delete all of the required issues. (The JIRA CLI plugin is usually a good option for automating certain types of tasks too, but as far as I can tell, it does not currently support the deletion of issues, so the REST API is your best bet.)