MIGRATION: Retain issue links

[toddfarmer]

ASF Jira provides the ability to link two Jira issues as related. In some cases - notably sub-tasks - those relationships are essential to meaningfully track work that is a component of a larger collection of work. We need to assess how best to express linked issues once migrated to GitHub issues, orchestrate this as part of migration (e.g., it's hard to include details about a GitHub child issue while migrating the parent issue if the child has not yet been migrated itself). It's relatively trivial to point back at the source ASF Jira issue (or issues) and force users wanting to navigate those relationships to do so via ASF Jira records. That's not very efficient or friendly, but may be sufficient as an associated cost of a one-time migration.

[assignUser]

Issues support checklist that are tracked in the issue header and visible from the issue overview. You just need to create a list of check boxes with - [ ] #issue_id

The sub-tasks are automatically linked to the main issue with a "tracked in" label

Additionally it also tracks completion status and displays a marker for the type of completion.

What was not possible with JIRA is possible with Issues:

Establishing the links via github issue id will need to happen in a two-pass process with the second pass removing placeholder ids and looking up the correct ids and editing them in or something of the sort. An interesting problem 🤓

[assignUser]

It's relatively trivial to point back at the source ASF Jira issue (or issues) and force users wanting to navigate those relationships to do so via ASF Jira records. That's not very efficient or friendly, but may be sufficient as an associated cost of a one-time migration.

Strong -1 to a half:half situation, which would also means that we need to import open and closed/resolved issues.

[toddfarmer]

Just for clarity, where in a GitHub issue will we retain a consolidated list of sub-tasks? I ask because, for example, the description of #14542 contains a list of sub-tasks, but that is not editable for me. I can add a comment with a link to a new sub-task, but ... does that mean readers will have to scan all content for links to evaluate whether a sub-task is referenced? This probably matters in the context of the import, because if a system user account is leveraged to perform the import, and a list of sub-tasks is placed in the issue description, it's possible that very few (if any?) privileged users can modify that list (both to add and remove listed sub-tasks).

[toddfarmer]

I'm finding associating issues to be the most challenging problem of the proposed migration. There are a few options:

Listing associated issues in issue description

This is problematic for three reasons. First, only a select number of people can update the description to modify the related issues list. These individuals may not include an assignee. Second, the related issues are unstructured. Yes, we can propose standard formatting as has been described above, but ... there's no guarantee that people will follow it, and any slight deviation will make it difficult to extract related issues for any sort of reporting. Third, it's virtually impossible to filter on that content. Yes, the target related issue will have a back-link, but this persist even after it is removed from the source issue, meaning there will be false positive back-links.

Adding associated issues in comments

This has the same problem, but it adds the additional challenge that the related issues are disbursed across any number of issue comments. It allows different users to add new related issue links, partially mitigating the first problem above, but does not solve the problem of removing links to issues that are no longer considered related. Again, only a select group of individuals can modify comments made by a given user.

Using a related issue project

This is demonstrated in this issue:

Projects allow for custom fields to be defined and used, and in this example, child issues can be associated with a parent issue. The management of this isn't easy - every new parent issue requires adding a value to the custom field picklist, and adding an issue to a project can also only be done by a limited number of people. However, these operations could be automated with GitHub actions. While the custom fields in a project cannot be used in normal issue filtering, projects themselves have search and dashboard capabilities that can use these fields to filter:

[jorisvandenbossche]

First, only a select number of people can update the description to modify the related issues list. These individuals may not include an assignee.

I am not sure this will be a problem in practice: 1) we can give triage rights to regular contributors that are not committers (we just haven't been doing that now, because we didn't need that for JIRA, 2) in most cases it is the person who opened the issue who that will create a list of subtasks, in which case they can edit it anyway (this is probably not true for migrated issues, since they don't become the author of that issue? But also here, we still have item 1))
So personally my guess is that this will be OK in practice, and it is still something we can address later and try to find a different solution (or helper bots) if it turns out to give problems.

I would say that using task lists for the subtask relation on JIRA is, despite its limitations, by far the easiest.

I think we could explore using GitHub Project features, but that feels something we should experiment with while using it, and something we can start to use later. I personally wouldn't already use this for the migration.

[jorisvandenbossche]

the target related issue will have a back-link, but this https://github.com/toddfarmer/test_import/issues/95, meaning there will be false positive back-links.

The screenshot above in #14520 (comment) shows a "Tracked by .." label just below the issue title. I would expect that such a back-link gets updated?

[toddfarmer]

@jorisvandenbossche : I see what you're saying now, thanks. Some observations:

1. The "tracked in" only gets managed when the sub-tasks are listed in the issue description, not when they are added in comments.
2. The "tracked in" does get updated when sub-tasks get removed, though the activity stream in the previously-referenced issue still cites the (no-longer) referencing parent issue..
3. The syntax required to track issues as sub-tasks adds a checkbox, which only partially syncs to the referenced issue status. This issue shows an unchecked box for the completed sub-task, because I closed the sub-task first, then created the parent issue without explicitly adding the "X" to the checkbox. That results in a different view than this issue, which had the checkbox created empty as well, but then the sub-task was completed, and the checkbox automatically updated.
4. The checkbox state seems to drive some aspects of the sub-task tracking. See the difference between this example and this example. The first shows "1 of 2 tasks", the other shows "2 tasks".

All that said, I'm happy to add the necessary listing of sub-tasks to the parent issue description as part of the initial issue migration, and we can evaluate whether further changes are needed going forward.

[toddfarmer]

FYI, I've implemented the post-import update to the migrated issue description to add a list of sub-tasks to the parent issue. See this example - but ignore the rest of the issue description field formatting (it turns out the conversion code to get markdown out of Jira syntax fails to account for Jira's auto-numbering ordered list syntax, and interprets each line as heading instead of an ordered list element - now tracked in #14628).

[jorisvandenbossche]

That example looks good to me!

1. The "tracked in" only gets managed when the sub-tasks are listed in the issue description, not when they are added in comments.

I think that is fine. We also only want to use task lists in the main issue description I think, not spread across comments (for the purpose of having a "parent" issue with sub tasks)

2. The "tracked in" does get updated when sub-tasks get removed, though the activity stream in the previously-referenced issue still cites the (no-longer) referencing parent issue..

I think that is also fine. The issue was referenced at some point by the parent, so this information can still be useful when reading that issue, even thought it might no longer be an "official" sub task.

3. The syntax required to track issues as sub-tasks adds a checkbox, which only partially syncs to the referenced issue status. This issue shows an unchecked box for the completed sub-task, because I closed the sub-task first, then created the parent issue without explicitly adding the "X" to the checkbox.

I would find this only a minor issue, that can easily be corrected by the people actively working on an issue with subtasks (it's easy to quickly mark one as completed if you see a mismatch like this. In fact, if you make manual task lists that include more text than just the issue number, you actually always have to do this manually, as github then won't automatically tick the box).
But maybe this could even be fixed in the script that adds the subtasks in a second step? (as I suppose it is known at that point if the JIRA/issue is open or closed?)

[jorisvandenbossche]

Aside from the "subtasks" we have been discussing now, the original issue is about issue links in general I think? We should also try to keep those "Issue links" (and their types, like "is related to", "duplicates", "depends on", ..).
Could those also be handled in a second pass? And added as normal lists (instead of task lists)?

[toddfarmer]

Yes, my plan is to add all formally related issue links - those where a link has been added, rather than simply mentioned inline in an issue comment - to a comment at the end, along with issue watchers/author/etc. This does mean that any inline references to other issues in comments will point back to the ASF Jira issue that was originally referenced, which will people to the migrated issue on GitHub. Not ideal, but probably sufficient?

[toddfarmer]

This sample issue provides an example of the related (non-subtask) issue handling I propose.

[toddfarmer]

But maybe this could even be fixed in the script that adds the subtasks in a second step? (as I suppose it is known at that point if the JIRA/issue is open or closed?)

The import script knows the state of the subtask issues at the time of import, and it's setting the checkbox appropriately for the status at that time. I think that's what you are proposing here, but please let me know if I've misunderstood.

[jorisvandenbossche]

https://github.com/toddfarmer/test_import/issues/304 provides an example of the related (non-subtask) issue handling I propose.

Looks good!

The import script knows the state of the subtask issues at the time of import, and it's setting the checkbox appropriately for the status at that time. I think that's what you are proposing here, but please let me know if I've misunderstood.

Ah, that sounds good then. Because you brought up the issue (and showed an example of the not-correctly-synced checkbox in https://github.com/toddfarmer/test-arrow-config/issues/23), I thought it was a problem with the current migration strategy for subtasks. But it sounds like you already handle setting the checkbox state.