Annotation sessions

[mzur]

In order to better support different kinds of annotation studies, we came up with the concept of annotation sessions:

A transect can have zero or more annotation sessions. Annotation sessions can be created, edited and deleted only by users with admin permissions for the transect. Annotation sessions cannot be deleted once there are annotations or image labels belonging to them (see below). Each annotation session has a name and a short description (like projects).

Annotation sessions may have a start date and/or an end date (with a day as smallest possible unit). Annotation sessions without start date are active for all days prior to the end date. Annotation sessions without end date are active for all days after and including the start date. Annotation sessions without both dates are always active. Annotation sessions with start and end date are active for all days after and including the start date up to and excluding the end date. Annotation sessions which are not active are inactive.

Annotation sessions may be restricted to a set of users. Each of these users must be able to access the transect (be guest, editor or admin in one of the projects). If an annotation session is restricted to a set of users, it is valid for all of these users and invalid for all other users. If an annotation session is unrestricted, it is valid for all users of the transect.

If an annotation session is active and valid for a user, it is running for this user.

For each user and transect, there must not be more than one running annotation session at any given time. There is one exception: There may be exactly two valid and active annotation sessions for a user at the same time if one has no start and end date and the other has both a start and end date. In this case the annotation session with start and end date is considered running and the other is not (the former overrides the latter). This means that unrestricted annotation sessions (where the exception does not apply) must not have an overlapping time frame where more than one of them is active. However, multiple annotation sessions may be active at the same time if they are restricted and for each user only one of them is valid.

Annotation sessions may be restricted to a subset of images of the transect. If an annotation session is running for a user, all images, the annotation session is restricted to, are valid for the user and all other images are invalid. Users may not access invalid images. This holds for all API endpoints, the transect overview and the annotation tool. Furthermore each user is not allowed to create or modify annotations or image labels belonging to invalid images. In particular, annotation sessions may be restricted to an empty subset of images of the transect. This means that all users for which the annotation session is running may not access, create or modify anything in the transect.

If an annotation session is running for a user, all newly created annotations or image labels of this user belong to the running annotation session.

If an annotation session is running for a user, the user may only see annotations and image labels belonging to this annotation session. Annotation sessions may or may not allow users to see annotations or image labels of other users which belong to the same annotation session.

Users may access, modify or delete only those annotations or image labels that belong to the currently running annotation session. This means that users won't be able to see, modify or delete their annotations or image labels of inactive annotation sessions. However, annotation sessions may or may not allow those users for which the annotation session is valid to access (but not modify or delete) the annotations or image labels (only their own or all, see above) even after the annotation session became inactive. This means that users may choose from different annotation sessions that are valid for them and access different versions of the transect overview or the annotation tool. Examples:

/transects/:id/annotate/:id (replaces /annotate/:id)

Shows the annotation tool for an image of the currently running session.

/transects/:id

Shows the transect overview for the currently running session.

/transects/:id/session/:number

Shows the transect overview for session number :number if the session is running or if the user is allowed to access the inactive but valid session. The last case returns the transect view in 'read only' mode (no image labels can be created/deleted).

/transects/:id/session/:number/annotate/:id

Shows the annotation tool for an image of session number :number if the session is running or if the user is allowed to access the inactive but valid session. The last case returns the annotation tool in 'read only' mode (no annotations or annotation labels can be created/modified/deleted).

New transects get an unrestricted annotation session without start and end date by default (the default annotation session). There is no distinction between the default annotation session and any other annotation session, which means that it can be modified or deleted just the same.

[mzur]

Ideas/questions:

• Offer prearranged studies:
  1. The transect gets a single session with a given start and end date (default duration 24/48h?), and a subset of x randomly chosen images. The default session of the transect (if there is one) will get the start date of the newly created session as end date so users won't be able to create new annotations or image labels after the session has ended.
  2. In addition to the default session, the transect gets a session from date a to b (48h?) with a subset of x randomly chosen images, then a session from date b to c, restricted to an empty set of images and finally a session from date c to d with a subset of y <= x images, randomly chosen from the images used in the first session.
• Should this be incorporated into the core application (plus transects/annotations module) or be a new separate module?

[mzur]

The approach described above has lots of nice features but since these features are probably only interesting for a minority of users, I believe it is way too complex to implement. It would touch a large portion of the application and would require lots of complex validation and authorization rules. Here is a much simpler approach that might offer enough features for annotation studies:

Annotation sessions belong to transects, have a start and end date (in days?), name and description. They can be created and modified by transect admins. Annotation sessions don't affect annotations directly. Instead, annotations can be assigned to an annotation session by their creation date. All annotations that were created after the begin date and before the end date of an annotation session belong to this session.

Annotation sessions have two options that can be switched on or off:

1. While the annotation session is active, users are not able to see annotations of other users.
2. While the annotation session is active, users are not able to see their own annotations that were created before the annotation session started.

This should statisfy the needs of different annotation studies where the intra and inter observer variablility should be measured.

The differences to the approach described above are these:

• Annotation sessions cannot be restricted to a subset of transect images. During the annotation session users can freely annotate all images of the transect. If only a subset is of interest for a study, the annotations need to be "manually" filtered later (not in Dias).
• Annotation sessions cannot be restricted to a subset of users. Like with the subset of images, if only annotations of some users are of interest, the annotations need to be filtered later on. This also means that only one annotation session can be active at a time. There cannot be different groups of users working in different annotation sessions at the same time.
• Annotation sessions affect only annotations, not image labels. Technically the image labels may be filtered by their creation date, too, but this probably doesn't make a lot of sense.

In combination with some other new features, this concept might come out nicely:

• Similar to labels, users or shapes, annotations in the annotation tool can be filtered by annotation session. This way, users only see the annotations that were created during the session (but are not forced to do so).
• Annotation sessions can be freely deleted by transect admins since this does not affect the annotations. They can be freely created "in the past", too, e.g. when the user wants to have a report of a certain time period.
• Transect reports (as opposed to project reports). Maybe "annotation session reports"? These reports can be restricted to a certain annotation session and contain only the annotations of this session. This is not possible for project reports since the user would have to choose an annotation session for each transect of the project.

Ideas that I dont want to implement right now since there are other more important issues:

• An option to lock annotations if they are x hours/days old. All annotations that were not modified for the time period of x cannot be changed (transformed, deleted, etc) any more.
• An option to lock a transect. When this option is active, the transect is in "read only" mode and no annotations/image labels can be added, changed or deleted. This lock may get a start and end date, too, so it can be combined with annotation sessions.
• Prearranged studies can still be offered. These studies are a combination of annotation sessions and transect locks. E.g. an annotation session for x days, then a transect lock for y days and finally another annotation session for x days.

[mzur]

Endpoints that are affected by this change. Checked if they were already updated.

Forbid displaying of 'hidden' annotations

core:

• GET|HEAD api/v1/images/{image}/annotations
• GET|HEAD annotations/{id}
• GET|HEAD api/v1/annotations/{annotation}
• GET|HEAD api/v1/annotations/{annotation}/labels
• GET|HEAD api/v1/transects/{transect}/annotations (deleted)

ate:

• GET|HEAD api/v1/annotations/{id}/patch
• GET|HEAD api/v1/transects/{tid}/annotations/filter/label/{lid}

annotations:

• GET|HEAD api/v1/transects/{id}/images/filter/annotations
• GET|HEAD api/v1/transects/{tid}/images/filter/annotation-label/{lid}
• GET|HEAD api/v1/transects/{tid}/images/filter/annotation-user/{uid}

[mzur]

We have to decide if we implement #18 before or after we publish this feature.

In any way we have to create a detailled manual for annotation sessions before this can be published. The manual should be scenario-oriented, which means that we have to describe what users should do if they want to perform a study to measure:

1. intra observer agreement
2. inter observer agreement
3. intra- and inter observer agreement

I think a detailled written manual with some images would be the best choice.

The manual must include a warning that an annotation session affects all users of all projects that are attached to the transect (which might not be a problem any more with #18 implemented).

Todo:

• Implement Exclude users from annotation sessions #18
• Create manual (also describe how Exclude users from annotation sessions #18 affects the behavior of the a/s, the annotation filter of the annotation tool and the reports)
• Require additional confirmation for update of a/s if there are annotations that would no longer belong to it after the change. Also require confirmation for deletion of a/s if there are annotations belonging to it.