Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow Independent Subviews via API #5253

Open
wants to merge 68 commits into
base: production
Choose a base branch
from
Open

Conversation

melton-jason
Copy link
Contributor

Fixes #114
Alternative implementation of #3125

New Functionality

Independent Subviews

An Independent Subview can be distinguished from its dependent counterpart (for both button/non-button subviews) by a magnifying glass which is used to search existing related records, or a link icon

An independent subview in subform mode with no related records

subview_indepdenet

An independent subview in subform mode with no related records

subview_independent_related

Querying to add to a -to-many independent subview

subview_to_many_add_via_query.mov

Querying to add to a -to-one independent subview

subview_to_one_add_via_query.mov

Unlike dependent records, because the records can exist without each other (hence independent), removing a related record from an independent collection only removes it from the associated relationship and does not delete the record.

💡 Unless a resource is explicitly saved/deleted in a dialog (in which case only that specific resource is modified), no changes are made to any of the related records until the "base" record is saved

Saveblockers propagate to the "base" resource

When the subview is not being rendered as a button, if there are any Saveblockers they will be propagated to the "base" record.

subview_save_blocker.mov

Adding new records to an Independent Subview

Because a viewname can be specified on the Form Definition, there is some special logic used when creating new related records to add to the collection.

When the viewname of the Subview is the same view as the default DataEntry form for the table, then the new resource is directly added to the Collection.

subview_base_form_add.mov

Otherwise, if the viewname for the independent subview is not the default DataEntry view, then a separate dialog will be used to create the new resource

subview_different_form_add.mov

(In the above example, a CollectionObject view called AccessionItems is used)

Version Control

To handle version control and ensure that the related records do not get out of date, in the background Specify will only care about the values of the related objects (CollectionObjects related to an Accession for example) when they are explicitly changed.
In other words, you should only experience an out-of-date error on the "base" record if you attempt to save the "base" record when a change to a related record was already made (i.e., the version has been incremented) and you have made changes to the related record from the "base" record.

In other words, the following is okay (as well as adding related records and then modifying them not in the Independent Subview and then saving the "base" record):

subview_independent_version_ctrl.mov

But the below workflow will result in an error:

subview_version_ctl_error.mov

Loading Indicator while fetching collections

subview_loading.mov

Finding Independent Relationships

You can use the Database Schema viewer in UserTools to find independent relationships on specific tables:

finding_independence.mov

For convenience, i've also compiled a list of every independent relationship in Specify organized by table in a json format: independent_master_list.json

You can open the file in any text editor.

(note that PaleoContext and CollectingEvent may be dependent/independent depending on the Discipline -> IsPaleoContextEmbedded and Collection -> IsEmbeddedCollectingEvent respectively)

Checklist

  • Self-review the PR after opening it to make sure the changes look good
    and self-explanatory (or properly documented)
  • Add automated tests
    • though I'd like to add more complex tests
  • Add relevant issue to release milestone

Testing instructions

When testing Independent -to-many relationships, please test Accession -> collectionObjects and RepositoryAgreement -> accessions. These are pretty popular!

Unless otherwise stated in the testing instruction, a specific step (or chain of steps) should be able to be performed on any Independent Subview (-to-one, -to-many, as button, etc.)

Querying related records

  • Using the QueryBuilder interface, make sure you can only select a single record when adding to a -to-one independent relationship (save the "base" record after adding the record)
  • Using the QueryBuilder interface, make sure you can select multiple records when adding to a -to-many independent relationship(save the "base" record after adding the records)

Creating new related records

  • For an independent subview which uses the default DataEntry view for the table (usually just the name of the table. If viewname is omitted in the subview cell, this view is used by default), ensure that no dialog is used when creating a new associated related record using the independent subview.
  • For an independent subview which uses a different view from the default DataEntry view for the table, ensure that a dialog is displayed and resource creation occurs within that dialog (once the resource is saved, it should be automatically added to the independent subview)
  • For more information and examples, see the Adding new records to an Independent Subview section earlier in this PR

Permissions

  • Ensure the buttons for Viewing, Searching/Querying, Creating, and Removing are only present if the logged in user has the correct permissions (All operations should require read permissions, and Query/Create/Remove should require update permissions on the related record table)

Version control

    1. Record the version and/or timestampModifiedof one or more related records before adding/editing them via an independent subview
    1. Add the related record to an independent subview (or edit/remove the related record if already present in the independent subview) and save the "base" record
  • Ensure that the version has increased by one and that the timestampModified has been updated since the recorded value for each related record
    1. Add one or more related records to an independent subview
    • Do not yet save the "base" record!
    1. In a separate tab, edit one or more of the related records and save them
  • Ensure the "base" record can still be saved
    1. Find an independent subview with one or more related records (add them if needed. "base" record saving is optional at this point)
    1. In a separate tab, edit one or more of the related records and save them
    1. Explicitly modify one or more of the related records which you saved in step 2 in the independent subview
  • Ensure an 'out-of-date' error is raised when the "base" record is saved.

Misc.

  • When working with an independent -to-one relationship (such as CollectionObject -> cataloger), make sure that the 🔍 and ➕ icons are disabled for the Subviewwhen there is one related record
  • When you have one or more related records blocking the deletion of a "base" record, ensure that you can remove the association(s) from the "base" record, save the "base" record, and then delete the "base" record.
  • Find or create a record on the one side of a one-to-many record (i.e., Accession with collectionObjects) with a lot (probably 80+) of related records and make sure a loading indicator is displayed while the Subview is loading.

maxpatiiuk and others added 30 commits March 8, 2023 15:50
Triggered by dfaa5d0 on branch refs/heads/issue-114
Triggered by 1359149 on branch refs/heads/issue-114
This is not likely the behavior we want, so this is something to be discussed.
See related #3127
@melton-jason
Copy link
Contributor Author

melton-jason commented Sep 9, 2024

@maxpatiiuk @realVinayak

Still need to handle some edge cases, write/update tests, and fix some known bugs (like the first bug from #5253 (comment)), but generally how about the approach in the latest implementation?

The API is changed to accept a dictionary like the following:

{
   "update":[
      {
         "text1":"creates a new related record and sets the relatonship"
      },
      {
         "id": 1,
         "text1":"Updates an existing related record and sets the relationship"
      }
   ],
   "remove":[
      "/api/specify/table/id"
   ]
}
  • everything still happens in a transaction
  • the payload to the backend is only what is strictly required by the consumer of the API to modify the Collection
  • updates to each related record is made using update_obj so business rules are always run and versions always checked
  • fetch/edit times for the Collection happen substantially faster because Specify no longer needs to check/update each related record in the relationship as well

The collection on the frontend is now a modified LazyCollection, and only the first 20 records are fetched initially. Any modifications to the Collection now only update those added/modified/removed (and no longer scales linearly with the length of the Collection).

Loading a VERY large Collection

Screen.Recording.2024-09-09.at.9.31.05.AM.mov

Updating a single related record

Screen.Recording.2024-09-09.at.9.38.58.AM.mov

Copy link
Collaborator

@combs-a combs-a left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Testing instructions

When testing Independent -to-many relationships, please test Accession -> collectionObjects and RepositoryAgreement -> accessions. These are pretty popular!

Unless otherwise stated in the testing instruction, a specific step (or chain of steps) should be able to be performed on any Independent Subview (-to-one, -to-many, as button, etc.)

Querying related records

  • Using the QueryBuilder interface, make sure you can only select a single record when adding to a -to-one independent relationship (save the "base" record after adding the record)
  • Using the QueryBuilder interface, make sure you can select multiple records when adding to a -to-many independent relationship(save the "base" record after adding the records)

Creating new related records

  • For an independent subview which uses the default DataEntry view for the table (usually just the name of the table. If viewname is omitted in the subview cell, this view is used by default), ensure that no dialog is used when creating a new associated related record using the independent subview.
  • For an independent subview which uses a different view from the default DataEntry view for the table, ensure that a dialog is displayed and resource creation occurs within that dialog (once the resource is saved, it should be automatically added to the independent subview)
  • For more information and examples, see the Adding new records to an Independent Subview section earlier in this PR

Permissions

  • Ensure the buttons for Viewing, Searching/Querying, Creating, and Removing are only present if the logged in user has the correct permissions (All operations should require read permissions, and Query/Create/Remove should require update permissions on the related record table)

Version control

    1. Record the version and/or timestampModifiedof one or more related records before adding/editing them via an independent subview
    1. Add the related record to an independent subview (or edit/remove the related record if already present in the independent subview) and save the "base" record
  • Ensure that the version has increased by one and that the timestampModified has been updated since the recorded value for each related record
    1. Add one or more related records to an independent subview
    • Do not yet save the "base" record!
    1. In a separate tab, edit one or more of the related records and save them
  • Ensure the "base" record can still be saved
    1. Find an independent subview with one or more related records (add them if needed. "base" record saving is optional at this point)
    1. In a separate tab, edit one or more of the related records and save them
    1. Explicitly modify one or more of the related records which you saved in step 2 in the independent subview
  • Ensure an 'out-of-date' error is raised when the "base" record is saved.

Misc.

  • When working with an independent -to-one relationship (such as CollectionObject -> cataloger), make sure that the 🔍 and ➕ icons are disabled for the Subviewwhen there is one related record
  • When you have one or more related records blocking the deletion of a "base" record, ensure that you can remove the association(s) from the "base" record, save the "base" record, and then delete the "base" record.
  • Find or create a record on the one side of a one-to-many record (i.e., Accession with collectionObjects) with a lot (probably 80+) of related records and make sure a loading indicator is displayed while the Subview is loading.

Tested using ChadronTest in the test panel.

Ran into a weird behavior--when deleting an item from a subview, it'll sometimes use the loading indicator until you save. Didn't have this issue from the Accession side of this relationship, just the Collection Object was having issues.

loadafterdelete.mp4

Had permission issues with embedded Collecting Events--when adding a Collection Object to an Accession (without editing it), Specify thinks that Collecting Event is being updated. Maybe not a bug but wanted to mention it in case it's not intended. The user role I made for this only had permissions for Accessions and read/update for Collection Objects. Same issue when adding an Accession to a Collection Object.

image

Additionally, after creating a bunch of Collection Objects via the grid subform in accessions (~120), the accession that they were created on only shows up to 24 of them, but you can still access them in subform mode. Exiting the tab, reloading, and clearing the cache did nothing to fix this. Adding this amount of Collection Objects through the Query Builder did display everything in grid view, as expected.

image

Good work so far Jason, this looks really good!

Copy link
Contributor

@pashiav pashiav left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Testing instructions

When testing Independent -to-many relationships, please test Accession -> collectionObjects and RepositoryAgreement -> accessions. These are pretty popular!

Unless otherwise stated in the testing instruction, a specific step (or chain of steps) should be able to be performed on any Independent Subview (-to-one, -to-many, as button, etc.)

Querying related records

  • Using the QueryBuilder interface, make sure you can only select a single record when adding to a -to-one independent relationship (save the "base" record after adding the record)
  • Using the QueryBuilder interface, make sure you can select multiple records when adding to a -to-many independent relationship(save the "base" record after adding the records)

Creating new related records

  • For an independent subview which uses the default DataEntry view for the table (usually just the name of the table. If viewname is omitted in the subview cell, this view is used by default), ensure that no dialog is used when creating a new associated related record using the independent subview.
  • For an independent subview which uses a different view from the default DataEntry view for the table, ensure that a dialog is displayed and resource creation occurs within that dialog (once the resource is saved, it should be automatically added to the independent subview)
  • For more information and examples, see the Adding new records to an Independent Subview section earlier in this PR

Permissions

  • Ensure the buttons for Viewing, Searching/Querying, Creating, and Removing are only present if the logged in user has the correct permissions (All operations should require read permissions, and Query/Create/Remove should require update permissions on the related record table)

Version control

    1. Record the version and/or timestampModifiedof one or more related records before adding/editing them via an independent subview
    1. Add the related record to an independent subview (or edit/remove the related record if already present in the independent subview) and save the "base" record
  • Ensure that the version has increased by one and that the timestampModified has been updated since the recorded value for each related record
    1. Add one or more related records to an independent subview
    • Do not yet save the "base" record!
    1. In a separate tab, edit one or more of the related records and save them
  • Ensure the "base" record can still be saved
    1. Find an independent subview with one or more related records (add them if needed. "base" record saving is optional at this point)
    1. In a separate tab, edit one or more of the related records and save them
    1. Explicitly modify one or more of the related records which you saved in step 2 in the independent subview
  • Ensure an 'out-of-date' error is raised when the "base" record is saved.

Misc.

  • When working with an independent -to-one relationship (such as CollectionObject -> cataloger), make sure that the 🔍 and ➕ icons are disabled for the Subviewwhen there is one related record
  • When you have one or more related records blocking the deletion of a "base" record, ensure that you can remove the association(s) from the "base" record, save the "base" record, and then delete the "base" record.
  • Find or create a record on the one side of a one-to-many record (i.e., Accession with collectionObjects) with a lot (probably 80+) of related records and make sure a loading indicator is displayed while the Subview is loading.

Tested on large db um_herb and smaller dbmcnb

Known issues: Saving new changes to forms with independent subviews work great, but saving them a second time causes an BusinessRuleException.
Specify 7 Crash Report - 2024-09-09T14_09_42.627Z.txt

The grid-type subview only shows 20 items max. This happens on edge too, but leaving it here as a note.

Screen.Recording.2024-09-09.at.1.56.40.PM.mov

I did not run into any other errors or issues. Looking great, Jason!

Copy link

@Areyes42 Areyes42 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Testing instructions

When testing Independent -to-many relationships, please test Accession -> collectionObjects and RepositoryAgreement -> accessions. These are pretty popular!

Unless otherwise stated in the testing instruction, a specific step (or chain of steps) should be able to be performed on any Independent Subview (-to-one, -to-many, as button, etc.)

Querying related records

  • Using the QueryBuilder interface, make sure you can only select a single record when adding to a -to-one independent relationship (save the "base" record after adding the record)
  • Using the QueryBuilder interface, make sure you can select multiple records when adding to a -to-many independent relationship(save the "base" record after adding the records)

Creating new related records

  • For an independent subview which uses the default DataEntry view for the table (usually just the name of the table. If viewname is omitted in the subview cell, this view is used by default), ensure that no dialog is used when creating a new associated related record using the independent subview.
  • For an independent subview which uses a different view from the default DataEntry view for the table, ensure that a dialog is displayed and resource creation occurs within that dialog (once the resource is saved, it should be automatically added to the independent subview)
  • For more information and examples, see the Adding new records to an Independent Subview section earlier in this PR

Permissions

  • Ensure the buttons for Viewing, Searching/Querying, Creating, and Removing are only present if the logged in user has the correct permissions (All operations should require read permissions, and Query/Create/Remove should require update permissions on the related record table)

Version control

    1. Record the version and/or timestampModifiedof one or more related records before adding/editing them via an independent subview
    1. Add the related record to an independent subview (or edit/remove the related record if already present in the independent subview) and save the "base" record
  • Ensure that the version has increased by one and that the timestampModified has been updated since the recorded value for each related record

    1. Add one or more related records to an independent subview
    • Do not yet save the "base" record!
    1. In a separate tab, edit one or more of the related records and save them
  • Ensure the "base" record can still be saved

    1. Find an independent subview with one or more related records (add them if needed. "base" record saving is optional at this point)
    1. In a separate tab, edit one or more of the related records and save them
    1. Explicitly modify one or more of the related records which you saved in step 2 in the independent subview
  • Ensure an 'out-of-date' error is raised when the "base" record is saved.

Misc.

  • When working with an independent -to-one relationship (such as CollectionObject -> cataloger), make sure that the 🔍 and ➕ icons are disabled for the Subviewwhen there is one related record
  • When you have one or more related records blocking the deletion of a "base" record, ensure that you can remove the association(s) from the "base" record, save the "base" record, and then delete the "base" record.
  • Find or create a record on the one side of a one-to-many record (i.e., Accession with collectionObjects) with a lot (probably 80+) of related records and make sure a loading indicator is displayed while the Subview is loading.

Tested with KU_Fish_5_16_23

Everything I tested looked good or was already mentioned by someone earlier.

I noticed that adding data to different types of subviews doesn't save correctly. Also, if you try switching between subform and grid after saving the form, it causes a weird visual error, which only gets fixed by refreshing the page.

Screen.Recording.2024-09-09.at.2.48.30.PM.mov

Copy link
Member

@maxpatiiuk maxpatiiuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️

@melton-jason
Copy link
Contributor Author

melton-jason commented Sep 11, 2024

Thanks for the reviews everyone!

From @combs-a, #5253 (review)

Ran into a weird behavior--when deleting an item from a subview, it'll sometimes use the loading indicator until you save.

Fixed!

Had permission issues with embedded Collecting Events--when adding a Collection Object to an Accession (without editing it), Specify thinks that Collecting Event is being updated. Maybe not a bug but wanted to mention it in case it's not intended.

This is actually an Issue with how the permissions are setup: not specific/caused by the independent subview PR.
That error is raised whenever a CollectionObject is updated (because the CollectingEvent needs to be passed inline with the CollectionObject to make any updates to the CollectionObject. The same is true for all dependent relationships).
Very related to #2714 (comment), and this can be fixed in #4893.

Here's a video of the issue even on edge:

Screen.Recording.2024-09-11.at.10.18.51.AM.mov

Also from @pashiav, #5253 (review)

The grid-type subview only shows 20 items max. This happens on edge too, but leaving it here as a note.

Yes, this is a bug in edge right now which also unfortunately affects this PR now because I made independent subviews fetch related records lazily.

Specifically, records in Independent Subviews are only fetched in chunks of 20 at a time. So for an example if you have 1,000 Collection Objects related to an Accession, only the first 20 should initially be fetched. If you navigate to the 21st record, then the next 20 records are fetched (and so on).

In Grid mode there's supposed to be a feature which fetches the next 20 records whenever you scroll to the bottom of the Subview, but that feature is broken currently.

Working on a fix!

A temporary workaround is to switch to "Form" mode, navigate to the 21st record, and then switch to Grid view. The 20-> 40 records should now be accessible. (This process can be repeated for every 20 records).


From @Areyes42, #5253 (review)

I noticed that adding data to different types of subviews doesn't save correctly. Also, if you try switching between subform and grid after saving the form, it causes a weird visual error, which only gets fixed by refreshing the page.

These were issues specifically with Dependent relationships/subviews. Should be fixed!
Can we verify that dependent subviews are behaving correctly?

specifyweb/specify/api.py Outdated Show resolved Hide resolved
ids_to_fetch.append(fk_id)

if fk_model is not None:
cached_objs = {item.id: obj_to_data(item) for item in get_model(fk_model).objects.filter(id__in=ids_to_fetch).select_for_update()}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice attempt! Note that this can be simplified by simply excluding the ones which reverse side as the current obj using an exclude(). In fact, you won’t even need lines 701-705 in that case. Plus, this is an optimization, on a lot of levels.

Copy link
Contributor

@pashiav pashiav left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Testing instructions

When testing Independent -to-many relationships, please test Accession -> collectionObjects and RepositoryAgreement -> accessions. These are pretty popular!

Unless otherwise stated in the testing instruction, a specific step (or chain of steps) should be able to be performed on any Independent Subview (-to-one, -to-many, as button, etc.)

Querying related records

  • Using the QueryBuilder interface, make sure you can only select a single record when adding to a -to-one independent relationship (save the "base" record after adding the record)
  • Using the QueryBuilder interface, make sure you can select multiple records when adding to a -to-many independent relationship(save the "base" record after adding the records)

Creating new related records

  • For an independent subview which uses the default DataEntry view for the table (usually just the name of the table. If viewname is omitted in the subview cell, this view is used by default), ensure that no dialog is used when creating a new associated related record using the independent subview.
  • For an independent subview which uses a different view from the default DataEntry view for the table, ensure that a dialog is displayed and resource creation occurs within that dialog (once the resource is saved, it should be automatically added to the independent subview)
  • For more information and examples, see the Adding new records to an Independent Subview section earlier in this PR

Permissions

  • Ensure the buttons for Viewing, Searching/Querying, Creating, and Removing are only present if the logged in user has the correct permissions (All operations should require read permissions, and Query/Create/Remove should require update permissions on the related record table)

Version control

    1. Record the version and/or timestampModifiedof one or more related records before adding/editing them via an independent subview
    1. Add the related record to an independent subview (or edit/remove the related record if already present in the independent subview) and save the "base" record
  • Ensure that the version has increased by one and that the timestampModified has been updated since the recorded value for each related record
    1. Add one or more related records to an independent subview
    • Do not yet save the "base" record!
    1. In a separate tab, edit one or more of the related records and save them
  • Ensure the "base" record can still be saved
    1. Find an independent subview with one or more related records (add them if needed. "base" record saving is optional at this point)
    1. In a separate tab, edit one or more of the related records and save them
    1. Explicitly modify one or more of the related records which you saved in step 2 in the independent subview
  • Ensure an 'out-of-date' error is raised when the "base" record is saved.

Misc.

  • When working with an independent -to-one relationship (such as CollectionObject -> cataloger), make sure that the 🔍 and ➕ icons are disabled for the Subviewwhen there is one related record
  • When you have one or more related records blocking the deletion of a "base" record, ensure that you can remove the association(s) from the "base" record, save the "base" record, and then delete the "base" record.
  • Find or create a record on the one side of a one-to-many record (i.e., Accession with collectionObjects) with a lot (probably 80+) of related records and make sure a loading indicator is displayed while the Subview is loading.



The following issues are ONLY FOR NEW RECORDS (Data Entry), not existing records. Everything seems to be working as expected on existing records.

1. Adding a new record (through both the + button and QueryBuilder) then deleting it causes it to load forever.

  • Note: Deleting through the grid subform does not having the excessive loading.
Screen.Recording.2024-09-11.at.12.15.30.PM.mov

2. The navigator is no longer displayed with the subform subview. With grid subview, you can see that the count of records is NaN.

Screen.Recording.2024-09-11.at.12.36.38.PM.mov
  • Expected counts and navigator (screenshot is from an existing record) :
    Screen Shot 2024-09-11 at 1 17 50 PM

  • When you add multiple records to a subform subview, but do not fill out all required fields, you get this save blocked message:
    Screen Shot 2024-09-11 at 12 42 59 PM But users cannot navigate between records to find the required field.


Also adding as a note because I don't think it's necessarily a big problem: when you add a new record to an independent subview that already has over 20 records, you can see that (in the navigator) the new record is added after the 20 initially fetched records - not after the last record in the subview. This could be confusing for users as seeing that adding a record will add it in the middle of the existing records, instead of after the expected last record.

Video: First, I show how a regular subview functions- new records are added to the last record. Then I show that in the independent subview, new records are added after the 20th record instead of the last record (24).

Screen.Recording.2024-09-11.at.1.44.40.PM.mov

@pashiav pashiav requested a review from a team September 11, 2024 19:03
Copy link
Collaborator

@lexiclevenger lexiclevenger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Testing instructions

When testing Independent -to-many relationships, please test Accession -> collectionObjects and RepositoryAgreement -> accessions. These are pretty popular!

Unless otherwise stated in the testing instruction, a specific step (or chain of steps) should be able to be performed on any Independent Subview (-to-one, -to-many, as button, etc.)

Querying related records

  • Using the QueryBuilder interface, make sure you can only select a single record when adding to a -to-one independent relationship (save the "base" record after adding the record)
  • Using the QueryBuilder interface, make sure you can select multiple records when adding to a -to-many independent relationship(save the "base" record after adding the records)

Creating new related records

  • For an independent subview which uses the default DataEntry view for the table (usually just the name of the table. If viewname is omitted in the subview cell, this view is used by default), ensure that no dialog is used when creating a new associated related record using the independent subview.
  • For an independent subview which uses a different view from the default DataEntry view for the table, ensure that a dialog is displayed and resource creation occurs within that dialog (once the resource is saved, it should be automatically added to the independent subview)
  • For more information and examples, see the Adding new records to an Independent Subview section earlier in this PR

Permissions

  • Ensure the buttons for Viewing, Searching/Querying, Creating, and Removing are only present if the logged in user has the correct permissions (All operations should require read permissions, and Query/Create/Remove should require update permissions on the related record table)

Version control

    1. Record the version and/or timestampModifiedof one or more related records before adding/editing them via an independent subview
    1. Add the related record to an independent subview (or edit/remove the related record if already present in the independent subview) and save the "base" record
  • Ensure that the version has increased by one and that the timestampModified has been updated since the recorded value for each related record
    1. Add one or more related records to an independent subview
    • Do not yet save the "base" record!
    1. In a separate tab, edit one or more of the related records and save them
  • Ensure the "base" record can still be saved
    1. Find an independent subview with one or more related records (add them if needed. "base" record saving is optional at this point)
    1. In a separate tab, edit one or more of the related records and save them
    1. Explicitly modify one or more of the related records which you saved in step 2 in the independent subview
  • Ensure an 'out-of-date' error is raised when the "base" record is saved.

Misc.

  • [x ] When working with an independent -to-one relationship (such as CollectionObject -> cataloger), make sure that the 🔍 and ➕ icons are disabled for the Subview when there is one related record
  • When you have one or more related records blocking the deletion of a "base" record, ensure that you can remove the association(s) from the "base" record, save the "base" record, and then delete the "base" record.
  • Find or create a record on the one side of a one-to-many record (i.e., Accession with collectionObjects) with a lot (probably 80+) of related records and make sure a loading indicator is displayed while the Subview is loading.

Looks awesome! I didn't come across issues with anything on the checklist, but here is something to consider.

Records can no longer be sorted using Order By in one-to-many independent subviews, even when you save to work around #4780. I'm not sure whether it needs to be fixed in this PR or can be fixed at a later date since there are already issues with Order By.

Screen.Recording.2024-09-11.at.3.33.48.PM.mov

Comment on lines 226 to 252
const self = this;
const records = Reflect.apply(
LazyCollection.prototype.parse,
this,
arguments
);

this._totalCount -= (this.removed as Set<string>).size;

return records.filter(
({ resource_uri }) => !(this.removed as Set<string>).has(resource_uri)
);
},
async fetch(options) {
if (this.related.isBeingInitialized()) {
return this;
}
this.filters[this.field.name.toLowerCase()] = this.related.id;

const offset =
this.length === 0 && this.removed.size > 0
? this.removed.size
: this.length;

options = { ...(options ?? {}), silent: true, offset };

return Reflect.apply(LazyCollection.prototype.fetch, this, [options]);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While this does work, a more simple solution be

  1. Defining a constant OFFSET = 20
  2. Have a local variable per collection defining the number of times fetch has been called
  3. Incrementing offset based on multiple of that variable

Benefits:

  1. This will remove the need for parse(resp) --- a record not fetched couldn't have been removed.
  2. Don't need to consider removed.size in offset calculation.
  3. Faster fetches (right now removed resources are still fetched -- quite unepected)

If you don't like variable, you could use a generator (see something similar on backend:

# Use this in places where we need to guarantee unique values. We could use random numbers, but using this
# makes things sane for debugging.
def get_unique_predicate(pre="predicate-") -> Generator[str, None, None]:
_id = 0
while True:
yield f"{pre}{_id}"
_id += 1

Copy link
Collaborator

@emenslin emenslin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Testing instructions

When testing Independent -to-many relationships, please test Accession -> collectionObjects and RepositoryAgreement -> accessions. These are pretty popular!

Unless otherwise stated in the testing instruction, a specific step (or chain of steps) should be able to be performed on any Independent Subview (-to-one, -to-many, as button, etc.)

Querying related records

  • Using the QueryBuilder interface, make sure you can only select a single record when adding to a -to-one independent relationship (save the "base" record after adding the record)
  • Using the QueryBuilder interface, make sure you can select multiple records when adding to a -to-many independent relationship(save the "base" record after adding the records)

Creating new related records

  • For an independent subview which uses the default DataEntry view for the table (usually just the name of the table. If viewname is omitted in the subview cell, this view is used by default), ensure that no dialog is used when creating a new associated related record using the independent subview.
  • For an independent subview which uses a different view from the default DataEntry view for the table, ensure that a dialog is displayed and resource creation occurs within that dialog (once the resource is saved, it should be automatically added to the independent subview)
  • For more information and examples, see the Adding new records to an Independent Subview section earlier in this PR

Permissions

  • Ensure the buttons for Viewing, Searching/Querying, Creating, and Removing are only present if the logged in user has the correct permissions (All operations should require read permissions, and Query/Create/Remove should require update permissions on the related record table)

Version control

    1. Record the version and/or timestampModifiedof one or more related records before adding/editing them via an independent subview
    1. Add the related record to an independent subview (or edit/remove the related record if already present in the independent subview) and save the "base" record
  • Ensure that the version has increased by one and that the timestampModified has been updated since the recorded value for each related record
    1. Add one or more related records to an independent subview
    • Do not yet save the "base" record!
    1. In a separate tab, edit one or more of the related records and save them
  • Ensure the "base" record can still be saved
    1. Find an independent subview with one or more related records (add them if needed. "base" record saving is optional at this point)
    1. In a separate tab, edit one or more of the related records and save them
    1. Explicitly modify one or more of the related records which you saved in step 2 in the independent subview
  • Ensure an 'out-of-date' error is raised when the "base" record is saved.

Misc.

  • When working with an independent -to-one relationship (such as CollectionObject -> cataloger), make sure that the 🔍 and ➕ icons are disabled for the Subviewwhen there is one related record
  • When you have one or more related records blocking the deletion of a "base" record, ensure that you can remove the association(s) from the "base" record, save the "base" record, and then delete the "base" record.
  • Find or create a record on the one side of a one-to-many record (i.e., Accession with collectionObjects) with a lot (probably 80+) of related records and make sure a loading indicator is displayed while the Subview is loading.

All the checks look good, just a few more issues.

  • Removing an accession from a repository agreement throws an error, I know it happens in other places but I am not sure where.
chrome_pXHj1Y0WSq.mp4

Specify 7 Crash Report - 2024-09-13T14_02_54.036Z.txt

  • ‘Order by’ not working properly, the changes of the previous ordering don't get set until you make a change to the subview options again. Trying to just change the record and save doesn’t work, it needs to be a change to the ‘order by’ options
chrome_125vrrxCog.mp4
  • Autonumber doesn’t show up properly when adding records to a subview with no viewname defined. It is actually working but you have to refresh to actually see it update.
chrome_GK7LFKfShJ.mp4
  • This might be the expected behavior but in this case cat # 10 is a part of accession 2024-AA-005 but I can then go to accession 2024-AA-004 and add cat # 10 to it and then it will just remove it from the previous accession. I can see how this might be the expected behavior but I wanted to mention it as it does feel a little weird.
chrome_iBfBZ3FZAr.mp4

@realVinayak
Copy link
Collaborator

realVinayak commented Sep 14, 2024

FYI: You'd want to set options.merge. Looking at the code, I'm able to produce a case where if you remove just the right object(s), you'd accidentally reset previously modified (on the frontend) data. So, basically, data entered by the user gets lost -- important bug! EDIT: I did test this on frontend, so not a false positive

Being vague on purpose as a challenge for you to figure out what those cases (not just 1 case) where that happens.

Spoiler Nope

Copy link
Contributor

@alesan99 alesan99 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Testing instructions

When testing Independent -to-many relationships, please test Accession -> collectionObjects and RepositoryAgreement -> accessions. These are pretty popular!

Unless otherwise stated in the testing instruction, a specific step (or chain of steps) should be able to be performed on any Independent Subview (-to-one, -to-many, as button, etc.)

Querying related records

  • Using the QueryBuilder interface, make sure you can only select a single record when adding to a -to-one independent relationship (save the "base" record after adding the record)
  • Using the QueryBuilder interface, make sure you can select multiple records when adding to a -to-many independent relationship(save the "base" record after adding the records)

Creating new related records

  • For an independent subview which uses the default DataEntry view for the table (usually just the name of the table. If viewname is omitted in the subview cell, this view is used by default), ensure that no dialog is used when creating a new associated related record using the independent subview.
  • For an independent subview which uses a different view from the default DataEntry view for the table, ensure that a dialog is displayed and resource creation occurs within that dialog (once the resource is saved, it should be automatically added to the independent subview)
  • For more information and examples, see the Adding new records to an Independent Subview section earlier in this PR

Permissions

  • Ensure the buttons for Viewing, Searching/Querying, Creating, and Removing are only present if the logged in user has the correct permissions (All operations should require read permissions, and Query/Create/Remove should require update permissions on the related record table)

Version control

    1. Record the version and/or timestampModifiedof one or more related records before adding/editing them via an independent subview
    1. Add the related record to an independent subview (or edit/remove the related record if already present in the independent subview) and save the "base" record
  • Ensure that the version has increased by one and that the timestampModified has been updated since the recorded value for each related record
    1. Add one or more related records to an independent subview
    • Do not yet save the "base" record!
    1. In a separate tab, edit one or more of the related records and save them
  • Ensure the "base" record can still be saved
    1. Find an independent subview with one or more related records (add them if needed. "base" record saving is optional at this point)
    1. In a separate tab, edit one or more of the related records and save them
    1. Explicitly modify one or more of the related records which you saved in step 2 in the independent subview
  • Ensure an 'out-of-date' error is raised when the "base" record is saved.

Misc.

  • When working with an independent -to-one relationship (such as CollectionObject -> cataloger), make sure that the 🔍 and ➕ icons are disabled for the Subviewwhen there is one related record
  • When you have one or more related records blocking the deletion of a "base" record, ensure that you can remove the association(s) from the "base" record, save the "base" record, and then delete the "base" record.
  • Find or create a record on the one side of a one-to-many record (i.e., Accession with collectionObjects) with a lot (probably 80+) of related records and make sure a loading indicator is displayed while the Subview is loading.

The testing instructions check out for Collection Objects and Accessions 👍

I am still experiencing some of the errors previously reported though. I think the behavior is slightly different now.

Setting order by on an Accession to-many causes my tab to freeze and eventually run out of memory

chrome_LPMpJOABm3.mp4

On a separate record I just got an error.

chrome_b1ROk8jZFg.mp4

Specify 7 Crash Report - 2024-09-18T14_31_40.231Z.txt

I also have a crash report from an error I got from removing a collection object from an accessions, though I could not verify if it still happens with the latest changes.
Specify 7 Crash Report - 2024-09-18T13_56_46.292Z.txt

@CarolineDenis CarolineDenis modified the milestones: 7.9.7, 7.9.8 Sep 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Dev Attention Needed
Development

Successfully merging this pull request may close these issues.

Add ability to add independent resources from the remote side of a one-to-many relationship.