From 7d78ca80d365f386a7818228f081382cb952da53 Mon Sep 17 00:00:00 2001 From: John Andersen Date: Thu, 1 Apr 2021 16:51:14 -0700 Subject: [PATCH 1/3] docs: arch: 2nd and 3rd party plugins Co-authored-by: Yash Lamba Co-authored-by: Saksham Arora Signed-off-by: John Andersen --- docs/arch/0001-2nd-and-3rd-party-plugins.rst | 210 +++++++++++++++++++ docs/arch/index.rst | 1 + 2 files changed, 211 insertions(+) create mode 100644 docs/arch/0001-2nd-and-3rd-party-plugins.rst diff --git a/docs/arch/0001-2nd-and-3rd-party-plugins.rst b/docs/arch/0001-2nd-and-3rd-party-plugins.rst new file mode 100644 index 0000000000..083ac32326 --- /dev/null +++ b/docs/arch/0001-2nd-and-3rd-party-plugins.rst @@ -0,0 +1,210 @@ +1. 2nd and 3rd party plugins +============================ + +Date: 2021-04-01 + +Status +------ + +Draft + +Context +------- + +DFFML currency has the main package and all plugins maintained within a single +repo. We always intended to support 3rd party plugins, meaning displaying them +on the main docs site as options for users. We are just now getting around to +it. + +We decided that we will have the main package, 2nd party plugins, and 3rd party +plugins. + +- The main package is ``dffml``. + +- 2nd party plugins are plugins that are maintained by the core maintainers and + who's repos reside in the ``dffml`` organization. + +- 3rd party plugins are hosted in user or other org repos and core maintainers + are not owners of those repos. + +We need to take the plugins that are currency maintained within the main repo +and put them in their own repos. We need the docs site to reflect the working / +not-working status of plugins on tutorial pages. We need to have a set of +plugins that we don't release unless those all work together. This is because a +core part of our functionality is letting users swap out underlying libraries, +which they can't do if they can't be installed together. + +What do we know +~~~~~~~~~~~~~~~ + +- Main package has no dependencies + +- The plugins page lists all plugins that are in ``dffml/plugins.py`` + +- There are tutorials that are associated with specific plugins + + - If a plugin's latest release doesn't pass CI against DFFML's latest + release, any tutorials should show that it's not working. + +- A main point of DFFML is to have a set of ML libraries that work together in + a single install environment so that a user can try multiple libraries and + choose the best one. + + - This means we have to know which plugins can be installed together. + +Decision +-------- + +- We want to do the compatibility matrix check in the main plugin and in each + plugin. + + - This lets the main plugin know at time of docs build, what status is for + each plugin. + + - This lets plugin authors know in PR CI, etc. if they are about to cause a + compatibility issue. + +- We need the ability to move things from level 1 to level 2 if we want to + deicide that that it's not longer a showstopper for release. + +- Working / not-working status of tutorials we want to show two things + This only applies to support levels 2 and 3. Because support levels 0 and 1 + must always work for release. + + - Does this tutorial work when other packages are installed latest / master? + + - Does this tutorial does work against all dependent packages for latest / + master? + + - Some plugins rely only on the main package + + - Main package never has any dependencies. So from a dependency checking + perspective, there should never be any issue. + + - Some plugin rely on other plugins as well + + - Main package never has any dependencies. So from a dependency checking + perspective, there should never be any issue. + +- To know which plugins can be installed together + + - Which plugins failure to validate against master branch warrant blocking + release + + - We need some sort of support level tracking. + + - Support level tracking means test against latest release and master + branch. + + - Possible support levels + + - 0 main package + + - 1 2nd party required pass for release + + - 2 2nd party not required pass for release + + - Note on tutorials that involve level 2 plugins to say they aren't + working at the moment + + - 3 3rd party not required pass for release + + - Note on tutorials that involve level 3 plugins to say they aren't + working at the moment + +Consequences +------------ + +- Based on support levels + + - The ``dffml/plugins.json`` should list the plugin, and it's support level + +- Documentation related to specific plugins + + - Tutorials that pip install packages of support level 2 or 3 must have some + element that the top of the page that can show the working / not-working + status. + + - When tutorials are tested, they only install the set of plugins that they + need. So a tutorial CI test will fail if those plugins do not work together. + Therefore, we display the warning because we know the tutorial works. If + there is a failure to install all support level 1 and 2 plugins together, we + know that we should display the warning. The tutorial works, but we're not + sure what other other plugins installed in a same environment might cause + dependency conflicts. + +- Matrix check, two perspectives (this translates into CI tests) + + - Main package + + - Support level 1 + + - For master, does installing all the plugins from their master zip + archive URL work when all given to ``pip install`` at the same time. + + - For latest release, does installing all the plugins by PyPi name work + when all given to ``pip install`` at the same time. + + - "work" here meaning does pip raise any issues about conflicting + dependency versions. + + - Support level 2 + + - For master + + - Does installing all the plugins in support levels 1 and 2 work from + their master zip archive URL work when all given to ``pip install`` + at the same time. + + - PASS: No warning on tutorials. + + - FAIL: Warning on tutorials, this may not work when other plugins + are install. This tutorial should still work when no other plugins + are installed. + + - For latest release, does installing all the plugins by PyPi name work + when all given to ``pip install`` at the same time. + + - Does installing all the plugins in support levels 1 and 2 work from + their PyPi name when all given to ``pip install`` at the same time. + + - PASS: No warning on tutorials. + + - FAIL: Warning on tutorials, this may not work when other plugins + are install. This tutorial should still work when no other plugins + are installed. + + - "work" here meaning does pip raise any issues about conflicting + dependency versions. + + - If they don't. Do we care about finding more info about which one's + are braking it. No, we do not care, because figuring out matrix is + exponential. + + - Support level 3 + + - Always have a warning on tutorials, this may not work when other plugins + are installed, because this a tutorial based on a third party plugin. + This tutorial should still work when no other plugins are installed. + In the event that it doesn't please report issues to third party here: + + + - Plugin package + + - Support level 1 + + - Fail CI if install of support level 1 plugins fails. + + - Support level 2 + + - Fail CI if install of support level 1 plugins fails. + + - If there is some way to warn via CI. Then warn if install of support + level 1 and 2 plugins fails. + + - Support level 3 + + - Fail CI if install of support level 1 plugins fails. + + - If there is some way to warn via CI. Then warn if install of support + level 1 and 2 plugins fails. diff --git a/docs/arch/index.rst b/docs/arch/index.rst index 3908b9a44e..7be21f5667 100644 --- a/docs/arch/index.rst +++ b/docs/arch/index.rst @@ -7,5 +7,6 @@ https://cognitect.com/blog/2011/11/15/documenting-architecture-decisions .. toctree:: :titlesonly: + 0001-2nd-and-3rd-party-plugins 0002-Object-Loading-and-Instantiation-in-Examples 0003-Config-Property-Mutable-vs-Immutable From 2197029229b61168e76cb02633e5fbf93c0e3481 Mon Sep 17 00:00:00 2001 From: John Andersen Date: Thu, 21 Oct 2021 07:58:15 -0700 Subject: [PATCH 2/3] Update 0001-2nd-and-3rd-party-plugins.rst --- docs/arch/0001-2nd-and-3rd-party-plugins.rst | 25 ++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/docs/arch/0001-2nd-and-3rd-party-plugins.rst b/docs/arch/0001-2nd-and-3rd-party-plugins.rst index 083ac32326..9ad846af2f 100644 --- a/docs/arch/0001-2nd-and-3rd-party-plugins.rst +++ b/docs/arch/0001-2nd-and-3rd-party-plugins.rst @@ -208,3 +208,28 @@ Consequences - If there is some way to warn via CI. Then warn if install of support level 1 and 2 plugins fails. + +Notes +----- + +tutorial check command + +this command will help us check if a tutorial by providing its URL is compatible with the locally installed version of the FML and all available plugins that are installed + +check if URL to tutorial we could push a Jason file or some sort of metadata into the I built doc so that we can check maybe a unique ID and the unique ID then you know build some Jason files well what we do is we output UM you know probably some kind of structure within the document that lets us determine what the versions of the plugins that the document were looking at was tested against + +dependency declaration + +we need some sort of file maybe espam format which declares all of our dependencies we can then do a you know take this format converted into something that can be pip installed and then we do a PIP download in one of the stages of the CI job we upload the downloaded artifact into the next stage of the CI job and then we you know install it or yeah no we we oh we set that as the index basically so we take these downloaded files and pip only well we expose them on a local file server or something or or maybe you can use the file URL for the index uhm URL of pip in this way the test cases are only allowed to install things which have been declared in whatever you know dependency format for example espam ah so + +PR validation + +essentially trigger a domino effect where we analyze the requirements files of all of the plugins that are either first or second party possibly support third party later somehow uhm and we build a dependency tree to understand which packages or which plugins are dependent on the plug in which is being changed in the original pull request we run the validation for the original pull request and then we run validation against you we trigger all of the CI runs of all of the downstream projects with the PR applied to with the original PR applied at if any of the downstream repos have would need to be changed for their CI to pass we can create PR's against those repos in the original PR we can provide overrides for each dependency so that when we trigger the validation or not dependency but downstream package so that when we trigger the validation for each downstream package we can say use this PR so if you've made an API breaking change and you need to go through all of the downstream dependencies are and make changes and submit PR that would make it OK then you go and then you specify you know all of those PRs which will be used when running the CI of the downstream dependencies respectively + +Repo locks + +for this what amounts to essentially a Poly repo structure to work we with the way that we're validating all of our poor requests against each other before merge we need to ensure that when the original PR is merged all the rest of the PR's associated with it that might you know fix API breaking changes in downstream dependent packages are also merged therefore we will need some sort of a system account or bot to which has which must approve every pull request and that bot we can make the logic so that if there is if an approved reviewer has approved the pull request then the bot will approve the pull request analyst initiate the locking procedure and rebate support request into the into the repo so when we have a change which effects more than one repo we will we will trigger rebase is into the respective repos main branches while all of those repos are locked in fact all of the reports will be locked within that within the main repo and the 2nd party org this is because we need to ensure that all of the changes get merged and there are no conflicts so that we end up in an unknown state which which would result in us ending up in an unknown state our state is known so long as we have tested all of the PR's involved against the main branch I or the you know the latest commit before rebase. When all PR's in a set across repos are approved the bot will merge starting with the farthest downstream PR at it will specify somehow version information to the CIA so that the C I can block waiting for the commit which was in the original PR to be merged before continuing this will ensure that the CI jobs do not run against a slightly outdated version of the original the repo which the original PR was made against + +Maintainer info + +for support level two or three plugins which might break with the application of a PR we should have some bot or some workflow comment to highlight which plugins would break if this PR was applied this information is purely for maintainers well as well as ah it's mainly for maintainers so that they understand whether they should request additional work be done or slight modifications or ah whether we need to plan and create issues to to potentially for example breaking a support level two plug-in we may be OK with that we just need to make sure that we're tracking it From a7b2b0585862bda883be5f475a50945f91043b2f Mon Sep 17 00:00:00 2001 From: John Andersen Date: Fri, 4 Mar 2022 12:47:14 -0800 Subject: [PATCH 3/3] Update docs/arch/0001-2nd-and-3rd-party-plugins.rst --- docs/arch/0001-2nd-and-3rd-party-plugins.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/arch/0001-2nd-and-3rd-party-plugins.rst b/docs/arch/0001-2nd-and-3rd-party-plugins.rst index 9ad846af2f..c3f9f57ac1 100644 --- a/docs/arch/0001-2nd-and-3rd-party-plugins.rst +++ b/docs/arch/0001-2nd-and-3rd-party-plugins.rst @@ -226,6 +226,8 @@ PR validation essentially trigger a domino effect where we analyze the requirements files of all of the plugins that are either first or second party possibly support third party later somehow uhm and we build a dependency tree to understand which packages or which plugins are dependent on the plug in which is being changed in the original pull request we run the validation for the original pull request and then we run validation against you we trigger all of the CI runs of all of the downstream projects with the PR applied to with the original PR applied at if any of the downstream repos have would need to be changed for their CI to pass we can create PR's against those repos in the original PR we can provide overrides for each dependency so that when we trigger the validation or not dependency but downstream package so that when we trigger the validation for each downstream package we can say use this PR so if you've made an API breaking change and you need to go through all of the downstream dependencies are and make changes and submit PR that would make it OK then you go and then you specify you know all of those PRs which will be used when running the CI of the downstream dependencies respectively +We should also make sure to support 3rd party plugin's abilities to revalidate against any of their dependencies, whenever one of their dependencies changes. Possibly some kind of service people can set as a webhook which is a sort of pubsub. The SCM sever such as GitHub publishes webhook events to the service (`dffml-service-sw-src-change-notify`). The service then relays to any listeners. Listeners are downstream projects. Downstream projects can register themselves with the listener to receive change events for any of their dependencies. Registration involves plugin based configurable callbacks. + Repo locks for this what amounts to essentially a Poly repo structure to work we with the way that we're validating all of our poor requests against each other before merge we need to ensure that when the original PR is merged all the rest of the PR's associated with it that might you know fix API breaking changes in downstream dependent packages are also merged therefore we will need some sort of a system account or bot to which has which must approve every pull request and that bot we can make the logic so that if there is if an approved reviewer has approved the pull request then the bot will approve the pull request analyst initiate the locking procedure and rebate support request into the into the repo so when we have a change which effects more than one repo we will we will trigger rebase is into the respective repos main branches while all of those repos are locked in fact all of the reports will be locked within that within the main repo and the 2nd party org this is because we need to ensure that all of the changes get merged and there are no conflicts so that we end up in an unknown state which which would result in us ending up in an unknown state our state is known so long as we have tested all of the PR's involved against the main branch I or the you know the latest commit before rebase. When all PR's in a set across repos are approved the bot will merge starting with the farthest downstream PR at it will specify somehow version information to the CIA so that the C I can block waiting for the commit which was in the original PR to be merged before continuing this will ensure that the CI jobs do not run against a slightly outdated version of the original the repo which the original PR was made against