Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add CustomPackageOrder #355

Open
wants to merge 10 commits into
base: main
Choose a base branch
from
Open

add CustomPackageOrder #355

wants to merge 10 commits into from

Conversation

pmolodo
Copy link
Contributor

@pmolodo pmolodo commented Oct 13, 2016 •
edited by JeanChristopheMorinPerso
Loading

Fixes #369.

This adds a new package orderer, which allows the user to specify and explicit ordering for versions in the package, using standard version-range strings. It also provides an easy way to specify a "default" version. See the docstring for CustomPackageOrder for more details.

The first commit adds the ability to configure which orderers you want using the standard rezconfig mechanisms; the last also adds a ReversePackageOrder, which does exactly what you'd think it does. ;)

@nerdvegas
Copy link
Contributor

Cheers Paul, this sounds good.

I have a question. I actually need an orderer where you can specify a
particular version for any number of packages. The search order would then
start at this version and work down, and then start at latest and work
down. In other words, if package X has versions [1,2,3,4,5], and I
specified package X-3 to the orderer, then the search order would become
[3,2,1,5,4]. Is that something your orderer can do, and if not, does it
make sense to extend it to be able to, or to write a new one?

The reason I want this orderer: I've found there are common misconceptions
about how patching (-p) works. People often expect that if they run
"rez-env -p X", then they'll only potentially get a newer version of X, not
other packages. I would like to provide a kind of patching that does give
this expected behavior. It would minimize the number of package version
changes. With this new orderer I can achieve this, by setting the
'preferred' version of each package to the currently resolved version,
except for those packages explicitly requested in the patch list.

Thanks,
A

On Thu, Oct 13, 2016 at 10:53 PM, Paul Molodowitch <[email protected]

wrote:

This adds a new package orderer, which allows the user to specify and
explit ordering for versions in the package, using standard version-range
strings. It also provides an easy way to specify a "default" version. See
the docstring for CustomPackageOrder for more details.

The first commit adds the ability to configure which orderers you want
using the standard rezconfig mechanisms; the last also adds a

ReversePackageOrder, which does exactly what you'd think it does. ;)

You can view, comment on, or merge this pull request online at:

#355
Commit Summary

  • ability to configure package orderers in rezconfig
  • add CustomPackageOrder
  • add ReversedPackageOrder

File Changes

Patch Links:

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#355, or mute the thread
https://github.com/notifications/unsubscribe-auth/ABjqSg6YXPiCKDMiJdesEPoA0wy2dOScks5qzrZggaJpZM4KWd2O
.

@pmolodo
Copy link
Contributor Author

pmolodo commented Nov 2, 2016

Yeah, you should be able to do this. The strings given the the orderer are
standard range-expressions, so if you wanted 3-and-lower, then everything
else, you'd just do:

package_orderers = [
{
'type': 'custom',
'packages': {
'X': ['<4'],
}
},
]

(...or, of course, if you wanted EXACTLY 3-or-lower, and didn't want things
like 3.1 to come before 4, you'd do "<=3")

This would give you your [3, 2, 1, 5, 4] order because the orderer will by
default automatically append any versions which don't match any
version-ranges in the list at the end. You can also change the place at
which "all other versions" are inserted in the order by using False or an
empty string, so these three are equivalent:

['<4']
['<4', '']
['', '4+']

  • Paul

On Mon, Oct 31, 2016 at 2:59 PM allan johns [email protected]
wrote:

Cheers Paul, this sounds good.

I have a question. I actually need an orderer where you can specify a
particular version for any number of packages. The search order would then
start at this version and work down, and then start at latest and work
down. In other words, if package X has versions [1,2,3,4,5], and I
specified package X-3 to the orderer, then the search order would become
[3,2,1,5,4]. Is that something your orderer can do, and if not, does it
make sense to extend it to be able to, or to write a new one?

The reason I want this orderer: I've found there are common misconceptions
about how patching (-p) works. People often expect that if they run
"rez-env -p X", then they'll only potentially get a newer version of X, not
other packages. I would like to provide a kind of patching that does give
this expected behavior. It would minimize the number of package version
changes. With this new orderer I can achieve this, by setting the
'preferred' version of each package to the currently resolved version,
except for those packages explicitly requested in the patch list.

Thanks,
A

On Thu, Oct 13, 2016 at 10:53 PM, Paul Molodowitch <[email protected]

wrote:

This adds a new package orderer, which allows the user to specify and
explit ordering for versions in the package, using standard version-range
strings. It also provides an easy way to specify a "default" version. See
the docstring for CustomPackageOrder for more details.

The first commit adds the ability to configure which orderers you want
using the standard rezconfig mechanisms; the last also adds a

ReversePackageOrder, which does exactly what you'd think it does. ;)

You can view, comment on, or merge this pull request online at:

#355
Commit Summary

  • ability to configure package orderers in rezconfig
  • add CustomPackageOrder
  • add ReversedPackageOrder

File Changes

Patch Links:

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#355, or mute the thread
<https://github.com/notifications/unsubscribe-auth/
ABjqSg6YXPiCKDMiJdesEPoA0wy2dOScks5qzrZggaJpZM4KWd2O>
.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#355 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAEYeD1xV4tj76jpr2Q6fcS8Nv0QCQK1ks5q5mTWgaJpZM4KWd2O
.

@pmolodo
Copy link
Contributor Author

pmolodo commented Nov 2, 2016

Actually, after re-reading your use case, it sounds like you'd want to use
<=.

This does bring up something which I've thought in the past would be nice,
though: a way to do something like this in version-range expressions:

foo<3.*

Last note - in your case, since you're creating an orderer programatically,
you can feed VersionRange objects directly into the constructor for
CustomPackageOrder, instead of having to parse strings...

On Wed, Nov 2, 2016 at 4:31 PM, Paul Molodowitch [email protected] wrote:

Yeah, you should be able to do this. The strings given the the orderer
are standard range-expressions, so if you wanted 3-and-lower, then
everything else, you'd just do:

package_orderers = [
{
'type': 'custom',
'packages': {
'X': ['<4'],
}
},
]

(...or, of course, if you wanted EXACTLY 3-or-lower, and didn't want
things like 3.1 to come before 4, you'd do "<=3")

This would give you your [3, 2, 1, 5, 4] order because the orderer will by
default automatically append any versions which don't match any
version-ranges in the list at the end. You can also change the place at
which "all other versions" are inserted in the order by using False or an
empty string, so these three are equivalent:

['<4']
['<4', '']
['', '4+']

  • Paul

On Mon, Oct 31, 2016 at 2:59 PM allan johns [email protected]
wrote:

Cheers Paul, this sounds good.

I have a question. I actually need an orderer where you can specify a
particular version for any number of packages. The search order would then
start at this version and work down, and then start at latest and work
down. In other words, if package X has versions [1,2,3,4,5], and I
specified package X-3 to the orderer, then the search order would become
[3,2,1,5,4]. Is that something your orderer can do, and if not, does it
make sense to extend it to be able to, or to write a new one?

The reason I want this orderer: I've found there are common misconceptions
about how patching (-p) works. People often expect that if they run
"rez-env -p X", then they'll only potentially get a newer version of X, not
other packages. I would like to provide a kind of patching that does give
this expected behavior. It would minimize the number of package version
changes. With this new orderer I can achieve this, by setting the
'preferred' version of each package to the currently resolved version,
except for those packages explicitly requested in the patch list.

Thanks,
A

On Thu, Oct 13, 2016 at 10:53 PM, Paul Molodowitch <
[email protected]

wrote:

This adds a new package orderer, which allows the user to specify and
explit ordering for versions in the package, using standard version-range
strings. It also provides an easy way to specify a "default" version. See
the docstring for CustomPackageOrder for more details.

The first commit adds the ability to configure which orderers you want
using the standard rezconfig mechanisms; the last also adds a

ReversePackageOrder, which does exactly what you'd think it does. ;)

You can view, comment on, or merge this pull request online at:

#355
Commit Summary

  • ability to configure package orderers in rezconfig
  • add CustomPackageOrder
  • add ReversedPackageOrder

File Changes

Patch Links:

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#355, or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABjqSg6YX
PiCKDMiJdesEPoA0wy2dOScks5qzrZggaJpZM4KWd2O>
.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#355 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAEYeD1xV4tj76jpr2Q6fcS8Nv0QCQK1ks5q5mTWgaJpZM4KWd2O
.

@pmolodo pmolodo mentioned this pull request Dec 9, 2016
Closed
@pmolodo pmolodo mentioned this pull request Mar 17, 2017
Open
@pmolodo
Copy link
Contributor Author

pmolodo commented Apr 1, 2017 •
edited
Loading

Completely rebased to apply against the latest master, and cleaned up a bit too!

The first two commits are bugfixes, which are also included in the "orderer_fixes" pull request (#413)

@bpabel
Copy link
Contributor

bpabel commented Apr 3, 2017

Do you have an example of an actual use case for this? In what scenario would a package want to use non-standard versioning to order packages as opposed to using a standardized versioning scheme?

@pmolodo
Copy link
Contributor Author

pmolodo commented Apr 3, 2017 •
edited
Loading

It's not about versioning-ordering, per-se, but solver-prioritization.

That is, this doesn't affect which versions are considered greater than others at all (and as such, I now realize my comment, referencing this PR in this thread #302 doesn't apply).

As a real-world example, you may want the solver to "prefer" maya-2016 over maya-2017, even though maya-2017 is still considered greater. So, if someone requested

rez-env maya>2016

..then they would still end up with maya-2017 (or maya-2016.5), since it's "greater" than 2016. However, if they just requested:

rez-env maya

...then you might want to have it resolve to maya-2016, because that version is "prioritized". That's exactly the use case we have at Luma most of the time - where we just want to have a studio-wide "default" version of something. However, this also allows us to get even more granular, and say things like, "We prefer version 2016 over 2017, and within 2016, we prefer service pack 2 over service pack 3... but we prefer 2017 over 2015... and within 2017, we prefer service pack 4 over service pack 5".

@bpabel
Copy link
Contributor

bpabel commented Apr 3, 2017 •
edited
Loading

Ah, I see now. We use a similar concept of "preferred versions", though we define them in curated "environment" packages.

For example, we have a package called env.maya that artists use to launch maya. Inside the env.maya package is where we specify all the packages we need to actually use maya with all our plugins and software packages. This env.maya package also specifies the specific maya version to use app.maya-2016. When we want to use a new version of maya, we update this env package and the artists just get the new version of maya. The same is true for when we update any of our tools or plugins; we update this env package to push the changes out to users, so that users don't ever need to know anything about package versions.

It sounds like what you're looking for is environment management. The old documentation explicitly made the distinction between package management and environment management and how rez was really just a platform to build environment management on top off. I'm not sure why it was left out of the newer docs.

@pmolodo
Copy link
Contributor Author

pmolodo commented Apr 3, 2017

Allan already incorporated the concept of configurable package-prioritization into rez - check out package_order.py. I'm simply extending this work (by making it configurable through rezconfig, adding a new orderer, and fixing some bugs).

@nerdvegas
Copy link
Contributor

nerdvegas commented Apr 3, 2017 via email

@bpabel
Copy link
Contributor

bpabel commented Apr 3, 2017

Thanks for the explanation.

So instead we've switched to a runtime solve type approach, where the env is resolved when you launch the tool.

This is also the way I've been using rez.

This causes another problem though - we also don't want the latest of every
package each time a tool is launched, because that is too unstable.

Isn't this the entire purpose of using explicit versions in a package manager? Instead of specifying bare package dependencies, like mypackage, shouldn't you be launching tools using more explicit versions, like mypackage-1.5? That way, you automatically pick up "bugfix" releases (i.e. mypackage-1.5.2), but for minor releases, they won't automatically be pushed out to production until you manually update the launcher environment to use mypackage-1.6.

It seems like what you're actually trying to do is environment management (i.e. for Lighting, I need these packages and versions). But instead of specifying the "preferred" versions in the "Lighting" environment, you're specifying those "preferred" versions in the packages themselves and never really explicitly defining the "Lighting" environment.

To me, the "preferred" version is subjective depending on the environment, and shouldn't be an attribute of the package.

@nerdvegas
Copy link
Contributor

nerdvegas commented Apr 3, 2017 via email

@bpabel
Copy link
Contributor

bpabel commented Apr 3, 2017

Perhaps there's a misunderstanding, these are not attributes of the package.

Yes, I thought this was a setting for packages.

Our 'soft' timestamp orderer is configured separately to packages (we have
a config, overridable per-show, that controls this)

So, are you using rez config files as the primary way of defining different environments? In your configs, do you have a soft timestamp defined for each of your packages, or is it a global timestamp that affects the preferred version of all packages? When you want to push out the latest releases, do you just update the timestamps in your config files? I'd be interested to see what one of your config files looks like. Perhaps there is a simpler way to push out changes than what I am doing.

@nerdvegas
Copy link
Contributor

nerdvegas commented Apr 4, 2017 via email

@mottosso
Copy link
Contributor

mottosso commented May 3, 2019

As a real-world example, you may want the solver to "prefer" maya-2016 over maya-2017, even though maya-2017 is still considered greater.

I'm looking to address this exact issue, but I'm having trouble figuring out how to actually use the existing package_order.py. Where does it apply? In my rezconfig.py? In an individual package.py? In every man's dream?

I'm digging through the source, and found a unit test for it, but it doesn't actually test it in-context with where it's used, but rather tests the individual features unit-test style.

Would it be possible to provide a short example?

@instinct-vfx
Copy link
Contributor

I think i would handle such a "default" on the context/request side rather than in the solver. Another possibility would be to set a package filter to filter our 2017 (that can be overriden in the shell if you absolutely want it). But again, i would say that this is to be solved before sending your request.

@pmolodo
Copy link
Contributor Author

pmolodo commented May 3, 2019 •
edited
Loading

So... this PR is now way out of date, and would probably need to be heavily refactored to get it to work with current source... but, we're going to need to update our internal version of rez at some point, and I'd LOVE to see it merged in.

Once it was, you would configure it in rezconfig.py. Here's an example:

package_ordererers = [
    {
        'type': 'custom',
        'packages': {
            'foo': ['3.7', '==2.8', '<3.7'],
            'bar': ['1.8.6', '1.8', '1.7.9', '1.7', '<1.7'],
        },
    },
    {
        'type': 'version_split',
        'packages': ['baz', 'bah'],
        'first_version': '4.0.5',
    },
]

The version expressions in the custom orderer use standard rez version range syntax, and are listed in order from most preferred to least preferred (with all other unmatching versions coming at the end).

This would mean that for foo, you would:

  • First prefer 3.7 (and 3.7.*)
  • Then prefer EXACTLY verison 2.8 (not 2.8.0, or 2.8.1, etc)
  • Then prefer any version less than 3.7
  • Then prefer all other versions (ie, > 3.7)

Within a specified version range, normal version ordering takes place - so, for instance, the exact version that exist for foo are:

3.9, 3.8.2, 3.7.8.1, 3.7.5, 3.7, 3.6, 2.8.5, 2.8, 2.4

...then the full ordered version preference for foo would be:

- 3.7.8.1, 3.7.5, 3.7
- 2.8
- 3.6, 2.8.5, 2.4
- 3.9, 3.8.2

Note also that the inclusion of the "version_split" orderer was simply to demonstrate how to include multiple orderers. You could easily have included those in the "custom" orderer by doing:

package_ordererers = [
    {
        'type': 'custom',
        'packages': {
            'foo': ['3.7', '==2.8', '<3.7'],
            'bar': ['1.8.6', '1.8', '1.7.9', '1.7', '<1.7'],
            'baz': ['<=4.0.5'],
            'bah': ['<=4.0.5'],
        },
    }
]

@mottosso
Copy link
Contributor

mottosso commented May 6, 2019

Once it was, you would configure it in rezconfig.py

Thanks @elrond79! Was it possible to also use package_order.py without this PR merged?

@pmolodo
Copy link
Contributor Author

pmolodo commented May 6, 2019

@mottosso - sadly, no. I believe the first commit in this PR addressed the configuration issue.

This was referenced May 21, 2019
Closed
Closed
@mottosso mottosso mentioned this pull request Jun 23, 2019
Open
@maxnbk maxnbk requested review from nerdvegas and a team as code owners November 15, 2022 23:58
@JeanChristopheMorinPerso JeanChristopheMorinPerso requested a review from a team as a code owner April 4, 2023 15:30
@isohedronpipeline isohedronpipeline mentioned this pull request Mar 7, 2024
Closed
@isohedronpipeline isohedronpipeline mentioned this pull request Mar 15, 2024
Merged
This was referenced Apr 2, 2024
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nerdvegas nerdvegas Awaiting requested review from nerdvegas

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

PackageOrderer classes don't apply to variants
5 participants
@pmolodo @nerdvegas @bpabel @mottosso @instinct-vfx