Skip to content

Commit

Permalink
Merge branch 'master' into feature/storage/stg71-base
Browse files Browse the repository at this point in the history
  • Loading branch information
@gapra-msft
gapra-msft committed Jan 8, 2020
2 parents b03044f + 0f27612 commit 2b32afb
Show file tree
Hide file tree
Showing 482 changed files with 39,480 additions and 3,397 deletions.
41 changes: 21 additions & 20 deletions .github/CODEOWNERS
View file
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
Expand Up @@ -6,33 +6,34 @@
###########

# Catch all
/sdk/ @joshfree @jonathangiles
/sdk/ @joshfree @jonathangiles

# Service teams
/sdk/appconfiguration/ @mssfang @alzimmermsft @conniey @sima-zhu
/sdk/batch/ @xingwu1 @bgklein @matthchr
/sdk/core/ @alzimmermsft @jianghaolu @srnagar @hemanttanwar
/sdk/cosmos/ @moderakh @christopheranderson @kushagraThapar
/sdk/eventhubs/ @conniey @srnagar @mssfang
/sdk/identity/ @jianghaolu @g2vinay @hemanttanwar
/sdk/keyvault/ @g2vinay @samvaity
/sdk/servicebus/ @yvgopal @nemakam
/sdk/tracing/ @samvaity @alzimmermsft
/sdk/appconfiguration/ @mssfang @alzimmermsft @conniey @sima-zhu
/sdk/batch/ @xingwu1 @bgklein @matthchr
/sdk/core/ @alzimmermsft @jianghaolu @srnagar @hemanttanwar
/sdk/cosmos/ @moderakh @christopheranderson @kushagraThapar
/sdk/eventhubs/ @conniey @srnagar @mssfang
/sdk/identity/ @jianghaolu @g2vinay @hemanttanwar
/sdk/keyvault/ @g2vinay @samvaity
/sdk/servicebus/ @yvgopal @nemakam
/sdk/textanalytics/ @samvaity @mssfang
/sdk/tracing/ @samvaity @alzimmermsft

/sdk/storage/ @rickle-msft @jaschrep-msft @gapra-msft @alzimmermsft @sima-zhu
/sdk/storage/ @rickle-msft @jaschrep-msft @gapra-msft @alzimmermsft @sima-zhu

# Management Plane
/**/resource-manager/ @yaohaizh
/sdk/**/azure-mgmt-*/ @yaohaizh
/**/resource-manager/ @yaohaizh
/sdk/**/azure-mgmt-*/ @yaohaizh

###########
# Eng Sys
###########
/eng/ @JimSuplizio @mitchdenny @weshaggard @danieljurek
/eng/code-quality-reports/ @mssfang @JonathanGiles
/eng/jacoco-test-coverage/ @srnagar @JonathanGiles
/eng/spotbugs-aggregate-report/ @srnagar @JonathanGiles
/parent/ @JimSuplizio
/eng/ @JimSuplizio @mitchdenny @weshaggard @danieljurek
/eng/code-quality-reports/ @mssfang @JonathanGiles
/eng/jacoco-test-coverage/ @srnagar @JonathanGiles
/eng/spotbugs-aggregate-report/ @srnagar @JonathanGiles
/parent/ @JimSuplizio

/**/tests.yml @danieljurek
/**/ci.yml @mitchdenny
/**/tests.yml @danieljurek
/**/ci.yml @mitchdenny
69 changes: 22 additions & 47 deletions CONTRIBUTING.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -87,31 +87,31 @@ The dependency-version should be set to the most recent released version and the

`com.azure:azure-identity;1.0.0-preview.4;1.0.0-preview.5`

Note: In the case of a new or unreleased artifact both versions will be the same.
Note: In the case of a new artifact both versions will be the same. In the case of a released artifact, the dependecny version should be the latest released version.

### Libraries vs External Dependencies

Libraries refer to things that are built and released as part of the Azure SDK. Libraries have a current version and a dependency version.

External Dependencies refer to dependencies for things that are not built and released as part of the Azure SDK regardless of the source. External Dependencies will only ever have one version.
External Dependencies refer to dependencies for things that are not built and released as part of the Azure SDK regardless of the source. External Dependencies will only ever have a dependency version.

### Current version vs Dependency version
### Current version, Dependency version and Unreleased Dependency version

Current version - This is the version we should be using when defining a component in its POM file and also when dependent components are built within the same pipeline. The current version is the version currently in development.
Dependency version - This is the version we should be using when a given library is a dependency outside of a particular area. This should be the latest released version of the package.
Dependency version - This is the version we should be using when a given library is a dependency outside of a particular area. This should be the latest released version of the package whenever possible.
Unreleased Dependency version – Whenever possible, libraries should be using the latest released version for dependencies but there is the case where active development in one library is going to be needed by another library or libraries that are built in separate pipelines. These types of changes are specifically additive and not breaking. Once a library has GA’d, nothing short of breaking changes should ever force the dependency versions across the repo to an unreleased version. The reason for this is that it would prevent other libraries, that don’t need this change, from releasing.

For example: `com.azure:azure-storage-blob-batch` has dependencies on `com.azure:azure-core`, `com.azure:azure-core-http-netty` and `com.azure:azure-storage-blob`. Because `com.azure:azure-core` and `com.azure:azure-core-http-netty` are both built outside of azure-storage pipeline we should be using the released or *Dependency* versions of these when they're dependencies of another library. Similarly, libraries built as part of the same pipeline, that have interdependencies, should be using the Current version. Since `com.azure:azure-storage-blob-batch` and `com.azure:azure-storage-blob` are both built part of the azure-batch pipeline when `com.azure:azure-storage-blob` is declared as a dependency of `com.azure:azure-storage-blob-batch` it should be the *Current* version.
An example of Current vs Dependency versions: `com.azure:azure-storage-blob-batch` has dependencies on `com.azure:azure-core`, `com.azure:azure-core-http-netty` and `com.azure:azure-storage-blob`. Because `com.azure:azure-core` and `com.azure:azure-core-http-netty` are both built outside of azure-storage pipeline we should be using the released or *Dependency* versions of these when they're dependencies of another library. Similarly, libraries built as part of the same pipeline, that have interdependencies, should be using the Current version. Since `com.azure:azure-storage-blob-batch` and `com.azure:azure-storage-blob` are both built part of the azure-batch pipeline when `com.azure:azure-storage-blob` is declared as a dependency of `com.azure:azure-storage-blob-batch` it should be the *Current* version.

This is going to be especially important after GA when releases aren't going to be the entire Azure SDK every time. If we're releasing a patch for a targeted azure-storage fix then we shouldn't need to build and release azure-core, we should be targeting the released versions and only building/releasing that update to azure-storage. It's worth noting that right now, in the version_client.txt, the dependency/current versions are the same. This will change once we GA, at which point the current version should be ahead of the dependency version.

What about README files? Right now the README files in the repo end up getting into an odd state since things like samples and versions can get updated during the development process. We're in the process of versioning documentation with the releases which means that the docs are snapshot at the time of the release and then versioned and stored. This will allow the README files in the repo to have updated samples and versions that are setup for the next release.
An example of an Unreleased Dependency version: Additive, not breaking, API changes have been made to `com.azure:azure-core`. `com.azure:azure-storage-blob` has a dependency on `com.azure:azure-core` and requires the additive API change that has not yet been released. An unreleased entry needs to be created in [version_client.txt](./eng/versioning/version_client.txt), under the unreleased section, with the following format: `unreleased_<groupId>:<artifactId>;dependency-version`, in this example that would be `unreleased_com.azure:azure-core;1.2.0` (this should match the 'current' version of core). The dependency update tags in the pom files that required this dependency would now reference `{x-version-update;unreleased_com.azure:azure-core;dependency}`. Once the updated library has been released the unreleased dependency version should be removed and the POM file update tags should be referencing the released version.

### Tooling, version files and marker tags

All of the tooling lives under the **eng\versioning** directory.

- version_client.txt - Contains the Client library and versions
- version_data.txt - Contains Data library and versions
- external_dependencies.txt - Contains the external dependency versions
- update_versions.py - This is just a basic python script that will climb through the source tree and update POM and README files. The script utilizes tags within the files to do replacements and the tags are slightly different between the POM and README files.
- set_versions.py - This script should only be used by the build system when we start producing nightly ops builds.

Expand All @@ -135,56 +135,31 @@ In README files this ends up being slightly different. Because the version tag i
```
[//]: # ({x-version-update-end})

What if I've got something that, for whatever reason, shoudln't be updated? There's a tag for that.

`<!-- {x-version-exempt;<groupId>:<artifactId>;reason for excemption} -->`

In theory, absence of an x-version-update tag would do the same thing but the tooling is still being developed and eventually there will be checkin blockers if xml has a version element with no tag.

### What does the process look like?

Let's say we've GA'd and I need to tick up the version of azure-storage libraries how would I do it? Guidelines for incrementing versions after release can be found [here](https://github.com/Azure/azure-sdk/blob/master/docs/policies/releases.md#incrementing-after-release).

1. I'd open up eng\versioning\version_client.txt and update the current-versions of the libraries that are built and released as part of the azure storage pipeline. This list can be found in pom.service.xml under the sdk/storage directory. It's worth noting that any module entry starting with "../" are external module dependencies and not something that's released as part of the pipeline. Once we GA, these build dependencies for library components outside of a given area should go away and be replaced with downloading the appropriate dependency from Maven like we do for external dependencies.
1. I'd open up eng\versioning\version_client.txt and update the current-versions of the libraries that are built and released as part of the azure storage pipeline. This list can be found in pom.service.xml under the sdk/storage directory. It's worth noting that any module entry starting with "../" are external module dependencies and not something that's released as part of the pipeline. Dependencies for library components outside of a given area would be downloading the appropriate dependency from Maven like we do for external dependencies.
2. Execute the update_versions python script from the root of the enlistment
`python eng/versioning/update_versions.py --ut libary --bt client`
This will go through the entire source tree and update all of the references in the POM and README files with the updated versions. Git status will show all of the modified files.
3. Review and submit a PR with the modified files.

### Next steps: External dependencies, Management plane and service pipeline changes
### Next steps: Management plane

- External dependencies. Right now there are only version files for client and data (eng\versioning\version_\[client|data\].txt) which only encompass the built binaries for their respective tracks. External dependencies for both client and data are next on the list which should allow modification of the parent/pom.xml to remove the list of version properties and dependency management sections which brings things one step closer to not having to publish the parent pom.
- Management plane. Management is in the process of being moved to service pipeline builds. The versioning work needs to wait until that work is finished.
- Service pipeline changes. The service pipelines currently have to build not only the libraries that are part of that pipeline but also the Azure SDK libraries that are dependencies. Once we GA and can start targeting the released version of those packages and pulling them from Maven instead of building them. An good example of this would be in sdk/appconfiguration/pom.service.xml where to build azure-data-appconfiguration we end up building azure-core, azure-core-test and azure-core-http-netty along with azure-data-appconfiguration instead of just building azure-data-appconfiguration.

### How are versioning and dependencies going to impact development after GA?

As mentioned above, in the service pipeline changes, the plan after we GA is to start targeting the released version of the packages and pulling them from Maven. This is going to fundamentally change some aspects of the development process especially when work needs to be done on given library that requires dependency changes in one or more libraries.

- **Scenario 1: Making changes to a single library which is not a dependency of any other libraries:** This ends up being the most straightforward scenario and really isn't much different than it is today.
- [ ] Appropriately increase the version
- [ ] Make the code changes
- [ ] Submit the PR
- [ ] Merge the PR
- [ ] Publish the new version

- **Scenario 2: Making changes to a library that also requires dependency changes:** Right now things are in a state where dependency changes can be made along with libraries that depend on them because of the project dependencies in the service pom files. Local development isn't going to change that much except when changing the version of a library and its dependency or dependencies means that the service poms are going to have to be built, and installed, in the appropriate order. This is because these new versions of the library dependencies won't yet be released and Maven will need to find these in the local cache. The biggest change to the process is going to be around PRs and publishing. Separate PRs are going to have to be submitted in order, with dependencies being submitted first. This is necessary because the dependencies need to be published in order to allow things that depend on them to continue using the published version. Trying to submit everything in one PR would cause build breaks since the dependency being referenced is a version not yet published. An example of this would be making changes to `com.azure:azure-storage-common` that also required dependency changes to `com.azure:azure-identity`.
Changes are going to have to be made to `com.azure:azure-identity` first.
- [ ] Appropriately increase the version of `com.azure:azure-identity`
- [ ] Make the code changes
- [ ] Build and optionally install locally
This isn't completely necessary other than to install the updated version of the dependency into the local cache on the machine. The alternative to this would be to publish (DevOps or otherwise) and reference that version of the dependency after the release. Either one would allow `com.azure:azure-storage-common` to use the updated version of `com.azure:azure-identity`
- [ ] Submit the PR for the `com.azure:azure-identity`
- [ ] Merge the PR for the `com.azure:azure-identity`
- [ ] Publish the `com.azure:azure-identity` with the updated version.

Only after the dependency `com.azure:azure-identity` has been published can the PR for `com.azure:azure-storage-common` be created.
- [ ] Appropriately increase the version of `com.azure:azure-storage-common` and the dependency version of `com.azure:azure-identity` in its pom file.
- [ ] Make the code changes, if any
- [ ] Build/Test or whatever
- [ ] Submit the PR for `com.azure:azure-storage-common`
- [ ] Merge the PR for `com.azure:azure-storage-common`
- [ ] Publish the PR for `com.azure:azure-storage-common`

### Making changes to an already GA'd library that require other libraries to depend on the unreleased version

This is where the `unreleased_` dependency tags come into play. Using the Unreleased Dependency example above, where `com.azure:azure-storage-blob` has a dependency on an unreleased `com.azure:azure-core`:

- [ ] Make the additive changes to `com.azure:azure-core`
- [ ] In [version_client.txt](./eng/versioning/version_client.txt) add the entry for the unreleased azure core in the unreleased section at the bottom of the file. The entry would look like `unreleased_com.azure:azure-core;<version>`.
Note: The version of the library referenced in the unreleased version tag should match the current version of that library.
- [ ] In the pom.xml file for `com.azure:azure-storage-blob`, the dependency tag for `com.azure:azure-core` which was originally `{x-version-update;com.azure:azure-core-test;dependency}` would now become `{x-version-update;unreleased_com.azure:azure-core-test;dependency}`
After the unreleased version of `com.azure:azure-core` was released but before `com.azure:azure-storage-blob` has been released.
- [ ] In [version_client.txt](./eng/versioning/version_client.txt) the the dependency version of `com.azure:azure-core` would become the released version and the "unreleased_" entry, at this time, would be removed.
- [ ] In the pom.xml file for `com.azure:azure-storage-blob`, the dependency tag for `com.azure:azure-core` would get changed back to `{x-version-update;com.azure:azure-core-test;dependency}`

### Nightly package builds

Expand Down
Loading

0 comments on commit 2b32afb

Please sign in to comment.