Support All API Versions

[marstr]

As discussed way back in #517, we would like to support all API Versions that are supported by Azure Services. At the moment, if there are breaking changes to the service in terms of API Version, we must release a new version of the SDK with breaking changes. However, this conflates breaking changes that have been made to target the most recent version of Azure Services and changes that we've made to make the SDK more idiomatic and pleasurable for the developer to use.

What this has led to is forcing people to not adopt newer versions of services than they would have otherwise, or not adopt newer versions of the SDK. We would like to end this debacle once and for all.

The solution is generating out packages which target each version of the service that is supported. The product of our efforts to achieve this is currently previewed in the branch experimental/allAPIVersions. (note: dramatic changes to this branch will come without warning.)

[marstr]

Time for an update!

The experimental branch with the changes necessary to support all API Versions has been updated, and I've used the code in in to rewrite the VM Management sample. You can find the work I've done on that in the following branch: Azure-Samples/virtual-machine-go-manage adoptExperimental.

The experience to adopt my experimental changes to a code base using v7 of the Azure SDK for go were pretty trivial. By far the most work came from other roll-up changes; in particular adopting the ADAL updates in go-autorest v8 and the channel updates in azure-sdk-for-go v10. If you pare out those changes, it really was as easy just updating import statements to include the API Version that should be targeted.

My local development environment is to work from VS Code using the lukehoban's Go extension 0.6.61. It was able to automatically make the switch for all of the packages accept github.com/Azure/azure-sdk-for-go/arm/resources which needed to be updated to (the admittedly infuriatingly redundant) github.com/Azure/azure-sdk-for-go/arm/resources/resources/2017-05-10/resources. I also needed to github.com/Azure/azure-sdk-for-go/arm/compute/2015-06-10/compute to github.com/Azure/azure-sdk-for-go/arm/compute/2017-03-30/compute because of a property that was added between API Versions.

The result just worked, which was a beautiful feeling. The best part being that it demonstrated that the Azure Services are pretty resilient to the API Version that was used to call it. I intentionally didn't go about picking out API Versions of packages that I knew would fit together. This leaves me pretty optimistic that the goal of using this to prevent SDK breaking changes without introducing new service compatibility bugs will be realized. I should note though, that while I didn't run into this, there is the potential that large-code bases have the additional work of orchestrating changes API Versions across their entire code-base at once or fail to compile. Realistically though, that is a pretty easy task to accomplish, and a pretty modest price to pay for the stability that this will bring to our SDK.

[marstr]

Alright, we're going to raise the bar a little here. @devigned has been working on a strategy to help drive Azure Stack compatibility across all of the Azure SDKs, Go is not going to be an exception.

To support Azure Stack "profiles" we're going to introduce special profile packages which contain aliases of all of the model types and consts of a targeted package. The first way we thought we may do that would be to parse a package, iterate through all of the types we find, and create a copy of each type thusly:

package webApps

import (
    original "github.com/Azure/azure-sdk-for-go/arm/web/2016-08-01/webApps"
)

type GroupClient original.GroupClient

This is nice because a simple cast will solve most of the problems you encounter. However, this has a couple of drawbacks. For one, methods that are available on the original.DatabaseMetric are not exposed via the alias we create, that means that casting, while easy, will have to happen obnoxiously frequently. So how do we solve that? On to the second iteration of thinking:

package sql

import (
    original "github.com/Azure/azure-sdk-for-go/arm/web/2016-08-01/webApps"
)

type GroupClient struct {
    original.GroupClient
}

By including an anonymous original.GroupClient in our structure, we automatically get "dot" access to all of the methods associated with the original type. However... yet again there is a drawback. All of the methods that we have access to reference the types from the original package. Which means that to use our package folks would need to type out something like:

package userPackage

import (
    "github.com/Azure/azure-sdk-for-go/profile/[selected profile]/web/webApps"
    original "github.com/Azure/azure-sdk-for-go/arm/web/2016-08-01/webApps"
)

myClient.AddPremierAddOnSlot("myResourceGroup", "myName", "myPremierAddOnName", 
    webApps.PremierAddOn{
        original.PremierAddOn: original.PremierAddOn{
            // Initialization fields should be inserted here
        }
    }, "mySlot")

So, obviously that is a no-go. We are clearly fighting the type system at that point. Not to mention, we have actively missed one of the goals here, which is to not force people to think about which particular API Versions they're importing if they don't want to. So, we're back to the original proposal but modified a little bit. In the original world, we'd generate types to extend all of the types we found in the original package, plus all of the methods tied to those types. That gets us past the troubles we've mentioned, but there are still a few pock-marks left:

1. Even if people are using the same actual models across profiles, they have to cast to the common type to communicate.
2. Initialization still looks a bit awkward.
3. A ton of work has to go into the generator to support creating alias functions for each of the already supported extensions.

We sat around mashing our teeth together and generally being frustrated with this scenario, until @devigned came with news that saved the day. In go 1.9, there will be a new mechanism for type aliasing. This is perfect for us, because these aliases allow for different spellings of a single type, it allows us to alias every single type, and not bother regenerating functions. The methods will just be available, and it doesn't matter if you pass in the raw or profile specific version. All of this, and you only need a single import statement. Even better, if two profiles target the same API Version of a package, you can pass data between the two interchangeably. That brings us to the plan:

package webApps

import (
    original "github.com/Azure/arm/web/2016-08-01/webApps"
)

type GroupClient = original.GroupClient

type PremierAddOn = original.PremierAddOn

// Rinse and repeat for all other types

[tombuildsstuff]

@marstr just to ensure that I've understood this correctly - are the Profiles between Azure Public and Azure Stack going to be sufficiently different that we're going to need to use different versions of each Client to connect? In addition, is there some kind of plan available for how API's added to Azure Public get added to Azure Stack? (sorry if I've mis-understood something here!)

Thanks!

[marstr]

Good question, @tombuildsstuff. Profiles between Azure Stack and Azure Public will be the same. It just about availability of profiles in different environments. Every profile will be available in Azure Public, each instance of Azure Stack will have it's own available set of profiles. You should be able to use the same clients everywhere.

[flyinprogrammer]

@marstr I'm really excited to see this work get done so terraform can actually support vm scale sets. How can I help?

[marstr]

Ah, thanks for pinging me on this thread, @flyinprogrammer. I've forgotten to update here for a long time!

I'm happy to say, things are in the final stretch. In the next two days, I plan on cleaning up the experimental/allAPIVersions branch so that it will merge with master more cleanly. Once that's done, I want to update a bunch of the Azure-Samples to use the latest profile. I'll ping you on this thread when I'm ready to start that, I could definitely use some help rewriting samples.

[marstr]

I ended up changing tack a little, and instead of attempting to get the experimental branch to merge with master, I just rebased the commits relating to the tools I added. That progress can be tracked here: #767

[marstr]

Alright! experimental/allAPIVersions has been populated with a new services folder containing the latest and greatest from the new generator which you can see more progress in #793

[devigned]

PR for multi-api version support: #812

[marstr]

#812 is merged! We will roll out multiple API Versions in an additive way the next time dev and master merge, probably v11.2.0