Updated `tools/generator` to read from literate files #793

marstr · 2017-10-04T22:59:48Z

This is the generator for the next generation package in our SDK that provide multi-API Version and profile support.

The way we specified `go build` enumerated each package to be compiled. This created a lot of noise because folders that didn't contain any Go source themselves, but had children that did caused an error message. This way, we'll still get all of the protection of making sure that our entire repo builds, without the noise. Also, a couple of formatting changes in profileBuilder.

Also, further documentation improvements

marstr · 2017-10-06T20:41:00Z

Alright @jhendrixMSFT / @mcardosos: This PR is ready for feedback and merge. It is no longer a WIP

mcardosos · 2017-10-11T20:03:55Z

tools/profileBuilder/latest.go

 			currentGroup := operationGroup{
 				provider:     pathMatches[1],
-				resourceType: pathMatches[2],
-				group:        pathMatches[3],
+				resourceType: pathMatches[4],


resourceType: pathMatches[4], [](start = 4, length = 29)

does this also work with services that have a parentResourcePath ? What about graphrbac? Their paths are not like the rest of ARM

Yep, I've been adding in support for broader and broader sets of Swaggers. The regexp itself is structured to take care of those differences at the moment.

mcardosos · 2017-10-11T20:11:20Z

tools/generator/program.go

-		return
-	}
+// version should be set by the linker when compiling this program by providing the following arguments:
+// -X main.version=<version>


[](start = 19, length = 9)

Here do you mean the semantic version? or the git commit?

I've added more clarifying lines later locally.

Though to answer the immediate question, the git commit :)

Remind me plz, where do you use the git commit?

mcardosos · 2017-10-11T20:22:17Z

tools/generator/program.go

-	return regexp.MustCompile(strings.Join(elements, `[/\\]`))
+// getDefaultOutputBase returns the default location of the Azure-SDK-for-Go on your filesystem.
+func getDefaultOutputBase() string {
+	return strings.Replace(path.Join(goPath(), "src", "github.com", "Azure", "azure-sdk-for-go"), `\`, "/", -1)


normalizePath :D ?

Derp, yes good catch! Fixed locally

mcardosos · 2017-10-11T20:30:38Z

tools/generator/program.go

-	if output == nil {
-		output = ioutil.Discard
-	}
+	targetFiles = collection.AsEnumerable(flag.Args())


Args [](start = 44, length = 4)

Example?

These are the arguments to the program. https://godoc.org/flag#Args

Is that what you mean?

I mean an example of a program argument, what can I input as argument?

I just added a new commit which replaces the flag.Usage global variable function, the new tells the caller that the expected arguments are literate files.

mcardosos · 2017-10-11T20:43:14Z

tools/generator/program.go

+
+			// This function is a collection.Unfolder which takes a literate file as a path, and retrieves all configuration which applies to a package tag and Go.
+			return func(subject interface{}) collection.Enumerator {
+				results := make(chan interface{})


can it be a generationTuple from the beginning?

I'm afraid not :/

That's not to say there isn't anything we could do to make it nicer! But the generationTuple type would need to be significantly modified.

mcardosos · 2017-10-11T20:55:59Z

tools/generator/program.go

-		fmt.Println("Cloning Azure REST API Specs to: ", cloneLoc)
+		fmt.Println("Execution Time: ", time.Now().Sub(start))
+		fmt.Println("Generated: ", generatedCount)
+		fmt.Println("Formatted: ", formattedCount)


golint and go vet are useful too :D

Fair enough!

mcardosos · 2017-10-11T23:40:27Z

tools/generator/program.go

-	debugLog                    *log.Logger
-	version                     *semver.Version
+	targetFiles    collection.Enumerable
+	packageVersion string


packageVersion [](start = 1, length = 14)

Where is this set?

Fixed locally.

jhendrixMSFT · 2017-10-11T23:54:57Z

To take a step back, how come we need our own generation program? With all the language-specific generation options in the config files it seems that we should only need one tool to perform generation for any language.

mcardosos · 2017-10-12T00:02:21Z

BTW, how are you whitelisting dataplane stuff?

marstr · 2017-10-12T21:40:41Z

I already discussed all of the following with @mcardosos and @jhendrixMSFT offline, but I wanted to copy this over for community involvement.

@jhendrixMSFT says:
To take a step back, how come we need our own generation program? With all the language-specific generation options in the config files it seems that we should only need one tool to perform generation for any language.

Agreed, ideally we wouldn't have to have such a tool. However, at the moment it's the best thing we've got. I'm in talks with @johanste, @lmazuel, and @sergey-shandar about how we could extend our current tool chain to have some reusable means of generating SDKs from README tags.

@mcardosos says:
BTW, how are you whitelisting dataplane stuff?

Specifically, having a yaml $(tag)=='<tag name>' && $(go) with an output-folder attribute white lists a package to be generated. Generally though, I've basically taken the position that being in master of the public repository that documentation is being generated off of constitutes as being ready to be generated in Go. We get a lot of Issues files on us asking why we don't generate such Swaggers, I'd like to be out of that business. I believe @salameer is working on improving the Specs Repo processes to accomodate this thinking.

marstr · 2017-10-13T17:00:50Z

@mcardosos / @jhendrixMSFT any final thoughts before I merge this in?

mcardosos · 2017-10-13T17:07:48Z

Not many, not many

Previously, separate files in the profileBuilder had different build tags. This would cause errors when package `main` didn't have a `main` function.

marstr added 7 commits October 5, 2017 13:24

Removing generator to start again.

ca8c3ad

Beginning of nextGen tools/generator

72774e2

Formatting changes.

748596b

Adding gofmt call to generation process

9059d5b

Adding a description of the value of profiles.

52cd211

Adding Build phase and execution time notice

137d937

+Doc for each step of the generation pipeline

9f651f8

marstr force-pushed the dev branch from cb6e043 to 9f651f8 Compare October 5, 2017 20:27

marstr added 4 commits October 5, 2017 13:58

Adding documentation and fixing formatting

36e763c

Updating AutoRest Log Output location

e9a7c9d

Also, further documentation improvements

We can't have nice things.

eab171c

marstr requested review from mcardosos and jhendrixMSFT October 5, 2017 23:31

marstr self-assigned this Oct 5, 2017

marstr added 3 commits October 6, 2017 09:35

Updating managment path in profileBuilder

cb22960

Tinkering with profileBuilder regexp

5011bec

Adding missing Apache Header.

9ac693e

marstr mentioned this pull request Oct 9, 2017

General Availability of the Azure-Go-SDK #804

Closed

Fixing regex error

bd4c903

marstr mentioned this pull request Oct 11, 2017

Support All API Versions #631

Closed

mcardosos reviewed Oct 11, 2017

View reviewed changes

Adding Latest Strategy logging

6edc9d5

mcardosos reviewed Oct 11, 2017

View reviewed changes

Fixing SemVer'd package latest comparison

cd911f1

mcardosos reviewed Oct 11, 2017

View reviewed changes

Adding go vet generation step

1d71d3d

marstr added 3 commits October 12, 2017 16:09

Supporting nested RPs

b6fffab

Overriding generator Usage text.

907a3ff

Testing and fixing the List strategy

714052a

mcardosos approved these changes Oct 13, 2017

View reviewed changes

Updating profileBuilder build tags

0efea33

Previously, separate files in the profileBuilder had different build tags. This would cause errors when package `main` didn't have a `main` function.

marstr closed this Oct 13, 2017

marstr reopened this Oct 13, 2017

marstr merged commit e5041c6 into Azure:dev Oct 13, 2017

marstr mentioned this pull request Oct 25, 2017

Update tools/generator to allow for multi-swagger packages #770

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updated `tools/generator` to read from literate files #793

Updated `tools/generator` to read from literate files #793

marstr commented Oct 4, 2017 •

edited

Loading

marstr commented Oct 6, 2017

mcardosos Oct 11, 2017

marstr Oct 13, 2017 •

edited

Loading

mcardosos Oct 11, 2017

marstr Oct 11, 2017

mcardosos Oct 11, 2017

mcardosos Oct 11, 2017

marstr Oct 11, 2017

mcardosos Oct 11, 2017

marstr Oct 11, 2017

mcardosos Oct 11, 2017

marstr Oct 13, 2017

mcardosos Oct 11, 2017

marstr Oct 11, 2017

marstr Oct 11, 2017

mcardosos Oct 11, 2017

marstr Oct 11, 2017

mcardosos Oct 11, 2017

marstr Oct 11, 2017

jhendrixMSFT commented Oct 11, 2017

mcardosos commented Oct 12, 2017

marstr commented Oct 12, 2017

marstr commented Oct 13, 2017

mcardosos commented Oct 13, 2017

Updated tools/generator to read from literate files #793

Updated tools/generator to read from literate files #793

Conversation

marstr commented Oct 4, 2017 • edited Loading

marstr commented Oct 6, 2017

Choose a reason for hiding this comment

marstr Oct 13, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jhendrixMSFT commented Oct 11, 2017

mcardosos commented Oct 12, 2017

marstr commented Oct 12, 2017

marstr commented Oct 13, 2017

mcardosos commented Oct 13, 2017

Updated `tools/generator` to read from literate files #793

Updated `tools/generator` to read from literate files #793

marstr commented Oct 4, 2017 •

edited

Loading

marstr Oct 13, 2017 •

edited

Loading