Separate compiled App from CLI tooling

[paganotoni]

Currently Buffalo build does some transformations to the main to embed other things within the resulting binary. Besides the application the compiled binary ends up having tasks and migrations making the build process tricky.

It seems to me that it is a common Go pattern to have multiple binaries within a codebase. Assuming that's the case we could split the CLI functionality into a separate cmd/cli.

Implications

• The generated application would end up with two folders under cmd, cmd/app and cmd/cli.
• Generated Dockerfile should be updated to contain build step for the cli.
• We would also need to write a fixer for existing applications, that generates cmd/cli and fixes the Dockerfile.
• We would remove the transformMain part of our build process and maybe move closer to the go build command.

Thoughts on this? cc @stanislas-m @sio4 @fasmat @gobuffalo/all

[sio4]

I am not fully sure. My understanding is that, finally, we can build the application by just running go build ./cmd/app as the same as the standard go application? If so, I think that could be great! (Sorry, I am not fully catch up this part + your initiative)

[paganotoni]

Well, thats partially true. I say this because in the case of a webpack based app we would need to compile the assets before compiling the app (and with that embed the public folder).

This would put us closer to that but not there. This makes me think about a separate topic: turning Buffalo default apps to be Node-less, compile our assets with Esbuild and use something like the Tailwind CLI for the CSS.

That would modernize our stack and reduce our dependency on Node and webpack as those 2 deliver binaries we could use in our development workflows, that is something Rails and other frameworks are starting to do and IMO makes all the sense (Even tho I'm not a fan of "doing what other do" per se).

I wanted to get our conversation going on this topic. This one is a nice-to-have for me, and the end goal would require other changes.

[fasmat]

There is also an old proposal on that topic here: gobuffalo/buffalo#1791

I kinda like the idea. It would greatly simplify the CLI application, which then essentially only adds one source file per plugin that is needed for the application and is used for hot-reload development. Building the application then only requires go build.

Regarding frontend technologies: I usually separate backend and frontend into individual projects, so I'm open for suggestions.

[paganotoni]

Ah yes. We're kind of getting to the extensibility topic on Buffalo. Which I think is one of the importan topics for v1.0.0.

In terms of the Frontend, I think thats one of the competitive advantages of Buffalo, other Go Frameworks are mostly about the backend. However, if we could reduce the amount of things we manage there we would be in a better position than what we are. I keep thinking we should talk about that in a separate conversation. Will start it soon.

[sio4]

I don't have a very solid opinion but basically, I agree that the way of "thin cli with plugins" could be the best approach for both users and our teams. yeah, this is off-topic in the perspective of this thread.

The original reason I asked about it was that if we can just treat a buffalo app as a normal go application and can use go build for building, I guess something like debugging process, or other tool integration (not sure what exists), will be simpler than before too. It could be better if the buffalo app's structure or behavior is closer to the standard.

[paganotoni]

While thinking about this one I thought we could change this "TransformMain" strategy to be an API that could be used in cmd/app/main.go. So we explicitly show the application binary wrapping and simplify the maintenance of that part of the build process by moving closer to go build.

[paganotoni]

I'm thinking moving from :

func main() {
	app := actions.App()
	if err := app.Serve(); err != nil {
		log.Fatal(err)
	}
}

Into something like:

func main() {
        c := cli.Wrap(
          actions.App(),

          cli.WithMigrations(migrations) // Assuming we will embed migrations with the migrate task
          cli.WithTasks(tasks) // in case there is something we need to pull from there
          ... // Other binary commands
        )

        // Running the wrapped binary
        c.Run()
}

[sio4]

I love this approach!

[stanislas-m]

Well, in my opinion the main issue is the fact we rewrite the main to include migrations stuff. How about just generating the app the same way as now, but with the code we inject in main as part of the user app?
That means https://github.com/gobuffalo/cli/blob/main/internal/genny/build/templates/buffalo_build_main.go.plush becomes the new main instead of being a tmp thing.

[paganotoni]

What if we take this approach? (#129 (comment)). We would not change main.go that much and still don't bloat user codebase with build time things.

[paganotoni]

@stanislas-m Would love to read your thoughts on the wrapping approach.

[stanislas-m]

As long as the user "owns" the code and it's no longer magic wrapping at build time, it's fine to me. :)

[fasmat]

@paganotoni: I still think we should go with an approach similar to gobuffalo/buffalo#1791

A sample application with the proposed new layout would already be available here: https://github.com/markbates/bluffalo

Specifically this is the main.go for a new application: https://github.com/markbates/bluffalo/blob/master/coke/main.go

This is the cli package that is called from main.go in an application (part of generated code): https://github.com/markbates/bluffalo/blob/master/coke/cli/cli.go

It has 2 exported functions: Main() that serves as the actual entry point to the application and is NOT rewritten when building a binary with buffalo build and Buffalo() that the buffalo cli tool uses when calling subcommands like fix and generate. buffalo dev still calls the same Main() as the built binary.

[paganotoni]

Yes. I think something like that would allow us to provide extensibility in the Buffalo CLI. I have been maintaining @wawandco Buffalo CLI in something we call ox. Which follows something similar to that pattern.

https://oxcli.com/docs/plugins/architecture#the-general-picture

[paganotoni]

My desire is that the Buffalo CLI evolves to a point where my teams could sunset the Ox CLI and use Buffalo.

[sio4]

It seems like the "moving main.go into cmd/app" was already released as v0.18.4 but the other related things are not fully migrated to the new structure. Also, I found some issues with this approach.

1. buffalo task no longer works (I think this is one of the important parts of the buffalo cli)
   • buffalo routes doesn't work
   • bin/todo routes doesn't work (built app)
   • however, bin/todo task routes works
2. the name of app directory is always app (cmd/app) for all applications (is a good thing? a bad thing? not sure for now)

Also, having apps in cmd/ could be nice if users need multiple binaries for a single application, but actually, I am not fully sure if we need that for the generated applications while lightweight cli + plugins structure should be the best for CLI itself.

For the end-users, I think the simple structure with a single binary (command) for all they need to do for their service management is one of the best parts of using buffalo, also this change requires them to migrate their management(deployment) pipeline.

If we don't have a solid consensus on this approach yet, I would like to keep the original structure at least for now. Also, I would like to freeze the current version of buffalo (except for fixing broken things) as v1 (yes, I want to make this version as v1 to prevent confusion for existing users when we are going to the next version) soon and I hope we can implements any possible breaking changes on a new structure as v2.

What do you think? @paganotoni @fasmat @stanislas-m

[hdm]

Hi folks! Re: structure, I poked around with it both ways and it seems fine either way, but did notice that buffalo dev still uses ./main.go while buffalo build uses the new ./app/cmd directory. Adding a bit of code to copy main.go before build is fine, but seems redundant.

For the "other commands" issue, I treat the buffalo binary as a general purpose command with a hacky init() in main.go that redirects all but certain arguments to a grift:

func init() {
	// Convert the "server" argument to the no-args mode that buffalo expects
	if len(os.Args) == 2 && os.Args[1] == "server" {
		os.Setenv("MYAPP_SERVER_MODE", "true")
		os.Args = []string{os.Args[0]}
		actions.TaskMode = false
		return
	}

	// This process is running in task mode
	actions.TaskMode = true

	// Convert empty arguments to the CLI help command
	if len(os.Args) < 2 {
		os.Args = []string{os.Args[0], "t", "custom:cli", "help"}
		return
	}

	// Allow pass-through of common arguments
	switch os.Args[1] {
	case "migrate", "t", "tasks", "task":
		return
	}

	newArgs := []string{os.Args[0], "t", "custom:cli"}
	newArgs = append(newArgs, os.Args[1:]...)
	os.Args = newArgs
}

This lets ./binary server work as normal, but lets me also add ./binary <anything> by handling it in the custom:cli grift.

A less hacky way to do this would be awesome, but this works for now.