Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add an "export" command for libraries #369

Open
mvdan opened this issue Jun 21, 2021 · 15 comments
Open

add an "export" command for libraries #369

mvdan opened this issue Jun 21, 2021 · 15 comments
Labels
enhancement New feature or request

Comments

@mvdan
Copy link
Member

mvdan commented Jun 21, 2021

In the context of #276 (comment), I've been thinking that there are scenarios where one might want to obfuscate only some packages: when publishing or packaging an obfuscated library, to be consumed as regular Go packages via source code.

We do sort of support this already, via something like:

GOPRIVATE=my.corp/some/library garble -debugdir=out build my.corp/some/library/...

Then, the obfuscated source code will be under out/my.corp/some/library.

I think this use case is valid, given that it's not possible (or at least not easy) to distribute Go libraries without distributing source code too. One such example is unidoc, where its libraries like unipdf are obfuscated then pushed to GitHub.

We already have much of the machinery here, and garble is pretty advanced as a Go obfuscator, so I think it makes sense to take it one step further and make this easier to do. For example, via:

garble export out my.corp/some/library/...

Some key differences compared to how we currently obfuscate code:

  • Import paths should remain untouched, for the sake of keeping imports working.
  • Exported names should also remain untouched, such as types and functions.
  • Godoc comments of exported names should be left in place.
  • License comments should be left in place.

Internal packages such as my.corp/some/library/internal/foo would still be fully obfuscated, including changing their API names and import paths. That could be a way to tell what packages must remain importable and usable.

Otherwise, I think obfuscation could stay the same: stripping unexported names, position information, most comments, etc.

We might have to tweak our obfuscator to collapse newlines, too - right now, it does that via /*line comments for the compiler, but newlines in the printed source code remain, since those don't affect binaries. They affect published source code, though - and it's also likely that those compiler directives shouldn't be present in this new mode.

@mvdan
Copy link
Member Author

mvdan commented Jun 21, 2021

It's unclear to me if this means we should keep GOPRIVATE or GOGARBLE support around.

On one hand, it makes sense to keep GOGARBLE as a configuration option, because garble export out my.corp/some/library/... is implicitly using GOGARBLE=my.corp/some/library. We'd need to keep support for only obfuscating some packages but not others, too.

On the other hand, would anyone ever set GOGARBLE directly? When building manually, one would (presumably) always want to obfuscate all packages. When exporting a library like the example above, one would just want to obfuscate the library's packages.

One possible use case for GOGARBLE would be to emulate the garble export configuration for other commands, like GOGARBLE=my.corp/some/library go test my.corp/some/library/.... However, note that obfuscation of binaries would be different, so this wouldn't be testing the same code that got exported. Perhaps we could add something like garble export-test in the future.

Right now, I'm still leaning towards phasing out GOGARBLE/GOPRIVATE support.

@mvdan
Copy link
Member Author

mvdan commented Jun 21, 2021

Also, I think this command should be akin to garble build, in that packages should be built as they are obfuscated to ensure they haven't been broken in ways that trigger the parser or typechecker. We could later consider an option to skip building and just obfuscate.

mvdan added a commit to mvdan/garble-fork that referenced this issue Nov 16, 2021
In the added test case, "garble -literals build" would fail:

	--- FAIL: TestScripts/literals (8.29s)
		testscript.go:397:
			> env GOPRIVATE=test/main
			> garble -literals build
			[stderr]
			# test/main
			Usz1FmFm.go:1: cannot call non-function string (type int), declared at Usz1FmFm.go:1
			Usz1FmFm.go:1: string is not a type
			Usz1FmFm.go:1: cannot call non-function append (type int), declared at Usz1FmFm.go:1

That is, for input code such as:

	var append int
	println("foo")
	_ = append

We'd end up with obfuscated code like:

	var append int
	println(func() string {
		// obfuscation...
		x = append(x, ...)
		// obfuscation...
		return string(x)
	})
	_ = append

Which would then break, as the code is shadowing the "append" builtin.
To work around this, always obfuscate variable names, so we end up with:

	var mwu1xuNz int
	println(func() string {
		// obfuscation...
		x = append(x, ...)
		// obfuscation...
		return string(x)
	})
	_ = mwu1xuNz

This change shouldn't make the quality of our obfuscation stronger,
as local variable names do not currently end up in Go binaries.
However, this does make garble more consistent in treating identifiers,
and it completely avoids any issues related to shadowing builtins.

Moreover, this also paves the way for publishing obfuscated source code,
such as burrowers#369.

Fixes burrowers#417.
mvdan added a commit to mvdan/garble-fork that referenced this issue Nov 16, 2021
In the added test case, "garble -literals build" would fail:

	--- FAIL: TestScripts/literals (8.29s)
		testscript.go:397:
			> env GOPRIVATE=test/main
			> garble -literals build
			[stderr]
			# test/main
			Usz1FmFm.go:1: cannot call non-function string (type int), declared at Usz1FmFm.go:1
			Usz1FmFm.go:1: string is not a type
			Usz1FmFm.go:1: cannot call non-function append (type int), declared at Usz1FmFm.go:1

That is, for input code such as:

	var append int
	println("foo")
	_ = append

We'd end up with obfuscated code like:

	var append int
	println(func() string {
		// obfuscation...
		x = append(x, ...)
		// obfuscation...
		return string(x)
	})
	_ = append

Which would then break, as the code is shadowing the "append" builtin.
To work around this, always obfuscate variable names, so we end up with:

	var mwu1xuNz int
	println(func() string {
		// obfuscation...
		x = append(x, ...)
		// obfuscation...
		return string(x)
	})
	_ = mwu1xuNz

This change shouldn't make the quality of our obfuscation stronger,
as local variable names do not currently end up in Go binaries.
However, this does make garble more consistent in treating identifiers,
and it completely avoids any issues related to shadowing builtins.

Moreover, this also paves the way for publishing obfuscated source code,
such as burrowers#369.

Fixes burrowers#417.
lu4p pushed a commit that referenced this issue Nov 16, 2021
In the added test case, "garble -literals build" would fail:

	--- FAIL: TestScripts/literals (8.29s)
		testscript.go:397:
			> env GOPRIVATE=test/main
			> garble -literals build
			[stderr]
			# test/main
			Usz1FmFm.go:1: cannot call non-function string (type int), declared at Usz1FmFm.go:1
			Usz1FmFm.go:1: string is not a type
			Usz1FmFm.go:1: cannot call non-function append (type int), declared at Usz1FmFm.go:1

That is, for input code such as:

	var append int
	println("foo")
	_ = append

We'd end up with obfuscated code like:

	var append int
	println(func() string {
		// obfuscation...
		x = append(x, ...)
		// obfuscation...
		return string(x)
	})
	_ = append

Which would then break, as the code is shadowing the "append" builtin.
To work around this, always obfuscate variable names, so we end up with:

	var mwu1xuNz int
	println(func() string {
		// obfuscation...
		x = append(x, ...)
		// obfuscation...
		return string(x)
	})
	_ = mwu1xuNz

This change shouldn't make the quality of our obfuscation stronger,
as local variable names do not currently end up in Go binaries.
However, this does make garble more consistent in treating identifiers,
and it completely avoids any issues related to shadowing builtins.

Moreover, this also paves the way for publishing obfuscated source code,
such as #369.

Fixes #417.
@mvdan mvdan mentioned this issue May 5, 2023
@lu4p
Copy link
Member

lu4p commented Jul 15, 2023

I think unidocs obfuscation is cute, I might try to reverse it tomorrow.

@mvdan
Copy link
Member Author

mvdan commented Jul 15, 2023

I just realised what is the biggest hurdle with garble export: build tags. Right now, the way we obfuscate code relies on go/types and go/ssa, both of which "compile" each package with a set of types given some specific build tags.

But when exporting, we want to obfuscate all files - no matter the GOOS, GOARCH, or any other build tags. If a package has foo_linux.go and foo_windows.go, the user likely wants both to be obfuscated and exported.

I can think of two options:

  1. Only obfuscate and export for one set of build tags, e.g. the current GOOS and GOARCH. This is easy, but rather limiting.

  2. Have the user give us a list of build tag sets, e.g. garble export -tags="linux,amd64 windows,amd64 darwin,arm64", and we perform the obfuscation for each of them - joining the results at the end so that all files are obfuscated. Note that simply deduplicating obfuscated files won't be enough, since different build tags will currently result in different obfuscation. We would have to teach garble to obfuscate the same way for each of the build tag sets. This would also slow down the export, as we would do one obfuscated build per tag set.

A third option would be to obfuscate in some way that doesn't rely on build tags. But this would require giving up go/types and go/ssa, and effectively rewriting garble to be significantly less powerful.

@pagran
Copy link
Member

pagran commented Jul 15, 2023

2 - what if we consider each tag as a unique build and put all files obfuscated with this tag under this tag as well? This would highly increase the total size (and obfuscation time), but make the resulting package universal and with unique obfuscation for each tag (tags combination?).

@mvdan
Copy link
Member Author

mvdan commented Jul 16, 2023

This could multiply the size of Go modules though, so I'd rather avoid it unless there are significant benefits. I don't see any significant benefits, given that I expect that most users would want to publish all the "build tag variants" alongside each other. Spotting the common bits between them wouldn't be that hard.

@jitcor
Copy link

jitcor commented Dec 27, 2023

I want to publish a private Go library without exposing the source code. I really need this feature and I hope for support.

@wubin1989
Copy link

Any news? I really need this feature, too!

@lu4p
Copy link
Member

lu4p commented Apr 18, 2024

This issue is only aspirational no one is actually working on it or planning to.

@wubin1989 @jitcor
If you really need this you may need to do it yourself or fund someone to do it.

@jitcor
Copy link

jitcor commented May 14, 2024

So garble can realize the effect similar to unipdf? Or java proguard the kind of obfuscation effect, through the obfuscation configuration file to add obfuscation rules.

@lu4p
Copy link
Member

lu4p commented May 14, 2024

Yes garble export would generate code in a similar manner how unipdf is obfuscated.

@wubin1989
Copy link

Yes garble export would generate code in a similar manner how unipdf is obfuscated.

@lu4p do you mean we just need to add "export" command line subcommand, the feature has already implemented, right?

@lu4p
Copy link
Member

lu4p commented May 15, 2024

No this is not implemented

@yardenlaif
Copy link

I implemented something (minimal) like this here: https://github.com/yardenlaif/balagan
I could possibly add it to garble if it makes sense

@wubin1989
Copy link

@yardenlaif stared and forked. Many thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Development

No branches or pull requests

6 participants