-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Introduce storage factory framework and composable CLI #625
Conversation
32e84fa
to
7fc4b69
Compare
plugin/storage/factory.go
Outdated
if f.depStoreType == "" { | ||
f.depStoreType = f.spanStoreType | ||
} | ||
uniqueTypes := map[string]struct{}{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what black magic is this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
your example fails with explicit literals, but uq
works correctly.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't understand the why though, at runtime uq is also just literals no? so uq should fail too?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well, it doesn't fail if you remove uq2 from the example. If I were to guess, it's due to compiler optimization. When you init the map with the actual literals, compiler can check for uniqueness, and it's a reasonable behavior to fail since such statement is more likely to be a typo. But when you init with dynamic values, compiler cannot check anything, and the initialization is reduced to a sequence of m[k] = v
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
pretty nice. Things like this seem like they're a breaking change if someone had written their own main function to startup Jaeger. Do we need to cut a major release for things like this?
// See also | ||
// | ||
// plugin.Configurable | ||
type Factory interface { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe a Close() function too?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
no use case for it right now, if we need it in the future I'd rather use io.Closer
@@ -0,0 +1,44 @@ | |||
// Copyright (c) 2017 Uber Technologies, Inc. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can forsee us needing to support other storage factories like SamplingFactory. Would that mean we keep adding functions to this interface or create a new interface for each factory? If the latter, wouldn't it make more sense to split this factory into SpanFactory and DependencyFactory?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess the question is - how likely is it for someone to run SpanStore on C, DepsStore on ES, SamplingStore on smth else, etc.?
We could split the interfaces, but their implementations would still need to be in a single struct, both for meta-factory and for individual backends.
plugin/storage/factory.go
Outdated
if f.depStoreType == "" { | ||
f.depStoreType = f.spanStoreType | ||
} | ||
uniqueTypes := map[string]struct{}{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't understand the why though, at runtime uq is also just literals no? so uq should fail too?
A bit of a philosophical question. Strictly speaking, Jaeger is distributed in binary form, we haven't made commitment to preserve the internal APIs. Yes, some people may have forked and extended at the source level, because we don't have a good plugin framework, but I think these changes actually make it easier to extend, even if they are breaking. And this change is a step towards supporting external plugins. |
@objectiser @pavolloffay what do you think? |
988f748
to
b11efc9
Compare
I am looking at this now, so far it looks fine. Somehow I wanted to skip env var but I don't have a good alternative solution for it |
Haven't looked in detail at the code but the general approach looks good. In terms of the env var vs command line - although viper is used to generally manage the command line options, couldn't a hard coded solution just be used to scan for this one command line option (if not provided via env var), just to keep a consistent approach with all options? |
jc "github.com/jaegertracing/jaeger/thrift-gen/jaeger" | ||
zc "github.com/jaegertracing/jaeger/thrift-gen/zipkincore" | ||
) | ||
|
||
// standalone/main is a standalone full-stack jaeger backend, backed by a memory store | ||
func main() { | ||
if os.Getenv(storage.SpanStorageEnvVar) == "" { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Isn't dependency storage missing?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it defaults to span storage, I will add a comment
cmd/standalone/main.go
Outdated
if os.Getenv(storage.SpanStorageEnvVar) == "" { | ||
os.Setenv(storage.SpanStorageEnvVar, "memory") | ||
} | ||
storageFactory, err := storage.NewFactory() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What about passing values of SpanStorageType
and DependencyStorageType
to the factory and abstract the factory of env vars?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the main idea is to remove the dispatching logic from main(). It's certainly possible to de-couple the actual dispatching factory from the source of storage types parameters, e.g. something like
storageFactory, err := storage.NewFactory(storage.StorageTypesFromEnv())
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see, the code snippet looks better.
@objectiser yes, we can manually scan the Another interesting alternative might be to converge to a single binary where |
b11efc9
to
34a1380
Compare
34a1380
to
4661cb6
Compare
I ran a bunch of other experiments and still think env vars are the best option, because:
|
b0dcb36
to
00e6797
Compare
Signed-off-by: Yuri Shkuro <[email protected]>
8b41562
to
db3913a
Compare
Signed-off-by: Yuri Shkuro <[email protected]>
I ended up adding manual parsing for the deprecated switch, which will provide better backwards compatibility, but it would still not work with the existing Kubernetes templates since they are actually using config files |
Main changes:
--span-storage.type
CLI flagSPAN_STORAGE
(and optionallyDEPENDENCY_STORAGE
)storage.Factory
interface for creating different storage components likespanstore.Reader
plugin/storage.Factory
that instantiates concrete factories based on the above env varsplugin.Configurable
interface that allows them to define custom CLI flags and to be initialized from Viperenv
that prints help about configuring via env varsRelated to #422, #608