Proposal: Native Multi-tenancy Support #4055
Conversation
Signed-off-by: Abhishek357 <[email protected]>
There was a problem hiding this comment.
Amazing work!
And good idea to put things into GitHub - this way we can be easier aware of what's the proposal status.
Very well format proposal and I love the verbose sentences. Thanks @Abhishek357 💪🏽 Some small things to figure out still - in comments.
| ### Goals | ||
|
|
||
| 1. Introduce a mechanism to identify tenants to support multi-tenancy use cases. | ||
| 2. Isolate all the incoming query requests. |
There was a problem hiding this comment.
Nice! Is query API the only goal? If yes let's change this to:
| 2. Isolate all the incoming query requests. | |
| 2. Isolate the incoming read and write API requests across tenants if needed. |
If only query then let's make sure to put Metadata, Exemplars APIs etc to ### Non-Goals section
There was a problem hiding this comment.
yes, for now, let's focus on query API.
other opinions are welcomed
There was a problem hiding this comment.
We can make a note about store API(only the querier part) that we are focussing on.
There was a problem hiding this comment.
Well it would be nice to put it in other APIs too, but we can extend that later. Just important @Abhishek357 to decide now (:
|
|
||
| ### Solution | ||
|
|
||
| * Introduce a new flag to enable or disable the multi-tenancy mode. |
There was a problem hiding this comment.
I would love to have something more organic. E.g if you specify a certain tenant label the mode will be enabled (:
There was a problem hiding this comment.
Would be nice to say exactly how this flag could named, or the injection of tenant ID header/flag/parameter
There was a problem hiding this comment.
Would be nice to say exactly how this flag could named
receiver already have tenant flags and we will adopt them to be consistent.
There was a problem hiding this comment.
Ack, can you add note about that? And maybe provide the flag name here?
|
|
||
| * Introduce a new flag to enable or disable the multi-tenancy mode. | ||
| * When this flag is enabled we will modify the metrics which can be used to track usage characteristics of a tenant by injecting the tenant’s label (a const. Label ). | ||
| * We will make sure that the query request received has a tenant label and if not we have some default tenant label configured. |
There was a problem hiding this comment.
Why not blocking if tenant is not specified?
There was a problem hiding this comment.
we can but we can also have a default tenant label to use when none is specified.
There was a problem hiding this comment.
If we have switched on multi-tenancy mode, let's have no default label. Users know that if they have enabled multitenancy, they should provide a label for that, and not depend on a default tenant label. If they want a default behaviour, then it does not make sense to have multitenancy enabled. WDYT?
There was a problem hiding this comment.
Yes, I think that would be good. But we need to ensure how we do that on Receiver too (it would be good to do so)
| * We will enforce the tenant labels inside the Thanos code base(i.e we assume the authentication is done and the query has access to the tenant's data which it specifies) when we run components with a specified mode(i.e multi-tenancy flag is enabled). We will no longer require any label enforcing proxy. | ||
| * Support the cross tenant queries using regular expressions. | ||
| * Store API will advertise the tenant label and this how it will help us - | ||
| * Let's assume a case of hard tenancy, we have two store-gateway advertising different sets of tenants. |
There was a problem hiding this comment.
Do we plan to have a special Info method field in proto. If yes, do you think we could design it here in this doc?
Alternatively, why not reusing LabelSet? (I am not suggesting we should, but it would be nice to mention why not e.g in Alternatives section below)
There was a problem hiding this comment.
Is this suggestion about this proposal - https://thanos.io/tip/proposals/202101_endpoint_discovery.md/
| ### Solution | ||
|
|
||
| * Introduce a new flag to enable or disable the multi-tenancy mode. | ||
| * When this flag is enabled we will modify the metrics which can be used to track usage characteristics of a tenant by injecting the tenant’s label (a const. Label ). |
There was a problem hiding this comment.
For storing the tenant external label, we follow the current implementation of how Thanos does multitenancy, but for querying, we certainly need to define the implementation details 👍
| 2. We can use a prom-label proxy for label enforcement. | ||
| * It does not deal with cross-tenant queries. But with the proposed approach we can use regular expression for cross tenant queries. | ||
| * Difficult to configure and increases complexity. | ||
|
|
There was a problem hiding this comment.
- Alternative, reuse LabelSet in InfoAPI for Store and others API tenancy info propagation
There was a problem hiding this comment.
can you suggest why should we not use it?
There was a problem hiding this comment.
We can, was that a plan? I think we did not talk about advertising tenant labels yet, right? I just proposed the alternative, maybe it's worth moving to main idea, let's think about it (:
kakkoyun
changed the title
|
It'd be great if you could take a look @brancz :) |
Signed-off-by: Abhishek357 <[email protected]>
|
|
||
| 1. Allow admins to obtain tenant information on per tenant queries, operations, and ingestion. | ||
| 2. Sending tenant-specific alerts to different alert managers. | ||
| 3. Support cross tenant views. |
There was a problem hiding this comment.
I think we should elaborate on this, what exactly this use case looks like in practice?
There was a problem hiding this comment.
"User can query for multiple tenants" -> How you user specify those tenants?
cc @onprem
There was a problem hiding this comment.
Ideally we extend those use case (more descriptive) more life examples, showing also the value of this feature
e.g We can enable running the same Querier/Store for multiple shared tenants
|
|
||
| ### Goals | ||
|
|
||
| 1. Introduce a mechanism to identify tenants to support multi-tenancy use cases. |
There was a problem hiding this comment.
I think another goal is to have a way for user to specify which tenant it wants to access?
|
Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward? This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. |
Signed-off-by: Abhishek357 [email protected]