-
Notifications
You must be signed in to change notification settings - Fork 8.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Application Usage] Better SO management #87840
Comments
Pinging @elastic/kibana-core (Team:Core) |
It would be nice to have a bulk delete api for savedobjects too. |
tbh, I'm surprised we don't have this yet. However given the technical implications, it could probably benefit from its own issue |
++ @pgayvallet I think it already exists: #30503 |
@alexfrancoeur due to improvements in the saved objects API, we no longer need the transactional documents we were using for application usage, as we will be able to directly increment the daily application usage doc when receiving reports from the client-side. Now the question is: is it acceptable to totally drop support for the To understand the implications: Daily and total documents for application usage were added in The other option would be to still perform the improvement by directly writing to Note that in both cases, the cc @afharo: did I say anything wrong here 😅 ? |
Generally, if transactional support will be going away anyway, I'd prefer to make any drastic changes sooner than later. That being said, I'm not totally sure I understand the implications to analysis by moving off of transactional documents. Today the end result for analysis ends up being cluster wide time spent in and clicks into applications over a few given timeframes in a cluster snapshot. We are leveraging application usage telemetry to identify adoption of applications, so there may be a larger impact that we should discuss.
For analysis, we only have aggregate counts, so if individual transactions are being captured, I don't believe we ever see them. When you say lose the data, do you mean a reset? Meaning, the transactional documents go away, and we starting incrementing from 0 at the point of upgrade. I'd like to make this transition as seamless as possible. If we're talking about resetting the counters to 0 on an upgrade, I wonder if we can "fill in" some of these gaps on ingest somehow. I think it might be worth a quick live sync and include a few folks not on this issue to truly understand impact before making this decision. Happy to set something up. cc: @thesmallestduck |
Hi @alexfrancoeur, I tried to draw a diagram of the differences across versions: In 7.7 & 7.8: we only had transactional documents (containing 3 minutes worth of data from each user's browsers). Then the collector would read all of them and aggregate them into the last 7/30/90 days/total counters. With this strategy, we caused an explosion in the number of SOs. That explosion led to some highly used clusters failing to report these metrics because the transactional documents were more than 10k (apart from some other side effects, like migrations taking too long or import/export issues). In 7.9, due to an explosion in the number of SOs, we applied a rollup logic that runs every 30 minutes to pre-aggregate the transactional documents to daily objects. Reducing the number of transactional SOs to 10 docs per user max. We were still using transactional docs to avoid concurrency issues when editing the daily counters. Now that we have APIs that allow us to do so, we are considering atomically increasing the counters of the daily objects without relying on the rollup logic. So we won't generate any transactional documents anymore. The question is: should we keep the rollup logic to migrate the transactional documents to daily for the clusters that will upgrade to 7.13+? I hope I shed some light 😇 |
FWIW, as it didn't impact much of the code anyway, I preserved the transactional documents in #94923. The transactional->daily rolling is done only once, at startup, to preserve the BWC during migration from older versions. |
Now that SO's
incrementCounter
allows atomic custom increases (and for multiple fields at once), we can get rid of the transactional documents.Maybe we can keep the rollup logic throughout the rest of the 7.x series to make sure we rollup the transactional documents before completely removing them.
Alternatively, we could explore a new way of reporting this type of usage. Similar to how UI Counters work, we could only provide the deltas for the last 3 days. That would require a common API on the server-side to unify their logic (#81645)
The text was updated successfully, but these errors were encountered: