Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fleet] Prepare Fleet API for promotion to GA #123150

Closed
15 of 21 tasks
Tracked by #108524
joshdover opened this issue Jan 17, 2022 · 18 comments
Closed
15 of 21 tasks
Tracked by #108524

[Fleet] Prepare Fleet API for promotion to GA #123150

joshdover opened this issue Jan 17, 2022 · 18 comments
Assignees
Labels
enhancement New value added to drive a business result Team:Fleet Team label for Observability Data Collection Fleet team v8.8.0

Comments

@joshdover
Copy link
Contributor

joshdover commented Jan 17, 2022

One design goal of Fleet is to be 100% API-based in our business logic. This allows for easy external integration and management of the Fleet platform and paves the way for future 1st class features, such as Infrastructure-as-Code (IaC) tooling support (Terraform, etc.).

Today, Fleet publishes an OpenAPI spec which is currently marked as experimental.

There will be a beta period prior to the Fleet API's general availability. This beta period is intended to provide internal and external consumers of the Fleet API some time to begin adopting new features, and to provide feedback and input on the transition to General Availability.

1️⃣ Required for beta

Beta requirements are now fulfilled.

These issues must be completed before the Fleet API can enter a formal beta period.

"Recipe" documentation

We need some level of "how to use this API" documentation around the Fleet API that captures common operations and processes. These docs should include all of the following operations:

  • Add new integration
  • Update an Integration Config
  • Upgrade installed integration
  • Delete an integration config
  • Delete installed integrations
  • Create new agent policy
  • Update agent policy
  • Reassign agent
  • Update agent config
  • Upgrade agent version
  • Unenroll agent
  • Delete agent policy
  • Delete agent

Package Policy API improvements and breaking changes

The package policy API is difficult to use programmatically. It requires too much knowledge of the structure of specific integrations and their inputs. It should improved top-to-bottom to better support automation use cases at scale. Several of the changes below will be considered breaking, so we need to handle these before we can consider the Fleet API ready for a beta period.

Audit OpenAPI spec and remove unused endpoints

Any endpoint that is completely unused or doesn't have a use-case should be removed prior to beta to limit the API's surface area.

Update bulk actions API

In supporting #141567, we'll likely be making breaking changes to the bulk actions API response structure. We should make sure these changes land and documentation is accurate before we consider the Fleet API beta. cc @juliaElastic.

2️⃣ Required for GA

The GA requirements are now fulfilled

These issues don't block the initial beta, but they should be completed prior to the Fleet API transition into General Availability.

First-party reference documentation

Currently, Fleet maintains its own set of OpenAPI documentation in GitHub here. This documentation is not hosted or bundled with any Elastic-owned documentation. Typically, we use a link to a Swagger UI provider to allow for exploring the API spec. This is not scalable, and we need to provide first-party documentation for the Fleet API in order to promote it to GA.

It'd be great to get API docs on https://docs.elastic.co, which supports MDX docs. It's not clear, though, how we'd integrate our OpenAPI spec files into this docs system currently. https://github.com/elastic/docs.elastic.co/issues/37 tracks the implementation for a React component that consumes OpenAPI specs and produces a UI within the docs.elastic.co site.

[No longer required] API versioning + breaking change management

Note
The decision making process for API versioning for Kibana API's is still ongoing as we start to plan for the Elastic Serverless offering. The Fleet API's promotion to GA should not be blocked by this effort.

Another concern with the Fleet API moving forward is limiting breaking changes. There are two distinct efforts related to this:

  1. Implementation of versioning within the Kibana API - this is a higher level effort spanning all Kibana plugins that expose an API
  2. Automation and patterns around preventing/mitigating breaking changes from landing in the API
  • Determine a policy for deprecations and breaking changes that is compatible with Elastic's versioning policy for 8.0+
  • Depending on outcome of above, add a versioning scheme (see prior art from Kibana Platform)
  • Prevent the possibility of unplanned breaking changes by adding CI checks for detecting breaking changes on the OpenAPI spec

Other API fixes

Optional

These are "nice to have" improvements that aren't necessarily required for promoting the Fleet API to beta or GA.

@joshdover joshdover added enhancement New value added to drive a business result Team:Fleet Team label for Observability Data Collection Fleet team labels Jan 17, 2022
@juliaElastic
Copy link
Contributor

#121485 has a few downstream dependencies, does Fleet API GA have to depend on it?

@joshdover
Copy link
Contributor Author

Any breaking changes needed for RBAC evolution

@juliaElastic I don't believe we identified any needs for breaking changes on the HTTP API to support our RBAC evolution goals. Can you confirm and if so, check this one off the list?

@juliaElastic
Copy link
Contributor

Any breaking changes needed for RBAC evolution

@juliaElastic I don't believe we identified any needs for breaking changes on the HTTP API to support our RBAC evolution goals. Can you confirm and if so, check this one off the list?

Yes, we don't plan any breaking changes for RBAC. I'll close that one out.

@joshdover joshdover changed the title [Fleet] Promote Fleet API to GA [Fleet] Prepare Fleet API for promotion to GA Jul 19, 2022
@jen-huang
Copy link
Contributor

jen-huang commented Jul 21, 2022

@juliaElastic @kpollich Would it make sense to also include these?:

@jen-huang
Copy link
Contributor

@kpollich We've had a couple of recent issues related to the data stream API, we've discussed putting some time here to make it more fault-tolerant. Could improvements here be a candidate? Related issues:

@juliaElastic
Copy link
Contributor

@juliaElastic @kpollich Would it make sense to also include these?:

I think the Optimize issue does not change the API itself, so it can be done independently of moving to GA.
The deprecate issue can be included in this.

@juliaElastic
Copy link
Contributor

@joshdover
Copy link
Contributor Author

We may want to consider this one as well as a fix required for GA: #137005

@kpollich
Copy link
Member

@joshdover Thanks - added to the description.

@kpollich
Copy link
Member

It'd be great to integrate Fleet's OpenAPI docs with a first party Elastic docs site, preferably https://docs.elastic.co/. OpenAPI support isn't quite ready yet, but is planned and tracked here https://github.com/elastic/docs.elastic.co/issues/37.

We should be doing everything we can do clean up and improve our existing OpenAPI files to make sure they're ready for integration with a first-party docs site when possible.

@jlind23
Copy link
Contributor

jlind23 commented Sep 20, 2022

@nimarezainia can we make sure that the docs are good to be shipped before 8.5 official release?

@jlind23
Copy link
Contributor

jlind23 commented Oct 5, 2022

Update: Some more docs details will be added to: elastic/observability-docs#2180 then our writers will take it from there and create some getting started materials.

As the steps needed to get it to a Beta stage are now all done, i'll move this issue to a later iteration.

cc @kpollich @jen-huang

@kpollich
Copy link
Member

We need to make sure once we formally enter a beta period that any public-facing docs for the Fleet API or its OpenAPI spec to say beta rather than experimental.

@nimarezainia
Copy link
Contributor

marking it beta is the easy part. What do we want to collect from customers? a sample workflow or test guide would be great.

@kpollich
Copy link
Member

Relevant docs/OpenAPI issue: #137240

@kpollich
Copy link
Member

kpollich commented Feb 7, 2023

We need to address some issues with Fleet's OpenAPI spec to unblock the docs team: #149990

@jlind23
Copy link
Contributor

jlind23 commented Feb 14, 2023

@kpollich converting this to a meta issue as we already added the follow up ones in our sprint.

@jen-huang
Copy link
Contributor

@kpollich Per discussion, going to close this issue for planning purposes. Feel free to reference any follow up docs issues needed.

juliaElastic added a commit that referenced this issue Apr 26, 2023
## Summary

Small pr to remove Experimental from the Fleet openapi README.

Related to #123150
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New value added to drive a business result Team:Fleet Team label for Observability Data Collection Fleet team v8.8.0
Projects
None yet
Development

No branches or pull requests

6 participants