[Fleet] Improve public API for Agent Policy, Package Policy

[nchaulet]

Description

While working on proof of concept of a terraform provider for Fleet I had to work with the Fleet APIs outside of Kibana. I found these APIs really hard to consume as you need to provide to much information and have a lot of knowledge on how packages are structured.

To add a package you need to provide things that I think are not necessary like the package title, all the information on the variables (we can compute that)

Potential solutions

I think we can a new API that allow to return easily the inputs,streams, and variables for a package (eventually we can use that to display how to programatically configure an integration in the UI)

We could create a new version of the package policy API that accept something more similar to the preconfigure API and we can compute a lot more things based on the package server side (it will be less error prone too)

Something like this

{
  "package": {
    "name": "fleet_server",
    "version": "0.0.1"
  },
  "name": "Fleet Server",
  "inputs": [
    {
      "type": "fleet-server",
      "keep_enabled": true
    }
  ]
}

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[mostlyjason]

@nchaulet would this require a breaking change to switch to this improved format that we should aim for 8.0 or can we add it as an enhancement later?

[nchaulet]

@nchaulet would this require a breaking change to switch to this improved format that we should aim for 8.0 or can we add it as an enhancement later?

I think making this a breaking change is probably a better option so we will not have to maintains the old API for the whole 8.x cycle. Also in my opinion providing a good API here is a huge enhancement for users that can open the door to more IAC tooling.

[juliaElastic]

@nchaulet I had a look at this, and I think we can't remove as many properties from the package_policies API, there are a lot of options that can be customised from UI, so those cannot be removed, e.g. inputs.streams or inputs.vars.

Wouldn't it be better to create a parallel create API that would take the minimal information and enrich the rest from server side?

Also, is this a priority issue? Since it is not there in current Iteration.

[kpollich]

@nchaulet I had a look at this, and I think we can't remove as many properties from the package_policies API, there are a lot of options that can be customised from UI, so those cannot be removed, e.g. inputs.streams or inputs.vars.

Wouldn't it be better to create a parallel create API that would take the minimal information and enrich the rest from server side?

Also, is this a priority issue? Since it is not there in current Iteration.

I think the idea here would be to make those fields optional rather than required, so we can support a more minimal payload when users aren't submitting data via the policy editor. The logic here should be "anything that can be computed server-side from the package info is optional" - if that makes sense.

I am not in favor of a simplified special case endpoint that accepts this minimal payload, as that seems like introducing multiple sources of truth.

[juliaElastic]

@kpollich I see your point, so it sounds like this won't be a breaking change?
I would rather not introduce breaking change in 8.0 at this point, see this discussion: elastic/elastic-agent#111

The only tricky part might be to merge together properties overwritten from UI and default values.

[juliaElastic]

Regarding the update API, shouldn't we use HTTP PATCH instead of making fields optional in PUT? Though it would mean to have 2 APIs for update.

[juliaElastic]

@nchaulet did you mean to change anything in Agent Policy API too? I had a check and only name and namespace is required when creating or updating an agent policy, see NewAgentPolicy
That should be sufficient, do you agree?

[criamico]

Adding here the PR as it was not linked: #119739