Generic HTTP datasource

[blotus]

Add a new datasource that will expose a generic HTTP server to receive JSON formatted logs.

The datasource will be agnostic regarding the exact content of the logs, and if "complex" objects are received (for example an array of items), it will rely on the transform features to split the items.

Requirements:

• Authentication: Support basic auth and custom headers name/values (in the case of header, it should probably allow an AND between multiples headers as some services seem to send multiple auth-related headers). In P2, mTLS support.
• TLS Support

Optional:

• Source IP filtering

Questions:

• Input format: At the beginning, only support JSON data. Should we only support "one-shot" requests (ie, every items in an array) or NDJSON (one item per line).
• Maximum body size: should the user be able to configure a max body size to avoid potential memory issues when dealing with a lot of logs ?
• Chunk size configuration ?(mostly for streaming purpose)
• Compression support ? Rely on the Content-type provided by the client ?
• Stream support for decoding ? (ie, client open a persistent connection and stream data to crowdsec).
• Configuration of HTTP response code on success and success headers.
• Timeout configuration (should we kill the connection if no data is transmitted after some time)
• How to support client that perform a health-check ? Have a configuration with a custom route + return code ?

[github-actions]

@blotus: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

1. Check Crowdsec Documentation to see if your issue can be self resolved.
2. You can also join our Discord.
3. Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[github-actions]

@blotus: There are no 'kind' label on this issue. You need a 'kind' label to start the triage process.

• /kind feature
• /kind enhancement
• /kind refactoring
• /kind bug
• /kind packaging

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[LaurenceJJones]

Mark as solved due to #3294