Add integration tests for the shipper output

[rdner]

• An event batch is published from an input to the shipper gRPC server
• An event batch is not dropped when the gRPC server is not available but starts later
• An event batch is not dropped when ResourceExhausted code is returned from the gRPC

[rdner]

@cmacknz @faec

Looking at the interface again, it occurred to me that I never asked one of the questions I had:

elastic-agent-shipper/api/shipper.proto

Lines 13 to 21 in f2aa770

	// Publishes an event via the Elastic agent shipper.
	//
	// Blocks until all processing steps complete and data is written to the queue. Returns a
	// RESOURCE_EXHAUSTED gRPC status code if the queue is full.
	//
	// Inputs may execute multiple concurrent Produce requests for independent data streams.
	// The order in which concurrent requests complete is not guaranteed. Use sequential requests to
	// control ordering.
	rpc PublishEvents(PublishRequest) returns (PublishReply);

is it fair to assume that regardless the returned error I must always check the list of results and mark the listed events as "accepted to the queue"?

I'll give you an example:

1. I sent 50 events to the gRPC server
2. It responds with an error with the gRPC code ResourceExhausted
3. However, it also returns a list of 20 event results

The way I understand it now, I should treat these 20 events as accepted to the queue and I must retry the rest of the 30 events.

Can you please both confirm that this is the right behaviour that we're going to implement on the server?

Also, is this the case with the ResourceExhausted code only, or regardless the error I must always look the results up and retry the unaccepted events?

[cmacknz]

The way I understand it now, I should treat these 20 events as accepted to the queue and I must retry the rest of the 30 events.

Yes, I think it is reasonable for us to handle partially published batches. If we start implementing it and decide it is too difficult or complex we can re-evaluate that.

Also, is this the case with the ResourceExhausted code only, or regardless the error I must always look the results up and retry the unaccepted events?

ResourceExhausted is the most obvious case where some events could succeed but others may fail (because the queue is full). For now you can assume it is the only error with this behaviour.