Add error count metrics

[excalq]

Resolves #109. Adds metrics, test expected metrics, fixtures, structs, and snapshots.

Adds new metrics as follows:

# HELP logstash_stats_pipeline_plugin_bulk_requests_errors Number of bulk request errors.
# TYPE logstash_stats_pipeline_plugin_bulk_requests_errors counter
logstash_stats_pipeline_plugin_bulk_requests_errors{pipeline="telemetry-json",plugin="opensearch",plugin_id="6c1a6024",plugin_type="output"} 2
logstash_stats_pipeline_plugin_bulk_requests_errors{pipeline="telemetry-json",plugin="stdout",plugin_id="acba4948",plugin_type="output"} 0
logstash_stats_pipeline_plugin_bulk_requests_errors{pipeline="telemetry-legacy",plugin="opensearch",plugin_id="059ed188",plugin_type="output"} 0
logstash_stats_pipeline_plugin_bulk_requests_errors{pipeline="testing-stdout",plugin="stdout",plugin_id="fd7f574d",plugin_type="output"} 0

# HELP logstash_stats_pipeline_plugin_bulk_requests_responses Bulk request HTTP response counts by code.
# TYPE logstash_stats_pipeline_plugin_bulk_requests_responses counter
logstash_stats_pipeline_plugin_bulk_requests_responses{code="200",pipeline="telemetry-json",plugin="opensearch",plugin_id="6c1a6024",plugin_type="output"} 2
logstash_stats_pipeline_plugin_bulk_requests_responses{code="200",pipeline="telemetry-legacy",plugin="opensearch",plugin_id="059ed188",plugin_type="output"} 2

# HELP logstash_stats_pipeline_plugin_documents_non_retryable_failures Number of output events with non-retryable failures.
# TYPE logstash_stats_pipeline_plugin_documents_non_retryable_failures counter
logstash_stats_pipeline_plugin_documents_non_retryable_failures{pipeline="telemetry-json",plugin="opensearch",plugin_id="6c1a6024",plugin_type="output"} 2
logstash_stats_pipeline_plugin_documents_non_retryable_failures{pipeline="telemetry-json",plugin="stdout",plugin_id="acba4948",plugin_type="output"} 0
logstash_stats_pipeline_plugin_documents_non_retryable_failures{pipeline="telemetry-legacy",plugin="opensearch",plugin_id="059ed188",plugin_type="output"} 0
logstash_stats_pipeline_plugin_documents_non_retryable_failures{pipeline="testing-stdout",plugin="stdout",plugin_id="fd7f574d",plugin_type="output"} 0

# HELP logstash_stats_pipeline_plugin_documents_successes Number of successful bulk requests.
# TYPE logstash_stats_pipeline_plugin_documents_successes counter
logstash_stats_pipeline_plugin_documents_successes{pipeline="telemetry-json",plugin="opensearch",plugin_id="6c1a6024",plugin_type="output"} 0
logstash_stats_pipeline_plugin_documents_successes{pipeline="telemetry-json",plugin="stdout",plugin_id="acba4948",plugin_type="output"} 0
logstash_stats_pipeline_plugin_documents_successes{pipeline="telemetry-legacy",plugin="opensearch",plugin_id="059ed188",plugin_type="output"} 2
logstash_stats_pipeline_plugin_documents_successes{pipeline="testing-stdout",plugin="stdout",plugin_id="fd7f574d",plugin_type="output"} 0

Note, as I'm pre-populating all counters with 0 (except for bulk HTTP response codes), there is some additional nil checking and looping. Does this merit additional tests?

[codecov]

Codecov Report

Patch coverage: 100.00% and no project coverage change.

Comparison is base (607efa9) 100.00% compared to head (4e6143f) 100.00%.

Additional details and impacted files

@@            Coverage Diff            @@
##            master      #110   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           12        12           
  Lines          496       513   +17     
=========================================
+ Hits           496       513   +17     

Impacted Files	Coverage Δ
collectors/nodestats/pipeline_subcollector.go	100.00% <100.00%> (ø)

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[kuskoman]

@excalq lint seems to fail:

Run golangci-lint run
Error: collectors/nodestats/pipeline_subcollector.go:117:6: SA4022: the address of a variable cannot be nil (staticcheck)
		if &output.Documents != nil {
		   ^
Error: collectors/nodestats/pipeline_subcollector.go:122:6: SA4022: the address of a variable cannot be nil (staticcheck)
		if &output.BulkRequests != nil {
		   ^
Error: Process completed with exit code 1.

[excalq]

I learned something new in Golang: https://www.sohamkamani.com/golang/omitempty/
I'm going to remove the omitempty and the conditional checks, then we should ok.

[excalq]

@kuskoman I've updated the structs snapshot to fix Github's checks. This should be ready for review.

[kuskoman]

@excalq after merging master to the branch integration test seems to fail, could you re-generate the metrics?

[excalq]

@excalq after merging master to the branch integration test seems to fail, could you re-generate the metrics?

@kuskoman I'm having trouble reproducing this. make verify-metrics returns 0, and tests pass. What am I missing?