Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update host metrics receiver to use scraper controller #1949

Conversation

james-bebbington
Copy link
Member

@james-bebbington james-bebbington commented Oct 13, 2020

Update host metrics receiver to use scraper controller removing around 400 lines of code. There is some "scraper factory" code left over that might be able to be moved into the receiverhelper (or scraper) library. Will think about that for next phase.

Resolves #918


Sample Output:

Note in these examples we see quite a lot of errors scraping process data due to not running the Collector in "Administrative mode" on Windows:

Traces (from http://localhost:55679/debug/tracez?zspanname=scraper%2fprocess%2fMetricsScraped&ztype=2&zsubtype=0):

image

Metrics (from http://localhost:8888/metrics):

image

@codecov
Copy link

codecov bot commented Oct 13, 2020

Codecov Report

Merging #1949 into master will increase coverage by 0.09%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1949      +/-   ##
==========================================
+ Coverage   91.92%   92.01%   +0.09%     
==========================================
  Files         282      278       -4     
  Lines       16797    16746      -51     
==========================================
- Hits        15440    15409      -31     
+ Misses        933      921      -12     
+ Partials      424      416       -8     
Impacted Files Coverage Δ
...rnal/scraper/processesscraper/processes_scraper.go 77.77% <ø> (-2.23%) ⬇️
receiver/hostmetricsreceiver/internal/utils.go 100.00% <ø> (ø)
receiver/hostmetricsreceiver/factory.go 86.66% <100.00%> (+11.66%) ⬆️
...eceiver/internal/scraper/cpuscraper/cpu_scraper.go 100.00% <100.00%> (ø)
...icsreceiver/internal/scraper/cpuscraper/factory.go 100.00% <100.00%> (ø)
...nternal/scraper/diskscraper/disk_scraper_others.go 100.00% <100.00%> (ø)
...csreceiver/internal/scraper/diskscraper/factory.go 100.00% <100.00%> (ø)
...iver/internal/scraper/filesystemscraper/factory.go 88.88% <100.00%> (+3.17%) ⬆️
...al/scraper/filesystemscraper/filesystem_scraper.go 100.00% <100.00%> (ø)
...csreceiver/internal/scraper/loadscraper/factory.go 100.00% <100.00%> (ø)
... and 13 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 07d7a1f...98a76e5. Read the comment docs.

@asuresh4
Copy link
Member

@james-bebbington do you think it makes sense to split this PR? Asking since there are changes in componenterror and obsreport apart from updates to host metrics receiver.

@james-bebbington
Copy link
Member Author

james-bebbington commented Oct 15, 2020

@james-bebbington do you think it makes sense to split this PR? Asking since there are changes in componenterror and obsreport apart from updates to host metrics receiver.

Yes definitely.

This PR actually depends on #1961 (the first commit of this PR with changes to componenterror, obsreport, etc is in #1961). If you want you can just review the second commit of this PR, but otherwise I would recommend reviewing #1961 first and waiting for that to be merged before reviewing this.

@github-actions
Copy link
Contributor

This PR was marked stale due to lack of activity. It will be closed in 7 days.

@github-actions github-actions bot added the Stale label Oct 23, 2020
@james-bebbington
Copy link
Member Author

james-bebbington commented Oct 29, 2020

This is now unblocked and ready for review again

@@ -135,7 +144,7 @@ func (s *scraper) getProcessMetadata() ([]*processMetadata, error) {

executable, err := getProcessExecutable(handle)
if err != nil {
errs = append(errs, fmt.Errorf("error reading process name for pid %v: %w", pid, err))
errs = append(errs, consumererror.NewPartialScrapeError(fmt.Errorf("error reading process name for pid %v: %w", pid, err), 1))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Didn't think of this during the previous review but seeing it now a less verbose version might be

var errs consumererror.ScrapeErrors
errs.Add(fmt.Errorf("error reading process name for pid %v: %w", pid, err), 1))
errs.Add(...)
return errs.Combine()

Or something. Just an idea.

Copy link
Member Author

@james-bebbington james-bebbington Oct 30, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is a bit ugly yea.

Another option might be to just make a fmt-like constructor for PartialScrapeError so we can do something like

errs = append(errs, consumererror.NewScrapeError(1, "error reading process name for pid %v: %w", pid, err))

WDYT?

In any case I will probably do in a follow up PR as this one is very large already

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another option might be to just make a fmt-like constructor for PartialScrapeError so we can do something like

Yeah I think that'd be good. I was worried go vet might not catch fmt string errors but looks like it does from a quick test:

package main

import (
	"fmt"
)

func f(s string, args... interface{}) string {
   return fmt.Sprintf(s, args...)
  
}

func main() {
//	fmt.Printf("Hello, playground %s %s", "foo", 5)
	f("hello, playground %s %s", "foo", 3)
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But I still think it'd be nice to have a struct to accumulate errors into instead of having to do all the appends. Could give it an Addf for formatted strings as well.

Copy link
Member Author

@james-bebbington james-bebbington Oct 31, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea that seems like a good idea. Note we would also want a way to include regular unexpected errors that don't necessarily make sense to be represented as a scrape error.

Moved this discussion to an issue: #2046

@tigrannajaryan
Copy link
Member

OK to merge this?

@jrcamp
Copy link
Contributor

jrcamp commented Oct 30, 2020

OK to merge this?

@tigrannajaryan yes

@bogdandrutu bogdandrutu merged commit 5414c55 into open-telemetry:master Nov 1, 2020
MovieStoreGuy pushed a commit to atlassian-forks/opentelemetry-collector that referenced this pull request Nov 11, 2021
Troels51 pushed a commit to Troels51/opentelemetry-collector that referenced this pull request Jul 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Use obsreport.StartMetricsReceiveOp/EndMetricsReceiveOp in host scrapers
5 participants