-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Additional system attributes #31627
Comments
Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself. |
Server uptime is a crucial metrics for infrastructure observability point of view. Could anyone look into this request, as all other tool providing the same details. |
Pinging code owners for receiver/hostmetrics: @dmitryax @braydonk. See Adding Labels via Comments if you do not have permissions to add labels yourself. |
This seems to fit into Host Metrics receiver more, so I have updated the label. It has been requested before (the issue was closed as inactive): I continue to believe this is a reasonable addition. Would you be up for implementing it @bhupenbisht? |
@astencel-sumo sure i would like contribute.. pls let me know, how can i..? |
@bhupenbisht You can prepare a pull request implementing the change. See the original issue for tips on how to implement this (a new |
We do have an internal receiver/scraper to gather uptime, if still interested we could push it part of the hostmetrics receiver |
@kevinnoel-be this sounds great. Is this code open source - can you point to it to take a look? |
@andrzej-stencel It is in an internal/private OTel build, so you won't be able to see it. We created a separate receiver with scraper as we cannot extend the hostmetrics receiver and we didn't want to fork it only for this metric. I could take same time to port this, but wondering what would be an appropriate naming for this metric as I don't see much movement on open-telemetry/semantic-conventions#648
|
Understood, thanks for your response @kevinnoel-be. I believe Also see #14130 for previous discussion and considerations regarding implementing it. |
@kevinnoel-be do you want to prepare a PR adding the |
@andrzej-stencel I'd like to contribute as well, If it's okay with @kevinnoel-be, Can I take a shot at this? |
@kernelpanic77 Sure. |
Sure, thanks for offering your help @kernelpanic77! Assigning this isuse to you. |
I don't think we should add this metric to the hostmetrics receiver without adding it to semantic conventions. It's fine to add it under a feature flag as a PoC for the semantic conventions, but we must not risk deviating from semantic conventions here. |
Hi @kevinnoel-be, I understand that the repository is internal. Could you guide me on how you implemented the uptime metric, or is there a way I can take a look at the implementation in your fork? @andrzej-stencel @mx-psi we can create a draft PR for this until the semantic conventions is approved. |
@kernelpanic77 Created a new |
To be clear: to my knowledge, nobody is actively working on this on the semantic conventions side. I am happy to guide you through the process if you want to contribute it yourself to semantic conventions |
@kevinnoel-be is there any update on uptime metrics? We are looking forward |
@krantishetty Give me some time, I'm working on a draft PR. |
Sure, let's create a PR for semantic-conventions as well. Could you please guide me ? |
This is helpful @kevinnoel-be. thanks. |
@kernelpanic77 No worries! Take a look at this recent PR that adds another metric to system metrics: open-telemetry/semantic-conventions/pull/1078. You would have to file a PR with roughly the same structure, noting that the Markdown files are autogenerated (see here how this works and how to set up your development environment). |
This is covering only process uptime, however we are looking for server uptime which can be captured from /proc/uptime. PR 2824 not covered with server uptime |
@kevinnoel-be I believe that the existing code for the processes scraper, is already calling |
@krantishetty are you referring to application uptime ? Could you give an example of your usecase for more clarity. |
@kernelpanic77 i believe krantishetty is talking about otel process uptime. Here we are looking for server uptime. |
Im talking about the server uptime which is last reboot of the server. Suppose if i give command # uptime, it shows server uptime since last reboot. |
@kernelpanic77 any luck on system uptime ...? |
Hi |
Hey, |
@kernelpanic77 will you be working on this? |
Hi |
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> #### Description Add new uptime scraper to emit the `system.uptime` metric <!-- Issue number (e.g. #1234) or full URL to issue, if applicable. --> #### Link to tracking issue Fixes #31627 <!--Describe what testing was performed and which tests were added.--> #### Testing Unit test added. Also tested locally (Linux VM) with following config: ```yaml receivers: hostmetrics: scrapers: uptime: exporters: debug: verbosity: detailed service: pipelines: metrics: receivers: - hostmetrics exporters: - debug ``` <details><summary>Details</summary> <p> ```console $ make genotelcontribcol [...] $ make RUN_CONFIG=config-uptime.yaml run 2024-10-23T13:24:48.195+0200 info [email protected]/service.go:135 Setting up own telemetry... 2024-10-23T13:24:48.195+0200 info telemetry/metrics.go:70 Serving metrics {"address": "localhost:8888", "metrics level": "Normal"} 2024-10-23T13:24:48.195+0200 info builders/builders.go:26 Development component. May change in the future. {"kind": "exporter", "data_type": "metrics", "name": "debug"} 2024-10-23T13:24:48.199+0200 info [email protected]/service.go:207 Starting otelcontribcol... {"Version": "0.112.0-dev", "NumCPU": 7} 2024-10-23T13:24:48.199+0200 info extensions/extensions.go:39 Starting extensions... 2024-10-23T13:24:48.199+0200 info [email protected]/service.go:230 Everything is ready. Begin running and processing data. 2024-10-23T13:24:49.202+0200 info Metrics {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 1, "data points": 1} 2024-10-23T13:24:49.202+0200 info ResourceMetrics #0 Resource SchemaURL: https://opentelemetry.io/schemas/1.9.0 ScopeMetrics #0 ScopeMetrics SchemaURL: InstrumentationScope github.com/open-telemetry/opentelemetry-collector-contrib/receiver/hostmetricsreceiver/internal/scraper/uptimescraper 0.112.0-dev Metric #0 Descriptor: -> Name: system.uptime -> Description: The time the system has been running -> Unit: s -> DataType: Sum -> IsMonotonic: false -> AggregationTemporality: Cumulative NumberDataPoints #0 StartTimestamp: 2024-10-22 18:23:22 +0000 UTC Timestamp: 2024-10-23 11:24:49.202512199 +0000 UTC Value: 61287.000000 {"kind": "exporter", "data_type": "metrics", "name": "debug"} 2024-10-23T13:25:49.206+0200 info Metrics {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 1, "data points": 1} 2024-10-23T13:25:49.206+0200 info ResourceMetrics #0 Resource SchemaURL: https://opentelemetry.io/schemas/1.9.0 ScopeMetrics #0 ScopeMetrics SchemaURL: InstrumentationScope github.com/open-telemetry/opentelemetry-collector-contrib/receiver/hostmetricsreceiver/internal/scraper/uptimescraper 0.112.0-dev Metric #0 Descriptor: -> Name: system.uptime -> Description: The time the system has been running -> Unit: s -> DataType: Sum -> IsMonotonic: false -> AggregationTemporality: Cumulative NumberDataPoints #0 StartTimestamp: 2024-10-22 18:23:22 +0000 UTC Timestamp: 2024-10-23 11:25:49.206657828 +0000 UTC Value: 61347.000000 {"kind": "exporter", "data_type": "metrics", "name": "debug"} 2024-10-23T13:26:49.205+0200 info Metrics {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 1, "data points": 1} 2024-10-23T13:26:49.205+0200 info ResourceMetrics #0 Resource SchemaURL: https://opentelemetry.io/schemas/1.9.0 ScopeMetrics #0 ScopeMetrics SchemaURL: InstrumentationScope github.com/open-telemetry/opentelemetry-collector-contrib/receiver/hostmetricsreceiver/internal/scraper/uptimescraper 0.112.0-dev Metric #0 Descriptor: -> Name: system.uptime -> Description: The time the system has been running -> Unit: s -> DataType: Sum -> IsMonotonic: false -> AggregationTemporality: Cumulative NumberDataPoints #0 StartTimestamp: 2024-10-22 18:23:22 +0000 UTC Timestamp: 2024-10-23 11:26:49.2051249 +0000 UTC Value: 61407.000000 {"kind": "exporter", "data_type": "metrics", "name": "debug"} 2024-10-23T13:27:49.210+0200 info Metrics {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 1, "data points": 1} 2024-10-23T13:27:49.210+0200 info ResourceMetrics #0 Resource SchemaURL: https://opentelemetry.io/schemas/1.9.0 ScopeMetrics #0 ScopeMetrics SchemaURL: InstrumentationScope github.com/open-telemetry/opentelemetry-collector-contrib/receiver/hostmetricsreceiver/internal/scraper/uptimescraper 0.112.0-dev Metric #0 Descriptor: -> Name: system.uptime -> Description: The time the system has been running -> Unit: s -> DataType: Sum -> IsMonotonic: false -> AggregationTemporality: Cumulative NumberDataPoints #0 StartTimestamp: 2024-10-22 18:23:22 +0000 UTC Timestamp: 2024-10-23 11:27:49.21007371 +0000 UTC Value: 61467.000000 {"kind": "exporter", "data_type": "metrics", "name": "debug"} ^C 2024-10-23T13:27:54.204+0200 info [email protected]/collector.go:328 Received signal from OS {"signal": "interrupt"} 2024-10-23T13:27:54.204+0200 info [email protected]/service.go:266 Starting shutdown... 2024-10-23T13:27:54.204+0200 info extensions/extensions.go:66 Stopping extensions... 2024-10-23T13:27:54.204+0200 info [email protected]/service.go:280 Shutdown complete. ``` </p> </details> <!--Describe the documentation added.--> #### Documentation Updated readme & generated scraper docs <!--Please delete paragraphs that you did not use before submitting.-->
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> #### Description Add new uptime scraper to emit the `system.uptime` metric <!-- Issue number (e.g. open-telemetry#1234) or full URL to issue, if applicable. --> #### Link to tracking issue Fixes open-telemetry#31627 <!--Describe what testing was performed and which tests were added.--> #### Testing Unit test added. Also tested locally (Linux VM) with following config: ```yaml receivers: hostmetrics: scrapers: uptime: exporters: debug: verbosity: detailed service: pipelines: metrics: receivers: - hostmetrics exporters: - debug ``` <details><summary>Details</summary> <p> ```console $ make genotelcontribcol [...] $ make RUN_CONFIG=config-uptime.yaml run 2024-10-23T13:24:48.195+0200 info [email protected]/service.go:135 Setting up own telemetry... 2024-10-23T13:24:48.195+0200 info telemetry/metrics.go:70 Serving metrics {"address": "localhost:8888", "metrics level": "Normal"} 2024-10-23T13:24:48.195+0200 info builders/builders.go:26 Development component. May change in the future. {"kind": "exporter", "data_type": "metrics", "name": "debug"} 2024-10-23T13:24:48.199+0200 info [email protected]/service.go:207 Starting otelcontribcol... {"Version": "0.112.0-dev", "NumCPU": 7} 2024-10-23T13:24:48.199+0200 info extensions/extensions.go:39 Starting extensions... 2024-10-23T13:24:48.199+0200 info [email protected]/service.go:230 Everything is ready. Begin running and processing data. 2024-10-23T13:24:49.202+0200 info Metrics {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 1, "data points": 1} 2024-10-23T13:24:49.202+0200 info ResourceMetrics #0 Resource SchemaURL: https://opentelemetry.io/schemas/1.9.0 ScopeMetrics #0 ScopeMetrics SchemaURL: InstrumentationScope github.com/open-telemetry/opentelemetry-collector-contrib/receiver/hostmetricsreceiver/internal/scraper/uptimescraper 0.112.0-dev Metric #0 Descriptor: -> Name: system.uptime -> Description: The time the system has been running -> Unit: s -> DataType: Sum -> IsMonotonic: false -> AggregationTemporality: Cumulative NumberDataPoints #0 StartTimestamp: 2024-10-22 18:23:22 +0000 UTC Timestamp: 2024-10-23 11:24:49.202512199 +0000 UTC Value: 61287.000000 {"kind": "exporter", "data_type": "metrics", "name": "debug"} 2024-10-23T13:25:49.206+0200 info Metrics {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 1, "data points": 1} 2024-10-23T13:25:49.206+0200 info ResourceMetrics #0 Resource SchemaURL: https://opentelemetry.io/schemas/1.9.0 ScopeMetrics #0 ScopeMetrics SchemaURL: InstrumentationScope github.com/open-telemetry/opentelemetry-collector-contrib/receiver/hostmetricsreceiver/internal/scraper/uptimescraper 0.112.0-dev Metric #0 Descriptor: -> Name: system.uptime -> Description: The time the system has been running -> Unit: s -> DataType: Sum -> IsMonotonic: false -> AggregationTemporality: Cumulative NumberDataPoints #0 StartTimestamp: 2024-10-22 18:23:22 +0000 UTC Timestamp: 2024-10-23 11:25:49.206657828 +0000 UTC Value: 61347.000000 {"kind": "exporter", "data_type": "metrics", "name": "debug"} 2024-10-23T13:26:49.205+0200 info Metrics {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 1, "data points": 1} 2024-10-23T13:26:49.205+0200 info ResourceMetrics #0 Resource SchemaURL: https://opentelemetry.io/schemas/1.9.0 ScopeMetrics #0 ScopeMetrics SchemaURL: InstrumentationScope github.com/open-telemetry/opentelemetry-collector-contrib/receiver/hostmetricsreceiver/internal/scraper/uptimescraper 0.112.0-dev Metric #0 Descriptor: -> Name: system.uptime -> Description: The time the system has been running -> Unit: s -> DataType: Sum -> IsMonotonic: false -> AggregationTemporality: Cumulative NumberDataPoints #0 StartTimestamp: 2024-10-22 18:23:22 +0000 UTC Timestamp: 2024-10-23 11:26:49.2051249 +0000 UTC Value: 61407.000000 {"kind": "exporter", "data_type": "metrics", "name": "debug"} 2024-10-23T13:27:49.210+0200 info Metrics {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 1, "data points": 1} 2024-10-23T13:27:49.210+0200 info ResourceMetrics #0 Resource SchemaURL: https://opentelemetry.io/schemas/1.9.0 ScopeMetrics #0 ScopeMetrics SchemaURL: InstrumentationScope github.com/open-telemetry/opentelemetry-collector-contrib/receiver/hostmetricsreceiver/internal/scraper/uptimescraper 0.112.0-dev Metric #0 Descriptor: -> Name: system.uptime -> Description: The time the system has been running -> Unit: s -> DataType: Sum -> IsMonotonic: false -> AggregationTemporality: Cumulative NumberDataPoints #0 StartTimestamp: 2024-10-22 18:23:22 +0000 UTC Timestamp: 2024-10-23 11:27:49.21007371 +0000 UTC Value: 61467.000000 {"kind": "exporter", "data_type": "metrics", "name": "debug"} ^C 2024-10-23T13:27:54.204+0200 info [email protected]/collector.go:328 Received signal from OS {"signal": "interrupt"} 2024-10-23T13:27:54.204+0200 info [email protected]/service.go:266 Starting shutdown... 2024-10-23T13:27:54.204+0200 info extensions/extensions.go:66 Stopping extensions... 2024-10-23T13:27:54.204+0200 info [email protected]/service.go:280 Shutdown complete. ``` </p> </details> <!--Describe the documentation added.--> #### Documentation Updated readme & generated scraper docs <!--Please delete paragraphs that you did not use before submitting.-->
@kernelpanic77 .. hope same will work for windows servers.. |
@bhupenbisht I haven't tested Windows specifically but it should work as documented: Linux, Windows & Mac |
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> #### Description This PR makes some changes that I had hoped to suggest on #35954, however I was unable to make my comment in time before the PR was merged. The changes this PR makes are: * The `uptime` scraper is changed to be named `system` * The scraper is registered using the scraper name when adding to the scrapercontroller <!-- Issue number (e.g. #1234) or full URL to issue, if applicable. --> #### Link to tracking issue Amends the fix to #31627 (following a comment I made on #35932) <!--Describe what testing was performed and which tests were added.--> #### Testing <!--Describe the documentation added.--> #### Documentation The documentation that was added for the `uptime` scraper was renamed accordingly, and the description adjusted. <!--Please delete paragraphs that you did not use before submitting.-->
This feature will be in the v0.113.0 release! Thanks for the implementation @kevinnoel-be. Once v0.113.0 is out, you'll be able to add The metric is implemented as per the guidance in open-telemetry/oteps#185. (Haven't finished the work of adapting this OTEP into official semconv guidance, but it is for all intents and purposes our plan). In that OTEP you can find guidance for how to use the metric for common dashboard/alert scenarios. |
…-telemetry#36123) <!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> #### Description This PR makes some changes that I had hoped to suggest on open-telemetry#35954, however I was unable to make my comment in time before the PR was merged. The changes this PR makes are: * The `uptime` scraper is changed to be named `system` * The scraper is registered using the scraper name when adding to the scrapercontroller <!-- Issue number (e.g. open-telemetry#1234) or full URL to issue, if applicable. --> #### Link to tracking issue Amends the fix to open-telemetry#31627 (following a comment I made on open-telemetry#35932) <!--Describe what testing was performed and which tests were added.--> #### Testing <!--Describe the documentation added.--> #### Documentation The documentation that was added for the `uptime` scraper was renamed accordingly, and the description adjusted. <!--Please delete paragraphs that you did not use before submitting.-->
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> #### Description Add new uptime scraper to emit the `system.uptime` metric <!-- Issue number (e.g. open-telemetry#1234) or full URL to issue, if applicable. --> #### Link to tracking issue Fixes open-telemetry#31627 <!--Describe what testing was performed and which tests were added.--> #### Testing Unit test added. Also tested locally (Linux VM) with following config: ```yaml receivers: hostmetrics: scrapers: uptime: exporters: debug: verbosity: detailed service: pipelines: metrics: receivers: - hostmetrics exporters: - debug ``` <details><summary>Details</summary> <p> ```console $ make genotelcontribcol [...] $ make RUN_CONFIG=config-uptime.yaml run 2024-10-23T13:24:48.195+0200 info [email protected]/service.go:135 Setting up own telemetry... 2024-10-23T13:24:48.195+0200 info telemetry/metrics.go:70 Serving metrics {"address": "localhost:8888", "metrics level": "Normal"} 2024-10-23T13:24:48.195+0200 info builders/builders.go:26 Development component. May change in the future. {"kind": "exporter", "data_type": "metrics", "name": "debug"} 2024-10-23T13:24:48.199+0200 info [email protected]/service.go:207 Starting otelcontribcol... {"Version": "0.112.0-dev", "NumCPU": 7} 2024-10-23T13:24:48.199+0200 info extensions/extensions.go:39 Starting extensions... 2024-10-23T13:24:48.199+0200 info [email protected]/service.go:230 Everything is ready. Begin running and processing data. 2024-10-23T13:24:49.202+0200 info Metrics {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 1, "data points": 1} 2024-10-23T13:24:49.202+0200 info ResourceMetrics #0 Resource SchemaURL: https://opentelemetry.io/schemas/1.9.0 ScopeMetrics #0 ScopeMetrics SchemaURL: InstrumentationScope github.com/open-telemetry/opentelemetry-collector-contrib/receiver/hostmetricsreceiver/internal/scraper/uptimescraper 0.112.0-dev Metric #0 Descriptor: -> Name: system.uptime -> Description: The time the system has been running -> Unit: s -> DataType: Sum -> IsMonotonic: false -> AggregationTemporality: Cumulative NumberDataPoints #0 StartTimestamp: 2024-10-22 18:23:22 +0000 UTC Timestamp: 2024-10-23 11:24:49.202512199 +0000 UTC Value: 61287.000000 {"kind": "exporter", "data_type": "metrics", "name": "debug"} 2024-10-23T13:25:49.206+0200 info Metrics {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 1, "data points": 1} 2024-10-23T13:25:49.206+0200 info ResourceMetrics #0 Resource SchemaURL: https://opentelemetry.io/schemas/1.9.0 ScopeMetrics #0 ScopeMetrics SchemaURL: InstrumentationScope github.com/open-telemetry/opentelemetry-collector-contrib/receiver/hostmetricsreceiver/internal/scraper/uptimescraper 0.112.0-dev Metric #0 Descriptor: -> Name: system.uptime -> Description: The time the system has been running -> Unit: s -> DataType: Sum -> IsMonotonic: false -> AggregationTemporality: Cumulative NumberDataPoints #0 StartTimestamp: 2024-10-22 18:23:22 +0000 UTC Timestamp: 2024-10-23 11:25:49.206657828 +0000 UTC Value: 61347.000000 {"kind": "exporter", "data_type": "metrics", "name": "debug"} 2024-10-23T13:26:49.205+0200 info Metrics {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 1, "data points": 1} 2024-10-23T13:26:49.205+0200 info ResourceMetrics #0 Resource SchemaURL: https://opentelemetry.io/schemas/1.9.0 ScopeMetrics #0 ScopeMetrics SchemaURL: InstrumentationScope github.com/open-telemetry/opentelemetry-collector-contrib/receiver/hostmetricsreceiver/internal/scraper/uptimescraper 0.112.0-dev Metric #0 Descriptor: -> Name: system.uptime -> Description: The time the system has been running -> Unit: s -> DataType: Sum -> IsMonotonic: false -> AggregationTemporality: Cumulative NumberDataPoints #0 StartTimestamp: 2024-10-22 18:23:22 +0000 UTC Timestamp: 2024-10-23 11:26:49.2051249 +0000 UTC Value: 61407.000000 {"kind": "exporter", "data_type": "metrics", "name": "debug"} 2024-10-23T13:27:49.210+0200 info Metrics {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 1, "data points": 1} 2024-10-23T13:27:49.210+0200 info ResourceMetrics #0 Resource SchemaURL: https://opentelemetry.io/schemas/1.9.0 ScopeMetrics #0 ScopeMetrics SchemaURL: InstrumentationScope github.com/open-telemetry/opentelemetry-collector-contrib/receiver/hostmetricsreceiver/internal/scraper/uptimescraper 0.112.0-dev Metric #0 Descriptor: -> Name: system.uptime -> Description: The time the system has been running -> Unit: s -> DataType: Sum -> IsMonotonic: false -> AggregationTemporality: Cumulative NumberDataPoints #0 StartTimestamp: 2024-10-22 18:23:22 +0000 UTC Timestamp: 2024-10-23 11:27:49.21007371 +0000 UTC Value: 61467.000000 {"kind": "exporter", "data_type": "metrics", "name": "debug"} ^C 2024-10-23T13:27:54.204+0200 info [email protected]/collector.go:328 Received signal from OS {"signal": "interrupt"} 2024-10-23T13:27:54.204+0200 info [email protected]/service.go:266 Starting shutdown... 2024-10-23T13:27:54.204+0200 info extensions/extensions.go:66 Stopping extensions... 2024-10-23T13:27:54.204+0200 info [email protected]/service.go:280 Shutdown complete. ``` </p> </details> <!--Describe the documentation added.--> #### Documentation Updated readme & generated scraper docs <!--Please delete paragraphs that you did not use before submitting.-->
Component(s)
Describe the issue you're reporting
Looking for system uptime metric . This metrics would provide useful context on the machine that is generating telemetry and would be useful for infrastructure monitoring.
The text was updated successfully, but these errors were encountered: