Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Issue with Endpoints Port Forwarding outside of Otel. #35329

Open
VenuEmmadi opened this issue Sep 20, 2024 · 0 comments
Open

Issue with Endpoints Port Forwarding outside of Otel. #35329

VenuEmmadi opened this issue Sep 20, 2024 · 0 comments
Labels
bug Something isn't working needs triage New item requiring triage

Comments

@VenuEmmadi
Copy link

Component(s)

No response

What happened?

PortForwardIssueOpenTelemetry.docx

Collector version

v0.109.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

receivers:
     otlp:
       protocols:
         grpc:
   extensions:
     health_check:
       path: "/"
   service:
     extensions: [health_check]
     pipelines:
       metrics:
         receivers: [otlp]

Log output

curl -v http://localhost:13133
  * processing: http://localhost:13133
  *   Trying [::1]:13133...
  * Connected to localhost (::1) port 13133
  > GET / HTTP/1.1
  > Host: localhost:13133
  > User-Agent: curl/8.2.1
  > Accept: */*
  >
  * Recv failure: Connection was aborted
  * Closing connection
  curl: (56) Recv failure: Connection was aborted

Additional context

Title: Issues with Endpoints Port Forwarding outside of OpenTelemetry Collector application

Title: Issues with Endpoints Port Forwarding outside of OpenTelemetry Collector application

  1. Description:
    I am encountering issues with Endpoints like the health check and gRPC endpoints in the OpenTelemetry Collector. The problems are particularly evident when these endpoints are not explicitly configured, resulting in connection aborts. Below are the detailed observations and configurations related to this issue.

Issue:

When attempting to access the health check and gRPC endpoints of the OpenTelemetry Collector, the connections are being aborted
Example:

  • Health Check Endpoint: http://localhost:13133
  • gRPC Endpoint: http://localhost:4318

Expected Behavior according to documentation:

  • Health Check Endpoint: Should respond with a status indicating the health of the collector.
  • gRPC Endpoint: Should be accessible for receiving gRPC requests.
  1. Actual Behavior:

  • Health Check Endpoint: When accessed without an explicitly defined endpoint, the response is an error indicating that the connection was aborted.
    The following curl command results in:

    curl -v http://localhost:13133/

    GET / HTTP/1.1
    Host: localhost:13133
    User-Agent: curl/8.2.1
    Accept: /

    • Recv failure: Connection was aborted
    • Closing connection
      curl: (56) Recv failure: Connection was aborted
      From Browser the output is as below:
      Its Showwing “ERR_EMPTY_RESPONSE”.

  • gRPC Endpoint: Similarly, when not configured explicitly, attempts to connect to the gRPC endpoint result in connection aborts. However, when explicitly configured, the connection succeeds.

Configuration Files:

  1. Configuration Without Explicit Endpoint Definitions:

    receivers:
  otlp:
    protocols:
      grpc:
extensions:
  health_check:
    path: "/"
service:
  extensions: [health_check]
  pipelines:
    metrics:
      receivers: [otlp]

    With this configuration, the health check and gRPC endpoints fail to respond properly as shown below.
    curl -v http://localhost:13133/

GET / HTTP/1.1
Host: localhost:13133
User-Agent: curl/8.2.1
Accept: /

  • Recv failure: Connection was aborted
  • Closing connection
    curl: (56) Recv failure: Connection was aborted
    From Browser the output is as below:
    Its Showwing “ERR_EMPTY_RESPONSE”.

  1. Configuration With Explicit Endpoint Definitions:

    receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "[0.0.0.0:4318](https://apc01.safelinks.protection.outlook.com/?url=http%3A%2F%2F0.0.0.0%3A4318%2F&data=05%7C02%7Cvenumadhav.emmadi%40infosys.com%7C53c4f24a94a5494fd36e08dcd8f33866%7C63ce7d592f3e42cda8ccbe764cff5eb6%7C0%7C0%7C638623786360026286%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=nYm5wAc1ovdmsl0%2FiDx3Qa%2Bczc4RU79%2F0Fvm7cR6zFA%3D&reserved=0)"
extensions:
  health_check:
    endpoint: "[0.0.0.0:13133](https://apc01.safelinks.protection.outlook.com/?url=http%3A%2F%2F0.0.0.0%3A13133%2F&data=05%7C02%7Cvenumadhav.emmadi%40infosys.com%7C53c4f24a94a5494fd36e08dcd8f33866%7C63ce7d592f3e42cda8ccbe764cff5eb6%7C0%7C0%7C638623786360048365%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=TyHVueKfQN%2FHUg5%2FpeHVxDY81%2FPEzgi%2BsSlTZ2t2HBU%3D&reserved=0)"
    path: "/"
service:
  extensions: [health_check]
  pipelines:
    metrics:
      receivers: [otlp]

With this configuration, both endpoints work correctly. Specifically:

curl -v http://localhost:13133/
* processing: http://localhost:13133/
*   Trying [::1]:13133...
* Connected to localhost (::1) port 13133
> GET / HTTP/1.1
> Host: localhost:13133
> User-Agent: curl/8.2.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Date: Thu, 19 Sep 2024 21:32:46 GMT
< Content-Length: 98
<
{"status":"Server available","upSince":"2024-09-19T21:24:23.69654773Z","uptime":"8m23.117058632s"}

From Browser the output is as below:
3. Expected Behavior:

  • Health Check Endpoint: Should respond with a status indicating the health of the collector.
  • gRPC Endpoint: Should be accessible for receiving gRPC requests.
    Below output should be given by the server even without explicitly given the endpoint.
    curl -v http://localhost:13133/

    GET / HTTP/1.1
    Host: localhost:13133
    User-Agent: curl/8.2.1
    Accept: /

    < HTTP/1.1 200 OK
    < Content-Type: application/json
    < Date: Thu, 19 Sep 2024 21:32:46 GMT
    < Content-Length: 98
    <
    {"status":"Server available","upSince":"2024-09-19T21:24:23.69654773Z","uptime":"8m23.117058632s"}
    From Browser:
  1. Additional Information:

The issue seems to be related to the default behavior of endpoint configurations in the OpenTelemetry Collector. When endpoints are not explicitly defined, the connections to the health check and gRPC services fail. Explicitly defining these endpoints resolves the issue, allowing for proper communication and status responses.

  1. Resolution:

  2. Deploy the OpenTelemetry Collector using the configuration without explicit endpoint definitions.

  3. Attempt to access the health check endpoint at http://localhost:13133.

  4. Attempt to access the gRPC endpoint at http://localhost:4318.

  5. Observe the connection abort errors.

  6. Modify the configuration to include explicit endpoint definitions for both services.

  7. Retry accessing the endpoints and confirm that they respond correctly.

This issue needs further investigation to understand why its behaving like above in new versions.

References:

  • OpenTelemetry Collector Documentation
  • Issue Tracker
  1. Description:
    I am encountering issues with Endpoints like the health check and gRPC endpoints in the OpenTelemetry Collector. The problems are particularly evident when these endpoints are not explicitly configured, resulting in connection aborts. Below are the detailed observations and configurations related to this issue.

Issue:

When attempting to access the health check and gRPC endpoints of the OpenTelemetry Collector, the connections are being aborted
Example:

  • Health Check Endpoint: http://localhost:13133
  • gRPC Endpoint: http://localhost:4318

Expected Behavior according to documentation:

  • Health Check Endpoint: Should respond with a status indicating the health of the collector.
  • gRPC Endpoint: Should be accessible for receiving gRPC requests.
  1. Actual Behavior:

  • Health Check Endpoint: When accessed without an explicitly defined endpoint, the response is an error indicating that the connection was aborted.
    The following curl command results in:

    curl -v http://localhost:13133/

    GET / HTTP/1.1
    Host: localhost:13133
    User-Agent: curl/8.2.1
    Accept: /

    • Recv failure: Connection was aborted
    • Closing connection
      curl: (56) Recv failure: Connection was aborted
      From Browser the output is as below:
      Its Showwing “ERR_EMPTY_RESPONSE”.

  • gRPC Endpoint: Similarly, when not configured explicitly, attempts to connect to the gRPC endpoint result in connection aborts. However, when explicitly configured, the connection succeeds.

Configuration Files:

  1. Configuration Without Explicit Endpoint Definitions:

    receivers:
  otlp:
    protocols:
      grpc:
extensions:
  health_check:
    path: "/"
service:
  extensions: [health_check]
  pipelines:
    metrics:
      receivers: [otlp]

    With this configuration, the health check and gRPC endpoints fail to respond properly as shown below.
    curl -v http://localhost:13133/

GET / HTTP/1.1
Host: localhost:13133
User-Agent: curl/8.2.1
Accept: /

  • Recv failure: Connection was aborted
  • Closing connection
    curl: (56) Recv failure: Connection was aborted
    From Browser the output is as below:
    Its Showwing “ERR_EMPTY_RESPONSE”.

  1. Configuration With Explicit Endpoint Definitions:

    receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "[0.0.0.0:4318](https://apc01.safelinks.protection.outlook.com/?url=http%3A%2F%2F0.0.0.0%3A4318%2F&data=05%7C02%7Cvenumadhav.emmadi%40infosys.com%7C53c4f24a94a5494fd36e08dcd8f33866%7C63ce7d592f3e42cda8ccbe764cff5eb6%7C0%7C0%7C638623786360026286%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=nYm5wAc1ovdmsl0%2FiDx3Qa%2Bczc4RU79%2F0Fvm7cR6zFA%3D&reserved=0)"
extensions:
  health_check:
    endpoint: "[0.0.0.0:13133](https://apc01.safelinks.protection.outlook.com/?url=http%3A%2F%2F0.0.0.0%3A13133%2F&data=05%7C02%7Cvenumadhav.emmadi%40infosys.com%7C53c4f24a94a5494fd36e08dcd8f33866%7C63ce7d592f3e42cda8ccbe764cff5eb6%7C0%7C0%7C638623786360048365%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=TyHVueKfQN%2FHUg5%2FpeHVxDY81%2FPEzgi%2BsSlTZ2t2HBU%3D&reserved=0)"
    path: "/"
service:
  extensions: [health_check]
  pipelines:
    metrics:
      receivers: [otlp]

With this configuration, both endpoints work correctly. Specifically:

curl -v http://localhost:13133/
* processing: http://localhost:13133/
*   Trying [::1]:13133...
* Connected to localhost (::1) port 13133
> GET / HTTP/1.1
> Host: localhost:13133
> User-Agent: curl/8.2.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Date: Thu, 19 Sep 2024 21:32:46 GMT
< Content-Length: 98
<
{"status":"Server available","upSince":"2024-09-19T21:24:23.69654773Z","uptime":"8m23.117058632s"}

From Browser the output is as below:
3. Expected Behavior:

  • Health Check Endpoint: Should respond with a status indicating the health of the collector.
  • gRPC Endpoint: Should be accessible for receiving gRPC requests.
    Below output should be given by the server even without explicitly given the endpoint.
    curl -v http://localhost:13133/

    GET / HTTP/1.1
    Host: localhost:13133
    User-Agent: curl/8.2.1
    Accept: /

    < HTTP/1.1 200 OK
    < Content-Type: application/json
    < Date: Thu, 19 Sep 2024 21:32:46 GMT
    < Content-Length: 98
    <
    {"status":"Server available","upSince":"2024-09-19T21:24:23.69654773Z","uptime":"8m23.117058632s"}
    From Browser:
  1. Additional Information:

The issue seems to be related to the default behavior of endpoint configurations in the OpenTelemetry Collector. When endpoints are not explicitly defined, the connections to the health check and gRPC services fail. Explicitly defining these endpoints resolves the issue, allowing for proper communication and status responses.

  1. Resolution:

  2. Deploy the OpenTelemetry Collector using the configuration without explicit endpoint definitions.

  3. Attempt to access the health check endpoint at http://localhost:13133.

  4. Attempt to access the gRPC endpoint at http://localhost:4318.

  5. Observe the connection abort errors.

  6. Modify the configuration to include explicit endpoint definitions for both services.

  7. Retry accessing the endpoints and confirm that they respond correctly.

This issue needs further investigation to understand why its behaving like above in new versions.

References:

  • OpenTelemetry Collector Documentation
  • Issue Tracker
@VenuEmmadi VenuEmmadi added bug Something isn't working needs triage New item requiring triage labels Sep 20, 2024
This was referenced Sep 24, 2024
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working needs triage New item requiring triage
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@VenuEmmadi