Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The pulsar-proxy fails to start with 401 when deploying with Pulsar 3.1.1 #403

Closed
apigeontx opened this issue Nov 2, 2023 · 5 comments
Closed

The pulsar-proxy fails to start with 401 when deploying with Pulsar 3.1.1 #403

apigeontx opened this issue Nov 2, 2023 · 5 comments

Comments

@apigeontx
Copy link

apigeontx commented Nov 2, 2023 •
edited
Loading

Describe the bug
I've modified the Chart.yaml to use appVersion: 3.1.1 and I'm installing the Chart on Kubernetes v1.25.9. All services are starting fine (pulsar-bookie, pulsar-broker, pulsar-recovery, pulsar-toolset and pulsar-zookeeper), but the pulsar-proxy is failing to start. The pulsar-proxy log reveals:

2023-11-02T17:01:52,196+0000 [pulsar-external-web-5-3] WARN  org.apache.pulsar.broker.web.AuthenticationFilter - [10.77.223.28] Failed to authenticate HTTP request: Authentication required
2023-11-02T17:01:52,197+0000 [pulsar-external-web-5-4] INFO  org.eclipse.jetty.server.RequestLog - 10.77.223.28 - - [02/Nov/2023:17:01:52 +0000] "GET /status.html HTTP/1.1" 401 550 "-" "kube-probe/1.25" 1
2023-11-02T17:01:52,198+0000 [pulsar-external-web-5-3] INFO  org.eclipse.jetty.server.RequestLog - 10.77.223.28 - - [02/Nov/2023:17:01:52 +0000] "GET /status.html HTTP/1.1" 401 550 "-" "kube-probe/1.25" 1

If I exec into the pulsar-proxy and try curl http://localhost/status.html, it shows:

<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 401 Authentication required</title>
</head>
<body><h2>HTTP ERROR 401 Authentication required</h2>
<table>
<tr><th>URI:</th><td>/status.html</td></tr>
<tr><th>STATUS:</th><td>401</td></tr>
<tr><th>MESSAGE:</th><td>Authentication required</td></tr>
<tr><th>SERVLET:</th><td>org.glassfish.jersey.servlet.ServletContainer-2ec3633f</td></tr>
</table>
<hr/><a href="https://eclipse.org/jetty">Powered by Jetty:// 9.4.51.v20230217</a><hr/>

</body>
</html>

In the configMap, I'm specifying authenticateMetricsEndpoint: "false" and I also tried to set the webServicePort to 8080.

I have the livenessProbe and readinessProbe set with:

      httpGet:
        path: /status.html
        port: 80
        scheme: HTTP

This is working fine, if I use Pulsar 2.11.2

One more note, I did try the latest changes from #356, but still didn't work.

Note that if I set the webServicePortTls in the pulsar-proxy configMap and change the livenessProbe/readinessProbe to use

          exec:
            command:
            - sh
            - -c
            - curl --fail --key <PATH_TO_KEY> --cert <PATH_TO_CERT> -k "https://localhost/status.html" | grep -q "OK"

the pulsar-proxy is coming up, but I'm wondering why it's not working with webServicePort: "80".

Desktop (please complete the following information):

  • OS: Red Hat Enterprise Linux Server release 7.9 (Maipo), el7uek.x86_64
@Awsmsniper
Copy link

I also encountered this problem

@Awsmsniper
Copy link

@lhotari

@alexandrutopala
Copy link

@apigeontx can you explain how you managed to change the liveness probe?

I'm facing the same issue with the sn-platform:1.11.2 and pulsar-operator:0.17.2 charts deployed in a Kubernetes v1.27.4.

I've searched through the templates and CRD but found no field for liveness/readiness probes for PulsarProxy kind.

@hadican
Copy link
Contributor

hadican commented Dec 27, 2023

I'm getting this too. I think something changed at 3.x

When you enable auth, status.html expects JWT token too. IMHO, health check files should be skipped.

For example, this returns "OK"

export PROXY_ADMIN_TOKEN="something"
curl -H "Authorization: Bearer ${PROXY_ADMIN_TOKEN}" "http://pod_ip:8080/status.html

@lhotari
Copy link
Member

lhotari commented Jan 26, 2024

This is fixed in 3.0.2 and 3.1.2 versions of Pulsar. It's this fix: apache/pulsar#21428 .

@lhotari lhotari closed this as completed Jan 26, 2024
@lhotari lhotari mentioned this issue Jan 26, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@lhotari @hadican @apigeontx @Awsmsniper @alexandrutopala