AWS deploy failing

[TomAugspurger]

Started with my migration away from prometheus operator: https://github.com/pangeo-data/pangeo-cloud-federation/actions/runs/345529754

I'll look into it today.

[TomAugspurger]

Seeing this in the logs while deploying

Error syncing load balancer: failed to ensure load balancer: LoadBalancerIP cannot be specified for AWS ELB

This is from the ingress-nginx specifed in icesat2/secrets/staging.yaml that's used for grafana ingress. I see some references to this in, e.g. kubernetes/cloud-provider-aws#48, but I can't seem to find a solution. @salvis2 does anything come to you immediately? If not then I'll keep digging into it.

[TomAugspurger]

https://github.com/pangeo-data/pangeo-cloud-federation/runs/1364947717?check_suite_focus=true passed with the loadBalancerIP for grafana unset.

I'll see if I can figure out how to get a static IP to point to.

[cspencerjones]

Is this why I keep getting This site can’t be reached?

[scottyhq]

Is this why I keep getting This site can’t be reached?

@cspencerjones are you trying to access the grafana monitoring page? that is what this is referring to, I think the aws hub itself is running fine

@TomAugspurger we used to be able to see hub metrics at https://grafana.staging.aws-uswest2-binder.pangeo.io/ , which no longer works I believe due to the changes linked above. Would be great to have that back up so that we can investigate activity and costs over the last two weeks https://github.com/pangeo-data/jupyterhub-monitoring

[TomAugspurger]

I think I had some trouble with how AWS does ingress. Some issue with assigning a fixed IP to an ELB, so that we can point the DNS entry to it. I don’t believe I was able to resolve it. Sebastian had a comment in the AWS secrets file about issues with getting the IP. I don’t recall the details though. People with access to the kubernetes cluster can access grafana still with `kubectl port-forward` to the ingress service. IIRC grafana uses port 9000.

…

On Nov 13, 2020, at 11:36 AM, Scott Henderson ***@***.***> wrote: Is this why I keep getting This site can’t be reached? @cspencerjones <https://github.com/cspencerjones> are you trying to access the grafana monitoring page? that is what this is referring to, I think the aws hub itself is running fin @TomAugspurger <https://github.com/TomAugspurger> we used to be able to see hub metrics at https://grafana.staging.aws-uswest2-binder.pangeo.io/ <https://grafana.staging.aws-uswest2-binder.pangeo.io/> , which no longer works I believe due to the changes linked above. Would be great to have that back up so that we can investigate activity and costs over the last two weeks https://github.com/pangeo-data/jupyterhub-monitoring <https://github.com/pangeo-data/jupyterhub-monitoring> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#849 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKAOIQRF3OVO2BRDYPHG23SPVVAZANCNFSM4TLJQY6Q>.

[scottyhq]

People with access to the kubernetes cluster can access grafana still with kubectl port-forward to the ingress service. IIRC grafana uses port 9000.

Ok, thanks for the tip! I'll check it out

[cspencerjones]

@cspencerjones are you trying to access the grafana monitoring page? that is what this is referring to, I think the aws hub itself is running fine

No I guess this is separate. I experienced frequent This site can’t be reached messages earlier in the week. I'll make a separate issue if the problem persists. Thanks!

[scottyhq]

closed by #895