Jeff
Jeff Cloud Systems Architect
1 min read

Prometheus Readiness Probe Failure

Prometheus Readiness Probe Failure

Prometheus Readiness Probe Failure

I was recently tweakin my Prometheus deployment on Kubernetes when the readiness probe started to fail with a 404. The issue started when I wanted to add my external URL for Alert Manager. I added the --web.external-url=https://my.url to my yaml:

args:
  - --storage.tsdb.retention.time=8d
  - --config.file=/etc/config/prometheus.yml
  - --storage.tsdb.path=/data
  - --web.enable-lifecycle=true
  - --web.external-url=https://my.url

After a re-deploy, I only had 1/2 containers up (I also run jimmidyson/configmap-reload) and had the following entries in my Kubernetes events

2m    Warning    Unhealthy    pod/promtheus-server-xxxxxx-xxxx    Readiness probe failed: HTTP probe failed with statuscode: 404
4m    Warning    Unhealthy    pod/promtheus-server-xxxxxx-xxxx    Liveness probe failed: HTTP probe failed with statuscode: 404

It turns out that when adding --web.external-url=https://my.url you also need to add --web.route-prefix=/

args:
  - --storage.tsdb.retention.time=8d
  - --config.file=/etc/config/prometheus.yml
  - --storage.tsdb.path=/data
  - --web.enable-lifecycle=true
  - --web.external-url=https://my.url
  - --web.route-prefix=/

If you're using the helm chart, add this to your values.yaml 

server:
  extraArgs:
    web.external-url: "https://my.url"
    web.route-prefix: "/"