how to set up prometheus with grafana docker-compose not showing metrics

Why Prometheus and Grafana Look Fine But Show Nothing

You spin up Prometheus and Grafana in docker-compose, the containers start cleanly, both UIs load, and yet your dashboards stay empty. No error banners, no crashed containers, just a blank graph mocking you. This is one of the most common monitoring setup failures, and almost every case traces back to one of three things: container networking, scrape target configuration, or the Grafana datasource URL.

The error from the forum post is actually a perfect example of this failure mode. The user's app exposes a completely valid /metrics endpoint with proper # HELP and # TYPE lines, gauges for player counts, memory usage, CPU percent, all correctly formatted. But Grafana threw a JSON parsing error: ReadObject: expect { or , or } or n, but found #. That error is the giveaway. Grafana's Prometheus datasource expects to talk to Prometheus's HTTP API, which returns JSON. If Grafana is instead pointed straight at an app's raw /metrics text output, it chokes immediately because that's plain text exposition format, not JSON. This almost always means the datasource URL field in Grafana was set to the application's metrics endpoint instead of the Prometheus server address.

The Three Places This Breaks in Docker Compose

When everything runs on bare metal on one machine, localhost means the same thing everywhere. Inside docker-compose, it doesn't. Each container gets its own network namespace, so localhost inside the Grafana container refers to Grafana itself, not Prometheus, not your app. This single fact causes most of the "metrics load but don't appear" reports.

1. Prometheus Can't Reach Your App's Scrape Target

If your prometheus.yml has something like this:

scrape_configs:
  - job_name: 'my-app'
    static_configs:
      - targets: ['localhost:8080']

And your app runs in its own service container named app, Prometheus will fail silently on that target. It's not an error you see in Grafana, it's an error you have to go find in Prometheus's own Targets page. Fix it by using the docker-compose service name as the hostname, since compose sets up internal DNS resolution between services on the same network automatically:

scrape_configs:
  - job_name: 'my-app'
    static_configs:
      - targets: ['app:8080']

Always check http://localhost:9090/targets in your browser after any config change. If the target shows as DOWN with a connection refused error, your networking or port is wrong, not your PromQL query.

2. Grafana's Datasource URL Points to the Wrong Place

This is the exact mistake behind the JSON parsing error above. In Grafana, when adding a Prometheus datasource, the URL field needs the Prometheus server's address, reachable from inside the Grafana container, not from your host machine and not the app's metrics path.

Inside docker-compose this should be:

http://prometheus:9090

Not http://localhost:9090, and definitely not http://app:8080/metrics. Grafana queries Prometheus's /api/v1/query endpoints internally. It never talks to your app's /metrics endpoint directly, Prometheus does that scraping on its own schedule. If you're tempted to paste your app's metrics URL into Grafana thinking that's the data source, that's the misunderstanding causing the exact error in the forum post.

3. The Compose Network Isn't What You Think It Is

By default, docker-compose creates a single bridge network and attaches every service in the file to it, letting services resolve each other by service name. But if you're running multiple compose files, using external: true networks, or mixing network_mode: host on one service, containers can end up unable to see each other at all. Run this to confirm:

docker network inspect <project>_default

Check that both the prometheus and grafana containers, along with your app, are listed under Containers. If your app is missing, it's on a different network and no hostname resolution will save you.

A Working docker-compose Setup

Here's a minimal, correctly wired setup covering all three failure points:

version: '3.8'
services:
  app:
    build: ./app
    ports:
      - "8080:8080"

  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"
    depends_on:
      - app

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    depends_on:
      - prometheus

And prometheus.yml sitting next to it:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'app'
    static_configs:
      - targets: ['app:8080']

Notice app:8080, not localhost:8080. The mounted config file is another common trip-up point, if you mount the wrong path or forget to restart Prometheus after editing it, your changes never take effect since Prometheus doesn't hot-reload the file by default without a reload signal or restart. This kind of volume mounting mistake is the same category of problem covered in setting up a docker development environment with proper volume mounts, where getting host-to-container paths wrong quietly breaks the whole stack.

Confirming Metrics Actually Flow Before Blaming Grafana

Don't touch Grafana until Prometheus itself confirms it's scraping successfully. Three checks, in order:

• Open http://localhost:9090/targets and confirm the target state is UP, not DOWN or UNKNOWN.
• Run a raw query in the Prometheus UI itself, like memory_bytes, and confirm a value comes back.
• Only then go into Grafana, add the datasource with http://prometheus:9090, click Save & Test, and confirm the green success message.

If step one fails, it's networking or scrape config. If step two fails, it's usually a metrics naming mismatch or the app isn't actually exposing on the path Prometheus expects (/metrics by default, configurable via metrics_path in the scrape config). If step three fails after one and two pass, it's the datasource URL, almost certainly still pointing at localhost or the app instead of the Prometheus container name.

Why Dashboards Sometimes Show "No Data" Even When Everything's Connected

Once networking and the datasource are correct, there's a second-tier issue that trips people up: gauge metrics sitting at zero, like the players_ingame or chat_rate values in the original forum example, will render as flat empty-looking lines. This isn't a connection problem, it's just an inactive metric. Test with a metric guaranteed to have movement, like memory_bytes, before assuming your whole pipeline is broken over a genuinely idle gauge.

Also check the dashboard's time range picker. Grafana defaults often show the last 6 hours, and if your containers just started, there's no historical data to plot yet, only a single flat point. Give it a few scrape intervals and narrow the range to "Last 5 minutes" while testing.

Preventing This From Happening Again

Treat container hostnames as the only addressing method inside compose networks, and reserve localhost for testing from your own machine's browser against exposed ports. Keep a habit of checking the Prometheus Targets page as your first debugging step, before ever opening Grafana. This same host-resolution confusion shows up constantly in other containerized stacks too, it's the identical root cause behind many nginx 502 errors in docker setups, where a proxy container can't resolve or reach an upstream service by the name you assumed it had.

Write your scrape configs and datasource URLs with the mental model that every container is its own isolated machine, addressable only by its compose service name on the shared network. Once that clicks, these particular "metrics exist but don't show up" bugs mostly stop happening.