All guides
  1. Home
  2. Technical guides
  3. Monitoring

MonitoringIntermediate level

Grafana and Prometheus on Kubernetes: monitoring on AKS, GKE, EKS and IONOS

9 min read

All guides

Grafana and Prometheus on Kubernetes: monitoring on AKS, GKE, EKS and IONOS

A practical guide to setting up monitoring with Prometheus and Grafana on a managed Kubernetes cluster: AKS (Azure), GKE (Google Cloud), EKS (AWS) or IONOS Managed Kubernetes. With YAML manifests ready to adapt, credentials in Secrets, access only through the internal network and the differences between clouds. All the manifests and commands are also in the README.md on the left.

What you’ll learn

  • Deploying Prometheus and Grafana on Kubernetes with your own YAML manifests, without depending on a chart.
  • Monitoring a service with /metrics protected by username, password and HTTPS, and discovering pods automatically.
  • Storing passwords and certificates in Secrets, not in ConfigMaps or in the YAML.
  • Giving private access to Grafana on AKS, GKE, EKS and IONOS.
  • Reloading the configuration, checking the status and solving the most common errors.

Prerequisites

  • An AKS, GKE, EKS or IONOS Managed Kubernetes cluster, and kubectl connected to it (see the Kubernetes administration guide).
  • Permissions to create namespaces, ClusterRoles and persistent volumes.
  • Windows with PowerShell; Docker Desktop only to validate the configuration.
  • On EKS, the Amazon EBS CSI driver add-on for the disks, and the AWS Load Balancer Controller if you are going to use an internal load balancer.
  • The image versions are examples: always pin a specific version (never latest) and check the latest stable one in the Prometheus and Grafana release notes.
  • If your cluster only pulls images from a private registry, copy prom/prometheus and grafana/grafana there and add its imagePullSecrets.

Step by step

1. Prepare the namespace and check the storage

Prometheus and Grafana store data on disk, so they need a persistent volume. Check that you are on the right cluster and which StorageClass will be used by default:

# The right cluster and a dedicated namespace for monitoring
kubectl config current-context
kubectl create namespace monitoring

# Which StorageClass will be used for the disks (the one marked "(default)")
kubectl get storageclass

If none shows as (default), add storageClassName to the volumeClaimTemplates in steps 5 and 6. On EKS, without the EBS CSI driver the volumes stay in Pending, and on new clusters gp2 is no longer marked as the default.

2. Store the credentials in Secrets

Passwords don’t belong in the Prometheus ConfigMap or in the repository: anyone with read access to the namespace or to Git would see them. Create them as Secrets from local files, which you delete afterwards:

# Password and certificate authority (CA) certificate of the service you are going to monitor
kubectl create secret generic prometheus-target -n monitoring `
  --from-file=password=.\target-password.txt `
  --from-file=ca.crt=.\internal-ca.crt

# Grafana administrator user and password
kubectl create secret generic grafana-admin -n monitoring `
  --from-literal=admin-user=admin `
  --from-file=admin-password=.\grafana-password.txt

# Delete the local files with the passwords once the Secret has been created
Remove-Item .\target-password.txt, .\grafana-password.txt

For the service you are going to monitor, use a user that can only read metrics. If its certificate is signed by an internal CA, keep the ca.crt: that way Prometheus validates the certificate instead of skipping the check.

3. Configure Prometheus

The configuration goes in a ConfigMap. This example monitors three things: Prometheus itself, a service with protected /metrics (with password_file and ca_file, which read the Secret from step 2) and the cluster pods that request it with annotations:

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: monitoring
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s
      evaluation_interval: 15s
    scrape_configs:
      # Prometheus itself
      - job_name: prometheus
        static_configs:
          - targets: ['localhost:9090']

      # A service with /metrics protected by username and password, and HTTPS with its own CA
      - job_name: my-service
        scrape_interval: 30s
        scheme: https
        metrics_path: /metrics
        basic_auth:
          username: metrics-reader
          password_file: /etc/prometheus/secrets/password
        tls_config:
          ca_file: /etc/prometheus/secrets/ca.crt
        static_configs:
          - targets: ['metrics.my-company.internal:8443']

      # Cluster pods that request it with prometheus.io/* annotations
      - job_name: pods
        kubernetes_sd_configs:
          - role: pod
        relabel_configs:
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
            action: keep
            regex: "true"
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
            action: replace
            target_label: __metrics_path__
            regex: (.+)
          - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
            action: replace
            regex: ([^:]+)(?::\d+)?;(\d+)
            replacement: $1:$2
            target_label: __address__
          - source_labels: [__meta_kubernetes_namespace]
            target_label: namespace
          - source_labels: [__meta_kubernetes_pod_name]
            target_label: pod

For Prometheus to monitor a pod, add prometheus.io/scrape: "true", prometheus.io/port and, if it is not /metrics, prometheus.io/path to its template. Before applying it, validate the configuration with promtool:

# Validate the configuration before applying it (requires Docker Desktop)
# Save the prometheus.yml block to a separate file for this check
docker run --rm -v "${PWD}:/cfg" --entrypoint promtool prom/prometheus:v3.2.1 check config /cfg/prometheus.yml

4. Grant read permissions to discover pods

Pod discovery queries the Kubernetes API, so Prometheus needs its own service account with read-only permissions:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus-read
rules:
  - apiGroups: [""]
    resources: [nodes, nodes/metrics, services, endpoints, pods]
    verbs: [get, list, watch]
  - nonResourceURLs: [/metrics]
    verbs: [get]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus-read
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus-read
subjects:
  - kind: ServiceAccount
    name: prometheus
    namespace: monitoring

5. Deploy Prometheus

A StatefulSet with its own disk, running without root privileges (the nobody user, 65534, from the official image), with health probes, memory limits and 15 days of retention. --web.enable-lifecycle lets you reload the configuration without restarting:

apiVersion: v1
kind: Service
metadata:
  name: prometheus
  namespace: monitoring
spec:
  selector:
    app: prometheus
  ports:
    - name: web
      port: 9090
      targetPort: 9090
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: prometheus
  namespace: monitoring
spec:
  serviceName: prometheus
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      serviceAccountName: prometheus
      securityContext:
        runAsNonRoot: true
        runAsUser: 65534
        runAsGroup: 65534
        fsGroup: 65534
      # nodeSelector:            # optional: a node pool dedicated to monitoring
      #   pool: monitoring
      containers:
        - name: prometheus
          image: prom/prometheus:v3.2.1
          args:
            - --config.file=/etc/prometheus/prometheus.yml
            - --storage.tsdb.path=/prometheus
            - --storage.tsdb.retention.time=15d
            - --web.enable-lifecycle
          ports:
            - containerPort: 9090
          readinessProbe:
            httpGet: { path: /-/ready, port: 9090 }
          livenessProbe:
            httpGet: { path: /-/healthy, port: 9090 }
            initialDelaySeconds: 30
          resources:
            requests: { cpu: 250m, memory: 1Gi }
            limits: { memory: 2Gi }
          securityContext:
            allowPrivilegeEscalation: false
          volumeMounts:
            - name: config
              mountPath: /etc/prometheus
            - name: secrets
              mountPath: /etc/prometheus/secrets
              readOnly: true
            - name: data
              mountPath: /prometheus
      volumes:
        - name: config
          configMap:
            name: prometheus-config
        - name: secrets
          secret:
            secretName: prometheus-target
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: [ReadWriteOnce]
        resources:
          requests:
            storage: 20Gi

The disk size depends on how many series you monitor and on the retention: start with 20 GB and keep an eye on usage (step 9).

6. Deploy Grafana with Prometheus already configured

Grafana reads the administrator user from the Secret and the data source from a provisioning file, so it starts already connected to Prometheus through its internal cluster name:

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-datasources
  namespace: monitoring
data:
  prometheus.yaml: |
    apiVersion: 1
    datasources:
      - name: Prometheus
        type: prometheus
        access: proxy
        url: http://prometheus.monitoring.svc.cluster.local:9090
        isDefault: true
---
apiVersion: v1
kind: Service
metadata:
  name: grafana
  namespace: monitoring
spec:
  selector:
    app: grafana
  ports:
    - name: web
      port: 3000
      targetPort: 3000
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: grafana
  namespace: monitoring
spec:
  serviceName: grafana
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 472
        fsGroup: 472
      containers:
        - name: grafana
          image: grafana/grafana:11.6.0
          ports:
            - containerPort: 3000
          env:
            - name: GF_SECURITY_ADMIN_USER
              valueFrom:
                secretKeyRef: { name: grafana-admin, key: admin-user }
            - name: GF_SECURITY_ADMIN_PASSWORD
              valueFrom:
                secretKeyRef: { name: grafana-admin, key: admin-password }
          readinessProbe:
            httpGet: { path: /api/health, port: 3000 }
          resources:
            requests: { cpu: 100m, memory: 256Mi }
            limits: { memory: 512Mi }
          securityContext:
            allowPrivilegeEscalation: false
          volumeMounts:
            - name: data
              mountPath: /var/lib/grafana
            - name: datasources
              mountPath: /etc/grafana/provisioning/datasources
              readOnly: true
      volumes:
        - name: datasources
          configMap:
            name: grafana-datasources
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: [ReadWriteOnce]
        resources:
          requests:
            storage: 5Gi

Apply everything and wait for the pods to be ready:

kubectl apply -f .\prometheus-config.yaml
kubectl apply -f .\prometheus-rbac.yaml
kubectl apply -f .\prometheus.yaml
kubectl apply -f .\grafana.yaml

kubectl get pods,pvc,svc -n monitoring
kubectl rollout status statefulset/prometheus -n monitoring
kubectl rollout status statefulset/grafana -n monitoring

7. Access Grafana securely

To start with, you don’t need to expose anything: kubectl port-forward opens a tunnel from your machine, authenticated with your cluster credentials:

# Grafana at http://localhost:3000 (username and password from the grafana-admin Secret)
kubectl port-forward svc/grafana 3000:3000 -n monitoring

# Prometheus at http://localhost:9090 (Status > Targets: all targets UP)
kubectl port-forward svc/prometheus 9090:9090 -n monitoring

In Grafana, go to Explore, choose Prometheus and query up: each monitored target appears with value 1 if it responds.

8. Publish Grafana on the internal network only

When the team needs access without kubectl, use an internal load balancer, with a private IP on the cluster network, and also restrict which networks can connect:

apiVersion: v1
kind: Service
metadata:
  name: grafana-internal
  namespace: monitoring
  annotations:
    # Only the line for your cloud:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"        # AKS
    # networking.gke.io/load-balancer-type: "Internal"                     # GKE
    # service.beta.kubernetes.io/aws-load-balancer-type: "external"        # EKS (AWS Load Balancer Controller)
    # service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"   # EKS
    # service.beta.kubernetes.io/aws-load-balancer-scheme: "internal"      # EKS (already the default value)
spec:
  type: LoadBalancer
  loadBalancerSourceRanges:
    - 10.20.0.0/16          # only your office network or your VPN, never 0.0.0.0/0
  selector:
    app: grafana
  ports:
    - name: web
      port: 3000
      targetPort: 3000
CloudInternal load balancerNotes
AKSazure-load-balancer-internal: "true"Private IP on the cluster VNet.
GKEnetworking.gke.io/load-balancer-type: "Internal"Private IP on the VPC.
EKSaws-load-balancer-scheme: "internal" with the AWS Load Balancer ControllerCreates an internal Network Load Balancer.
IONOSThere is no internal load balancer for ServicesLoadBalancer Services only work on public node pools and attach a fixed public IP to a node: for Grafana, port-forward or an Ingress with TLS and restricted access is a better option.

loadBalancerSourceRanges restricts the source IPs: GKE applies it on internal and external load balancers; on EKS it only takes effect if the load balancer preserves the client IP (with nlb-target-type: "ip" this is disabled by default). On any cloud, reinforce it with network rules (NSG, VPC firewall or security groups).

Never publish Grafana or Prometheus with 0.0.0.0/0: Prometheus has no username or password by default, and anyone who reaches it can read all the metrics.

9. Day-to-day operations

To change what is monitored, edit the ConfigMap, apply it and ask Prometheus to reload. Here you also have how to check the status, read the logs and keep an eye on disk space:

# Change the Prometheus configuration without restarting it
kubectl apply -f .\prometheus-config.yaml
# (wait about a minute for the pod to see the new ConfigMap)
kubectl port-forward svc/prometheus 9090:9090 -n monitoring
curl.exe -X POST http://localhost:9090/-/reload      # in another PowerShell window

# Check the status
curl.exe http://localhost:9090/-/ready
kubectl logs statefulset/prometheus -n monitoring --tail=50
kubectl logs statefulset/grafana -n monitoring --tail=50

# Prometheus disk space
kubectl exec prometheus-0 -n monitoring -- df -h /prometheus

If the reload fails, Prometheus keeps the previous configuration and says so in its logs: fix the error and reload again.

Common errors

The target shows as DOWN with “x509: certificate signed by unknown authority”

Prometheus does not trust the service certificate. Add the CA that signs it in tls_config.ca_file (step 2). Don’t use insecure_skip_verify: true: anyone could then impersonate the service and receive the password.

The target shows as DOWN with “401 Unauthorized”

The username or password is wrong, or the file has a trailing line break. Create the Secret again with kubectl create secret … --dry-run=client -o yaml | kubectl apply -f - and restart the pod.

The pod stays in Pending

It is usually the volume: kubectl describe pvc -n monitoring tells you why. There is no default StorageClass, the EBS CSI driver is missing on EKS, or the nodeSelector does not match any node.

CrashLoopBackOff with “permission denied” on /prometheus or /var/lib/grafana

The disk does not have permissions for the container user. Check that the pod’s securityContext includes fsGroup (65534 for Prometheus, 472 for Grafana).

Grafana cannot connect to Prometheus

Check the data source URL: inside the cluster it is http://prometheus.monitoring.svc.cluster.local:9090, never localhost. Also check that the Prometheus pod is ready.

The load balancer stays with EXTERNAL-IP “pending”

The cloud could not create it: look at the events with kubectl describe svc grafana-internal -n monitoring. On EKS, check that the AWS Load Balancer Controller is installed; on IONOS, that the node pool is public.

Annotated pods do not appear in Targets

The RBAC from step 4 is missing (the Prometheus logs show “forbidden”), or the annotations are on the Deployment rather than on the pod template. The value of prometheus.io/scrape must be the text "true", in quotes.

Next steps

  • Define alerts with Alertmanager (for example, target down or disk almost full) and send them to the team’s channel.
  • Monitor the cluster itself with kube-state-metrics and node-exporter, or move to the kube-prometheus-stack Helm chart when you need the whole set.
  • Store the Grafana dashboards as code (dashboard provisioning) in the repository.
  • For long retention or several clusters, consider a managed Prometheus service from your cloud, or Thanos or Mimir.
  • Deploy the manifests from a pipeline (see the CI/CD with GitHub Actions guide).

Share

More guides on

See all guides

Other technical guides

See all guides

IntegrationIntermediate

TIBCO ActiveSpaces 5.2: basic administration commands

The day-to-day commands in TIBCO ActiveSpaces 5.2: realm service on FTL servers, tibdg, grid definition, tables, starting processes, status, scaling with copysets, checkpoints and logs.

Read the guide 8 min read

APIsIntermediate

Basic structure of an API in IBM API Connect 10

How an API is structured in IBM API Connect 10: OpenAPI contract with x-ibm-configuration, Client ID and Client Secret, product with plans and limits, and validation and publishing with the apic CLI.

Read the guide 7 min read

DevOpsBasic

Basic CI/CD structure with GitHub Actions

A complete pipeline with GitHub Actions: tests on every push and pull request, a Docker image in GitHub Container Registry and deployment to a protected environment, with a workflow ready to copy.

Read the guide 6 min read

DevelopmentBasic

Java Spring Boot development best practices

The practices that make the biggest difference in a Spring Boot service: structure, constructor injection, DTOs, validation, errors, configuration, transactions, logs, comments, tests and automated quality.

Read the guide 7 min read

Contact

Build your next competitive advantage today

Let’s talk about how we can help you. We will reply with a clear technical proposal, with no obligation.

Request information

Fields marked with are required.