# Grafana and Prometheus on Kubernetes (AKS, GKE, EKS and IONOS)

A Bitflow Solutions guide. Web version: https://bitflow-solutions.com/en/guias-tecnicas/grafana-prometheus-kubernetes-monitoring/

## What you’ll learn

- Deploying **Prometheus** and **Grafana** on Kubernetes with your own YAML manifests, without depending on a chart.
- Monitoring a service with `/metrics` protected by username, password and HTTPS, and discovering pods automatically.
- Storing passwords and certificates in **Secrets**, not in ConfigMaps or in the YAML.
- Giving private access to Grafana on **AKS**, **GKE**, **EKS** and **IONOS**.
- Reloading the configuration, checking the status and solving the most common errors.

## Prerequisites

- An **AKS**, **GKE**, **EKS** or **IONOS Managed Kubernetes** cluster, and `kubectl` connected to it (see the Kubernetes administration guide).
- Permissions to create namespaces, ClusterRoles and persistent volumes.
- **Windows** with **PowerShell**; **Docker Desktop** only to validate the configuration.
- On **EKS**, the **Amazon EBS CSI driver** add-on for the disks, and the **AWS Load Balancer Controller** if you are going to use an internal load balancer.
- The image versions are examples: always pin a specific version (never `latest`) and check the latest stable one in the Prometheus and Grafana release notes.
- If your cluster only pulls images from a private registry, copy `prom/prometheus` and `grafana/grafana` there and add its `imagePullSecrets`.

## Step by step

### 1. Namespace and storage

```powershell
# The right cluster and a dedicated namespace for monitoring
kubectl config current-context
kubectl create namespace monitoring

# Which StorageClass will be used for the disks (the one marked "(default)")
kubectl get storageclass
```

### 2. Secrets

```powershell
# Password and certificate authority (CA) certificate of the service you are going to monitor
kubectl create secret generic prometheus-target -n monitoring `
  --from-file=password=.\target-password.txt `
  --from-file=ca.crt=.\internal-ca.crt

# Grafana administrator user and password
kubectl create secret generic grafana-admin -n monitoring `
  --from-literal=admin-user=admin `
  --from-file=admin-password=.\grafana-password.txt

# Delete the local files with the passwords once the Secret has been created
Remove-Item .\target-password.txt, .\grafana-password.txt
```

### 3. prometheus-config.yaml

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: monitoring
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s
      evaluation_interval: 15s
    scrape_configs:
      # Prometheus itself
      - job_name: prometheus
        static_configs:
          - targets: ['localhost:9090']

      # A service with /metrics protected by username and password, and HTTPS with its own CA
      - job_name: my-service
        scrape_interval: 30s
        scheme: https
        metrics_path: /metrics
        basic_auth:
          username: metrics-reader
          password_file: /etc/prometheus/secrets/password
        tls_config:
          ca_file: /etc/prometheus/secrets/ca.crt
        static_configs:
          - targets: ['metrics.my-company.internal:8443']

      # Cluster pods that request it with prometheus.io/* annotations
      - job_name: pods
        kubernetes_sd_configs:
          - role: pod
        relabel_configs:
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
            action: keep
            regex: "true"
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
            action: replace
            target_label: __metrics_path__
            regex: (.+)
          - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
            action: replace
            regex: ([^:]+)(?::\d+)?;(\d+)
            replacement: $1:$2
            target_label: __address__
          - source_labels: [__meta_kubernetes_namespace]
            target_label: namespace
          - source_labels: [__meta_kubernetes_pod_name]
            target_label: pod
```

Validate:

```powershell
# Validate the configuration before applying it (requires Docker Desktop)
# Save the prometheus.yml block to a separate file for this check
docker run --rm -v "${PWD}:/cfg" --entrypoint promtool prom/prometheus:v3.2.1 check config /cfg/prometheus.yml
```

### 4. prometheus-rbac.yaml

```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus-read
rules:
  - apiGroups: [""]
    resources: [nodes, nodes/metrics, services, endpoints, pods]
    verbs: [get, list, watch]
  - nonResourceURLs: [/metrics]
    verbs: [get]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus-read
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus-read
subjects:
  - kind: ServiceAccount
    name: prometheus
    namespace: monitoring
```

### 5. prometheus.yaml

```yaml
apiVersion: v1
kind: Service
metadata:
  name: prometheus
  namespace: monitoring
spec:
  selector:
    app: prometheus
  ports:
    - name: web
      port: 9090
      targetPort: 9090
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: prometheus
  namespace: monitoring
spec:
  serviceName: prometheus
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      serviceAccountName: prometheus
      securityContext:
        runAsNonRoot: true
        runAsUser: 65534
        runAsGroup: 65534
        fsGroup: 65534
      # nodeSelector:            # optional: a node pool dedicated to monitoring
      #   pool: monitoring
      containers:
        - name: prometheus
          image: prom/prometheus:v3.2.1
          args:
            - --config.file=/etc/prometheus/prometheus.yml
            - --storage.tsdb.path=/prometheus
            - --storage.tsdb.retention.time=15d
            - --web.enable-lifecycle
          ports:
            - containerPort: 9090
          readinessProbe:
            httpGet: { path: /-/ready, port: 9090 }
          livenessProbe:
            httpGet: { path: /-/healthy, port: 9090 }
            initialDelaySeconds: 30
          resources:
            requests: { cpu: 250m, memory: 1Gi }
            limits: { memory: 2Gi }
          securityContext:
            allowPrivilegeEscalation: false
          volumeMounts:
            - name: config
              mountPath: /etc/prometheus
            - name: secrets
              mountPath: /etc/prometheus/secrets
              readOnly: true
            - name: data
              mountPath: /prometheus
      volumes:
        - name: config
          configMap:
            name: prometheus-config
        - name: secrets
          secret:
            secretName: prometheus-target
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: [ReadWriteOnce]
        resources:
          requests:
            storage: 20Gi
```

### 6. grafana.yaml

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-datasources
  namespace: monitoring
data:
  prometheus.yaml: |
    apiVersion: 1
    datasources:
      - name: Prometheus
        type: prometheus
        access: proxy
        url: http://prometheus.monitoring.svc.cluster.local:9090
        isDefault: true
---
apiVersion: v1
kind: Service
metadata:
  name: grafana
  namespace: monitoring
spec:
  selector:
    app: grafana
  ports:
    - name: web
      port: 3000
      targetPort: 3000
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: grafana
  namespace: monitoring
spec:
  serviceName: grafana
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 472
        fsGroup: 472
      containers:
        - name: grafana
          image: grafana/grafana:11.6.0
          ports:
            - containerPort: 3000
          env:
            - name: GF_SECURITY_ADMIN_USER
              valueFrom:
                secretKeyRef: { name: grafana-admin, key: admin-user }
            - name: GF_SECURITY_ADMIN_PASSWORD
              valueFrom:
                secretKeyRef: { name: grafana-admin, key: admin-password }
          readinessProbe:
            httpGet: { path: /api/health, port: 3000 }
          resources:
            requests: { cpu: 100m, memory: 256Mi }
            limits: { memory: 512Mi }
          securityContext:
            allowPrivilegeEscalation: false
          volumeMounts:
            - name: data
              mountPath: /var/lib/grafana
            - name: datasources
              mountPath: /etc/grafana/provisioning/datasources
              readOnly: true
      volumes:
        - name: datasources
          configMap:
            name: grafana-datasources
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: [ReadWriteOnce]
        resources:
          requests:
            storage: 5Gi
```

Apply:

```powershell
kubectl apply -f .\prometheus-config.yaml
kubectl apply -f .\prometheus-rbac.yaml
kubectl apply -f .\prometheus.yaml
kubectl apply -f .\grafana.yaml

kubectl get pods,pvc,svc -n monitoring
kubectl rollout status statefulset/prometheus -n monitoring
kubectl rollout status statefulset/grafana -n monitoring
```

### 7. Access with port-forward

```powershell
# Grafana at http://localhost:3000 (username and password from the grafana-admin Secret)
kubectl port-forward svc/grafana 3000:3000 -n monitoring

# Prometheus at http://localhost:9090 (Status > Targets: all targets UP)
kubectl port-forward svc/prometheus 9090:9090 -n monitoring
```

### 8. grafana-internal.yaml (internal load balancer)

```yaml
apiVersion: v1
kind: Service
metadata:
  name: grafana-internal
  namespace: monitoring
  annotations:
    # Only the line for your cloud:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"        # AKS
    # networking.gke.io/load-balancer-type: "Internal"                     # GKE
    # service.beta.kubernetes.io/aws-load-balancer-type: "external"        # EKS (AWS Load Balancer Controller)
    # service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"   # EKS
    # service.beta.kubernetes.io/aws-load-balancer-scheme: "internal"      # EKS (already the default value)
spec:
  type: LoadBalancer
  loadBalancerSourceRanges:
    - 10.20.0.0/16          # only your office network or your VPN, never 0.0.0.0/0
  selector:
    app: grafana
  ports:
    - name: web
      port: 3000
      targetPort: 3000
```

| Cloud | Internal load balancer |
|---|---|
| AKS | service.beta.kubernetes.io/azure-load-balancer-internal: "true" |
| GKE | networking.gke.io/load-balancer-type: "Internal" |
| EKS | aws-load-balancer-scheme: "internal" (AWS Load Balancer Controller) |
| IONOS | None: port-forward or Ingress with TLS and restricted access |

### 9. Operations

```powershell
# Change the Prometheus configuration without restarting it
kubectl apply -f .\prometheus-config.yaml
# (wait about a minute for the pod to see the new ConfigMap)
kubectl port-forward svc/prometheus 9090:9090 -n monitoring
curl.exe -X POST http://localhost:9090/-/reload      # in another PowerShell window

# Check the status
curl.exe http://localhost:9090/-/ready
kubectl logs statefulset/prometheus -n monitoring --tail=50
kubectl logs statefulset/grafana -n monitoring --tail=50

# Prometheus disk space
kubectl exec prometheus-0 -n monitoring -- df -h /prometheus
```

## Common errors

- **The target shows as DOWN with “x509: certificate signed by unknown authority”**: Prometheus does not trust the service certificate. Add the CA that signs it in `tls_config.ca_file` (step 2). Don’t use `insecure_skip_verify: true`: anyone could then impersonate the service and receive the password.
- **The target shows as DOWN with “401 Unauthorized”**: The username or password is wrong, or the file has a trailing line break. Create the Secret again with `kubectl create secret … --dry-run=client -o yaml | kubectl apply -f -` and restart the pod.
- **The pod stays in Pending**: It is usually the volume: `kubectl describe pvc -n monitoring` tells you why. There is no default StorageClass, the EBS CSI driver is missing on EKS, or the `nodeSelector` does not match any node.
- **CrashLoopBackOff with “permission denied” on /prometheus or /var/lib/grafana**: The disk does not have permissions for the container user. Check that the pod’s `securityContext` includes `fsGroup` (65534 for Prometheus, 472 for Grafana).
- **Grafana cannot connect to Prometheus**: Check the data source URL: inside the cluster it is `http://prometheus.monitoring.svc.cluster.local:9090`, never `localhost`. Also check that the Prometheus pod is ready.
- **The load balancer stays with EXTERNAL-IP “pending”**: The cloud could not create it: look at the events with `kubectl describe svc grafana-internal -n monitoring`. On EKS, check that the AWS Load Balancer Controller is installed; on IONOS, that the node pool is public.
- **Annotated pods do not appear in Targets**: The RBAC from step 4 is missing (the Prometheus logs show “forbidden”), or the annotations are on the Deployment rather than on the pod template. The value of `prometheus.io/scrape` must be the text `"true"`, in quotes.

## Next steps

- Define alerts with **Alertmanager** (for example, target down or disk almost full) and send them to the team’s channel.
- Monitor the cluster itself with **kube-state-metrics** and **node-exporter**, or move to the **kube-prometheus-stack** Helm chart when you need the whole set.
- Store the Grafana dashboards as code (dashboard provisioning) in the repository.
- For long retention or several clusters, consider a managed Prometheus service from your cloud, or Thanos or Mimir.
- Deploy the manifests from a pipeline (see the CI/CD with GitHub Actions guide).
