A practical guide to setting up monitoring with Prometheus and Grafana on a managed Kubernetes cluster: AKS (Azure), GKE (Google Cloud), EKS (AWS) or IONOS Managed Kubernetes. With YAML manifests ready to adapt, credentials in Secrets, access only through the internal network and the differences between clouds. All the manifests and commands are also in the README.md on the left.
What you’ll learn
- Deploying Prometheus and Grafana on Kubernetes with your own YAML manifests, without depending on a chart.
- Monitoring a service with
/metricsprotected by username, password and HTTPS, and discovering pods automatically. - Storing passwords and certificates in Secrets, not in ConfigMaps or in the YAML.
- Giving private access to Grafana on AKS, GKE, EKS and IONOS.
- Reloading the configuration, checking the status and solving the most common errors.
Prerequisites
- An AKS, GKE, EKS or IONOS Managed Kubernetes cluster, and
kubectlconnected to it (see the Kubernetes administration guide). - Permissions to create namespaces, ClusterRoles and persistent volumes.
- Windows with PowerShell; Docker Desktop only to validate the configuration.
- On EKS, the Amazon EBS CSI driver add-on for the disks, and the AWS Load Balancer Controller if you are going to use an internal load balancer.
- The image versions are examples: always pin a specific version (never
latest) and check the latest stable one in the Prometheus and Grafana release notes. - If your cluster only pulls images from a private registry, copy
prom/prometheusandgrafana/grafanathere and add itsimagePullSecrets.
Step by step
1. Prepare the namespace and check the storage
Prometheus and Grafana store data on disk, so they need a persistent volume. Check that you are on the right cluster and which StorageClass will be used by default:
# The right cluster and a dedicated namespace for monitoring
kubectl config current-context
kubectl create namespace monitoring
# Which StorageClass will be used for the disks (the one marked "(default)")
kubectl get storageclass
If none shows as (default), add storageClassName to the volumeClaimTemplates in steps 5 and 6. On EKS, without the EBS CSI driver the volumes stay in Pending, and on new clusters gp2 is no longer marked as the default.
2. Store the credentials in Secrets
Passwords don’t belong in the Prometheus ConfigMap or in the repository: anyone with read access to the namespace or to Git would see them. Create them as Secrets from local files, which you delete afterwards:
# Password and certificate authority (CA) certificate of the service you are going to monitor
kubectl create secret generic prometheus-target -n monitoring `
--from-file=password=.\target-password.txt `
--from-file=ca.crt=.\internal-ca.crt
# Grafana administrator user and password
kubectl create secret generic grafana-admin -n monitoring `
--from-literal=admin-user=admin `
--from-file=admin-password=.\grafana-password.txt
# Delete the local files with the passwords once the Secret has been created
Remove-Item .\target-password.txt, .\grafana-password.txt
For the service you are going to monitor, use a user that can only read metrics. If its certificate is signed by an internal CA, keep the ca.crt: that way Prometheus validates the certificate instead of skipping the check.
3. Configure Prometheus
The configuration goes in a ConfigMap. This example monitors three things: Prometheus itself, a service with protected /metrics (with password_file and ca_file, which read the Secret from step 2) and the cluster pods that request it with annotations:
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-config
namespace: monitoring
data:
prometheus.yml: |
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
# Prometheus itself
- job_name: prometheus
static_configs:
- targets: ['localhost:9090']
# A service with /metrics protected by username and password, and HTTPS with its own CA
- job_name: my-service
scrape_interval: 30s
scheme: https
metrics_path: /metrics
basic_auth:
username: metrics-reader
password_file: /etc/prometheus/secrets/password
tls_config:
ca_file: /etc/prometheus/secrets/ca.crt
static_configs:
- targets: ['metrics.my-company.internal:8443']
# Cluster pods that request it with prometheus.io/* annotations
- job_name: pods
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: "true"
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
target_label: __address__
- source_labels: [__meta_kubernetes_namespace]
target_label: namespace
- source_labels: [__meta_kubernetes_pod_name]
target_label: pod
For Prometheus to monitor a pod, add prometheus.io/scrape: "true", prometheus.io/port and, if it is not /metrics, prometheus.io/path to its template. Before applying it, validate the configuration with promtool:
# Validate the configuration before applying it (requires Docker Desktop)
# Save the prometheus.yml block to a separate file for this check
docker run --rm -v "${PWD}:/cfg" --entrypoint promtool prom/prometheus:v3.2.1 check config /cfg/prometheus.yml
4. Grant read permissions to discover pods
Pod discovery queries the Kubernetes API, so Prometheus needs its own service account with read-only permissions:
apiVersion: v1
kind: ServiceAccount
metadata:
name: prometheus
namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: prometheus-read
rules:
- apiGroups: [""]
resources: [nodes, nodes/metrics, services, endpoints, pods]
verbs: [get, list, watch]
- nonResourceURLs: [/metrics]
verbs: [get]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: prometheus-read
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: prometheus-read
subjects:
- kind: ServiceAccount
name: prometheus
namespace: monitoring
5. Deploy Prometheus
A StatefulSet with its own disk, running without root privileges (the nobody user, 65534, from the official image), with health probes, memory limits and 15 days of retention. --web.enable-lifecycle lets you reload the configuration without restarting:
apiVersion: v1
kind: Service
metadata:
name: prometheus
namespace: monitoring
spec:
selector:
app: prometheus
ports:
- name: web
port: 9090
targetPort: 9090
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: prometheus
namespace: monitoring
spec:
serviceName: prometheus
replicas: 1
selector:
matchLabels:
app: prometheus
template:
metadata:
labels:
app: prometheus
spec:
serviceAccountName: prometheus
securityContext:
runAsNonRoot: true
runAsUser: 65534
runAsGroup: 65534
fsGroup: 65534
# nodeSelector: # optional: a node pool dedicated to monitoring
# pool: monitoring
containers:
- name: prometheus
image: prom/prometheus:v3.2.1
args:
- --config.file=/etc/prometheus/prometheus.yml
- --storage.tsdb.path=/prometheus
- --storage.tsdb.retention.time=15d
- --web.enable-lifecycle
ports:
- containerPort: 9090
readinessProbe:
httpGet: { path: /-/ready, port: 9090 }
livenessProbe:
httpGet: { path: /-/healthy, port: 9090 }
initialDelaySeconds: 30
resources:
requests: { cpu: 250m, memory: 1Gi }
limits: { memory: 2Gi }
securityContext:
allowPrivilegeEscalation: false
volumeMounts:
- name: config
mountPath: /etc/prometheus
- name: secrets
mountPath: /etc/prometheus/secrets
readOnly: true
- name: data
mountPath: /prometheus
volumes:
- name: config
configMap:
name: prometheus-config
- name: secrets
secret:
secretName: prometheus-target
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: [ReadWriteOnce]
resources:
requests:
storage: 20Gi
The disk size depends on how many series you monitor and on the retention: start with 20 GB and keep an eye on usage (step 9).
6. Deploy Grafana with Prometheus already configured
Grafana reads the administrator user from the Secret and the data source from a provisioning file, so it starts already connected to Prometheus through its internal cluster name:
apiVersion: v1
kind: ConfigMap
metadata:
name: grafana-datasources
namespace: monitoring
data:
prometheus.yaml: |
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://prometheus.monitoring.svc.cluster.local:9090
isDefault: true
---
apiVersion: v1
kind: Service
metadata:
name: grafana
namespace: monitoring
spec:
selector:
app: grafana
ports:
- name: web
port: 3000
targetPort: 3000
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: grafana
namespace: monitoring
spec:
serviceName: grafana
replicas: 1
selector:
matchLabels:
app: grafana
template:
metadata:
labels:
app: grafana
spec:
securityContext:
runAsNonRoot: true
runAsUser: 472
fsGroup: 472
containers:
- name: grafana
image: grafana/grafana:11.6.0
ports:
- containerPort: 3000
env:
- name: GF_SECURITY_ADMIN_USER
valueFrom:
secretKeyRef: { name: grafana-admin, key: admin-user }
- name: GF_SECURITY_ADMIN_PASSWORD
valueFrom:
secretKeyRef: { name: grafana-admin, key: admin-password }
readinessProbe:
httpGet: { path: /api/health, port: 3000 }
resources:
requests: { cpu: 100m, memory: 256Mi }
limits: { memory: 512Mi }
securityContext:
allowPrivilegeEscalation: false
volumeMounts:
- name: data
mountPath: /var/lib/grafana
- name: datasources
mountPath: /etc/grafana/provisioning/datasources
readOnly: true
volumes:
- name: datasources
configMap:
name: grafana-datasources
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: [ReadWriteOnce]
resources:
requests:
storage: 5Gi
Apply everything and wait for the pods to be ready:
kubectl apply -f .\prometheus-config.yaml
kubectl apply -f .\prometheus-rbac.yaml
kubectl apply -f .\prometheus.yaml
kubectl apply -f .\grafana.yaml
kubectl get pods,pvc,svc -n monitoring
kubectl rollout status statefulset/prometheus -n monitoring
kubectl rollout status statefulset/grafana -n monitoring
7. Access Grafana securely
To start with, you don’t need to expose anything: kubectl port-forward opens a tunnel from your machine, authenticated with your cluster credentials:
# Grafana at http://localhost:3000 (username and password from the grafana-admin Secret)
kubectl port-forward svc/grafana 3000:3000 -n monitoring
# Prometheus at http://localhost:9090 (Status > Targets: all targets UP)
kubectl port-forward svc/prometheus 9090:9090 -n monitoring
In Grafana, go to Explore, choose Prometheus and query up: each monitored target appears with value 1 if it responds.
8. Publish Grafana on the internal network only
When the team needs access without kubectl, use an internal load balancer, with a private IP on the cluster network, and also restrict which networks can connect:
apiVersion: v1
kind: Service
metadata:
name: grafana-internal
namespace: monitoring
annotations:
# Only the line for your cloud:
service.beta.kubernetes.io/azure-load-balancer-internal: "true" # AKS
# networking.gke.io/load-balancer-type: "Internal" # GKE
# service.beta.kubernetes.io/aws-load-balancer-type: "external" # EKS (AWS Load Balancer Controller)
# service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip" # EKS
# service.beta.kubernetes.io/aws-load-balancer-scheme: "internal" # EKS (already the default value)
spec:
type: LoadBalancer
loadBalancerSourceRanges:
- 10.20.0.0/16 # only your office network or your VPN, never 0.0.0.0/0
selector:
app: grafana
ports:
- name: web
port: 3000
targetPort: 3000
| Cloud | Internal load balancer | Notes |
|---|---|---|
| AKS | azure-load-balancer-internal: "true" | Private IP on the cluster VNet. |
| GKE | networking.gke.io/load-balancer-type: "Internal" | Private IP on the VPC. |
| EKS | aws-load-balancer-scheme: "internal" with the AWS Load Balancer Controller | Creates an internal Network Load Balancer. |
| IONOS | There is no internal load balancer for Services | LoadBalancer Services only work on public node pools and attach a fixed public IP to a node: for Grafana, port-forward or an Ingress with TLS and restricted access is a better option. |
loadBalancerSourceRanges restricts the source IPs: GKE applies it on internal and external load balancers; on EKS it only takes effect if the load balancer preserves the client IP (with nlb-target-type: "ip" this is disabled by default). On any cloud, reinforce it with network rules (NSG, VPC firewall or security groups).
Never publish Grafana or Prometheus with 0.0.0.0/0: Prometheus has no username or password by default, and anyone who reaches it can read all the metrics.
9. Day-to-day operations
To change what is monitored, edit the ConfigMap, apply it and ask Prometheus to reload. Here you also have how to check the status, read the logs and keep an eye on disk space:
# Change the Prometheus configuration without restarting it
kubectl apply -f .\prometheus-config.yaml
# (wait about a minute for the pod to see the new ConfigMap)
kubectl port-forward svc/prometheus 9090:9090 -n monitoring
curl.exe -X POST http://localhost:9090/-/reload # in another PowerShell window
# Check the status
curl.exe http://localhost:9090/-/ready
kubectl logs statefulset/prometheus -n monitoring --tail=50
kubectl logs statefulset/grafana -n monitoring --tail=50
# Prometheus disk space
kubectl exec prometheus-0 -n monitoring -- df -h /prometheus
If the reload fails, Prometheus keeps the previous configuration and says so in its logs: fix the error and reload again.
Common errors
The target shows as DOWN with “x509: certificate signed by unknown authority”
Prometheus does not trust the service certificate. Add the CA that signs it in tls_config.ca_file (step 2). Don’t use insecure_skip_verify: true: anyone could then impersonate the service and receive the password.
The target shows as DOWN with “401 Unauthorized”
The username or password is wrong, or the file has a trailing line break. Create the Secret again with kubectl create secret … --dry-run=client -o yaml | kubectl apply -f - and restart the pod.
The pod stays in Pending
It is usually the volume: kubectl describe pvc -n monitoring tells you why. There is no default StorageClass, the EBS CSI driver is missing on EKS, or the nodeSelector does not match any node.
CrashLoopBackOff with “permission denied” on /prometheus or /var/lib/grafana
The disk does not have permissions for the container user. Check that the pod’s securityContext includes fsGroup (65534 for Prometheus, 472 for Grafana).
Grafana cannot connect to Prometheus
Check the data source URL: inside the cluster it is http://prometheus.monitoring.svc.cluster.local:9090, never localhost. Also check that the Prometheus pod is ready.
The load balancer stays with EXTERNAL-IP “pending”
The cloud could not create it: look at the events with kubectl describe svc grafana-internal -n monitoring. On EKS, check that the AWS Load Balancer Controller is installed; on IONOS, that the node pool is public.
Annotated pods do not appear in Targets
The RBAC from step 4 is missing (the Prometheus logs show “forbidden”), or the annotations are on the Deployment rather than on the pod template. The value of prometheus.io/scrape must be the text "true", in quotes.
Next steps
- Define alerts with Alertmanager (for example, target down or disk almost full) and send them to the team’s channel.
- Monitor the cluster itself with kube-state-metrics and node-exporter, or move to the kube-prometheus-stack Helm chart when you need the whole set.
- Store the Grafana dashboards as code (dashboard provisioning) in the repository.
- For long retention or several clusters, consider a managed Prometheus service from your cloud, or Thanos or Mimir.
- Deploy the manifests from a pipeline (see the CI/CD with GitHub Actions guide).