v2.41.1

Configuring Prometheus in EKS Anywhere package spec

Example

apiVersion: packages.eks.amazonaws.com/v1alpha1
kind: Package
metadata:
  name: generated-prometheus
  namespace: eksa-packages-<cluster-name>
spec:
  packageName: prometheus
  targetNamespace: observability
  config: |
    server:
      replicaCount: 2
      statefulSet:
        enabled: true

Configurable parameters and default values under spec.config

Parameter Description Default
General
rbac.create Specifies if clusterRole / role and clusterRoleBinding / roleBinding will be created for prometheus-server and node-exporter true
sourceRegistry Specifies image source registry for prometheus-server and node-exporter "783794618700.dkr.ecr.us-west-2.amazonaws.com"
Node-Exporter
nodeExporter.enabled Indicates if node-exporter is enabled true
nodeExporter.hostNetwork Indicates if node-exporter shares the host network namespace true
nodeExporter.hostPID Indicates if node-exporter shares the host process ID namespace true
nodeExporter.image.pullPolicy Specifies node-exporter image pull policy: IfNotPresent, Always, Never "IfNotPresent"
nodeExporter.image.repository Specifies node-exporter image repository "prometheus/node-exporter"
nodeExporter.resources Specifies resource requests and limits of the node-exporter container. Refer to the Kubernetes API documentation ResourceRequirements field for more details {}
nodeExporter.service Specifies how to expose node-exporter as a network service See footnote 1
nodeExporter.tolerations Specifies node tolerations for node-exporter scheduling to nodes with taints. Refer to the Kubernetes API documentation toleration field for more details. See footnote 2
serviceAccounts.nodeExporter.annotations Specifies node-exporter service account annotations {}
serviceAccounts.nodeExporter.create Indicates if node-exporter service account will be created true
serviceAccounts.nodeExporter.name Specifies node-exporter service account name ""
Prometheus-Server
server.enabled Indicates if prometheus-server is enabled true
server.global.evaluation_interval Specifies how frequently the prometheus-server rules are evaluated "1m"
server.global.scrape_interval Specifies how frequently prometheus-server will scrape targets "1m"
server.global.scrape_timeout Specifies how long until a prometheus-server scrape request times out "10s"
server.image.pullPolicy Specifies prometheus-server image pull policy: IfNotPresent, Always, Never "IfNotPresent"
server.image.repository Specifies prometheus-server image repository "prometheus/prometheus"
server.name Specifies prometheus-server container name "server"
server.persistentVolume.accessModes Specifies prometheus-server data Persistent Volume access modes "ReadWriteOnce"
server.persistentVolume.enabled Indicates if prometheus-server will create/use a Persistent Volume Claim true
server.persistentVolume.existingClaim Specifies prometheus-server data Persistent Volume existing claim name. It requires server.persistentVolume.enabled: true. If defined, PVC must be created manually before volume will be bound ""
server.persistentVolume.size Specifies prometheus-server data Persistent Volume size "8Gi"
server.remoteRead Specifies prometheus-server remote read configs. Refer to Prometheus docs remote_read for more details []
server.remoteWrite Specifies prometheus-server remote write configs. Refer to Prometheus docs remote_write for more details []
server.replicaCount Specifies the replicaCount for prometheus-server deployment / statefulSet. Note: server.statefulSet.enabled should be set to true if server.replicaCount is greater than 1 1
server.resources Specifies resource requests and limits of the prometheus-server container. Refer to the Kubernetes API documentation ResourceRequirements field for more details {}
server.retention Specifies prometheus-server data retention period "15d"
server.service Specifies how to expose prometheus-server as a network service See footnote 3
server.statefulSet.enabled Indicates if prometheus-server is deployed as a statefulSet. If set to false, prometheus-server will be deployed as a deployment false
serverFiles.“prometheus.yml”.scrape_configs Specifies a set of targets and parameters for prometheus-server describing how to scrape them. Refer to Prometheus docs scrape_config for more details See footnote 4
serviceAccounts.server.annotations Specifies prometheus-server service account annotations {}
serviceAccounts.server.create Indicates if prometheus-server service account will be created true
serviceAccounts.server.name Specifies prometheus-server service account name ""

  1. Node-exporter service is exposed as a clusterIP with port: 9100 (controlled by nodeExporter.service.servicePort below) and targetPort: 9100 (controlled by nodeExporter.service.hostPort below) by default. Note the annotation prometheus.io/scrape: "true" is mandatory in order for node-exporter to be discovered by prometheus-server as a scrape target. See below specification for details.

    apiVersion: packages.eks.amazonaws.com/v1alpha1
    kind: Package
    ...
    spec:
      config: |
        nodeExporter:
          service:
            annotations:
              prometheus.io/scrape: "true"
            hostPort: 9100
            servicePort: 9100
            type: ClusterIP    
    
     ↩︎
  2. Node-exporter pods have the following toleration by default, which allows daemonSet to be scheduled on control plane node.

    apiVersion: packages.eks.amazonaws.com/v1alpha1
    kind: Package
    ...
    spec:
      config: |
        nodeExporter:
          tolerations:
            # For K8 version prior to 1.24
            - key: "node-role.kubernetes.io/master"
              operator: "Exists"
              effect: "NoSchedule"
            # For K8 version 1.24+
            - key: "node-role.kubernetes.io/control-plane"
              operator: "Exists"
              effect: "NoSchedule"    
    
     ↩︎
  3. Prometheus-server service is exposed as a clusterIP with port: 9090 (controlled by server.service.servicePort below) and targetPort: 9090 (not overridable) by default. See below specification for details.

    apiVersion: packages.eks.amazonaws.com/v1alpha1
    kind: Package
    ...
    spec:
      config: |
        server:
          service:
            enabled: true
            servicePort: 9090
            type: ClusterIP    
    
     ↩︎
  4. Prometheus-server by default has the following scrape configs.

    apiVersion: packages.eks.amazonaws.com/v1alpha1
    kind: Package
    ...
    spec:
      config: | 
        serverFiles:
          prometheus.yml:
            scrape_configs:
              - job_name: prometheus
                honor_timestamps: true
                scrape_interval: 1m
                scrape_timeout: 10s
                metrics_path: /metrics
                scheme: http
                follow_redirects: true
                enable_http2: true
                static_configs:
                - targets:
                  - localhost:9090
              - job_name: kubernetes-apiservers
                honor_timestamps: true
                scrape_interval: 1m
                scrape_timeout: 10s
                metrics_path: /metrics
                scheme: https
                authorization:
                  type: Bearer
                  credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
                tls_config:
                  ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
                  insecure_skip_verify: false
                follow_redirects: true
                enable_http2: true
                relabel_configs:
                - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
                  separator: ;
                  regex: default;kubernetes;https
                  replacement: $1
                  action: keep
                kubernetes_sd_configs:
                - role: endpoints
                  kubeconfig_file: ""
                  follow_redirects: true
                  enable_http2: true
              - job_name: kubernetes-nodes
                honor_timestamps: true
                scrape_interval: 1m
                scrape_timeout: 10s
                metrics_path: /metrics
                scheme: https
                authorization:
                  type: Bearer
                  credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
                tls_config:
                  ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
                  insecure_skip_verify: false
                follow_redirects: true
                enable_http2: true
                relabel_configs:
                - separator: ;
                  regex: __meta_kubernetes_node_label_(.+)
                  replacement: $1
                  action: labelmap
                - separator: ;
                  regex: (.*)
                  target_label: __address__
                  replacement: kubernetes.default.svc:443
                  action: replace
                - source_labels: [__meta_kubernetes_node_name]
                  separator: ;
                  regex: (.+)
                  target_label: __metrics_path__
                  replacement: /api/v1/nodes/$1/proxy/metrics
                  action: replace
                kubernetes_sd_configs:
                - role: node
                  kubeconfig_file: ""
                  follow_redirects: true
                  enable_http2: true
              - job_name: kubernetes-nodes-cadvisor
                honor_timestamps: true
                scrape_interval: 1m
                scrape_timeout: 10s
                metrics_path: /metrics
                scheme: https
                authorization:
                  type: Bearer
                  credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
                tls_config:
                  ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
                  insecure_skip_verify: false
                follow_redirects: true
                enable_http2: true
                relabel_configs:
                - separator: ;
                  regex: __meta_kubernetes_node_label_(.+)
                  replacement: $1
                  action: labelmap
                - separator: ;
                  regex: (.*)
                  target_label: __address__
                  replacement: kubernetes.default.svc:443
                  action: replace
                - source_labels: [__meta_kubernetes_node_name]
                  separator: ;
                  regex: (.+)
                  target_label: __metrics_path__
                  replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
                  action: replace
                kubernetes_sd_configs:
                - role: node
                  kubeconfig_file: ""
                  follow_redirects: true
                  enable_http2: true
              - job_name: kubernetes-service-endpoints
                honor_labels: true
                honor_timestamps: true
                scrape_interval: 1m
                scrape_timeout: 10s
                metrics_path: /metrics
                scheme: http
                follow_redirects: true
                enable_http2: true
                relabel_configs:
                - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
                  separator: ;
                  regex: "true"
                  replacement: $1
                  action: keep
                - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape_slow]
                  separator: ;
                  regex: "true"
                  replacement: $1
                  action: drop
                - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
                  separator: ;
                  regex: (https?)
                  target_label: __scheme__
                  replacement: $1
                  action: replace
                - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
                  separator: ;
                  regex: (.+)
                  target_label: __metrics_path__
                  replacement: $1
                  action: replace
                - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
                  separator: ;
                  regex: (.+?)(?::\d+)?;(\d+)
                  target_label: __address__
                  replacement: $1:$2
                  action: replace
                - separator: ;
                  regex: __meta_kubernetes_service_annotation_prometheus_io_param_(.+)
                  replacement: __param_$1
                  action: labelmap
                - separator: ;
                  regex: __meta_kubernetes_service_label_(.+)
                  replacement: $1
                  action: labelmap
                - source_labels: [__meta_kubernetes_namespace]
                  separator: ;
                  regex: (.*)
                  target_label: namespace
                  replacement: $1
                  action: replace
                - source_labels: [__meta_kubernetes_service_name]
                  separator: ;
                  regex: (.*)
                  target_label: service
                  replacement: $1
                  action: replace
                - source_labels: [__meta_kubernetes_pod_node_name]
                  separator: ;
                  regex: (.*)
                  target_label: node
                  replacement: $1
                  action: replace
                kubernetes_sd_configs:
                - role: endpoints
                  kubeconfig_file: ""
                  follow_redirects: true
                  enable_http2: true
              - job_name: kubernetes-service-endpoints-slow
                honor_labels: true
                honor_timestamps: true
                scrape_interval: 5m
                scrape_timeout: 30s
                metrics_path: /metrics
                scheme: http
                follow_redirects: true
                enable_http2: true
                relabel_configs:
                - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape_slow]
                  separator: ;
                  regex: "true"
                  replacement: $1
                  action: keep
                - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
                  separator: ;
                  regex: (https?)
                  target_label: __scheme__
                  replacement: $1
                  action: replace
                - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
                  separator: ;
                  regex: (.+)
                  target_label: __metrics_path__
                  replacement: $1
                  action: replace
                - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
                  separator: ;
                  regex: (.+?)(?::\d+)?;(\d+)
                  target_label: __address__
                  replacement: $1:$2
                  action: replace
                - separator: ;
                  regex: __meta_kubernetes_service_annotation_prometheus_io_param_(.+)
                  replacement: __param_$1
                  action: labelmap
                - separator: ;
                  regex: __meta_kubernetes_service_label_(.+)
                  replacement: $1
                  action: labelmap
                - source_labels: [__meta_kubernetes_namespace]
                  separator: ;
                  regex: (.*)
                  target_label: namespace
                  replacement: $1
                  action: replace
                - source_labels: [__meta_kubernetes_service_name]
                  separator: ;
                  regex: (.*)
                  target_label: service
                  replacement: $1
                  action: replace
                - source_labels: [__meta_kubernetes_pod_node_name]
                  separator: ;
                  regex: (.*)
                  target_label: node
                  replacement: $1
                  action: replace
                kubernetes_sd_configs:
                - role: endpoints
                  kubeconfig_file: ""
                  follow_redirects: true
                  enable_http2: true
              - job_name: prometheus-pushgateway
                honor_labels: true
                honor_timestamps: true
                scrape_interval: 1m
                scrape_timeout: 10s
                metrics_path: /metrics
                scheme: http
                follow_redirects: true
                enable_http2: true
                relabel_configs:
                - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
                  separator: ;
                  regex: pushgateway
                  replacement: $1
                  action: keep
                kubernetes_sd_configs:
                - role: service
                  kubeconfig_file: ""
                  follow_redirects: true
                  enable_http2: true
              - job_name: kubernetes-services
                honor_labels: true
                honor_timestamps: true
                params:
                  module:
                  - http_2xx
                scrape_interval: 1m
                scrape_timeout: 10s
                metrics_path: /probe
                scheme: http
                follow_redirects: true
                enable_http2: true
                relabel_configs:
                - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
                  separator: ;
                  regex: "true"
                  replacement: $1
                  action: keep
                - source_labels: [__address__]
                  separator: ;
                  regex: (.*)
                  target_label: __param_target
                  replacement: $1
                  action: replace
                - separator: ;
                  regex: (.*)
                  target_label: __address__
                  replacement: blackbox
                  action: replace
                - source_labels: [__param_target]
                  separator: ;
                  regex: (.*)
                  target_label: instance
                  replacement: $1
                  action: replace
                - separator: ;
                  regex: __meta_kubernetes_service_label_(.+)
                  replacement: $1
                  action: labelmap
                - source_labels: [__meta_kubernetes_namespace]
                  separator: ;
                  regex: (.*)
                  target_label: namespace
                  replacement: $1
                  action: replace
                - source_labels: [__meta_kubernetes_service_name]
                  separator: ;
                  regex: (.*)
                  target_label: service
                  replacement: $1
                  action: replace
                kubernetes_sd_configs:
                - role: service
                  kubeconfig_file: ""
                  follow_redirects: true
                  enable_http2: true
              - job_name: kubernetes-pods
                honor_labels: true
                honor_timestamps: true
                scrape_interval: 1m
                scrape_timeout: 10s
                metrics_path: /metrics
                scheme: http
                follow_redirects: true
                enable_http2: true
                relabel_configs:
                - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
                  separator: ;
                  regex: "true"
                  replacement: $1
                  action: keep
                - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape_slow]
                  separator: ;
                  regex: "true"
                  replacement: $1
                  action: drop
                - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]
                  separator: ;
                  regex: (https?)
                  target_label: __scheme__
                  replacement: $1
                  action: replace
                - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
                  separator: ;
                  regex: (.+)
                  target_label: __metrics_path__
                  replacement: $1
                  action: replace
                - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
                  separator: ;
                  regex: (.+?)(?::\d+)?;(\d+)
                  target_label: __address__
                  replacement: $1:$2
                  action: replace
                - separator: ;
                  regex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+)
                  replacement: __param_$1
                  action: labelmap
                - separator: ;
                  regex: __meta_kubernetes_pod_label_(.+)
                  replacement: $1
                  action: labelmap
                - source_labels: [__meta_kubernetes_namespace]
                  separator: ;
                  regex: (.*)
                  target_label: namespace
                  replacement: $1
                  action: replace
                - source_labels: [__meta_kubernetes_pod_name]
                  separator: ;
                  regex: (.*)
                  target_label: pod
                  replacement: $1
                  action: replace
                - source_labels: [__meta_kubernetes_pod_phase]
                  separator: ;
                  regex: Pending|Succeeded|Failed|Completed
                  replacement: $1
                  action: drop
                kubernetes_sd_configs:
                - role: pod
                  kubeconfig_file: ""
                  follow_redirects: true
                  enable_http2: true
              - job_name: kubernetes-pods-slow
                honor_labels: true
                honor_timestamps: true
                scrape_interval: 5m
                scrape_timeout: 30s
                metrics_path: /metrics
                scheme: http
                follow_redirects: true
                enable_http2: true
                relabel_configs:
                - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape_slow]
                  separator: ;
                  regex: "true"
                  replacement: $1
                  action: keep
                - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]
                  separator: ;
                  regex: (https?)
                  target_label: __scheme__
                  replacement: $1
                  action: replace
                - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
                  separator: ;
                  regex: (.+)
                  target_label: __metrics_path__
                  replacement: $1
                  action: replace
                - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
                  separator: ;
                  regex: (.+?)(?::\d+)?;(\d+)
                  target_label: __address__
                  replacement: $1:$2
                  action: replace
                - separator: ;
                  regex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+)
                  replacement: __param_$1
                  action: labelmap
                - separator: ;
                  regex: __meta_kubernetes_pod_label_(.+)
                  replacement: $1
                  action: labelmap
                - source_labels: [__meta_kubernetes_namespace]
                  separator: ;
                  regex: (.*)
                  target_label: namespace
                  replacement: $1
                  action: replace
                - source_labels: [__meta_kubernetes_pod_name]
                  separator: ;
                  regex: (.*)
                  target_label: pod
                  replacement: $1
                  action: replace
                - source_labels: [__meta_kubernetes_pod_phase]
                  separator: ;
                  regex: Pending|Succeeded|Failed|Completed
                  replacement: $1
                  action: drop
                kubernetes_sd_configs:
                - role: pod
                  kubeconfig_file: ""
                  follow_redirects: true
                  enable_http2: true
    
     ↩︎