v3.15.1

Starting with v3.12.0, the default scrape jobs are defined under scrapeConfigs, keyed by job name. To change a default job, set its fields under its key; to turn it off, set its enabled field to false. To add a scrape job, add a key under scrapeConfigs.

Jobs set under serverFiles."prometheus.yml".scrape_configs are added to the scrapeConfigs jobs, not used instead of them. A serverFiles job with the same job_name as a scrapeConfigs job replaces that job. Other job names must be unique across scrapeConfigs, serverFiles and extraScrapeConfigs, or Prometheus rejects the configuration.

The package build 3.15.1-383aa49f… (the CurrentVersion column of kubectl get packages) doesn’t apply serverFiles."prometheus.yml".scrape_configs jobs, and setting extraScrapeConfigs makes its configuration invalid. With that build, use scrapeConfigs only.

Configuring Prometheus in EKS Anywhere package spec

Example

apiVersion: packages.eks.amazonaws.com/v1alpha1
kind: Package
metadata:
  name: generated-prometheus
  namespace: eksa-packages-<cluster-name>
spec:
  packageName: prometheus
  targetNamespace: observability
  config: |
    server:
      replicaCount: 2
      statefulSet:
        enabled: true

Configurable parameters and default values under spec.config

Parameter Description Default
General
rbac.create Specifies if clusterRole / role and clusterRoleBinding / roleBinding will be created for prometheus-server and node-exporter true
sourceRegistry Specifies image source registry for prometheus-server and node-exporter "783794618700.dkr.ecr.us-west-2.amazonaws.com"
Node-Exporter
nodeExporter.enabled Indicates if node-exporter is enabled true
nodeExporter.hostNetwork Indicates if node-exporter shares the host network namespace true
nodeExporter.hostPID Indicates if node-exporter shares the host process ID namespace true
nodeExporter.image.pullPolicy Specifies node-exporter image pull policy: IfNotPresent, Always, Never "IfNotPresent"
nodeExporter.image.repository Specifies node-exporter image repository "prometheus/node-exporter"
nodeExporter.resources Specifies resource requests and limits of the node-exporter container. Refer to the Kubernetes API documentation ResourceRequirements field for more details {}
nodeExporter.service Specifies how to expose node-exporter as a network service See footnote 1
nodeExporter.tolerations Specifies node tolerations for node-exporter scheduling to nodes with taints. Refer to the Kubernetes API documentation toleration field for more details. See footnote 2
serviceAccounts.nodeExporter.annotations Specifies node-exporter service account annotations {}
serviceAccounts.nodeExporter.create Indicates if node-exporter service account will be created true
serviceAccounts.nodeExporter.name Specifies node-exporter service account name ""
Prometheus-Server
server.global.evaluation_interval Specifies how frequently the prometheus-server rules are evaluated "1m"
server.global.scrape_interval Specifies how frequently prometheus-server will scrape targets "1m"
server.global.scrape_timeout Specifies how long until a prometheus-server scrape request times out "10s"
server.image.pullPolicy Specifies prometheus-server image pull policy: IfNotPresent, Always, Never Not set; Kubernetes uses IfNotPresent
server.image.repository Specifies prometheus-server image repository "prometheus/prometheus"
server.name Specifies prometheus-server container name "server"
server.persistentVolume.accessModes Specifies prometheus-server data Persistent Volume access modes "ReadWriteOnce"
server.persistentVolume.enabled Indicates if prometheus-server will create/use a Persistent Volume Claim true
server.persistentVolume.existingClaim Specifies prometheus-server data Persistent Volume existing claim name. It requires server.persistentVolume.enabled: true. If defined, PVC must be created manually before volume will be bound ""
server.persistentVolume.size Specifies prometheus-server data Persistent Volume size "8Gi"
server.remoteRead Specifies prometheus-server remote read configs. Refer to Prometheus docs remote_read for more details []
server.remoteWrite Specifies prometheus-server remote write configs. Refer to Prometheus docs remote_write for more details []
server.replicaCount Specifies the replicaCount for prometheus-server deployment / statefulSet. Note: server.statefulSet.enabled should be set to true if server.replicaCount is greater than 1 1
server.resources Specifies resource requests and limits of the prometheus-server container. Refer to the Kubernetes API documentation ResourceRequirements field for more details {}
server.retention Specifies prometheus-server data retention period "15d"
server.service Specifies how to expose prometheus-server as a network service See footnote 3
server.statefulSet.enabled Indicates if prometheus-server is deployed as a statefulSet. If set to false, prometheus-server will be deployed as a deployment false
scrapeConfigs Specifies prometheus-server scrape jobs, keyed by job name. Each entry takes the fields of a Prometheus scrape_config , plus enabled (defaults to true), job_name (defaults to the key), and pre_relabel_configs / post_relabel_configs, templated YAML strings whose rules are added before and after relabel_configs See footnote 4
serverFiles.“prometheus.yml”.scrape_configs Specifies additional prometheus-server scrape jobs, as a list of Prometheus scrape_config entries. They are added to the scrapeConfigs jobs; a job with the same job_name as a scrapeConfigs job replaces that job. Not applied by package build 3.15.1-383aa49f… Not set
extraScrapeConfigs Specifies additional prometheus-server scrape jobs, as a templated YAML string with a list of Prometheus scrape_config entries. Their job names must not match any other job. Don’t set it with package build 3.15.1-383aa49f… ""
serviceAccounts.server.annotations Specifies prometheus-server service account annotations {}
serviceAccounts.server.create Indicates if prometheus-server service account will be created true
serviceAccounts.server.name Specifies prometheus-server service account name ""

  1. Node-exporter service is exposed as a clusterIP with port: 9100 (controlled by nodeExporter.service.servicePort below) and targetPort: 9100 (controlled by nodeExporter.service.hostPort below) by default. Note the annotation prometheus.io/scrape: "true" is mandatory in order for node-exporter to be discovered by prometheus-server as a scrape target. See below specification for details.

    apiVersion: packages.eks.amazonaws.com/v1alpha1
    kind: Package
    ...
    spec:
      config: |
        nodeExporter:
          service:
            annotations:
              prometheus.io/scrape: "true"
            hostPort: 9100
            servicePort: 9100
            type: ClusterIP    
    
     ↩︎
  2. Node-exporter pods have the following toleration by default, which allows daemonSet to be scheduled on control plane node.

    apiVersion: packages.eks.amazonaws.com/v1alpha1
    kind: Package
    ...
    spec:
      config: |
        nodeExporter:
          tolerations:
            # For K8 version prior to 1.24
            - key: "node-role.kubernetes.io/master"
              operator: "Exists"
              effect: "NoSchedule"
            # For K8 version 1.24+
            - key: "node-role.kubernetes.io/control-plane"
              operator: "Exists"
              effect: "NoSchedule"    
    
     ↩︎
  3. Prometheus-server service is exposed as a clusterIP with port: 9090 (controlled by server.service.servicePort below) and targetPort: 9090 (not overridable) by default. See below specification for details.

    apiVersion: packages.eks.amazonaws.com/v1alpha1
    kind: Package
    ...
    spec:
      config: |
        server:
          service:
            enabled: true
            servicePort: 9090
            type: ClusterIP    
    
     ↩︎
  4. Prometheus-server by default has the following scrape configs.

    apiVersion: packages.eks.amazonaws.com/v1alpha1
    kind: Package
    ...
    spec:
      config: |
        scrapeConfigs:
          prometheus:
            enabled: true
            job_name: ""
            static_configs:
              - targets:
                  - localhost:9090
          kubernetes-api-servers:
            enabled: true
            job_name: ""
            kubernetes_sd_configs:
              - role: endpointslice
            scheme: https
            tls_config:
              ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
            bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
            relabel_configs:
              - source_labels:
                - __meta_kubernetes_namespace
                - __meta_kubernetes_service_name
                - __meta_kubernetes_endpointslice_port_name
                action: keep
                regex: default;kubernetes;https
            pre_relabel_configs: ""
            post_relabel_configs: ""
          kubernetes-nodes:
            enabled: true
            job_name: ""
            scheme: https
            tls_config:
              ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
              insecure_skip_verify: true
            bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
            kubernetes_sd_configs:
              - role: node
            relabel_configs:
              - action: labelmap
                regex: __meta_kubernetes_node_label_(.+)
            pre_relabel_configs: ""
            post_relabel_configs: ""
          kubernetes-nodes-cadvisor:
            enabled: true
            job_name: ""
            scheme: https
            tls_config:
              ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
              insecure_skip_verify: true
            bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
            metrics_path: /metrics/cadvisor
            kubernetes_sd_configs:
              - role: node
            relabel_configs:
              - action: labelmap
                regex: __meta_kubernetes_node_label_(.+)
              - source_labels: [__metrics_path__]
                target_label: metrics_path
            pre_relabel_configs: ""
            post_relabel_configs: ""
          kubernetes-service-endpoints:
            enabled: true
            job_name: ""
            honor_labels: true
            kubernetes_sd_configs:
              - role: endpointslice
            relabel_configs:
              - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
                action: keep
                regex: true
              - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape_slow]
                action: drop
                regex: true
              - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
                action: replace
                target_label: __scheme__
                regex: (https?)
              - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
                action: replace
                target_label: __metrics_path__
                regex: (.+)
              - source_labels:
                - __address__
                - __meta_kubernetes_service_annotation_prometheus_io_port
                action: replace
                target_label: __address__
                regex: (.+?)(?::\d+)?;(\d+)
                replacement: $1:$2
              - action: labelmap
                regex: __meta_kubernetes_service_annotation_prometheus_io_param_(.+)
                replacement: __param_$1
              - action: labelmap
                regex: __meta_kubernetes_service_label_(.+)
              - source_labels: [__meta_kubernetes_namespace]
                action: replace
                target_label: namespace
              - source_labels: [__meta_kubernetes_service_name]
                action: replace
                target_label: service
              - source_labels: [__meta_kubernetes_pod_node_name]
                action: replace
                target_label: node
            pre_relabel_configs: ""
            post_relabel_configs: ""
          kubernetes-service-endpoints-slow:
            enabled: true
            job_name: ""
            honor_labels: true
            scrape_interval: 5m
            scrape_timeout: 30s
            kubernetes_sd_configs:
              - role: endpointslice
            relabel_configs:
              - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape_slow]
                action: keep
                regex: true
              - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
                action: replace
                target_label: __scheme__
                regex: (https?)
              - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
                action: replace
                target_label: __metrics_path__
                regex: (.+)
              - source_labels:
                - __address__
                - __meta_kubernetes_service_annotation_prometheus_io_port
                action: replace
                target_label: __address__
                regex: (.+?)(?::\d+)?;(\d+)
                replacement: $1:$2
              - action: labelmap
                regex: __meta_kubernetes_service_annotation_prometheus_io_param_(.+)
                replacement: __param_$1
              - action: labelmap
                regex: __meta_kubernetes_service_label_(.+)
              - source_labels: [__meta_kubernetes_namespace]
                action: replace
                target_label: namespace
              - source_labels: [__meta_kubernetes_service_name]
                action: replace
                target_label: service
              - source_labels: [__meta_kubernetes_pod_node_name]
                action: replace
                target_label: node
          prometheus-pushgateway:
            enabled: true
            job_name: ""
            honor_labels: true
            kubernetes_sd_configs:
              - role: service
            relabel_configs:
              - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
                action: keep
                regex: pushgateway
            pre_relabel_configs: ""
            post_relabel_configs: ""
          kubernetes-services:
            enabled: true
            job_name: ""
            honor_labels: true
            metrics_path: /probe
            params:
              module: [http_2xx]
            kubernetes_sd_configs:
              - role: service
            relabel_configs:
              - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
                action: keep
                regex: true
              - source_labels: [__address__]
                target_label: __param_target
              - target_label: __address__
                replacement: blackbox
              - source_labels: [__param_target]
                target_label: instance
              - action: labelmap
                regex: __meta_kubernetes_service_label_(.+)
              - source_labels: [__meta_kubernetes_namespace]
                target_label: namespace
              - source_labels: [__meta_kubernetes_service_name]
                target_label: service
            pre_relabel_configs: ""
            post_relabel_configs: ""
          kubernetes-pods:
            enabled: true
            job_name: ""
            honor_labels: true
            kubernetes_sd_configs:
              - role: pod
            relabel_configs:
              - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
                action: keep
                regex: true
              - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape_slow]
                action: drop
                regex: true
              - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]
                action: replace
                regex: (https?)
                target_label: __scheme__
              - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
                action: replace
                target_label: __metrics_path__
                regex: (.+)
              - source_labels:
                - __meta_kubernetes_pod_annotation_prometheus_io_port
                - __meta_kubernetes_pod_ip
                action: replace
                regex: (\d+);(([A-Fa-f0-9]{1,4}::?){1,7}[A-Fa-f0-9]{1,4})
                replacement: '[$2]:$1'
                target_label: __address__
              - source_labels:
                - __meta_kubernetes_pod_annotation_prometheus_io_port
                - __meta_kubernetes_pod_ip
                action: replace
                regex: (\d+);((([0-9]+?)(\.|$)){4})
                replacement: $2:$1
                target_label: __address__
              - action: labelmap
                regex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+)
                replacement: __param_$1
              - action: labelmap
                regex: __meta_kubernetes_pod_label_(.+)
              - source_labels: [__meta_kubernetes_namespace]
                action: replace
                target_label: namespace
              - source_labels: [__meta_kubernetes_pod_name]
                action: replace
                target_label: pod
              - source_labels: [__meta_kubernetes_pod_phase]
                regex: Pending|Succeeded|Failed|Completed
                action: drop
              - source_labels: [__meta_kubernetes_pod_node_name]
                action: replace
                target_label: node
            pre_relabel_configs: ""
            post_relabel_configs: ""
          kubernetes-pods-slow:
            enabled: true
            job_name: ""
            honor_labels: true
            scrape_interval: 5m
            scrape_timeout: 30s
            kubernetes_sd_configs:
              - role: pod
            relabel_configs:
              - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape_slow]
                action: keep
                regex: true
              - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]
                action: replace
                regex: (https?)
                target_label: __scheme__
              - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
                action: replace
                target_label: __metrics_path__
                regex: (.+)
              - source_labels:
                - __meta_kubernetes_pod_annotation_prometheus_io_port
                - __meta_kubernetes_pod_ip
                action: replace
                regex: (\d+);(([A-Fa-f0-9]{1,4}::?){1,7}[A-Fa-f0-9]{1,4})
                replacement: '[$2]:$1'
                target_label: __address__
              - source_labels:
                - __meta_kubernetes_pod_annotation_prometheus_io_port
                - __meta_kubernetes_pod_ip
                action: replace
                regex: (\d+);((([0-9]+?)(\.|$)){4})
                replacement: $2:$1
                target_label: __address__
              - action: labelmap
                regex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+)
                replacement: __param_$1
              - action: labelmap
                regex: __meta_kubernetes_pod_label_(.+)
              - source_labels: [__meta_kubernetes_namespace]
                action: replace
                target_label: namespace
              - source_labels: [__meta_kubernetes_pod_name]
                action: replace
                target_label: pod
              - source_labels: [__meta_kubernetes_pod_phase]
                regex: Pending|Succeeded|Failed|Completed
                action: drop
              - source_labels: [__meta_kubernetes_pod_node_name]
                action: replace
                target_label: node
            pre_relabel_configs: ""
            post_relabel_configs: ""    
    
     ↩︎