prometheus relabeling

Prometheus relabeling 설명

https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config

https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config

 

kubernetes_sd_config에는 role이 여러가지 존재합니다.

• node
• service
• pod
• endpoints
• endpointslice
• ingress

각 role마다 사용할 수 있는 meta label이 다릅니다.

 

Servicemonitor를 이용하면 role이 endpoints로 지정되는데,

https://prometheus.io/docs/prometheus/latest/configuration/configuration/#endpoints

위 링크의 설명을 읽어보면,

endpoint가 service로부터 비롯된 것이면 service role에서 사용할 수 있는 meta label을 사용할 수 있고,

target이 pod인 경우에는 pod role에서 사용할 수 있는 meta label을 사용할 수 있습니다.

 

아래는 각 role마다 사용할 수 있는 meta label이고,

Servicemonitor를 사용할 시 role이 endpoints로 부여되므로 이를 기준으로 사용할 수 있을만한 endpoints, service, pod의 meta label만 발췌해봤습니다.

( node, endpointslice, ingress role의 meta label은 위의 링크 참조 )

---

endpoints

The endpoints role discovers targets from listed endpoints of a service. For each endpoint address one target is discovered per port. If the endpoint is backed by a pod, all additional container ports of the pod, not bound to an endpoint port, are discovered as targets as well.

Available meta labels:

• __meta_kubernetes_namespace: The namespace of the endpoints object.
• __meta_kubernetes_endpoints_name: The names of the endpoints object.
• __meta_kubernetes_endpoints_label_<labelname>: Each label from the endpoints object.
• __meta_kubernetes_endpoints_labelpresent_<labelname>: true for each label from the endpoints object.
• For all targets discovered directly from the endpoints list (those not additionally inferred from underlying pods), the following labels are attached:
  • __meta_kubernetes_endpoint_hostname: Hostname of the endpoint.
  • __meta_kubernetes_endpoint_node_name: Name of the node hosting the endpoint.
  • __meta_kubernetes_endpoint_ready: Set to true or false for the endpoint's ready state.
  • __meta_kubernetes_endpoint_port_name: Name of the endpoint port.
  • __meta_kubernetes_endpoint_port_protocol: Protocol of the endpoint port.
  • __meta_kubernetes_endpoint_address_target_kind: Kind of the endpoint address target.
  • __meta_kubernetes_endpoint_address_target_name: Name of the endpoint address target.

pod

The pod role discovers all pods and exposes their containers as targets. For each declared port of a container, a single target is generated. If a container has no specified ports, a port-free target per container is created for manually adding a port via relabeling.

Available meta labels:

• __meta_kubernetes_namespace: The namespace of the pod object.
• __meta_kubernetes_pod_name: The name of the pod object.
• __meta_kubernetes_pod_ip: The pod IP of the pod object.
• __meta_kubernetes_pod_label_<labelname>: Each label from the pod object.
• __meta_kubernetes_pod_labelpresent_<labelname>: true for each label from the pod object.
• __meta_kubernetes_pod_annotation_<annotationname>: Each annotation from the pod object.
• __meta_kubernetes_pod_annotationpresent_<annotationname>: true for each annotation from the pod object.
• __meta_kubernetes_pod_container_init: true if the container is an InitContainer
• __meta_kubernetes_pod_container_name: Name of the container the target address points to.
• __meta_kubernetes_pod_container_id: ID of the container the target address points to. The ID is in the form <type>://<container_id>.
• __meta_kubernetes_pod_container_image: The image the container is using.
• __meta_kubernetes_pod_container_port_name: Name of the container port.
• __meta_kubernetes_pod_container_port_number: Number of the container port.
• __meta_kubernetes_pod_container_port_protocol: Protocol of the container port.
• __meta_kubernetes_pod_ready: Set to true or false for the pod's ready state.
• __meta_kubernetes_pod_phase: Set to Pending, Running, Succeeded, Failed or Unknown in the lifecycle.
• __meta_kubernetes_pod_node_name: The name of the node the pod is scheduled onto.
• __meta_kubernetes_pod_host_ip: The current host IP of the pod object.
• __meta_kubernetes_pod_uid: The UID of the pod object.
• __meta_kubernetes_pod_controller_kind: Object kind of the pod controller.
• __meta_kubernetes_pod_controller_name: Name of the pod controller.

service

The service role discovers a target for each service port for each service. This is generally useful for blackbox monitoring of a service. The address will be set to the Kubernetes DNS name of the service and respective service port.

Available meta labels:

• __meta_kubernetes_namespace: The namespace of the service object.
• __meta_kubernetes_service_annotation_<annotationname>: Each annotation from the service object.
• __meta_kubernetes_service_annotationpresent_<annotationname>: "true" for each annotation of the service object.
• __meta_kubernetes_service_cluster_ip: The cluster IP address of the service. (Does not apply to services of type ExternalName)
• __meta_kubernetes_service_loadbalancer_ip: The IP address of the loadbalancer. (Applies to services of type LoadBalancer)
• __meta_kubernetes_service_external_name: The DNS name of the service. (Applies to services of type ExternalName)
• __meta_kubernetes_service_label_<labelname>: Each label from the service object.
• __meta_kubernetes_service_labelpresent_<labelname>: true for each label of the service object.
• __meta_kubernetes_service_name: The name of the service object.
• __meta_kubernetes_service_port_name: Name of the service port for the target.
• __meta_kubernetes_service_port_number: Number of the service port for the target.
• __meta_kubernetes_service_port_protocol: Protocol of the service port for the target.
• __meta_kubernetes_service_type: The type of the service.

---

 

relabeling 적용

그래서, nodename이라는 새로운 공통 라벨을 만들기로 결정하고,

모든 Servicemonitor에 relabel 옵션으로 예를 들면 아래와 같이 적용

prometheus-node-exporter:
  prometheus:
    monitor:
      enabled: true
      relabelings:
      - sourceLabels: [__meta_kubernetes_pod_node_name]
        separator: ;
        regex: ^(.*)$
        targetLabel: nodename
        replacement: $1
        action: replace

 

node exporter에 node role 적용 테스트

Servicemonitor로 메트릭을 수집하게 되면 endpoints role을 사용하게 되는데, 이를 node role로 사용해보고 싶어서 테스트 해봤다.

kube-prometheus-stack helm chart를 이용하여 적용해봤고, 방법은

values.yaml에서 prometheus.prometheusSpec에서 아래와 같이 additionalScrapeConfigs를 추가

prometheus:
	prometheusSpec:
    additionalScrapeConfigs: 
    - job_name: node-exporter
      relabel_configs:
      - source_labels: [__address__]
        action: replace
        regex: ([^:]+):.*
        replacement: $1:9100
        target_label: __address__
      - source_labels: [__meta_kubernetes_node_name]
        target_label: name
      - source_labels: [__meta_kubernetes_node_label_beta_kubernetes_io_arch]
        target_label: arch
      - source_labels: [__meta_kubernetes_node_label_beta_kubernetes_io_instance_type]
        target_label: instance_type
      - source_labels: [__meta_kubernetes_node_label_kubernetes_io_os]
        target_label: os
      - source_labels: [__meta_kubernetes_node_label_topology_kubernetes_io_region]
        target_label: region
      - source_labels: [__meta_kubernetes_node_label_topology_kubernetes_io_zone]
        target_label: zone
      - source_labels: [__meta_kubernetes_node_label_dedicated] # or any other custom label
        target_label: dedicated
      kubernetes_sd_configs:
      - role: node

적용하여 배포하면, prometheus pod에

/etc/prometheus/config_out/prometheus.env.yaml 경로에서 config를 확인해 볼 수 있다.

 

그리고 메트릭을 조회해보면, 아래와 같이 node 정보가 포함된 메트릭을 확인할 수 있다.