方案

1. app in k8s
   • k8s discover # 推荐
   • serviceAccount
   • 注册中心
2. app not in k8s
   • serviceAccount
   • 注册中心

文档地址

https://github.com/prometheus/prometheus/blob/main/documentation/examples/prometheus-kubernetes.yml

以service为维度监控endpoients（pod)

secret additional-scrape-configs 增加如下配置

- job_name: "kubernetes-service-endpoints"

  kubernetes_sd_configs:
    - role: endpoints
  relabel_configs:
     #  Example relabel to scrape only endpoints that have
     #  "example.io/should_be_scraped = true" annotation.
      - source_labels: [__meta_kubernetes_service_annotation_example_io_should_be_scraped]
        action: keep
        regex: true

     #  Example relabel to customize metric path based on endpoints
     #  "example.io/metric_path = <metric path>" annotation.
      - source_labels: [__meta_kubernetes_service_annotation_example_io_metric_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)

     #  Example relabel to scrape only single, desired port for the service based
     #  on endpoints "example.io/scrape_port = <port>" annotation.
      - source_labels: [__address__, __meta_kubernetes_service_annotation_example_io_scrape_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__

    #  Example relabel to configure scrape scheme for all service scrape targets
    #  based on endpoints "example.io/scrape_scheme = <scheme>" annotation.
      - source_labels: [__meta_kubernetes_service_annotation_example_io_scrape_scheme]
        action: replace
        target_label: __scheme__
        regex: (https?)
    - action: labelmap
      regex: __meta_kubernetes_service_label_(.+)
    - source_labels: [__meta_kubernetes_namespace]
      action: replace
      target_label: kubernetes_namespace
    - source_labels: [__meta_kubernetes_service_name]

service注解

metadata:
  annotations:
    prometheus.io/path: /actuator/prometheus
    prometheus.io/scheme: http
    prometheus.io/scrape: "true"
    prometheus.io/port: "12000"  # 如pod中应用端口与监控端口不一致，需填写应用端口

效果图

serviceAccout（k8s应用）

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: rabbitmq
  namespace: monitoring
spec:
  endpoints:
  - port: rabbitmq15672
    interval: 30s
    path: /api/metrics
  namespaceSelector:
    matchNames:
    - middleware
  selector:
    matchLabels:
      app: rabbitmq-common-qa

serviceAccout（非k8s应用）

apiVersion: v1
kind: Service
metadata:
  name: rabbitmq
  namespace: monitoring
  labels:
    app: rabbitmq
spec:
  type: ClusterIP
  clusterIP: None
  ports:
  - name: http
    port: 15672
    protocol: TCP

---

apiVersion: v1
kind: Endpoints
metadata:
  name: rabbitmq
  namespace: monitoring
  labels:
    app: rabbitmq
subsets:
- addresses:
  - ip: 192.168.10.98
  - ip: 192.168.10.99
  ports:
  - name: http
    port: 15672
    protocol: TCP

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app: rabbitmq
  name: rabbitmq
  namespace: monitoring
spec:
  endpoints:
  - port: http
    interval: 90s
    path: /api/metrics
  selector:
    matchLabels:
      app: rabbitmq

从注册中心获取监控目标（eureka）

- job_name: 'eureka'
  metrics_path: '/actuator/prometheus'
  eureka_sd_configs:
    - server: http://8999.pr.xx.corp:8999/eureka

  relabel_configs:
    - source_labels: [__address__]
      regex: "192.[0-9]+.[0-9]+.[0-9]+:[0-9]+"
      action: keep
    - source_labels: [__meta_eureka_app_instance_port]
      target_label:  "appid"
    - source_labels: [__address__]
      target_label:  "instance"

从注册中心获取监控目标（consul）

  - job_name: 'consul'
    metrics_path: '/actuator/prometheus'
    scrape_interval: 5s
    consul_sd_configs:
    - server : 'consul.xxx.com:8500'
      services: ['springboot']

    relabel_configs:
    - source_labels: [__meta_consul_service_id]
      target_label:  "application"
    - source_labels: [__meta_consul_metadata_hostname]
      target_label:  "hostname"