prometheus 监控

2026-06-24

Prometheus 安装/监控/告警/图表展示
- Promethues 监控服务安装
- Promethues 自动化监控
- Grafana 仪表盘
- Alertmanager 告警

1.0 各监控服务的安装

基于Linux 的 Promethues 各类服务安装搭建

IP	OS	HostName	备注
192.168.10.118	Centos7.9	ops-monitor	Prometheus相关服务
192.168.10.131	Centos7.9	ops-master01	绑定域名k8s-vip.ink8s.com
192.168.10.132	Centos7.9	ops-node01	-
192.168.10.133	Centos7.9	ops-node02	-
192.168.10.134	Centos7.9	ops-node03	-

1.1 Prometheus Server 安装

Prometheus Server 二进制下载和安装
启动项配置

# wget https://github.com/prometheus/prometheus/releases/download/v3.5.4/prometheus-3.5.4.linux-amd64.tar.gz
# mkdir -p /data/apps/monitor/prometheus/data/
# tar -xf prometheus-3.5.4.linux-amd64.tar.gz
# mv prometheus-3.5.4.linux-amd64/* /data/apps/monitor/prometheus/
# /data/apps/monitor/prometheus/prometheus -h							# 查看帮助

# cat /usr/lib/systemd/system/prometheus.service 
[Unit]
Description=prometheus

[Service]
ExecStart=/data/apps/monitor/prometheus/prometheus \
          --config.file=/data/apps/monitor/prometheus/prometheus.yml \
          --storage.tsdb.path=/data/apps/monitor/prometheus/data/ \
          --storage.tsdb.retention.time=180d
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure

[Install]
WantedBy=multi-user.target

# systemctl daemon-reload
# systemctl enable prometheus.service --now
# /data/apps/monitor/prometheus/promtool check config /data/apps/monitor/prometheus/prometheus.yml	# 主配置文件检查

// 访问: http://IP:9090

1.2 Alertermanager 安装

Alertermanager 二进制下载和安装
启动项配置

wget https://github.com/prometheus/alertmanager/releases/download/v0.33.0/alertmanager-0.33.0.linux-amd64.tar.gz
# tar -xf alertmanager-0.33.0.linux-amd64.tar.gz
# mv alertmanager-0.33.0.linux-amd64 /data/apps/monitor/alertmanager

# cat /usr/lib/systemd/system/alertermanager.service
[Unit]
Description=alertmanager System
Documentation=alertmanager System

[Service]
ExecStart= /data/apps/monitor/alertmanager/alertmanager --config.file=/data/apps/monitor/alertmanager/alertmanager.yml
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure

[Install]
WantedBy=multi-user.target

# systemctl daemon-reload 
# systemctl enable alertermanager.service --now
# /data/apps/monitor/alertmanager/amtool check-config /data/apps/monitor/alertmanager/alertmanager.yml		# 告警配置文件检查

// 访问: http://IP:9093/

1.3 Exporter 安装

1.3.1 Node_exporter 安装

1
2
3

# wget https://github.com/prometheus/node_exporter/releases/download/v1.11.1/node_exporter-1.11.1.linux-amd64.tar.gz
# tar -xf node_exporter-1.11.1.linux-amd64.tar.gz
# mv node_exporter-1.11.1.linux-amd64 /usr/local/node_exporter

# cat /usr/lib/systemd/system/node_exporter.service 
[Unit]
Description=node_exporter

[Service]
ExecStart=/usr/local/node_exporter/node_exporter
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure

[Install]
WantedBy=multi-user.target

# systemctl daemon-reload 
# systemctl enable node_exporter.service --now

// 访问: http://IP:9100/metrics

1.4 Granfana 安装

Granfana 二进制下载和安装
启动项配置

1
2
3

# wget https://dl.grafana.com/grafana/release/13.1.0/grafana_13.1.0_28013217238_linux_amd64.tar.gz
# tar -xf grafana_13.1.0_28013217238_linux_amd64.tar.gz
# mv grafana-13.1.0 /data/apps/grafana

# cat /usr/lib/systemd/system/grafana.service
[Unit]
Description=grafana

[Service]
ExecStart=/data/apps/grafana/bin/grafana server -homepath=/data/apps/grafana/
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure

[Install]
WantedBy=multi-user.target

# systemctl daemon-reload 
# systemctl enable grafana.service -now

//访问: http://IP:3000 {admin:admin}

2.0 Promethues 自动化监控

Prometheus 支持两种被监控方式:
- 静态配置: 手动配置
- 服务发现: 动态发现需要监控的 target 实例
  - file_sd_configs
  - consul_sd_configs
  - kubernetes_sd_configs

2.1 基于文件服务发现

file_sd_configs(基于文件服务发现)

# cat /data/apps/monitor/prometheus/prometheus.yml 
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  scrape_timeout: 10s

alerting:
  alertmanagers:
    - static_configs:
        - targets:

rule_files:

scrape_configs:
  - job_name: "file_sd"
    file_sd_configs:
    # 采用子配置文件方式
    - files: ['/data/apps/monitor/prometheus/file_sd_config/*.yml']

# mkdir -p /data/apps/monitor/prometheus/file_sd_config/
# /data/apps/monitor/prometheus/promtool check config /data/apps/monitor/prometheus/prometheus.yml

# cat /data/apps/monitor/prometheus/file_sd_config/node.yml 
- targets:
  - 192.168.10.118:9100
  # 后续直接往下添加节点会自动周期更新配置，无需重启

# kill -HUP 11433		# 11433 为prometheus PID进程。这种方式不用重启prometheus即可生效配置文件

upload successful

2.2 基于 consul 服务发现

consul_sd_configs（基于 consul 服务发现）

upload successful

2.2.1 快速部署 consul

1	# docker run --restart=always --name consul -d -p 8500:8500 harbor.inadm.com/ops_monitor/consul:1.15.4

2.2.2 consul 服务应用

通过在 consul 注册服务或注销服务(监控 targets)
Prometheus 一直监视(watch)consul 服务，当发现 consul 中符合要求的服务有新变化是更新 Prometheus 监控对象

// 使用 API 把这里的启动的 node_exporter 服务注册到 consul
# curl -X PUT -d '{"id": "ops-monitor01","name": "node-exporter","address": "192.168.10.118","port": 9100,"tags": ["service"],"checks": [{"http": "http://192.168.10.118:9100","interval": "5s"}]}' http://192.168.10.118:8500/v1/agent/service/register
    id:        实例(instance)
    name:      类似job,类似prometheus中的一个分组
    tags:      注册的标签
    checks:    健康检查,检查这个地址是否可以正常访问

// 注销服务
# curl --request PUT "http://192.168.10.118:8500/v1/agent/service/deregister/ops-monitor01"

upload successful

2.2.3 主配置文件修改

# cat /data/apps/monitor/prometheus/prometheus.yml 
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  scrape_timeout: 10s

alerting:
  alertmanagers:
    - static_configs:
        - targets:

rule_files:

scrape_configs:
  - job_name: "host_monitor"
    consul_sd_configs:
    - server: 192.168.10.118:8500
      services: ['node-exporter']

# /data/apps/monitor/prometheus/promtool check config /data/apps/monitor/prometheus/prometheus.yml
# kill -HUP 11433		# 11433 为prometheus PID进程。这种方式不用重启prometheus即可生效配置文件

upload successful

2.3 监控 kubernetes 集群

kubernetes_sd_configs (监控 kubernetes): 集群外 Prometheus + 全部指标经 API Server 代理抓取

监控指标	具体实现	举例
node 资源利用率	node-exporter	节点CPU，内存利用率
pod 资源利用率	cAdvisor	容器CPU，内存利用率
k8s 资源对象状态	kube-state-metrics	pod/deploy/svc

upload successful

服务发现类型	描述
service	发现集群中的节点，默认地址为kubelet的HTTP端口
pod	容器CPU，内存利用率
endpoints	从service列表中的endpoint发现pod为目标
ingress	发现ingress路径为目标

2.3.1 RBAC 授权

// kubectl api-resources		# 版本号确认
# cat prometheus_rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: kube-system
  labels:
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/component: monitor

---
#   ClusterRole: 授予服务发现与代理抓取所需权限
#    - nodes/nodes/proxy: 节点发现 + 通过 API Server 代理抓取 kubelet/cAdvisor 指标
#    - pods/services/endpoints/endpointslices/namespaces: 服务发现
#    - ingresses/configmaps/secrets: 辅助发现与配置读取
#    - /metrics, /metrics/cadvisor 非资源 URL: 代理访问
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
  labels:
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/component: monitor
rules:
  - apiGroups: [""]
    resources:
      - nodes
      - nodes/proxy          # API Server 代理抓取 kubelet/cAdvisor(/metrics, /metrics/cadvisor)
      - services
      - services/proxy       # API Server 服务代理: /api/v1/namespaces/<ns>/services/<svc>:<port>/proxy/
      - pods
      - pods/proxy           # API Server Pod 代理: /api/v1/namespaces/<ns>/pods/<pod>:<port>/proxy/
      - endpoints
      - endpointslices       # 1.21+ 内置, 也保留 discovery.k8s.io 组兼容旧版
      - namespaces
      - configmaps
      - secrets
    verbs: ["get", "list", "watch"]
  - apiGroups: ["networking.k8s.io"]
    resources: ["ingresses"]
    verbs: ["get", "list", "watch"]
  - nonResourceURLs: ["/metrics", "/metrics/cadvisor"]
    verbs: ["get"]

---
# 3) ClusterRoleBinding: 将 ClusterRole 绑定到 ServiceAccount
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
  labels:
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/component: monitor
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
  - kind: ServiceAccount
    name: prometheus
    namespace: kube-system

---
# Secret: 为 SA 生成长期有效的 token(K8S 1.24+ 不再自动生成 token Secret)
#    说明: 该 Secret 关联 SA 后, controller 会自动填入 token 与 ca.crt,
#          token 在 SA 存在期间长期有效(不自动过期), 适合集群外长期使用。
apiVersion: v1
kind: Secret
metadata:
  name: prometheus-token
  namespace: kube-system
  annotations:
    kubernetes.io/service-account.name: prometheus
  labels:
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/component: monitor
type: kubernetes.io/service-account-token

# kubectl apply -f prometheus_rbac.yaml

2.3.2 准备ca与token

// 在 Prometheus 监控主机上操作
# mkdir -p /data/apps/monitor/prometheus/{certs,tokens}

// 将k8s master 上的ca证书cp到prometheus server端
# scp [email protected]:/etc/kubernetes/pki/ca.crt /data/apps/monitor/prometheus/certs/ca.crt

// 获取token信息并保存到prometheus服务器上
# kubectl -n kube-system get secret prometheus-token -o jsonpath='{.data.token}' | base64 -d	# 在master节点执行
# vim /data/apps/monitor/prometheus/tokens/prometheus.token					# 将获取的到token写入到prometheus服务器上
# chmod 644 /data/apps/monitor/prometheus/certs/ca.crt
# chmod 600 /data/apps/monitor/prometheus/tokens/prometheus.token

2.3.3 监控k8s资源对象

部署 kube-state-metrics

# cat kube-state-metrics-deploy.yaml
# =============================================================================
# kube-state-metrics 部署清单
# 版本: v2.19.1 (latest, 支持 K8S 1.28)
# 用途: 暴露 K8S 资源对象状态指标
#       (Deployment 副本数、Pod 状态、Node 资源、PV 状态等)
# 部署: kubectl apply -f kube-state-metrics-deploy.yaml
# 验证: kubectl -n kube-system get pods -l app.kubernetes.io/name=kube-state-metrics
#       Prometheus Targets: kubernetes-service-endpoints 应出现 1/1 UP
# =============================================================================

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-state-metrics
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kube-state-metrics
rules:
  # 需要读取各类资源对象的元数据来生成指标
  - apiGroups: [""]
    resources:
      - nodes
      - pods
      - services
      - resourcequotas
      - replicationcontrollers
      - limitranges
      - persistentvolumes
      - persistentvolumeclaims
      - configmaps
      - secrets
      - endpoints
      - namespaces
      - serviceaccounts
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources:
      - daemonsets
      - deployments
      - replicasets
      - statefulsets
    verbs: ["get", "list", "watch"]
  - apiGroups: ["batch"]
    resources:
      - cronjobs
      - jobs
    verbs: ["get", "list", "watch"]
  - apiGroups: ["autoscaling"]
    resources:
      - horizontalpodautoscalers
    verbs: ["get", "list", "watch"]
  - apiGroups: ["policy"]
    resources:
      - poddisruptionbudgets
    verbs: ["get", "list", "watch"]
  - apiGroups: ["certificates.k8s.io"]
    resources:
      - certificatesigningrequests
    verbs: ["get", "list", "watch"]
  - apiGroups: ["storage.k8s.io"]
    resources:
      - storageclasses
      - volumeattachments
    verbs: ["get", "list", "watch"]
  - apiGroups: ["networking.k8s.io"]
    resources:
      - networkpolicies
      - ingresses
    verbs: ["get", "list", "watch"]
  - apiGroups: ["coordination.k8s.io"]
    resources:
      - leases
    verbs: ["get", "list", "watch"]
  - apiGroups: ["discovery.k8s.io"]
    resources:
      - endpointslices
    verbs: ["get", "list", "watch"]
  - apiGroups: ["scheduling.k8s.io"]
    resources:
      - priorityclasses
    verbs: ["get", "list", "watch"]
  - apiGroups: ["node.k8s.io"]
    resources:
      - runtimeclasses
    verbs: ["get", "list", "watch"]
  - apiGroups: ["admissionregistration.k8s.io"]
    resources:
      - mutatingwebhookconfigurations
      - validatingwebhookconfigurations
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kube-state-metrics
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kube-state-metrics
subjects:
  - kind: ServiceAccount
    name: kube-state-metrics
    namespace: kube-system
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-state-metrics
  namespace: kube-system
  labels:
    app.kubernetes.io/name: kube-state-metrics
    app.kubernetes.io/version: "v2.19.1"
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: kube-state-metrics
  template:
    metadata:
      labels:
        app.kubernetes.io/name: kube-state-metrics
        app.kubernetes.io/version: "v2.19.1"
    spec:
      serviceAccountName: kube-state-metrics
      containers:
        - name: kube-state-metrics
          image: registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.19.1
          imagePullPolicy: IfNotPresent
          args:
            - '--port=8080'          # metrics 端口
            - '--telemetry-port=8081'  # 自身 metrics 端口
          ports:
            - name: metrics
              containerPort: 8080
              protocol: TCP
            - name: telemetry
              containerPort: 8081
              protocol: TCP
          livenessProbe:
            httpGet:
              path: /livez
              port: 8080
            initialDelaySeconds: 5
            timeoutSeconds: 5
          readinessProbe:
            httpGet:
              path: /readyz
              port: 8081
            initialDelaySeconds: 5
            timeoutSeconds: 5
            failureThreshold: 5
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: 300m
              memory: 256Mi
      tolerations:
        - operator: Exists
          effect: NoSchedule
      nodeSelector:
        kubernetes.io/os: linux
---
apiVersion: v1
kind: Service
metadata:
  name: kube-state-metrics
  namespace: kube-system
  labels:
    app.kubernetes.io/name: kube-state-metrics
  # ★ 关键: 加这三个注解, Prometheus 的 service-endpoints job 会自动发现并抓取
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8080"
    prometheus.io/path: "/metrics"
spec:
  selector:
    app.kubernetes.io/name: kube-state-metrics
  ports:
    - name: metrics
      port: 8080
      targetPort: 8080
      protocol: TCP
    - name: telemetry
      port: 8081
      targetPort: 8081
      protocol: TCP
  type: ClusterIP


# kubectl apply -f kube-state-metrics-deploy.yaml

网络打通

1 2	// prometheus 和 k8s 的集群 Pod 不是同一网段，则需要在 promethues 监控主机上添加路由 # ip route add 10.1.0.0/16 via 192.168.10.131

2.3.4 主配置文件配置

# cat /data/apps/monitor/prometheus/prometheus.yml
# =============================================================================
# Prometheus 3.5.4 主配置 - 集群外监控 K8S 1.28.0 集群
# 部署主机: 192.168.10.118, 安装目录: /data/apps/monitor/prometheus
# 发现方式: kubernetes_sd_configs (通过 API Server https://k8s-vip.inadm.com:6443)
# 抓取策略: 所有指标统一通过 API Server 代理抓取
#           - nodes/cAdvisor:   /api/v1/nodes/<n>/proxy/metrics
#           - service-endpoints: /api/v1/namespaces/<ns>/services/<s>:<port>/proxy/
#           - pods:             /api/v1/namespaces/<ns>/pods/<p>:<port>/proxy/
#           避免 Prometheus 直连集群内 ClusterIP/PodIP, 仅需打通到 API Server 的 6443
# 前置文件:
#   /data/apps/monitor/prometheus/certs/ca.crt            (集群 CA 证书)
#   /data/apps/monitor/prometheus/tokens/prometheus.token (SA token)
# =============================================================================

global:
  scrape_interval: 30s          # 抓取间隔
  scrape_timeout: 15s           # 单次抓取超时(代理路径略慢, 适当放宽)
  evaluation_interval: 30s      # 规则评估间隔
  external_labels:
    cluster: 'k8s-prod'         # 多集群场景下区分来源

# 告警规则文件(按需启用)
rule_files:
  # - /data/apps/monitor/prometheus/rules/*.yml

scrape_configs:

  # ---------------------------------------------------------------------------
  # 1) 监控 K8S API Server 自身指标
  #    利用 endpoints 角色, 筛选 default 命名空间下 kubernetes 服务的 https 端口
  # ---------------------------------------------------------------------------
  - job_name: 'kubernetes-apiservers'
    kubernetes_sd_configs:
      - role: endpoints
        api_server: https://k8s-vip.inadm.com:6443
        tls_config:
          ca_file: /data/apps/monitor/prometheus/certs/ca.crt
        bearer_token_file: /data/apps/monitor/prometheus/tokens/prometheus.token
    scheme: https
    tls_config:
      ca_file: /data/apps/monitor/prometheus/certs/ca.crt
    bearer_token_file: /data/apps/monitor/prometheus/tokens/prometheus.token
    relabel_configs:
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
        action: keep
        regex: default;kubernetes;https
      - target_label: component
        replacement: apiserver
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace

  # ---------------------------------------------------------------------------
  # 2) 监控 Kubelet 自身指标(/metrics)
  #    通过 API Server 代理访问每个节点的 kubelet
  # ---------------------------------------------------------------------------
  - job_name: 'kubernetes-nodes'
    kubernetes_sd_configs:
      - role: node
        api_server: https://k8s-vip.inadm.com:6443
        tls_config:
          ca_file: /data/apps/monitor/prometheus/certs/ca.crt
        bearer_token_file: /data/apps/monitor/prometheus/tokens/prometheus.token
    scheme: https
    tls_config:
      ca_file: /data/apps/monitor/prometheus/certs/ca.crt
    bearer_token_file: /data/apps/monitor/prometheus/tokens/prometheus.token
    relabel_configs:
      # 把节点上的标签映射成 Prometheus 标签
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      # 抓取地址统一指向 API Server
      - target_label: __address__
        replacement: k8s-vip.inadm.com:6443
      # 抓取路径改写为 API Server 代理路径: /api/v1/nodes/<节点名>/proxy/metrics
      - source_labels: [__meta_kubernetes_node_name]
        target_label: __metrics_path__
        regex: (.+)
        replacement: /api/v1/nodes/${1}/proxy/metrics
      # 保留节点名便于查询
      - source_labels: [__meta_kubernetes_node_name]
        target_label: node
      - source_labels: [__meta_kubernetes_node_address_InternalIP]
        target_label: internal_ip

  # ---------------------------------------------------------------------------
  # 3) 监控 cAdvisor 容器指标(/metrics/cadvisor)
  #    同样通过 API Server 代理, 提供每个容器的 CPU/内存/网络/文件系统指标
  # ---------------------------------------------------------------------------
  - job_name: 'kubernetes-cadvisor'
    kubernetes_sd_configs:
      - role: node
        api_server: https://k8s-vip.inadm.com:6443
        tls_config:
          ca_file: /data/apps/monitor/prometheus/certs/ca.crt
        bearer_token_file: /data/apps/monitor/prometheus/tokens/prometheus.token
    scheme: https
    tls_config:
      ca_file: /data/apps/monitor/prometheus/certs/ca.crt
    bearer_token_file: /data/apps/monitor/prometheus/tokens/prometheus.token
    relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: k8s-vip.inadm.com:6443
      - source_labels: [__meta_kubernetes_node_name]
        target_label: __metrics_path__
        regex: (.+)
        replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
      - source_labels: [__meta_kubernetes_node_name]
        target_label: node
      - source_labels: [__meta_kubernetes_node_address_InternalIP]
        target_label: internal_ip
    # 丢弃 cAdvisor 中已知无用/高基数的指标, 降低存储压力
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: 'container_(network_tcp_usage_total|network_udp_usage_total|tasks_state|cpu_load_average_10s)'
        action: drop

  # ---------------------------------------------------------------------------
  # 4) 监控带注解的 Service Endpoints (通过 API Server 服务代理抓取)
  #    业务 Service 打上注解即可被自动发现并经 API Server 代理抓取:
  #      prometheus.io/scrape: "true"
  #      prometheus.io/port:   "9100"    ← 端口号(数字)
  #      prometheus.io/path:   "/metrics"
  #      prometheus.io/scheme: "http"       (注解中的 scheme 不影响实际连接,
  #                                         因为实际请求都走 API Server HTTPS 代理)
  #    代理路径格式:
  #      /api/v1/namespaces/<ns>/services/<svc>:<port>/proxy/<path>
  # ---------------------------------------------------------------------------
  - job_name: 'kubernetes-service-endpoints'
    kubernetes_sd_configs:
      - role: endpoints
        api_server: https://k8s-vip.inadm.com:6443
        tls_config:
          ca_file: /data/apps/monitor/prometheus/certs/ca.crt
        bearer_token_file: /data/apps/monitor/prometheus/tokens/prometheus.token
    # 全部走 API Server HTTPS 代理
    scheme: https
    tls_config:
      ca_file: /data/apps/monitor/prometheus/certs/ca.crt
    bearer_token_file: /data/apps/monitor/prometheus/tokens/prometheus.token
    relabel_configs:
      # 仅抓取声明了 prometheus.io/scrape=true 的 service
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      # ★ 端口处理: 优先用注解端口, 否则用 endpoint 的端口号
      # 第 1 步: 如果注解指定了端口, 直接写入 __tmp_port
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_port]
        regex: '(.+)'
        target_label: __tmp_port
        action: replace
      # 第 2 步: 如果注解端口为空, 则从 __address__ (格式 host:port) 提取端口号
      # Prometheus relabel 在 source_labels 匹配失败时不会执行 replacement,
      # 因此当 __tmp_port 已有值时此步自动跳过(正则不匹配)
      - source_labels: [__tmp_port, __address__]
        regex: '^$;([^:]+):([0-9]+)'
        replacement: '${2}'
        target_label: __tmp_port
        action: replace
      # ★ 路径处理: 优先用注解路径, 否则默认 /metrics
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
        regex: '(.+)'
        target_label: __metrics_path__
        action: replace
      # ★ 核心: 将 __metrics_path__ 改写为 API Server 服务代理路径
      # 代理路径 = /api/v1/namespaces/<ns>/services/<svc>:<port>/proxy<original_path>
      - source_labels: [__meta_kubernetes_namespace]
        regex: '(.+)'
        target_label: __tmp_ns
        action: replace
      - source_labels: [__meta_kubernetes_service_name]
        regex: '(.+)'
        target_label: __tmp_svc
        action: replace
      # 拼装最终代理路径
      - source_labels: [__tmp_ns, __tmp_svc, __tmp_port, __metrics_path__]
        regex: '(.+);(.+);(.+);(.+)'
        replacement: '/api/v1/namespaces/${1}/services/${2}:${3}/proxy${4}'
        target_label: __metrics_path__
        action: replace
      # 地址统一改写为 API Server
      - target_label: __address__
        replacement: k8s-vip.inadm.com:6443
      # 清理临时标签
      - regex: '__tmp_(ns|svc|port)'
        action: labeldrop
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_service_name]
        target_label: service

  # ---------------------------------------------------------------------------
  # 5) 监控带注解的 Pod (通过 API Server Pod 代理抓取)
  #    业务 Pod 打上注解即可被自动发现并经 API Server 代理抓取:
  #      prometheus.io/scrape: "true"
  #      prometheus.io/port:   "8080"
  #      prometheus.io/path:   "/metrics"
  #    代理路径格式:
  #      /api/v1/namespaces/<ns>/pods/<pod>:<port_num>/proxy/<path>
  # ---------------------------------------------------------------------------
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
        api_server: https://k8s-vip.inadm.com:6443
        tls_config:
          ca_file: /data/apps/monitor/prometheus/certs/ca.crt
        bearer_token_file: /data/apps/monitor/prometheus/tokens/prometheus.token
    # 全部走 API Server HTTPS 代理
    scheme: https
    tls_config:
      ca_file: /data/apps/monitor/prometheus/certs/ca.crt
    bearer_token_file: /data/apps/monitor/prometheus/tokens/prometheus.token
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      # ★ 端口处理: 优先用注解端口, 否则用 pod 的端口号
      # 第 1 步: 如果注解指定了端口, 直接写入 __tmp_port
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
        regex: '(.+)'
        target_label: __tmp_port
        action: replace
      # 第 2 步: 如果注解端口为空, 则从 __address__ (格式 host:port) 提取端口号
      - source_labels: [__tmp_port, __address__]
        regex: '^$;([^:]+):([0-9]+)'
        replacement: '${2}'
        target_label: __tmp_port
        action: replace
      # ★ 路径处理: 优先用注解路径, 否则默认 /metrics
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        regex: '(.+)'
        target_label: __metrics_path__
        action: replace
      # ★ 核心: 将 __metrics_path__ 改写为 API Server Pod 代理路径
      - source_labels: [__meta_kubernetes_namespace]
        regex: '(.+)'
        target_label: __tmp_ns
        action: replace
      - source_labels: [__meta_kubernetes_pod_name]
        regex: '(.+)'
        target_label: __tmp_pod
        action: replace
      - source_labels: [__tmp_ns, __tmp_pod, __tmp_port, __metrics_path__]
        regex: '(.+);(.+);(.+);(.+)'
        replacement: '/api/v1/namespaces/${1}/pods/${2}:${3}/proxy${4}'
        target_label: __metrics_path__
        action: replace
      # 地址统一改写为 API Server
      - target_label: __address__
        replacement: k8s-vip.inadm.com:6443
      # 清理临时标签
      - regex: '__tmp_(ns|pod|port)'
        action: labeldrop
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod
      - source_labels: [__meta_kubernetes_pod_node_name]
        target_label: node

  # ---------------------------------------------------------------------------
  # 6) (可选) node-exporter 抓取 —— 前提: 集群已用 DaemonSet 部署 node-exporter
  #    通过 API Server pod 代理抓取, 避免直连 Pod IP(ClusterIP 段不可达)
  #    若未部署可先注释掉, 不影响其它 job
  # ---------------------------------------------------------------------------
  - job_name: 'kubernetes-node-exporter'
    kubernetes_sd_configs:
      - role: pod
        api_server: https://k8s-vip.inadm.com:6443
        tls_config:
          ca_file: /data/apps/monitor/prometheus/certs/ca.crt
        bearer_token_file: /data/apps/monitor/prometheus/tokens/prometheus.token
    scheme: https
    tls_config:
      ca_file: /data/apps/monitor/prometheus/certs/ca.crt
    bearer_token_file: /data/apps/monitor/prometheus/tokens/prometheus.token
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        action: keep
        regex: node-exporter
      # 改写为 API Server pod 代理路径
      - source_labels: [
          __meta_kubernetes_namespace,
          __meta_kubernetes_pod_name
        ]
        target_label: __metrics_path__
        regex: (.+);(.+)
        replacement: /api/v1/namespaces/$1/pods/$2:9100/proxy/metrics
      - target_label: __address__
        replacement: k8s-vip.inadm.com:6443
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_node_name]
        target_label: node
      - source_labels: [__meta_kubernetes_pod_host_ip]
        target_label: host_ip


# /data/apps/monitor/prometheus/promtool check config /data/apps/monitor/prometheus/prometheus.yml
# kill -HUP 11433

upload successful

3.0 Grafana 仪表盘

3.1 Grafana 基础配置

3.1.1 数据源配置

连接 –> 数据源 –> Connection: (http://192.168.10.118:9090)

upload successful

3.1.2 导入仪表板

左侧点击”仪表板” –> 右上角”新建” –> 选择”Import dashbaord”

upload successful

3.1.3 仪表板ID清单

监控对象	仪表盘名称	Grafana_ID	说明
node-exporter	Node Exporter Full	1860	最全面的主机监控，CPU/内存/磁盘/网络
cAdvisor	Kubernetes Pods Monitoring	14282	容器/Pod 资源监控
cAdvisor	Kubernetes Cluster Monitoring (cAdvisor)	3119	集群级容器监控
kube-state-metrics	Kubernetes Cluster (Prometheus)	15757	资源对象状态，Deployment/Pod/Node 等
kube-state-metrics	Kubernetes Stateful Metrics	13105	另一个 kube-state-metrics 视图
API Server	Kubernetes API Server	17688	API Server 性能指标
K8S 总览	Kubernetes All-in-One	15661	综合视图，包含多个组件

3.2 图形面板

Grafana 中所有的面板均以插件形式使用，Grafana 提供了各种可视化来支持不同的用例:
- Time series（时间序列）默认的也是主要的图形可视化面板
- State timeline（状态时间表）状态随时间变化
- Status history（状态历史记录）
- Bar chart（条形图）
- Histogram（直方图）
- Heatmap（热力图）
- Pie chart （饼状图）
- Stat（统计数据）
- Gauge、Bar gauge、Table（表格）
- Logs（日志）
- Node Graph（节点图）
- Dashboard list（仪表板列表）
- Alert list（报警列表）
- Text panel（文本面板、支持markdown和html）
- News Panel（新闻面板，可以显示RSS摘要）
其它面板信息获取

3.2.1 Time series 时间序列

Time series（时间序列）默认的也是主要的图形可视化面板

3.2.1.1 CPU使用率公式

准备好一个CPU使用率公式,用于下面创建时间序列图时使用

1 2	// CPU 使用率 (1 - sum(rate(node_cpu_seconds_total{mode="idle"}[5m])) by (instance) / sum(rate(node_cpu_seconds_total[5m])) by (instance) ) * 100

3.2.1.2 创建时间序列自定义图

点击左侧”仪表板” –> 点击右上角”新建” –> 选择”新建仪表板” –> 点击”+” –> 选择”配置可视化” –> 点击右侧”所有可视化” –> 单击”Time series”: 如图配置最后保存

upload successful

3.2.1.3 配置自定义图设置

可针对这个自定义的仪表盘进行配置: 单击右侧齿轮”数据面板选项” –> 单击 “View all settings”
可以配置名称、标签、变量等

upload successful

3.2.1.4 过滤节点IP地址

upload successful

3.2.1.5 标签替换

针对单个图表将 “语句中的标签进行变量替换”，修改完成后保存

1
2

// CPU使用率 添加 instance=~"$HOST_IP" 变量
(1 - sum(rate(node_cpu_seconds_total{instance=~"$HOST_IP" ,mode="idle"}[5m])) by (instance) / sum(rate(node_cpu_seconds_total{instance=~"$HOST_IP"}[5m])) by (instance) ) * 100

upload successful

4.0 Alertmanager

本文作者： [email protected]
本文链接： https://www.ink8s.com/2026/06/24/prometheus-监控/
版权声明： 本博客所有文章除特别声明外，均采用 MIT 许可协议。转载请注明出处！