prometheus 监控

2026-06-24

Prometheus 安装/监控/告警/图表
- Promethues 监控服务安装
- Promethues 自动化监控
- Alertmanager 告警
- Grafana 仪表盘

1.0 各监控服务的安装

基于Linux 的 Promethues 各类服务安装搭建

IP	OS	HostName	备注
192.168.10.118	Centos7.9	ops-monitor	Prometheus相关服务
192.168.10.131	Centos7.9	ops-master01	绑定域名k8s-vip.ink8s.com
192.168.10.132	Centos7.9	ops-node01	-
192.168.10.133	Centos7.9	ops-node02	-
192.168.10.134	Centos7.9	ops-node03	-

1.1 Prometheus Server 安装

Prometheus Server 二进制下载和安装
启动项配置

# wget https://github.com/prometheus/prometheus/releases/download/v3.5.4/prometheus-3.5.4.linux-amd64.tar.gz
# mkdir -p /data/apps/monitor/prometheus/data/
# tar -xf prometheus-3.5.4.linux-amd64.tar.gz
# mv prometheus-3.5.4.linux-amd64/* /data/apps/monitor/prometheus/
# /data/apps/monitor/prometheus/prometheus -h							# 查看帮助

# cat /usr/lib/systemd/system/prometheus.service 
[Unit]
Description=prometheus

[Service]
ExecStart=/data/apps/monitor/prometheus/prometheus \
          --config.file=/data/apps/monitor/prometheus/prometheus.yml \
          --storage.tsdb.path=/data/apps/monitor/prometheus/data/ \
          --storage.tsdb.retention.time=180d \
          --web.enable-lifecycle \
          --web.listen-address=0.0.0.0:9090
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure

[Install]
WantedBy=multi-user.target

# systemctl daemon-reload
# systemctl enable prometheus.service --now
# /data/apps/monitor/prometheus/promtool check config /data/apps/monitor/prometheus/prometheus.yml	# 主配置文件检查

// 访问: http://IP:9090

1.2 Alertermanager 安装

Alertermanager 二进制下载和安装
启动项配置

wget https://github.com/prometheus/alertmanager/releases/download/v0.33.0/alertmanager-0.33.0.linux-amd64.tar.gz
# tar -xf alertmanager-0.33.0.linux-amd64.tar.gz
# mv alertmanager-0.33.0.linux-amd64 /data/apps/monitor/alertmanager

# cat /usr/lib/systemd/system/alertermanager.service
[Unit]
Description=Alertmanager
After=network.target

[Service]
Type=simple
User=root
Group=root
WorkingDirectory=/data/apps/monitor/alertmanager
ExecStart=/data/apps/monitor/alertmanager/alertmanager \
          --config.file=/data/apps/monitor/alertmanager/alertmanager.yml \
          --storage.path=/data/apps/monitor/alertmanager/data \
          --web.listen-address=0.0.0.0:9093 \
          --web.external-url=http://192.168.10.118:9093 
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

# systemctl daemon-reload 
# systemctl enable alertermanager.service --now
# /data/apps/monitor/alertmanager/amtool check-config /data/apps/monitor/alertmanager/alertmanager.yml		# 告警配置文件检查

// 访问: http://IP:9093/

1.3 Exporter 安装

1.3.1 Node_exporter 安装

1
2
3

# wget https://github.com/prometheus/node_exporter/releases/download/v1.11.1/node_exporter-1.11.1.linux-amd64.tar.gz
# tar -xf node_exporter-1.11.1.linux-amd64.tar.gz
# mv node_exporter-1.11.1.linux-amd64 /usr/local/node_exporter

# cat /usr/lib/systemd/system/node_exporter.service 
[Unit]
Description=node_exporter

[Service]
ExecStart=/usr/local/node_exporter/node_exporter --collector.systemd
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure

[Install]
WantedBy=multi-user.target

# systemctl daemon-reload 
# systemctl enable node_exporter.service --now

// 访问: http://IP:9100/metrics

1.4 Granfana 安装

Granfana 二进制下载和安装
启动项配置

1
2
3

# wget https://dl.grafana.com/grafana/release/13.1.0/grafana_13.1.0_28013217238_linux_amd64.tar.gz
# tar -xf grafana_13.1.0_28013217238_linux_amd64.tar.gz
# mv grafana-13.1.0 /data/apps/grafana

# cat /usr/lib/systemd/system/grafana.service
[Unit]
Description=grafana

[Service]
ExecStart=/data/apps/grafana/bin/grafana server -homepath=/data/apps/grafana/
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure

[Install]
WantedBy=multi-user.target

# systemctl daemon-reload 
# systemctl enable grafana.service -now

//访问: http://IP:3000 {admin:admin}

2.0 Promethues 自动化监控

Prometheus 支持两种被监控方式:
- 静态配置: 手动配置
- 服务发现: 动态发现需要监控的 target 实例
  - file_sd_configs
  - consul_sd_configs
  - kubernetes_sd_configs

2.1 基于文件服务发现

file_sd_configs(基于文件服务发现)

# cat /data/apps/monitor/prometheus/prometheus.yml 
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  scrape_timeout: 10s

alerting:
  alertmanagers:
    - static_configs:
        - targets:

rule_files:

scrape_configs:
  - job_name: "file_sd"
    file_sd_configs:
    # 采用子配置文件方式
    - files: ['/data/apps/monitor/prometheus/file_sd_config/*.yml']

# mkdir -p /data/apps/monitor/prometheus/file_sd_config/
# /data/apps/monitor/prometheus/promtool check config /data/apps/monitor/prometheus/prometheus.yml

# cat /data/apps/monitor/prometheus/file_sd_config/node.yml 
- targets:
  - 192.168.10.118:9100
  # 后续直接往下添加节点会自动周期更新配置，无需重启

# kill -HUP 11433		# 11433 为prometheus PID进程。这种方式不用重启prometheus即可生效配置文件

upload successful

2.2 基于 consul 服务发现

consul_sd_configs（基于 consul 服务发现）

upload successful

2.2.1 快速部署 consul

1	# docker run --restart=always --name consul -d -p 8500:8500 harbor.ink8s.com/ops_monitor/consul:1.15.4

2.2.2 consul 服务应用

通过在 consul 注册服务或注销服务(监控 targets)
Prometheus 一直监视(watch)consul 服务，当发现 consul 中符合要求的服务有新变化是更新 Prometheus 监控对象

// 使用 API 把这里的启动的 node_exporter 服务注册到 consul
# curl -X PUT -d '{"id": "ops-monitor01","name": "node-exporter","address": "192.168.10.118","port": 9100,"tags": ["service"],"checks": [{"http": "http://192.168.10.118:9100","interval": "5s"}]}' http://192.168.10.118:8500/v1/agent/service/register
    id:        实例(instance)
    name:      类似job,类似prometheus中的一个分组
    tags:      注册的标签
    checks:    健康检查,检查这个地址是否可以正常访问

// 注销服务
# curl --request PUT "http://192.168.10.118:8500/v1/agent/service/deregister/ops-monitor01"

upload successful

2.2.3 主配置文件修改

# cat /data/apps/monitor/prometheus/prometheus.yml 
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  scrape_timeout: 10s

alerting:
  alertmanagers:
    - static_configs:
        - targets:

rule_files:

scrape_configs:
  - job_name: "host_monitor"
    consul_sd_configs:
    - server: 192.168.10.118:8500
      services: ['node-exporter']

# /data/apps/monitor/prometheus/promtool check config /data/apps/monitor/prometheus/prometheus.yml
# kill -HUP 11433		# 11433 为prometheus PID进程。这种方式不用重启prometheus即可生效配置文件

upload successful

2.3 监控 kubernetes 集群

kubernetes_sd_configs (监控 kubernetes): 集群外 Prometheus + 全部指标经 API Server 代理抓取

监控指标	具体实现	举例
node 资源利用率	node-exporter	节点CPU，内存利用率
pod 资源利用率	cAdvisor	容器CPU，内存利用率
k8s 资源对象状态	kube-state-metrics	pod/deploy/svc

upload successful

服务发现类型	描述
service	发现集群中的节点，默认地址为kubelet的HTTP端口
pod	容器CPU，内存利用率
endpoints	从service列表中的endpoint发现pod为目标
ingress	发现ingress路径为目标

2.3.1 RBAC 授权

// kubectl api-resources		# 版本号确认
# cat prometheus_rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: kube-system
  labels:
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/component: monitor

---
#   ClusterRole: 授予服务发现与代理抓取所需权限
#    - nodes/nodes/proxy: 节点发现 + 通过 API Server 代理抓取 kubelet/cAdvisor 指标
#    - pods/services/endpoints/endpointslices/namespaces: 服务发现
#    - ingresses/configmaps/secrets: 辅助发现与配置读取
#    - /metrics, /metrics/cadvisor 非资源 URL: 代理访问
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
  labels:
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/component: monitor
rules:
  - apiGroups: [""]
    resources:
      - nodes
      - nodes/proxy          # API Server 代理抓取 kubelet/cAdvisor(/metrics, /metrics/cadvisor)
      - services
      - services/proxy       # API Server 服务代理: /api/v1/namespaces/<ns>/services/<svc>:<port>/proxy/
      - pods
      - pods/proxy           # API Server Pod 代理: /api/v1/namespaces/<ns>/pods/<pod>:<port>/proxy/
      - endpoints
      - endpointslices       # 1.21+ 内置, 也保留 discovery.k8s.io 组兼容旧版
      - namespaces
      - configmaps
      - secrets
    verbs: ["get", "list", "watch"]
  - apiGroups: ["networking.k8s.io"]
    resources: ["ingresses"]
    verbs: ["get", "list", "watch"]
  - nonResourceURLs: ["/metrics", "/metrics/cadvisor"]
    verbs: ["get"]

---
# 3) ClusterRoleBinding: 将 ClusterRole 绑定到 ServiceAccount
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
  labels:
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/component: monitor
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
  - kind: ServiceAccount
    name: prometheus
    namespace: kube-system

---
# Secret: 为 SA 生成长期有效的 token(K8S 1.24+ 不再自动生成 token Secret)
#    说明: 该 Secret 关联 SA 后, controller 会自动填入 token 与 ca.crt,
#          token 在 SA 存在期间长期有效(不自动过期), 适合集群外长期使用。
apiVersion: v1
kind: Secret
metadata:
  name: prometheus-token
  namespace: kube-system
  annotations:
    kubernetes.io/service-account.name: prometheus
  labels:
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/component: monitor
type: kubernetes.io/service-account-token

# kubectl apply -f prometheus_rbac.yaml

2.3.2 准备ca与token

// 在 Prometheus 监控主机上操作
# mkdir -p /data/apps/monitor/prometheus/{certs,tokens}

// 将k8s master 上的ca证书cp到prometheus server端
# scp [email protected]:/etc/kubernetes/pki/ca.crt /data/apps/monitor/prometheus/certs/ca.crt

// 获取token信息并保存到prometheus服务器上
# kubectl -n kube-system get secret prometheus-token -o jsonpath='{.data.token}' | base64 -d	# 在master节点执行
# vim /data/apps/monitor/prometheus/tokens/prometheus.token					# 将获取的到token写入到prometheus服务器上
# chmod 644 /data/apps/monitor/prometheus/certs/ca.crt
# chmod 600 /data/apps/monitor/prometheus/tokens/prometheus.token

2.3.3 监控k8s资源对象

部署 kube-state-metrics

# cat kube-state-metrics-deploy.yaml
# =============================================================================
# kube-state-metrics 部署清单
# 版本: v2.19.1 (latest, 支持 K8S 1.28)
# 用途: 暴露 K8S 资源对象状态指标
#       (Deployment 副本数、Pod 状态、Node 资源、PV 状态等)
# 部署: kubectl apply -f kube-state-metrics-deploy.yaml
# 验证: kubectl -n kube-system get pods -l app.kubernetes.io/name=kube-state-metrics
#       Prometheus Targets: kubernetes-service-endpoints 应出现 1/1 UP
# =============================================================================

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-state-metrics
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kube-state-metrics
rules:
  # 需要读取各类资源对象的元数据来生成指标
  - apiGroups: [""]
    resources:
      - nodes
      - pods
      - services
      - resourcequotas
      - replicationcontrollers
      - limitranges
      - persistentvolumes
      - persistentvolumeclaims
      - configmaps
      - secrets
      - endpoints
      - namespaces
      - serviceaccounts
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources:
      - daemonsets
      - deployments
      - replicasets
      - statefulsets
    verbs: ["get", "list", "watch"]
  - apiGroups: ["batch"]
    resources:
      - cronjobs
      - jobs
    verbs: ["get", "list", "watch"]
  - apiGroups: ["autoscaling"]
    resources:
      - horizontalpodautoscalers
    verbs: ["get", "list", "watch"]
  - apiGroups: ["policy"]
    resources:
      - poddisruptionbudgets
    verbs: ["get", "list", "watch"]
  - apiGroups: ["certificates.k8s.io"]
    resources:
      - certificatesigningrequests
    verbs: ["get", "list", "watch"]
  - apiGroups: ["storage.k8s.io"]
    resources:
      - storageclasses
      - volumeattachments
    verbs: ["get", "list", "watch"]
  - apiGroups: ["networking.k8s.io"]
    resources:
      - networkpolicies
      - ingresses
    verbs: ["get", "list", "watch"]
  - apiGroups: ["coordination.k8s.io"]
    resources:
      - leases
    verbs: ["get", "list", "watch"]
  - apiGroups: ["discovery.k8s.io"]
    resources:
      - endpointslices
    verbs: ["get", "list", "watch"]
  - apiGroups: ["scheduling.k8s.io"]
    resources:
      - priorityclasses
    verbs: ["get", "list", "watch"]
  - apiGroups: ["node.k8s.io"]
    resources:
      - runtimeclasses
    verbs: ["get", "list", "watch"]
  - apiGroups: ["admissionregistration.k8s.io"]
    resources:
      - mutatingwebhookconfigurations
      - validatingwebhookconfigurations
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kube-state-metrics
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kube-state-metrics
subjects:
  - kind: ServiceAccount
    name: kube-state-metrics
    namespace: kube-system
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-state-metrics
  namespace: kube-system
  labels:
    app.kubernetes.io/name: kube-state-metrics
    app.kubernetes.io/version: "v2.19.1"
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: kube-state-metrics
  template:
    metadata:
      labels:
        app.kubernetes.io/name: kube-state-metrics
        app.kubernetes.io/version: "v2.19.1"
    spec:
      serviceAccountName: kube-state-metrics
      containers:
        - name: kube-state-metrics
          image: registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.19.1
          imagePullPolicy: IfNotPresent
          args:
            - '--port=8080'          # metrics 端口
            - '--telemetry-port=8081'  # 自身 metrics 端口
          ports:
            - name: metrics
              containerPort: 8080
              protocol: TCP
            - name: telemetry
              containerPort: 8081
              protocol: TCP
          livenessProbe:
            httpGet:
              path: /livez
              port: 8080
            initialDelaySeconds: 5
            timeoutSeconds: 5
          readinessProbe:
            httpGet:
              path: /readyz
              port: 8081
            initialDelaySeconds: 5
            timeoutSeconds: 5
            failureThreshold: 5
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: 300m
              memory: 256Mi
      tolerations:
        - operator: Exists
          effect: NoSchedule
      nodeSelector:
        kubernetes.io/os: linux
---
apiVersion: v1
kind: Service
metadata:
  name: kube-state-metrics
  namespace: kube-system
  labels:
    app.kubernetes.io/name: kube-state-metrics
  # ★ 关键: 加这三个注解, Prometheus 的 service-endpoints job 会自动发现并抓取
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8080"
    prometheus.io/path: "/metrics"
spec:
  selector:
    app.kubernetes.io/name: kube-state-metrics
  ports:
    - name: metrics
      port: 8080
      targetPort: 8080
      protocol: TCP
    - name: telemetry
      port: 8081
      targetPort: 8081
      protocol: TCP
  type: ClusterIP


# kubectl apply -f kube-state-metrics-deploy.yaml

网络打通

1 2	// prometheus 和 k8s 的集群 Pod 不是同一网段，则需要在 promethues 监控主机上添加路由 # ip route add 10.1.0.0/16 via 192.168.10.131

2.3.4 主配置文件配置

# cat /data/apps/monitor/prometheus/prometheus.yml
# =============================================================================
# Prometheus 3.5.4 主配置 - 集群外监控 K8S 1.28.0 集群
# 部署主机: 192.168.10.118, 安装目录: /data/apps/monitor/prometheus
# 发现方式: kubernetes_sd_configs (通过 API Server https://k8s-vip.ink8s.com:6443)
# 抓取策略: 所有指标统一通过 API Server 代理抓取
#           - nodes/cAdvisor:   /api/v1/nodes/<n>/proxy/metrics
#           - service-endpoints: /api/v1/namespaces/<ns>/services/<s>:<port>/proxy/
#           - pods:             /api/v1/namespaces/<ns>/pods/<p>:<port>/proxy/
#           避免 Prometheus 直连集群内 ClusterIP/PodIP, 仅需打通到 API Server 的 6443
# 前置文件:
#   /data/apps/monitor/prometheus/certs/ca.crt            (集群 CA 证书)
#   /data/apps/monitor/prometheus/tokens/prometheus.token (SA token)
# =============================================================================

global:
  scrape_interval: 30s          # 抓取间隔
  scrape_timeout: 15s           # 单次抓取超时(代理路径略慢, 适当放宽)
  evaluation_interval: 30s      # 规则评估间隔
  external_labels:
    cluster: 'k8s-prod'         # 多集群场景下区分来源

# 告警规则文件(按需启用)
rule_files:
  # - /data/apps/monitor/prometheus/rules/*.yml

scrape_configs:

  # ---------------------------------------------------------------------------
  # 1) 监控 K8S API Server 自身指标
  #    利用 endpoints 角色, 筛选 default 命名空间下 kubernetes 服务的 https 端口
  # ---------------------------------------------------------------------------
  - job_name: 'kubernetes-apiservers'
    kubernetes_sd_configs:
      - role: endpoints
        api_server: https://k8s-vip.ink8s.com:6443
        tls_config:
          ca_file: /data/apps/monitor/prometheus/certs/ca.crt
        bearer_token_file: /data/apps/monitor/prometheus/tokens/prometheus.token
    scheme: https
    tls_config:
      ca_file: /data/apps/monitor/prometheus/certs/ca.crt
    bearer_token_file: /data/apps/monitor/prometheus/tokens/prometheus.token
    relabel_configs:
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
        action: keep
        regex: default;kubernetes;https
      - target_label: component
        replacement: apiserver
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace

  # ---------------------------------------------------------------------------
  # 2) 监控 Kubelet 自身指标(/metrics)
  #    通过 API Server 代理访问每个节点的 kubelet
  # ---------------------------------------------------------------------------
  - job_name: 'kubernetes-nodes'
    kubernetes_sd_configs:
      - role: node
        api_server: https://k8s-vip.ink8s.com:6443
        tls_config:
          ca_file: /data/apps/monitor/prometheus/certs/ca.crt
        bearer_token_file: /data/apps/monitor/prometheus/tokens/prometheus.token
    scheme: https
    tls_config:
      ca_file: /data/apps/monitor/prometheus/certs/ca.crt
    bearer_token_file: /data/apps/monitor/prometheus/tokens/prometheus.token
    relabel_configs:
      # 把节点上的标签映射成 Prometheus 标签
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      # 抓取地址统一指向 API Server
      - target_label: __address__
        replacement: k8s-vip.ink8s.com:6443
      # 抓取路径改写为 API Server 代理路径: /api/v1/nodes/<节点名>/proxy/metrics
      - source_labels: [__meta_kubernetes_node_name]
        target_label: __metrics_path__
        regex: (.+)
        replacement: /api/v1/nodes/${1}/proxy/metrics
      # 保留节点名便于查询
      - source_labels: [__meta_kubernetes_node_name]
        target_label: node
      - source_labels: [__meta_kubernetes_node_address_InternalIP]
        target_label: internal_ip

  # ---------------------------------------------------------------------------
  # 3) 监控 cAdvisor 容器指标(/metrics/cadvisor)
  #    同样通过 API Server 代理, 提供每个容器的 CPU/内存/网络/文件系统指标
  # ---------------------------------------------------------------------------
  - job_name: 'kubernetes-cadvisor'
    kubernetes_sd_configs:
      - role: node
        api_server: https://k8s-vip.ink8s.com:6443
        tls_config:
          ca_file: /data/apps/monitor/prometheus/certs/ca.crt
        bearer_token_file: /data/apps/monitor/prometheus/tokens/prometheus.token
    scheme: https
    tls_config:
      ca_file: /data/apps/monitor/prometheus/certs/ca.crt
    bearer_token_file: /data/apps/monitor/prometheus/tokens/prometheus.token
    relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: k8s-vip.ink8s.com:6443
      - source_labels: [__meta_kubernetes_node_name]
        target_label: __metrics_path__
        regex: (.+)
        replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
      - source_labels: [__meta_kubernetes_node_name]
        target_label: node
      - source_labels: [__meta_kubernetes_node_address_InternalIP]
        target_label: internal_ip
    # 丢弃 cAdvisor 中已知无用/高基数的指标, 降低存储压力
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: 'container_(network_tcp_usage_total|network_udp_usage_total|tasks_state|cpu_load_average_10s)'
        action: drop

  # ---------------------------------------------------------------------------
  # 4) 监控带注解的 Service Endpoints (通过 API Server 服务代理抓取)
  #    业务 Service 打上注解即可被自动发现并经 API Server 代理抓取:
  #      prometheus.io/scrape: "true"
  #      prometheus.io/port:   "9100"    ← 端口号(数字)
  #      prometheus.io/path:   "/metrics"
  #      prometheus.io/scheme: "http"       (注解中的 scheme 不影响实际连接,
  #                                         因为实际请求都走 API Server HTTPS 代理)
  #    代理路径格式:
  #      /api/v1/namespaces/<ns>/services/<svc>:<port>/proxy/<path>
  # ---------------------------------------------------------------------------
  - job_name: 'kubernetes-service-endpoints'
    kubernetes_sd_configs:
      - role: endpoints
        api_server: https://k8s-vip.ink8s.com:6443
        tls_config:
          ca_file: /data/apps/monitor/prometheus/certs/ca.crt
        bearer_token_file: /data/apps/monitor/prometheus/tokens/prometheus.token
    # 全部走 API Server HTTPS 代理
    scheme: https
    tls_config:
      ca_file: /data/apps/monitor/prometheus/certs/ca.crt
    bearer_token_file: /data/apps/monitor/prometheus/tokens/prometheus.token
    relabel_configs:
      # 仅抓取声明了 prometheus.io/scrape=true 的 service
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      # ★ 端口处理: 优先用注解端口, 否则用 endpoint 的端口号
      # 第 1 步: 如果注解指定了端口, 直接写入 __tmp_port
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_port]
        regex: '(.+)'
        target_label: __tmp_port
        action: replace
      # 第 2 步: 如果注解端口为空, 则从 __address__ (格式 host:port) 提取端口号
      # Prometheus relabel 在 source_labels 匹配失败时不会执行 replacement,
      # 因此当 __tmp_port 已有值时此步自动跳过(正则不匹配)
      - source_labels: [__tmp_port, __address__]
        regex: '^$;([^:]+):([0-9]+)'
        replacement: '${2}'
        target_label: __tmp_port
        action: replace
      # ★ 路径处理: 优先用注解路径, 否则默认 /metrics
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
        regex: '(.+)'
        target_label: __metrics_path__
        action: replace
      # ★ 核心: 将 __metrics_path__ 改写为 API Server 服务代理路径
      # 代理路径 = /api/v1/namespaces/<ns>/services/<svc>:<port>/proxy<original_path>
      - source_labels: [__meta_kubernetes_namespace]
        regex: '(.+)'
        target_label: __tmp_ns
        action: replace
      - source_labels: [__meta_kubernetes_service_name]
        regex: '(.+)'
        target_label: __tmp_svc
        action: replace
      # 拼装最终代理路径
      - source_labels: [__tmp_ns, __tmp_svc, __tmp_port, __metrics_path__]
        regex: '(.+);(.+);(.+);(.+)'
        replacement: '/api/v1/namespaces/${1}/services/${2}:${3}/proxy${4}'
        target_label: __metrics_path__
        action: replace
      # 地址统一改写为 API Server
      - target_label: __address__
        replacement: k8s-vip.ink8s.com:6443
      # 清理临时标签
      - regex: '__tmp_(ns|svc|port)'
        action: labeldrop
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_service_name]
        target_label: service

  # ---------------------------------------------------------------------------
  # 5) 监控带注解的 Pod (通过 API Server Pod 代理抓取)
  #    业务 Pod 打上注解即可被自动发现并经 API Server 代理抓取:
  #      prometheus.io/scrape: "true"
  #      prometheus.io/port:   "8080"
  #      prometheus.io/path:   "/metrics"
  #    代理路径格式:
  #      /api/v1/namespaces/<ns>/pods/<pod>:<port_num>/proxy/<path>
  # ---------------------------------------------------------------------------
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
        api_server: https://k8s-vip.ink8s.com:6443
        tls_config:
          ca_file: /data/apps/monitor/prometheus/certs/ca.crt
        bearer_token_file: /data/apps/monitor/prometheus/tokens/prometheus.token
    # 全部走 API Server HTTPS 代理
    scheme: https
    tls_config:
      ca_file: /data/apps/monitor/prometheus/certs/ca.crt
    bearer_token_file: /data/apps/monitor/prometheus/tokens/prometheus.token
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      # ★ 端口处理: 优先用注解端口, 否则用 pod 的端口号
      # 第 1 步: 如果注解指定了端口, 直接写入 __tmp_port
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
        regex: '(.+)'
        target_label: __tmp_port
        action: replace
      # 第 2 步: 如果注解端口为空, 则从 __address__ (格式 host:port) 提取端口号
      - source_labels: [__tmp_port, __address__]
        regex: '^$;([^:]+):([0-9]+)'
        replacement: '${2}'
        target_label: __tmp_port
        action: replace
      # ★ 路径处理: 优先用注解路径, 否则默认 /metrics
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        regex: '(.+)'
        target_label: __metrics_path__
        action: replace
      # ★ 核心: 将 __metrics_path__ 改写为 API Server Pod 代理路径
      - source_labels: [__meta_kubernetes_namespace]
        regex: '(.+)'
        target_label: __tmp_ns
        action: replace
      - source_labels: [__meta_kubernetes_pod_name]
        regex: '(.+)'
        target_label: __tmp_pod
        action: replace
      - source_labels: [__tmp_ns, __tmp_pod, __tmp_port, __metrics_path__]
        regex: '(.+);(.+);(.+);(.+)'
        replacement: '/api/v1/namespaces/${1}/pods/${2}:${3}/proxy${4}'
        target_label: __metrics_path__
        action: replace
      # 地址统一改写为 API Server
      - target_label: __address__
        replacement: k8s-vip.ink8s.com:6443
      # 清理临时标签
      - regex: '__tmp_(ns|pod|port)'
        action: labeldrop
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod
      - source_labels: [__meta_kubernetes_pod_node_name]
        target_label: node

  # ---------------------------------------------------------------------------
  # 6) (可选) node-exporter 抓取 —— 前提: 集群已用 DaemonSet 部署 node-exporter
  #    通过 API Server pod 代理抓取, 避免直连 Pod IP(ClusterIP 段不可达)
  #    若未部署可先注释掉, 不影响其它 job
  # ---------------------------------------------------------------------------
  - job_name: 'kubernetes-node-exporter'
    kubernetes_sd_configs:
      - role: pod
        api_server: https://k8s-vip.ink8s.com:6443
        tls_config:
          ca_file: /data/apps/monitor/prometheus/certs/ca.crt
        bearer_token_file: /data/apps/monitor/prometheus/tokens/prometheus.token
    scheme: https
    tls_config:
      ca_file: /data/apps/monitor/prometheus/certs/ca.crt
    bearer_token_file: /data/apps/monitor/prometheus/tokens/prometheus.token
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        action: keep
        regex: node-exporter
      # 改写为 API Server pod 代理路径
      - source_labels: [
          __meta_kubernetes_namespace,
          __meta_kubernetes_pod_name
        ]
        target_label: __metrics_path__
        regex: (.+);(.+)
        replacement: /api/v1/namespaces/$1/pods/$2:9100/proxy/metrics
      - target_label: __address__
        replacement: k8s-vip.ink8s.com:6443
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_node_name]
        target_label: node
      - source_labels: [__meta_kubernetes_pod_host_ip]
        target_label: host_ip


# /data/apps/monitor/prometheus/promtool check config /data/apps/monitor/prometheus/prometheus.yml
# kill -HUP 11433

upload successful

3.0 Alertmanager 告警

告警触发:
- severity=critical -> 邮件+企微+钉钉
- severity=warning -> 企微+钉钉
- 告警恢复 -> 三通道全发恢复通知
告警抑制规则:
- 同节点critical触发 -> 抑制该节点warning
- 节点Down -> 抑制该节点 CPU/内存/磁盘/负载告警
- API Server Down -> 抑制 K8S 集群所有告警

3.1 告警规则

3.1.1 主机告警

主机告警:（node-alerts.yml，K8S 节点专用，8条）
筛选 job=”kubernetes-node-exporter”，定位标签为 node + instance

告警名	级别	触发条件	持续时间
NodeCPUHighUsage	warning	CPU > 85%	5m
NodeCPUCritical	critical	CPU > 95%	2m
NodeMemoryHighUsage	warning	内存 > 85%	5m
NodeMemoryCritical	critical	内存 > 95%	2m
NodeDiskHighUsage	warning	根分区 > 80%	5m
NodeDiskCritical	critical	根分区 > 90%	2m
NodeDiskWillFull	warning	预测 4h 内写满	15m
NodeDown	critical	node-exporter 不可达	1m

3.1.2 独立节点告警

独立节点告警:（consul-node-alerts.yml，Consul 发现，10条）
筛选 job=”host_monitor”，定位标签为 instance（IP:Port）
进程服务存活告警

告警名	级别	触发条件	持续时间
ConsulNodeCPUHighUsage	warning	CPU > 85%	5m
ConsulNodeCPUCritical	critical	CPU > 95%	2m
ConsulNodeMemoryHighUsage	warning	内存 > 85%	5m
ConsulNodeMemoryCritical	critical	内存 > 95%	2m
ConsulNodeDiskHighUsage	warning	根分区 > 80%	5m
ConsulNodeDiskCritical	critical	根分区 > 90%	2m
ConsulNodeDiskWillFull	warning	预测 4h 内写满	15m
ConsulNodeDown	critical	node-exporter 不可达	1m
ConsulNodeLoadHigh	warning	Load1 > CPU核数×2	1m
JenkinsProcessDown	critical	jenkins.service 未 active	1m

3.1.3 K8S 集群告警

K8S 集群告警（k8s-alerts.yml，共 15 条）

告警名	级别	触发条件	持续时间
PodCrashLooping	warning	5分钟内重启 > 3次	立即
PodNotReady	warning	Pod 未就绪	3m
PodOOMKilled	warning	容器被 OOM 杀死	立即
PodPending	warning	Pod 卡在 Pending	10m
PodFailed	critical	Pod Failed 状态	立即
ContainerCPUHighUsage	warning	容器 CPU > Limit 90%	5m
ContainerMemoryHighUsage	warning	容器内存 > Limit 90%	5m
DeploymentReplicasMismatch	warning	副本数不匹配	5m
DeploymentUnavailable	critical	无可用副本	1m
K8sNodeMemoryPressure	critical	节点内存压力	2m
K8sNodeDiskPressure	critical	节点磁盘压力	2m
APIServerDown	critical	API Server 不可达	1m
APIServerHighErrorRate	critical	节点 NotReady	2m
K8sNodeNotReady	warning	5xx 错误率 > 5%	5m
APIServerHighLatency	warning	P99 延迟 > 2s	5m

3.2 配置文件

1 2	# mkdir -p /data/apps/monitor/alertmanager/{data,templates} # mkdir -p /data/apps/monitor/prometheus/rules

3.2.1 Alertmanager 主配置

# cat /data/apps/monitor/alertmanager/alertmanager.yml 
# =============================================================================
# Alertmanager 主配置
# 版本: 0.33.0
# 支持通道: 邮件(Email) + 企业微信(WeChat Work) + 钉钉(DingTalk)
#
# 目录结构:
#   /data/apps/monitor/alertmanager/
#   ├── alertmanager.yml          ← 本文件
#   └── templates/
#       └── email.tmpl            ← 邮件模板（企微和钉钉通过 webhook 直发，无需模板）
#
# 说明:
#   - 告警路由策略: critical → 邮件+企微+钉钉三通道
#                  warning  → 企微+钉钉
#                  其他     → 仅邮件
#   - 告警分组: 同一 alertname 的告警合并发送, 避免轰炸
#   - 钉钉需配合 prometheus-webhook-dingtalk 转发器
# =============================================================================

global:
  # 一个告警触发后，如果在 resolve_timeout 内没有新告警，则认为已恢复并发送恢复通知
  resolve_timeout: 5m

  # ----------- 邮件全局配置 -----------
  smtp_smarthost: 'smtp.qiye.aliyun.com:465'      # ← 改成你的 SMTP 服务器:端口
  smtp_from: '[email protected]'   # ← 改成你的发件人邮箱
  smtp_auth_username: '[email protected]'
  smtp_auth_password: 'PASSWORD'    # ← 改成你的邮箱密码/授权码
  smtp_require_tls: false                     # 465端口用SSL, 设为false; 587端口用STARTTLS, 设为true

# 模板文件目录
templates:
  - '/data/apps/monitor/alertmanager/templates/*.tmpl'

# =============================================================================
# 路由规则
# =============================================================================
route:
  # 根路由分组规则: 按 alertname + cluster + namespace 合并
  group_by: ['alertname', 'cluster', 'namespace']
  group_wait: 30s          # 初次触发告警后等待 30s，等同组告警一起发
  group_interval: 5m       # 同组告警发送间隔
  repeat_interval: 4h      # 未恢复告警的重复发送间隔

  # 默认接收者: 发邮件
  receiver: 'email-ops'

  routes:
    # ----- critical 级别: 三通道全发 -----
    - match:
        severity: critical
      receiver: 'all-channels'
      group_wait: 10s         # critical 更快通知
      repeat_interval: 1h     # 未恢复每小时重复

    # ----- warning 级别: 企微 + 钉钉 -----
    - match:
        severity: warning
      receiver: 'im-channels'
      repeat_interval: 2h

    # ----- 告警已恢复: 发送恢复通知 -----
    - match:
        alertstate: resolved
      receiver: 'all-channels'
      group_wait: 10s

# =============================================================================
# 告警接收者定义
# =============================================================================
receivers:

  # ----------- 仅邮件 -----------
  - name: 'email-ops'
    email_configs:
      - to: '[email protected]'            # ← 改成收件人邮箱, 多个用逗号分隔
        send_resolved: true
        html: '{{ template "email.html" . }}'
        headers:
          Subject: '{{ template "email.subject" . }}'

  # ----------- 企微 + 钉钉 -----------
  - name: 'im-channels'
    webhook_configs:
      # 企业微信 Webhook (通过 Bot)
      - url: 'http://127.0.0.1:8061/'
        send_resolved: true
      # 钉钉 (通过 prometheus-webhook-dingtalk)
      - url: 'http://127.0.0.1:8060/dingtalk/ops/send'   # ← 改成 dingtalk-webhook 地址
        send_resolved: true
        max_alerts: 10

  # ----------- 三通道全发 -----------
  - name: 'all-channels'
    email_configs:
      - to: '[email protected]'            # ← 改成收件人邮箱
        send_resolved: true
        html: '{{ template "email.html" . }}'
        headers:
          Subject: '{{ template "email.subject" . }}'
    webhook_configs:
      # 企业微信 Webhook
      - url: 'http://127.0.0.1:8061/'
        send_resolved: true
      # 钉钉
      - url: 'http://127.0.0.1:8060/dingtalk/ops/send'   # ← 改成 dingtalk-webhook 地址
        send_resolved: true
        max_alerts: 10

# =============================================================================
# 告警抑制规则(避免噪音)
# =============================================================================
inhibit_rules:
  # 同一节点 critical 触发时, 抑制 warning
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'instance']

  # 节点 Down 时, 抑制该节点上的其他告警
  - source_match:
      alertname: 'NodeDown'
    target_match_re:
      alertname: 'NodeCPU.*|NodeMemory.*|NodeDisk.*|NodeLoad.*'
    equal: ['instance']

  # API Server Down 时, 抑制 K8S 相关告警
  - source_match:
      alertname: 'APIServerDown'
    target_match_re:
      alertname: 'K8sNode.*|Pod.*|Deployment.*'
    equal: ['cluster']

3.2.2 邮件模板

# cat /data/apps/monitor/alertmanager/templates/email.tmpl 
{{ define "email.subject" -}}
[{{ .Status | toUpper }}] {{ if eq .Status "firing" }}🔥{{ else }}✅{{ end }} {{ .GroupLabels.alertname }} - {{ .CommonLabels.cluster }}
{{- end }}

{{ define "email.html" }}
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<style>
  body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif; background:#f5f5f5; margin:0; padding:20px; }
  .container { max-width:700px; margin:0 auto; background:#fff; border-radius:8px; overflow:hidden; box-shadow:0 2px 8px rgba(0,0,0,.1); }
  .header { padding:20px 24px; color:#fff; }
  .header-firing { background: linear-gradient(135deg, #e74c3c, #c0392b); }
  .header-resolved { background: linear-gradient(135deg, #27ae60, #1e8449); }
  .header h1 { margin:0; font-size:20px; }
  .header .status-badge { display:inline-block; background:rgba(255,255,255,.2); border-radius:4px; padding:2px 10px; font-size:13px; margin-top:6px; }
  .content { padding:20px 24px; }
  .alert-card { border:1px solid #e8e8e8; border-radius:6px; margin-bottom:16px; overflow:hidden; }
  .alert-header { padding:10px 16px; font-weight:600; font-size:14px; }
  .alert-header-critical { background:#fdf0f0; color:#c0392b; border-left:4px solid #e74c3c; }
  .alert-header-warning  { background:#fef9e7; color:#d35400; border-left:4px solid #f39c12; }
  .alert-header-resolved { background:#eafaf1; color:#1e8449; border-left:4px solid #27ae60; }
  .alert-body { padding:10px 16px; font-size:13px; color:#555; }
  .label-row { margin:4px 0; }
  .label-key { color:#999; min-width:90px; display:inline-block; }
  .label-val { color:#333; font-weight:500; }
  .footer { padding:12px 24px; background:#f9f9f9; font-size:12px; color:#999; border-top:1px solid #eee; }
</style>
</head>
<body>
<div class="container">
  <div class="header {{ if eq .Status "firing" }}header-firing{{ else }}header-resolved{{ end }}">
    <h1>{{ if eq .Status "firing" }}🔥 告警触发{{ else }}✅ 告警恢复{{ end }}</h1>
    <span class="status-badge">{{ .Status | toUpper }} · {{ len .Alerts }} 条告警</span>
  </div>
  <div class="content">
    {{ range .Alerts }}
    <div class="alert-card">
      <div class="alert-header alert-header-{{ if .Labels.severity }}{{ .Labels.severity }}{{ else }}warning{{ end }}">
        {{ .Labels.alertname }}
        {{ if .Labels.severity }}[{{ .Labels.severity | toUpper }}]{{ end }}
      </div>
      <div class="alert-body">
        <div class="label-row"><span class="label-key">📋 描述:</span> <span class="label-val">{{ .Annotations.description }}</span></div>
        {{ if .Labels.node }}<div class="label-row"><span class="label-key">🖥 节点:</span> <span class="label-val">{{ .Labels.node }}</span></div>{{ end }}
        {{ if .Labels.pod }}<div class="label-row"><span class="label-key">🎯 Pod:</span> <span class="label-val">{{ .Labels.pod }}</span></div>{{ end }}
        {{ if .Labels.instance }}<div class="label-row"><span class="label-key">🔗 实例:</span> <span class="label-val">{{ .Labels.instance }}</span></div>{{ end }}
        <div class="label-row"><span class="label-key">⏰ 触发时间:</span> <span class="label-val">{{ .StartsAt.Local.Format "2006-01-02 15:04:05" }}</span></div>
        {{ if .EndsAt }}{{ if not .EndsAt.IsZero }}
        <div class="label-row"><span class="label-key">✅ 恢复时间:</span> <span class="label-val">{{ .EndsAt.Local.Format "2006-01-02 15:04:05" }}</span></div>
        {{ end }}{{ end }}
        {{ if .GeneratorURL }}
        <div class="label-row" style="margin-top:8px">
          <a href="{{ .GeneratorURL }}" style="color:#3498db;font-size:12px">🔍 查看 Prometheus 详情 →</a>
        </div>
        {{ end }}
      </div>
    </div>
    {{ end }}
  </div>
  <div class="footer">
    由 Prometheus Alertmanager 自动发送 · 集群: {{ if .CommonLabels.cluster }}{{ .CommonLabels.cluster }}{{ else }}k8s-prod{{ end }}
  </div>
</div>
</body>
</html>
{{ end }}

3.2.3 主机告警配置

# cat /data/apps/monitor/prometheus/rules/node-alerts.yml 
groups:
  - name: node.rules
    interval: 30s   # 该 group 内规则评估间隔(覆盖全局)
    rules:

      # -------------------------------------------------------------------------
      # CPU
      # -------------------------------------------------------------------------
      - alert: NodeCPUHighUsage
        expr: |
          (1 - avg by(instance, node) (rate(node_cpu_seconds_total{job="kubernetes-node-exporter", mode="idle"}[5m]))) * 100 > 85
        for: 5m
        labels:
          severity: warning
          team: ops
        annotations:
          summary: "节点 CPU 使用率过高"
          description: "节点 {{ $labels.node }}({{ $labels.instance }}) CPU 使用率已超过 85%，当前值: {{ $value | printf \"%.1f\" }}%，持续 5 分钟"

      - alert: NodeCPUCritical
        expr: |
          (1 - avg by(instance, node) (rate(node_cpu_seconds_total{job="kubernetes-node-exporter", mode="idle"}[5m]))) * 100 > 95
        for: 2m
        labels:
          severity: critical
          team: ops
        annotations:
          summary: "节点 CPU 使用率严重过高"
          description: "节点 {{ $labels.node }}({{ $labels.instance }}) CPU 使用率已超过 95%，当前值: {{ $value | printf \"%.1f\" }}%，持续 2 分钟"

      # -------------------------------------------------------------------------
      # 内存
      # -------------------------------------------------------------------------
      - alert: NodeMemoryHighUsage
        expr: |
          (1 - (node_memory_MemAvailable_bytes{job="kubernetes-node-exporter"} / node_memory_MemTotal_bytes{job="kubernetes-node-exporter"})) * 100 > 85
        for: 5m
        labels:
          severity: warning
          team: ops
        annotations:
          summary: "节点内存使用率过高"
          description: "节点 {{ $labels.node }}({{ $labels.instance }}) 内存使用率已超过 85%，当前值: {{ $value | printf \"%.1f\" }}%，持续 5 分钟"

      - alert: NodeMemoryCritical
        expr: |
          (1 - (node_memory_MemAvailable_bytes{job="kubernetes-node-exporter"} / node_memory_MemTotal_bytes{job="kubernetes-node-exporter"})) * 100 > 95
        for: 2m
        labels:
          severity: critical
          team: ops
        annotations:
          summary: "节点内存使用率严重过高"
          description: "节点 {{ $labels.node }}({{ $labels.instance }}) 内存使用率已超过 95%，当前值: {{ $value | printf \"%.1f\" }}%，持续 2 分钟"

      # -------------------------------------------------------------------------
      # 磁盘
      # -------------------------------------------------------------------------
      - alert: NodeDiskHighUsage
        expr: |
          (1 - (node_filesystem_avail_bytes{job="kubernetes-node-exporter", mountpoint="/", fstype!~"tmpfs|devtmpfs"} / node_filesystem_size_bytes{job="kubernetes-node-exporter", mountpoint="/", fstype!~"tmpfs|devtmpfs"})) * 100 > 80
        for: 5m
        labels:
          severity: warning
          team: ops
        annotations:
          summary: "节点磁盘使用率过高"
          description: "节点 {{ $labels.node }}({{ $labels.instance }}) 根分区使用率已超过 80%，当前值: {{ $value | printf \"%.1f\" }}%"

      - alert: NodeDiskCritical
        expr: |
          (1 - (node_filesystem_avail_bytes{job="kubernetes-node-exporter", mountpoint="/", fstype!~"tmpfs|devtmpfs"} / node_filesystem_size_bytes{job="kubernetes-node-exporter", mountpoint="/", fstype!~"tmpfs|devtmpfs"})) * 100 > 90
        for: 2m
        labels:
          severity: critical
          team: ops
        annotations:
          summary: "节点磁盘使用率严重过高"
          description: "节点 {{ $labels.node }}({{ $labels.instance }}) 根分区使用率已超过 90%，当前值: {{ $value | printf \"%.1f\" }}%，请立即清理"

      - alert: NodeDiskWillFull
        expr: |
          predict_linear(node_filesystem_avail_bytes{job="kubernetes-node-exporter", mountpoint="/", fstype!~"tmpfs|devtmpfs"}[1h], 4*3600) < 0
        for: 15m
        labels:
          severity: warning
          team: ops
        annotations:
          summary: "节点磁盘预测 4 小时内耗尽"
          description: "节点 {{ $labels.node }}({{ $labels.instance }}) 根分区按当前趋势预测将在 4 小时内写满，请尽快处理"

      # -------------------------------------------------------------------------
      # 节点存活
      # -------------------------------------------------------------------------
      - alert: NodeDown
        expr: |
          up{job="kubernetes-node-exporter"} == 0
        for: 1m
        labels:
          severity: critical
          team: ops
        annotations:
          summary: "节点 node-exporter 不可达"
          description: "节点 {{ $labels.node }}({{ $labels.instance }}) 的 node-exporter 已停止响应超过 1 分钟"

      # -------------------------------------------------------------------------
      # 系统负载
      # -------------------------------------------------------------------------
      - alert: NodeLoadHigh
        expr: |
          node_load1{job="kubernetes-node-exporter"} > count by (instance, node) (node_cpu_seconds_total{job="kubernetes-node-exporter", mode="idle"}) * 2
        for: 5m
        labels:
          severity: warning
          team: ops
        annotations:
          summary: "节点系统负载过高"
          description: "节点 {{ $labels.node }}({{ $labels.instance }}) 1分钟负载为 {{ $value | printf \"%.2f\" }}，超过 CPU 核数的 2 倍，持续 5 分钟"

3.2.4 K8S 集群告警配置

# cat /data/apps/monitor/prometheus/rules/k8s-alerts.yml 
groups:
- interval: 30s
  name: k8s-pod.rules
  rules:
  - alert: PodCrashLooping
    annotations:
      description: 命名空间 {{ $labels.exported_namespace }} 下 Pod {{ $labels.pod }} 容器 {{ $labels.container }} 在 15 分钟内重启了 {{ $value | printf "%.0f" }} 次
      summary: Pod 频繁重启
    expr: 'increase(kube_pod_container_status_restarts_total[15m]) > 3

      '
    for: 0m
    labels:
      severity: warning
      team: ops
  - alert: PodNotReady
    annotations:
      description: 命名空间 {{ $labels.exported_namespace }} 下 Pod {{ $labels.pod }} 已超过 3 分钟处于未就绪状态
      summary: Pod 未就绪
    expr: 'kube_pod_status_ready{condition="true"} == 0

      '
    for: 3m
    labels:
      severity: warning
      team: ops
  - alert: PodOOMKilled
    annotations:
      description: 命名空间 {{ $labels.exported_namespace }} 下 Pod {{ $labels.pod }} 容器 {{ $labels.container }} 因内存不足(OOMKilled)被终止，请检查内存限制是否合理
      summary: Pod 容器 OOM 被杀
    expr: 'kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1 or kube_pod_container_status_terminated_reason{reason="OOMKilled"} == 1

      '
    for: 0m
    labels:
      severity: warning
      team: ops
  - alert: PodPending
    annotations:
      description: 命名空间 {{ $labels.exported_namespace }} 下 Pod {{ $labels.pod }} 已超过 10 分钟处于 Pending 状态，可能存在资源不足或调度问题
      summary: Pod 长时间处于 Pending 状态
    expr: 'kube_pod_status_phase{phase="Pending"} == 1

      '
    for: 10m
    labels:
      severity: warning
      team: ops
  - alert: PodFailed
    annotations:
      description: 命名空间 {{ $labels.exported_namespace }} 下 Pod {{ $labels.pod }} 处于 Failed 状态，请立即排查
      summary: Pod 处于 Failed 状态
    expr: 'kube_pod_status_phase{phase="Failed"} == 1

      '
    for: 0m
    labels:
      severity: critical
      team: ops
  - alert: ContainerCPUHighUsage
    annotations:
      description: 命名空间 {{ $labels.namespace }} / Pod {{ $labels.pod }} / 容器 {{ $labels.container }} CPU 使用率已达 Limit 的 {{ $value | printf "%.1f" }}%
      summary: 容器 CPU 使用率超过 Limit 90%
    expr: '(sum(rate(container_cpu_usage_seconds_total{container!="POD",container!=""}[5m])) by (namespace, pod, container)

      / sum(container_spec_cpu_quota{container!="POD",container!=""}/container_spec_cpu_period{container!="POD",container!=""}) by (namespace, pod, container)) * 100 > 90

      '
    for: 5m
    labels:
      severity: warning
      team: ops
  - alert: ContainerMemoryHighUsage
    annotations:
      description: 命名空间 {{ $labels.namespace }} / Pod {{ $labels.pod }} / 容器 {{ $labels.container }} 内存使用已达 Limit 的 {{ $value | printf "%.1f" }}%，OOM 风险高
      summary: 容器内存使用率超过 Limit 90%
    expr: "(\n  container_memory_working_set_bytes{container!=\"POD\",container!=\"\"}\n  / on(namespace, pod, container)\n  (container_spec_memory_limit_bytes{container!=\"POD\",container!=\"\"} > 0)\n\
      ) * 100 > 90"
    for: 5m
    labels:
      severity: warning
      team: ops
- interval: 30s
  name: k8s-deployment.rules
  rules:
  - alert: DeploymentReplicasMismatch
    annotations:
      description: 命名空间 {{ $labels.exported_namespace }} Deployment {{ $labels.deployment }} 期望 {{ $labels.replicas }} 个副本，但可用副本数不足，持续 5 分钟
      summary: Deployment 副本数不匹配
    expr: 'kube_deployment_spec_replicas != kube_deployment_status_replicas_available

      '
    for: 5m
    labels:
      severity: warning
      team: ops
  - alert: DeploymentUnavailable
    annotations:
      description: 命名空间 {{ $labels.exported_namespace }} Deployment {{ $labels.deployment }} 已无可用副本，服务中断！
      summary: Deployment 全部副本不可用
    expr: 'kube_deployment_status_replicas_available == 0

      '
    for: 1m
    labels:
      severity: critical
      team: ops
- interval: 30s
  name: k8s-node.rules
  rules:
  - alert: K8sNodeNotReady
    annotations:
      description: K8S 节点 {{ $labels.node }} 处于 NotReady 状态超过 2 分钟，请立即检查
      summary: K8S 节点 NotReady
    expr: 'kube_node_status_condition{condition="Ready",status="true"} == 0

      '
    for: 2m
    labels:
      severity: critical
      team: ops
  - alert: K8sNodeMemoryPressure
    annotations:
      description: K8S 节点 {{ $labels.node }} 处于 MemoryPressure 状态，节点内存严重不足
      summary: K8S 节点内存压力
    expr: 'kube_node_status_condition{condition="MemoryPressure",status="true"} == 1

      '
    for: 2m
    labels:
      severity: critical
      team: ops
  - alert: K8sNodeDiskPressure
    annotations:
      description: K8S 节点 {{ $labels.node }} 处于 DiskPressure 状态，节点磁盘严重不足
      summary: K8S 节点磁盘压力
    expr: 'kube_node_status_condition{condition="DiskPressure",status="true"} == 1

      '
    for: 2m
    labels:
      severity: critical
      team: ops
- interval: 30s
  name: k8s-apiserver.rules
  rules:
  - alert: APIServerDown
    annotations:
      description: Kubernetes API Server ({{ $labels.instance }}) 已超过 1 分钟无响应，集群控制面异常！
      summary: Kubernetes API Server 不可达
    expr: 'up{job="kubernetes-apiservers"} == 0

      '
    for: 1m
    labels:
      severity: critical
      team: ops
  - alert: APIServerHighErrorRate
    annotations:
      description: 'Kubernetes API Server 5xx 错误率超过 5%，当前值: {{ $value | printf "%.2f" }}%'
      summary: API Server 错误率过高
    expr: 'sum(rate(apiserver_request_total{job="kubernetes-apiservers",code=~"5.."}[5m]))

      / sum(rate(apiserver_request_total{job="kubernetes-apiservers"}[5m])) * 100 > 5

      '
    for: 5m
    labels:
      severity: warning
      team: ops
  - alert: APIServerHighLatency
    annotations:
      description: 'Kubernetes API Server P99 延迟超过 2 秒，当前值: {{ $value | printf "%.2f" }}s'
      summary: API Server 请求延迟过高
    expr: 'histogram_quantile(0.99, sum(rate(apiserver_request_duration_seconds_bucket{job="kubernetes-apiservers", verb!~"WATCH|LIST"}[5m])) by (le)) > 2

      '
    for: 5m
    labels:
      severity: warning
      team: ops

3.2.5 独立节点告警配置

# cat /data/apps/monitor/prometheus/rules/consul-node-alerts.yml 
groups:
- interval: 30s
  name: consul-node.rules
  rules:
  - alert: ConsulNodeCPUHighUsage
    annotations:
      description: '主机 {{ $labels.nodename }} ({{ $labels.instance }}) CPU 使用率已超过 85%，当前值: {{ $value | printf "%.1f" }}%，持续 5 分钟'
      summary: 独立节点 CPU 使用率过高
    expr: "(\n  (1 - avg by(instance) (rate(node_cpu_seconds_total{job=\"host_monitor\", mode=\"idle\"}[5m]))) * 100\n)\n* on(instance) group_left(nodename)\nnode_uname_info{job=\"host_monitor\"}\n> 85\n"
    for: 5m
    labels:
      severity: warning
      team: ops
  - alert: ConsulNodeCPUCritical
    annotations:
      description: '主机 {{ $labels.nodename }} ({{ $labels.instance }}) CPU 使用率已超过 95%，当前值: {{ $value | printf "%.1f" }}%，持续 2 分钟'
      summary: 独立节点 CPU 使用率严重过高
    expr: "(\n  (1 - avg by(instance) (rate(node_cpu_seconds_total{job=\"host_monitor\", mode=\"idle\"}[5m]))) * 100\n)\n* on(instance) group_left(nodename)\nnode_uname_info{job=\"host_monitor\"}\n> 95\n"
    for: 2m
    labels:
      severity: critical
      team: ops
  - alert: ConsulNodeMemoryHighUsage
    annotations:
      description: '主机 {{ $labels.nodename }} ({{ $labels.instance }}) 内存使用率已超过 85%，当前值: {{ $value | printf "%.1f" }}%，持续 5 分钟'
      summary: 独立节点内存使用率过高
    expr: "(\n  (1 - (node_memory_MemAvailable_bytes{job=\"host_monitor\"} / node_memory_MemTotal_bytes{job=\"host_monitor\"})) * 100\n)\n* on(instance) group_left(nodename)\nnode_uname_info{job=\"host_monitor\"}\n> 85\n"
    for: 5m
    labels:
      severity: warning
      team: ops
  - alert: ConsulNodeMemoryCritical
    annotations:
      description: '主机 {{ $labels.nodename }} ({{ $labels.instance }}) 内存使用率已超过 95%，当前值: {{ $value | printf "%.1f" }}%，持续 2 分钟'
      summary: 独立节点内存使用率严重过高
    expr: "(\n  (1 - (node_memory_MemAvailable_bytes{job=\"host_monitor\"} / node_memory_MemTotal_bytes{job=\"host_monitor\"})) * 100\n)\n* on(instance) group_left(nodename)\nnode_uname_info{job=\"host_monitor\"}\n> 95\n"
    for: 2m
    labels:
      severity: critical
      team: ops
  - alert: ConsulNodeDiskHighUsage
    annotations:
      description: '主机 {{ $labels.nodename }} ({{ $labels.instance }}) 根分区使用率已超过 80%，当前值: {{ $value | printf "%.1f" }}%'
      summary: 独立节点磁盘使用率过高
    expr: "(\n  (1 - (node_filesystem_avail_bytes{job=\"host_monitor\", mountpoint=\"/\", fstype!~\"tmpfs|devtmpfs\"} / node_filesystem_size_bytes{job=\"host_monitor\", mountpoint=\"/\", fstype!~\"tmpfs|devtmpfs\"})) * 100\n)\n* on(instance) group_left(nodename)\nnode_uname_info{job=\"host_monitor\"}\n\
      > 80\n"
    for: 5m
    labels:
      severity: warning
      team: ops
  - alert: ConsulNodeDiskCritical
    annotations:
      description: '主机 {{ $labels.nodename }} ({{ $labels.instance }}) 根分区使用率已超过 90%，当前值: {{ $value | printf "%.1f" }}%，请立即清理'
      summary: 独立节点磁盘使用率严重过高
    expr: "(\n  (1 - (node_filesystem_avail_bytes{job=\"host_monitor\", mountpoint=\"/\", fstype!~\"tmpfs|devtmpfs\"} / node_filesystem_size_bytes{job=\"host_monitor\", mountpoint=\"/\", fstype!~\"tmpfs|devtmpfs\"})) * 100\n)\n* on(instance) group_left(nodename)\nnode_uname_info{job=\"host_monitor\"}\n\
      > 90\n"
    for: 2m
    labels:
      severity: critical
      team: ops
  - alert: ConsulNodeDiskWillFull
    annotations:
      description: 主机 {{ $labels.nodename }} ({{ $labels.instance }}) 根分区按当前趋势预测将在 4 小时内写满，请尽快处理
      summary: 独立节点磁盘预测 4 小时内耗尽
    expr: 'predict_linear(node_filesystem_avail_bytes{job="host_monitor", mountpoint="/", fstype!~"tmpfs|devtmpfs"}[1h], 4*3600)

      * on(instance) group_left(nodename)

      node_uname_info{job="host_monitor"}

      < 0

      '
    for: 15m
    labels:
      severity: warning
      team: ops
  - alert: ConsulNodeDown
    annotations:
      description: 主机 {{ $labels.nodename }} ({{ $labels.instance }}) 的 node-exporter 已停止响应超过 1 分钟，主机可能宕机
      summary: 独立节点 node-exporter 不可达
    expr: up{job="host_monitor"} == 0
    for: 1m
    labels:
      severity: critical
      team: ops
  - alert: ConsulNodeLoadHigh
    annotations:
      description: 主机 {{ $labels.nodename }} ({{ $labels.instance }}) 1分钟负载为 {{ $value | printf "%.2f" }}，超过 CPU 核数的 2 倍
      summary: 独立节点系统负载过高
    expr: "node_load1{job=\"host_monitor\"}\n* on(instance) group_left(nodename)\nnode_uname_info{job=\"host_monitor\"}\n> (\n  count by(instance) (node_cpu_seconds_total{job=\"host_monitor\", mode=\"idle\"})\n  * on(instance) group_left(nodename)\n  node_uname_info{job=\"host_monitor\"}\n) * 2\n"
    for: 5m
    labels:
      severity: warning
      team: ops
  - alert: JenkinsProcessDown
    annotations:
      description: 主机 {{ $labels.nodename }} ({{ $labels.instance }}) 的 Jenkins 不可用：主机不可达或 jenkins.service 未运行，请检查
      summary: Jenkins 服务不可用
    expr: '# 情况1: 主机不可达

      up{job="host_monitor", instance=~"192.168.10.116:.*"} == 0

      or

      # 情况2: 主机可达但 jenkins.service 不在 active 状态

      up{job="host_monitor", instance=~"192.168.10.116:.*"} == 1

      unless on(instance)

      node_systemd_unit_state{name="jenkins.service", state="active"} == 1'
    for: 2m
    labels:
      severity: critical
      team: ops

3.2.6 钉钉配置

# wget https://github.com/timonwong/prometheus-webhook-dingtalk/releases/download/v2.1.0/prometheus-webhook-dingtalk-2.1.0.linux-amd64.tar.gz
# tar -xzf prometheus-webhook-dingtalk-2.1.0.linux-amd64.tar.gz
# cp prometheus-webhook-dingtalk-2.1.0.linux-amd64/prometheus-webhook-dingtalk /usr/local/bin/
# mv prometheus-webhook-dingtalk-2.1.0.linux-amd64 /data/apps/monitor/dingtalk/
# mkdir -p /data/apps/monitor/dingtalk/templates

# cat /data/apps/monitor/dingtalk/config.yml 
timeout: 5s
# 并发请求数

# 自定义消息模板
templates:
  - /data/apps/monitor/dingtalk/templates/*.tmpl

targets:
  # ── ops 群机器人 ──────────────────────────────────────────
  ops:
    url: https://oapi.dingtalk.com/robot/send?access_token=TOKEN_ID
    # 如果钉钉机器人开启了"加签"安全设置, 取消注释并填写 Secret
    # secret: YOUR-DINGTALK-SECRET
    mention:
      # 发送时 @指定手机号的人 (可选)
      # mobiles: ['138XXXXXXXX']
      all: false
    message:
      title: '{{ template "dingtalk.title" . }}'
      text: '{{ template "dingtalk.content" . }}'

3.2.7 钉钉模板

# cat /data/apps/monitor/dingtalk/templates/dingtalk.tmpl 
{{ define "dingtalk.title" -}}
{{ if eq .Status "firing" }}🔥 告警触发{{ else }}✅ 告警恢复{{ end }}: {{ .GroupLabels.alertname }}
{{ end }}

{{ define "dingtalk.content" -}}
{{ if eq .Status "firing" -}}
## 🔥 告警触发 ({{ .Alerts.Firing | len }} 条)
{{ else -}}
## ✅ 告警恢复 ({{ .Alerts.Resolved | len }} 条)
{{ end }}

{{ if .CommonLabels.cluster }}**集群:** {{ .CommonLabels.cluster }}
{{ end }}
{{ if .CommonLabels.severity }}**级别:** {{ .CommonLabels.severity }}
{{ end }}

{{ range .Alerts }}
---
**{{ .Labels.alertname }}** `{{ .Labels.severity }}`

> {{ .Annotations.description }}

{{ if .Labels.instance }}- 实例: {{ .Labels.instance }}{{ end }}
{{ if .Labels.pod }}- Pod: {{ .Labels.pod }}{{ end }}
{{ if .Labels.node }}- 节点: {{ .Labels.node }}{{ end }}
- 触发时间: {{ .StartsAt.Local.Format "2006-01-02 15:04:05" }}
{{ end }}
{{ end }}

钉钉systemd

# cat /usr/lib/systemd/system/dingtalk-webhook.service 
[Unit]
Description=Prometheus DingTalk Webhook
After=network.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecStart=/usr/local/bin/prometheus-webhook-dingtalk \
  --config.file=/data/apps/monitor/dingtalk/config.yml \
  --web.listen-address=127.0.0.1:8060
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target


# systemctl daemon-reload

3.2.8 中转脚本

Python 中转服务，把 Alertmanager 格式转成企微机器人格式

# yum install -y python3
# pip3 install requests
# mkdir -p /data/apps/monitor/wechat-relay

# cat /data/apps/monitor/wechat-relay/wechat-relay.py 
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
企业微信机器人告警转发 — Alertmanager Webhook → 企业微信 Bot
兼容 Python 3.6+ (CentOS 7 默认)
"""
import json, os, sys
from datetime import datetime, timedelta
from http.server import HTTPServer, BaseHTTPRequestHandler
from urllib import request, error

WECHAT_KEY = os.environ.get("WECHAT_KEY", "YOUR-WECHAT-BOT-KEY")
WECHAT_URL = "https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key={}".format(WECHAT_KEY)


def parse_time(iso_str):
    if not iso_str:
        return ""
    s = iso_str.replace("Z", "").replace("z", "")
    for fmt in ["%Y-%m-%dT%H:%M:%S.%f", "%Y-%m-%dT%H:%M:%S"]:
        try:
            dt = datetime.strptime(s, fmt)
            dt = dt + timedelta(hours=8)
            return dt.strftime("%Y-%m-%d %H:%M:%S")
        except ValueError:
            continue
    return s


def build_markdown(data):
    status = data.get("status", "firing")
    alerts = data.get("alerts", [])
    group_labels = data.get("groupLabels", {})

    lines = []
    red_open = '<font color="warning">'
    red_close = '</font>'
    green_open = '<font color="info">'
    green_close = '</font>'

    if status == "firing":
        alert_count = sum(1 for a in alerts if a.get("status") == "firing")
        lines.append('## {}🔥 告警触发{}  ({}条)'.format(red_open, red_close, alert_count))
    else:
        lines.append('## {}✅ 告警恢复{}  ({}条)'.format(green_open, green_close, len(alerts)))

    cluster = group_labels.get("cluster", "")
    if cluster and cluster != "standalone":
        lines.append("> 集群: **{}**".format(cluster))

    severity = group_labels.get("severity", "")
    if severity:
        label = "严重" if severity == "critical" else "警告"
        lines.append("> 级别: **{}**".format(label))

    lines.append("")
    for alert in alerts[:5]:
        labels = alert.get("labels", {})
        annotations = alert.get("annotations", {})
        name = labels.get("alertname", "Unknown")
        sev = labels.get("severity", "")
        desc = annotations.get("description", annotations.get("summary", ""))
        instance = labels.get("instance", "") or labels.get("node", "")
        starts_at = parse_time(alert.get("startsAt", ""))

        lines.append("> **{}** `{}`".format(name, sev))
        if desc:
            lines.append("> {}".format(desc[:120]))
        if instance:
            lines.append("> 📍 {}".format(instance))
        if starts_at:
            lines.append("> ⏰ {}".format(starts_at))
        lines.append("")

    if len(alerts) > 5:
        lines.append("> ... 还有 {} 条".format(len(alerts) - 5))

    external_url = data.get("externalURL", "")
    if external_url:
        lines.append("")
        lines.append("[📊 查看 Prometheus]({})".format(external_url))

    return "\n".join(lines)


class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        content_length = int(self.headers.get("Content-Length", 0))
        body = self.rfile.read(content_length)
        try:
            data = json.loads(body)
        except Exception:
            self.send_error(400, "Invalid JSON")
            return

        md = build_markdown(data)
        payload = json.dumps({
            "msgtype": "markdown",
            "markdown": {"content": md}
        }).encode("utf-8")

        try:
            req = request.Request(WECHAT_URL, data=payload,
                                  headers={"Content-Type": "application/json"})
            resp = request.urlopen(req, timeout=10)
            status_code = resp.getcode()
            resp_body = resp.read().decode()
            print("[{}] {}".format(status_code, resp_body), file=sys.stderr)
            self.send_response(200)
            self.end_headers()
            self.wfile.write(b"ok")
        except error.HTTPError as e:
            print("[{}] {}".format(e.code, e.read().decode()), file=sys.stderr)
            self.send_response(502)
            self.end_headers()
            self.wfile.write("wechat api error: {}".format(e.code).encode())

    def log_message(self, fmt, *args):
        print("[wechat-relay] {}".format(args[0]), file=sys.stderr)


if __name__ == "__main__":
    port = int(os.environ.get("LISTEN_PORT", "8061"))
    server = HTTPServer(("127.0.0.1", port), Handler)
    print("WeChat relay listening on 127.0.0.1:{}".format(port), file=sys.stderr)
    server.serve_forever()

# chmod +x /data/apps/monitor/wechat-relay/wechat-relay.py
# cat /usr/lib/systemd/system/wechat-relay.service
[Unit]
Description=WeChat Alert Relay
After=network.target

[Service]
Type=simple
User=prometheus
Group=prometheus
WorkingDirectory=/data/apps/monitor/wechat-relay
Environment=WECHAT_KEY=TOKEN_ID
ExecStart=/usr/bin/python3 /data/apps/monitor/wechat-relay/wechat-relay.py
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

systemctl daemon-reload
systemctl enable --now wechat-relay
systemctl status wechat-relay

3.2.9 prometheus 主配置

# cat /data/apps/monitor/prometheus/prometheus.yml
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - 127.0.0.1:9093
    timeout: 10s
global:
  evaluation_interval: 30s
  scrape_interval: 30s
  scrape_timeout: 15s
rule_files:
- /data/apps/monitor/prometheus/rules/*.yml
scrape_configs:
- consul_sd_configs:
  - server: 192.168.10.118:8500
    services:
    - node-exporter
  job_name: host_monitor
  relabel_configs:
  - replacement: standalone
    target_label: cluster
  - source_labels:
    - __meta_consul_service_id
    target_label: nodename
- bearer_token_file: /data/apps/monitor/prometheus/tokens/prometheus.token
  job_name: kubernetes-apiservers
  kubernetes_sd_configs:
  - api_server: https://k8s-vip.ink8s.com:6443
    bearer_token_file: /data/apps/monitor/prometheus/tokens/prometheus.token
    role: endpoints
    tls_config:
      ca_file: /data/apps/monitor/prometheus/certs/ca.crt
  relabel_configs:
  - action: keep
    regex: default;kubernetes;https
    source_labels:
    - __meta_kubernetes_namespace
    - __meta_kubernetes_service_name
    - __meta_kubernetes_endpoint_port_name
  - replacement: apiserver
    target_label: component
  - source_labels:
    - __meta_kubernetes_namespace
    target_label: namespace
  - replacement: k8s-prod
    source_labels:
    - __address__
    target_label: cluster
  scheme: https
  tls_config:
    ca_file: /data/apps/monitor/prometheus/certs/ca.crt
- bearer_token_file: /data/apps/monitor/prometheus/tokens/prometheus.token
  job_name: kubernetes-nodes
  kubernetes_sd_configs:
  - api_server: https://k8s-vip.ink8s.com:6443
    bearer_token_file: /data/apps/monitor/prometheus/tokens/prometheus.token
    role: node
    tls_config:
      ca_file: /data/apps/monitor/prometheus/certs/ca.crt
  relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_node_label_(.+)
  - replacement: k8s-vip.ink8s.com:6443
    target_label: __address__
  - regex: (.+)
    replacement: /api/v1/nodes/${1}/proxy/metrics
    source_labels:
    - __meta_kubernetes_node_name
    target_label: __metrics_path__
  - source_labels:
    - __meta_kubernetes_node_name
    target_label: node
  - source_labels:
    - __meta_kubernetes_node_address_InternalIP
    target_label: internal_ip
  - replacement: k8s-prod
    source_labels:
    - __address__
    target_label: cluster
  scheme: https
  tls_config:
    ca_file: /data/apps/monitor/prometheus/certs/ca.crt
- bearer_token_file: /data/apps/monitor/prometheus/tokens/prometheus.token
  job_name: kubernetes-cadvisor
  kubernetes_sd_configs:
  - api_server: https://k8s-vip.ink8s.com:6443
    bearer_token_file: /data/apps/monitor/prometheus/tokens/prometheus.token
    role: node
    tls_config:
      ca_file: /data/apps/monitor/prometheus/certs/ca.crt
  metric_relabel_configs:
  - action: drop
    regex: container_(network_tcp_usage_total|network_udp_usage_total|tasks_state|cpu_load_average_10s)
    source_labels:
    - __name__
  relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_node_label_(.+)
  - replacement: k8s-vip.ink8s.com:6443
    target_label: __address__
  - regex: (.+)
    replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
    source_labels:
    - __meta_kubernetes_node_name
    target_label: __metrics_path__
  - source_labels:
    - __meta_kubernetes_node_name
    target_label: node
  - source_labels:
    - __meta_kubernetes_node_address_InternalIP
    target_label: internal_ip
  - replacement: k8s-prod
    source_labels:
    - __address__
    target_label: cluster
  scheme: https
  tls_config:
    ca_file: /data/apps/monitor/prometheus/certs/ca.crt
- bearer_token_file: /data/apps/monitor/prometheus/tokens/prometheus.token
  job_name: kubernetes-service-endpoints
  kubernetes_sd_configs:
  - api_server: https://k8s-vip.ink8s.com:6443
    bearer_token_file: /data/apps/monitor/prometheus/tokens/prometheus.token
    role: endpoints
    tls_config:
      ca_file: /data/apps/monitor/prometheus/certs/ca.crt
  relabel_configs:
  - action: keep
    regex: true
    source_labels:
    - __meta_kubernetes_service_annotation_prometheus_io_scrape
  - action: replace
    regex: (.+)
    source_labels:
    - __meta_kubernetes_service_annotation_prometheus_io_port
    target_label: __tmp_port
  - action: replace
    regex: ^$;([^:]+):([0-9]+)
    replacement: ${2}
    source_labels:
    - __tmp_port
    - __address__
    target_label: __tmp_port
  - action: replace
    regex: (.+)
    source_labels:
    - __meta_kubernetes_service_annotation_prometheus_io_path
    target_label: __metrics_path__
  - action: replace
    regex: (.+)
    source_labels:
    - __meta_kubernetes_namespace
    target_label: __tmp_ns
  - action: replace
    regex: (.+)
    source_labels:
    - __meta_kubernetes_service_name
    target_label: __tmp_svc
  - action: replace
    regex: (.+);(.+);(.+);(.+)
    replacement: /api/v1/namespaces/${1}/services/${2}:${3}/proxy${4}
    source_labels:
    - __tmp_ns
    - __tmp_svc
    - __tmp_port
    - __metrics_path__
    target_label: __metrics_path__
  - replacement: k8s-vip.ink8s.com:6443
    target_label: __address__
  - action: labeldrop
    regex: __tmp_(ns|svc|port)
  - action: labelmap
    regex: __meta_kubernetes_service_label_(.+)
  - source_labels:
    - __meta_kubernetes_namespace
    target_label: namespace
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: service
  - replacement: k8s-prod
    source_labels:
    - __address__
    target_label: cluster
  scheme: https
  tls_config:
    ca_file: /data/apps/monitor/prometheus/certs/ca.crt
- bearer_token_file: /data/apps/monitor/prometheus/tokens/prometheus.token
  job_name: kubernetes-pods
  kubernetes_sd_configs:
  - api_server: https://k8s-vip.ink8s.com:6443
    bearer_token_file: /data/apps/monitor/prometheus/tokens/prometheus.token
    role: pod
    tls_config:
      ca_file: /data/apps/monitor/prometheus/certs/ca.crt
  relabel_configs:
  - action: keep
    regex: true
    source_labels:
    - __meta_kubernetes_pod_annotation_prometheus_io_scrape
  - action: replace
    regex: (.+)
    source_labels:
    - __meta_kubernetes_pod_annotation_prometheus_io_port
    target_label: __tmp_port
  - action: replace
    regex: ^$;([^:]+):([0-9]+)
    replacement: ${2}
    source_labels:
    - __tmp_port
    - __address__
    target_label: __tmp_port
  - action: replace
    regex: (.+)
    source_labels:
    - __meta_kubernetes_pod_annotation_prometheus_io_path
    target_label: __metrics_path__
  - action: replace
    regex: (.+)
    source_labels:
    - __meta_kubernetes_namespace
    target_label: __tmp_ns
  - action: replace
    regex: (.+)
    source_labels:
    - __meta_kubernetes_pod_name
    target_label: __tmp_pod
  - action: replace
    regex: (.+);(.+);(.+);(.+)
    replacement: /api/v1/namespaces/${1}/pods/${2}:${3}/proxy${4}
    source_labels:
    - __tmp_ns
    - __tmp_pod
    - __tmp_port
    - __metrics_path__
    target_label: __metrics_path__
  - replacement: k8s-vip.ink8s.com:6443
    target_label: __address__
  - action: labeldrop
    regex: __tmp_(ns|pod|port)
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
  - source_labels:
    - __meta_kubernetes_namespace
    target_label: namespace
  - source_labels:
    - __meta_kubernetes_pod_name
    target_label: pod
  - source_labels:
    - __meta_kubernetes_pod_node_name
    target_label: node
  - replacement: k8s-prod
    source_labels:
    - __address__
    target_label: cluster
  scheme: https
  tls_config:
    ca_file: /data/apps/monitor/prometheus/certs/ca.crt
- bearer_token_file: /data/apps/monitor/prometheus/tokens/prometheus.token
  job_name: kubernetes-node-exporter
  kubernetes_sd_configs:
  - api_server: https://k8s-vip.ink8s.com:6443
    bearer_token_file: /data/apps/monitor/prometheus/tokens/prometheus.token
    role: pod
    tls_config:
      ca_file: /data/apps/monitor/prometheus/certs/ca.crt
  relabel_configs:
  - action: keep
    regex: node-exporter
    source_labels:
    - __meta_kubernetes_pod_label_app
  - regex: (.+);(.+)
    replacement: /api/v1/namespaces/$1/pods/$2:9100/proxy/metrics
    source_labels:
    - __meta_kubernetes_namespace
    - __meta_kubernetes_pod_name
    target_label: __metrics_path__
  - replacement: k8s-vip.ink8s.com:6443
    target_label: __address__
  - source_labels:
    - __meta_kubernetes_namespace
    target_label: namespace
  - source_labels:
    - __meta_kubernetes_pod_node_name
    target_label: node
  - source_labels:
    - __meta_kubernetes_pod_host_ip
    target_label: host_ip
  - replacement: k8s-prod
    source_labels:
    - __address__
    target_label: cluster
  scheme: https
  tls_config:
    ca_file: /data/apps/monitor/prometheus/certs/ca.crt

// 重启服务
# systemctl restart alertmanager.service
# systemctl start dingtalk-webhook.service
# systemctl enable --now dingtalk-webhook
# kill -HUP PROMETHEUS_PID

3.3 告警测试

邮件、企微、钉钉三通道告警测试

/data/apps/monitor/alertmanager/amtool alert add \
  alertname=TestAlert \
  severity=critical \
  --alertmanager.url=http://127.0.0.1:9093 \
  --annotation=summary='测试告警' \
  --annotation=description='三通道告警测试，收到说明配置成功'

3.4 告警过滤

如报警过于频繁，或者已经在处理问题了，但是还在发送告警通知，这时可以对报警信息进行静默设置

3.4.1 静默通知

在 Alertmanager 的后台页面中提供了静默操作的入口

upload successful

3.4.2 抑制

抑制是指当某次告警发出后，可以停止重复发送由此告警引发的其他告警的机制，比如现在有一台服务器宕机了，上面跑了很多服务都设置了告警，那么肯定会收到大量无用的告警信息，这个时候抑制就非常有用了，可以有效的防止告警风暴

inhibit_rules:
  # 同一节点 critical 触发时, 抑制 warning
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'instance']

  # 节点 Down -> 该 instance 上的 Node 级别告警
  - source_match:
      alertname: 'NodeDown'
    target_match_re:
      alertname: 'NodeCPU.*|NodeMemory.*|NodeDisk.*|NodeLoad.*'
    equal: ['instance']

  # 节点 Down -> 该 node 上的 K8S 告警(Pod/容器等)
  - source_match:
      alertname: 'NodeDown'
    target_match_re:
      alertname: 'K8sNode.*|Pod.*|Deployment.*|Container.*'
    equal: ['node']

  # K8s 节点 NotReady -> 该 node 上的 Pod/容器告警
  - source_match:
      alertname: 'K8sNodeNotReady'
    target_match_re:
      alertname: 'Pod.*|Deployment.*|Container.*'
    equal: ['node']

  # API Server Down -> 全集群 K8S 告警
  - source_match:
      alertname: 'APIServerDown'
    target_match_re:
      alertname: 'K8sNode.*|Pod.*|Deployment.*|Container.*'
    equal: ['cluster']

  # ConsulNodeDown -> all Consul node and process alerts on same instance
  - source_match:
      alertname: 'ConsulNodeDown'
    target_match_re:
      alertname: 'ConsulNodeCPU.*|ConsulNodeMemory.*|ConsulNodeDisk.*|ConsulNodeLoad.*|JenkinsProcessDown'
    equal: ['instance']

# systemctl restart alertmanager.service

4.0 Grafana 仪表盘

4.1 Grafana 基础配置

4.1.1 数据源配置

连接 –> 数据源 –> Connection: (http://192.168.10.118:9090)

upload successful

4.1.2 导入仪表板

左侧点击”仪表板” –> 右上角”新建” –> 选择”Import dashbaord”

upload successful

4.1.3 仪表板ID清单

监控对象	仪表盘名称	Grafana_ID	说明
node-exporter	Node Exporter Full	1860	最全面的主机监控，CPU/内存/磁盘/网络
cAdvisor	Kubernetes Pods Monitoring	14282	容器/Pod 资源监控
cAdvisor	Kubernetes Cluster Monitoring (cAdvisor)	3119	集群级容器监控
kube-state-metrics	Kubernetes Cluster (Prometheus)	15757	资源对象状态，Deployment/Pod/Node 等
kube-state-metrics	Kubernetes Stateful Metrics	13105	另一个 kube-state-metrics 视图
API Server	Kubernetes API Server	17688	API Server 性能指标
K8S 总览	Kubernetes All-in-One	15661	综合视图，包含多个组件

4.2 图形面板

Grafana 中所有的面板均以插件形式使用，Grafana 提供了各种可视化来支持不同的用例:
- Time series（时间序列）默认的也是主要的图形可视化面板
- State timeline（状态时间表）状态随时间变化
- Status history（状态历史记录）
- Bar chart（条形图）
- Histogram（直方图）
- Heatmap（热力图）
- Pie chart （饼状图）
- Stat（统计数据）
- Gauge、Bar gauge、Table（表格）
- Logs（日志）
- Node Graph（节点图）
- Dashboard list（仪表板列表）
- Alert list（报警列表）
- Text panel（文本面板、支持markdown和html）
- News Panel（新闻面板，可以显示RSS摘要）
其它面板信息获取

4.2.1 Time series 时间序列

Time series（时间序列）默认的也是主要的图形可视化面板

4.2.1.1 CPU使用率公式

准备好一个CPU使用率公式,用于下面创建时间序列图时使用

1 2	// CPU 使用率 (1 - sum(rate(node_cpu_seconds_total{mode="idle"}[5m])) by (instance) / sum(rate(node_cpu_seconds_total[5m])) by (instance) ) * 100

4.2.1.2 创建时间序列自定义图

点击左侧”仪表板” –> 点击右上角”新建” –> 选择”新建仪表板” –> 点击”+” –> 选择”配置可视化” –> 点击右侧”所有可视化” –> 单击”Time series”: 如图配置最后保存

upload successful

4.2.1.3 配置自定义图设置

可针对这个自定义的仪表盘进行配置: 单击右侧齿轮”数据面板选项” –> 单击 “View all settings”
可以配置名称、标签、变量等

upload successful

4.2.1.4 过滤节点IP地址

upload successful

4.2.1.5 标签替换

针对单个图表将 “语句中的标签进行变量替换”，修改完成后保存

1
2

// CPU使用率 添加 instance=~"$HOST_IP" 变量
(1 - sum(rate(node_cpu_seconds_total{instance=~"$HOST_IP" ,mode="idle"}[5m])) by (instance) / sum(rate(node_cpu_seconds_total{instance=~"$HOST_IP"}[5m])) by (instance) ) * 100

upload successful

本文作者： [email protected]
本文链接： https://www.ink8s.com/2026/06/24/prometheus-监控/
版权声明： 本博客所有文章除特别声明外，均采用 MIT 许可协议。转载请注明出处！