jenkins ci/cd 部署

2025-09-23

Jenkins CI/CD 部署
- Linux Jenkins 安装
- Linux Jenkins CI/CD
- Linux Jenkins K8S CI/CD
- K8S Jenkins 安装
- K8S Jenkins CI/CD

1.0 Linux Jenkins 安装

在Linux主机环境下安装Jenkins服务

1.1 Jenkins 官方下载

war 包官方下载地址:
- https://www.jenkins.io/download/
- https://get.jenkins.io/war-stable/

Java 支持的政策

1.2 Jenkins 依赖配置

JDK 21 配置

# tar -xf jdk-21.0.10_linux-x64_bin.tar.gz
# mv jdk-21.0.10 /usr/local/
# ln -s /usr/local/jdk-21.0.10/ /usr/local/jdk
# cat /etc/profile.d/jdk.sh 
export JAVA_HOME=/usr/local/jdk
export PATH=$PATH:$JAVA_HOME/bin
export JENKINS_HOME=/data/apps/jenkins/jenkins-work

# source /etc/profile.d/jdk.sh

JDK 1.8 配置

# cat /etc/profile.d/jdk.sh 
export JAVA_HOME=/usr/local/jdk
export PATH=$PATH:$JAVA_HOME/bin
export JRE_HOME=$JAVA_HOME/jre
export JENKINS_HOME=/data/apps/jenkins/jenkins-work

# source /etc/profile.d/jdk.sh

maven 3.8.5 配置

# tar -xf apache-maven-3.8.5-bin.tar.gz
# mv apache-maven-3.8.5 /usr/local/
# ln -s /usr/local/apache-maven-3.8.5/ /usr/local/maven
# cat /etc/profile.d/maven.sh 
export MAVEN_HOME=/usr/local/maven
export PATH=${PATH}:${MAVEN_HOME}/bin

source /etc/profile.d/maven.sh

1.3 Jenkins 安装启动

安装

# mkdir -p /data/apps/jenkins/jenkins-work
# wget -P /data/apps/jenkins/ https://get.jenkins.io/war-stable/2.361.1/jenkins.war
# chmod +x /data/apps/jenkins/jenkins.war
// 准备 tls 证书，例如：jenkins.ink8s.com.jks

Jenkins 启动项

# cat /usr/lib/systemd/system/jenkins.service 
[Unit]
Description=Jenkins
After=network.target

[Service]
Type=simple
EnvironmentFile=/data/apps/jenkins/jenkins
ExecStart=/usr/local/jdk/bin/java -jar /data/apps/jenkins/jenkins.war \
          --logfile=/data/apps/jenkins/jenkins.log \
          --httpPort=80 --httpsPort=443 \
          --httpsKeyStore=/data/apps/jenkins/jenkins.ink8s.com.jks  \
          --httpsKeyStorePassword=xxxxxxxx
Restart=always
ExecStop=/usr/bin/kill -s QUIT $MAINPID
User=root
RestartSec=20
TimeoutStopSec=0
SendSIGKILL=on

[Install]
WantedBy=multi-user.target

# java -jar /data/apps/jenkins/jenkins.war --help                # 查看帮助
# systemctl enable jenkins
# systemctl start jenkins

1.4 Jenkins 页面配置

1、配置解析,浏览器访问: https://jenkins.ink8s.com/
2、自定义Jenkins: 选择插件来安装(不安装任何插件)
3、创建第一个管理员用户
- 用户名: admin
- 密码: ink8s.com
- 确认密码: ink8s.com
- 全名: admin
- 电子邮件: admin@ink8s.com

1.5 Jenkins 插件安装

更新插件下载源

// 更新插件之前,将国外下载源改为国内

# cd /root/.jenkins/updates
# sed -i 's/https:\/\/updates.jenkins.io\/download/https:\/\/mirrors.tuna.tsinghua.edu.cn\/jenkins/g' default.json
# sed -i 's/https:\/\/www.google.com/https:\/\/www.baidu.com/g' default.json
// 重启

常用插件:
- blue ocean (流水线可视化经典版界面)
- Localization (语言)
- Git (拉取代码)
- Git Parameter (Git参数化构建)
- Eclipse Temurin(JDK Tool) (选择 jdk 版本进行构建)
- Pipeline (流水线)
- Kubernetes (连接Kubernetes动态创建Slave代理)
- Config File Provider (存储配置文件)
- Extended Choice Parameter (扩展选择框参数，支持多选)
- Pipeline Stage View (阶段视图)
手动上传插件(上传 .hpi 文件方式来安装插件)
- jenkins 官网下载插件: https://plugins.jenkins.io/
- 清华源下载插件: https://mirrors.tuna.tsinghua.edu.cn/jenkins/plugins/git/latest/
- 上传 .phi 插件: 系统设置 –> 插件管理 –> 高级
离线安装插件:
1. 将其它机器上已经安装好的 plugins 目录打包,拷贝到需要安装插件的 jenkins 机器上
2. war 包安装的 jenkins, 默认 plugins 在 /root/.jenkins/

2.0 Linux Jenkins CI/CD

基于Linux环境的Jenkins部署java应用到Linux主机
提前配置Jenkins主机与java应用主机的ssh-key互通
java应用主机应用目录提前创建

2.1 Jenkins 构建前环境配置

2.1.1 JDK1.8 配置

Jenkins安装依赖的是jdk21。但众多java后端代码编译依然依赖jdk1.8,所以这里单独配置一个jdk1.8。
将jdk1.8放到指定目录然后在Jenkins页面进行JDK配置,用于在Pipeline中引用。

upload successful

2.1.2 mvn 编译参数说明

# mvn clean package -Dmaven.test.skip=true -P prod -am -amd -pl ${build_options}    # 打包指定模块,同时打包依赖于所指定模块的模块。常用!
    -h                            # 帮助
    -P                            # 指定编译环境
    -am,--als-make                # 同时打包所指定模块的依赖模块
    -adm,--also-make-dependents   # 同时打包依赖于所指定模块的模块
    -pl,--projects <arg>          # 打包指定模块,逗号分隔 module1,module2,module3,不会打包指定模块依赖包
    -rf,--resume-from <arg>       # 从所指定模块顺序开始打包
    clean                         # 删除之前构建生成的所有文件。完全重新构建
# mvn clean package -Dmaven.test.ksip=true -P prod    # 打包所有模块

2.1.3 jenkins 配置gitlab验证

系统管理 –> 凭据管理 –> 全局凭据 –> Add Credentials:
- 范围: 全局(Jenkins,nodes,items,all child items, etc)
- 用户名: Gitlab 有权限拉取代码的用户
- 密码: Gitlab 密码
- ID: 不填写(自动生成)
- 描述: GITLAB_AUTH(根据自己需求填写描述)

upload successful

2.1.4 软件包安装

Jenkins 本地拉取代码需要使用Git命令

git 安装

// Git 官网: https://git-scm.com/

# wget https://mirrors.edge.kernel.org/pub/software/scm/git/git-2.39.0.tar.gz --no-check-certificate
# tar -xf git-2.39.0.tar.gz 
# cd git-2.39.0/
# yum -y install curl-devel expat-devel gettext-devel openssl-devel zlib-devel gcc perl-ExtUtils-MakeMaker
# make prefix=/usr/local/git all
# make prefix=/usr/local/git install
# rm -f /usr/bin/git
# ln -s /usr/local/git/bin/git /usr/bin/git
# /usr/bin/git --version

2.1.5 nodejs 配置

Jenkins 前端编译以来 nodejs

nodejs 配置

# tar -xf node-v16.18.0-linux-x64.tar.xz
# mv node-v16.18.0-linux-x64 /usr/local/lib/
# ln -s /usr/local/lib/node-v16.18.0-linux-x64/ /usr/local/lib/nodejs
# cat /etc/profile.d/nodejs18.sh 
#!/bin/bash
export NODEJS=/usr/local/lib/nodejs
export PATH=$NODEJS/bin:$PATH

# source /etc/profile.d/nodejs18.sh

2.2 Pipeline 前后端流水线

2.2.1 业务主机环境配置

jar包启动以来jdk1.8需要配置

systemd 配置举例

// 每个模块的启动项都需要实现配置并开机自启
# /usr/lib/systemd/system/ink8s-merchant-api.service
[Unit]
Description=ink8s end
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
WorkingDirectory=/data/apps/
ExecStart=/usr/bin/sh -c 'exec /usr/local/jdk/bin/java -jar /data/apps/ink8s-merchant-api-1.0.0-SNAPSHOT.jar --spring.profiles.active=test'
LimitNOFILE=65536
KillMode=process
Restart=on-failure

[Install]
WantedBy=multi-user.target

# systemctl daemon-reload
# systemctl enable ink8s-merchant-api

2.2.2 java 后端多模块并发部署

CD 部分采用并发部署
首次构建需要构建2次才能部署成功

Pipeline

pipeline {
  agent any
  parameters {
    gitParameter(
      name: 'BRANCH', 
      type: 'PT_BRANCH',
      branchFilter: '.*', 
      defaultValue: 'origin/master', 
      selectedValue: 'NONE',
      sortMode: 'NONE',
      tagFilter: '*', 
      quickFilterEnabled: true,
      description: '选择要构建的分支')
    extendedChoice(
      description: '选择构建模块', 
      multiSelectDelimiter: ',', 
      name: 'build_options', 
      quoteValue: true, 
      saveJSONParameterToFile: false, 
      type: 'PT_MULTI_SELECT', 
      value: 'ink8s-merchant-api,ink8s-admin-api,ink8s-warehouse-api,ink8s-warehouse-third-api,ink8s-platform-api', 
      visibleItemCount: 5)
    choice (choices: ['no', 'yes'], description: '是否部署所有服务', name: 'DEPLOY_ALL')
  } // parameters end

  environment {
    gitlab_domain = "gitlab.ink8s.com"
    gitlab_item_groups = "ops_code"
    gitlab_code = "ink8s_code"
    gitlab_address = "https://${gitlab_domain}/${gitlab_item_groups}/${gitlab_code}.git"
    gitlab_auth = "9e92ff5d-7f89-4084-bc18-571d90917469"
    host01 = "10.0.0.10"
    dest_opt_dir = '/opt'
    dest_app_dir = "/data/apps"
    app_version = "1.0.0-SNAPSHOT"
  }

  options {
    buildDiscarder logRotator(
      daysToKeepStr: '5', 
      numToKeepStr: '6'
    )
  }

  stages {
    stage('拉取代码') {
      steps {
        checkout([
          $class: 'GitSCM', 
          branches: [[name: '${BRANCH}']], 
          extensions: [], 
          userRemoteConfigs: [[
            credentialsId: "${gitlab_auth}", 
            url: "${gitlab_address}"
          ]]
        ])    
      } // steps end
    } // stage end

    stage('检查模块') {
      steps {
        sh '''#!/bin/bash
          # 服务的模块
          Service_Module=(ink8s-merchant-api ink8s-admin-api ink8s-warehouse-api ink8s-warehouse-third-api ink8s-platform-api)
          # 更新的模块
          Update_Module="$(echo ${build_options})"
          echo "${Update_Module}"

          # 确认哪一些模块更新
          for UM in ${Update_Module}; do
            for SM in ${Service_Module[@]}; do
              if [[ "${UM}" =~ "${SM}" ]]; then
                echo "${SM}=ok" >> var.txt
              fi
            done
          done
        '''
        // 制作自定义变量
        script {
          env.MERCHANTAPI = sh(returnStdout: true, script: "grep 'ink8s-merchant-api' var.txt | awk -F '=' '{print \$2}'").trim()
          env.ADMINAPI = sh(returnStdout: true, script: "grep 'ink8s-admin-api' var.txt | awk -F '=' '{print \$2}'").trim()
          env.WAREHOUSEAPI = sh(returnStdout: true, script: "grep 'ink8s-warehouse-api' var.txt | awk -F '=' '{print \$2}'").trim()
          env.WAREHOUSETHIRDAPI = sh(returnStdout: true, script: "grep 'ink8s-warehouse-third-api' var.txt | awk -F '=' '{print \$2}'").trim()
          env.PLATFORMAPI = sh(returnStdout: true, script: "grep 'ink8s-platform-api' var.txt | awk -F '=' '{print \$2}'").trim()
        }
      }
    } // stage end

    stage('更新的模块'){
      steps {
        sh 'echo "merchant-api: ${MERCHANTAPI}"'
        sh 'echo "admin-api: ${ADMINAPI}"'
        sh 'echo "warehouse-api: ${WAREHOUSEAPI}"'
        sh 'echo "warehouse-third-api: ${WAREHOUSETHIRDAPI}"'
        sh 'echo "platform-api: ${PLATFORMAPI}"'
        sh 'echo > var.txt'
      }
    }

  // 编译代码
  stage('编译代码'){
    tools { jdk 'jdk1.8' }
    steps {
      sh '''
        /usr/local/maven/bin/mvn package -Dmaven.test.skip=true -pl ${build_options} -am -P test
      '''
    } // steps
  } // stage end

  // 服务部署
  stage('服务部署'){
    parallel {
      stage('部署 merchant-api'){
        when { expression {MERCHANTAPI == "ok" || DEPLOY_ALL == "yes" } }
        environment{ APP_NAME = "ink8s-merchant-api" }
        steps {
          sh '''
            cd $(find ./ -type d -name "${APP_NAME}")
            pwd
            jar_dir_pkg=$(ls target/*.jar)
            ssh root@${host01} "mkdir -p ${dest_opt_dir}/${APP_NAME}-${BUILD_ID}"
            scp ${jar_dir_pkg} root@${host01}:${dest_opt_dir}/${APP_NAME}-${BUILD_ID}/
            ssh root@${host01} "systemctl stop ${APP_NAME}"
            ssh root@${host01} "rm -f ${dest_app_dir}/${APP_NAME}/${APP_NAME}-${app_version}.jar"
            ssh root@${host01} "mkdir -p ${dest_app_dir}/${APP_NAME}/"
            ssh root@${host01} "ln -s ${dest_opt_dir}/${APP_NAME}-${BUILD_ID}/${APP_NAME}-${app_version}.jar ${dest_app_dir}/${APP_NAME}/${APP_NAME}-${app_version}.jar"
            ssh root@${host01} "systemctl start ${APP_NAME}"
          '''
        } // steps
      } // stage end

      stage('部署 admin-api'){
        when { expression {ADMINAPI == "ok" || DEPLOY_ALL == "yes" } }
        environment{ APP_NAME = "ink8s-admin-api" }
        steps {
          sh '''
            cd $(find ./ -type d -name "${APP_NAME}")
            pwd
            jar_dir_pkg=$(ls target/*.jar)
            ssh root@${host01} "mkdir -p ${dest_opt_dir}/${APP_NAME}-${BUILD_ID}"
            scp ${jar_dir_pkg} root@${host01}:${dest_opt_dir}/${APP_NAME}-${BUILD_ID}/
            ssh root@${host01} "systemctl stop ${APP_NAME}"
            ssh root@${host01} "rm -f ${dest_app_dir}/${APP_NAME}/${APP_NAME}-${app_version}.jar"
            ssh root@${host01} "mkdir -p ${dest_app_dir}/${APP_NAME}/"
            ssh root@${host01} "ln -s ${dest_opt_dir}/${APP_NAME}-${BUILD_ID}/${APP_NAME}-${app_version}.jar ${dest_app_dir}/${APP_NAME}/${APP_NAME}-${app_version}.jar"
            ssh root@${host01} "systemctl start ${APP_NAME}"
          '''
        } // steps
      } // stage end

      stage('部署 warehouse-api'){
        when { expression {WAREHOUSEAPI == "ok" || DEPLOY_ALL == "yes" } }
        environment{ APP_NAME = "ink8s-warehouse-api" }
        steps {
          sh '''
            cd $(find ./ -type d -name "${APP_NAME}")
            pwd
            jar_dir_pkg=$(ls target/*.jar)
            ssh root@${host01} "mkdir -p ${dest_opt_dir}/${APP_NAME}-${BUILD_ID}"
            scp ${jar_dir_pkg} root@${host01}:${dest_opt_dir}/${APP_NAME}-${BUILD_ID}/
            ssh root@${host01} "systemctl stop ${APP_NAME}"
            ssh root@${host01} "rm -f ${dest_app_dir}/${APP_NAME}/${APP_NAME}-${app_version}.jar"
            ssh root@${host01} "mkdir -p ${dest_app_dir}/${APP_NAME}/"
            ssh root@${host01} "ln -s ${dest_opt_dir}/${APP_NAME}-${BUILD_ID}/${APP_NAME}-${app_version}.jar ${dest_app_dir}/${APP_NAME}/${APP_NAME}-${app_version}.jar"
            ssh root@${host01} "systemctl start ${APP_NAME}"
          '''
        } // steps
      } // stage end

      stage('部署 warehouse-third-api'){
        when { expression {WAREHOUSETHIRDAPI == "ok" || DEPLOY_ALL == "yes" } }
        environment{ APP_NAME = "ink8s-warehouse-third-api" }
        steps {
          sh '''
            cd $(find ./ -type d -name "${APP_NAME}")
            pwd
            jar_dir_pkg=$(ls target/*.jar)
            ssh root@${host01} "mkdir -p ${dest_opt_dir}/${APP_NAME}-${BUILD_ID}"
            scp ${jar_dir_pkg} root@${host01}:${dest_opt_dir}/${APP_NAME}-${BUILD_ID}/
            ssh root@${host01} "systemctl stop ${APP_NAME}"
            ssh root@${host01} "rm -f ${dest_app_dir}/${APP_NAME}/${APP_NAME}-${app_version}.jar"
            ssh root@${host01} "mkdir -p ${dest_app_dir}/${APP_NAME}/"
            ssh root@${host01} "ln -s ${dest_opt_dir}/${APP_NAME}-${BUILD_ID}/${APP_NAME}-${app_version}.jar ${dest_app_dir}/${APP_NAME}/${APP_NAME}-${app_version}.jar"
            ssh root@${host01} "systemctl start ${APP_NAME}"
          '''
        } // steps
      } // stage end

      stage('部署 platform-api'){
        when { expression {PLATFORMAPI == "ok" || DEPLOY_ALL == "yes" } }
        environment{ APP_NAME = "ink8s-platform-api" }
        steps {
          sh '''
            cd $(find ./ -type d -name "${APP_NAME}")
            pwd
            jar_dir_pkg=$(ls target/*.jar)
            ssh root@${host01} "mkdir -p ${dest_opt_dir}/${APP_NAME}-${BUILD_ID}"
            scp ${jar_dir_pkg} root@${host01}:${dest_opt_dir}/${APP_NAME}-${BUILD_ID}/
            ssh root@${host01} "systemctl stop ${APP_NAME}"
            ssh root@${host01} "rm -f ${dest_app_dir}/${APP_NAME}/${APP_NAME}-${app_version}.jar"
            ssh root@${host01} "mkdir -p ${dest_app_dir}/${APP_NAME}/"
            ssh root@${host01} "ln -s ${dest_opt_dir}/${APP_NAME}-${BUILD_ID}/${APP_NAME}-${app_version}.jar ${dest_app_dir}/${APP_NAME}/${APP_NAME}-${app_version}.jar"
            ssh root@${host01} "systemctl start ${APP_NAME}"
          '''
        } // steps
      } // stage end

    } // parallel end

  } // stage end
  
  
  } // stages
} // pipeline

2.2.3 vue 前端构建

Pipeline

pipeline {
    agent any

    parameters {
        gitParameter branch: '', branchFilter: '.*', 
                     defaultValue: 'origin/master', 
                     description: '选择要发布的分支', name: 'Branch', 
                     quickFilterEnabled: false, selectedValue: 'NONE', 
                     sortMode: 'NONE', tagFilter: '*', type: 'GitParameterDefinition'
    }

    environment {
        domian_gitlab = "gitlab.ink8s.com"
        project = "ops_code"
        app_name = "ink8s-front-merchant"
        gitlab_addr = "https://${domian_gitlab}/${project}/${app_name}.git"
        gitlab_auth = "9e92ff5d-7f89-4084-bc18-571d90917469"
        host01 = "10.0.0.10"
        app_dir = "/data/apps"
        web_dir = "ink8s-front"
    }

    stages{
        stage('获取代码'){
            steps{
                checkout([$class: 'GitSCM', 
                branches: [[name: '${Branch}']], 
                extensions: [], 
                userRemoteConfigs: [[credentialsId: "${gitlab_auth}", 
                url: "${gitlab_addr}"]]])    
            }
        }
        stage('编译代码'){
            steps{
                sh """
                   /usr/local/lib/nodejs/bin/npm install --unsafe-perm
                   /usr/local/lib/nodejs/bin/npm run build:stage
                """
            }
        }
        stage('打包代码'){
            steps{
                sh """
                   /usr/bin/tar -zcf /opt/${web_dir}_${env.BUILD_ID}.tar.gz ./dist
                """
            }
        }
        stage('部署代码') {
            steps {
                sh """
                    scp /opt/${web_dir}_${env.BUILD_ID}.tar.gz root@${host01}:/opt/
                    ssh root@${host01} 'mkdir -p /opt/${web_dir}_${env.BUILD_ID}'
                    ssh root@${host01} 'tar -xf /opt/${web_dir}_${env.BUILD_ID}.tar.gz -C /opt/${web_dir}_${env.BUILD_ID}/'
                    ssh root@${host01} '[ -d ${app_dir}/${web_dir} ] && unlink ${app_dir}/${web_dir}/dist'
                    ssh root@${host01} '/bin/ln -s /opt/${web_dir}_${env.BUILD_ID}/dist ${app_dir}/${web_dir}/'
                  """
                }
            }
        }
    }

3.0 Linux Jenkins K8S CI/CD

基于Linux环境的Jenkins部署java应用到K8S环境
jdk、nodejs、插件等基础环境延用(1.0|2.0)部分

3.1 Jenkins 构建前环境配置

3.1.1 创建应用部署空间等信息

// 创建空间
# kubectl create ns ink8s-item

// k8s 的 harbor 仓库认证创建
# kubectl create secret docker-registry harbor-auth \
--docker-server=harbor.ink8s.com \
--docker-username=admin \
--docker-password=ink8s.com \
-n ink8s-item

// 所有K8S节点登录 harbor 仓库
# nerctl login harbor.ink8s.com
  用户名: admin
  密码: ink8s.com

3.1.2 K8S应用部署权限

将 kubectl 和 .kube 目录 scp 到 jenkins 服务器

// 将 k8s master 服务器的 kubectl 文件拷贝到 jenkins 服务器
# scp /usr/bin/kubectl [email protected]:/usr/bin/

// 将 k8s master 的角色权限拷贝到 jenkins 服务器，用于 apply 部署 deploy.yaml 文件
# scp -r /root/.kube [email protected]:/root/

3.1.3 k8s接入外部服务

3.2 Pipeline 前后端流水线

3.2.1 后端模块部署文件

此处仅采用一个模块的相关文件作为参考，实际应是每个构建模块都需配置相关文件

3.2.1.1 Dockerfile 文件

Dockerfile 文件需要存放于 ink8s/ink8s-admin-api/Dockerfile 路径（此路径为gitlab代码存放路径）

Dockerfile

# FROM openjdk:8-jre-alpine
FROM harbor.ink8s.com/ops-jenkins/openjdk:8-jre

COPY ./target/*.jar /ink8s-admin-api-1.0.0-SNAPSHOT.jar
COPY ./entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh

EXPOSE 9196
CMD ["/bin/sh", "-c", "/entrypoint.sh"]

3.2.1.2 entrypoint.sh 启动脚本文件

entrypoint.sh 文件需要存放于 ink8s/ink8s-admin-api/entrypoint.sh 路径（此路径为gitlab代码存放路径）

entrypoint.sh

# 设定端口，默认不传参则为9196端口
# PARAMS="--server.port=${Server_Port:-9196}"

# JVM堆内存设定
JAVA_OPTS="-Xms${XMS_OPTS:-128m} -Xmx${XMX_OPTS:-2048m}"

# 启动环境
INK8S_START_EVN="--spring.profiles.active=test"

# 启动命令（指定jvm堆内存选项、jar包，最后跟上params参数）
# java ${JAVA_OPTS} -jar /ink8s-admin-api-1.0.0-SNAPSHOT.jar ${INK8S_START_EVN}
java ${JAVA_OPTS} -jar /ink8s-admin-api-1.0.0-SNAPSHOT.jar ${INK8S_START_EVN}

3.2.1.3 deploy.yaml 文件

deploy.yaml 文件需要存放与 ink8s/nk8s-admin-api/deploy.yaml 路径（此路径为gitlab代码存放路径）

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ink8s-admin-api
  namespace: {NAME_SPACE}
spec:
  replicas: {REPLICAS_NUM}
  selector:
    matchLabels:
      app: admin-api
  template:
    metadata:
      labels:
        app: admin-api
    spec:
      imagePullSecrets:
      - name: harbor-auth
      containers:
      - name: admin-api
        image: {IMAGE_NAME}
        ports:
        - containerPort: 9196
        livenessProbe:
          tcpSocket:
            port: 9196
          initialDelaySeconds: 60
          periodSeconds: 10
          timeoutSeconds: 10

---
apiVersion: v1
kind: Service
metadata:
  name: ink8s-admin-api-svc
  namespace: {NAME_SPACE}
spec:
  selector:
    app: admin-api
  ports:
  - port: 80
    targetPort: 9196
    protocol: TCP

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ink8s-admin-api-ing
  namespace: {NAME_SPACE}
spec:
  ingressClassName: "nginx"
  rules:
  - host: "{host}"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: ink8s-admin-api-svc
            port:
              number: 80

3.2.1.4 中间件连接配置修改

MySQL、Redis等等
路径：ink8s/ink8s-admin-api/src/main/resources/application-test.yaml（此路径为gitlab代码存放路径）

spring:
  datasource:
    dynamic:
      primary: ink8s_v3
      datasource:
        ink8s_v3:
          url: jdbc:mysql://ink8s-mysql.ink8s-item:3306/ink8s_v3?useUnicode=true&characterEncoding=utf-8&useSSL=false&allowMultiQueries=true&serverTimezone=Asia/Shanghai&zeroDateTimeBehavior=convertToNull
          username: root
          password: ink8s.com
  redis:
    database: 10
    host: ink8s-redis.ink8s-item
    port: 6379
    password: ink8s.com
    lettuce:
      pool:
        max-wait: -1ms
        max-idle: 10
        min-idle: 0
        max-active: 10


...

3.2.1.5 Jenkinsfile 文件

Pipeline 仅展示3个模块,可供参考
路径: ink8s/ink8s-ops/Jenkinsfile（此路径为gitlab代码存放路径）

Jenkinsfile

pipeline {
  agent any
  parameters {
    gitParameter(
      name: 'BRANCH', 
      type: 'PT_BRANCH',
      branchFilter: '.*', 
      defaultValue: 'origin/master', 
      selectedValue: 'NONE',
      sortMode: 'NONE',
      tagFilter: '*', 
      quickFilterEnabled: true,
      description: '选择要构建的分支')
    extendedChoice(
      description: '选择构建模块', 
      multiSelectDelimiter: ',', 
      name: 'build_options', 
      quoteValue: true, 
      saveJSONParameterToFile: false, 
      type: 'PT_MULTI_SELECT', 
      // 仅验证这三个模块: ink8s-merchant-api、ink8s-admin-api、ink8s-platform-api
      value: 'ink8s-merchant-api,ink8s-admin-api,ink8s-platform-api', 
      visibleItemCount: 3)
    choice (choices: ['1', '3'], description: '副本数', name: 'REPLICAS_NUM')
    choice (choices: ['ink8s-item'], description: '命名空间', name: 'NAME_SPACE')
    choice (choices: ['no', 'yes'], description: '是否部署所有服务', name: 'DEPLOY_ALL')
  } // parameters end

  environment {
    // 域名
    gitlab_domain = "gitlab.ink8s.com"
    harbor_domian = "harbor.ink8s.com"
    // 验证
    gitlab_auth = "9e92ff5d-7f89-4084-bc18-571d90917469"
    harbor_auth = "86fe9fc0-c8b3-4cbe-9da8-976854179c6c"
    // 项目地址
    gitlab_item_groups = "ops_code"
    harbor_groups = "ink8s_java_service"
    gitlab_code = "ink8s"
    gitlab_address = "https://${gitlab_domain}/${gitlab_item_groups}/${gitlab_code}.git"
  }

  options {
    buildDiscarder logRotator(
      daysToKeepStr: '5', 
      numToKeepStr: '6'
    )
  }

  stages {
    stage('拉取代码') {
      steps {
        checkout([
          $class: 'GitSCM', 
          branches: [[name: '${BRANCH}']], 
          extensions: [], 
          userRemoteConfigs: [[
            credentialsId: "${gitlab_auth}", 
            url: "${gitlab_address}"
          ]]
        ])    
      } // steps end
    } // stage end

    stage('检查模块') {
      steps {
        sh '''#!/bin/bash
          # 服务的模块
          Service_Module=(ink8s-merchant-api ink8s-admin-api ink8s-platform-api)
          # 更新的模块
          Update_Module="$(echo ${build_options})"
          echo "${Update_Module}"

          # 确认哪一些模块更新
          for UM in ${Update_Module}; do
            for SM in ${Service_Module[@]}; do
              if [[ "${UM}" =~ "${SM}" ]]; then
                echo "${SM}=ok" >> var.txt
              fi
            done
          done
        '''
        // 制作自定义变量
        script {
          env.MERCHANTAPI = sh(returnStdout: true, script: "grep 'ink8s-merchant-api' var.txt | awk -F '=' '{print \$2}'").trim()
          env.ADMINAPI = sh(returnStdout: true, script: "grep 'ink8s-admin-api' var.txt | awk -F '=' '{print \$2}'").trim()
          env.PLATFORMAPI = sh(returnStdout: true, script: "grep 'ink8s-platform-api' var.txt | awk -F '=' '{print \$2}'").trim()
        } // script end
      } // steps end
    } // 检查模块 stage end

    stage('更新的模块'){
      steps {
        sh 'echo "merchant-api: ${MERCHANTAPI}"'
        sh 'echo "admin-api: ${ADMINAPI}"'
        sh 'echo "platform-api: ${PLATFORMAPI}"'
        sh 'echo > var.txt'
      }
    }

    // 编译代码
    stage('编译代码'){
      tools { jdk 'jdk1.8' }
      steps {
        sh '''
          /usr/local/maven/bin/mvn package -Dmaven.test.skip=true -pl ${build_options} -am -P test
        '''
      } // steps
    } // 编译代码 stage end

    // 镜像版本
    stage('镜像版本'){
      steps {
          script {
            env.COMMIT_ID = sh(returnStdout: true, script: "git log -n1 --pretty=format:'%h'").trim()
            env.BUILD_TIME = sh(returnStdout: true, script: "date +%Y%m%d_%H%M%S").trim()
            env.IMAGE_TAG = COMMIT_ID + "_" +  BUILD_TIME
          }
          sh 'echo "镜像的TAG: ${IMAGE_TAG}"'
      }
    } // 镜像版本 stage end

    // 镜像打包
    stage('镜像打包'){
      parallel {
        stage('镜像 merchant-api'){
          when { expression {MERCHANTAPI == "ok" || DEPLOY_ALL == "yes" } }
          environment{ APP_NAME = "ink8s-merchant-api" }
          steps {
            sh '''
              cd $(find ./ -type d -name "${APP_NAME}")
              pwd
              nerdctl build -t ${harbor_domian}/${harbor_groups}/${APP_NAME}:${IMAGE_TAG} .
              nerdctl push ${harbor_domian}/${harbor_groups}/${APP_NAME}:${IMAGE_TAG}
              nerdctl rmi ${harbor_domian}/${harbor_groups}/${APP_NAME}:${IMAGE_TAG}
            '''
          } // steps
        } // stage end

        stage('镜像 admin-api'){
          when { expression {ADMINAPI == "ok" || DEPLOY_ALL == "yes" } }
          environment{ APP_NAME = "ink8s-admin-api" }
          steps {
            sh '''
              cd $(find ./ -type d -name "${APP_NAME}")
              pwd
              nerdctl build -t ${harbor_domian}/${harbor_groups}/${APP_NAME}:${IMAGE_TAG} .
              nerdctl push ${harbor_domian}/${harbor_groups}/${APP_NAME}:${IMAGE_TAG}
              nerdctl rmi ${harbor_domian}/${harbor_groups}/${APP_NAME}:${IMAGE_TAG}
            '''
          } // steps
        } // stage end

        stage('镜像 platform-api'){
          when { expression {PLATFORMAPI == "ok" || DEPLOY_ALL == "yes" } }
          environment{ APP_NAME = "ink8s-platform-api" }
          steps {
            sh '''
              cd $(find ./ -type d -name "${APP_NAME}")
              pwd
              nerdctl build -t ${harbor_domian}/${harbor_groups}/${APP_NAME}:${IMAGE_TAG} .
              nerdctl push ${harbor_domian}/${harbor_groups}/${APP_NAME}:${IMAGE_TAG}
              nerdctl rmi ${harbor_domian}/${harbor_groups}/${APP_NAME}:${IMAGE_TAG}
            '''
          } // steps
        } // stage end

      } // parallel end
    } // 镜像打包 stage end

    // 服务交付
    stage('服务交付') {
      parallel {
        stage('交付 admin-api') {
          when { expression {ADMINAPI == "ok" || DEPLOY_ALL == "yes" } }
          environment { 
            APP_NAME = "ink8s-admin-api" 
            ING_HOSTS = "ink8s-admin-api.ink8s.com"
            }
          steps {
            sh '''
              cd $(find ./ -type d -name "${APP_NAME}")
              cat deploy.yaml
              sed -i "s#{NAME_SPACE}#${NAME_SPACE}#g" deploy.yaml
              sed -i "s#{REPLICAS_NUM}#${REPLICAS_NUM}#g" deploy.yaml
              sed -i "s#{host}#${ING_HOSTS}#g" deploy.yaml
              sed -i "s#{IMAGE_NAME}#${harbor_domian}/${harbor_groups}/${APP_NAME}:${IMAGE_TAG}#g" deploy.yaml
              cat deploy.yaml
            '''
            sh '''
              cd $(find ./ -type d -name "${APP_NAME}")
              kubectl apply -f deploy.yaml -n ${NAME_SPACE}
            '''
          } // steps end
        } // stage end

        stage('交付 merchant-api') {
          when { expression {MERCHANTAPI == "ok" || DEPLOY_ALL == "yes" } }
          environment { 
            APP_NAME = "ink8s-merchant-api" 
            ING_HOSTS = "ink8s-merchant-api.ink8s.com"
            }
          steps {
            sh '''
              cd $(find ./ -type d -name "${APP_NAME}")
              cat deploy.yaml
              sed -i "s#{NAME_SPACE}#${NAME_SPACE}#g" deploy.yaml
              sed -i "s#{REPLICAS_NUM}#${REPLICAS_NUM}#g" deploy.yaml
              sed -i "s#{host}#${ING_HOSTS}#g" deploy.yaml
              sed -i "s#{IMAGE_NAME}#${harbor_domian}/${harbor_groups}/${APP_NAME}:${IMAGE_TAG}#g" deploy.yaml
              cat deploy.yaml
            '''
            sh '''
              cd $(find ./ -type d -name "${APP_NAME}")
              kubectl apply -f deploy.yaml -n ${NAME_SPACE}
            '''
          } // steps end
        } // stage end

        stage('交付 platform-api') {
          when { expression {PLATFORMAPI == "ok" || DEPLOY_ALL == "yes" } }
          environment { 
            APP_NAME = "ink8s-platform-api" 
            ING_HOSTS = "ink8s-platform-api.ink8s.com"
            }
          steps {
            sh '''
              cd $(find ./ -type d -name "${APP_NAME}")
              cat deploy.yaml
              sed -i "s#{NAME_SPACE}#${NAME_SPACE}#g" deploy.yaml
              sed -i "s#{REPLICAS_NUM}#${REPLICAS_NUM}#g" deploy.yaml
              sed -i "s#{host}#${ING_HOSTS}#g" deploy.yaml
              sed -i "s#{IMAGE_NAME}#${harbor_domian}/${harbor_groups}/${APP_NAME}:${IMAGE_TAG}#g" deploy.yaml
              cat deploy.yaml
            '''
            sh '''
              cd $(find ./ -type d -name "${APP_NAME}")
              kubectl apply -f deploy.yaml -n ${NAME_SPACE}
            '''
          }
        }

      } // parallel end
    } // stage 服务交付
  } // stages
} // pipeline

3.2.2 前端部署文件

3.2.2.1 Dockerfile

FROM harbor.ink8s.com/ops_apps_tools/nginx:1.20

COPY ./dist/ /code
COPY ./test.ink8s.com.conf /etc/nginx/conf.d/test.ink8s.com.conf
RUN rm -f /etc/nginx/conf.d/default.conf

3.2.2.2 nginx配置文件

根据实际情况配置nginx配置文件，这里仅供参考

server {
    listen  80;
    server_name test.ink8s.com;

    if ($http_user_agent ~* "scrapy|httpclient|python|bench") {
       return 403;
    }

    location / {
        root /code/;
        index index.html index.htm;
        try_files $uri $uri/ /index.html;
    }
}

3.2.2.3 deploy 文件

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ink8s-www
  namespace: {NAME_SPACE}
spec:
  replicas: {REPLICAS_NUM}
  selector:
    matchLabels:
      app: ink8s
  template:
    metadata:
      labels:
        app: ink8s
    spec:
      imagePullSecrets:
      - name: harbor-auth

      containers:
      - name: ink8s-www
        image: {IMAGE_NAME}
        ports:
        - containerPort: 80
        readinessProbe:         # 就绪探针，不就绪则从负载均衡移除
          tcpSocket:
            port: 80
          initialDelaySeconds: 60
          periodSeconds: 10
          timeoutSeconds: 10

        livenessProbe:          # 存活探针，不存活会重启
          tcpSocket:
            port: 80
          initialDelaySeconds: 60
          periodSeconds: 10
          timeoutSeconds: 10

---
apiVersion: v1
kind: Service
metadata:
  name: ink8s-svc
  namespace: {NAME_SPACE}
spec:
  selector:
    app: ink8s
  ports:
  - port: 80
    targetPort: 80

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ink8s-ingress
  namespace: {NAME_SPACE}
spec:
  ingressClassName: "nginx"
  tls:
  - hosts:
    - "{host}"
    secretName: test-ink8s-com
  rules:
  - host: "{host}"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: ink8s-svc
            port:
              number: 80

3.2.2.3 tls 配置

1	# kubectl create secret tls test-ink8s-com -n ink8s-item --key=test.ink8s.com.key --cert=test.ink8s.com.pem

3.2.2.4 Jenkinsfile 文件

pipeline {
  agent any
  parameters {
    gitParameter(
      name: 'BRANCH', 
      type: 'PT_BRANCH',
      branchFilter: '.*', 
      defaultValue: 'origin/master', 
      selectedValue: 'NONE',
      sortMode: 'NONE',
      tagFilter: '*', 
      quickFilterEnabled: true,
      description: '选择要构建的分支')
    choice (choices: ['1', '3'], description: '副本数', name: 'REPLICAS_NUM')
    choice (choices: ['ink8s-item'], description: '命名空间', name: 'NAME_SPACE')
    choice (choices: ['no', 'yes'], description: '是否执行回滚？', name: 'DO_ROLLBACK')
  } // parameters end

  environment {
    // 域名
    gitlab_domain = "gitlab.ink8s.com"
    harbor_domian = "harbor.ink8s.com"
    // 验证
    gitlab_auth = "9e92ff5d-7f89-4084-bc18-571d90917469"
    harbor_auth = "86fe9fc0-c8b3-4cbe-9da8-976854179c6c"
    // 项目地址
    gitlab_item_groups = "ops_code"
    harbor_groups = "ink8s_java_service"
    gitlab_code = "ink8s-front-merchant"
    gitlab_address = "https://${gitlab_domain}/${gitlab_item_groups}/${gitlab_code}.git"
  }

  options {
    buildDiscarder logRotator(
      daysToKeepStr: '5', 
      numToKeepStr: '6'
    )
  }

  stages {
    stage('拉取代码') {
      steps {
        checkout([
          $class: 'GitSCM', 
          branches: [[name: '${BRANCH}']], 
          extensions: [], 
          userRemoteConfigs: [[
            credentialsId: "${gitlab_auth}", 
            url: "${gitlab_address}"
          ]]
        ])    
      } // steps end
    } // stage end

    stage('编译代码'){
      steps{
        sh """
          /usr/local/lib/nodejs/bin/npm install --unsafe-perm
          /usr/local/lib/nodejs/bin/npm run build:stage
        """
      }
    }

    // 镜像版本
    stage('镜像版本'){
      steps {
          script {
            env.COMMIT_ID = sh(returnStdout: true, script: "git log -n1 --pretty=format:'%h'").trim()
            env.BUILD_TIME = sh(returnStdout: true, script: "date +%Y%m%d_%H%M%S").trim()
            env.IMAGE_TAG = COMMIT_ID + "_" +  BUILD_TIME
          }
          sh 'echo "镜像的TAG: ${IMAGE_TAG}"'
      }
    } // 镜像版本 stage end

    stage('ink8s 前端镜像') {
      environment { APP_NAME = "ink8s-www" }
      steps {
        sh '''
          nerdctl build -t ${harbor_domian}/${harbor_groups}/${APP_NAME}:${IMAGE_TAG} .
          nerdctl push ${harbor_domian}/${harbor_groups}/${APP_NAME}:${IMAGE_TAG}
          nerdctl rmi ${harbor_domian}/${harbor_groups}/${APP_NAME}:${IMAGE_TAG}
        '''
      }
    }


    stage('交付 ink8s 前端') {
      environment {
        APP_NAME = "ink8s-www"
        ING_HOSTS = "test.ink8s.com"
      }
      steps {
        sh '''
          cat deploy.yaml
          sed -i "s#{NAME_SPACE}#${NAME_SPACE}#g" deploy.yaml
          sed -i "s#{REPLICAS_NUM}#${REPLICAS_NUM}#g" deploy.yaml
          sed -i "s#{host}#${ING_HOSTS}#g" deploy.yaml
          sed -i "s#{IMAGE_NAME}#${harbor_domian}/${harbor_groups}/${APP_NAME}:${IMAGE_TAG}#g" deploy.yaml
        '''
        sh 'cat deploy.yaml'

        sh '''
          kubectl apply -f deploy.yaml -n ${NAME_SPACE}
        '''
      }
    }

  } // stages
} // pipeline

4.0 K8S Jenkins 安装

4.1 Jenkins 安装

// 创建空间
# kubectl create ns ops

// k8s 的 harbor 仓库认证创建
# kubectl create secret docker-registry harbor-auth \
--docker-server=harbor.ink8s.com \
--docker-username=admin \
--docker-password=ink8s.com \
-n ops

// 所有K8S节点登录 harbor 仓库
# nerctl login harbor.ink8s.com
  用户名: admin
  密码: ink8s.com

4.1.1 Jenkins 镜像下载

1
2
3

# docker pull jenkins/jenkins:2.346.3-2-lts
# docker tag jenkins/jenkins:2.346.3-2-lts harbor.ink8s.com/ops-jenkins/jenkins:2.346
# docker push harbor.ink8s.com/ops-jenkins/jenkins:2.346

4.1.2 Jenkins RBAC授权

创建 RBAC，为 Jenkins 授权

# serviceaccount
apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins
  namespace: ops

---
# clusterRole
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: jenkins
  namespace: ops
rules:
  - apiGroups: ["extensions", "apps"]
    resources: ["deployments", "ingresses"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create", "delete", "get", "list", "patch", "update", "watch"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create", "delete", "get", "list", "patch", "update", "watch"]
  - apiGroups: [""]
    resources: ["pods/log", "events"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get"]

---
# clusterrolebinding
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: jenkins
  namespace: ops
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: jenkins
subjects:
  - kind: ServiceAccount
    name: jenkins
    namespace: ops

4.1.3 部署 Jenkins

1	# kubectl create secret tls jenkins-ink8s-com --cert=jenkins.ink8s.com.pem --key=jenkins.ink8s.com.key -n ops

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: jenkins
  namespace: ops
spec:
  serviceName: "jenkins-svc"
  selector:
    matchLabels:
      app: jenkins
  template:
    metadata:
      labels:
        app: jenkins
    spec:
      serviceAccount: jenkins
      imagePullSecrets:
        - name: harbor-auth
      containers:
        - name: jenkins
          image: harbor.ink8s.com/ops-jenkins/jenkins:2.346
          imagePullPolicy: IfNotPresent
          securityContext:
            privileged: true
            runAsUser: 0
          env:
            - name: JAVA_OPTS
              value: -Duser.timezone=Asia/Shanghai
          ports:
            - name: http
              containerPort: 8080
            - name: agent
              containerPort: 50000
          resources:
            limits:
              cpu: 1500m
              memory: 2048Mi
          readinessProbe:
            httpGet:
              path: /login
              port: 8080
            initialDelaySeconds: 60
            timeoutSeconds: 5
            failureThreshold: 12
          volumeMounts:
            - name: data
              mountPath: /var/jenkins_home
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: "csi-rbd-sc"
        resources:
          requests:
            storage: 5Gi

---
apiVersion: v1
kind: Service
metadata:
  name: jenkins-svc
  namespace: ops
spec:
  clusterIP: None
  selector:
    app: jenkins
  ports:
    - name: http
      port: 8080
      targetPort: 8080
    - name: agent
      port: 50000
      targetPort: 50000

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: jenkins-ingress
  namespace: ops
spec:
  tls:
    - hosts:
        - jenkins.ink8s.com
      secretName: jenkins-ink8s-com
  ingressClassName: "nginx"
  rules:
    - host: jenkins.ink8s.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: jenkins-svc
                port:
                  name: http

4.1.4 初始化 Jenkins

配置解析并访问Jenkins: https://jenkins.ink8s.com

4.1.5 插件安装

中文插件: localization: chinese
git插件: git、gitlab
sonar插件: sonarqube scanner
pipeline插件: pipeline、stage view、blue ocean
kubernetes插件: kubernetes
kubeconfig插件 Config File Provider
扩展选择框，支持多选 Extended Choice Parameter
Git参数化构建 Git Parameter

4.2 Jenkins Slave 架构

4.2.1 架构说明

所谓 Jenkins Master/Slave 架构,及在Master上进行任务分配。然后由Slave来完成,不过Slave运行方式有2种:
1. 静态Slave: 需要固定的节点,配置其对应环境,手动注册到Master,然后执行任务,任务完成节点处于空闲等待状态
2. 动态Slave: 由Master动态创建Slave的Pod,自动注册到Master,然后执行任务,任务结束Pod自动销毁

4.2.1.1 静态Jenkins Slave

能够分担主节点上的压力,加快构建速度(所有任务都由Master执行,造成构建速度缓慢,且任务多会出现排队现象)
能够将特定的任务在特定的主机上运行(比如: 不同任务需要不同的编译环境)
缺点:
1. Master发生单点故障时,整个Jenkins都没办法使用
2. 每个Slave的环境不一样,用于完成不同项目的编译打包工作,但这些不同环境的配置管理及维护都特别困难
3. 有的Slave构建任务频繁,可能出现排队等待,而有的Slave又处于空闲状态,所以会出现资源分配严重不均衡
4. 因为每个Slave都需要一台虚拟机,当Slave空闲时,等于就空跑,资源浪费明显

4.2.1.2 动态Jenkins Slave

所谓动态Slave，就是根据任务进行动态供应和动态删除。Jenkins Master 和 Jenkins Slave 都是以Pod的形式运行在Kubernetes集群节点上，Master 运行在其中一个节点上,其配置数据存储在一个持久卷声明中。而Slave则随机运行在各个节点上，但它不会一直处于运行状态，而是根据需求动态创建并自动删除
工作流程: 当 Jenkins Master 收到构建请求,会根据标签动态创建一个Pod,该Pod就是Jenkins Slave,然后Jenkins Slave会自动注册到Jenkins Master上。当Slave运行完任务后,会自动释放,相关的Pod也会被删除
优点:
1. 高可用性: (当Jenkins Master 故障时,kubernetes会自动创建一个新的Jenkins Master容器,并将持久卷挂载至新创建的容器,保证数据不会丢失,从而实现Jenkins的高可用性)
2. 高可用扩展性: (当kubernetes集群因资源不足而导致任务长时间排队等待时,可以向集群新增节点,来缓解压力)
3. 资源分配合理: (kubernetes动态分配Slave至空闲节点,避免因单个节点资源利用率高而导致排队等待)

4.2.2 Jenkins 动态 Slave 配置

系统管理 -> 节点管理 -> Configure Clouds -> Add a New Cloud -> Kubernetes:
- 点击: Kubernetes Cloud details …
  - Kubernetes 地址: https://kubernetes.default.svc.cluster.local
  - Kubernetes 命名空间: ops # 这里表示拉起来的 pod 是在什么空间下运行
  - Jenkins 地址: http://jenkins-svc.ops.svc.cluster.local:8080
  - Jenkins 通道: jenkins-svc.ops.svc.cluster.local:50000
解析jenkins地址: # dig @10.96.0.10 jenkins-svc.ops.svc.cluster.local +short

4.2.3 测试动态 Slave

pipeline {
    agent {
        kubernetes {
            cloud 'kubernetes'
        }
    }
    stages {
        stage('Hello') {
            steps {
                echo "Hello World"
                sh 'hostname'
            }
        }
    }
}

4.3 Jenkins 动态 Slave Pod模板

jnlp 镜像是用来连接Jenkins Master以及共享Master的WORKSPACE。但该镜像并没有maven、docker、kubectl等常用命令
为此需要指定几个镜像,后期通过pipeline将不同的任务交由同一个pod的不同容器来执行

4.3.1 Maven

Dockerfile

# cd /root/ops/maven
# ls
Dockerfile  settings_docker.xml

# cat Dockerfile 
FROM maven:3.8.6-openjdk-8

ADD ./settings_docker.xml /usr/share/maven/conf/settings.xml
RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

# docker build -t harbor.ink8s.com/ops/maven:3.8.6 .
# docker push harbor.ink8s.com/ops/maven:3.8.6

settings_docker.xml

# cat settings_docker.xml        # 修改依赖地址为阿里云地址
...
  <mirrors>
    <!-- mirror
     | Specifies a repository mirror site to use instead of a given repository. The repository that
     | this mirror serves has an ID that matches the mirrorOf element of this mirror. IDs are used
     | for inheritance and direct lookup purposes, and must be unique across the set of mirrors.
     |
    <mirror>
      <id>mirrorId</id>
      <mirrorOf>repositoryId</mirrorOf>
      <name>Human Readable Name for this Mirror.</name>
      <url>http://my.repository.com/repo/path</url>
    </mirror>
     -->
<mirror>
      <id>alimaven</id>
      <name>aliyun maven</name>
      <url>http://maven.aliyun.com/nexus/content/groups/public/</url>
      <mirrorOf>central</mirrorOf>       
</mirror>
  </mirrors>
...

4.3.2 NodeJs

Dockerfile

# cp /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/epel.repo ./
# cat Dockerfile 
FROM centos:7
RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
RUN curl --silent --location https://rpm.nodesource.com/setup_14.x |bash -
RUN yum install nodejs gcc-c++ make vim -y && \
    yum clean all

# docker build -t harbor.ink8s.com/ops/nodejs:14.20 .
# docker push harbor.ink8s.com/ops/nodejs:14.20

4.3.3 Docker

镜像 build 时的 Dockerfile 和 entrypoint.sh 2个文件在 gitlab 代码中， build 时会用到

docker

1
2
3

# docker pull docker:20.10
# docker tag docker:20.10 harbor.ink8s.com/ops/docker:20.10
# docker push harbor.ink8s.com/ops/docker:20.10

4.3.4 Containerd

1.24.0 以后的 kubernetes 版本采用 containerd 方式，如下为使用 containerd 起 Pod

1. 准备 containerd 所需文件
# wget https://github.com/containerd/nerdctl/releases/download/v1.5.0/nerdctl-full-1.5.0-linux-amd64.tar.gz
# tar -xf nerdctl-full-1.5.0-linux-amd64.tar.gz 
// 将 nerdctl buildctl buildkitd 3个文件放到当前目录

2. 系统所需文件
  config selinux 文件。默认 centos7 基础镜像的 selinux 是开启状态，里面设置为 disabled 状态
  CentOS-Base.repo epel.repo docker-ce.repo

3. config.toml containerd 的配置文件。需要将里面的设置调整好

4. supervisord.conf 多进程启动服务

// Dockerfile 文件(里面文件需要提前准备好到当前目录)
# cat Dockerfile
FROM harbor.ink8s.com/ops_apps_tools/centos:7
COPY CentOS-Base.repo /etc/yum.repos.d/
COPY epel.repo /etc/yum.repos.d/
COPY docker-ce.repo /etc/yum.repos.d/
COPY config /etc/selinux/config
RUN yum -y install containerd supervisor --nogpgcheck
RUN mkdir -p /etc/containerd
RUN yum clean all && rm -rf /var/cache/yum/*
COPY config.toml /etc/containerd/
COPY nerdctl buildctl buildkitd /usr/local/bin/
COPY supervisord.conf /etc/supervisord.conf
RUN mkdir -p /var/run/buildkit && chmod 777 /var/run/buildkit
CMD ["/usr/bin/supervisord", "-c", "/etc/supervisord.conf"]

# cat supervisord.conf                # buildkitd 服务必须在前面先启动，否则打包时会找不到 /var/run/buildkit/buildkitd.sock 文件
[supervisord]
nodaemon=true
logfile=/var/log/supervisord.log
pidfile=/var/run/supervisord.pid

[program:containerd]
command=/usr/bin/containerd
autostart=true
autorestart=true
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0

[program:buildkitd]
command=/usr/local/bin/buildkitd
autostart=true
autorestart=true
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0

# nerdctl build -t harbor.ink8s.com/ops-jenkins/containerd:1.6.33-v1 .        # 这里打包有个bug，如果不加"-v1"类似这种方式,镜像无法运行起来
# nerdctl push harbor.ink8s.com/ops-jenkins/containerd:1.6.33-v1

// 使用的时候同 docker 方式一样，采用 volumeMounts
// 可参考 pipeline 文件中的配置！！！
- name: containerd
  image: harbor.ink8s.com/ops-jenkins/containerd:1.6.33
  imagePullPolicy: IfNotPresent
  command: ["cat"]
  tty: true
  volumeMounts:
    - name: containerdsock
      mountPath: /run/containerd/containerd.sock

4.3.5 kubectl

Dockerfile

# cat Dockerfile
FROM centos:7

RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \
    echo 'Asia/Shanghai' >/etc/timezone
COPY kubectl /usr/bin/
RUN chmod +x /usr/bin/kubectl

# docker build -t harbor.ink8s.com/ops/kubectl:1.23.0 .
# docker push harbor.ink8s.com/ops/kubectl:1.23.0

4.3.6 测试模板镜像

运行一个流水线任务,定义Pod模板中的容器名称以及容器镜像地址,而后定义任务,不同的任务由不同的容器来执行

maven-rbd-pvc.yaml


# 由于 jenkins_pipeline_template_test.yaml 文件中的 maven 容器无法进行挂载，所以先单独创建，方能成功。
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: maven-rbd-pvc
  namespace: ops
spec:
  storageClassName: csi-rbd-sc
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 5Gi

jenkins_pipeline_template_test.yaml

pipeline {
  agent {
    kubernetes {
      cloud 'kubernetes'
      yaml """
apiVersion: v1
kind: Pod
metadata:
  name: test
  namespace: ops
spec:
  imagePullSecrets:
    - name: harbor-auth
  containers:
    - name: maven
      image: harbor.ink8s.com/ops-jenkins/maven:3.8.6
      imagePullPolicy: IfNotPresent
      command: ["cat"]
      tty: true
      volumenMounts:
        - name: maven-vms
          mountPath: /root/.m2

    - name: nodejs
      image: harbor.ink8s.com/ops-jenkins/nodejs:16.20
      imagePullPolicy: IfNotPresent
      command: ["cat"]
      tty: true

    - name: sonar
      image: harbor.ink8s.com/-ops-jenkins/sonar-scanner:2.3.0
      imagePullPolicy: IfNotPresent
      command: ["cat"]
      tty: true

    - name: containerd
      image: harbor.ink8s.com/ops-jenkins/containerd:1.6.33
      imagePullPolicy: IfNotPresent
      command: ["cat"]
      tty: true
      volumeMounts:
        - name: containerdsock
          mountPath: /run/containerd/containerd.sock

    - name: kubectl
      image: harbor.ink8s.com/ops-jenkins/kubectl:1.28.0
      imagePullPolicy: IfNotPresent
      command: ["cat"]
      tty: true

  volumes:
    - name: containerdsock
      hostPath:
        path: /run/containerd/containerd.sock
    - name: maven-vms
      persistentVolumeClaim:
        claimName: maven-rbd-pvc
        readOnly: false
"""
    }
  }
  stages {
    stage('maven测试') {
      steps {
        container('maven'){
            sh 'mvn --version'
        }
      }
    }
    stage('nodejs测试') {
      steps {
        container('nodejs'){
            sh 'node -v'
        }
      }
    }
    stage('containerd测试') {
      steps {
        container('containerd') {
            sh '/usr/local/bin/nerdctl ps -l'
        }
      }
    }
    stage('kubectl测试') {
      steps {
        container('kubectl'){
            sh 'kubectl version'
        }
      }
    }
  }
}

5.0 K8S Jenkins CI/CD

有点懒,不是很想更新。有需要的话加微信催我更新吧…

本文作者： [email protected]
本文链接： https://www.ink8s.com/2025/09/23/jenkins-ci-cd-部署/
版权声明： 本博客所有文章除特别声明外，均采用 MIT 许可协议。转载请注明出处！