misterli's Blog.

k8s之纵向扩缩容vpa

k8s vpa 动态扩缩容
字数统计: 2.4k阅读时长: 11 min
2020/09/02

vpa

纵向容器自动缩放器（VPA）使用户无需设置最新的资源限制和对容器中容器的要求。配置后，它将根据使用情况自动设置请求，从而允许在节点上进行适当的调度，以便为每个Pod提供适当的资源量。它还将保持限制和初始容器配置中指定的请求之间的比率。

它既可以根据资源的使用情况来缩减对资源过度使用的Pod，也可以对资源需求不足的向上扩展Pod。

Kubernetes VPA 包含以下组件：

  • Recommender：用于根据监控指标结合内置机制给出资源建议值
  • Updater：用于实时更新 pod resource requests
  • History Storage：用于采集和存储监控数据
  • Admission Controller: 用于在 pod 创建时修改 resource requests

架构图

VPA体系结构图

主要流程是：Recommender在启动时从History Storage获取历史数据，根据内置机制修改VPA API object资源建议值。Updater监听VPA API object，依据建议值动态修改 pod resource requests。VPA Admission Controller则是用于 pod 创建时修改 pod resource requests。History Storage则是通过Kubernetes Metrics API采集和存储监控数据。

CPU 和内存的建议值均是依据历史数据+固定机制计算而成。

在 Kubernetes VPA 中缺少资源回收的机制，但Recommender却可以配合Updater动态修改 pod resource requests 的值。也就是说 pod resource requests - 推荐值 = 资源回收值。这间接实现了资源回收的功能。

VPA工作流程

空值

  1. VPA连续检查您在设置过程中配置的指标值，默认间隔为10秒
  2. 如果达到阈值，VPA会尝试更改分配的内存和/或CPU
  3. VPA主要更新部署或复制控制器规范中的资源
  4. 重新启动Pod后，新资源将全部应用于创建的实例。

使用VPA时需要考虑以下几点：

  • 如果不重新启动Pod，将无法更改资源。到目前为止，主要的合理性在于，这样的变化可能会导致很多不稳定。因此，考虑重新启动Pod并根据新分配的资源对其进行调度。
  • VPA和HPA尚不兼容，不能在同一pod上使用。如果要在同一群集中同时使用它们，请确保在设置中分开它们的作用域。
  • VPA仅根据观察到的过去和当前资源使用情况来调整容器的资源请求。它没有设置资源限制。对于行为不当的应用程序可能会出现问题，这些应用程序开始使用越来越多的资源，导致pod被Kubernetes杀死。

安装

当前默认版本是Vertical Pod Autoscaler 0.8.0

1
2
3
git clone https://github.com/kubernetes/autoscaler.git
cd autoscaler
./hack/vpa-up.sh

注意：该脚本当前读取环境变量：$REGISTRY和$TAG。除非要使用VPA的非默认版本，否则请确保未设置它们。

如果出现如下报错，我们可以升级openssl到支持-addext选项的版本，或在下载via-release-0.8这个分支的代码并使用./hack/vpa-up.sh 

1
2
ERROR: Failed to create CA certificate for self-signing. If the error is "unknown option -addext", update your openssl version or deploy VPA from the vpa-release-0.8 branch.
deployment.apps/vpa-admission-controller created

安装好之后会创建三个pod，这三个pod使用的镜像需要翻墙下载

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
[root@master01 vertical-pod-autoscaler]# kubectl -n kube-system  get pod 
NAME                                                READY   STATUS    RESTARTS   AGE
calico-kube-controllers-688c5dc8c7-rcn2t            1/1     Running   0          59d
calico-node-n7sj8                                   1/1     Running   7          89d
calico-node-vk9c9                                   1/1     Running   1          89d
coredns-6955765f44-mgcc4                            1/1     Running   0          59d
coredns-6955765f44-vwzxt                            1/1     Running   0          59d
csi-disk-provisioner-0                              2/2     Running   0          17d
csi-diskplugin-jmw74                                2/2     Running   0          17d
csi-diskplugin-mv99m                                2/2     Running   0          17d
dex-1591193803-6844b4846c-4wwzg                     1/1     Running   1          19d
dex-k8s-authenticator-1591193977-54f66bd94f-tv2rl   1/1     Running   1          59d
etcd-master01                                       1/1     Running   8          89d
kube-apiserver-master01                             1/1     Running   0          19d
kube-controller-manager-master01                    1/1     Running   0          13d
kube-proxy-s98fh                                    1/1     Running   0          59d
kube-proxy-zn2l6                                    1/1     Running   0          59d
kube-scheduler-master01                             1/1     Running   0          13d
kube-sealyun-lvscare-node01                         1/1     Running   0          89d
kuboard-85d499bd9b-xmvtp                            1/1     Running   0          15d
metrics-server-6b4d999dcb-xzx4p                     1/1     Running   0          13d
snapshot-controller-0                               1/1     Running   0          14d
traefik-0                                           1/1     Running   0          20d
vpa-admission-controller-6b88678dcb-d5vb6           1/1     Running   0          7s
vpa-recommender-5fc9b4c4cb-d5j8l                    1/1     Running   0          32s
vpa-updater-865849df48-rcl94                        1/1     Running   0          40s
[root@master01 vertical-pod-autoscaler]# kubectl -n kube-system  get deployments.apps  -o wide
NAME                               READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS                IMAGES     
....
vpa-admission-controller           1/1     1            1           62m   admission-controller      us.gcr.io/k8s-artifacts-prod/autoscaling/vpa-admission-controller:0.8.0   app=vpa-admission-controller
vpa-recommender                    1/1     1            1           62m   recommender               us.gcr.io/k8s-artifacts-prod/autoscaling/vpa-recommender:0.8.0            app=vpa-recommender
vpa-updater                        1/1     1            1           62m   updater                   us.gcr.io/k8s-artifacts-prod/autoscaling/vpa-updater:0.8.0                app=vpa-updater

这里可以使用 misterli/vpa-recommender:0.8.0,misterli/vpa-updater:0.8.0,misterli/vpa-admission-controller:0.8.0这三个镜像进行替换

使用

  • VPA 有以下四种更新策略：

    •Initial：仅在 Pod 创建时修改资源请求，以后都不再修改。

    •Auto：默认策略，在 Pod 创建时修改资源请求，并且在 Pod 更新时也会修改。

    •Recreate：类似 Auto，在 Pod 的创建和更新时都会修改资源请求，不同的是，只要Pod 中的请求值与新的推荐值不同，VPA 都会驱逐该 Pod，然后使用新的推荐值重新启一个。因此，一般不使用该策略，而是使用 Auto，除非你真的需要保证请求值是最新的推荐值。

    •Off：不改变 Pod 的资源请求，不过仍然会在 VPA 中设置资源的推荐值。

我们部署一个使用vpa的示例：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
---
apiVersion: "autoscaling.k8s.io/v1beta2"
kind: VerticalPodAutoscaler
metadata:
  name: hamster-vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: hamster
  updatePolicy:
    updateMode: "Auto"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hamster
spec:
  selector:
    matchLabels:
      app: hamster
  replicas: 2
  template:
    metadata:
      labels:
        app: hamster
    spec:
      containers:
        - name: hamster
          image: ubuntu:16.04
          resources:
            requests:
              cpu: 100m
              memory: 50Mi
          command: ["/bin/sh"]
          args:
            - "-c"
            - "while true; do timeout 0.5s yes >/dev/null; sleep 0.5s; done"

我们发现稍等片刻vpa会对pod的资源进行动态更新

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
[root@master01 vertical-pod-autoscaler]# kubectl apply -f test.yaml 
verticalpodautoscaler.autoscaling.k8s.io/hamster-vpa created
deployment.apps/hamster created
[root@master01 vertical-pod-autoscaler]# kubectl get pod -w 
NAME                              READY   STATUS    RESTARTS   AGE
event-exporter-56b4dbc8fb-gsmx4   1/1     Running   0          40d
hamster-bb5bb8754-krljt           1/1     Running   0          21s
hamster-bb5bb8754-nvhf2           1/1     Running   0          21s
hamster-bb5bb8754-krljt           1/1     Terminating   0          92s
hamster-bb5bb8754-qs8dl           0/1     Pending       0          0s
hamster-bb5bb8754-qs8dl           0/1     Pending       0          0s
hamster-bb5bb8754-qs8dl           0/1     ContainerCreating   0          0s
hamster-bb5bb8754-qs8dl           0/1     ContainerCreating   0          0s
hamster-bb5bb8754-qs8dl           1/1     Running             0          1s
hamster-bb5bb8754-krljt           0/1     Terminating         0          2m3s
hamster-bb5bb8754-krljt           0/1     Terminating         0          2m4s
hamster-bb5bb8754-krljt           0/1     Terminating         0          2m4s
hamster-bb5bb8754-nvhf2           1/1     Terminating         0          2m32s
hamster-bb5bb8754-d2rnh           0/1     Pending             0          0s
hamster-bb5bb8754-d2rnh           0/1     Pending             0          0s
hamster-bb5bb8754-d2rnh           0/1     ContainerCreating   0          0s
hamster-bb5bb8754-d2rnh           0/1     ContainerCreating   0          1s
hamster-bb5bb8754-d2rnh           1/1     Running             0          2s

查看一下vpa的详情可以看到vpa建议的值

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
[root@master01 vertical-pod-autoscaler]# kubectl describe vpa hamster-vpa 
Name:         hamster-vpa
Namespace:    default
.....
  Recommendation:
    Container Recommendations:
      Container Name:  hamster
      Lower Bound:
        Cpu:     579m
        Memory:  262144k
      Target:
        Cpu:     587m
        Memory:  262144k
      Uncapped Target:
        Cpu:     587m
        Memory:  262144k
      Upper Bound:
        Cpu:     4554m
        Memory:  262144k
Events:          <none>

查看当前pod的资源限制，pod的requests已经被动态调整了

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
[root@master01 vertical-pod-autoscaler]# kubectl get pod hamster-bb5bb8754-kp9jk  -o yaml 
apiVersion: v1
kind: Pod
....
    name: hamster
    resources:
      requests:
        cpu: 587m
        memory: 262144k
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-jqvcc
      readOnly: true
.....

还可以限制动态调整的范围

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
---
apiVersion: "autoscaling.k8s.io/v1beta2"
kind: VerticalPodAutoscaler
metadata:
  name: hamster-vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: hamster
  resourcePolicy:
    containerPolicies:
      - containerName: '*'
        minAllowed:
          cpu: 100m
          memory: 50Mi
        maxAllowed:
          cpu: 1
          memory: 500Mi
        controlledResources: ["cpu", "memory"]
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hamster
spec:
  selector:
    matchLabels:
      app: hamster
  replicas: 2
  template:
    metadata:
      labels:
        app: hamster
    spec:
      containers:
        - name: hamster
          image: ubuntu:16.04
          resources:
            requests:
              cpu: 100m
              memory: 50Mi
          command: ["/bin/sh"]
          args:
            - "-c"
            - "while true; do timeout 0.5s yes >/dev/null; sleep 0.5s; done"

上面的yaml创建了一个包含2个Pod的deployment，每个Pod运行一个请求100m cpu 的容器，并尝试使用略高于500m cpu的容器。该命令还会创建一个指向deployment的VPA配置。VPA将观察Pod的行为，大约5分钟后，它们应使用更高的CPU请求进行更新（请注意，VPA不会在deployment中修改模板，但Pod的实际请求会被更新）。要查看VPA配置和当前推荐的资源请求:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
[root@master01 vertical-pod-autoscaler]# kubectl describe vpa hamster-vpa 
Name:         hamster-vpa
Namespace:    default
...
Status:
  Conditions:
    Last Transition Time:  2020-09-01T03:55:47Z
    Status:                True
    Type:                  RecommendationProvided
  Recommendation:
    Container Recommendations:
      Container Name:  hamster
      Lower Bound:
        Cpu:     552m
        Memory:  262144k
      Target:
        Cpu:     587m
        Memory:  262144k
      Uncapped Target:
        Cpu:     587m
        Memory:  262144k
      Upper Bound:
        Cpu:     1
        Memory:  370459537
Events:          <none>

查看pod 详情，发现此时 resources字段已经被更改

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[root@master01 vertical-pod-autoscaler]# kubectl get pod hamster-bb5bb8754-gcrbc  -o yaml 
apiVersion: v1
kind: Pod
   ....
    image: ubuntu:16.04
    imagePullPolicy: IfNotPresent
    name: hamster
    resources:
      requests:
        cpu: 587m
        memory: 262144k
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    ...
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
[root@master01 vertical-pod-autoscaler]# kubectl get pod -w 
NAME                              READY   STATUS    RESTARTS   AGE
event-exporter-56b4dbc8fb-gsmx4   1/1     Running   0          40d
hamster-bb5bb8754-ghggr           1/1     Running   0          5s
hamster-bb5bb8754-xnndg           1/1     Running   0          5s
hamster-bb5bb8754-xnndg           1/1     Terminating   0          62s
hamster-bb5bb8754-gcrbc           0/1     Pending       0          0s
hamster-bb5bb8754-gcrbc           0/1     Pending       0          0s
hamster-bb5bb8754-gcrbc           0/1     ContainerCreating   0          0s
hamster-bb5bb8754-gcrbc           0/1     ContainerCreating   0          1s
hamster-bb5bb8754-gcrbc           1/1     Running             0          1s
hamster-bb5bb8754-xnndg           0/1     Terminating         0          93s
hamster-bb5bb8754-xnndg           0/1     Terminating         0          94s
hamster-bb5bb8754-xnndg           0/1     Terminating         0          94s
hamster-bb5bb8754-ghggr           1/1     Terminating         0          2m2s
hamster-bb5bb8754-gwxgw           0/1     Pending             0          0s
hamster-bb5bb8754-gwxgw           0/1     Pending             0          0s
hamster-bb5bb8754-gwxgw           0/1     ContainerCreating   0          0s
hamster-bb5bb8754-gwxgw           0/1     ContainerCreating   0          1s
hamster-bb5bb8754-gwxgw           1/1     Running             0          2s

event事件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
0s          Normal    EvictedByVPA        pod/hamster-bb5bb8754-xnndg     Pod was evicted by VPA Updater to apply resource recommendation.
0s          Normal    SuccessfulCreate    replicaset/hamster-bb5bb8754    Created pod: hamster-bb5bb8754-gcrbc
0s          Normal    Scheduled           pod/hamster-bb5bb8754-gcrbc     Successfully assigned default/hamster-bb5bb8754-gcrbc to node01
0s          Normal    Killing             pod/hamster-bb5bb8754-xnndg     Stopping container hamster
0s          Normal    Pulled              pod/hamster-bb5bb8754-gcrbc     Container image "ubuntu:16.04" already present on machine
0s          Normal    Created             pod/hamster-bb5bb8754-gcrbc     Created container hamster
0s          Normal    Started             pod/hamster-bb5bb8754-gcrbc     Started container hamster
0s          Normal    EvictedByVPA        pod/hamster-bb5bb8754-ghggr     Pod was evicted by VPA Updater to apply resource recommendation.
0s          Normal    Killing             pod/hamster-bb5bb8754-ghggr     Stopping container hamster
0s          Normal    SuccessfulCreate    replicaset/hamster-bb5bb8754    Created pod: hamster-bb5bb8754-gwxgw
0s          Normal    Scheduled           pod/hamster-bb5bb8754-gwxgw     Successfully assigned default/hamster-bb5bb8754-gwxgw to node01
0s          Normal    Pulled              pod/hamster-bb5bb8754-gwxgw     Container image "ubuntu:16.04" already present on machine
0s          Normal    Created             pod/hamster-bb5bb8754-gwxgw     Created container hamster
0s          Normal    Started             pod/hamster-bb5bb8754-gwxgw     Started container hamster

原文作者：Mister Li

原文链接：https://www.lishuai.fun/2020/09/02/k8s-vpa/

发表日期：September 2nd 2020, 2:07:39 pm

更新日期：May 3rd 2022, 11:25:55 pm

版权声明：本文采用知识共享署名-非商业性使用 4.0 国际许可协议进行许可

CATALOG
  1. 1. vpa
    1. 1.1. 架构图
    2. 1.2. VPA工作流程
    3. 1.3. 安装
    4. 1.4. 使用
Total : 46
2023
2022
2021
2020
kubernetes rbac k8s vulnerability ingress-nginx RBAC argocd devops gitlab harbor slb hugo pv csi s3 kubesphere prometheus sso kubevela k8s准入控制 kyverno longhorn oomkill event configmap secret sync ingress cert-manager chaos-mesh minio vpa 动态扩缩容 k8s kubevela oamm 报警 ssl-exporter traefik 混沌工程 rke k8s pvc nfs prometheus redis umami etcd kuboard docker ci buildkit redis