misterli's Blog.

扫描k8s集群中漏洞

k8s vulnerability
字数统计: 1.5k阅读时长: 7 min
2021/01/06

Kubei是一个漏洞扫描和CIS Docker基准测试工具，它能够对kubernetes集群进行准确，即时的风险评估。Kubei扫描Kubernetes集群中正在使用的所有图像，包括应用程序Pod和系统Pod的镜像。它不会扫描整个镜像注册表，也不需要与CI / CD进行初步集成。

它是一种可配置的工具，可以定义扫描范围（目标名称空间），速度和关注的漏洞级别，还提供了图形用户界面。

安装要求：

Kubernetes集群已准备就绪，并且已~/.kube/config为目标集群正确配置了kubeconfig

需要的权限

  1. 在集群范围内读取secret。这是用于获取私有镜像仓库的镜像的拉取凭据。
  2. 列出集群范围内的Pod。
  3. 在集群范围内创建job。创建job将扫描其名称空间中的目标容器

部署

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
---
apiVersion: v1
kind: Namespace
metadata:
  name: kubei
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kubei
  namespace: kubei
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kubei
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["list"]
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs: ["create","delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubei
subjects:
- kind: ServiceAccount
  name: kubei
  namespace: kubei
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kubei
---
apiVersion: v1
kind: Service
metadata:
  name: clair
  namespace: kubei
  labels:
    app: clair
spec:
  type: ClusterIP
  ports:
  - port: 6060
    protocol: TCP
    name: apiserver
  - port: 6061
    protocol: TCP
    name: health
  selector:
    app: clair
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: clair
  namespace: kubei
  labels:
    app: clair
spec:
  replicas: 1
  selector:
    matchLabels:
      app: clair
  template:
    metadata:
      labels:
        app: clair
        kubeiShouldScan: "false"
    spec:
      initContainers:
      - name: check-db-ready
        image: postgres:12-alpine
        command: ['sh', '-c',  'until pg_isready -h postgres -p 5432;  do echo waiting for database; sleep 2; done;']
      containers:
      - name: clair
        image: gcr.io/portshift-release/clair/clair-local-scan
        imagePullPolicy: Always
        ports:
        - containerPort: 6060
        - containerPort: 6061
        resources:
          limits:
            cpu: 2000m
            memory: 6G
          requests:
            cpu: 700m
            memory: 3G
# uncomment the following lines in case Kubei is running behind proxy. update PROXY_IP & PROXY_PORT according to your proxy settings
#        env:
#        - name: HTTPS_PROXY
#          value: "{ PROXY_IP }:{ PROXY_PORT }"
#        - name: HTTP_PROXY
#          value: "{ PROXY_IP }:{ PROXY_PORT }"
#        - name: NO_PROXY
#          value: "postgres"
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: postgres
  name: postgres
  namespace: kubei
spec:
  type: ClusterIP
  ports:
    - name: db
      port: 5432
      protocol: TCP
  selector:
    app: postgres
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: clair-postgres
  namespace: kubei
  labels:
    app: postgres
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
        kubeiShouldScan: "false"
    spec:
      containers:
      - name: clair-db  
        image: gcr.io/portshift-release/clair/clair-db
        imagePullPolicy: Always
        ports:
        - containerPort: 5432
        resources:
          limits:
            cpu: 1500m
            memory: 1G
          requests:
            cpu: 500m
            memory: 400Mi
---
apiVersion: v1
kind: Service
metadata:
  namespace: kubei
  name: kubei
  labels:
    app: kubei
spec:
  type: ClusterIP
  ports:
  - port: 8080
    protocol: TCP
    name: http-webapp
  - port: 8081
    protocol: TCP
    name: http-klar-result
  selector:
    app: kubei
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kubei
  namespace: kubei
  labels:
    app: kubei
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kubei
  template:
    metadata:
      labels:
        app: kubei
        kubeiShouldScan: "false"
    spec:
      serviceAccountName: kubei
      initContainers:
      - name: init-clairsvc
        image: yauritux/busybox-curl
        args:
          - /bin/sh
          - -c
          - >
            set -x;
            while [ $(curl -sw '%{http_code}' "http://clair.kubei:6060/v1/namespaces" -o /dev/null) -ne 200 ]; do
              echo "waiting for clair to be ready";
              sleep 15;
            done
      containers:
      - name: kubei
        image: gcr.io/development-infra-208909/kubei:1.0.8
        imagePullPolicy: Always
        env:
        - name: "KLAR_IMAGE_NAME"
          value: "gcr.io/development-infra-208909/klar:1.0.7"
        - name: "DOCKLE_IMAGE_NAME"
          value: "gcr.io/development-infra-208909/dockle:1.0.0"
        - name: "MAX_PARALLELISM" # max number of scans that will run simultaneously. defaults to 10
          value: "10"
        - name: "TARGET_NAMESPACE" # empty = scan all namespaces
          value: ""
        - name: "SEVERITY_THRESHOLD" # minimum level of vulnerability to report. defaults to MEDIUM
          value: "MEDIUM"
        - name: "IGNORE_NAMESPACES" # a list of namespaces to ignore. defaults no namespace to ignore
          value: "istio-system,kube-system"
        - name: "DELETE_JOB_POLICY"
          value: "Successful"
        - name: "SCANNER_SERVICE_ACCOUNT" # Defaults to 'default' service account
          value: ""
        - name: "REGISTRY_INSECURE" # Allow scanner to access insecure registries (HTTP only). Default is `false`.
          value: "false"
# uncomment the following lines in case Kubei is running behind proxy. update PROXY_IP & PROXY_PORT according to your proxy settings
#        - name: SCANNER_HTTPS_PROXY
#          value: "{ PROXY_IP }:{ PROXY_PORT }"
#        - name: SCANNER_HTTP_PROXY
#          value: "{ PROXY_IP }:{ PROXY_PORT }"
        ports:
        - containerPort: 8080
        - containerPort: 8081
        resources:
          limits:
            cpu: 100m
            memory: 100Mi
          requests:
            cpu: 10m
            memory: 20Mi

这里有部分镜像因为特殊原因无法拉取，这里我已经下载并上传到dockerhub,对应如下，修改即可。

gcr.io/portshift-release/clair/clair-local-scan —>misterli/clair-local-scan:v2.1.4.ubi.2

gcr.io/portshift-release/clair/clair-db —> misterli/clair-db:12.2.ubi.2

gcr.io/development-infra-208909/kubei:1.0.8 —> misterli/kubei:1.0.8

gcr.io/development-infra-208909/klar:1.0.7 —> misterli/development-infra-208909-klar:1.0.7

gcr.io/development-infra-208909/dockle:1.0.0 —> misterli/evelopment-infra-208909-dockle:1.0.0

配置说明

1、设置扫描范围，设置变量IGNORE_NAMESPACES以忽略特定的名称空间。设置变量TARGET_NAMESPACE为扫描特定的名称空间，或保留为空以扫描所有名称空间

2、设置扫描速度，变量MAX_PARALLELISM设置最大同时扫描数量

3、设置严重性级别阈值，SEVERITY_THRESHOLD将报告严重性级别高于或等于阈值的漏洞。支持的水平Unknown，Negligible，Low，Medium，High，Critical，Defcon1。默认值为Medium

4、设置删除作业策略,设置变量DELETE_JOB_POLICY以定义是否删除已完成的扫描仪作业。支持的值为：

  • All -所有作业将被删除。
  • Successful -仅成功的作业将被删除（默认）。
  • Never -作业将永远不会被删除

5、禁用CIS Docker基准测试,将变量SHOULD_SCAN_DOCKERFILE设置为false

6、设置扫描仪服务帐户。将变量SCANNER_SERVICE_ACCOUNT设置为扫描仪作业要使用的服务帐户名称。默认为default服务帐户

7、扫描不安全的注册表。设置变量REGISTRY_INSECURE，以允许扫描程序访问不安全的注册表（仅HTTP）。默认值为false。

执行扫描

我们部署一个ingress用来查看dashboard

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: kubei
  namespace: kubei
spec:
  entryPoints:
    - web
  routes:
  - match: Host(`kubei.lishuai.fun`)
    kind: Rule
    services:
    - name: kubei
      port: 8080
    middlewares:  # 使用http重定向搭配https的中间件
    - name: redirect-https
      namespace: default
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: kubei-tls
  namespace: kubei
spec:
  entryPoints:
    - websecure
  routes:
  - match: Host(`kubei.lishuai.fun`)
    kind: Rule
    services:
    - name: kubei
      port: 8080
  tls:
    certResolver: myresolver

查看dashboard 如下：

image-20210105172003260

我们点击右上角的go来执行扫描，点击后集群会为每个pod创建一个用来执行扫描的job。

1
2
3
4
5
6
7
8
9
[root@master-01 deploy]# kubectl get job --all-namespaces  -w 
NAMESPACE              NAME                                                            COMPLETIONS   DURATION   AGE
kubernetes-dashboard   scanner-dashboard-69f24894-91cc-4cf4-9dc8-d5c02dc7b7db          0/1           21m        21m
longhorn-system        scanner-csi-attacher-31757dfc-f223-4ecc-878f-cd2ac2d41949       1/1           7s         17s
monitoring             scanner-configmap-reload-230805e3-b3c1-4c00-ac35-4e28b4c17c58   0/1           17s        17s
monitoring             scanner-prometheus-d6ca5856-424f-4019-8812-07f8cabae041         0/1           10s        10s
version-checker        scanner-version-checker-93b0f404-41cd-4088-b082-8622bd25f200    0/1                      0s
version-checker        scanner-version-checker-93b0f404-41cd-4088-b082-8622bd25f200    0/1           0s         0s
monitoring             scanner-prometheus-d6ca5856-424f-4019-8812-07f8cabae041         0/1           13s        13s

稍等再查看dashboard就会出现扫描结果

image-20210105180124460

image-20210105180157128

原文作者：Mister Li

原文链接：https://www.lishuai.fun/2021/01/06/Scan-the-K8S-cluster-for-vulnerabilities/

发表日期：January 6th 2021, 11:26:23 am

更新日期：May 3rd 2022, 11:25:55 pm

版权声明：本文采用知识共享署名-非商业性使用 4.0 国际许可协议进行许可

CATALOG
  1. 1. 配置说明
  2. 2. 执行扫描
Total : 48
2024
2023
2022
2021
2020
kubernetes rbac k8s vulnerability ingress-nginx RBAC argocd devops gitlab harbor slb hugo pv csi s3 kubesphere prometheus sso kubevela k8s准入控制 kyverno longhorn oomkill event configmap secret sync ingress cert-manager chaos-mesh minio vpa 动态扩缩容 k8s kubevela oamm 报警 ssl-exporter traefik 混沌工程 rke k8s pvc nfs prometheus redis umami etcd kuboard docker ci buildkit redis dockerhub proxy Cloudflare