misterli's Blog.

扫描k8s集群中漏洞

字数统计: 1.5k阅读时长: 7 min
2021/01/06

Kubei是一个漏洞扫描和CIS Docker基准测试工具,它能够对kubernetes集群进行准确,即时的风险评估。Kubei扫描Kubernetes集群中正在使用的所有图像,包括应用程序Pod和系统Pod的镜像。它不会扫描整个镜像注册表,也不需要与CI / CD进行初步集成。

它是一种可配置的工具,可以定义扫描范围(目标名称空间),速度和关注的漏洞级别,还提供了图形用户界面。

安装要求:

Kubernetes集群已准备就绪,并且已~/.kube/config为目标集群正确配置了kubeconfig

需要的权限

  1. 在集群范围内读取secret。这是用于获取私有镜像仓库的镜像的拉取凭据。
  2. 列出集群范围内的Pod。
  3. 在集群范围内创建job。创建job将扫描其名称空间中的目标容器

部署

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
---
apiVersion: v1
kind: Namespace
metadata:
name: kubei
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: kubei
namespace: kubei
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: kubei
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["list"]
- apiGroups: ["batch"]
resources: ["jobs"]
verbs: ["create","delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: kubei
subjects:
- kind: ServiceAccount
name: kubei
namespace: kubei
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: kubei
---
apiVersion: v1
kind: Service
metadata:
name: clair
namespace: kubei
labels:
app: clair
spec:
type: ClusterIP
ports:
- port: 6060
protocol: TCP
name: apiserver
- port: 6061
protocol: TCP
name: health
selector:
app: clair
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: clair
namespace: kubei
labels:
app: clair
spec:
replicas: 1
selector:
matchLabels:
app: clair
template:
metadata:
labels:
app: clair
kubeiShouldScan: "false"
spec:
initContainers:
- name: check-db-ready
image: postgres:12-alpine
command: ['sh', '-c', 'until pg_isready -h postgres -p 5432; do echo waiting for database; sleep 2; done;']
containers:
- name: clair
image: gcr.io/portshift-release/clair/clair-local-scan
imagePullPolicy: Always
ports:
- containerPort: 6060
- containerPort: 6061
resources:
limits:
cpu: 2000m
memory: 6G
requests:
cpu: 700m
memory: 3G
# uncomment the following lines in case Kubei is running behind proxy. update PROXY_IP & PROXY_PORT according to your proxy settings
# env:
# - name: HTTPS_PROXY
# value: "{ PROXY_IP }:{ PROXY_PORT }"
# - name: HTTP_PROXY
# value: "{ PROXY_IP }:{ PROXY_PORT }"
# - name: NO_PROXY
# value: "postgres"
---
apiVersion: v1
kind: Service
metadata:
labels:
app: postgres
name: postgres
namespace: kubei
spec:
type: ClusterIP
ports:
- name: db
port: 5432
protocol: TCP
selector:
app: postgres
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: clair-postgres
namespace: kubei
labels:
app: postgres
spec:
replicas: 1
selector:
matchLabels:
app: postgres
template:
metadata:
labels:
app: postgres
kubeiShouldScan: "false"
spec:
containers:
- name: clair-db
image: gcr.io/portshift-release/clair/clair-db
imagePullPolicy: Always
ports:
- containerPort: 5432
resources:
limits:
cpu: 1500m
memory: 1G
requests:
cpu: 500m
memory: 400Mi
---
apiVersion: v1
kind: Service
metadata:
namespace: kubei
name: kubei
labels:
app: kubei
spec:
type: ClusterIP
ports:
- port: 8080
protocol: TCP
name: http-webapp
- port: 8081
protocol: TCP
name: http-klar-result
selector:
app: kubei
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: kubei
namespace: kubei
labels:
app: kubei
spec:
replicas: 1
selector:
matchLabels:
app: kubei
template:
metadata:
labels:
app: kubei
kubeiShouldScan: "false"
spec:
serviceAccountName: kubei
initContainers:
- name: init-clairsvc
image: yauritux/busybox-curl
args:
- /bin/sh
- -c
- >
set -x;
while [ $(curl -sw '%{http_code}' "http://clair.kubei:6060/v1/namespaces" -o /dev/null) -ne 200 ]; do
echo "waiting for clair to be ready";
sleep 15;
done
containers:
- name: kubei
image: gcr.io/development-infra-208909/kubei:1.0.8
imagePullPolicy: Always
env:
- name: "KLAR_IMAGE_NAME"
value: "gcr.io/development-infra-208909/klar:1.0.7"
- name: "DOCKLE_IMAGE_NAME"
value: "gcr.io/development-infra-208909/dockle:1.0.0"
- name: "MAX_PARALLELISM" # max number of scans that will run simultaneously. defaults to 10
value: "10"
- name: "TARGET_NAMESPACE" # empty = scan all namespaces
value: ""
- name: "SEVERITY_THRESHOLD" # minimum level of vulnerability to report. defaults to MEDIUM
value: "MEDIUM"
- name: "IGNORE_NAMESPACES" # a list of namespaces to ignore. defaults no namespace to ignore
value: "istio-system,kube-system"
- name: "DELETE_JOB_POLICY"
value: "Successful"
- name: "SCANNER_SERVICE_ACCOUNT" # Defaults to 'default' service account
value: ""
- name: "REGISTRY_INSECURE" # Allow scanner to access insecure registries (HTTP only). Default is `false`.
value: "false"
# uncomment the following lines in case Kubei is running behind proxy. update PROXY_IP & PROXY_PORT according to your proxy settings
# - name: SCANNER_HTTPS_PROXY
# value: "{ PROXY_IP }:{ PROXY_PORT }"
# - name: SCANNER_HTTP_PROXY
# value: "{ PROXY_IP }:{ PROXY_PORT }"
ports:
- containerPort: 8080
- containerPort: 8081
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 20Mi

这里有部分镜像因为特殊原因无法拉取,这里我已经下载并上传到dockerhub,对应如下,修改即可。

gcr.io/portshift-release/clair/clair-local-scan —>misterli/clair-local-scan:v2.1.4.ubi.2

gcr.io/portshift-release/clair/clair-db —> misterli/clair-db:12.2.ubi.2

gcr.io/development-infra-208909/kubei:1.0.8 —> misterli/kubei:1.0.8

gcr.io/development-infra-208909/klar:1.0.7 —> misterli/development-infra-208909-klar:1.0.7

gcr.io/development-infra-208909/dockle:1.0.0 —> misterli/evelopment-infra-208909-dockle:1.0.0

配置说明

1、设置扫描范围,设置变量IGNORE_NAMESPACES以忽略特定的名称空间。设置变量TARGET_NAMESPACE为扫描特定的名称空间,或保留为空以扫描所有名称空间

2、设置扫描速度,变量MAX_PARALLELISM设置最大同时扫描数量

3、设置严重性级别阈值,SEVERITY_THRESHOLD将报告严重性级别高于或等于阈值的漏洞。支持的水平UnknownNegligibleLowMediumHighCriticalDefcon1。默认值为Medium

4、设置删除作业策略,设置变量DELETE_JOB_POLICY以定义是否删除已完成的扫描仪作业。支持的值为:

  • All -所有作业将被删除。
  • Successful -仅成功的作业将被删除(默认)。
  • Never -作业将永远不会被删除

5、禁用CIS Docker基准测试,将变量SHOULD_SCAN_DOCKERFILE设置为false

6、设置扫描仪服务帐户。将变量SCANNER_SERVICE_ACCOUNT设置为扫描仪作业要使用的服务帐户名称。默认为default服务帐户

7、扫描不安全的注册表。设置变量REGISTRY_INSECURE,以允许扫描程序访问不安全的注册表(仅HTTP)。默认值为false

执行扫描

我们部署一个ingress用来查看dashboard

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: kubei
namespace: kubei
spec:
entryPoints:
- web
routes:
- match: Host(`kubei.lishuai.fun`)
kind: Rule
services:
- name: kubei
port: 8080
middlewares: # 使用http重定向搭配https的中间件
- name: redirect-https
namespace: default
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: kubei-tls
namespace: kubei
spec:
entryPoints:
- websecure
routes:
- match: Host(`kubei.lishuai.fun`)
kind: Rule
services:
- name: kubei
port: 8080
tls:
certResolver: myresolver

查看dashboard 如下:

image-20210105172003260

我们点击右上角的go来执行扫描,点击后集群会为每个pod创建一个用来执行扫描的job。

1
2
3
4
5
6
7
8
9
[root@master-01 deploy]# kubectl get job --all-namespaces  -w 
NAMESPACE NAME COMPLETIONS DURATION AGE
kubernetes-dashboard scanner-dashboard-69f24894-91cc-4cf4-9dc8-d5c02dc7b7db 0/1 21m 21m
longhorn-system scanner-csi-attacher-31757dfc-f223-4ecc-878f-cd2ac2d41949 1/1 7s 17s
monitoring scanner-configmap-reload-230805e3-b3c1-4c00-ac35-4e28b4c17c58 0/1 17s 17s
monitoring scanner-prometheus-d6ca5856-424f-4019-8812-07f8cabae041 0/1 10s 10s
version-checker scanner-version-checker-93b0f404-41cd-4088-b082-8622bd25f200 0/1 0s
version-checker scanner-version-checker-93b0f404-41cd-4088-b082-8622bd25f200 0/1 0s 0s
monitoring scanner-prometheus-d6ca5856-424f-4019-8812-07f8cabae041 0/1 13s 13s

稍等再查看dashboard就会出现扫描结果

image-20210105180124460

image-20210105180157128

CATALOG
  1. 1. 配置说明
  2. 2. 执行扫描