misterli's Blog.

三个小工具带你进一步了解K8S集群内RBAC

字数统计: 1.9k阅读时长: 10 min
2021/08/10

利用k8s审计日志生成RBAC规则

简介

很多时候我们在k8s上安装服务会遇到各种各样的权限问题,有时候为某个用户或者serviceaccount对象生成一个合适的role会比较头疼,这里推荐一个工具audit2rbac,它可以根据k8s的审计日志,为指定用户或者serviceaccount对象生成它们所需要的role.

audit2rbac下载地址: https://github.com/liggitt/audit2rbac/releases

前提要求

1、集群已经开启审计日志,且日志格式为json格式,开启审计日志可以参考https://kubernetes.io/docs/tasks/debug-application-cluster/audit/#advanced-audit

2、建议日志级别设置为Metadata,还可以减少日志大小

使用

我们这里已经开启了审计日志,这里截取一小段日志内容如下:

1
2
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"571b8d06-aa30-4aec-87cb-7bef2ef88d18","stage":"ResponseComplete","requestURI":"/apis/coordination.k8s.io/v1/namespaces/longhorn-system/leases/external-resizer-driver-longhorn-io","verb":"update","user":{"username":"system:serviceaccount:longhorn-system:longhorn-service-account","uid":"cdb0a05f-170d-4f02-aeec-88af904e68f7","groups":["system:serviceaccounts","system:serviceaccounts:longhorn-system","system:authenticated"]},"sourceIPs":["172.20.166.16"],"userAgent":"csi-resizer/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"leases","namespace":"longhorn-system","name":"external-resizer-driver-longhorn-io","uid":"81766194-e2e3-4edd-83d7-788a07562b91","apiGroup":"coordination.k8s.io","apiVersion":"v1","resourceVersion":"18772044"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2021-01-06T03:02:52.709670Z","stageTimestamp":"2021-01-06T03:02:52.710917Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"longhorn-bind\" of ClusterRole \"longhorn-role\" to ServiceAccount \"longhorn-service-account/longhorn-system\""}}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"0795fecc-38ea-46d7-a27d-6e73e6a27cd8","stage":"ResponseComplete","requestURI":"/apis/coordination.k8s.io/v1/namespaces/longhorn-system/leases/driver-longhorn-io","verb":"get","user":{"username":"system:serviceaccount:longhorn-system:longhorn-service-account","uid":"cdb0a05f-170d-4f02-aeec-88af904e68f7","groups":["system:serviceaccounts","system:serviceaccounts:longhorn-system","system:authenticated"]},"sourceIPs":["172.20.166.16"],"userAgent":"csi-provisioner/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"leases","namespace":"longhorn-system","name":"driver-longhorn-io","apiGroup":"coordination.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2021-01-06T03:02:52.713255Z","stageTimestamp":"2021-01-06T03:02:52.713894Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"longhorn-bind\" of ClusterRole \"longhorn-role\" to ServiceAccount \"longhorn-service-account/longhorn-system\""}}

接下来我们使用audit2rbac为serviceaccount:longhorn-system:longhorn-service-account生成rbac role

1
2
3
4
5
6
[root@master-01 audit2rbac]# ./audit2rbac -f /var/log/kube-audit/audit-log.json --serviceaccount longhorn-system:longhorn-service-account > longhorn-service-account-role.yaml
Opening audit source...
Loading events......................................
Evaluating API calls...
Generating roles...
Complete!

查看一下生成的role

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
annotations:
audit2rbac.liggitt.net/version: v0.8.0
labels:
audit2rbac.liggitt.net/generated: "true"
audit2rbac.liggitt.net/user: system-serviceaccount-longhorn-system-longhorn-service-account
name: audit2rbac:system:serviceaccount:longhorn-system:longhorn-service-account
namespace: longhorn-system
rules:
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- get
- patch
- update
- apiGroups:
- longhorn.io
resourceNames:
- pvc-50b55b24-fdd4-4714-be16-49c84fea4e1c-e-4020fcd8
resources:
- engines/status
verbs:
- get
- patch
- update
- apiGroups:
- longhorn.io
resourceNames:
- node-01
resources:
- nodes/status
verbs:
- get
- patch
- update
- apiGroups:
- longhorn.io
resourceNames:
- pvc-50b55b24-fdd4-4714-be16-49c84fea4e1c
resources:
- volumes/status
verbs:
- get
- patch
- update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
audit2rbac.liggitt.net/version: v0.8.0
labels:
audit2rbac.liggitt.net/generated: "true"
audit2rbac.liggitt.net/user: system-serviceaccount-longhorn-system-longhorn-service-account
name: audit2rbac:system:serviceaccount:longhorn-system:longhorn-service-account
rules:
- apiGroups:
- ""
resources:
- nodes
- persistentvolumeclaims
- persistentvolumes
- pods
verbs:
- get
- list
- watch
- apiGroups:
- apps
resources:
- daemonsets
- deployments
verbs:
- get
- list
- watch
- apiGroups:
- batch
resources:
- cronjobs
verbs:
- get
- list
- watch
- apiGroups:
- longhorn.io
resources:
- engineimages
- engines
- instancemanagers
- nodes
- replicas
- settings
- volumes
verbs:
- get
- list
- watch
- apiGroups:
- scheduling.k8s.io
resources:
- priorityclasses
verbs:
- get
- list
- watch
- apiGroups:
- storage.k8s.io
resources:
- csinodes
- storageclasses
- volumeattachments
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
annotations:
audit2rbac.liggitt.net/version: v0.8.0
labels:
audit2rbac.liggitt.net/generated: "true"
audit2rbac.liggitt.net/user: system-serviceaccount-longhorn-system-longhorn-service-account
name: audit2rbac:system:serviceaccount:longhorn-system:longhorn-service-account
namespace: longhorn-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: audit2rbac:system:serviceaccount:longhorn-system:longhorn-service-account
subjects:
- kind: ServiceAccount
name: longhorn-service-account
namespace: longhorn-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
annotations:
audit2rbac.liggitt.net/version: v0.8.0
labels:
audit2rbac.liggitt.net/generated: "true"
audit2rbac.liggitt.net/user: system-serviceaccount-longhorn-system-longhorn-service-account
name: audit2rbac:system:serviceaccount:longhorn-system:longhorn-service-account
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: audit2rbac:system:serviceaccount:longhorn-system:longhorn-service-account
subjects:
- kind: ServiceAccount
name: longhorn-service-account
namespace: longhorn-system

可视化集群中的rbac

rback可以使用Kubernetes集群的所有与RBAC相关的信息去生成service account,roles,和访问规则之间的关系图

安装

1
2
curl -sL https://github.com/team-soteria/rback/releases/download/v0.4.0/linux_rback -o rback 
chmod +x rback && sudo mv rback /usr/local/bin

使用

我们可以在本地运行并输出为result.dot文件,利用该文件可以在线或者本地渲染图形

1
kubectl get sa,roles,rolebindings,clusterroles,clusterrolebindings --all-namespaces -o json | rback > result.dot

在线渲染

有许多Graphviz(dot)在线可视化工具可用,例如,使用magjac.com/graphviz-visual-editor/dreampuf.github.io/GraphvizOnline

我们打开result.dot,把文件内容复制到上面网站即可

image-20210113112859183

这里节选一部分下载到本地的图形(展示所有namespace中图形太长了)

image-20210113113039225

本地渲染

这里以mac上操作示例

安装graphviz brew install graphviz,然后执行下面操作

1
kubectl get sa,roles,rolebindings,clusterroles,clusterrolebindings --all-namespaces -o json | rback | dot -Tpng  > /tmp/rback.png && open /tmp/rback.png

使用rbac-lookup查看集群内rbac绑定

简介

RBAC Lookup 是一个 CLI,可让您轻松查找绑定到任何用户、service accout 、group的 Kubernetes roles 和 cluster roles 。

安装

https://github.com/FairwindsOps/rbac-lookup/releases页面下载

1
2
3
4
5
6
wget https://github.com/FairwindsOps/rbac-lookup/releases/download/v0.6.4/rbac-lookup_0.6.4_Linux_x86_64.tar.gz
tar -xt rbac-lookup_0.6.4_Linux_x86_64.tar.g
mv rbac-lookup /usr/local/bin

[root@master-01 permission-manager]# rbac-lookup version
Version:0.6.4 Commit:3c86486a94e56e762b2a46d899e5f7b235c74ca8

使用

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
[root@master-01 permission-manager]# rbac-lookup  -h
rbac-lookup provides a missing Kubernetes API to view RBAC bindings by user

Usage:
rbac-lookup [subject query] [flags]
rbac-lookup [command]

Available Commands:
help Help about any command
version Print the version number of rbac-lookup

Flags:
--context string context to use for Kubernetes config
--gke enable GKE integration
-h, --help help for rbac-lookup
-k, --kind string filter by this RBAC subject kind (user, group, serviceaccount)
--kubeconfig string config file location
-o, --output string output format (normal, wide)

我们可以查看特定用户、service accout 、group绑定的role

1
2
3
4
5
[root@master-01 permission-manager]# rbac-lookup lishuai 
SUBJECT SCOPE ROLE
lishuai@example.com devops ClusterRole/edit
lishuai@example.com test ClusterRole/edit
lishuai@liangla.mobi cluster-wide ClusterRole/cluster-admin

-o wide 选项可以输出更多信息

1
2
3
4
5
[root@master-01 permission-manager]# rbac-lookup lishuai -o wide
SUBJECT SCOPE ROLE SOURCE
User/lishuai@example.com test ClusterRole/edit RoleBinding/lishuai-access-lishuai-edit
User/lishuai@example.com devops ClusterRole/edit RoleBinding/lishuai-access-lishuai-edit
User/lishuai@liangla.mobi cluster-wide ClusterRole/cluster-admin ClusterRoleBinding/git-912988434

也可以按RBAC Subject类型进行分类,参数--kind or -k 可以接受usergroupserviceaccount作为值。

1
2
3
4
5
6
7
8
9
10
11
12
13
[root@master-01 permission-manager]# rbac-lookup li --output wide -k serviceaccount
SUBJECT SCOPE ROLE SOURCE
ServiceAccount/:argocd-application-controller argocd Role/argocd-application-controller RoleBinding/argocd-application-controller
ServiceAccount/argocd:argocd-application-controller cluster-wide ClusterRole/argocd-application-controller ClusterRoleBinding/argocd-application-controller
ServiceAccount/argocd:argocd-applicationset-controller argocd Role/argocd-applicationset-controller RoleBinding/argocd-applicationset-controller
ServiceAccount/devops:nfs-client-provisioner cluster-wide ClusterRole/nfs-client-provisioner-runner ClusterRoleBinding/run-nfs-client-provisioner
ServiceAccount/devops:nfs-client-provisioner devops Role/leader-locking-nfs-client-provisioner RoleBinding/leader-locking-nfs-client-provisioner
ServiceAccount/kube-system:calico-kube-controllers cluster-wide ClusterRole/calico-kube-controllers ClusterRoleBinding/calico-kube-controllers
ServiceAccount/kube-system:calico-node cluster-wide ClusterRole/calico-node ClusterRoleBinding/calico-node
ServiceAccount/kube-system:endpointslice-controller cluster-wide ClusterRole/system:controller:endpointslice-controller ClusterRoleBinding/system:controller:endpointslice-controller
ServiceAccount/kube-system:replicaset-controller cluster-wide ClusterRole/system:controller:replicaset-controller ClusterRoleBinding/system:controller:replicaset-controller
ServiceAccount/kube-system:replication-controller cluster-wide ClusterRole/system:controller:replication-controller ClusterRoleBinding/system:controller:replication-controller

CATALOG
  1. 1. 利用k8s审计日志生成RBAC规则
    1. 1.1. 简介
    2. 1.2. 前提要求
    3. 1.3. 使用
  2. 2. 可视化集群中的rbac
    1. 2.1. 安装
    2. 2.2. 使用
      1. 2.2.1. 在线渲染
      2. 2.2.2. 本地渲染
  3. 3. 使用rbac-lookup查看集群内rbac绑定
    1. 3.1. 简介
    2. 3.2. 安装
    3. 3.3. 使用