misterli's Blog.

misterli's Blog.

欢迎您访问misterli's Blog.

记录一次kyverno重启解决过程

记一次kyverno重启解决
收到报警kyverno的pod一直重启,查看一下发现重启了1w+次,还是有点疯狂

1
2
3
4
# lishuai @ MacBook-Pro in ~/.kube [10:22:22]
$ kubectl --kubeconfig config-test -n kyverno get pod
NAME READY STATUS RESTARTS AGE
kyverno-6d75c9bcbc-9wrrp 0/1 Error 10739 38d

日志如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
$ kubectl --kubeconfig config-test  -n kyverno logs -f kyverno-6d75c9bcbc-9cwmr
I1106 02:30:05.596951 1 version.go:17] "msg"="Kyverno" "Version"="v1.1.5"
I1106 02:30:05.597059 1 version.go:18] "msg"="Kyverno" "BuildHash"="(HEAD/7d76a4566784d366e93a8e0023cec7a9724ad465"
I1106 02:30:05.597072 1 version.go:19] "msg"="Kyverno" "BuildTime"="2020-04-28_11:05:16AM"
I1106 02:30:05.597394 1 config.go:79] CreateClientConfig "msg"="Using in-cluster configuration"
I1106 02:30:05.599167 1 client.go:246] Client/Poll "msg"="starting registered resources sync" "period"=10000000000
I1106 02:30:05.674374 1 util.go:69] CRDInstalled "msg"="CRD found" "kind"="ClusterPolicy"
I1106 02:30:05.674959 1 util.go:69] CRDInstalled "msg"="CRD found" "kind"="ClusterPolicyViolation"
I1106 02:30:05.675425 1 util.go:69] CRDInstalled "msg"="CRD found" "kind"="PolicyViolation"
I1106 02:30:05.677486 1 dynamicconfig.go:68] ConfigData "msg"="init configuration from commandline arguments"
I1106 02:30:05.677675 1 dynamicconfig.go:170] ConfigData "msg"="Init resource filters" "filters"=[{"Kind":"Event","Namespace":"*","Name":"*"},{"Kind":"*","Namespace":"kube-system","Name":"*"},{"Kind":"*","Namespace":"kube-public","Name":"*"},{"Kind":"*","Namespace":"kube-node-lease","Name":"*"},{"Kind":"Node","Namespace":"*","Name":"*"},{"Kind":"APIService","Namespace":"*","Name":"*"},{"Kind":"TokenReview","Namespace":"*","Name":"*"},{"Kind":"SubjectAccessReview","Namespace":"*","Name":"*"},{"Kind":"*","Namespace":"kyverno","Name":"*"}]
I1106 02:30:05.684098 1 certificates.go:28] Client "msg"="Generating new key/certificate pair for TLS"
I1106 02:30:06.136691 1 certificates.go:89] Client/submitAndApproveCertificateRequest "msg"="Old certificate request is deleted"
I1106 02:30:06.144440 1 certificates.go:98] Client/submitAndApproveCertificateRequest "msg"="Certificate request created" "name"="kyverno-svc.kyverno.cert-request"
I1106 02:30:06.158198 1 certificates.go:113] Client/submitAndApproveCertificateRequest "msg"="Certificate request is approved" "name"="kyverno-svc.kyverno.cert-request"
E1106 02:30:06.179647 1 main.go:239] setup "msg"="Failed to initialize TLS key/certificate pair" "error"="Unable to save TLS pair to the cluster: name is required"

看日志描述说证书无法保存到集群。

查看发现集群中已经存在证书

1
2
3
4
5
$ kubectl --kubeconfig config-test  -n kyverno get secrets
NAME TYPE DATA AGE
default-token-pdbwt kubernetes.io/service-account-token 3 223d
kubernetes.io/service-account-token 3 223d
kyverno-svc.kyverno.svc.kyverno-tls-pair kubernetes.io/tls 2 223d

查看证书内容发现也没什么问题

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ kubectl --kubeconfig config-test  -n kyverno get secrets kyverno-svc.kyverno.svc.kyverno-tls-pair -o yaml
apiVersion: v1
data:
tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURZakNDQWtxZ0F3SUJBZ0lVYmgzSk4zeExPQ1lGWXI2aU1jY2lXS1RhNkJ3d0RRWUpLb1pJaHZjTkFRRUwKQlFBd0VqRVFNQTRHQTFVRUF4TUhhM1ZpWlMxallUQWVGdzB5TURBek1qY3dOelEzTURCYUZ3MHlNVEF6TWpjdwpOelEzTURCYU1CWXhGREFTQmdOVkJBTVRDMnQ1ZG1WeWJtOHRjM1pqTUlJQklqQU5CZ2txaGtpRzl3MEJBUUVGCkFBT0NBUThBTUlJQkNnS0NBUUVBMXRVNnBaV2dsQzZwYnhVTG42NzhvYnJuVzlTdWdjb1BseTNPZk5ha09BRkgKdlpYWE91elRGTVA5OTVxK3J4eGNNTlpiMEU4ZWhmNVhpVDJ3YnhvcHRqSVNIRk0weEdIeStEL0dXa1daVElQeAo0K1VQRGxzKzVWZjZtekRPR1pFMzk0cTlmbHp2ZGtrYUtQaWhhS1FVNkdlVjRucnhpRy9qdFI5a1hqNW5OTGZpCjFySmVEdVg0bTRuWWFtODVQY056cWl3TTVmL2I3K3grNU80ZWZpWVUxK2VtUDZsVlFOejZFbkhVVkd3akFMR1UKOHNxdjBWL2NJbHFuR0tGa3pYRnMrOUovTzJzUXREdjd1ZWZ3YXgzZEI3RjBFWEM3Y1A2OUZQK2ZTMWxGYkZZcApPYkFpM3ZmYy9tZm5OaVJJV3JXQ1RrT3YrSkRFV0tNejArbS8xaWQ2QVFJREFRQUJvNEdyTUlHb01BNEdBMVVkCkR3RUIvd1FFQXdJRm9EQWRCZ05WSFNVRUZqQVVCZ2dyQmdFRkJRY0RBUVlJS3dZQkJRVUhBd0l3REFZRFZSMFQKQVFIL0JBSXdBREFkQmdOVkhRNEVGZ1FVUXNBL3JmK0hPRHFGZExYNEs4TWI4bGEzYUdJd1NnWURWUjBSQkVNdwpRWUlMYTNsMlpYSnVieTF6ZG1PQ0UydDVkbVZ5Ym04dGMzWmpMbXQ1ZG1WeWJtK0NGMnQ1ZG1WeWJtOHRjM1pqCkxtdDVkbVZ5Ym04dWMzWmpod1FLS3dBQk1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQWN1R1Arc2N4Wk5TSUQKRCs5UWE1bnViVEp5SS9xSVhxWUM2S3cyakRDMk1XQzJXTDl1ZXNiN1lCejd4SEVQVFFnR2lNUHpqZm9lNi9iSQpjeXBhckZVMGdRSFZQZFhVZFNSaU5qckplUm54UlhiT0pRY24rQlJpb1VaNkw0SUlZQmlIcjV4VEhHSnJ3eHJnCjNYalBSZ1pDSHdTVkJoWnd4YlArN0VIeEdLVjlyRmRacXRWZ3dObDdGNUdrRjlnc0hqRmw5eFcyMUNaZTlEa04KM1dkVW0vOW9mZ2VHSTRzalQ0cHFnb1F0cXVOM2tRVnFkc1VSQ0p3OXlwK1lrMjJ3RWZJcW93d2VSYzMybHRLZQozRVNjclhEL3hIbVJ3czY3bkYxcXJvZWFCMk51eW1jdEhZMUMrYkovdFBUWnB2eDIzWllvSExLaHl5RGsxYktBCk9yeWd1QU00Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
tls.key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUVwQUlCQUFLQ0FRRUExdFU2cFpXZ2xDNnBieFVMbjY3OG9icm5XOVN1Z2NvUGx5M09mTmFrT0FGSHZaWFgKT3V6VEZNUDk5NXErcnh4Y01OWmIwRThlaGY1WGlUMndieG9wdGpJU0hGTTB4R0h5K0QvR1drV1pUSVB4NCtVUApEbHMrNVZmNm16RE9HWkUzOTRxOWZsenZka2thS1BpaGFLUVU2R2VWNG5yeGlHL2p0UjlrWGo1bk5MZmkxckplCkR1WDRtNG5ZYW04NVBjTnpxaXdNNWYvYjcreCs1TzRlZmlZVTErZW1QNmxWUU56NkVuSFVWR3dqQUxHVThzcXYKMFYvY0lscW5HS0ZrelhGcys5Si9PMnNRdER2N3VlZndheDNkQjdGMEVYQzdjUDY5RlArZlMxbEZiRllwT2JBaQozdmZjL21mbk5pUklXcldDVGtPditKREVXS016MCttLzFpZDZBUUlEQVFBQkFvSUJBQ2tYOEhmci95TlpLWi9OCjdzTkV2WjVTR2g4K3Q0S3NHLzlYQzhCbGJsUW9Lb2poT0tKVTJxdUdNZlpDNjJhampoN3BZZmFlcThBRnZzakoKdkE0RWV5WVd2ZEFkT21LMk9idXl0MFpkT2MyaEQ0d0FMTGthU3hXamxwUkk2YU9LVzZKR0w2a1VMZG42Y2I2VQprSXRybDNROUhEYU9QZFZUVWNNN2xmOVJBSHpjdGhhczNaSXFmNSt0b05iRUhvQnpxTTYrN2NWVlNlVW5EN1VqCmlQNXlhY2F1bkcycDRDZkJLZFNqUnZZUG5uZmFnNGNIWkV2b3BleGo1a1pVTWlra0VLMG8zYlY5cHRqUktXVHkKUjMyTWVYMGl2M0tmdnpLVUUrK2d0TmFQalB5UGkwZEdmbkk2bUlFMHVhNks0U3RaMlNxN25oVENvRU16MDFJTQoxS2gzZ0FFQ2dZRUE3SUNoUGZrQTRrNkp5amNtQ0FWZ0ZlbjJVbFN1bUNnT2N0UWpMcmtFZ1pydHJ4MWpBV0s5CjczOE00WUpCTElyQ2ppVHZIQjM4d3pKb0FaSGxRV3lNTTE0a2JsK3FBL2gzZkJHdmFmTEZPVGNNbHpWcWV6WkMKcXhGRWdBWno3R0xmdExuOURBa0JBSW5aTG9XN21ZZ25sZmI3V1ZSVjI0N3p4VDhkNThWbmVVRUNnWUVBNkl0RAoyTFo4VktuWnB6QVVwS0VRV1VwdEJOQlF5bGpyZ2U2KzRxajJSeFhzVjVrVU43d2RER3J3RU5xSFZsNk5RemdtCjBvZlVjRHl4dkFhYVQ0cnE1K2Y3Ui9LT3BMTElOZ1FmY0dzeUdyY1NZOXM1V1MxdlVTbXpoQlZFQjVoUmlXOHAKdmZLeU1LMGowWjR1aDN5UGR2SzJicy8vcXNCei91RkFCcTVSRU1FQ2dZRUFxbGdyeTF0aWk1NU9HTnlJQkJiNwpFazJtSWI3azBxdG5YTVgzWVZ2YUp3L1VTdUU3d20vQXBwUTRUdVZtMUJKTjk5d2FiWUliNE95WmhTZjBuSjcyCmpMa3VQR0dqTDZEelR1WGVGczNKeUdBaUxYZEg3dDh5UGN6K0xjaDREcmRZc2UrVWwrcVVVakwzdjA2THhSWVEKalMrTDh0ZVB6OGl6UkVzbDJ4NlFYUUVDZ1lCVVBXYjFrWjNXbWJVRUVMSFp0Wk1UbFplS24rQTBmU1BMYk81dgpjNS9MdnBCZ1owN2dwZCtzQ08wd1hjbWJLeU5uVDJjWTZ5VzFCdmVuMG9pQitpUUFvSlB4eTFlTEtFekk3Sk5yCkNSb2NmV2RIRHpwbUtNUmpsWVMzZTNDcWc2NDk2Q3dwNkVwT3dkbnc3S21VWVRZamMrZE1tMExWMjJQcDJEVjIKZGgxZHdRS0JnUURPOHBIN1V3WVk4UDVxR1pJL3NncGYwRlBFdndLWUE1d2NiTS9HL2V6SDlnSE1XMWdPUkoxRQpLNzYxWGFJYzIrRnFLV1d2c3oyMjhKZFFlbDk2ZVZqM1BPZk1ncFZGNmVyL2NBQWxXVUlRVWQ3Qm1rK3k0dTdiCk91SU9wK1h4bDdRaEJBblIxZ0hJRDc2SmhydmFXK3UyRUNCQzRpdklsRkh2M0ZXaXR1U3NyQT09Ci0tLS0tRU5EIFBSSVZBVEUgS0VZLS0tLS0K
kind: Secret
metadata:
creationTimestamp: "2020-03-27T07:51:31Z"
name: kyverno-svc.kyverno.svc.kyverno-tls-pair
namespace: kyverno
resourceVersion: "234527968"
selfLink: /api/v1/namespaces/kyverno/secrets/kyverno-svc.kyverno.svc.kyverno-tls-pair
uid: c5b6cbee-6fff-11ea-8fe8-fa163e10b76c
type: kubernetes.io/tls

解析一下证书,发现也都正常

使用harbor代理缓存docker hub

Harbor 镜像代理缓存

概述

harbor v2.1 新增了代理缓存的功能,类似nexus可以使用harbor代理并缓存来自公共或私有镜像仓库的图像。并且从harbor v2.1.1开始,代理和缓存功能已更新,以与docker hub 的速率限制保持一致

我们可以使用代理缓存功能让一些访问受限环境能够访问互联网上的镜像,并且如果没有某个镜像,此时客户端第一次发起pull image 请求会从指定的代理仓库下载并缓存到harbor的仓库里,下次别的客户端再需要pull 这个镜像就无需从公网再去下载该镜像了,从而避免占用过多带宽或被docker hub 速率限制。

Harbor仅支持Docker Hub和Harbor注册中心的代理缓存。

创建代理缓存 项目

要使用harbor的代理缓存,我们需要先创建一个仓库,仓库指定使用哪个镜像仓库作为被代理缓存的镜像仓库,再创建一个开启镜像代理的项目,选择是用之前创建的仓库。

1、创建仓库

image-20201105132320193

提供者有以下几种,根据自己实际情况选择即可,我们这里选择docker hub

一次cert-manager无法续签

一次周末突然收到报警说证书即将到期,实际上我们的证书是使用cert-manager生成并自动续签的,按理说不会出现即将到期的问题

image-20200923112218825

查看cert-manager-cainjector日志发现有如下报错

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
I0920 21:36:32.855359       1 controller.go:170] cert-manager/inject-controller "level"=1 "msg"="updated object" "resource_kind"="APIService" "resource_name"="v1beta1.webhook.cert-manager.io" "resource_namespace"=""
I0920 21:36:32.855410 1 controller.go:242] cert-manager/controller-runtime/controller "level"=1 "msg"="Successfully Reconciled" "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.webhook.cert-manager.io"}
E0920 21:36:32.855257 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-dashboard-tls\" not found" "certificate"={"Namespace":"kube-system","Name":"com-dashboard-tls"} "secret"={"Namespace":"kube-system","Name":"com-dashboard-tls"}
E0920 21:36:32.855516 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-dashboard-tls\" not found" "certificate"={"Namespace":"kube-system","Name":"com-dashboard-tls"} "secret"={"Namespace":"kube-system","Name":"com-dashboard-tls"}
E0920 21:36:32.855690 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-dashboard-tls\" not found" "certificate"={"Namespace":"kube-system","Name":"com-dashboard-tls"} "secret"={"Namespace":"kube-system","Name":"com-dashboard-tls"}
E0920 21:36:32.855742 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-dashboard-tls\" not found" "certificate"={"Namespace":"kube-system","Name":"com-dashboard-tls"} "secret"={"Namespace":"kube-system","Name":"com-dashboard-tls"}
E0920 21:36:32.856413 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-kibana-tls\" not found" "certificate"={"Namespace":"elastic","Name":"com-kibana-tls"} "secret"={"Namespace":"elastic","Name":"com-kibana-tls"}
E0920 21:36:32.856444 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-kibana-tls\" not found" "certificate"={"Namespace":"elastic","Name":"com-kibana-tls"} "secret"={"Namespace":"elastic","Name":"com-kibana-tls"}
E0920 21:36:32.856578 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-kibana-tls\" not found" "certificate"={"Namespace":"elastic","Name":"com-kibana-tls"} "secret"={"Namespace":"elastic","Name":"com-kibana-tls"}
E0920 21:36:32.856607 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-kibana-tls\" not found" "certificate"={"Namespace":"elastic","Name":"com-kibana-tls"} "secret"={"Namespace":"elastic","Name":"com-kibana-tls"}
E0920 21:36:32.856710 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-kibana-tls\" not found" "certificate"={"Namespace":"elastic","Name":"com-kibana-tls"} "secret"={"Namespace":"elastic","Name":"com-kibana-tls"}
E0920 21:36:32.856745 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-kibana-tls\" not found" "certificate"={"Namespace":"elastic","Name":"com-kibana-tls"} "secret"={"Namespace":"elastic","Name":"com-kibana-tls"}
E0920 21:36:32.856793 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-kibana-tls\" not found" "certificate"={"Namespace":"elastic","Name":"com-kibana-tls"} "secret"={"Namespace":"elastic","Name":"com-kibana-tls"}
E0920 21:36:32.856881 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-kibana-tls\" not found" "certificate"={"Namespace":"elastic","Name":"com-kibana-tls"} "secret"={"Namespace":"elastic","Name":"com-kibana-tls"}
E0920 21:36:32.857349 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-dataworks-job-tls\" not found" "certificate"={"Namespace":"dataworks","Name":"com-dataworks-job-tls"} "secret"={"Namespace":"dataworks","Name":"com-dataworks-job-tls"}
E0920 21:36:32.857377 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-dataworks-job-tls\" not found" "certificate"={"Namespace":"dataworks","Name":"com-dataworks-job-tls"} "secret"={"Namespace":"dataworks","Name":"com-dataworks-job-tls"}
E0920 21:36:32.857427 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-dataworks-job-tls\" not found" "certificate"={"Namespace":"dataworks","Name":"com-dataworks-job-tls"} "secret"={"Namespace":"dataworks","Name":"com-dataworks-job-tls"}
E0920 21:36:32.857456 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-dataworks-job-tls\" not found" "certificate"={"Namespace":"dataworks","Name":"com-dataworks-job-tls"} "secret"={"Namespace":"dataworks","Name":"com-dataworks-job-tls"}
E0920 21:36:32.857630 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-dataworks-job-tls\" not found" "certificate"={"Namespace":"dataworks","Name":"com-dataworks-job-tls"} "secret"={"Namespace":"dataworks","Name":"com-dataworks-job-tls"}
E0920 21:36:32.857728 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-dataworks-job-tls\" not found" "certificate"={"Namespace":"dataworks","Name":"com-dataworks-job-tls"} "secret"={"Namespace":"dataworks","Name":"com-dataworks-job-tls"}
E0920 21:36:32.857764 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-dataworks-job-tls\" not found" "certificate"={"Namespace":"dataworks","Name":"com-dataworks-job-tls"} "secret"={"Namespace":"dataworks","Name":"com-dataworks-job-tls"}
E0920 21:36:32.857809 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-dataworks-job-tls\" not found" "certificate"={"Namespace":"dataworks","Name":"com-dataworks-job-tls"} "secret"={"Namespace":"dataworks","Name":"com-dataworks-job-tls"}
E0920 21:36:32.858259 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-dataworks-tls\" not found" "certificate"={"Namespace":"dataworks","Name":"com-dataworks-tls"} "secret"={"Namespace":"dataworks","Name":"com-dataworks-tls"}
E0920 21:36:32.858287 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-dataworks-tls\" not found" "certificate"={"Namespace":"dataworks","Name":"com-dataworks-tls"} "secret"={"Namespace":"dataworks","Name":"com-dataworks-tls"}
E0920 21:36:32.858358 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-dataworks-tls\" not found" "certificate"={"Namespace":"dataworks","Name":"com-dataworks-tls"} "secret"={"Namespace":"dataworks","Name":"com-dataworks-tls"}
E0920 21:36:32.858443 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-dataworks-tls\" not found" "certificate"={"Namespace":"dataworks","Name":"com-dataworks-tls"} "secret"={"Namespace":"dataworks","Name":"com-dataworks-tls"}
E0920 21:36:32.858480 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-dataworks-tls\" not found" "certificate"={"Namespace":"dataworks","Name":"com-dataworks-tls"} "secret"={"Namespace":"dataworks","Name":"com-dataworks-tls"}
E0920 21:36:32.858535 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-dataworks-tls\" not found" "certificate"={"Namespace":"dataworks","Name":"com-dataworks-tls"} "secret"={"Namespace":"dataworks","Name":"com-dataworks-tls"}
E0920 21:36:32.858641 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-dataworks-tls\" not found" "certificate"={"Namespace":"dataworks","Name":"com-dataworks-tls"} "secret"={"Namespace":"dataworks","Name":"com-dataworks-tls"}
E0920 21:36:32.858666 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-dataworks-tls\" not found" "certificate"={"Namespace":"dataworks","Name":"com-dataworks-tls"} "secret"={"Namespace":"dataworks","Name":"com-dataworks-tls"}
E0920 21:36:32.859427 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"cn-grafana-tls\" not found" "certificate"={"Namespace":"monitoring","Name":"cn-grafana-tls"} "secret"={"Namespace":"monitoring","Name":"cn-grafana-tls"}
E0920 21:36:32.859457 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"cn-grafana-tls\" not found" "certificate"={"Namespace":"monitoring","Name":"cn-grafana-tls"} "secret"={"Namespace":"monitoring","Name":"cn-grafana-tls"}
E0920 21:36:32.859508 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"cn-grafana-tls\" not found" "certificate"={"Namespace":"monitoring","Name":"cn-grafana-tls"} "secret"={"Namespace":"monitoring","Name":"cn-grafana-tls"}
E0920 21:36:32.859538 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"cn-grafana-tls\" not found" "certificate"={"Namespace":"monitoring","Name":"cn-grafana-tls"} "secret"={"Namespace":"monitoring","Name":"cn-grafana-tls"}
E0920 21:36:32.859636 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"cn-grafana-tls\" not found" "certificate"={"Namespace":"monitoring","Name":"cn-grafana-tls"} "secret"={"Namespace":"monitoring","Name":"cn-grafana-tls"}
E0920 21:36:32.859659 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"cn-grafana-tls\" not found" "certificate"={"Namespace":"monitoring","Name":"cn-grafana-tls"} "secret"={"Namespace":"monitoring","Name":"cn-grafana-tls"}
E0920 21:36:32.859727 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"cn-grafana-tls\" not found" "certificate"={"Namespace":"monitoring","Name":"cn-grafana-tls"} "secret"={"Namespace":"monitoring","Name":"cn-grafana-tls"}
E0920 21:36:32.862350 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"cn-grafana-tls\" not found" "certificate"={"Namespace":"monitoring","Name":"cn-grafana-tls"} "secret"={"Namespace":"monitoring","Name":"cn-grafana-tls"}
E0921 03:50:36.814654 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-kibana-tls\" not found" "certificate"={"Namespace":"elastic","Name":"com-kibana-tls"} "secret"={"Namespace":"elastic","Name":"com-kibana-tls"}
E0921 03:50:36.814654 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-kibana-tls\" not found" "certificate"={"Namespace":"elastic","Name":"com-kibana-tls"} "secret"={"Namespace":"elastic","Name":"com-kibana-tls"}
E0921 03:50:36.814747 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-kibana-tls\" not found" "certificate"={"Namespace":"elastic","Name":"com-kibana-tls"} "secret"={"Namespace":"elastic","Name":"com-kibana-tls"}
E0921 03:50:36.814748 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-kibana-tls\" not found" "certificate"={"Namespace":"elastic","Name":"com-kibana-tls"} "secret"={"Namespace":"elastic","Name":"com-kibana-tls"}

上述报错提示说在某些名称空间找不到与证书文件对应的certificate

查看对应的secret和certificate如下

1
2
3
4
5
$ kubectl get secrets -n monitoring
NAME TYPE DATA AGE
alertmanager-prometheus-operator-monito-alertmanager Opaque 1 82d
cn-grafana-tls kubernetes.io/tls 3 642d
cn-monitor-tls kubernetes.io/tls 3 642d
1
2
3
$ kubectl get certificate -n monitoring
NAME READY SECRET AGE
cn-monitor-tls True cn-monitor-tls 82d

发现实际上cn-grafana-tls是缺少对应的certificate的,同时com-kibana-tls这个secret缺少对应的certificate和ingress

使用unable to fetch certificate that owns the secret 在google搜索找到几个相关的issue

使用traefik为服务配置sso

使用traefik为服务配置sso

ForwardAuth

我们使用中经常会遇到一些网页没有登录身份验证,比如prometheus ui 和traefik ui 等,我们通常使用basic-auth 做登陆身份验证,这种虽然简单但是不利于实际使用中我们针对用户做限制,比如不希望某些人访问,basic-auth 无法实现类似的限制功能。

traefik有一个中间件ForwardAuth,ForwardAuth中间件可以将身份验证委派给外部服务。如果服务响应代码为2XX,则将授予访问权限并执行原始请求。否则,将返回来自身份验证服务器的响应。

AuthForward

配置如下:

1
2
3
4
5
6
7
8
9
10
11
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
name: test-auth
spec:
forwardAuth:
address: https://example.com/auth
authResponseHeaders:
- X-Forwarded-User
# trustForwardHeader: true

address选项定义外部身份验证服务器地址。

authResponseHeaders选项定义要从外部身份验证服务器复制到请求的标头列表

trustForwardHeader选项设置true表示信任所有现有的X-Forwarded-*标头

k8s之纵向扩缩容vpa

vpa

纵向容器自动缩放器(VPA)使用户无需设置最新的资源限制和对容器中容器的要求。配置后,它将根据使用情况自动设置请求,从而允许在节点上进行适当的调度,以便为每个Pod提供适当的资源量。它还将保持限制和初始容器配置中指定的请求之间的比率。

它既可以根据资源的使用情况来缩减对资源过度使用的Pod,也可以对资源需求不足的向上扩展Pod。

Kubernetes VPA 包含以下组件:

  • Recommender:用于根据监控指标结合内置机制给出资源建议值
  • Updater:用于实时更新 pod resource requests
  • History Storage:用于采集和存储监控数据
  • Admission Controller: 用于在 pod 创建时修改 resource requests

架构图

VPA体系结构图

主要流程是:Recommender在启动时从History Storage获取历史数据,根据内置机制修改VPA API object资源建议值。Updater监听VPA API object,依据建议值动态修改 pod resource requests。VPA Admission Controller则是用于 pod 创建时修改 pod resource requests。History Storage则是通过Kubernetes Metrics API采集和存储监控数据。

CPU 和内存的建议值均是依据历史数据+固定机制计算而成。

在 Kubernetes VPA 中缺少资源回收的机制,但Recommender却可以配合Updater动态修改 pod resource requests 的值。也就是说 pod resource requests - 推荐值 = 资源回收值。这间接实现了资源回收的功能。

使用hugo搭建博客

使用hugo搭建个人博客

Hugo是由Go语言实现的静态网站生成器。简单、易用、高效、易扩展、快速部署。

项目地址:https://github.com/gohugoio/hugo.git

安装hugo

这里使用二进制安装,到 Hugo Releases 下载对应的操作系统版本的Hugo二进制文件

1
2
3
wget https://github.com/gohugoio/hugo/releases/download/v0.69.2/hugo_0.69.2_Linux-64bit.tar.gz
tar xf hugo_0.69.2_Linux-64bit.tar.gz
mv hugo /usr/bin/

image-20200501161515193

解压后得到一个hugo文件,我们将文件复制到/usr/bin/下

生成站点

使用hugo在当前目录下生成一个名为lishuai的站点

avatar
misterli
大风起兮云飞扬
FRIENDS
baidu google