

misterli's Blog.

misterli's Blog.

一次cert-manager无法续签



一次cert-manager无法续签

k8s ingress cert-manager

字数统计: 3.8k阅读时长: 22 min

 2020/09/23   Share

• 
• 
• 
• 
• 

一次周末突然收到报警说证书即将到期，实际上我们的证书是使用cert-manager生成并自动续签的，按理说不会出现即将到期的问题

查看cert-manager-cainjector日志发现有如下报错

I0920 21:36:32.855359       1 controller.go:170] cert-manager/inject-controller "level"=1 "msg"="updated object" "resource_kind"="APIService" "resource_name"="v1beta1.webhook.cert-manager.io" "resource_namespace"=""
I0920 21:36:32.855410       1 controller.go:242] cert-manager/controller-runtime/controller "level"=1 "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.webhook.cert-manager.io"}
E0920 21:36:32.855257       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-dashboard-tls\" not found" "certificate"={"Namespace":"kube-system","Name":"com-dashboard-tls"} "secret"={"Namespace":"kube-system","Name":"com-dashboard-tls"}
E0920 21:36:32.855516       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-dashboard-tls\" not found" "certificate"={"Namespace":"kube-system","Name":"com-dashboard-tls"} "secret"={"Namespace":"kube-system","Name":"com-dashboard-tls"}
E0920 21:36:32.855690       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-dashboard-tls\" not found" "certificate"={"Namespace":"kube-system","Name":"com-dashboard-tls"} "secret"={"Namespace":"kube-system","Name":"com-dashboard-tls"}
E0920 21:36:32.855742       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-dashboard-tls\" not found" "certificate"={"Namespace":"kube-system","Name":"com-dashboard-tls"} "secret"={"Namespace":"kube-system","Name":"com-dashboard-tls"}
E0920 21:36:32.856413       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-kibana-tls\" not found" "certificate"={"Namespace":"elastic","Name":"com-kibana-tls"} "secret"={"Namespace":"elastic","Name":"com-kibana-tls"}
E0920 21:36:32.856444       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-kibana-tls\" not found" "certificate"={"Namespace":"elastic","Name":"com-kibana-tls"} "secret"={"Namespace":"elastic","Name":"com-kibana-tls"}
E0920 21:36:32.856578       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-kibana-tls\" not found" "certificate"={"Namespace":"elastic","Name":"com-kibana-tls"} "secret"={"Namespace":"elastic","Name":"com-kibana-tls"}
E0920 21:36:32.856607       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-kibana-tls\" not found" "certificate"={"Namespace":"elastic","Name":"com-kibana-tls"} "secret"={"Namespace":"elastic","Name":"com-kibana-tls"}
E0920 21:36:32.856710       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-kibana-tls\" not found" "certificate"={"Namespace":"elastic","Name":"com-kibana-tls"} "secret"={"Namespace":"elastic","Name":"com-kibana-tls"}
E0920 21:36:32.856745       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-kibana-tls\" not found" "certificate"={"Namespace":"elastic","Name":"com-kibana-tls"} "secret"={"Namespace":"elastic","Name":"com-kibana-tls"}
E0920 21:36:32.856793       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-kibana-tls\" not found" "certificate"={"Namespace":"elastic","Name":"com-kibana-tls"} "secret"={"Namespace":"elastic","Name":"com-kibana-tls"}
E0920 21:36:32.856881       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-kibana-tls\" not found" "certificate"={"Namespace":"elastic","Name":"com-kibana-tls"} "secret"={"Namespace":"elastic","Name":"com-kibana-tls"}
E0920 21:36:32.857349       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-dataworks-job-tls\" not found" "certificate"={"Namespace":"dataworks","Name":"com-dataworks-job-tls"} "secret"={"Namespace":"dataworks","Name":"com-dataworks-job-tls"}
E0920 21:36:32.857377       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-dataworks-job-tls\" not found" "certificate"={"Namespace":"dataworks","Name":"com-dataworks-job-tls"} "secret"={"Namespace":"dataworks","Name":"com-dataworks-job-tls"}
E0920 21:36:32.857427       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-dataworks-job-tls\" not found" "certificate"={"Namespace":"dataworks","Name":"com-dataworks-job-tls"} "secret"={"Namespace":"dataworks","Name":"com-dataworks-job-tls"}
E0920 21:36:32.857456       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-dataworks-job-tls\" not found" "certificate"={"Namespace":"dataworks","Name":"com-dataworks-job-tls"} "secret"={"Namespace":"dataworks","Name":"com-dataworks-job-tls"}
E0920 21:36:32.857630       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-dataworks-job-tls\" not found" "certificate"={"Namespace":"dataworks","Name":"com-dataworks-job-tls"} "secret"={"Namespace":"dataworks","Name":"com-dataworks-job-tls"}
E0920 21:36:32.857728       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-dataworks-job-tls\" not found" "certificate"={"Namespace":"dataworks","Name":"com-dataworks-job-tls"} "secret"={"Namespace":"dataworks","Name":"com-dataworks-job-tls"}
E0920 21:36:32.857764       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-dataworks-job-tls\" not found" "certificate"={"Namespace":"dataworks","Name":"com-dataworks-job-tls"} "secret"={"Namespace":"dataworks","Name":"com-dataworks-job-tls"}
E0920 21:36:32.857809       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-dataworks-job-tls\" not found" "certificate"={"Namespace":"dataworks","Name":"com-dataworks-job-tls"} "secret"={"Namespace":"dataworks","Name":"com-dataworks-job-tls"}
E0920 21:36:32.858259       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-dataworks-tls\" not found" "certificate"={"Namespace":"dataworks","Name":"com-dataworks-tls"} "secret"={"Namespace":"dataworks","Name":"com-dataworks-tls"}
E0920 21:36:32.858287       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-dataworks-tls\" not found" "certificate"={"Namespace":"dataworks","Name":"com-dataworks-tls"} "secret"={"Namespace":"dataworks","Name":"com-dataworks-tls"}
E0920 21:36:32.858358       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-dataworks-tls\" not found" "certificate"={"Namespace":"dataworks","Name":"com-dataworks-tls"} "secret"={"Namespace":"dataworks","Name":"com-dataworks-tls"}
E0920 21:36:32.858443       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-dataworks-tls\" not found" "certificate"={"Namespace":"dataworks","Name":"com-dataworks-tls"} "secret"={"Namespace":"dataworks","Name":"com-dataworks-tls"}
E0920 21:36:32.858480       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-dataworks-tls\" not found" "certificate"={"Namespace":"dataworks","Name":"com-dataworks-tls"} "secret"={"Namespace":"dataworks","Name":"com-dataworks-tls"}
E0920 21:36:32.858535       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-dataworks-tls\" not found" "certificate"={"Namespace":"dataworks","Name":"com-dataworks-tls"} "secret"={"Namespace":"dataworks","Name":"com-dataworks-tls"}
E0920 21:36:32.858641       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-dataworks-tls\" not found" "certificate"={"Namespace":"dataworks","Name":"com-dataworks-tls"} "secret"={"Namespace":"dataworks","Name":"com-dataworks-tls"}
E0920 21:36:32.858666       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-dataworks-tls\" not found" "certificate"={"Namespace":"dataworks","Name":"com-dataworks-tls"} "secret"={"Namespace":"dataworks","Name":"com-dataworks-tls"}
E0920 21:36:32.859427       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"cn-grafana-tls\" not found" "certificate"={"Namespace":"monitoring","Name":"cn-grafana-tls"} "secret"={"Namespace":"monitoring","Name":"cn-grafana-tls"}
E0920 21:36:32.859457       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"cn-grafana-tls\" not found" "certificate"={"Namespace":"monitoring","Name":"cn-grafana-tls"} "secret"={"Namespace":"monitoring","Name":"cn-grafana-tls"}
E0920 21:36:32.859508       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"cn-grafana-tls\" not found" "certificate"={"Namespace":"monitoring","Name":"cn-grafana-tls"} "secret"={"Namespace":"monitoring","Name":"cn-grafana-tls"}
E0920 21:36:32.859538       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"cn-grafana-tls\" not found" "certificate"={"Namespace":"monitoring","Name":"cn-grafana-tls"} "secret"={"Namespace":"monitoring","Name":"cn-grafana-tls"}
E0920 21:36:32.859636       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"cn-grafana-tls\" not found" "certificate"={"Namespace":"monitoring","Name":"cn-grafana-tls"} "secret"={"Namespace":"monitoring","Name":"cn-grafana-tls"}
E0920 21:36:32.859659       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"cn-grafana-tls\" not found" "certificate"={"Namespace":"monitoring","Name":"cn-grafana-tls"} "secret"={"Namespace":"monitoring","Name":"cn-grafana-tls"}
E0920 21:36:32.859727       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"cn-grafana-tls\" not found" "certificate"={"Namespace":"monitoring","Name":"cn-grafana-tls"} "secret"={"Namespace":"monitoring","Name":"cn-grafana-tls"}
E0920 21:36:32.862350       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"cn-grafana-tls\" not found" "certificate"={"Namespace":"monitoring","Name":"cn-grafana-tls"} "secret"={"Namespace":"monitoring","Name":"cn-grafana-tls"}
E0921 03:50:36.814654       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-kibana-tls\" not found" "certificate"={"Namespace":"elastic","Name":"com-kibana-tls"} "secret"={"Namespace":"elastic","Name":"com-kibana-tls"}
E0921 03:50:36.814654       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-kibana-tls\" not found" "certificate"={"Namespace":"elastic","Name":"com-kibana-tls"} "secret"={"Namespace":"elastic","Name":"com-kibana-tls"}
E0921 03:50:36.814747       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-kibana-tls\" not found" "certificate"={"Namespace":"elastic","Name":"com-kibana-tls"} "secret"={"Namespace":"elastic","Name":"com-kibana-tls"}
E0921 03:50:36.814748       1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"com-kibana-tls\" not found" "certificate"={"Namespace":"elastic","Name":"com-kibana-tls"} "secret"={"Namespace":"elastic","Name":"com-kibana-tls"}

上述报错提示说在某些名称空间找不到与证书文件对应的certificate

查看对应的secret和certificate如下

$ kubectl get secrets -n monitoring
NAME                                                          TYPE                                  DATA   AGE
alertmanager-prometheus-operator-monito-alertmanager          Opaque                                1      82d
cn-grafana-tls                                                kubernetes.io/tls                     3      642d
cn-monitor-tls                                                kubernetes.io/tls                     3      642d

$ kubectl get certificate -n monitoring
NAME             READY   SECRET           AGE
cn-monitor-tls   True    cn-monitor-tls   82d

发现实际上cn-grafana-tls是缺少对应的certificate的，同时com-kibana-tls这个secret缺少对应的certificate和ingress

使用unable to fetch certificate that owns the secret 在google搜索找到几个相关的issue

https://github.com/jetstack/cert-manager/issues/1944

https://github.com/jetstack/cert-manager/issues/1489

https://github.com/SeldonIO/seldon-core/issues/2101

这个问题像是之前某次删除ingress资源但是cert-manager并没有同步删除secret,如果想同步删除可以在cert-manager控制其中加入–enable-certificate-owner-ref这个标志。(可能和cert-manager升级有关系)

但是我们这里没有设置该标志，官方提供了一个清理脚本，https://github.com/richstokes/k8s-scripts/tree/master/clean-orphaned-secrets-cert-manager

执行脚本 bash xxx you-namespace 如 bash clean.sh monitoring

清理之后发现会新创建certificate并绑定对应的证书secret

$ kubectl get certificate -n monitoring
NAME             READY   SECRET           AGE
cn-grafana-tls   True    cn-grafana-tls   47h
cn-monitor-tls   True    cn-monitor-tls   82d

原文作者：Mister Li

原文链接：https://www.lishuai.fun/2020/09/23/cert-manager%E6%97%A0%E6%B3%95%E7%BB%AD%E7%AD%BE/

发表日期：September 23rd 2020, 10:32:39 pm

更新日期：May 4th 2022, 12:34:08 am

版权声明：本文采用知识共享署名-非商业性使用 4.0 国际许可协议进行许可

• Next Post
  使用harbor代理缓存docker hub
• Previous Post
  使用traefik为服务配置sso

Powered by Hexotheme Archer

©豫ICP备19035521号

PV: :)

CATALOG

• Archive
• Tag
• Cate

Total : 48



2024

• 09/11 prometheus监控集群外机器的node-exporter的两种方式
• 06/11 Cloudflare 代理dockerhub的两种方式

2023

• 05/05 记一次kuboard故障—Etcd空间爆满
• 05/05 K8s环境清理节点上旧的镜像
• 05/05 docker/buildkit 构建问题
• 05/05 分析redis 大key
• 05/05 Umami 安装并升级到v2.x

2022

• 06/15 SSO之使用Pomerium保护您的网站
• 05/09 使用sloop 监控kubernetes event
• 05/02 阿里云ACK集群 Ingress-nginx配置日志输出request body和request headers信息
• 04/06 K8s RBAC工具之akcess

2021

• 12/31 使用s3(minio)为kubernetes提供pv存储
• 09/28 Kubernetes 的 API 流量查看器---mizu
• 09/22 记一次longhorn 组件重启导致pv无法正常挂载
• 09/09 在Gitlab中规范提交的commit message的格式
• 09/06 监控minio的多种姿势
• 09/01 在k8s上部署支持多租户的minio
• 08/25 k8s 开启临时容器进行debug
• 08/12 史上最全之K8s使用nfs作为存储卷的五种方式
• 08/10 三个小工具带你进一步了解K8S集群内RBAC
• 08/09 使用Permission manager动态为k8s集群创建用户及kubeconfig
• 08/05 使用RBAC Manager 简化Kubernetes 中的授权
• 08/02 使用gitlab账号登陆Kubesphere
• 08/01 k8s安装cert-manager及签发泛域名证书
• 07/16 kubesphere 3.1.1 接入自定义 Prometheus
• 07/13 阿里云slb https代理harbor 总结
• 07/06 prometheus使用missing-container-metrics监控pod oomkill
• 04/01 使用ssl-exporter监控ssl证书过期时间
• 01/27 prometheus使用PrometheusAlert进行聚合报警
• 01/22 chaos-mesh 混沌工程（下）
• 01/21 Chaos-mesh 混沌工程(上）
• 01/06 扫描k8s集群中漏洞
• 01/06 利用k8s审计日志生成RBAC规则
• 01/05 同步secret和config到指定namespace

2020

• 12/08 kubevela之appfile详解（下）
• 12/08 kubevela之appfile详解（上）
• 12/03 kubevela小试牛刀（1）
• 12/02 使用Rancher的rke安装k8s
• 11/27 Prometheus监控redis及redis-cluster
• 11/18 Argo CD使用ding talk通知同步状态
• 11/17 k8s集群中pod镜像版本检查
• 11/16 阿里云rds 备份恢复到本地
• 11/06 记录一次kyverno重启解决过程
• 11/05 使用harbor代理缓存docker hub
• 09/23 一次cert-manager无法续签
• 09/07 使用traefik为服务配置sso
• 09/02 k8s之纵向扩缩容vpa
• 09/01 使用hugo搭建博客

 kubernetes  rbac  k8s  vulnerability  ingress-nginx  RBAC  argocd  devops  gitlab  harbor  slb  hugo  pv  csi  s3  kubesphere  prometheus  sso  kubevela  k8s准入控制  kyverno  longhorn  oomkill  event  configmap  secret  sync  ingress  cert-manager  chaos-mesh  minio  vpa  动态扩缩容  k8s kubevela  oamm  报警  ssl-exporter  traefik  混沌工程  rke k8s  pvc  nfs  prometheus redis  umami  etcd  kuboard  docker  ci  buildkit  redis  dockerhub  proxy  Cloudflare



缺失模块，请参考主题文档进行安装配置：https://github.com/fi3ework/hexo-theme-archer#%E5%AE%89%E8%A3%85%E4%B8%BB%E9%A2%98



