So this is my current setup. I have a kubernetes cluster deployed with kubeadm. The loadbalancer is created with metallb. The metallb ip pool is a pool of private ips, so it cant be used in the public.

To access the public ip, a proxy server with HAProxy installed on it is used. At the moment, if I use letsencrypt-staging the certificate is issued i.e READY: True but with letsencrypt-prod the certificate is not issued i.e READY: False.

This is the HAProxy configuration; note the HAProxy server is not part of the cluster and it has a public IP. The puvlic IP is used in an A record in my DNS.

global
    log /dev/log local0
    log /dev/log local1 notice
    daemon
    maxconn 2048

defaults
    log     global
    mode    tcp
    option  tcplog
    timeout connect 5s
    timeout client  1m
    timeout server  1m

frontend https_passthrough
    bind *:443
    mode tcp
    option tcplog
    default_backend k8s_https_backend

frontend http_frontend
    bind *:80
    mode http
    option httplog
    default_backend k8s_http_backend 

backend k8s_https_backend
    mode tcp
    option tcplog
    option ssl-hello-chk
    balance roundrobin
    server k8s_ingress 192.168.55.100:443 check

backend k8s_http_backend
    mode http
    balance roundrobin
    server k8s_ingress 192.168.55.100:80 check

192.168.55.100 is the metallb IP.

This is my Ingress File

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argocd-ingress
  namespace: argocd
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
    nginx.ingress.kubernetes.io/ssl-passthrough: "false"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - argocd.*******.com
    secretName: argocd-server-tls
  rules:
  - host: argocd.*******.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: argocd-server
            port:
              number: 80

This is the clusterissuers I am currently using

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    # Let’s Encrypt STAGING endpoint (for testing)
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: cloud_admin@****.com
    privateKeySecretRef:
      name: letsencrypt-staging
    solvers:
      - http01:
          ingress:
            ingressClassName: nginx
            ingressTemplate:
              metadata:
                annotations:
                  cert-manager.io/disable-http01-validation-check: "true"
                  nginx.ingress.kubernetes.io/ssl-redirect: "false"
                  kubernetes.io/ingress.class: "nginx"
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: cloud_admin@****.com
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - http01:
        ingress:
          ingressClassName: nginx
          ingressTemplate:
              metadata:
                annotations:
                  cert-manager.io/disable-http01-validation-check: "true"
                  nginx.ingress.kubernetes.io/ssl-redirect: "false"
                  kubernetes.io/ingress.class: "nginx"

This is the challenge from the certificate issuing process

kubectl describe challenge -n argocd                argocd-server-tls-1-3416849378-2890379253
Name:         argocd-server-tls-1-3416849378-2890379253
Namespace:    argocd
Labels:       
Annotations:  
API Version:  acme.cert-manager.io/v1
Kind:         Challenge
Metadata:
  Creation Timestamp:  2025-11-13T11:40:40Z
  Finalizers:
    acme.cert-manager.io/finalizer
  Generation:  1
  Owner References:
    API Version:           acme.cert-manager.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Order
    Name:                  argocd-server-tls-1-3416849378
    UID:                   b05a264c-8f0f-45df-a552-c82b24cd1459
  Resource Version:        8523463
  UID:                     40cfc24b-3fc8-40a5-ac78-cd4f7e69dc2f
Spec:
  Authorization URL:  https://acme-v02.api.letsencrypt.org/acme/authz/2791831726/612221305556
  Dns Name:           argocd.****.com
  Issuer Ref:
    Group:  cert-manager.io
    Kind:   ClusterIssuer
    Name:   letsencrypt-prod
  Key:      SeXBB2IXSOaLRvVaHE32ilsl6Bae610pAARP5nFDzPA.pF3ApJXaciE0g_bs-yPkuzyaJN7zxN24sRVZ3OIeSn0
  Solver:
    http01:
      Ingress:
        Ingress Class Name:  nginx
        Ingress Template:
          Metadata:
            Annotations:
              cert-manager.io/disable-http01-validation-check:  true
              kubernetes.io/ingress.class:                      nginx
              nginx.ingress.kubernetes.io/ssl-redirect:         false
        Name:                                                   argocd-ingress
  Token:                                                        SeXBB2IXSOaLRvVaHE32ilsl6Bae610pAARP5nFDzPA
  Type:                                                         HTTP-01
  URL:                                                          https://acme-v02.api.letsencrypt.org/acme/chall/2791831726/612221305556/3lDCQg
  Wildcard:                                                     false
Status:
  Presented:   true
  Processing:  true
  Reason:      Waiting for HTTP-01 challenge propagation: failed to perform self check GET request 'http://argocd.****.com/.well-known/acme-challenge/SeXBB2IXSOaLRvVaHE32ilsl6Bae610pAARP5nFDzPA': Get "http://argocd.****.com/.well-known/acme-challenge/SeXBB2IXSOaLRvVaHE32ilsl6Bae610pAARP5nFDzPA": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
  State:       pending
Events:
  Type    Reason     Age                    From                     Message
  ----    ------     ----                   ----                     -------
  Normal  Started    5m59s                  cert-manager-challenges  Challenge scheduled for processing
  Normal  Presented  5m48s (x2 over 5m58s)  cert-manager-challenges  Presented challenge using HTTP-01 challenge mechanism

How can I get this to work i.e issue a prod certification to my subdomain