SSL-problem for http port in OpenShift/Kubernetes

AlexGro · March 17, 2021, 11:01am

Dear Support-Team,

our environment is an OpenShift-Cluster in the version 4.5.30, which is based on Kubernetes.
We deployed the axon-server from our own Dockerfile which is based on your axon-server-image but added our keystore to the image.
Without SSL everything works perfect and we reach the gui over the url:
axon-staging.intern.company.de

Dockerfile

FROM docker.io/axoniq/axonserver:4.3.3
USER 0
COPY ./axon.p12 /resources/axon.p12
COPY ./axon.cer /resources/axon.cer
COPY ./key.pem /resources/key.pem

The axon-dev.p12 is a keystore which contains the certification chain:
Root.ca
Issuing.ca
Axon-staging.certificate

The axon-staging-cert is a self-signed cert from our issuing.ca
We set following environment variables for the start of the container:

ENV-VARS

  - name: TZ
    value: Europe/Berlin
  - name: AXONIQ_AXONSERVER_NAME
    value: axonserver-staging
  - name: AXONIQ_AXONSERVER_HOSTNAME
    value: axonserver-staging
  - name: axoniq.axonserver.accesscontrol.enabled
    value: 'true'
  - name: axon.axonserver.token
    valueFrom:
      secretKeyRef:
        name: axonserver-token
        key: WebHookSecretKey

SSL enabled for HTTP server

      - name: server.ssl.key-store-type
        value: PKCS12
      - name: server.ssl.key-store
        value: /resources/axon.p12
      - name: server.ssl.key-store-password
        value: axonserver
      - name: server.ssl.key-alias
        value: axonserver 
      - name: security.require-ssl
        value: 'true'

SSL enabled for gRPC server

      - name: axoniq.axonserver.ssl.enabled
        value: 'true'
      - name: axoniq.axonserver.ssl.cert-chain-file
        value: /resources/axon.cer
      - name: axoniq.axonserver.ssl.private-key-file
        value: /resources/key.pem

The SSL for the gRPC port seems to work, because when we only encrypt the gRPC port, we don’t see the yellow warning „ssl disabled“ in the GUI.
Unfortunately we cant reach the gui over the http port, when we encrypt the http-connection.
We get the following error:

Bad Request
This combination of host and port requires TLS.

When we open the certificate on the URL it says that the certificate is valid.
So we think that Axon can’t handle the certificate.

What is our problem and how can we solve it?
Thank you very much in advance

Kind regards,
Alexander

milendyankov · March 17, 2021, 12:14pm

Hello and welcome Alexander,

I do hope someone can help you with the issue at hand soon.

Meanwhile after reading

Dear Support-Team

I feel I need to to clarify few things and set the expectations straight. This is an open and publicly available discussion platform and not an official support channel. That means

everything you post here will be publicly available and searchable to anyone out there. Please be careful to not post any sensitive data (certificates, passwords, keys, …)
the AxonIQ team is doing its best to answer questions and engage in discussions here but it’s a best effort based approach. There is no guarantee a solution will be provided nor there are specific response times promises.

I’m sorry if I’m pointing out the obvious. I just wanted to make sure there is no misunderstanding about it. If you are representing a customer and looking for official support channels please DM me and I’ll try to help. Otherwise you are more than welcome to discuss just about anything Axon related here.

AlexGro · March 17, 2021, 12:28pm

Hello Milen,

thank you for your answer.
I know that this is not the support team and that the community might help us, I just expressed myself poorly. I’m sorry if you misunderstood it.

We hope that the community is able to help us, if not, we might reach out to you.
Thank you very much.

Bert_Laverman · March 17, 2021, 12:35pm

Alexander,
I see some mixed things in here, so please bear with me. You say you can “open the certificate on the URL” but also that you “encrypt the http-connection”, and that confuses me a bit as to what you actually did. Do you have the tool “curl”? Can you try “curl -v https://axon-staging.intern.company.de/actuator/info” and tell me what output this generates?

The assumption is here that you have a Service that either uses a LoadBalancer type, or else you have an Ingress defined. Please note that you may have more components in play here than just Axon Server, and the front-end ingress has a valid certificate, while the connection between that ingress controller and Axon Server is not using TLS.

Some other things I notice: you use “AXONIQ_AXONSERVER_NAME” (all caps) as an environment variable, and that will indeed be translated by Spring-boot to the correct property, but others you write in lowercase. Wouldn’t it be easier to use a ConfigMap or Secret so you can use an actual property file? We have some examples of how that works in the Running Axon Server GitHub repo. Similarly, I would suggest that you mount the certificates using a Secret rather than as copied files in a Docker image. Otherwise, that would require you to create a new image whenever the certificate changes. (e.g. when its validity expires)

Cheers,
Bert Laverman

AlexGro · March 17, 2021, 2:31pm

Hello Bert,
thank you for your reply.
When we encrypt the openshift route, we see the valid certification.

Actually we did have the service configured as LoadBalancer. I’ve changed it to ClusterIP but the result is the same.
Your idea of using a ConfigMap is good, we will take a look at it, after we get the server running.

The curl command shows the following:

* About to connect() to staging-axon.intern.company.de port 443 (#0)
*   Trying 172.xx.xx.xx...
* Connected to staging-axon.intern.company (172.xx.xx.xx) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: E=hostmaster@company.de,CN=staging-axon.intern.company,OU=Openshift Operations,O=COMPANY,L=XX,ST=XX,C=XX
*       start date: Mar 12 10:46:53 2021 GMT
*       expire date: Mar 12 10:46:53 2023 GMT
*       common name: staging-axon.intern.company
*       issuer: CN=company Issuing CA,DC=intern,DC=company,DC=de
> GET /actuator/info HTTP/1.1
> User-Agent: curl/7.29.0
> Host: staging-axon.intern.company.de
> Accept: */*
> 
< HTTP/1.1 400 
< Content-Type: text/plain;charset=ISO-8859-1
< Connection: close
< Set-Cookie: e848677014042990746df48xxx=77da5bdf2xx; path=/; HttpOnly; Secure
< 
Bad Request
This combination of host and port requires TLS.
* Closing connection 0

We are going to look at the problem now from the OpenShift side - we think the problem might be as you expected the service.

Thank you very much for your caring, Bert.

Kind regards,
Alexander

Edit: Because I am new I cant post more than 2 links - so please see the URL as it is, ended with .de

AlexGro · March 17, 2021, 2:53pm

Hello Bert,
we are using the docker.io/axoniq/axonserver:4.3.3 with self-signed-certificates.
Is it possible in this version?
Furthermore we are using the same certificate for the http and for the gRPC port - is this correct?
Thank you in advance
Alexander

Bert_Laverman · March 17, 2021, 2:58pm

Alexander,
yes, TLS is no problem for Axon Server SE. Please look at the example in the Running Axon Server repository, in folder “1-local/5-tls-se”.

Bert

AlexGro · March 18, 2021, 6:38am

Hello everyone,
the problem was in our OpenShift Route.
We used “tls: edge”. This means something like the ingress gives the data unencrypted to the cluster.
We now used “tls: passthrough” for the route. This seems to work and the gui is using https.

Thanks to everyone who helped us, especially Bert who gave us the right direction!