Hi everybody,
I’m deploying Axoniq Server EE (3 nodes) inside a mesh (istio) on kubernetes (1.18).
Everything is set with mutual tls.
There is an entry point with an ingress gateway (with TLS and certificate from cert-manager/letsencrypt)
Everything is working fine but to setup SSL this is a different story. By using a mesh with mtls we don’t need to setup SSL on ui and grpc in Axon directly.
Unfortunately for traffic going to grpc, we need to use grpc over http2. So to connect with an http tls connection that will be upgraded.
We are trying to connect with a spring boot application without success (unencrypted connection and exception during the connection).
I have not found any documentation about that.
Is it doable and I just missing some settings ?
It is not doable and I really need to set a dedicated loadbalancer for Axon with SSL configured in it ?
What is your advise ?
Thanks
Hi!
Now that sounds like an interesting setup, but I am a bit unclear about what exactly the setup has become and what fails. You are setting up a 3 node EE cluster, but using something (Istio?) to “inject” the TLS? So does AxonServer actually know about it? (Is “axoniq.axonserver.ssl.enabled” set to true and are certificates configured?)
As for your remark that “unfortunately … we need to use gRPC over HTTP/2”: AFAIK gRPC is HTTP/2 based, where the protocol is switched after the connection is established. That is how it is supposed to work.
Now connecting to a TLS protected endpoint with Java is indeed difficult to troubleshoot because the Java runtime is not very clear with its error messages. For example, if you connect the Axon client app to an SSL enabled AxonServer instance, without specifying “axon.axonserver.ssl-enabled=true” in the app’s properties, it will fail the connection with an IOException, without any further explanations. Is that perhaps what is happening to you?
As far as I know, you can configure an NGINX Ingress controller to pass gRPC traffic, but this is dependent on the Kubernetes implementation how. GCE’s default Ingress Controller is rather closed off, so unless you’re very skilled in adjusting it, you may not be able to pull that off. Having a single Ingress controller feed to both gRPC and HTTP backends is not something I have been able to get working.
Please let me know how I could help you with this by providing a bit more detail. As you are an EE user, feel free to use the support email address if you want to provide non-public details.
Cheers,
Bert Laverman
Hi Bert,
Thanks for your reply.
So first of all you solved my main problem that was due to missing axon.axonserver.ssl-enabled=true. This is really my bad as I have not find any documentation about those parameters in axon spring boot (but this is just my limited skill on it).
Anyway I will explain the setup to share it with everyone.
So yes I’m building a 3 node cluster.
About SSL, Axon is not aware of that ( axoniq.axonserver.ssl.enabled is set to false). The same for the ui (which dispay SSL disabled).
Istio is netwrok mesh. I’m not using Nginx Ingress Controller but Istio Ingress Controller (based on Envoy).
Despite grpc is based on http2, internally istio needs to know which one is really in used and so we need to explicitly set it to grpc (in a virtualservice kind. see it as an abstraction on top of a kubernetes service).
Istio is using gateway definition. So before your post I added a gateway and set GRPC protocol (plain text) on port 8124.
I switched it to HTTPS (HTTP2) on port 443.
So with ssl-enabled parameters, the client is now able to connect on the istio ingress controller with HTTP2+tls and to discuss with the server (through the virtualservice set with grpc protocol).
The only remaining point for us is to be able to use grpc mutual tls but this is an open bug/feature on istio side (it will permit to transparently use ssl for internal traffic too).
As a source:
Mickey,
thank you for your reply. I’m happy the client connection issue is solved. Let me know how you get on with Istio, it is still a bit of undiscovered territory for me.
Cheers,
Bert