Mutual TLS communication between Istio mesh

Mutual TLS communication is about trusting each other between client and server. It is different from standard TLS that only client need to trust the server. In mutual TLS, server also need to trust client.

In Istio mesh, this type of communication can be archive directly because all of the proxy share same root CA. Also mutual TLS communication between multiple Istio mesh also can be archive by sharing the root CA via multicluster configuration. But what should we do when we cannot share the root CA to other?

This article give you another prespective how to archive mutual TLS communication between Istio mesh using 3rd root CA that both of the mesh agree and trust, as shown in picture below:

_config.yml

Environment

two Kubernetes cluster with Istio mesh enabled
a domain (that use to access the service) that resolve to istio-ingressgateway (that should be public, we can simulate using dnsmasq but I am to lazy to do that)
step & step-ca to generate certificate

Configuration

Certificate

Before configuring the cluster, we need to generate 3rd root CA, client & server certificate that will be use to make mutual TLS, we will using step certificate for that.

step certificate create a-b-root-ca root-cert.pem root-key.pem --profile root-ca \
--kty RSA --no-password --insecure --not-after 87600h

step certificate create cluster-a-client cluster-a-client-cert.pem cluster-a-client-key.pem \
--profile leaf --no-password --insecure --ca ./root-cert.pem --ca-key ./root-key.pem --kty RSA \
--size 2048 --not-after 8760h --san helloworld.zufar.io

step certificate create cluster-b-server cluster-b-server-cert.pem cluster-b-server-key.pem \
--profile leaf --no-password --insecure --ca ./root-cert.pem --ca-key ./root-key.pem --kty RSA \
--size 2048 --not-after 8760h --san helloworld.zufar.io

Note: I still need more context how we define the subject alternative name, will comeback here to fix this if necessary. Because we will using helloworld.zufar.io as the service domain.

Cluster A

Cluster A will act as client side of the mutual TLS communication.

Create namespace client

apiVersion: v1
kind: Namespace
metadata:
  name: client
  labels:
    istio-injection: enabled

Create certificate for client

kubectl -n client create secret generic \
cluster-a-client --from-file=cluster-a-client-cert.pem \
--from-file=cluster-a-client-key.pem --from-file=root-cert.pem

Create client deployment and mount the certificate via Istio annotation

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sleep
  namespace: client
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sleep
  template:
    metadata:
      annotations:                                                                                       
        sidecar.istio.io/userVolumeMount: '[{"name":"cluster-a-client", "mountPath":"/etc/cluster-a-client", "readonly":true}]'
        sidecar.istio.io/userVolume: '[{"name":"cluster-a-client", "secret":{"secretName":"cluster-a-client"}}]'
      labels:
        app: sleep
    spec:
      containers:
      - name: sleep
        image: curlimages/curl
        command: ["/bin/sleep", "3650d"]
        imagePullPolicy: IfNotPresent

This annotation will mount certificate inside a secret to istio-proxy sidecar in the /etc/cluster-a-client.

Create VirtualService object

virtualService will upgrade HTTP connection to HTTPS connection when accessing helloworld.zufar.io

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: helloworld-zufar-io-vs
  namespace: client
spec:
  hosts:
  - helloworld.zufar.io
  http:
  - match:
    - port: 80
    route:
    - destination:
        host: helloworld.zufar.io
        subset: tls-origination
        port:
          number: 443

VirtualService add a subset name tls-origination to HTTPS connection. This subset will be used by DestinationRule.

Create ServiceEntry object

ServiceEntry object will enable Istio control feature when accessing helloworld.zufar.io. We will inject certificate so we need the Istio control feature

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: helloworld-zufar-io-se
  namespace: client
spec:
  hosts:
  - helloworld.zufar.io
  ports:
  - number: 443
    name: https-port-for-tls-origination
    protocol: HTTPS
  resolution: DNS
  location: MESH_EXTERNAL

Create DestinationRule object

DestinationRule object will inject certificate that require by mutual TLS

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: helloworld-zufar-io-dr
  namespace: client
spec:
  host: helloworld.zufar.io
  subsets:
  - name: tls-origination
    trafficPolicy:
      portLevelSettings:
      - port:
          number: 443
        tls:
          mode: MUTUAL
          clientCertificate: /etc/cluster-a-client/cluster-a-client-cert.pem
          privateKey: /etc/cluster-a-client/cluster-a-client-key.pem
          caCertificates: /etc/cluster-a-client/root-cert.pem

Thats all on the client side, for making mutual TLS communication, client can use HTTP communication http://helloworld.zufar.io and no code change. Istio sidecar will upgrade to HTTPS and enable mutual TLS!

Cluster B

Cluster B will act as server side, have service helloworld inside. The traffic will enter istio-ingressgateway and terminate mutual TLS there. Istio then forward traffic as plain http to helloworld service.

Create certificate for istio-ingressgateway

The certificate must be on istio-system namespace. Or if you have custom istio-ingressgateway, it must place in the same namespace as the istio-ingressgateway.

kubectl create -n istio-system secret generic cluster-b-server-cert \
--from-file=key=cluster-b-server-key.pem \
--from-file=cert=cluster-b-server-cert.pem \
--from-file=cacert=root-cert.pem

Create namespace service

apiVersion: v1
kind: Namespace
metadata:
  name: service
  labels:
    istio-injection: enabled

Create deployment and service for helloworld

apiVersion: v1
kind: Service
metadata:
  name: helloworld
  namespace: service
  labels:
    app: helloworld
spec:
  ports:
  - port: 5000
    name: http
  selector:
    app: helloworld
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: helloworld-v1
  namespace: service
  labels:
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: helloworld
      version: v1
  template:
    metadata:
      labels:
        app: helloworld
        version: v1
    spec:
      containers:
      - name: helloworld
        image: docker.io/istio/examples-helloworld-v1
        resources:
          requests:
            cpu: "100m"
        imagePullPolicy: IfNotPresent #Always
        ports:
        - containerPort: 5000

Create Gateway object

Gateway object will serve the HTTPS communication and add the certificate to istio-ingressgateway.

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: service-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      tls:
        httpsRedirect: true
      hosts:
        - "*"
    - port:
        number: 443
        name: https
        protocol: HTTPS
      hosts:
        - "*"
      tls:
        mode: MUTUAL
        credentialName: cluster-b-server-cert

Note: I need more context about why the hosts should asterisk (*). Currently, my understanding is because the root CA is created without any DNS attach to it. I will get back as I get more understanding about this.

Create VirtualService object

VirtualService object will match host helloworld.zufar.io from service-gateway and route to the correct helloworld service in service namespace.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: helloworld-zufar-io-vs
  namespace: istio-system
spec:
  hosts:
  - helloworld.zufar.io
  gateways:
  - service-gateway
  http:
  - route:
    - destination:
        host: helloworld.service.svc.cluster.local
        port:
          number: 5000

Testing

We cannot se the mutual TLS communication because it is abstracted by Istio sidecar and Istio-ingressgateway. Client can use HTTP communication http://helloworld.zufar.io and Istio sidecar will upgrade to HTTPS and enable mutual TLS. To see the packet, we can use wireshark with sniff to capture the packet. We can capture packet inside sidecar (istio-proxy) with this command from cluster A:

kubectl sniff POD_NAME -c istio-proxy -n client \
-f "port 443" \
-p -o - | /Applications/Wireshark.app/Contents/MacOS/Wireshark -i -

Conclusion

Mutual TLS enabled trust between client and server. Mutual TLS can be archive by difference approach, via Istio multicluster or via this way. Thank you!

Written on June 12, 2020