TLS: Support separate certificate for HTTPS like Consul does

[olljanat]

Proposal

Consul supports configuration where tls.defaults is used to configure internal CA for mTLS in gRPC and RPC protocols.
Documentation also mentions that for security reason mTLS CA should be dedicated for this purpose.

Same time another CA (private or public) can be used for HTTPS traffic.

Full configuration example:

tls {
    defaults{
      verify_incoming = true
      verify_outgoing = true
      verify_server_hostname = true
      ca_file = "/opt/consul/agent-certs/ca.crt"
      cert_file = "/opt/consul/agent-certs/agent.crt"
      key_file = "/opt/consul/agent-certs/agent.key"
    }
}

tls {
    https {
        verify_incoming = false
        verify_outgoing = true
        ca_file = "/opt/node-cert/ca.crt"
        cert_file = "/opt/node-cert/node.crt"
        key_file = "/opt/node-cert/node.key"
    }
}

Currently Nomad supports only one CA which is used for all purposes and it would be nice to have this same functionality available in it.

Use-cases

Target is use mTLS for everything and avoid certificate errors when admins accessing to Consul UI with server name/IP address without going through load balancer/reverse proxy.

Attempted Solutions

Option 1: Enable TLS only to RPC
It is possible to enable TLS to RPC only and use non-encrypted HTTP for web traffic like this:

tls {
    http = false
    rpc = true
    ca_file = "/etc/nomad.d/agent-certs/ca.crt"
    cert_file = "/etc/nomad.d/agent-certs/agent.crt"
    key_file = "/etc/nomad.d/agent-certs/agent.key"
    rpc_upgrade_mode = false
}

However nowadays web browsers warns about that situation.

Option 2: Use same TLS for everything
It is possible to use certificate from external CA for mTLS too and it is possible to increase security by enabling verify_server_hostname = true setting because then these certificates must also contain name server.<region>.nomad

tls {
    http = true
    rpc = true
    ca_file = "/opt/node-cert/ca.crt"
    cert_file = "/opt/node-cert/node.crt"
    key_file = "/opt/node-cert/node.key"
    rpc_upgrade_mode = false
    verify_https_client = false
    verify_server_hostname = true
}

However it is bit hacky solution and that why not preferred.

Option 3: Offline Root CA + dedicated CA in Vault
I haven't fully tested this configuration but it seems that it would be possible to use Vault with external root to issue these certificates and then do configuration in way that:

1. External clients trust to offline root CA
2. ca_file value in Nomad configuration uses intermediate CA instead of actual root CA.

However, I'm not sure if that causes some other issues which I haven't think about yet.

[jrasell]

Hi @olljanat and thanks for raising this issue. This is an interesting idea and something that makes sense; I'll add it to our backlog, but note we would likely need some discussion on it as the changes would be broad and will need to be backwards compatible.

[olljanat]

@jrasell Sure backward compatibility is important.

It seems that in Consul it was handled by leaving old syntax intact so it is also possible configure example ca_file in root level of config file instead of tls.defaults.ca_file and it still can be overwritten with tls.https.ca_file

So in Nomad configuration like this would be backward compatible:

tls {
    http = true
    rpc = true
    ca_file = "/path/to/default-ca.crt"
    https {
        ca_file = "/path/to/https-ca.crt"
    }
}