TLS Handshake Error with Vault after Airflow and provider updates

[khe-cw-de]

Apache Airflow Provider(s)

hashicorp

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==8.19.0
apache-airflow-providers-common-compat==1.2.1
apache-airflow-providers-common-io==1.4.2
apache-airflow-providers-fab==1.5.0
apache-airflow-providers-ftp==3.11.1
apache-airflow-providers-hashicorp=4.0.0
apache-airflow-providers-http==4.13.3
apache-airflow-providers-imap==3.7.0
apache-airflow-providers-microsoft-azure>=9.0.1
apache-airflow-providers-microsoft-mssql==3.9.2
apache-airflow-providers-postgres>=5.10.2
apache-airflow-providers-smtp==1.8.0
apache-airflow-providers-sqlite==3.9.0

Apache Airflow version

2.10.3

Operating System

apache/airflow:2.10.3-python3.11

Deployment

Other Docker-based deployment

Deployment details

Hashicorp Vault version: 1.16.6+ent

What happened

After updating Airflow (from 2.9.0 to 2.10.3) as well as Hashicorp provider (from 3.6.4 to 4.0.0), secrets and connections available in our running Vault instance are not found anymore during DAG runs / compilation.

It seems that this is related to a TLS handshake error to our running Vault instance first observed after the update:

Jan 08 14:04:10 CWCOL0VXDTIMG02 vault[817]: 2025-01-08T14:04:10.018+0100 [INFO]  http: TLS handshake error from 10.74.73.21:56698: EOF
Jan 08 14:04:10 CWCOL0VXDTIMG02 vault[817]: 2025-01-08T14:04:10.193+0100 [INFO]  http: TLS handshake error from 10.74.73.21:56748: EOF
Jan 08 14:04:10 CWCOL0VXDTIMG02 vault[817]: 2025-01-08T14:04:10.306+0100 [INFO]  http: TLS handshake error from 10.74.73.21:56788: EOF

What you think should happen instead

When downgrading to Airflow 2.9.0 as well as Hashicorp provider 3.6.4, this error doesn't occur, and all Vault secrets and connections can be accessed again from within the DAG run.

How to reproduce

1. Set up an Apache Airflow environment with version 2.10.3 using the apache/airflow:2.10.3-python3.11 Docker image.
2. Configure the HashiCorp Vault connection in Airflow using the apache-airflow-providers-hashicorp==4.0.0 provider. Ensure the Vault instance is accessible.
3. Verify that the Vault instance is running with version 1.16.6+ent and is properly configured to store secrets/connections.
4. Attempt to retrieve a secret from the Vault using an Airflow task or within the DAG.
5. Observe the logs for TLS handshake errors, as described in the "What happened" section.

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[potiuk]

I think it might be related to a kubernetes bug and it is triggered via different things being upgraded as well.

I guess together with the upgrade you upgraded to a newer version of K8S and you hit this issue kubernetes/kubernetes#109022 - or maybe hvac version incompatibility (hvac is the hashicorp library used to communicate with hashicorp). You might want to narrow it down by limiting things to upgrade and diagnose what is the faulty component:

1. downgrade proivder separately on Airflow 2.10.0 - without downloading hvac
2. downgrade hvac separately to the version that was present in constraints of Airlfow 2.9.0
3. downgrade k8s version (or any other thing you upgraded that might contain the faulty go version).

You can then see which one works. My guess it's the hvac upgrade that caused it, then you can report the issue to hvac repository.

Please let us know what your investigation brings.