Airbyte support for non-HTTP async connectors

[nitesh-sinha]

Hi team,

Does anyone know if Airbyte supports async non-HTTP source development via their CDK? We're evaluating NATS(instead of Kafka) for our project and the recommended NATS Python client is an async one(https://github.com/nats-io/asyncio-nats) - uses asyncio. So, our overridden read method for NATS source contains an async call(using Python's await keyword) using NATS Python SDK. However this async call will require the read method to be an async def which does not conform to the BaseSource class' read method definition. Essentially it looks like Airbyte's Python CDK is not meant to be used for developing connectors which support asyncio based calls.

Furthermore Airbyte's Python CDK README(https://github.com/airbytehq/airbyte/blob/master/airbyte-cdk/python/README.md) mentions CDK for Async HTTP endpoints (request-poll-wait style endpoints) as an upcoming feature but it has no mention of Async non-HTTP SDK support.

I have also checked Facebook marketing source connector provided by Airbyte which uses async_job but they don't use python's asyncio functionality itself, rather FB marketing provides async REST APIs which the Airbyte connector code simply calls as a client. So, thats unrelated to our use-case.

Wondering if there is any supported way to use asyncio based third party source's SDK to develop an AIrbyte connector.

Thanks
Nitesh