New Destination: Apache Hudi

[vingov]

Tell us about the new connector you’d like to have

• Which source and which destination?

• Java destination for writing Apache Hudi dataset, an open file format that supports ACID transactions.

• Do you need a specific version of the underlying data source e.g: you specifically need support for an older version of the API or DB?

• Any Hudi Latest version >= 0.10.0

Describe the context around this new connector

• Why do you need this integration? How does your team intend to use the data? This helps us understand the use case.

• Uber uses the Apache Hudi format for raw data, to support pulling data from multiple sources Airbyte is an ideal choice since it has support for a long tail of source connectors.

• How often do you want to run syncs?

• Every time there is fresh data in the upstream, mostly 6 to 8 times syncing incremental data.

• If this is an API source connector, which entities/endpoints do you need supported?

• n/a

• If the connector is for a paid service, can we name you as a mutual user when we subscribe for an account? Which company should we name?

• n/a

Describe the alternative you are considering or using

What are you considering doing if you don’t have this integration through Airbyte?
Writing a custom connector using Marmaray

Are you willing to submit a PR?

Yes

[vinothchandar]

This was a PoC done by someone in the Hudi community https://h7kanna.medium.com/why-and-how-i-integrated-airbyte-and-apache-hudi-c18aff3af21a

[vasudevanubrolu]

Hi @vingov @vinothchandar @marcosmarxm if there is still a requirement I would like to pick this up

[marcosmarxm]

@vasudevanubrolu right now the team is working to change how destinations works. I don't think we want to implement right now.