Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[590] Add Hudi HMS Catalog Sync implementation #639

Merged
merged 1 commit into from
Feb 13, 2025
Merged

[590] Add Hudi HMS Catalog Sync implementation #639

merged 1 commit into from
Feb 13, 2025

Conversation

vamsikarnika
Copy link

@vamsikarnika vamsikarnika commented Feb 5, 2025 •
edited
Loading

#590

Important Read

  • Please ensure the GitHub issue is mentioned at the beginning of the PR

What is the purpose of the pull request

Added support for syncing a source table to HMS catalog in Hudi table format

Brief change log

  • Added HudiPartitionSyncTool to sync the hudi partitions to catalogs
  • AddedCatalogPartitionSync operations to support partitions sync to HMS and Glue

Verify this pull request

  • Added unit tests
  • Manually verified the change by running a job locally

vamsikarnika
vamsikarnika commented Feb 5, 2025
View reviewed changes
xtable-core/pom.xml Outdated Show resolved Hide resolved
@vamsikarnika vamsikarnika changed the title Add implementation for Hudi HMS Catalog Sync [590] Add Hudi HMS Catalog Sync implementation Feb 5, 2025
@vamsikarnika vamsikarnika force-pushed the hudi_hms_catalog_sync_v1 branch from 6022968 to ca4cbba Compare February 5, 2025 12:13
@vinishjail97 vinishjail97 self-assigned this Feb 6, 2025
@vamsikarnika vamsikarnika force-pushed the hudi_hms_catalog_sync_v1 branch from baa81dd to 75bb800 Compare February 10, 2025 17:05
@vinishjail97
Copy link
Contributor

@vamsikarnika Can we split this into two PR's ?

  1. Changes in xtable-core.
  2. Changes in xtable-hive-metastore.

@vamsikarnika vamsikarnika force-pushed the hudi_hms_catalog_sync_v1 branch from 17c2d49 to c37ebb5 Compare February 11, 2025 14:30
@vamsikarnika vamsikarnika force-pushed the hudi_hms_catalog_sync_v1 branch from 5fb9ab6 to abbab03 Compare February 11, 2025 15:49
vinishjail97
vinishjail97 reviewed Feb 12, 2025
View reviewed changes
xtable-core/src/main/java/org/apache/xtable/catalog/Partition.java Outdated Show resolved Hide resolved
xtable-core/src/main/java/org/apache/xtable/catalog/PartitionEvent.java Outdated Show resolved Hide resolved
xtable-core/src/main/java/org/apache/xtable/catalog/PartitionSyncTool.java Outdated Show resolved Hide resolved
xtable-core/src/main/java/org/apache/xtable/hudi/HudiPartitionSyncTool.java Outdated Show resolved Hide resolved
xtable-core/src/main/java/org/apache/xtable/hudi/HudiPartitionSyncTool.java Outdated Show resolved Hide resolved
xtable-core/src/main/java/org/apache/xtable/hudi/HudiSparkDataSourceTableUtils.java Outdated Show resolved Hide resolved
xtable-core/src/main/java/org/apache/xtable/hudi/HudiInputFormatUtils.java Outdated Show resolved Hide resolved
@vinishjail97
Copy link
Contributor

LGTM, can you squash into a single commit before merging ?

@vamsikarnika vamsikarnika force-pushed the hudi_hms_catalog_sync_v1 branch from 4402783 to accb5ab Compare February 13, 2025 06:26
@vamsikarnika vamsikarnika force-pushed the hudi_hms_catalog_sync_v1 branch from 540c04e to a5d138f Compare February 13, 2025 07:20
@vinishjail97 vinishjail97 merged commit 2e71e15 into apache:main Feb 13, 2025
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vinishjail97 vinishjail97 vinishjail97 approved these changes

Assignees

@vinishjail97 vinishjail97

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@vamsikarnika @vinishjail97