Skip to content

Commit

Permalink
[Bugfix] Populate table name from the identifier in Iceberge conversion
Browse files Browse the repository at this point in the history
What is the problem?
While converting the iceberg table to other format such as Hudi,
the icerberg source table do not populate the table name. This is
due to iceberg table's behavior as it is treated as Hadoop tables.
This leads to table identified as table-location, leading to confusing
conversation.

Solution:
This commit handles the conversation logic, when icebege table manager
provides HadoopTable, it populate the table name from provided input
TableIdentifier. This ensures that source table name is carried over
to the transformation.

Testing:
- Added unit test to cover this scenario.

Co-authored-by: Tim Brown <[email protected]>
  • Loading branch information
jalpan-randeri and the-other-tim-brown committed Feb 13, 2025
1 parent 14967dd commit 828112b
Show file tree
Hide file tree
Showing 2 changed files with 10 additions and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -120,10 +120,18 @@ public InternalTable getTable(Snapshot snapshot) {
irPartitionFields.size() > 0
? DataLayoutStrategy.HIVE_STYLE_PARTITION
: DataLayoutStrategy.FLAT;
// When the table name is not explicitly specified, Iceberg assumes the table is HDFS-based,
// treating the table name as the location in HDFS. This assumption can lead to mismatches
// during metadata conversion. To mitigate this issue, we rely on the table name provided in the
// source configuration of the conversation so target matches the user's expectations.
// See https://github.com/apache/incubator-xtable/issues/494
return InternalTable.builder()
.tableFormat(TableFormat.ICEBERG)
.basePath(iceTable.location())
.name(iceTable.name())
.name(
iceTable.name().contains(iceTable.location())
? sourceTableConfig.getName()
: iceTable.name())
.partitioningFields(irPartitionFields)
.latestCommitTime(Instant.ofEpochMilli(snapshot.timestampMillis()))
.readSchema(irSchema)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,7 @@ void getCurrentTableTest() {
.build();
validateTable(
internalTable,
testIcebergTable.getBasePath(),
testIcebergTable.getTableName(),
TableFormat.ICEBERG,
internalSchema,
DataLayoutStrategy.FLAT,
Expand Down

0 comments on commit 828112b

Please sign in to comment.