Handle timestamp_ntz in delta conversion target #647
Conversation
| @@ -66,9 +66,9 @@ | |||
| import org.apache.xtable.spi.sync.ConversionTarget; | |||
|
|
|||
| public class DeltaConversionTarget implements ConversionTarget { | |||
| private static final String MIN_READER_VERSION = String.valueOf(1); | |||
| private static final String MIN_READER_VERSION = String.valueOf(3); | |||
There was a problem hiding this comment.
Handle this gracefully by calling the function which upgrades version in delta codebase, this was the pending comment from the previous PR that wasn't addressed.
There was a problem hiding this comment.
Is this change related to timestamp_ntz? Increasing the min_reader version could break certain consumers using old libraries. If it's unrelated, could we create a separate issue for this and perhaps make it configurable?
There was a problem hiding this comment.
Delta does the automatic version upgrade today based on the schema of the table and other properties, if we call the same function when initializing the delta table it shouldn't break anything.
https://docs.delta.io/2.0.0/versioning.html
When creating a table, Delta Lake chooses the minimum required protocol version based on table characteristics such as the schema or table properties
| @@ -66,9 +66,9 @@ | |||
| import org.apache.xtable.spi.sync.ConversionTarget; | |||
|
|
|||
| public class DeltaConversionTarget implements ConversionTarget { | |||
| private static final String MIN_READER_VERSION = String.valueOf(1); | |||
| private static final String MIN_READER_VERSION = String.valueOf(3); | |||
There was a problem hiding this comment.
Is this change related to timestamp_ntz? Increasing the min_reader version could break certain consumers using old libraries. If it's unrelated, could we create a separate issue for this and perhaps make it configurable?
| @@ -61,6 +61,11 @@ public class DeltaSchemaExtractor { | |||
| private static final String DELTA_COLUMN_MAPPING_ID = "delta.columnMapping.id"; | |||
| private static final String COMMENT = "comment"; | |||
| private static final DeltaSchemaExtractor INSTANCE = new DeltaSchemaExtractor(); | |||
| // Timestamps in Delta are microsecond precision by default | |||
There was a problem hiding this comment.
Does XTable need to handle nanoseconds precision?
Important Read
What is the purpose of the pull request
Handle timestamp_ntz in delta target, this needs to be done in a better way by finding this function in delta codebase.
https://docs.delta.io/2.0.0/versioning.html
When creating a table, Delta Lake chooses the minimum required protocol version based on table characteristics such as the schema or table propertiesBrief change log
(for example:)
Verify this pull request
(Please pick either of the following options)
This pull request is a trivial rework / code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)