Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

writetime and TTL are not preserved if columns are not explicitly configured in cdm.properties #218

Closed
icaredb opened this issue Nov 15, 2023 · 1 comment

Comments

@icaredb
Copy link

icaredb commented Nov 15, 2023

With the following settings in cdm.properties, the writetime and TTL values are not preserved when data is migrated to target cluster. Note spark.cdm.schema.origin.column.ttl.names and spark.cdm.schema.origin.column.writetime.names are not explicitly set.

spark.cdm.schema.origin.column.ttl.automatic     true
spark.cdm.schema.origin.column.writetime.automatic  true

Although the automatic TTLCols and WriteTimestampCols are detected correctly, logging shows feature writetimeTTL is disabled:

23/11/15 01:10:08 INFO WritetimeTTL: PARAM -- Automatic TTLCols: [c3, c4]
23/11/15 01:10:08 INFO WritetimeTTL: PARAM -- Automatic WriteTimestampCols: [c3, c4]
23/11/15 01:10:08 INFO WritetimeTTL: Feature WritetimeTTL is disabled

Reproducing steps:

  1. prepare on origin cluster
CREATE TABLE ks1.cdm (
    pk1 ascii,
    ck1 ascii,
    c3 ascii,
    c4 ascii,
    PRIMARY KEY (pk1, ck1)
) ;
insert into cdm(pk1,ck1,c3,c4) values ('1','x','x1','y1');
insert into cdm(pk1,ck1,c3,c4) values ('2','x','x2','y2');
update cdm set c3='x1x' where pk1='1' and ck1='x';

cqlsh> select c3,writetime(c3),c4,writetime(c4) from ks1.cdm;

 c3  | writetime(c3)    | c4 | writetime(c4)
-----+------------------+----+------------------
  x2 | 1699617875598019 | y2 | 1699617875598019
 x1x | 1699617884895474 | y1 | 1699617863817553
  1. cdm.properties contents set as below:
spark.cdm.schema.origin.column.ttl.automatic     true
#spark.cdm.schema.origin.column.ttl.names       c3
spark.cdm.schema.origin.column.writetime.automatic  true
#spark.cdm.schema.origin.column.writetime.names   c3
  1. writetime is not preserved after data migration on the target cluster:
cqlsh> select c3,writetime(c3),c4,writetime(c4) from ks1.cdm;

 c3  | writetime(c3)    | c4 | writetime(c4)
-----+------------------+----+------------------
  x2 | 1699869971644859 | y2 | 1699869971644859
 x1x | 1699870639789613 | y1 | 1699870639789613
icaredb added a commit to icaredb/cassandra-data-migrator that referenced this issue Nov 15, 2023
Suggested fix for the following issue:
datastax#218
writetime and TTL are not preserved if columns are not explicitly configured in cdm.properties
icaredb added a commit to icaredb/cassandra-data-migrator that referenced this issue Nov 15, 2023
Fix for Jira datastax#218
writetime and TTL are not preserved if columns are not explicitly configured in cdm.properties
@icaredb icaredb mentioned this issue Nov 15, 2023
3 tasks
msmygit pushed a commit to icaredb/cassandra-data-migrator that referenced this issue Nov 15, 2023
Suggested fix for the following issue:
datastax#218
writetime and TTL are not preserved if columns are not explicitly configured in cdm.properties
msmygit pushed a commit to icaredb/cassandra-data-migrator that referenced this issue Nov 15, 2023
Fix for Jira datastax#218
writetime and TTL are not preserved if columns are not explicitly configured in cdm.properties
@msmygit
Copy link
Collaborator

msmygit commented Dec 19, 2023

Implemented as part of #226

@msmygit msmygit closed this as completed Dec 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants