Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Source][Utilization] Enable processing multiple C* in a single source instance #94

Open
aymkhalil opened this issue Oct 24, 2022 · 1 comment

Comments

@aymkhalil
Copy link
Contributor

aymkhalil commented Oct 24, 2022

Today, the C* source connectors only allows 1:1 between tables and sinks. In order to increate the utilization of the underling resources associated with a single source instance (e.g. Memory footprint a single sink is ~500MB, which does not scale well if the user has 10s or 100s of tables), the proposal is to enable users to configure multiple tables in their source config.

Proposed source config:

configs:
  contactPoints": "localhost",
  loadBalancing.localDc": "Cassandra" , "outputFormat": "key-value-avro"
  tables:
    ks1:
      table1:
        events.topic": "persistent://public/default/events-ks1.table1"
        data.topic": "persistent://public/default/data-ks1.table1"
    ks2:
      table2:
        events.topic": "persistent://public/default/events-ks2.table2"
        data.topic": "persistent://public/default/data-ks2.table2"
@aymkhalil
Copy link
Contributor Author

Alternatively, we can keep the config as close as possible to today's single table configs by replacing data.topic with destination-topic-name

@aymkhalil aymkhalil changed the title [Source][Utilization] Enable processing multiple C* in a single sink instance [Source][Utilization] Enable processing multiple C* in a single source instance Mar 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant