Problem when configuring a connection using the API with two identical streams in different schemas in Postgres

[dantonbertuol]

Topic

API Connection

Relevant information

I am performing data integration between Postgres and MinIO.

In Postgres I have two schemas, "client_one" and "client_two", so I am filtering them when creating the source. In both schemas I have the "customers" table.

When I create a connection to S3, through the Airbyte UI, both streams are displayed to select, so I select both.

However, when I need to patch the connection via the API, and I enter the name of the table, Airbyte selects only one of the streams for replication, this must be a BUG because they both have the same name and via the API I don't have a way to differentiate them.

I'm running Airbyte 0.58.0 via Docker, on WSL 2.

Source Configuration:

Connection:

Postman:

Connection Updated via the API:

Video: https://github.com/airbytehq/airbyte/assets/32557284/76159010-f9c3-46ba-9e2b-b9bbd73cee70

[marcosmarxm]

I requested assistance from the connector team regarding about this question. @dantonbertuol, I believe a temporary solution is to create separate connections for each schema. Set distinct prefixes for the tables so they are easily identifiable in the destination database.

[marcosmarxm]

@dantonbertuol If you pick the "Destination Namespace" setting and select Source-defined, the destination will replicate the source schemas without any issue. If you want to send both tables to a new schema, you'll need to split the source into two. It's important to use table prefix to avoid duplicate table names in the schema.

[dantonbertuol]

Hello, thanks for responding.

My problem is different, I will send it to the destination, obeying the source namespace. However, when I make a change through the API, it only marks one of the streams that has the same name, and I would like it to mark all of them.

Did you manage to watch the video?

[AHulshoff]

Wouldn't it be better if the stream prefix could use templating? I mean something like ${SCHEMA}_. That would create more flexibility and could be used with destination namespace and still keep tables separated with schema name as the prefix of the stream.