[source-mssql] ENH: doing full discover during read

[rodireich]

Connector Name

source-mssql

Connector Version

4.1.18

What step the error happened?

During the sync

Relevant information

This is coming from a customer with a few thousands of tables and some 20,000 columns overall.
They notice during the read that a full discover taking place from Airbyte which is straining the database.
This is coming from a call to discoverWithoutSystemTables during Read.

The reason we make this call is to go over all configured cursor and validate their existence and datatype in validateCursorFieldForIncrementalTables.

Since platform now has a DISCOVER_REFRESH_WINDOW_MINUTES parameter that allows for disabling or controlling the rate of automated discovers, source-mssql shouldn't do a discover on each run. we should be able to make this call more efficient by not attempting to read an entire db schema but only the tables we're interested in (tables that are included in a sync, that have a cursor field etc.)

Relevant log output

Contribute

• Yes, I want to contribute