[source-intercom] Intercom Incremental connector does not get all historical data and does not update state

[HoudaSebti]

Connector Name

source-intercom

Connector Version

Intercom v0.9.0

What step the error happened?

During the sync

Relevant information

Hello team !

Context:
We're using Airbyte Cloud.
We have created a new Intercom Connection with Sync mode: Incremental Append.

• Enabled stream of the connector: Conversations (Source is Intercom)
• Destination: Bigquery
  We've left all the connection configuration to default ones.
  As expected, the very first sync runs directly after the connector is created.
  It takes almost 4 hours and finishes by fetching ~95K records.

**Issue 1:**The true number of conversations we have is around 150K records. so we're almost 60K records short for historical data.

I checked the logs, I don't see any rate limit errors related to the Intercom API nor do I see any other kind of errors.

Issue 2: What's even more awkward is that the connector's state does not get updated after this first sync is done. The state remains empty as if no initial sync had ran.
I tried inputing some specific connection states to make the connector fetch only more recent data, and i notice that if the inputted start date is a bit old, the connector does not get all the records, but when the date is relatively recent (dating from one week ago for example), we get almost all records.

Example of connection state i manually inputted:
[ { "streamDescriptor": { "name": "conversations" }, "streamState": { "updated_at": 1736936245, "prior_state": { "updated_at": 1704151708, "prior_state": { "updated_at": 1704151708 } } } } ]

So I'm suspecting some kind of data leak that gets more important with larger amount of fetched records .

• Is this issue know for intercom connectors with large volumes of data or when fetching historical data ?
• Why do you think the connection state does not get updated after a sync had been completed ?

Please find attached the logs of the very first sync run of my incremental connector.

intercom_conversations___bigquery_intercom__fix_2_logs_26850667_txt.txt

Thanks a lot and feel free if you need further details !

Relevant log output

Contribute

• Yes, I want to contribute