Stitch - Github Integration 500 Error

[usamachaudharypray]

Hi,

I hope everyone is doing well. Yesterday after 3:00PM PST, Our stitch's Github integration started to throw 500 Internal server error (CRITICAL HTTP-error-code: 500, Error: An error has occurred at Github's end.) while we have not changed anything at our end in integration's settings. We assumed that it is due to any Github service being down but they are all working fine. Details of error are below:

2022-02-23 23:41:09,851Z    tap - INFO METRIC Point(metric_type='timer', metric='http_request_duration', value=1.0166196823120117, tags={'status': 'failed', 'endpoint': 'releases'})
2022-02-23 23:41:09,886Z target - INFO Serializing batch with 1020 messages for table releases
2022-02-23 23:41:09,916Z target - INFO Sending batch of 800459 bytes to https://api.stitchdata.com/v2/import/batch
2022-02-23 23:41:10,391Z    tap - INFO [smart-services] event successfully sent to kafka: com.stitchdata.streamRecordCount [5] at offset None
2022-02-23 23:41:10,858Z    tap - INFO [smart-services] event successfully sent to kafka: com.stitchdata.streamRecordCount [1] at offset None
2022-02-23 23:41:10,858Z    tap - INFO replicated 1020 records from "releases" endpoint
2022-02-23 23:41:10,858Z    tap - CRITICAL HTTP-error-code: 500, Error: An error has occurred at Github's end.
2022-02-23 23:41:10,858Z    tap - Traceback (most recent call last):
2022-02-23 23:41:10,858Z    tap -   File "tap-env/bin/tap-github", line 33, in <module>
2022-02-23 23:41:10,858Z    tap -     sys.exit(load_entry_point('tap-github==1.10.4', 'console_scripts', 'tap-github')())
2022-02-23 23:41:10,858Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/singer/utils.py", line 229, in wrapped
2022-02-23 23:41:10,859Z    tap -     return fnc(*args, **kwargs)
2022-02-23 23:41:10,859Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_github/__init__.py", line 1190, in main
2022-02-23 23:41:10,859Z    tap -     do_sync(args.config, args.state, catalog)
2022-02-23 23:41:10,859Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_github/__init__.py", line 1154, in do_sync
2022-02-23 23:41:10,859Z    tap -     state = sync_func(stream_schema, repo, state, mdata, start_date)
2022-02-23 23:41:10,859Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_github/__init__.py", line 769, in get_all_releases
2022-02-23 23:41:10,859Z    tap -     'https://api.github.com/repos/{}/releases?sort=created_at&direction=desc'.format(repo_path)
2022-02-23 23:41:10,859Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_github/__init__.py", line 228, in authed_get_all_pages
2022-02-23 23:41:10,859Z    tap -     r = authed_get(source, url, headers)
2022-02-23 23:41:10,859Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/backoff/_sync.py", line 94, in retry
2022-02-23 23:41:10,859Z    tap -     ret = target(*args, **kwargs)
2022-02-23 23:41:10,859Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_github/__init__.py", line 218, in authed_get
2022-02-23 23:41:10,859Z    tap -     raise_for_error(resp, source)
2022-02-23 23:41:10,859Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_github/__init__.py", line 192, in raise_for_error
2022-02-23 23:41:10,859Z    tap -     raise exc(message) from None
2022-02-23 23:41:10,859Z    tap - tap_github.InternalServerError: HTTP-error-code: 500, Error: An error has occurred at Github's end.
2022-02-23 23:41:10,860Z   main - INFO Target exited normally with status 0
2022-02-23 23:41:10,867Z target - INFO replicated 1020 records from "releases" endpoint
2022-02-23 23:41:10,868Z target - I/O operation on closed file.
2022-02-23 23:41:11,387Z   main - INFO [smart-services] event successfully sent to kafka: com.stitchdata.extractionJobFinished [44] at offset None
2022-02-23 23:41:11,390Z   main - INFO No tunnel subprocess to tear down
2022-02-23 23:41:11,390Z   main - INFO Exit status is: Discovery succeeded. Tap failed with code 1 and error message: "HTTP-error-code: 500, Error: An error has occurred at Github's end.". Target succeeded.

[jcw-]

I am also seeing this, started happening in roughly the past 12 hours.

HTTP-error-code: 500, Error: An error has occurred at Github's end.

Seems to coincide with v1.10.4 being released perhaps?

2022-02-24 07:26:14,874Z target - INFO replicated 1020 records from "releases" endpoint
2022-02-24 07:26:14,874Z target - INFO Requests complete, stopping loop
2022-02-24 07:26:14,884Z    tap - INFO replicated 1020 records from "releases" endpoint
2022-02-24 07:26:14,884Z    tap - CRITICAL HTTP-error-code: 500, Error: An error has occurred at Github's end.
2022-02-24 07:26:14,884Z    tap - Traceback (most recent call last):
2022-02-24 07:26:14,884Z    tap -   File "tap-env/bin/tap-github", line 33, in <module>
2022-02-24 07:26:14,884Z    tap -     sys.exit(load_entry_point('tap-github==1.10.4', 'console_scripts', 'tap-github')())
2022-02-24 07:26:14,884Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/singer/utils.py", line 229, in wrapped
2022-02-24 07:26:14,884Z    tap -     return fnc(*args, **kwargs)
2022-02-24 07:26:14,884Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_github/__init__.py", line 1190, in main
2022-02-24 07:26:14,884Z    tap -     do_sync(args.config, args.state, catalog)
2022-02-24 07:26:14,884Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_github/__init__.py", line 1154, in do_sync
2022-02-24 07:26:14,884Z    tap -     state = sync_func(stream_schema, repo, state, mdata, start_date)
2022-02-24 07:26:14,884Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_github/__init__.py", line 769, in get_all_releases
2022-02-24 07:26:14,885Z    tap -     '[https://api.github.com/repos/{}/releases?sort=created_at&direction=desc'.format(repo_path)](https://api.github.com/repos/%7B%7D/releases?sort=created_at&direction=desc%27.format(repo_path))
2022-02-24 07:26:14,885Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_github/__init__.py", line 228, in authed_get_all_pages
2022-02-24 07:26:14,885Z    tap -     r = authed_get(source, url, headers)
2022-02-24 07:26:14,885Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/backoff/_sync.py", line 94, in retry
2022-02-24 07:26:14,885Z    tap -     ret = target(*args, **kwargs)
2022-02-24 07:26:14,885Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_github/__init__.py", line 218, in authed_get
2022-02-24 07:26:14,885Z    tap -     raise_for_error(resp, source)
2022-02-24 07:26:14,885Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_github/__init__.py", line 192, in raise_for_error
2022-02-24 07:26:14,885Z    tap -     raise exc(message) from None
2022-02-24 07:26:14,885Z    tap - tap_github.InternalServerError: HTTP-error-code: 500, Error: An error has occurred at Github's end.
2022-02-24 07:26:14,885Z   main - INFO Target exited normally with status 0
2022-02-24 07:26:16,349Z   main - INFO [smart-services] event successfully sent to kafka: com.stitchdata.extractionJobFinished [5] at offset None
2022-02-24 07:26:16,352Z   main - INFO No tunnel subprocess to tear down
2022-02-24 07:26:16,353Z   main - INFO Exit status is: Discovery succeeded. Tap failed with code 1 and error message: "HTTP-error-code: 500, Error: An error has occurred at Github's end.". Target succeeded.

[jcw-]

This appears to have started working again on Feb 25th.

fwiw, my attempts to raise this problem received zero responses on the reported issue (this ticket), the dedicated channel for this tap on the slack community where I originally reported it, or in the #general channel of the slack where I tried to escalate it

it would be nice to know what the process is intended to be for raising these kinds of issues if the GitHub issues and slack aren't being checked very frequently?

Update: Got a response on Slack - looks like this was due to mislav/hub#2920