feat: unique connection counting #9074
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
1 Skipped Deployment
|
Dependency ReviewThe following issues were found:
OpenSSF Scorecard
Scanned Files
|
kwasniew
commented
Jan 9, 2025
| @@ -66,6 +66,11 @@ export function responseTimeMetrics( | |||
| req.query.appName; | |||
| } | |||
|
|
|||
| const connectionId = req.headers['x-unleash-connection-id']; | |||
There was a problem hiding this comment.
this is where it starts. on each detected connection id received we let the interested parties know
kwasniew
commented
Jan 9, 2025
| this.activeHour = new Date().getHours(); | ||
| } | ||
|
|
||
| listen() { |
There was a problem hiding this comment.
this service is interested in new connection ids received
kwasniew
commented
Jan 9, 2025
|
|
||
| async count(connectionId: string) { | ||
| if (!this.flagResolver.isEnabled('uniqueSdkTracking')) return; | ||
| this.hll.add(HyperLogLog.hash(connectionId)); |
There was a problem hiding this comment.
this data structure tracks unique connections
kwasniew
commented
Jan 9, 2025
| @@ -179,4 +180,10 @@ export const scheduleServices = async ( | |||
| minutesToMilliseconds(15), | |||
| 'cleanUpIntegrationEvents', | |||
| ); | |||
|
|
|||
| schedulerService.schedule( | |||
There was a problem hiding this comment.
sync in-memory HyperLogLogs with DB
kwasniew
commented
Jan 9, 2025
| this.hll.add(HyperLogLog.hash(connectionId)); | ||
| } | ||
|
|
||
| async sync(): Promise<void> { |
There was a problem hiding this comment.
this logic will get detailed coverage. We track current and previous HyperLogLog in 2 DB rows. If we found the hour has changed we start a new current bucket and copy current to previous. We have to accommodate for multiple pods where any pod can be the first one to start a current bucket.
kwasniew
commented
Jan 9, 2025
| ( | ||
| id VARCHAR(255) NOT NULL, | ||
| updated_at TIMESTAMP DEFAULT now(), | ||
| hll BYTEA NOT NULL, |
There was a problem hiding this comment.
hyperlog is a buffer that we translate to BYTEA type in postgres
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
About the changes
E2E POC for unique connection counting using windowed HyperLogLog. We want to know how many unique connections from SDKs we had in the previous hour and in the current hour.
Algorithm:
Followup:
Important files
Discussion points