Concurrency support on recorder and config specification #176

tonyredondo · 2020-02-24T11:01:05Z

Closes #173

This PR adds support for concurrency levels in the recorder, following the configuration specification, also adds some other missing keys from the configuration.

Examples:

Concurrency level: 1 , Time: 2min, 13sec

2020/02/24 11:52:40 agent.go:212: environment variable $SCOPE_DSN not found
2020/02/24 11:52:40 agent.go:220: API key found in the native app configuration
2020/02/24 11:52:40 agent.go:233: API endpoint found in the native app configuration
2020/02/24 11:52:40 recorder.go:88: recorder frequency: 1s
2020/02/24 11:52:40 recorder.go:89: recorder concurrency level: 1
2020/02/24 11:52:41 recorder.go:158: sending 4221 spans in 5 batches
2020/02/24 11:52:41 ntp.go:44: ntp offset: 5.699595ms
2020/02/24 11:52:41 recorder.go:217: sending batch 1/5 with 1000 spans
2020/02/24 11:52:45 agent.go:393: Scope agent is stopping gracefully...
2020/02/24 11:52:50 recorder.go:217: sending batch 2/5 with 1000 spans
2020/02/24 11:52:56 recorder.go:217: sending batch 3/5 with 1000 spans
2020/02/24 11:53:02 recorder.go:217: sending batch 4/5 with 1000 spans
2020/02/24 11:53:08 recorder.go:217: sending batch 5/5 with 221 spans
2020/02/24 11:53:10 recorder.go:158: sending 15880 spans in 16 batches
2020/02/24 11:53:10 recorder.go:217: sending batch 1/16 with 1000 spans
2020/02/24 11:53:17 recorder.go:217: sending batch 2/16 with 1000 spans
2020/02/24 11:53:23 recorder.go:217: sending batch 3/16 with 1000 spans
2020/02/24 11:53:30 recorder.go:217: sending batch 4/16 with 1000 spans
2020/02/24 11:53:37 recorder.go:217: sending batch 5/16 with 1000 spans
2020/02/24 11:53:43 recorder.go:217: sending batch 6/16 with 1000 spans
2020/02/24 11:53:48 recorder.go:217: sending batch 7/16 with 1000 spans
2020/02/24 11:53:53 recorder.go:217: sending batch 8/16 with 1000 spans
2020/02/24 11:53:59 recorder.go:217: sending batch 9/16 with 1000 spans
2020/02/24 11:54:07 recorder.go:217: sending batch 10/16 with 1000 spans
2020/02/24 11:54:13 recorder.go:217: sending batch 11/16 with 1000 spans
2020/02/24 11:54:20 recorder.go:217: sending batch 12/16 with 1000 spans
2020/02/24 11:54:26 recorder.go:217: sending batch 13/16 with 1000 spans
2020/02/24 11:54:32 recorder.go:217: sending batch 14/16 with 1000 spans
2020/02/24 11:54:38 recorder.go:217: sending batch 15/16 with 1000 spans
2020/02/24 11:54:44 recorder.go:217: sending batch 16/16 with 880 spans
2020/02/24 11:54:53 recorder.go:158: sending 0 spans in 1 batches

Concurrency level: 5 , Time: 55sec

2020/02/24 11:49:00 agent.go:212: environment variable $SCOPE_DSN not found
2020/02/24 11:49:00 agent.go:220: API key found in the native app configuration
2020/02/24 11:49:00 agent.go:233: API endpoint found in the native app configuration
2020/02/24 11:49:00 recorder.go:88: recorder frequency: 1s
2020/02/24 11:49:00 recorder.go:89: recorder concurrency level: 5
2020/02/24 11:49:01 recorder.go:158: sending 4020 spans in 5 batches
2020/02/24 11:49:01 ntp.go:44: ntp offset: 4.941899ms
2020/02/24 11:49:01 recorder.go:217: sending batch 5/5 with 20 spans
2020/02/24 11:49:01 recorder.go:217: sending batch 1/5 with 1000 spans
2020/02/24 11:49:01 recorder.go:217: sending batch 2/5 with 1000 spans
2020/02/24 11:49:01 recorder.go:217: sending batch 3/5 with 1000 spans
2020/02/24 11:49:01 recorder.go:217: sending batch 4/5 with 1000 spans
2020/02/24 11:49:05 agent.go:393: Scope agent is stopping gracefully...
2020/02/24 11:49:12 recorder.go:158: sending 16081 spans in 17 batches
2020/02/24 11:49:13 recorder.go:217: sending batch 1/17 with 1000 spans
2020/02/24 11:49:13 recorder.go:217: sending batch 3/17 with 1000 spans
2020/02/24 11:49:13 recorder.go:217: sending batch 4/17 with 1000 spans
2020/02/24 11:49:13 recorder.go:217: sending batch 5/17 with 1000 spans
2020/02/24 11:49:13 recorder.go:217: sending batch 2/17 with 1000 spans
2020/02/24 11:49:25 recorder.go:217: sending batch 6/17 with 1000 spans
2020/02/24 11:49:26 recorder.go:217: sending batch 8/17 with 1000 spans
2020/02/24 11:49:26 recorder.go:217: sending batch 7/17 with 1000 spans
2020/02/24 11:49:26 recorder.go:217: sending batch 9/17 with 1000 spans
2020/02/24 11:49:27 recorder.go:217: sending batch 10/17 with 1000 spans
2020/02/24 11:49:35 recorder.go:217: sending batch 11/17 with 1000 spans
2020/02/24 11:49:37 recorder.go:217: sending batch 12/17 with 1000 spans
2020/02/24 11:49:38 recorder.go:217: sending batch 13/17 with 1000 spans
2020/02/24 11:49:38 recorder.go:217: sending batch 14/17 with 1000 spans
2020/02/24 11:49:38 recorder.go:217: sending batch 15/17 with 1000 spans
2020/02/24 11:49:46 recorder.go:217: sending batch 16/17 with 1000 spans
2020/02/24 11:49:47 recorder.go:217: sending batch 17/17 with 81 spans
2020/02/24 11:49:55 recorder.go:158: sending 0 spans in 1 batches
2020/02/24 11:49:55 recorder.go:158: sending 0 spans in 1 batches

Concurrency level: 20 , Time: 40sec

2020/02/24 12:06:30 agent.go:212: environment variable $SCOPE_DSN not found
2020/02/24 12:06:30 agent.go:220: API key found in the native app configuration
2020/02/24 12:06:30 agent.go:233: API endpoint found in the native app configuration
2020/02/24 12:06:30 recorder.go:88: recorder frequency: 1s
2020/02/24 12:06:30 recorder.go:89: recorder concurrency level: 20
2020/02/24 12:06:31 recorder.go:158: sending 4020 spans in 5 batches
2020/02/24 12:06:31 ntp.go:44: ntp offset: 955.156µs
2020/02/24 12:06:31 recorder.go:217: sending batch 5/5 with 20 spans
2020/02/24 12:06:32 recorder.go:217: sending batch 3/5 with 1000 spans
2020/02/24 12:06:32 recorder.go:217: sending batch 4/5 with 1000 spans
2020/02/24 12:06:32 recorder.go:217: sending batch 1/5 with 1000 spans
2020/02/24 12:06:32 recorder.go:217: sending batch 2/5 with 1000 spans
2020/02/24 12:06:35 agent.go:393: Scope agent is stopping gracefully...
2020/02/24 12:06:43 recorder.go:158: sending 16081 spans in 17 batches
2020/02/24 12:06:43 recorder.go:217: sending batch 17/17 with 81 spans
2020/02/24 12:06:44 recorder.go:217: sending batch 6/17 with 1000 spans
2020/02/24 12:06:44 recorder.go:217: sending batch 9/17 with 1000 spans
2020/02/24 12:06:44 recorder.go:217: sending batch 8/17 with 1000 spans
2020/02/24 12:06:44 recorder.go:217: sending batch 15/17 with 1000 spans
2020/02/24 12:06:44 recorder.go:217: sending batch 1/17 with 1000 spans
2020/02/24 12:06:44 recorder.go:217: sending batch 5/17 with 1000 spans
2020/02/24 12:06:44 recorder.go:217: sending batch 10/17 with 1000 spans
2020/02/24 12:06:44 recorder.go:217: sending batch 3/17 with 1000 spans
2020/02/24 12:06:44 recorder.go:217: sending batch 14/17 with 1000 spans
2020/02/24 12:06:44 recorder.go:217: sending batch 7/17 with 1000 spans
2020/02/24 12:06:44 recorder.go:217: sending batch 11/17 with 1000 spans
2020/02/24 12:06:44 recorder.go:217: sending batch 16/17 with 1000 spans
2020/02/24 12:06:44 recorder.go:217: sending batch 12/17 with 1000 spans
2020/02/24 12:06:44 recorder.go:217: sending batch 2/17 with 1000 spans
2020/02/24 12:06:44 recorder.go:217: sending batch 4/17 with 1000 spans
2020/02/24 12:06:44 recorder.go:217: sending batch 13/17 with 1000 spans
2020/02/24 12:07:10 recorder.go:158: sending 0 spans in 1 batches
2020/02/24 12:07:10 recorder.go:158: sending 0 spans in 1 batches

drodriguezhdez · 2020-03-05T13:25:50Z

env/vars.go

+	ScopeInstrumentationDbStatementValues               = newBooleanEnvVar(false, "SCOPE_INSTRUMENTATION_DB_STATEMENT_VALUES")
+	ScopeTracerDispatcherHealthcheckFrequency           = newIntEnvVar(60000, "SCOPE_TRACER_DISPATCHER_HEALTHCHECK_FRECUENCY")
+	ScopeTracerDispatcherHealthcheckFrequencyInTestMode = newIntEnvVar(1000, "SCOPE_TRACER_DISPATCHER_HEALTHCHECK_FRECUENCY_IN_TESTMODE")
+	ScopeTracerDispatcherConcurrencyLevel               = newIntEnvVar(1, "SCOPE_TRACER_DISPATCHER_CONCURRENCY_LEVEL")


By default, we want "1" as concurrency level in Go?

unless we decide new defaults for this setting? that is the default in .NET too

fermayo · 2020-03-11T08:46:19Z

agent/recorder.go

 	r.url = agent.getUrl("api/agent/ingest")
 	r.client = &http.Client{}
 	r.stats = &RecorderStats{}
+	r.s = semaphore.NewWeighted(int64(r.concurrencyLevel))


Why not use a fixed pool of go routines that read from a channel? Like https://gobyexample.com/worker-pools

I've found easier to implement (less code modification) with a semaphore... But yeah we can do it also with two channels... Do you prefer with a channel implementation?

tonyredondo · 2020-03-17T14:14:39Z

This PR was superseed by #199

tonyredondo self-assigned this Feb 24, 2020

tonyredondo requested review from fermayo and drodriguezhdez February 24, 2020 11:01

tonyredondo force-pushed the scope-spec branch from bd7285d to 3deb861 Compare March 2, 2020 16:22

drodriguezhdez reviewed Mar 5, 2020

View reviewed changes

fermayo reviewed Mar 11, 2020

View reviewed changes

tonyredondo force-pushed the scope-spec branch from 3deb861 to bab1bee Compare March 11, 2020 17:03

scope specification support, initial

b1d0195

tonyredondo force-pushed the scope-spec branch from 8948a2a to 514deb7 Compare March 14, 2020 22:37

tonyredondo and others added 5 commits March 14, 2020 23:37

Adds concurrency support

dd102f5

comments

3c7719c

Fixes spans counters

403fb8c

fix span counters

514deb7

mod changes

958e9f8

tonyredondo closed this Mar 17, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Concurrency support on recorder and config specification #176

Concurrency support on recorder and config specification #176

tonyredondo commented Feb 24, 2020 •

edited

Loading

drodriguezhdez Mar 5, 2020

tonyredondo Mar 9, 2020

fermayo Mar 11, 2020

tonyredondo Mar 11, 2020

tonyredondo commented Mar 17, 2020

Concurrency support on recorder and config specification #176

Concurrency support on recorder and config specification #176

Conversation

tonyredondo commented Feb 24, 2020 • edited Loading

drodriguezhdez Mar 5, 2020

Choose a reason for hiding this comment

tonyredondo Mar 9, 2020

Choose a reason for hiding this comment

fermayo Mar 11, 2020

Choose a reason for hiding this comment

tonyredondo Mar 11, 2020

Choose a reason for hiding this comment

tonyredondo commented Mar 17, 2020

tonyredondo commented Feb 24, 2020 •

edited

Loading