Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[receiver/loadgen] Simulate multiple agents via new concurrency config #313

Merged
merged 17 commits into from
Jan 22, 2025

Conversation

carsonip
Copy link
Member

@carsonip carsonip commented Jan 17, 2025

Add a concurrency config to loadgenreceiver to simulate multiple agents (agent_replicas in apmsoak) sending otlp requests.

Fixes #305

@carsonip carsonip force-pushed the loadgen-multi-agent branch from 5552d83 to 43e0ea2 Compare January 17, 2025 19:24
@carsonip carsonip marked this pull request as ready for review January 17, 2025 22:10
@carsonip carsonip requested a review from a team as a code owner January 17, 2025 22:10
@carsonip carsonip changed the title [receiver/loadgen] Simulate multiple agents [receiver/loadgen] Simulate multiple agents via config concurrency Jan 20, 2025
@carsonip carsonip changed the title [receiver/loadgen] Simulate multiple agents via config concurrency [receiver/loadgen] Simulate multiple agents via new concurrency config Jan 20, 2025
receiver/loadgenreceiver/config.go Show resolved Hide resolved
type Config struct {
Metrics MetricsConfig `mapstructure:"metrics"`
Logs LogsConfig `mapstructure:"logs"`
Traces TracesConfig `mapstructure:"traces"`

// Concurrency is the amount of concurrency when sending to next consumer.
// The concurrent workers share the amount of workload, instead of multiplying the amount of workload,
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[to reviewer] I've made a design decision here for it to share the workload instead of multiplying the amount of workload. The implication is that when we use this config in otelbench, as concurrency increases, time/op will decrease. If it was implemented otherwise, the time/op will be equal or higher as concurrency increases.

@carsonip carsonip requested a review from axw January 20, 2025 12:00
receiver/loadgenreceiver/config.go Show resolved Hide resolved
loadgen/cmd/otelsoak/config.example.yaml Show resolved Hide resolved
receiver/loadgenreceiver/logs_test.go Outdated Show resolved Hide resolved
receiver/loadgenreceiver/internal/list/bounded_looping.go Outdated Show resolved Hide resolved
receiver/loadgenreceiver/internal/list/bounded_looping.go Outdated Show resolved Hide resolved
@carsonip carsonip requested a review from simitt January 21, 2025 21:43
Copy link

@simitt simitt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a comment for a small fix in the loop behavior

receiver/loadgenreceiver/factory.go Outdated Show resolved Hide resolved
receiver/loadgenreceiver/internal/list/looping.go Outdated Show resolved Hide resolved
@@ -1,5 +1,6 @@
receivers:
loadgen:
# concurrency: 16 # defaults to GOMAXPROCS
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of curiosity, how is otelsoak configured to stop at some point? I don't see a possibility to configure a time or similar?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does not terminate now. I'm not aware of a way to notify the otel collector to stop, unless we write code to make it panic or os.Exit

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, thanks for confirming; probably something for us to look into, but in another issue/PR

@carsonip carsonip requested a review from simitt January 22, 2025 10:56
@@ -1,5 +1,6 @@
receivers:
loadgen:
# concurrency: 16 # defaults to GOMAXPROCS
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, thanks for confirming; probably something for us to look into, but in another issue/PR

@carsonip carsonip enabled auto-merge (squash) January 22, 2025 11:55
Copy link
Contributor

@rogercoll rogercoll left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great!

@carsonip carsonip merged commit ec334ca into elastic:main Jan 22, 2025
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Load generator should be able to emulate multiple agents
3 participants