100% CPU Usage On Puma Workers at "idle" in Development.

[joshuapinter]

Hey There,

Enabling skylight in development causes a strange situation where Puma workers remain at about 100% CPU utilization even after the request has been complete.

Without skylight enabled in development and 5 seconds after the web request is complete:

# config.skylight.environments += [ "development" ]

With skylight enabled in development and 5 seconds after the web request is complete:

config.skylight.environments += [ "development" ]

And it stays like this in perpetuity.

The only thing that is being changed is that one line.

We checked in production/staging and are not seeing this CPU issue.

Happy to help diagnose but not sure what the next steps may be. For now, we have to disable it in development as it makes our development machines unusable.

Thanks.

Versions

ruby 2.7.2p137
Rails 5.2.6
skylight 5.1.1
puma 5.3.2

[wagenet]

@joshuapinter This is definitely surprising. We use puma ourselves and don't experience this issue so it seems likely to be something related to a specific configuration. Would you might sharing your Puma config?

[joshuapinter]

Thanks for the quick response. Here's the puma config we're using:

puma.rb

bind "tcp://0.0.0.0:3000" 
workers 8
threads 0, 1
preload_app!

[wagenet]

@joshuapinter thanks for sharing. Unfortunately, my colleague most equipped to investigate this is currently taking some PTO. I'll try to investigate some before he's back, but please bear with us if it takes a little bit for us to resolve this. Thanks.

[joshuapinter]

No worries, @wagenet. There's no rush, as the most important environments are production and staging. Keep me posted on anything I can do to help diagnose. Thanks.

[zvkemp]

Hi @joshuapinter, thanks for your patience! There is some additional information that might be helpful:

• It looks like you're on a Mac, is that accurate?
• Which processor architecture do you have? The output of uname -a may be helpful
• Using Activity Monitor, Could you sample the ruby process when it's under high CPU? A second or two should be enough to see what stack frames are active. You can email the result to us at [email protected].

[joshuapinter]

Hi @joshuapinter, thanks for your patience! There is some additional information that might be helpful:

Hi @zvkemp, thanks for the response.

• It looks like you're on a Mac, is that accurate?

Correct.

• Which processor architecture do you have? The output of uname -a may be helpful

Darwin 20.5.0 Darwin Kernel Version 20.5.0: Sat May  8 05:10:33 PDT 2021; root:xnu-7195.121.3~9/RELEASE_X86_64 x86_64

• Using Activity Monitor, Could you sample the ruby process when it's under high CPU? A second or two should be enough to see what stack frames are active. You can email the result to us at [email protected].

Sent!

Thanks.

[zvkemp]

Thanks for the sample. Skylight's task thread doesn't appear to be doing anything abnormal; as far as I can tell the only active thread (i.e., not waiting on a condition variable or IO) is Puma's reactor, so the question remains why the addition of Skylight would cause this. Can you try running Puma in single mode rather than cluster mode and let us know if the problem occurs there as well?

[joshuapinter]

Hey @zvkemp. I checked when running Puma in single mode (with workers commented out) and the issue was not present.

[jnimety]

Hello! I don't know if our issue is related but we're experiencing Puma workers randomly remaining at 100% in our production environment. There is this thread_pool issue that was fixed in Puma 5.4.0 puma/puma#2656 but it didn't fix the issue for us.