SystemStackError: stack level too deep

Problem

You are seeing a stack level too deep (SystemStackError) error after adding the newrelic_rpm gem to your application, or after upgrading to version 7.0.0 or higher.

Solution

In most cases, the Ruby agent is not the underlying issue. The issue is that method chaining (alias_method) and Module#prepend work together only in particular situations. When not used together properly, they can cause non-terminating recursions. The Ruby agent offers both Module#prepend and method chaining gem instrumentation to allow flexibility for customers. For more information about method chaining and Module#prepend incompatibility, read: Resolving Module#prepend and alias_method conflicts involving the New Relic Ruby agent

1. Get the full stack trace

The first thing you’ll need to do is to get the full stacktrace for the error you’re seeing. The reason it needs to be the full stack trace is because you will need to find the section in the stacktrace that shows the recursion to find the two locations that are in conflict. Things like Rails logs and Passenger logs often truncate the stack trace since it’s so long, so you may need to reproduce the error in an environment where you will be able to grab the full, untruncated stack trace. One way to do this is to call Exception#backtrace on the stack level too deep exception, which will return the full stack trace.

2. Find the conflicting code in the stack trace

Once you have the full stack trace, look for a section with repetition between one gem and the newrelic_rpm gem. This is likely the gem causing the conflict. Here is an example section of a stack trace that shows the locations in conflict with Module#prepend and method chaining.

/Users/user/.rvm/gems/ruby-2.6.5/gems/airbrake-11.0.1/lib/airbrake/rails/net_http.rb:11:in `block in request'
/Users/user/.rvm/gems/ruby-2.6.5/gems/airbrake-11.0.1/lib/airbrake/rack.rb:25:in `capture_timing'
/Users/user/.rvm/gems/ruby-2.6.5/gems/airbrake-11.0.1/lib/airbrake/rails/net_http.rb:10:in `request'
/Users/user/.rvm/gems/ruby-2.6.5/gems/newrelic_rpm-7.0.0/lib/new_relic/agent/instrumentation/net_http/chain.rb:16:in `block in request_with_newrelic_trace'
/Users/user/.rvm/gems/ruby-2.6.5/gems/newrelic_rpm-7.0.0/lib/new_relic/agent/instrumentation/net_http/instrumentation.rb:26:in `block (2 levels) in request_with_tracing'
/Users/user/.rvm/gems/ruby-2.6.5/gems/newrelic_rpm-7.0.0/lib/new_relic/agent/tracer.rb:371:in `capture_segment_error'
/Users/user/.rvm/gems/ruby-2.6.5/gems/newrelic_rpm-7.0.0/lib/new_relic/agent/instrumentation/net_http/instrumentation.rb:25:in `block in request_with_tracing'
/Users/user/.rvm/gems/ruby-2.6.5/gems/newrelic_rpm-7.0.0/lib/new_relic/agent.rb:501:in `disable_all_tracing'
/Users/user/.rvm/gems/ruby-2.6.5/gems/newrelic_rpm-7.0.0/lib/new_relic/agent/instrumentation/net_http/instrumentation.rb:24:in `request_with_tracing'
/Users/user/.rvm/gems/ruby-2.6.5/gems/newrelic_rpm-7.0.0/lib/new_relic/agent/instrumentation/net_http/chain.rb:16:in `request_with_newrelic_trace'
/Users/user/.rvm/gems/ruby-2.6.5/gems/airbrake-11.0.1/lib/airbrake/rails/net_http.rb:11:in `block in request'
/Users/user/.rvm/gems/ruby-2.6.5/gems/airbrake-11.0.1/lib/airbrake/rack.rb:25:in `capture_timing'
/Users/user/.rvm/gems/ruby-2.6.5/gems/airbrake-11.0.1/lib/airbrake/rails/net_http.rb:10:in `request'
/Users/user/.rvm/gems/ruby-2.6.5/gems/newrelic_rpm-7.0.0/lib/new_relic/agent/instrumentation/net_http/chain.rb:16:in `block in request_with_newrelic_trace'
/Users/user/.rvm/gems/ruby-2.6.5/gems/newrelic_rpm-7.0.0/lib/new_relic/agent/instrumentation/net_http/instrumentation.rb:26:in `block (2 levels) in request_with_tracing'
/Users/user/.rvm/gems/ruby-2.6.5/gems/newrelic_rpm-7.0.0/lib/new_relic/agent/tracer.rb:371:in `capture_segment_error'
/Users/user/.rvm/gems/ruby-2.6.5/gems/newrelic_rpm-7.0.0/lib/new_relic/agent/instrumentation/net_http/instrumentation.rb:25:in `block in request_with_tracing'
/Users/user/.rvm/gems/ruby-2.6.5/gems/newrelic_rpm-7.0.0/lib/new_relic/agent.rb:501:in `disable_all_tracing'
/Users/user/.rvm/gems/ruby-2.6.5/gems/newrelic_rpm-7.0.0/lib/new_relic/agent/instrumentation/net_http/instrumentation.rb:24:in `request_with_tracing'
/Users/user/.rvm/gems/ruby-2.6.5/gems/newrelic_rpm-7.0.0/lib/new_relic/agent/instrumentation/net_http/chain.rb:16:in `request_with_newrelic_trace'
/Users/user/.rvm/gems/ruby-2.6.5/gems/airbrake-11.0.1/lib/airbrake/rails/net_http.rb:11:in `block in request'
/Users/user/.rvm/gems/ruby-2.6.5/gems/airbrake-11.0.1/lib/airbrake/rack.rb:25:in `capture_timing'
/Users/user/.rvm/gems/ruby-2.6.5/gems/airbrake-11.0.1/lib/airbrake/rails/net_http.rb:10:in `request'

You can see which files are causing the conflict, including line numbers. This will allow you to know which gem instrumentation is part of this error. Using the example above, the two locations causing the recursion are:

/Users/user/.rvm/gems/ruby-2.6.5/gems/newrelic_rpm-7.0.0/lib/new_relic/agent/instrumentation/net_http/chain.rb:16:in `request_with_newrelic_trace'

/Users/user/.rvm/gems/ruby-2.6.5/gems/airbrake-11.0.1/lib/airbrake/rails/net_http.rb:11:in `block in request'

If you were to pull up the code in those two places, you’d see one uses prepend, and the other uses method chaining. In this specific example, the agent uses method chaining, and Airbrake is uses Module#prepend. Since the agent offers both method chaining and Module#prepend, you can configure the agent to use the strategy compatible with the conflicting gem.

3. Update config file

Taking a closer look at the line from the agent in the stack trace, you can see in the path name what gem instrumentation needs to be configured.

new_relic/agent/instrumentation/net_http/chain.rb:16

The gem name follows new_relic/agent/instrumentation/ in the file path. For this example, you can see that this is our Net::HTTP instrumentation, so you’ll want to use net_http in your config file to control this instrumentation. When looking at the file name, you can also see that this is using method chaining instrumentation. The file name for method chaining is /chain.rb, and for Module#prepend it would be /prepend.rb.

Add the config option to your newrelic.yml, using whatever gem instrumentation you found was part of the conflict. In this example, the error was raised due to a conflict with the method chaining instrumentation. To resolve it, we’ll set our Net::HTTP instrumentation to use Module#prepend:

instrumentation:
  net_http: prepend

If instead we saw the prepend.rb file was referenced in the stack trace, we would instead set this config option to chain.

For details about the agent’s available configuration options, please refer to the instrumentation section of our configuration documentation.

4. Still seeing an error?

If after following the above instructions you are still seeing a stack level too deep error, check and see if this is the same conflict and stack trace, or if this is a different conflict than the one you just fixed.

If this is a different conflict with a different stack trace, repeat the above steps with the new stack trace. That should resolve the newly surfaced conflict.

If the stack trace is the same as the first one, check to make sure the agent can load your configuration file. If the agent is having trouble locating or loading the configuration file, resolving that issue should allow the agent to use the gem instrumentation you need. Please see our configuration documentation for more information.

Cause

When one location in your application (or a gem used in your application) uses Module#prepend on a method that another location (or gem) later uses an alias_method on, it creates a non-terminating recursion and throws the SystemStackError: stack level too deep error.

With the 7.0 release of the Ruby agent, Module#prepend instrumentation has been added for all gem instrumentation that previously only used method chaining. Module#prepend will also be used by default in most cases. The agent still allows method chaining to be used for gem instrumentation, this behavior is controlled by the agent configuration.

When the agent uses the default value for gem instrumentation configuration, it will check the environment for common known conflicts with Module#prepend. If a gem that causes this conflict is detected, the agent instead installs the method chaining instrumentation.

However, we don’t know all of the possible conflicts. Any gem or application can add method chaining to any method. This is why we offer the option to configure which type of instrumentation is used.

Here are just a few examples of known conflicts the agent checks for and installs the compatible gem instrumentation.

Net::HTTP Instrumentation

Uses method chaining on Net::HTTP methods when Airbrake < 10.0.2 is being used. Airbrake 10.0.2+ updated to use Module#prepend on Net::HTTP, so the agent will install the prepend instrumentation when that version or higher is detected.

Resque Instrumentation

Uses method chaining on Reque methods when Airbrake < 11.0.3 is being used. Airbrake 11.0.3+ updated to use Module#prepend on Resque, so the agent will install the prepend instrumentation when that version or higher is detected.

Redis Instrumentation

Uses method chaining when PrometheusExporter is detected, as that gem uses method chaining on redis methods.