Changing Log Levels At Runtime - Rails

Question: how do you change the log level of a running Rails application?

Answer: You don't.

Technically the Rails docs inform us that we just need to do.

config.log_level = :warn # In any environment initializer, or
Rails.logger.level = 0 # at any time

However, that "at any time" is doing a lot of heavy lifting. You'd have to ssh into each server, edit the config file, restart the server, and then hope that you didn't make a typo.

As you can imagine, I'm here to show you something better, but first let's think about why you might want to change the log level of a running application in the first place and "what problem are we trying to solve".

Why change log levels?

Typicaly we want to change log levels because there's a bug that's hard to reproduce locally and there's just no substitute for understanding just exactly what is happening in our production or staging environment. Trying to use config.log_level = :debug to solve this is like trying to do surgery with a sledgehammer. You're going to end up with a lot of collateral damage (log aggregation expense) and a lot of wasted time.

This is because config.log_level = :debug is so non-specific. If you're trying to debug a user 4234's billing issue, do you need to see the logging from every single template render for every user? Probably not.

If you had a magic wand, what you'd really love would be able to target the logging to:

• The billing code
• The billing Sidekiq job
• User 4234
• Just for the next hour while you're debugging

But can we really do that? Yes, we can, and you're 10 minutes away from trying it yourself.

So, how do we do that?

First things first. We aren't going to change your log aggregator. You can still use DataDog of Logtail or whatever you like. We're just going to help you get much more value out of them.

Second, we aren't going to change your logging code. All of your Rails.logger.info or Rails.logger.debug are perfect just the way they are.

Here's what it does. Psuedo code is worth 1000 words, so here's the psuedo of what happens:

class PrefabLogger < ::Logger
  # path = app.models.billing.calculate_tax
  # level = :debug
  def log(message, path, level)
    Prefab.get("log-level-#{}") > level
      ...do the logging
    end
  end
end

class Prefab
  def get(key)
    @dynamic_config_map.get(key, current_context)
  end
end

Rails.logger = PrefabLogger

Now of course we've moved the heavy lifting to our @dynamic_config_map, but this is pretty simple to conceptualize. The map is a threadsafe Concurrent::Map.new. It will be populated from a CDN with the latest values, and it will be updated in real time as the values change. The values in the map can have rules inside them so that we can filter the log levels based on the context of the request.

To get the current_context in the above pseudo code, we'll just set some properties in an around_action in our ApplicationController.

Dynamic logging at it's core is just a special case of dynamic configuration. It's a simple solution that gives us a ton of power.

The User Experience

So, how do we actually use this? The Prefab UI has us covered. In the LogLevel UI, we'll see a list of all the log levels that are currently being used in our application. For any package, any class and even any method, we can simply click and change log level of any of these loggers.

We can also target specific loggers by using a targeted logger. This has the same targeting power as the Prefab Feature Flag system, so you'll have no problem laser targeting the loggers that you want to change.

That's it! Truly it's that simple. It's also very very easy to give this a try. Cut a branch, Signup, throw in your API key and set the logger, run your app and start using dynamic logging in just a few minutes. Full documentation for the ruby-sdk.

Conclusion

I hope you enjoyed this quick tour of dynamic logging. It's a simple solution to a common problem and once you get used to it, you'll wonder how you ever lived without it. Over time it will start to change how you think about logging. Without it, there's not a ton of point putting in Rails.logger.debug statements, because you're never going to see them, but with it, you can start to think about logging as an as-needed tool that you have in your pocket for when you need it most.

So, I encourage you to give dynamic logging a try, and experience the benefits of fine-tuning your log output. Happy debugging! 🚀