Introspecting Elixir — cheatsheet

OK, so you wrote your amazing Elixir or Phoenix application and deployed it. It all starts clean & nice but sooner or later tricky runtime issues will come up. Thankfully, when it comes to introspection, Elixir gives a great set of tools for plugging into running applications, both local and remote, and checking what the hell is going on there. Sections below present each of these tools in a brief manner for fast use when things go wrong.

Remote Setup

Prerequisite for further sections. Ensures that application will be run with name and cookie, making it possible to connect to it via Erlang Distribution.

When application is run via Mix (eg. mix phx.server):

Change the mix <task> command to elixir --name server@127.0.0.1 --cookie abc -S mix <task> . Restart running process if needed.

When application is run in release (eg. built via exrm):

Ensure the following content lands in running-config/vm.args in release:

-name server@127.0.0.1
-setcookie abc

Remote Shell

Opens and plugs Elixir console (IEx) to application running on remote node (hooked to already running VM instead of creating new one).

Useful for:

• Production environments (with attention to unintented side effects)
• Executing arbitrary code on the node
• Checking state at specific step via ETS queries & calls to running processes
• Executing most of introspection techniques described below

Steps:

1. Set name & cookie of runtime (described in Remote Setup section)
2. Forward port mapper (ssh user@myapp.com -L4369:localhost:4369)
3. Check out the port of remote app (epmd -names)
4. Forward app (ssh user@myapp.com -L9001:localhost:9001)
5. Connect IEx to remote app (iex --name shell@127.0.0.1 --cookie abc --remsh server@127.0.0.1)

More info:

• Connect to Running Elixir Applications with IEx Remote Shell
• Tracing and observing your remote node

Observer

Opens Erlang Observer (graphical tool) that allows to introspect running application in real-time and without blocking it.

Useful for:

• All environments
• Checking tree of applications and their processes
• Checking ETS tables
• Checking VM memory usage
• Checking process message queue consumption
• Checking process CPU usage
• Checking system stats

Steps:

1. Open remote shell (described in Remote Shell section)
2. Start observer (:observer.start)
3. Once done, stop observer (:observer.stop)

More info:

• Elixir in times of microservices
• Observing Processes in Elixir — The Little Elixir & OTP Guidebook

Pry

Plugs Elixir console to specific point in application code at runtime and blocking the runtime for purposes of introspection.

Useful for:

• Environments that may be blocked (usually only development)
• Introspecting local binding and process state at specific step
• Executing arbitrary code on the node with local binding as arguments
• Checking state at specific step via ETS queries & calls to running processes

Steps:

1. Open remote shell (described in Remote Shell section)
2. Require IEx module on top of target file (require IEx)
3. Add pry call in place that needs blocking & debugging (IEX.pry)
4. Once done, finish the pry session (respawn)

More info:

• Debugging techniques in Elixir
• Debugging Phoenix with IEx.pry

Debugger

Runs a debugging session against application by invoking Erlang Monitor (graphical tool) and blocking runtime execution.

Useful for:

• Environments that may be blocked (usually only development)
• Checking execution step by step with insight to local binding (Erlang-ish)
• Checking state at specific step via ETS queries & calls to running processes

Steps:

1. Open remote shell (described in Remote Shell section)
2. Start debugger (:debugger.start)
3. Introspect target module (:int.ni(MyApp.MyModule)
4. Add breakpoints in Monitor
5. Once done, stop debugger (:debugger.stop)

More info:

• Debugging techniques in Elixir

Tracer

Traces processes and function calls without blocking a running application.

Useful for:

• All environments
• Tracing arguments and results of calls without blocking them

Steps:

1. Install erlyberly
2. Set name & cookie of runtime (described in Remote Setup section)
3. Start erlyberly (java -jar target/*runnable.jar)
4. Connect with proper node name (server@127.0.0.1) and cookie (abc)

More info:

• How to trace Elixir nodes with Erlyberly

Hot Code Reload

Applies new code on running application without restarting it.

Useful for:

• Production environments (with attention to impact of each change)
• Logging local binding, process state or arbitrary output at specific step
• Testing code changes and patches in runtime

Steps:

1. Open remote shell (described in Remote Shell section)
2. Upload modified target file to remote server (or create it in-place)
3. Compile and reload modules from target file (c "lib/my_file.ex")

More info:

• Hot Code Reload in Elixir

Summary

I’ve refrained from any unnecessary comments in order to stick to the cheatsheet shape, but here are some extra points on the subject:

• Using Remote Shell the way I’ve presented is insecure due to lack of SSL in Erlang Distribution, but unfortunately an alternative technique described here (in which console is being opened directly on remote machine) won’t allow to run GUI tools which rules out Observer and Debugger. Look into SSH for Erlang Distribution and CLI Observer for ways to make things more secure.
• There are other Tracer tools besides erlyberly. Take a look at redbug, recon_trace or built-in Erlang dbg module for alternatives.
• It may be controversial to recommend Hot Code Reload as a debugging or introspection technique, but I think it’s worth knowing that such means exist and can be used — with extreme care if engaged on production. It’s definitely one of unique traits of Elixir to recompile and instantly start using a module in a running system without restarting it, which makes it more flexible even vs dynamic languages like Ruby in environments like Heroku where you can’t restart a server with changed code.

Finally, please do comment if any of described procedures can be simplified or refined, or if any of important methods for introspecting Elixir was omitted.