Akka Dead Letters and How to Monitor for Them

So we are a little way into this mini series on Akka now.

We have covered a few of the fundamental topics of Akka such as:

• Supervision
• Mailboxes
• Logging

When we talked about mailboxes, we discussed the fact that there were queues involved that are used as mailboxes.

Generally speaking where there are queues concerned, there is also the possibility of dead letters.

In this post, we will talk about what ‘dead letters’ are within Akka, and also look at how you can monitor for dead letters.

What Are Dead Letters?

In Akka, messages that can’t be delivered are routed to a synthetic actor which has the path “/deadLetters”. This is for NON transport lost messages.

Akka makes no guarantees for lost messages at the transport layer.

How Can You Monitor For Them?

Within Akka, there is a concept of a event bus. That is a bus that Akka uses internally to send messages. You can also use this event bus for other purposes, such as publishing/subscribing to certain types of messages.

It is kind of like a topic based subscription system.

We can use this event bus to monitor for dead letters.

It is also worth noting that the event bus can be used a general pub/sub mechanism, you can read more about it here: Akka Event Bus.

But for now, let’s get back to the problem at hand which is how do we monitor for dead letters?

So let’s start out with this actor code:

import akka.actor.Actor
import akka.event.Logging
 
class LifeCycleActor extends Actor {
  val log = Logging(context.system, this)
 
  log.info("LifeCycleActor: constructor")
 
  override def preStart { log.info("LifeCycleActor: preStart") }
 
  override def postStop { log.info("LifeCycleActor: postStop") }
 
  override def preRestart(reason: Throwable, message: Option[Any]) {
    log.info("LifeCycleActor: preRestart")
    log.info(s"LifeCycleActor reason: ${reason.getMessage}")
    log.info(s"LifeCycleActor message: ${message.getOrElse("")}")
    super.preRestart(reason, message)
  }
  override def postRestart(reason: Throwable) {
    log.info("LifeCycleActor: postRestart")
    log.info(s"LifeCycleActor reason: ${reason.getMessage}")
    super.postRestart(reason)
  }
  def receive = {
    case RestartMessage => throw new Exception("RestartMessage seen")
    case _ => log.info("LifeCycleActor : got a message")
  }
}

We will then use this actor which is setup to receive dead letters:

import akka.actor.{DeadLetter, Actor}
 
class DeadLetterMonitorActor 
  extends Actor 
  with akka.actor.ActorLogging {
  log.info("DeadLetterMonitorActor: constructor")
 
  def receive = {
    case d: DeadLetter => {
      log.error(s"DeadLetterMonitorActor : saw dead letter $d")
    }
    case _ => log.info("DeadLetterMonitorActor : got a message")
  }
}

We then use this demo code:

import akka.actor._
import scala.language.postfixOps
import scala.io.StdIn
import akka.event.Logging
 
class Runner {
 
  def Run(): Unit = {
 
    //create the actor system
    val system = ActorSystem("LifeCycleSystem")
 
    // default Actor constructor
    val lifeCycleActor =
      system.actorOf(Props[LifeCycleActor],
        name = "lifecycleactor")
 
    val deadLetterMonitorActor =
      system.actorOf(Props[DeadLetterMonitorActor],
        name = "deadlettermonitoractor")
 
    //subscribe to system wide event bus 'DeadLetter'
    system.eventStream.subscribe(
      deadLetterMonitorActor, classOf[DeadLetter])
 
    val log = Logging(system, classOf[Runner])
    log.debug("Runner IS UP BABY")
 
    log.debug("sending lifeCycleActor a few numbers")
    lifeCycleActor ! 100
    lifeCycleActor ! 200
    Thread.sleep(1000)
 
    log.debug("sending lifeCycleActor a poison pill (kill it)")
    lifeCycleActor ! PoisonPill
    Thread.sleep(1000)
    log.debug("sending lifeCycleActor a few numbers")
    lifeCycleActor ! 100
    lifeCycleActor ! 200
 
 
    log.debug("stop lifeCycleActor/deadLetterMonitorActor")
    system.stop(lifeCycleActor)
    system.stop(deadLetterMonitorActor)
 
    //shutdown the actor system
    log.debug("stop actor system")
    system.terminate()
 
    StdIn.readLine()
  }
}

To do the following:

• Create the 2 actors above
• Subscribe the DeadLetterMonitorActor (above) to the “DeadLetter” topic of the inbuilt akka EventStream (internal pub/sub bus)
• We then send a few messages to the LifeCycleActor
• Then send the LifeCylceActor a PoisonPill (Which terminates it)
• We then send a few more messages to the LifeCycleActor (which it doesn’t get cos it's dead like)
• We expect the DeadLetterMonitorActor to receive the DeadLetter messages and log these DeadLetter messages

If we run these bits of code, this is the result, where it can plainly be seen that the DeadLetterMonitorActor does receive the DeadLetter messages:

<font size="1">INFO [LifeCycleSystem-akka.actor.default-dispatcher-2] Slf4jLogger – Slf4jLogger started
16:55:49.546UTC DEBUG[LifeCycleSystem-akka.actor.default-dispatcher-2] EventStream – logger log1-Slf4jLogger started
16:55:49.547UTC DEBUG[LifeCycleSystem-akka.actor.default-dispatcher-2] EventStream – Default Loggers started
16:55:49.559UTC INFO [LifeCycleSystem-akka.actor.default-dispatcher-2] LifeCycleActor – LifeCycleActor: constructor
16:55:49.560UTC INFO [LifeCycleSystem-akka.actor.default-dispatcher-2] LifeCycleActor – LifeCycleActor: preStart
16:55:49.560UTC DEBUG[LifeCycleSystem-akka.actor.default-dispatcher-2] Runner – Runner IS UP BABY
16:55:49.561UTC DEBUG[LifeCycleSystem-akka.actor.default-dispatcher-2] Runner – sending lifeCycleActor a few numbers
16:55:49.561UTC INFO [LifeCycleSystem-akka.actor.default-dispatcher-2] DeadLetterMonitorActor – DeadLetterMonitorActor: constructor
16:55:49.563UTC INFO [LifeCycleSystem-akka.actor.default-dispatcher-2] LifeCycleActor – LifeCycleActor : got a message
16:55:49.563UTC INFO [LifeCycleSystem-akka.actor.default-dispatcher-2] LifeCycleActor – LifeCycleActor : got a message
16:55:50.561UTC DEBUG[LifeCycleSystem-akka.actor.default-dispatcher-2] Runner – sending lifeCycleActor a poison pill (kill it)
16:55:50.576UTC INFO [LifeCycleSystem-akka.actor.default-dispatcher-9] LifeCycleActor – LifeCycleActor: postStop
16:55:51.565UTC DEBUG[LifeCycleSystem-akka.actor.default-dispatcher-9] Runner – sending lifeCycleActor a few numbers
16:55:51.570UTC DEBUG[LifeCycleSystem-akka.actor.default-dispatcher-2] Runner – stop lifeCycleActor/deadLetterMonitorActor
16:55:51.573UTC DEBUG[LifeCycleSystem-akka.actor.default-dispatcher-7] Runner – stop actor system
16:55:51.581UTC INFO [LifeCycleSystem-akka.actor.default-dispatcher-9] RepointableActorRef – Message [java.lang.Integer]
    from Actor[akka://LifeCycleSystem/deadLetters] to Actor[akka://LifeCycleSystem/user/lifecycleactor#-912643257]
    was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration
    settings ‘akka.log-dead-letters’ and ‘akka.log-dead-letters-during-shutdown’.
16:55:51.581UTC INFO [LifeCycleSystem-akka.actor.default-dispatcher-9] RepointableActorRef – Message [java.lang.Integer]
    from Actor[akka://LifeCycleSystem/deadLetters] to Actor[akka://LifeCycleSystem/user/lifecycleactor#-912643257]
    was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration
    settings ‘akka.log-dead-letters’ and ‘akka.log-dead-letters-during-shutdown’.
16:55:51.588UTC ERROR[LifeCycleSystem-akka.actor.default-dispatcher-8] DeadLetterMonitorActor – DeadLetterMonitorActor :
    saw dead letter DeadLetter(100,Actor[akka://LifeCycleSystem/deadLetters],Actor[akka://LifeCycleSystem/user/lifecycleactor#-912643257])
16:55:51.590UTC ERROR[LifeCycleSystem-akka.actor.default-dispatcher-8] DeadLetterMonitorActor – DeadLetterMonitorActor :
    saw dead letter DeadLetter(200,Actor[akka://LifeCycleSystem/deadLetters],Actor[akka://LifeCycleSystem/user/lifecycleactor#-912643257])
16:55:51.609UTC INFO [LifeCycleSystem-akka.actor.default-dispatcher-6] LocalActorRef – Message [akka.actor.StopChild]
    from Actor[akka://LifeCycleSystem/deadLetters] to Actor[akka://LifeCycleSystem/user] was not delivered. [3] dead letters encountered.
    This logging can be turned off or adjusted with configuration settings ‘akka.log-dead-letters’ and ‘akka.log-dead-letters-during-shutdown’.
16:55:51.616UTC DEBUG[LifeCycleSystem-akka.actor.default-dispatcher-6] EventStream – shutting down: StandardOutLogger started</font>

NOTE: According to the official Akka docs “Dead letters are not propagated over the network, if you want to collect them in one place, you will have to subscribe one actor per network node and forward them manually.”

Are All Messages In DeadLetters A Problem

No, not all of the messages that end up in the deadLetters synthetic actor are problems. For example, suppose an actor is sent several Terminate messages. Only one of these will take effect, the others will likely end up in deadLetters, but this is not a concern.

The deadLetters actor and the monitoring of it is more of a debugging aid than anything else. You will have to use some modicum of common sense when examining the results of the deadLetters logging, which we discussed above

Where Is The Code?

As previously stated, all the code for this series will end up in this GitHub repo:

• https://github.com/sachabarber/SachaBarber.AkkaExamples