The Dev Tavern RSS Feed

5 Things that will change the way you use AWS Lambda

Sat, 03 Sep 2022 10:00:08 GMT

Introduction

AWS Lambda is one of the main precursors of Functions as a Service. This paradigm introduced a significant shift in how we develop applications and reason about them. The promise is simple: You push your code, specify your handler and the Functions runtime will take care of running it and scaling it on demand!

However, this apparent simplicity hides a learning curve that is not as flat as it seems. While using Lambda, I stumbled into many issues and learned many things that improved my usage over time and that I wish I had known from the start. Here is a short list of some of the most important things that changed how I use AWS Lambda.

Lambda timeouts cause cold starts!

Lambda functions can be configured to timeout after some time. This is useful to guarantee an upper limit to the execution time of function invocations. For a long time, they really seemed harmless to me. But in reality, their occurrence is much more problematic than you think, especially in high throughput or user facing apps. Why? Because they come with a high cost, both performance, and observability wise:

Performance wise: When a Lambda Function reaches a timeout (similarly to an out of memory error), Lambda does not only stop the current invocation. It kills the whole execution environment brutally with it! Think of it as a process sigkill against the entire execution environment. Consequently, the environment becomes non-reusable, and the following invocation needs to spin up a new environment again, incurring a cold start and causing a big latency spike in your endpoint. This behavior is documented in the AWS Lambda guide but is not visible enough considering its high impact. You can read about it in the invoke phase in the Lambda runtime environment page.

Function with a runtime timeout
Observability wise: Lambda timeouts are literally like unplugging the execution environment from the power source. The logs will be cut off, and your code won’t have the chance to report any metrics, logs, traces, or exceptions that would have been useful to debug the issue. In fact, you won’t have much more information in the logs than Task timed out, and you will likely have no clue where exactly your code hung.

For these reasons, Lambda timeouts should never be part of an expected workflow: If they happen, it means you have a bug in your application that you should investigate and fix. Treat them as an emergency exit door and set the threshold high enough to have them occur only on extremely rare occasions (ex. a bug that caused your code to hang). Instead, implement a custom timeout logic in your code if you want to set a hard limit to client calls. You can use this opportunity to report an informative error message to the user, a custom status code, and some timing metrics showing what took that much time.

Wondering how to report such custom metrics? Nice transition to the next bullet point!

Custom metrics from Lambdas via logs using EMF

AWS introduced a compelling way to report custom metrics from your Lambda functions: The Embedded Metrics Format (or EMF).

EMF lets you report custom metrics in the form of log statements shaped in a specific way described by the EMF specification. These logs are automatically intercepted, parsed, and transformed by CloudWatch into fully fledged metrics that you can graph and create alarms on. Your Lambda function can emit those logs using ordinary log reporters (ex. console.log for javascript, log4j for java, etc.) and does not need to perform network calls or depend on additional client libraries.

Here is an example of an EMF formatted log statement:

{
  "_aws": {
    "Timestamp": 1574109732004,
    "CloudWatchMetrics": [
      {
        "Namespace": "My Namespace",
        "Dimensions": [["functionVersion"]],
        "Metrics": [{ "Name": "time", "Unit": "Milliseconds" }]
      }
    ]
  },
  "functionVersion": "$LATEST",
  "time": 100,
  "requestId": "989ffbf8-9ace-4817-a57c-e4dd734019ee",
  "sessionId": "1234"
}

The _aws field contains the metric definition with its name, unit, and a list of dimensions. This needs to be present in every EMF log statement. The other fields contain the metric values, dimension values, and extra properties that attach additional context to the metric statement. The additional properties will not be captured by CloudWatch metrics but can be extremely useful when deep diving into the logs.

Optimize your Lambdas latency by running your init code once

Lambda execution environments are reused for multiple invocations. This is referred to as warm starts. Variables initialized outside your handler function ¹ will keep their value on subsequent invocations running on the same environment. This is a perfect place to store database connections, http pools, and other objects that are costly to create / initialize. That way, you won’t need to recreate them again on every invocation.

So instead of writing something like this (code example in kotlin):

class Handler: RequestHandler<Request, String> {

  /**
   * This is your handler function 👇 
   */
  override fun handleRequest(event: Request, context: Context): String {

    // dbPool initialized inside the handler function
    val dbPool = initializeDatabasePool();

    // Do something with your database pool
    ...

    return response;
  }
}

You could write this instead:

class Handler: RequestHandler<Request, String> {

  // dbPool initialized outside the handler function
  val dbPool = initializeDatabasePool();

  /**
   * This is your handler function 👇 
   */
  override fun handleRequest(event: Request, context: Context): String {

    // Do something with your database pool
    ...

    return response;
  }
}

The initializeDatabasePool() call in the second snippet will only execute during the init phase (cold starts), and the result will be kept in the dbPool variable for later invocations that reuse the execution environment.

Background code will continue to run on the next invocations

Background processes, threads, and promises that were initiated by your Lambda function and did not complete when the function ended will resume on subsequent invocations if Lambda reuses the execution environment. This can result in surprising behaviors that are very hard to debug.

The explanation is simple: When a function invocation is done, i.e. your handler function returned, the execution environment is frozen, and any code that was still running is paused. When a new invocation starts, the execution environment unfreezes, and all the incomplete processes resume.

This can happen much more easily than you think! Take a look at this innocent-looking javascript code:

const values = await Promise.all([
    asyncFunction1(), // returns a promise
    asyncFunction2(), // returns a promise
])

Promise.all waits for all the given promises until they all resolve. If any of them fail, Promise.all returns early and stops waiting for the other promises. In the example above, If the asyncFunction1 promise fails, the Promise.all call returns and throws an error without waiting for the completion of the asyncFunction2. In the meantime, asyncFunction2 is potentially still running in the background! If it didn’t complete before the Lambda function returns, it will continue running during the following invocation, potentially causing a resource leak.

Thus, to keep your lambda logic predictable, ensure you are not leaking any background threads or promises across function calls, as they won’t magically disappear!

One way to fix the example above is to use Promise.allSettled instead of Promise.all . Promise.allSettled waits for all the promises even if some of them fail.

Code size matters

A common mistake is thinking that the language runtime (ex. JVM) and the logic you put in your init code are the only factors that impact your functions’ cold starts. However, another critical factor that has a significant impact is the package size of your function.

During the init phase, Lambda starts by downloading the function’s package which is stored in an internal Amazon S3 bucket (or Amazon Elastic Container Registry if the function uses container packaging).

Cold start phases

Thus, the heavier your package is, the longer the download phase will be. It can even reach a point where it outweighs all the other cold start phases.

Thus, the package size should have an important consideration in your attempts to optimize lambdas latency. Common optimization techniques to reduce package sizes include: splitting the code, reducing dependencies, and other language specific techniques like tree shaking in the javascript world. In addition, keeping your package small will keep you further away from the dreaded 260Mb limit above which Lambda will not let you upload your code anymore. Believe me, the frustration is huge when this happens!

The handler function is the function that contains the logic that runs on every invocation.↩

Structured concurrency explained - Part 2: Exceptions and Cancellations

Wed, 23 Mar 2022 10:00:08 GMT

Introduction

In part 1 of this series, we introduced the concept of structured concurrency. We showed how it immediately makes concurrent programs safer, more correct, and easier to reason about. If you missed this part, I definitely recommend checking it out!

To achieve this safety in every situation, including when things fail, structured concurrency is very strict and opinionated about the way exceptions are handled and propagated throughout coroutines job hierarchies. This behavior can sometimes seem counterintuitive and confusing if not understood correctly. This is precisely what we will cover in this part 2 of the series. More specifically, we will go over the following topics:

Exception handling inside a coroutine scope
The confusing part!
Using supervisorScope when child failures are not fatal
Cancellation and blocking code

Ready? Let’s go!

Exception handling inside a coroutine scope

Let’s start our journey with this simple example: We want to create a simple pipeline between a Producer coroutine and a Consumer coroutine exchanging data through a channel. Both coroutines run concurrently within a coroutine scope:

The Producer continuously calls the fetchDataFromRemote() function to download some data from a remote service before piping it out to a channel.
The Consumer continuously polls data from the channel and prints it on the screen.

fun runPipeline() {
    coroutineScope {
        val channel = Channel<String>()

        launch(CoroutineName("Producer")) {
            while (true) {
                val data = fetchDataFromRemote()
                channel.send(data)
            }
        }

        launch(CoroutineName("Consumer")) {
            for (data in channel) {
                println(data)
            }
        }
    }
}

In the setup above, our coroutines job hierarchy would look like the following:

Coroutines job Hierarchy describing example 1

Now, let’s imagine that the fetchDataFromRemote() call throws an exception (ex. connectivity error).

fun runPipeline() {
    coroutineScope {
        val channel = Channel<String>()

        launch(CoroutineName("Producer")) {
            while (true) {
                // Boom!! Throws an exception!!!
                val data = fetchDataFromRemote()
                channel.send(data)
            }
        }

        launch(CoroutineName("Consumer")) {
            for (data in channel) {
                println(data)
            }
        }
    }
}

This would cause the Producer coroutine to fail. What do you think would happen then? Will the exception go unnoticed as we did not log it anywhere? Will we end up with the Consumer coroutine running alone forever, desperately waiting for some data to arrive in the channel?

Fortunately, none of those bad things would happen thanks to structured concurrency: The parent job will notice the Producer coroutine’s failure and will immediately take care of cancelling the other children within the scope (to prevent them from staying active forever) before bubbling the exception up to the outside.

Let’s review this workflow step by step (an animation below will further summarize it):

It all starts with the fetchFromRemote() throwing an exception, causing the Producer coroutine to fail.
The parent job notices the failure of one of its children and, consequently, marks itself as failed and cancels all its other children that are still running.
The child coroutines that are cancelled will throw a CancellationException on their next call to a suspending function (or immediately if they are already suspended). That’s how they trigger their exit.
Once all child coroutines are stopped, the original exception is forwarded out of the coroutineScope call.

These steps are summarized by the animation below¹ :

This is how structured concurrency saves us from leaking exceptions and coroutines when failures occur!

The confusing part

The error handling behavior we saw above is desirable in most scenarios. But sometimes, it might be extremely confusing. Especially when we want to handle the exceptions of the child coroutines ourselves.

Let’s demonstrate this using a slightly different example: We want to fetch and print the Github profile of a user on the screen. We decide that failures are not fatal: if we fail to fetch the Github profile of the user (ex. connectivity error), we just want to print an error message on the screen and prevent the exception from bubbling up.

coroutineScope {
    val profile = async(CoroutineName("Fetcher")) {
        // This call might fail!!
        fetchGithubProfile("wlezzar")
    }

    try {
        println(profile.await())
    } catch (err: Exception) {
        println("Failed to fetch data from github: ${err}")
    }
}

The associated job hierarchy would look like this:

Coroutines job Hierarchy describing example 2

We would think that catching the exception during the .await() call would safely prevent the exception from bubbling up. But, unfortunately, that’s not the case! In fact, if you run the program above and simulate a failure in the fetchGithubProfile call, the whole program will crash, and the exception will propagate outside of the coroutineScope. Why is that?

Again, the explanation is the same as in the previous example: by the time we catch the exception at the .await() call, it is already too late, the associated child coroutine (spun up using async ) would have already failed as we did not catch the exception inside the async { ... } block. Thus, the same mechanics as previously will happen: The parent job will notice its child failure, marks itself as failed, cancels the other remaining child coroutines, and bubbles the exception up. The try ... catch around the .await() only allows us not to exit immediately and gives us a chance to react to the failure (ex. do some resources cleaning).

To prevent this scenario from happening, we have two options:

Either we catch the exception inside the async block itself. This would prevent the child coroutine from failing in the first place.
Or we could use supervisorScope instead of coroutineScope to ignore child coroutine failures.

Using `supervisorScope` when child failures are not fatal

Sometimes, like in the last example, child coroutines failures are not fatal and should not cause the entire job hierarchy to fail. Instead, we would like to react to those failures in our own way. This is why supervisorScope exists. It lets us do exactly that.

supervisorScope is a slightly modified version of coroutineScope . Both have the same purpose of creating a new coroutine scope with an explicit lifetime in which we can spin up child coroutines. The difference between the two is that: supervisorScope uses a SupervisorJob as the parent job to hold the child coroutines whereas coroutineScope uses a regular Job instead.

SupervisorJob is a special implementation of Job that ignores child failures. When those occur, it does not cancel the other children coroutines and does not bubble up the exception.

Here is how it can be used in the previous example:

supervisorScope {
    val profile = async(CoroutineName("Fetcher")) { 
        fetchGithubProfil("wlezzar")
    }

    try {
        println(profile.await())
    } catch (err: Exception) {
        println("Failed to fetch profile data: ${err}")
    }
}

Now, when a failure happens in the fetchGithubProfile call, this will still cause the child coroutine to fail but the parent job (SupervisorJob) will ignore this failure, and the supervisorScope will not bubble it up to the outside.

One critical point to keep in mind: whenever we use a supervisor job as a parent in a scope, it becomes very important to catch the exceptions of child coroutines ourselves, otherwise they would be lost!

The way we handle a child coroutine’s exception when using a supervisor job as a parent depends on whether this coroutine was spinned up using launch or async:

Failures of coroutines spinned using async are caught by wrapping the .await() inside a try...catch.
On the other hand, failures of coroutines spinned up using launch can be handled using coroutine exception handlers.

Cancellation and blocking code

When failures occur, the parent job is responsible for triggering the cancellation of its child coroutines. However, cancellation is cooperative! This means that when coroutines are asked to cancel, they need to cooperate by listening to those cancellation signals and reacting appropriately (ex. by exiting). You cannot force them to! In this case, we say that a coroutine is cooperative to cancellation or cancellable.

In the Kotlin coroutines world, you rarely need to worry about making your coroutines cooperative because all the suspending functions (like delay, consuming from a channel, etc.) are natively already cancellable. Thus, your code will usually be cancellable right away without any extra effort.

However, when running a blocking code inside coroutines, you do need to worry about making it cooperative to cancellation.

Here is a hypothetical example:

// Blocking code
fun someSlowBlockingCode() {
    // Do some computation
}

val job = launch(Dispatchers.IO) {
    someSlowBlockingCode()
}

job.cancel()
job.join() // Will hang until the blocking code is done!

The call to someSlowBlockingCode() is not coroutines aware. It will not suspend and will never react to cancellations. Consequently, canceling the associated job will not cause the blocking function to interrupt. Instead, it will keep running until it’s done.

How can we fix this?

There are several approaches depending on whether you own or not the implementation of the blocking code.

One approach to make a blocking code cooperative to cancellations is to wrap it inside a runInterruptible block, which is specifically built for that purpose.

val job = launch(Dispatchers.IO) {
    runInterruptible {
        someSlowBlockingCode()
    }
}

job.cancel()
job.join()

When the coroutine is cancelled, runInterruptible interrupts the underlying thread before throwing a CancellationException to the outside. However, for this to work, the blocking call underneath must itself be cooperative to thread interupts which is not guaranteed! This is usually the case when:

The thread is blocked on Object.wait(), Thread.sleep(), or Thread.join()
Most java.util.concurrent structures are interruptible.
When using Java NIO (but not java.io).

You can find more information around thread interrupts in the Thread.interrupt() javadoc.

Despite not being a bullet proof solution, the runInterruptible function is still very useful in many situations, especially when you don’t have control over the implementation of the underlying blocking code.

Otherwise, if you do own the blocking code’s implementation, another way to make it cooperative to cancellations is by periodically watching the value of the isActive property of the enclosing coroutine scope. When this value is false, it means that the enclosing scope has been cancelled, and the blocking code should exit. Here is an example:

fun someSlowBlockingCode(scope: CoroutineScope) {
    while (scope.isActive) {
        // Do some short living blocking operation
    }
}

val job = launch(Dispatchers.IO) {
    someSlowBlockingCode()
}

job.cancel()
job.join()

Conclusion

Error handling in the context of coroutines is certainly one of the most challenging parts to understand and to master. Hopefully, this post gave you some clarity on its inner workings and the reasons why it was designed this way.

So far we have only seen how to create coroutine scopes inside suspend functions using coroutineScope and supervisorScope. In the third and last part of this series, we will discover another useful way of creating independent scopes that are not tied to the lifetime of a function call. This can be achieved using the CoroutineScope factory function. We will also talk about context elements and how they allow us to easily pass data to child coroutines. Stay tuned!

The names used to express the statuses of the jobs in the different steps are for clarity and illustration purpose. Please check the documentation of Job for the actual state machine and status names.↩

Structured concurrency explained - Part 1: Introduction

Sun, 13 Feb 2022 10:00:08 GMT

Introduction

Structured concurrency is a powerful concept that is important to understand to fully take advantage of the power of Kotlin coroutines.

Today’s post is the first of a series about Structured concurrency: an introduction that takes the challenge of explaining the concepts behind it in a straightforward and intuitive manner. It aims to trigger in you the “Aha!” moment that will make you say: Why doesn’t every single concurrency library use structured concurrency?! Stay tuned ;)

Unstructured concurrency

To understand something and build a good intuition around the subject, it is often helpful to start from the problems it tries to solve in the first place. So let’s consider a simple example program that does not use structured concurrency and see what problems can arise.

The following code creates three concurrent tasks that run in the background in separate threads. Each task’s job is to fetch a bunch of data from a remote service using the fetchData function and add the resulting number to the global sum.

val sum = AtomicInteger(0)

// Create 3 concurrent tasks that compute the total sum in parallel
repeat(3) {
    CompletableFuture.runAsync {
        val data: Int = fetchData()
        sum.addAndGet(data)
    }
}

// Do something useful in the meantime while the sum is being computed
Thread.sleep(2000)

// And then use the sum.
println("The final sum is: $sum")

You should have already spotted a few problems in the snippet above:

We never wait for the created tasks (futures) to finish their job before using the final sum. The fetchData call might take longer than expected in some of the futures, and we end up using an incomplete sum without being aware.
What if one of the futures fails? The fetchData() call might raise an exception because of a networking problem. The future’s failure will go unnoticed and will never be propagated to the outside. In other words, we can leak exceptions! Causing, once again, our sum to be incorrect without our awareness.
Last but not least: what if one of the futures gets stuck indefinitely and never ends? Let’s say the fetchData call hangs and never returns because of a deadlock or a bug in its implementation. In this case, our future and the thread hosting it will keep running in the background forever, consuming unnecessary resources. We will likely never be aware of it and will have no way of shutting it down. In other words, we can leak threads!

We can try to fix the problems above by adding some boilerplate code to keep track of the futures in a list, wait for them one by one to finish, and forward any exception that would be thrown.

val sum = AtomicInteger(0)

// A list to keep track of the created futures
val futures = ArrayList<Future<Void>>()

repeat(3) {
    val future = CompletableFuture.runAsync {
        val data: Int = fetchData()
        sum.addAndGet(data)
    }

    // Add the future to the list
    futures.add(future)
}

// Do something useful in the meantime in parallel
Thread.sleep(2000)

// Wait for all the futures and propagate exceptions
for (future in futures) {
    future.get()
}

// And then use the sum.
println("The final sum is: $sum")

What we did here is actually “Making our concurrency more structured”: Instead of launching the background futures in the air and losing sight of them, we keep track of them in a list and wait for their completion. That is exactly what we mean by structured concurrency: The start and the end of the concurrent paths are clear and explicit.

However, not only is this boilerplate code cumbersome to write, it is also far from being perfect:

What if the first future encountered by our for loop throws an exception? In that case, the for loop stops iterating and propagates the exception to the outside without waiting for the remaining futures. So if one of them hangs or takes an unreasonably long amount of time to finish, we would still be potentially leaking threads! To solve this edge case, we need to catch the exception and cancel the remaining futures that are still running before propagating the error to the outside. That requires a much more involved boilerplate code than what we wrote.
We didn’t handle user cancellations. What if we want to enable the user to cancel the sum computation in the middle? Doing so would require even more boilerplate code to listen for cancellation signals and to propagate them to the futures we launched one by one.

In summary, correctly structuring concurrency is complex and requires a lot of boilerplate code to make it bug free and 100% safe in all cases.

The good news is: structured concurrency is baked into the design of the Coroutines library, and we do not have to write any boilerplate code to use it and take advantage of it! So let’s check this out!

Structured concurrency with Kotlin Coroutines

Let’s rewrite our code using coroutines and see what that looks like:

suspend fun distributedSum() {
    val sum = AtomicInteger(0)

    // The coroutine scope acts like a parent that keeps track of
    // all the child coroutines created inside it
    coroutineScope {
        // Create 3 coroutines that compute the total sum concurrently
        repeat(3) {
            launch {
                val data: Int = fetchDataAsync()
                sum.addAndGet(data)
            }
        }
    }

    println("The final sum is: $sum")
}

This short and extremely simple code is already free from all the problems mentioned in the previous section: It cannot leak exceptions or coroutines, it will never print an incomplete sum or a wrong one, and cancellation is already possible!

Feels like magic? Let’s unpack it step by step to understand how.

coroutineScope { // this: CoroutineScope (receiver)
    ...
}

We start by calling the coroutineScope function and giving it a kotlin lambda inside which we can create any number of background coroutines using launch. The reason why we are able to use launch inside this lambda is because its receiver¹ is a CoroutineScope instance, of which launch is defined as an extension function.

coroutineScope { // this: CoroutineScope
    repeat(3) {
        // equivelent to: this.launch
        launch {
            ...
        }
    }
}

Any background coroutine created within this block is attached as a child to this CoroutineScope instance which keeps track of its completion and failure. It is similar to what we tried to achieve by manually keeping track of the completable futures in a list. But instead of a list, the CoroutineScope uses a fancier hierarchical data structure called Job. We will dig deeper into this data structure later on in this post.

Here are the goodies that come with this setup:

We don’t need to keep track of the background coroutines ourselves, the CoroutineScope itself already does, and the coroutineScope call will suspend until all of them complete or fail. Thus, we know for sure that after this call, all our background coroutines are done; and we can then confidently use the computed sum.
If any of the background coroutines fails for a reason, the exception is caught by the coroutineScope and propagated to the outside. But first, it takes care of canceling the remaining coroutines that are still running², thus making sure that no background coroutine is still pending after the exception is propagated.

Our background coroutines now have a clear and explicit lifetime defined by the coroutineScope block beyond which it is impossible to leak any coroutines or exceptions. That’s the core idea of structured concurrency: “every time our control splits into multiple concurrent paths, we make sure they join up again” ³. As a result, our program instantly becomes safer and easier to reason about.

The CoroutineScope and the Jobs hierarchy

Let’s take a closer look at the signature of the launch coroutine builder function:

public fun CoroutineScope.launch(...): Job

The launch function requires a CoroutineScope to be usable. But why is that?

The reason is to prevent background coroutines from being created in the air without having a parent CoroutineScope instance to keep track of them. This is the case for all coroutine builder functions, which is how Kotlin enforces structured concurrency.

The CoroutineScope instance keeps track of its children coroutines using a hierarchical data structure called Job. You can think of a job as a simple tree that:

Stores children Job instances to keep track of them (which themselves can have other nested children Job instances).
And has a status (active, completed, etc.).

Structured concurrency - Job Hierarchy

Coroutine builder functions (like launch) also create their own CoroutineScope instance as the receiver of their lambda block, making it possible to create nested coroutines! That enables building complex hierarchies of concurrently running background coroutines without ever leaking a single one of them.

coroutineScope { // this: CoroutineScope
    launch(CoroutineName("Coroutine A")) { // this: CoroutineScope (nested CoroutineScope with its own Job)
        launch(CoroutineName("Coroutine A.1")) { ... }
        launch(CoroutineName("Coroutine A.2")) { ... }
    }

    launch(CoroutineName("Coroutine B")) { ... }
}

The code above would create the following hierarchy:

Structured concurrency - Complex hierarchy

The coroutineScope call will suspend and not complete until Coroutine A and Coroutine B are done. Coroutine A, on the other hand, will not complete until Coroutine A.1 and Coroutine A.2 are done. If Coroutine A.1 fails for some reason, then Coroutine A will fail as a whole, propagating the exception to the parent job, causing coroutineScope itself to fail. Any coroutine still running in this hierarchy will get canceled during the process.

Difference between CoroutineScope and CoroutineContext

One aspect that may confuse a lot of Kotliners is the difference between a coroutineContext and a CoroutineScope.

If you check the source code of CoroutineScope, you will find the following:

public interface CoroutineScope {
    public val coroutineContext: CoroutineContext
}

Why do we need a CoroutineScope then? Why not just use CoroutineContext instead? The difference between both concepts as outlined by Roman Elizarov is in their intended purpose.

A CoroutineContext is simply a data structure holding 0 or several context elements. It is just like a HashMap and serves as a container that coroutines and their schedulers can access at any point in time to retrieve useful information from like the Dispatcher, the coroutine’s name, etc.

On the other hand, a CoroutineScope is like a “manager and a tracker for coroutines”. It is the entity that can create background coroutines using coroutine builder functions defined as extension methods to this class (like launch).

To be able to fullfill its job of creating such coroutines and keeping track of them, the CoroutineScope needs basically two things:

A Job instance to keep track of the children coroutines (to enforce structured concurrency).
Some optional coroutine context elements that we want to propagate to all children coroutines created inside this scope (ex. the Dispatcher).

It turns out that both the requirements above can be satisfied by having a single coroutineContext property:

The coroutineContext property can store the Job instance that will serve as a parent to the children coroutines created by this CoroutineScope. All the functions that instantiate a CoroutineScope will create such a Job and put it there.
It will also hold any additional context element that we want to propagate to the children coroutines.

Roman Elizarov explains this concept clearly in his article about the topic. I highly encourage you to read it as well as his other writings about Kotlin ;).

Conclusion

Structured concurrency makes your concurrency intensive programs safer and more understandable. It frees you from all the boilerplate code that you would otherwise need to write to synchronize your background tasks and make sure you are not leaking any of them.

The next topic of this series will be about Error handling and cancellation, maybe one of the trickiest aspects of structured concurrency. Stay tuned if you don’t want to miss out on that one ;) .

The lambda receiver is the instance referred to by the this keyword inside this lambda’s block.↩
This behavior can be overridden using Supervisor jobs. We will talk mpre about this in the second part of this series around Error Handling and cancellation.↩
Quote taken from Notes on structured concurrency or go statement considered harmful ↩

Why are my coroutines slow?

Mon, 27 Dec 2021 10:47:08 GMT

Introduction

Coroutines are one of Kotlin’s killer features! They do a great job simplifying concurrency intensive programs, making them feel more accessible and intuitive to write.

However, coroutines are also easy enough to misuse. One of the common mistakes that programmers make is accidentally injecting blocking calls inside coroutines (we will see what that means in a second), causing the whole application to perform slower and feel unresponsive. It might not even be you who introduced this blocking call in the first place, but a library function marked as suspend that you are using deep inside your program! And when this happens, chances are that it will take more than a few minutes to troubleshoot and figure out why.

This blog post is here to hopefully save you from wasting your time as I once did! 😅 . We will see how to quickly detect illegitimate blocking calls with BlockHound. But before that, we will cover some theory around how coroutines work, what causes them to be blocked, and a few strategies to handle blocking calls within a coroutines-based application. If you are already a coroutines expert and are not interested in the theory part, feel free to skip to the last part of this post, where we demo the usage of BlockHound.

The symptoms: what causes coroutines to become slow?

When you start feeling that your coroutines based application is becoming slow and unresponsive, in most cases (bets on the table!), the cause is one of these two:

The CPU of your machine is the bottleneck! This is probably the first thing to check. The coroutines machinery is a complex beast and needs CPU to run! Orchestrating and scheduling the many coroutines you have in your application is, in fact, a CPU-intensive task. If all the cores of your machines are nearing 100% when your program is running, then the whole machine will become slow, not only your coroutines! The way out in this case is to either optimize your program to use less CPU, distribute the work more evenly across the available cores, or simply upgrade your machine.
You are inadvertently blocking your coroutines! This mistake is unfortunately easy to make but often hard to find and recover from. This is what we will focus on in the rest of this post.

How is a coroutine blocked?

Let’s take a step back and try to form a high-level understanding of how coroutines work behind the scenes and what blocking them means.

launch {
    println("Hey I'm a coroutine!!")
    delay(5000) // suspends
    println("I'm done!")
}

Coroutines are often referred to as “lightweight threads”. When a coroutine is created, like in the example above, it runs concurrently alongside other coroutines. However, the analogy with threads stops pretty much here.

Coroutines are run by one or a small number of threads (typically the number of cores in your machine). Those threads are called workers in Kotlin’s jargon. One Worker (thread) juggles between several coroutines. It starts working on a coroutine, then suddenly stops its execution in the middle (suspension) to resume working on a completely unrelated coroutine (continuation). This suspension and continuation all happen behind the scenes with the help of some magic from the Kotlin compiler!

A worker (thread) is only allowed to do this jump from one coroutine to the other when it meets a suspension point inside the coroutine it is currently running. A suspension point is declared via the suspendCoroutine function or the likes. In practice, however, you will rarely encounter those functions directly. Instead, you will typically use higher-level functions marked with the suspend keyword that use these lower-level constructs behind the scenes.

Problems start when your coroutine takes a long time executing code without suspending. Consider the following example:

suspend fun httpRequest(url: String): String {
    ...
}

fun saveResponse(response: String, filename: String) {
    ...
}

launch {
    val response = httpRequest("https://example.com/data.json") // Takes about 5s!
    saveResponse(response, filename="/tmp/response.txt") // Takes about 2s!
}

In the code above, the coroutine created via the launch builder starts by performing an http request to fetch some remote data. This call takes around 5s. As it is suspending (i.e., the function is marked with the suspend keyword, and we assume the author’s implementation is correctly suspending), our Worker is not blocked during that time! It will jump out of this coroutine’s execution to progress on other coroutines while the http call is in progress. Thus, the Worker’s time here is not being wasted. So far, so good!

Now, 5s have elapsed, the http call is done, our worker thread comes back to resume from where it left off. The next step is to write the response into a file. This step takes around 2s. However, the call is not suspending! (Note the absence of the suspend keyword in the saveResponse function’s signature). Consequently, the worker thread has no other choice than to sit idle waiting for the call to finish. It cannot jump out to make progress on any other coroutine. In other words, our worker thread is blocked!

In a multithreaded runtime like Kotlin’s, the default coroutine dispatcher usually has multiple worker threads, so the issue is not as bad as if it were to happen in a single-threaded runtime like Javascript’s or Python’s. But still, if you block all your worker threads this way, no progress is made on any of the other coroutines, and that’s when you start noticing slowness or unresponsiveness in your app!

What should I do if I need to block inside my coroutines?

In the previous section, we have seen that blocking calls inside coroutines should be avoided. However, in reality, this is not something we can do all the time! Sometimes you might need to use a library function that synchronously performs blocking tasks, and you don’t find an asynchronous alternative (with futures or callbacks). There are, unfortunately, tons of examples in the Java ecosystem. A few that I encountered myself include the Kubernetes java client or the Kafka consumer client.

So how can we perform those blocking calls within our coroutines based application while keeping it performant and responsive?

The general idea is to run them in a separate worker pool or dedicated threads to keep your main worker pool¹, where most of your coroutines are running, free from blocking code. There are multiple ways how you can achieve this in practice.

Using Dispatchers.IO

The first solution is the simplest and the most widely used.

The idea is to isolate your blocking code by running it within the Dispatchers.IO, a global shared pool of worker threads whose primary purpose is to run blocking tasks. This worker pool contains 64 threads by default, making it less sensitive to blocking tasks. You would need to make all of the 64 threads simultaneously blocked for it to start causing slowness or unresponsiveness to your app!

Using this dispatcher can be done as the following:

launch {
    val response = httpCient.request()
    withContext(Dispatchers.IO) {
        saveResponse(response, filename="/tmp/response.txt")
    }
}

It is easy to feel safe using this dispatcher widely across your application whenever you have a blocking call to make. However, you need to be careful: this pool has a limited number of workers. And as it is global, shared across your whole application, and might also be used by your library functions without you knowing, it’s much easier to exhaust than you might think! It happened to me more than once that I ran into performance issues in my coroutines and then realized that I was saturating the Dispatchers.IO threads!

That’s why I’m usually reluctant to using this dispatcher unless I’m writing a small application or when the predictability of performance is not critical.

The good news is: these issues are entirely mitigated by using Dispatchers.IO views, a feature available since version 1.6.0 of kotlinx-coroutines-core, making this dispatcher much more useful and safer to use!

Creating views from the `Dispatchers.IO`

Since version 1.6.0 of kotlinx-coroutines-core, it is possible to create “views” from coroutine dipsatchers. This feature, especially when used on Dispatchers.IO, is a very effective way to isolate blocking calls.

Let’s check an example of the usage before explaining the underlying concepts:

val dispatcher = Dispatchers.IO.limitedParallelism(100)

launch {
    val response = httpCient.request()
    withContext(dispatcher) {
        saveResponse(response, filename="/tmp/response.txt")
    }
}

The limitedParallelism function creates a “view” of the current dispatcher that limits the parallelism to the given value (100 in the example above). The resulting view uses the original dispatcher for execution but guarantees that no more than the specified number of coroutines are executed simultaneously.

Generally, you cannot create a view with a parallelism level that is superior to the original dispatcher’s parallelism level. However, the Dispatchers.IO has a unique property of elasticity when creating views, allowing you to create as many views as you want with the desired level of parallelism. You can read more about this important behavior here. That’s the reason why, in the example above, we were able to create a view with a parallelism level of 100 even though the Dispatchers.IO is capped to 64 threads! That was not a mistake 😉.

This is my default approach for handling blocking calls in most situations unless I am using an older version of Coroutines.

Using a dedicated thread pool as a Coroutine Dispatcher

You can also create a dedicated thread pool to offload your blocking calls. This approach is a good alternative if you don’t want to rely on the shared Dispatchers.IO (before dispatcher views became available).

val dispatcher = newFixedThreadPoolContext(10)

// or
val dispatcher = Executors.newCachedThreadPool().asCoroutineDispatcher()

launch(dispatcher) {
    ...
}

// It becomes your responsibility to close the resulting dispatcher!
dispatcher.close()

The resulting dispatcher becomes your responsibility to manage. In particular, you need to remember to close it when you are done with it.

Spinning up a dedicated thread to run a long-running blocking task

I often use this approach when I have a long-running blocking task that produces some data that I want to pass to my coroutines through a Channel or a Flow. A nice example of this is using the Kafka consumer client and making it poll records from Kafka as a Flow:

fun <K, V> KafkaConsumer<K, V>.pollAsFlow(topics: List<String>): Flow<ConsumerRecord<K, V>> =
    callbackFlow {
        thread(name = "kafka-consumer-flow-thread") {
            try {
                subscribe(topics)
                // `isActive` here refers to the producer scope of the `callbackFlow`
                while (isActive) {
                    // Consume new records from Kafka (blocking)
                    val records = poll(Duration.ofSeconds((3)))
                    for (record in records) {
                        // Send the record downstream using the flow's `trySendBlocking` method
                        trySendBlocking(record).getOrThrow()
                    }
                    commitSync() // Blocking
                }
            } catch (exception: Exception) {
                close(exception)
            } finally {
                close()
            }
        }

        awaitClose {}
    }

// usage
launch {
    consumer
        .pollAsFlow(listOf("my_topic"))
        .collect { println(it) }
}

If you are not familiar with flows, I highly recommend checking them out! They will very likely level up your asynchronous programming skills!

Using Blockhound to detect blocking calls inside coroutines

Enough with theory! Let’s get to the fun part!

In practice, one efficient way to detect blocking calls inside your coroutines is by using BlockHound.

BlockHound is a Java Agent built by the Reactor team that detects blocking calls from non-blocking threads. When it does, BlockHound throws an exception with a helpful error message pointing us to the exact place where the blocking call occurred. It is suitable for local debugging during development, unit, and integration tests.

But first, we need to instruct BlockHound what threads we consider non-blocking (i.e., in which we want to forbid blocking calls). This is done through integrations. BlockHound already provides built-in integrations with Reactor & RxJava, allowing it to accurately detect illegitimate blocking calls in both frameworks’ event loops.

Fortunately for us, the Kotlin team also built a BlockHound integration for Kotlin Coroutines. It is packaged & distributed via the kotlinx-coroutines-debug project.

This integration configures BlockHound to intercept blocking calls in Coroutines launched in the Dispatchers.Default worker pool. This is an important thing to keep in mind when using this integration: Blocking calls in coroutines spun up in other dispatchers (including those you build yourself) will not be intercepted. If this part is not clear, no worries, it will become more apparent after the demo below.

To demonstrate the usage of BlockHound, let’s consider a short Kotlin code with a suspend function containing an ugly blocking call:

/**
* A suspend function with an awful blocking sleep!
*/
suspend fun httpCall() {
    Thread.sleep(200)
    delay(5000)
}

fun main(): Unit = runBlocking {
    httpCall()
}

Obviously, in reality, the evil blocking calls inside your coroutines will rarely be this visible! Usually, you will have to find them hidden deep inside your functions in unexpected places (such as middlewares, callbacks, etc.).

To use Blockhound, we need to inject its maven coordinates in our build system and kotlinx-coroutines-debug as it provides the coroutines Blockhound integration. The example below uses Gradle:

dependencies {
    implementation(platform("org.jetbrains.kotlin:kotlin-bom"))

    implementation("org.jetbrains.kotlin:kotlin-stdlib-jdk8")
    implementation("org.jetbrains.kotlinx:kotlinx-coroutines-core:1.5.2")

    implementation("org.jetbrains.kotlinx:kotlinx-coroutines-debug:1.5.2")
    implementation("io.projectreactor.tools:blockhound:1.0.6.RELEASE")
}

To prepare BlockHound to monitor our threads, we should call the BlockHound.install() method and supply it with a CoroutinesBlockHoundIntegration instance:

import kotlinx.coroutines.debug.CoroutinesBlockHoundIntegration
import kotlinx.coroutines.delay
import kotlinx.coroutines.runBlocking
import reactor.blockhound.BlockHound

/**
* A suspend function with an awful blocking sleep!
*/

suspend fun httpCall() {
    Thread.sleep(200)
    delay(5000)
}

fun main(): Unit = runBlocking {
    BlockHound.install(CoroutinesBlockHoundIntegration())
    httpCall()
}

Now, if you run this program, surprisingly, nothing happens! BlockHound did not intercept our blocking call. As I mentioned above, this is due to how this integration is built: it only intercepts blocking calls when running in the context of Dispatchers.Default (or Dispatchers.Main). So let’s modify our program to use that context instead:

import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.debug.CoroutinesBlockHoundIntegration
import kotlinx.coroutines.delay
import kotlinx.coroutines.runBlocking
import reactor.blockhound.BlockHound

/**
* A suspend function with an awful blocking sleep!
*/

suspend fun httpCall() {
    Thread.sleep(200)
    delay(5000)
}

fun main(): Unit = runBlocking(Dispatchers.Default) {
    BlockHound.install(CoroutinesBlockHoundIntegration())
    httpCall()
}

Now, if we rerun our program, we will get a nice error message pointing it to the exact place where the blocking call is happening:

Exception in thread "main" reactor.blockhound.BlockingOperationError: Blocking call! java.lang.Thread.sleep
        at java.base/java.lang.Thread.sleep(Thread.java)
        at coroutines.debug.MainKt.httpCall(Main.kt:13)
        at coroutines.debug.MainKt$main$1.invokeSuspend(Main.kt:19)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)

Conclusion

Accidental blocking calls are one of the main causes of performance issues in Coroutines-based applications. They can come from your library functions, middlewares, or other unexpected places. Using BlockHound in your automated tests should allow you to detect them preemptively before they reach your production environment. Once you identify your guilty blocking calls, you can use the strategies we have seen earlier to deal with them.

Last but not least, I am currently creating an online course dedicated to learning Kotlin Coroutines. If you are interested, make sure to check it out and subscribe for updates to be notified of its release 😉.

Usually the Dispatchers.Default↩

The Dev Tavern RSS Feed

5 Things that will change the way you use AWS Lambda

Introduction

Lambda timeouts cause cold starts!

Custom metrics from Lambdas via logs using EMF

Optimize your Lambdas latency by running your init code once

Background code will continue to run on the next invocations

Code size matters

Structured concurrency explained - Part 2: Exceptions and Cancellations

Introduction

Exception handling inside a coroutine scope

The confusing part

Using supervisorScope when child failures are not fatal

Cancellation and blocking code

Conclusion

Structured concurrency explained - Part 1: Introduction

Introduction

Unstructured concurrency

Structured concurrency with Kotlin Coroutines

The CoroutineScope and the Jobs hierarchy

Difference between CoroutineScope and CoroutineContext

Conclusion

Why are my coroutines slow?

Introduction

The symptoms: what causes coroutines to become slow?

How is a coroutine blocked?

What should I do if I need to block inside my coroutines?

Using Dispatchers.IO

Creating views from the Dispatchers.IO

Using a dedicated thread pool as a Coroutine Dispatcher

Spinning up a dedicated thread to run a long-running blocking task

Using Blockhound to detect blocking calls inside coroutines

Conclusion

Using `supervisorScope` when child failures are not fatal

Creating views from the `Dispatchers.IO`