swift-nio/IntegrationTests/tests_04_performance/test_01_resources

Allocation Counting Test

This briefly describes how the allocation counting test works.

How does it work?

This is possibly the simplest implementation that counts memory allocations (malloc and friends) and frees (mostly free). It just maintains two atomic variables which count the number of mallocs and the number of frees respectively. We run a simple HTTP1 example -- 1000 requests and responses generated by a simple SwiftNIO based client and server -- and then evaluate the number of mallocs and frees. The difference mallocs - frees should be pretty much 0 and the number of mallocs should remain stable (or decrease) across commits. We can't establish a perfect baseline as the exact number of allocations depends on your operating system, libc and Swift version.

How are the functions hooked?

Usually in UNIX it's enough to just define a function, for example

void free(void *ptr) { ... }

in the main binary and all modules will use this free function instead of the real one from the libc. For Linux, this is exactly what we're doing, the bootstrap binary defines such a free function in its main.c. On Darwin (macOS/iOS/...) however that is not the case and you need to use dyld's interpose feature. The odd thing is that dyld's interposing only works if it's in a .dylib and not from a binary's main executable. Therefore we need to build a slightly strange SwiftPM package:

• bootstrap: The main executable's main module (written in C) so we can hook the free function on Linux.
• BootstrapSwift: A SwiftPM module (written in Swift) called in from bootstrap which implements the actual SwiftNIO benchmark (and therefore depends on the NIO module).
• HookedFunctions: A separate SwiftPM package that builds a shared library (.so on Linux, .dylib on Darwin) which contains the replacement_malloc, replacement_free, etc functions which just increment an atomic integers representing the number of operations. On Darwin, we use DYLD_INTERPOSE in this module, interposing libc functions with our replacement_ functions. This needs to be a separate SwiftPM package as otherwise its code would just live inside of the bootstrap executable and the dyld interposing feature wouldn't work.
• AtomicCounter: SwiftPM package (written in C) that implements the atomic counters. It needs to be a separate package as both BoostrapSwift (to read the allocation counter) as well as HookedFree (to increment the allocation counter) depend on it.

What benchmark is run?

We run a single TCP connection over which 1000 HTTP requests are made by a client written in NIO, responded to by a server also written in NIO. We re-run the benchmark 10 times and return the lowest number of allocations that has been made.

Why do I have to set a baseline?

By default this test should always succeed as it doesn't actually compare the number of allocations to a certain number. The reason is that this number varies ever so slightly between operating systems and Swift versions. At the time of writing on macOS we got roughly 326k allocations and on Linux 322k allocations for 1000 HTTP requests & responses. To set a baseline simply run

export MAX_ALLOCS_ALLOWED_1000_reqs_1_conn=327000

or similar to set the maximum number of allocations allowed. If the benchmark exceeds these allocations the test will fail.