Motivation:
Running the script failed as it looked for a CI build for Swift 5.5,
which is not run by our CI anymore.
Modifications:
* don't look for CI builds for Swift 5.5
* additionally, also check for CI build for Swift 5.9
Motivation:
Channels can read `ChannelOptions.maxMessagesPerRead` times from a
socket in each read cycle. They typically re-use the same buffer for
each read and rely on it CoWing if necessary. If we read more than once
in a cycle then we may CoW the buffer. Instead of reusing one buffer we
can reuse a pool of buffers limited by `maxMessagesPerRead` and cycle
through each, reducing the chance of CoWing the buffers.
Modifications:
- Extend `RecvByteBufferAllocator` to provide the size of the next
buffer with a default implementation returning `nil`.
- Add an recv buffer pool which lazily grows up to a fixed size and
attempts to reuse buffers where possible if doing so avoids CoWing.
Results:
Fewer allocations
Motivation:
Cocoapods appears to have a different idea of the dependency graph to
SPM which has led to a handful of build failures (e.g.
https://github.com/apple/swift-nio/issues/2073).
This appears to have originated when we dropped the explicit dependency
on `CNIOAtomics` from `NIO` (https://github.com/apple/swift-nio/pull/1719).
We can work around this by listing all transitive dependencies as
requirements in the generated podspecs.
Modifications:
- Add a script to list transitive dependencies for a given module.
- Fix a bug in build_podspecs.sh which did a `pod repo update` even if
the pods were not being uploaded.
- Update build_podspecs.sh to use transitive dependencies.
- Use Python3 in 'dev/stackdiff-dtrace.py' and update soundness.sh
Result:
Podspecs should include all transitive dependencies.
We regressed allocs in nightly, probably for a Swift bug. Fixing it is
tracked in #2070, for now we're going to raise the limits so CI isn't
moaning all the time.
Motivation:
Many network protocols (especially for example NFS) have quite a number
of integer values next to each other. In NIO, you'd normally parse/write
them with multiple read/writeInteger calls.
Unfortunately, that's a bit wasteful because we're checking the bounds
as well as the CoW state every time.
Modifications:
- Provide read/writeMultipleIntegers for up to 15 FixedWidthIntegers.
- Benchmarks
Result:
Faster code. For 10 UInt32s, this is a 5x performance win on my machine,
see benchmarks.
Make Swift 5.2 the minimum requirement, dropping support for Swift 5.0 and 5.1.
Motivation:
Whenever we have problems, Swift 5.0 and 5.1 seem to be the culprits. Dropping support for these very old versions will require less maintenance and free up our time to work on new features.
Modifications:
Set the tools version in Package.swift to 5.2
Remove CI configurations for 5.0 and 5.1
Update the various readmes to reflect that this change will be rolled out in NIO 2.30.0
Result:
Swift 5.2 is the minimum version of Swift required to use NIO.
Motivation:
We need license headers but the soundness script couldn't check if the
scripts don't have a `.sh` extension.
Modification:
- Add missing license headers.
- Better license checking (search for shell shebangs too)
Result:
Proper license attribution.
Motivation:
The allocation limit updater script is handy but so far only worked for
the SwiftNIO core repo.
Modifications:
- allow the user to pass other URL prefixes and repos
Result:
Can be used for other repos
Motivation:
We often hit a bunch of problems with LLDB, especially on Linux. So
maybe it would be cool if we could smoke test LLDB using SwiftNIO.
Modifications:
Add script `dev/lldb-smoker` which does the following things:
- takes 2000 random picks amongst all the files in the package
- for each of those files, pick a random line number
- write an LLDB script which sets break points for each of the file/line
pairs (one-shot breakpoints)
- runs the program
- for each breakpoint hit, run `frame variable` to list all the
variables
Result:
LLDB smoke test
Motivation:
5.0 is several years old now, and while we still support it there is
beginning to be some conflict with performance optimizations for newer
Swift versions. We should stop counting the allocation counts on this
version now: we no longer care as much about regressions.
Modifications:
- Stop setting alloc counter limits on 5.0
- Remove the script that tries to add them back.
Result:
No longer measure allocs on 5.0
Motivation:
When doing bigger alloc count changes, it's really tedious to apply them
manually given that the CI has the right numbers already.
Modifications:
Add a script to download and update them automatically.
Result:
Less toil.
Motivation:
The dev/alloc-limits-from-test-output script is handy to get correctly
formatted (and sorted) alloc counter limits from test output.
Modifications:
When running the alloc counter script itself, it outputs suggested
`export MAX_ALLOCS_ALLOWED` lines. We're now also parsing those, so you
can easily go from `export ...*` to the docker-compose form :).
Result:
Even easier alloc counter limits.
Motivation:
Usually, we add a more or less random number of slack allocations to
make sure the tests don't spuriously fail. This makes it quite costly to
support new Swift versions.
Modifications:
Add a script which can spit you out the right allocation limits
including slack.
Result:
Easier to support new Swift versions
Motivation:
Some of the output from 'malloc-aggregation.d' trips up the
'stackdiff-dtrace.py' script -- especially when used with the output
from recent 5.4 builds. The regex for numbers includes hex which trips
us up when we try to convert that match to an int. Since we only use
this pattern for matching allocation counts, where we don't expect hex,
we can just modify the pattern.
Also the input parsing is a little tricky to follow without context so I
tidied it up little.
Modifications:
- relax the number regex (we don't expect counts to be in hex)
- drop the stack regex (it wasn't used)
- aggregate the current stack in a list (instead of a string)
- filter out non-stack lines (which were erroneously used to key the
first stack!)
- a few comments
Result:
stackdiff-dtrace.py doesn't blow up on output from newer builds
Motivation:
The Swift runtime is now using malloc_zone_*, we need to implement
replacements for these too. This is just a first pass, eventually, we
should implement _all_ replacements as `malloc_zone_memalign` which is
powerful enough to implement all others.
Modifications:
Provide new replacements.
Result:
Alloc tests work again on macOS.
Co-authored-by: Cory Benfield <lukasa@apple.com>
Allocation diffing script no longer drops data.
Motivation:
Looking at allocations is hard enough without the script getting the allocation counts wrong.
Modifications:
Where multiple allocations resolved to the same "key" only the last allocation was accounted for. I have changed the script to store all allocations - still keyed in the same way. When diffing the total number of allocations is used - if they are different the total and all contributing stack traces are output.
Added a total for all allocations everywhere at the end together with a difference number.
Output all stack traces before and after contributing to a detected diff.
Update the documentation to reflect changes.
Change the threshold for reporting diffs from > 1000 to >= 1000. This means if allocations go from 1000 to 2000 they are reported as a difference rather than a new allocation.
Result:
It is now easier - if somewhat more verbose to compare allocations.
Motivation:
The make-single-file-spm script used to use a hidden folder in /tmp
which isn't very helpful when tar'ing up the result for a bug report.
Modifications:
Switch to a non-hidden folder.
Result:
Easier bugs.swift.org reports.
Motivation:
We recently grew a script to diff the output of `malloc-aggregation.d`,
we should document how this can be used when debugging allocation
regressions.
Modifications:
- Update the doc to include an example of using the script to debug an
allocation regression
- Fix a bug in the script where before/after were mixed up
- Fix a bug in the script where usage would print '()\n' instead of '\n'
Result:
- Better info on debugging allocations
Co-authored-by: Johannes Weiss <johannesweiss@apple.com>
Motivation:
NIO has allocation counter tests, NIO also has dev/malloc-aggregation.d
which can give you aggregated stacks that allocated alongside how many
times they allocated. What's missing is a script that can diff two
aggregated stack traces.
Modifications:
Add dev/stacktrace-dtrace which allows aggregated stack trace diffing.
Result:
Easier to track performance regressions.
Motivation:
make-single-file-spm has a weird bug where it would chop off trailing
`n` characters in all lines due to a badly set `IFS` bash variable.
Modifications:
Set `IFS=""` which is correct.
Result:
make-single-file-spm will not chop off trailing `n`s.
Motivation:
make-single-file-spm is a simple bash script that can turn a single file
into a SwiftPM project with multiple modules (all defined in a single
file with special markers such as `// MODULE: Foo`. This is mostly
useful for performance evaluation of Swift becuase in almost all cases,
you'll need multiple modules which is quite a bit of work to set up.
Modifications:
Add make-single-file-spm
Result:
More useful scripts in the NIO repo.
Motivation:
When counting the number of allocations it's important not to forget an
allocation function that Swift uses.
Modifications:
add posix_memalign to dev/malloc-aggregation.d
Result:
malloc aggregations accurate with recent Swift versions
Motivation:
The dtrace scripts dev/malloc-aggregation.d was missing a bunch of more
rarely used memory allocation functions.
Modifications:
Added the more rarely used functions.
Result:
Better output when using dev/malloc-aggregation.d
Motivation:
dtrace is extrmely useful when debugging allocation related issues. This
adds two scripts that have helped me many times.
Modifications:
- `dev/malloc-aggregation.d` which prints an aggregation of all
stacks that have allocated
- `dev/boxed-existentials.d` which live prints all allocations of boxed
existentials
Result:
sharing debugging tools is great
Felix Schlegel