Motivation:
Swift 5.5 provides concurrency support on almost all platforms where it
is available. However, the Xcode 13 GM currently provides a macOS 11 SDK
with Swift 5.5, and the concurrency features are not available in that
SDK. As a result, NIO's concurrency features cause compile errors when
building the concurrency library on macOS using Xcode 13 GM.
Modifications:
- Only build the concurrency features when the concurrency library is
present.
Result:
We can build NIO using Xcode 13 GM.
Motivation:
In February, @weissi noticed that the nightly builds of Swift could no
longer compile the unit tests. We disabled them at that time and
reported an upstream bug, https://bugs.swift.org/browse/SR-14268. This
is not an ideal situation, but the Swift core team has provided a
workaround we can use to re-enable the tests.
Modifications:
Provide free functions that wrap some specific extension methods that
are called in unit tests, and amend the call sites to use them.
Result:
We can re-enable the nightly tests.
Same issue as fixed#1733 for ByteToMessageHandler.
### Motivation:
`NIOSingleStepByteToMessageProcessor` called out to the decoder's shouldReclaimBytes method after every parsing attempt, even if the decoder returned a value (which means continue in the `decodeLoop`).
That's quite pointless because we won't add any bytes into the buffer before we're trying the decoder again. Further it is a huge performance penalty for a use-cases in which we only consume small frames from the buffer. (Like reading data rows in a database client)
### Modifications:
- Only ask the decoder if we should reclaim bytes if the decoder actually is not able to process more frames from the input buffer.
Motivation:
Xcode 13 GM shipped with a Swift overlay for libsystem in macOS that
marked free's first argument as non-nullable. This leads to an awkward
breakage for us, because we're trying to hold a reference to free as a
function pointer, and to do that we had an explicit type annotation.
We'd like to keep NIO compiling in Xcode 13 GM.
Modifications:
- Defined the free function as a thunk that passes through to the
underlying OS free call, but takes its first argument as non-nullable.
Result:
Should compile on the Xcode 13 GM again.
Motivation:
Follow up to https://github.com/apple/swift-nio/pull/1952#discussion_r707351015
The OWS rule from the HTTP semantics draft only considers 'SP' and
'HTAB' to be whitespace. We also (unnecessarily) consider CR and LF to
be whitespace.
Modifications:
- Remove CR and LF from the characters we consider to be whitespace
- Update tests
Result:
We no longer consider CR or LF to be whitespace when trimming
whitespace when producing the canonical form of header values.
Motivation:
When getting the canonical form of header values any ascii whitespace
is stripped from the values. The current implementation does this by
doing equality checks on `Character` which is quite expensive.
Modifications:
- Update the `Substring.trimWhitespace()` function to trim on a UTF8
view
Result:
Retrieving the canonical form of header values is cheaper, significantly
so when values contain whitespace.
Motivation:
To justify performance changes we need to measure the code being
changed. We believe that `HTTPHeaders.subscript(canonicalForm:)` is a
little slow.
Modifications:
- Add allocation and performance tests for fetching header values in
their canonical form
Results:
More benchmarks!
Motivation:
It's currently unclear that SwiftNIO1 is end of life.
Modifications:
Update the README to set a deadline on patches to NIO1
Result:
Everyone knows where the core team stands on supporting NIO1
Co-authored-by: George Barnett <gbarnett@apple.com>
Motivation:
In #1888 a C compiler setting was added to Package.swift for CNIODarwin.
This isn't reflected in the associated CNIODarwin podspec so the pod
fails to build.
Modifications:
Modify the build_podspecs.sh script to add the missing compiler setting
for CNIODarwin.
Result:
CNIODarwin pod builds.
Motivation:
Our kqueue filter set differ logic attempted to stack allocate some
kevent objects and then treat them as a fixed size array by creating a
tuple and then grabbing a pointer to it. That's fairly gratuitously
unsafe, and we should shrink the amount of code that can touch unsafe
stuff.
Modifications:
- Defined KeventTriple, a value type that encapsulates a fixed-size
array and a length.
- Rewrote calculateKQueueFilterSetChanges to use the new type.
Result:
Safer code in the NIO core
Co-authored-by: George Barnett <gbarnett@apple.com>
* Fix doc generation and jazzy version
From: https://github.com/apple/swift-nio-extras/pull/141
motivation: fix ci
changes:
* only install ruby and jazzy on focal since jazzy id not supported onlder versions of ubuntu
* fix doc generations script adjusting it to latest source-kittent syntax
* Always install ruby: we need it for generating a test manifest
- Use correct @available checks in `NIOAsyncAwaitDemo` (post WWDC '21)
- Remove `import _Concurrency` statements
Result:
- Cleaner imports
- static-stdlib linking works again on 5.5 (it does not if we explicitly `import _Concurrency`)
Co-authored-by: George Barnett <gbarnett@apple.com>
Motivation:
We should explain when we want users to use specific products, so that
they can understand what to import and when.
Modifications:
- Added some text to the README to clarify what modules exist.
Result:
Should be easier to understand what there is in the NIO ecosystem
Co-authored-by: Fabian Fett <fabianfett@apple.com>
Motivation:
The remaining NIO code really conceptually belongs in a module called
NIOPosix, and NIOCore should really be called NIO. We can't really do
that last step, but we can prepare by pushing the bulk of the remaining
code into a module called NIOPosix.
Modifications:
- Move NIO to NIOPosix
- Make NIO an umbrella module.
Result:
NIOPosix exists.
Motivation:
In #1935 I removed NIO as a dependency of a number of our library
targets. This unfortunately led to some downstream breakage in modules
that were (incorrectly) assuming they could import NIO without
expressing a dependency on it in their Package.swift, e.g.
swift-server/swift-aws-lambda-runtime#218.
While a forums thread
(https://forums.swift.org/t/semantic-versioning-should-removing-a-dependency-be-a-semver-major/51179)
is ongoing to discuss the implications of this, we should re-add the
dependency to undo the breakage against main. I've validated this
locally: merely having the dependency is enough, we don't have to use
it.
Modifications:
- Re-add NIO as a dependency to:
- _NIOConcurrency
- NIOFoundationCompat
- NIOHTTP1
- NIOTLS
- NIOWebSocket
Result:
Broken downstreams can build again.
Motivation:
As we've largely completed our move to split out our core abstractions,
we now have an opportunity to clean up our dependencies and imports. We
should arrange for everything to only import NIO if it actually needs
it, and to correctly express dependencies on NIOCore and NIOEmbedded
where they exist.
We aren't yet splitting out tests that only test functionality in
NIOCore, that will follow in a separate patch.
Modifications:
- Fixed up imports
- Made sure our protocols only require NIOCore.
Result:
Better expression of dependencies.
Co-authored-by: George Barnett <gbarnett@apple.com>
Motivation:
EmbeddedChannel is an important testing tool, and we want to use it
without needing to bring along the POSIX layer. They are not tightly
coupled. However, it also doesn't belong naturally in NIOCore, so we
should probably put it in its own place.
Modifications:
- Moved EmbeddedChannel and EmbeddedEventLoop to NIOEmbedded.
- Moved the tests to NIOEmbeddedTests
- Duplicated some test helpers
Result:
Easy to use EmbeddedChannel without the POSIX layer.
Motivation:
I'd like to move EmbeddedChannel and friends out of the main NIO
repository and into their own module. Unfortunately, EmbeddedChannel
shares the PriorityQueue implementation we wrote with the various POSIX
channels. To avoid duplicating the code, we should pull it out to its
own module.
However, we've never wanted to commit to the API of this data structure,
and the same is true now. To that end, I'm pulling it into an
underscored module that is not a product of the package.
We could have used the `@_spi` annotation here but honestly I'm a bit
nervous about doing that at the low-level of NIO itself, as if the Swift
team does change the spelling of it at any point in the future we'll be
in real trouble. This way works almost as well, and makes our intent a
lot clearer.
Modifications:
- Extracted Heap and PriorityQueue to a new module.
- Made everything @inlinable to do our best to make performance
acceptable.
Result:
We can re-use PriorityQueue.
Motivation:
MultcastChannel is a general abstraction for expressing multicast
capabilities on a given Channel. This abstraction doesn't have any
particularly tight tie to the POSIX layer, so it belongs in NIOCore.
This is also expressed in terms of NIONetworkDevice, so we need to move
that over. That also encourages us to bring over
System.enumerateDevices, and given that System.coreCount is also fairly
general-purpose we may as well bring it along too.
Modifications:
- Move MulticastChannel to NIOCore
- Move NIONetworkDevice to NIOCore
- Move System to NIOCore
Result:
More general-purpose abstractions in NIOCore.
Motivation:
SocketOptionProvider is a general interface for setting socket options
that are "wider" than a single platform Int. There is nothing inherent
in NIO to require the definition of this protocol to be there, so we
can move it to NIOCore.
Modifications:
- Move the definition of SocketOptionProvider to NIOCore.
Result:
SocketOptionProvider is generally available.
Motivation:
NIO supplies a number of first-party ChannelHandlers that cover specific
use-cases. These aren't tied to the POSIX IO layer and so they should
come out.
Modifications:
- Move ChannelHandlers.swift to NIOCore
- Move NIOCloseOnErrorHandler.swift to NIOCore
Result:
Helper handlers in NIOCore
Motivation:
Our basic Channel Handlers don't need to be in NIO anymore: they aren't
realistically tied to that code. So we can move them to NIOCore.
Modifications:
- Move Codecs to NIOCore
- Move SingleStepByteToMessageDecoder to NIOCore.
Result:
Even more stuff in NIOCore
Motivation:
MCB is a companion part of circular buffer, and should also be a core
component of NIO.
Modifications:
Move MarkedCircularBuffer to NIOCore
Result:
MCB is in the common code.
Motivation:
Unbreak Android build from NIOCore not having these constants defined.
Modifications:
Copy the same constants that Channel was using in NIO over.
Result:
Android compiles again and the same tests pass.
Return `Task` from `EventLoopPromise.completeWithAsync(_:)` wrapper.
Motivation:
Bridging an `async` function into NIO is done by calling `completeWithAsync(_ body:)` on an `EventLoopPromise`. This spins up a `Task` and `await`s the result of the `async` closure that was passed in and uses the result to fulfil the promise with the value of the function, if it was successful, or with the error, if the function threw.
In some cases we may want to cancel this operation from the outside.
Modifications:
This patch adds a discardable return value to `EventLoopPromise.completeWithAsync(_:)` which is the `Task` it creates.
Result:
Users of `EventLoopPromise.completeWithAsync(_:)` are now able to explicitly cancel the `Task` that is being used to fulfil the promise.
Motivation:
The UniversalBootstrap abstraction allows you to write code that is
agnostic to the specific event loop and channel implementation. This is
likely to be used in places that don't want to have to express a
concrete requirement for a specific channel or loop, and so properly
belongs in NIOCore.
Modifications:
- Move Universal Bootstrap
- Move TCP convenience options, which are required by the universal
bootstrap.
Result:
Universal Bootstrap will be in NIOCore.
Motivation:
The most important API surface area in NIO are the Channel abstractions.
These are shared in all NIO programs, and are also used by several
projects to implement their I/O abstraction. There are several moving
parts to this abstraction, all of which are moving:
- Channel itself
- ChannelPipeline
- ChannelHandler
As these all move, they force several other pieces of API to move with
them. Most notably they force us to move NIOAny, which also forces us to
move FileHandle and FileRegion. That also forces us to bring over part
of our syscall abstraction. This duplication is acceptable due to its
minimal surface area, but it is definitely a flaw in our abstraction
design that we had to do that at all.
We also need to move the channel option abstraction, AddressedEnvelope,
and the DeadChannel.
Modifications:
- Moved a bunch of the Channel abstraction over.
- Moved Channel-associated types.
Result:
Channel will be part of NIOCore.
Motivation:
For Channel implementations, we implemented ChannelCore.unwrapData to
allow them to unwrap a NIOAny into their expected data type. This
remains our recommended flow, as Channels are supposed to know what data
type they can send. However, in a few cases the Channel has a runtime
type: in particular, EmbeddedChannel is capable of tolerating nearly
anything.
As we're coming to split the code apart, we'll likely want
EmbeddedChannel to live in a different module than NIOCore. To that end,
we need to add new API to ChannelCore to allow the specific
EmbeddedChannel use-case.
Modifications:
- Implement ChannelCore.tryUnwrapData.
- Use this within EmbeddedChannel.
Result:
EmbeddedChannel can attempt to unwrap things.
Motivation:
When pulling out the NIOBSDSocket later in #1907 we dropped some of the
socket option fixes for 32 bit platforms, which broke them.
Modifications:
Restore the socket options.
Result:
32-bit build should be working again.
Motivation:
Issue for #1891 to add JSONSerialization in NIOFoundationCompat.
Modifications:
Adds:
JSONSerialization+ByteBuffer.swift
JSONSerialization+ByteBufferTest.swift
JSONSerialization+ByteBufferTest+XCTest.swift
Result:
Support for transforming byteBuffer data to a Foundation object.
Co-authored-by: Cory Benfield <lukasa@apple.com>
Motivation:
A while back we added a ChannelCore extension method that would allow
removing handlers from a channel pipeline, in order to implement
Channels outside of NIO. This extension worked well, but it did assume
that you had a Channel to give it. There's no reason it needed a Channel
instead of just a ChannelPipeline, but it was based on the assumption
that Self would conform to Channel. Of course, this is usually true, but
it turns out not to always be true, and EmbeddedChannel is one example
where the Channel and ChannelCore are separate.
To that end, we shouldn't require using the Channel here, just the
ChannelPipeline. This will also let us force our NIO channels onto the
public API, which will make migrating to NIOCore easier.
Modifications:
- Deprecate ChannelCore.removeHandlers(channel:)
- Implement ChannelCore.removeHandlers(pipeline:)
- Swap existing Channel implementations to use the new method.
Result:
More consistent usage of our APIs.
Co-authored-by: George Barnett <gbarnett@apple.com>
Motivation:
The soundness check is failing because of the generated year range is
not as it should be. Moreover when re-generating the linux tests for the
length prefix tests (which are failing soundness) the generated header
has extra whitespace before the year; this also fails the soundness
check.
Modifications:
- Strip whitespace from the copyright year range generated for linux
tests
- Re-generate the test manifest for ByteBufferLengthPrefixTests
Result:
- Soundness is happier
A convenience method to write a message and prepend its length in front of it.
### Motivation:
Many protocols use a length field to split a byte stream into multiple separate messages. `swift-nio-extras` supports these kind of protocols through `LengthFieldPrepender` and friends. However, sometimes a channel handler is not the right tool, especially if the a message itself contains multiple dynamic length fields. One example is [RSocket Routing Metadata Extensions](https://github.com/rsocket/rsocket/blob/master/Extensions/Routing.md).
A `ByteBuffer` is probably a better tool to create this message. This can be tricky to get right if strings are part of the message. A naive solution like the following has two issues:
```swift
func add(tag: String, to buffer: inout ByteBuffer) {
let length = tag.count
buffer.writeInteger(UInt8(length))
buffer.writeString(tag)
}
```
The `count` property in Swift is different than in most other languages. It does **not** return the number of UTF-8 code points and can result in incorrect behavior if multi-byte characters are used (e.g. Emojis). Furthermore, the size of the string could be larger than `UInt8` can represent.
### Modifications:
This PR introduces a new method on `ByteBuffer` called `writeLengthPrefix(endianness:as:writeMessage:)`. It solves both problems mentioned above. It first reserves space for the length prefix and delegates the actually writing of the message to `writeMessage`. Afterwards, it counts how many bytes were written during the call to `writeMessage` and then sets it on the previously reserved space in front of the actually message.
### Result:
We can now write arbitrary content to a buffer and prefix it with the number of bytes written.
```swift
func add(tag: String, to buffer: inout ByteBuffer) {
buffer.writeLengthPrefix(as: UInt8.self) { buffer in
buffer.writeString(tag)
}
}
```
Motivation:
This is the next stage of our move of common interchange objects to
NIOCore and out of the NIO module. This time around we need
SocketAddress, which is part of the Channel API. Sadly, SocketAddress is
not as clean as some of the other ports, because it leaks a number of
POSIX-y concepts. This also forces us to bring along NIOBSDSocket, which
ideally we would not move, but we foolishly exposed as API on
SocketAddress.
Modifications:
- Move NIOBSDSocket API components into NIOCore.
- Split out the internal abstractions for NIOBSDSocket and leave those
in NIO.
- Move SocketAddress into NIOCore.
Result:
SocketAddress will be sitting in NIOCore.
Motivation:
NIO provides some scope for performing "synchronous" ChannelPipeline
operations. These are only safe to perform from the event loop to which
the pipeline is bound, and they avoid the need to check what thread
context you're running on in release mode.
Several of the built-in channels currently skip the public, thread-safe
API to ChannelPipeline in favour of directly calling into the
synchronous pipeline interface. This is going to cause us some trouble
when we pull ChannelPipeline out into NIOCore.
To that end, we should add the rest of the ChannelPipeline API surface
area to the SyncOperations structure and convert ourselves over to using
them.
Modifications:
- Implemented the pipeline operations on SyncOperations
- Added a test
- Swapped our channels to use the new APIs.
Result:
Better abstraction between ChannelPipeline and the Channels
Co-authored-by: George Barnett <gbarnett@apple.com>
Motivation:
GetaddrinfoResolver is blocking the calling thread for anything that
uses a `ClientBootstrap`. Often, `ClientBootstrap`s are used on
EventLoops which obviously creates a problem because you mustn't block
EventLoops.
Modifications:
- Offload the `getaddrinfo` call to a `DispatchQueue`.
- Keep the semantics:
* only do one concurrent `getaddrinfo` call per SelectableEventLoop
* prevent thread explosion on `SelectableEventLoop`
Result:
Less EL blocking.
Co-authored-by: George Barnett <gbarnett@apple.com>
Co-authored-by: Cory Benfield <lukasa@apple.com>
Motivation:
Instead of reproducing partial information, it makes more sense to link to the SSWG's guide which is more complete.
Modification:
Remove incomplete information and swap for a link.
Result:
Better docs.
Co-authored-by: Cory Benfield <lukasa@apple.com>
Motivation:
The next step on extracting the base abstractions is to pull out
EventLoop, EventLoopGroup and related types, and also EventLoopPromise
and EventLoopFuture. These types are fundamental to the API.
Modifications:
- Extract EventLoop, EventLoopGroup, and related types.
- Extract EventLoopPromise and EventLoopFuture.
- Add new @testable imports.
- Split MultiThreadedEventLoopGroup into new file, appropriately named.
Result:
Better separation.
Motivation:
In #1904 we included a change to add a debugging hook to confirm a group
could safely be shut down. Sadly, the default behaviour was too
aggressive, and would cause failures in downstream projects.
Modifications:
- Change the default behaviour of _preconditionSafeToSyncShutdown to be
a no-op, as it currently is in NIO programs today.
Result:
Downstream projects won't break.
Cory Benfield