Motivation:
The `HTTPServerUpgradeHandler` removes itself from the pipeline after an
upgrade and unbuffers any unconsumed reads. If an upgrade starts but
does not complete successfully then the pipeline may be left in an
unknown state.
If, for example, the failure occurs before the user provided upgrade
handler is run then unbuffered writes may be unwrapped by the wrong
channel handler as the wrong type leading to a crash.
Modifications:
- Only remove the upgrade handler and forward buffered writes if all
parts of the upgrade complete successfully.
- If part of the upgrade fails then fire an error into the channel
pipeline without removing the server upgrade handler.
- Remove a few unnecessary `map`s.
- Make `httpEncoder` non-optional since and remove associated dead code
since it can never be `nil`.
Result:
Upgrade handling is safer.
# Motivation
We landed the async bridge types a while back but never added allocation and performance tests. Since we expect these types to be used performance critical paths we really should cover those with tests.
# Modification
Extends the allocation counter scaffolding to support async tests. Furthermore, add allocations tests for both the writer and producer. Lastly, I a also added a performance test for the producer.
# Result
We now have baseline tests for the `NIOAsyncWriter` and `NIOAsyncSequenceProducer`
Co-authored-by: Cory Benfield <lukasa@apple.com>
Motivation
Get the Android build working again
Modifications
- Modify close/open/readdir arguments, as they can be null
- Remove mkpath_np, as it's not there on Android
Result
Android builds again and the same tests pass
Motivation:
As 5.7 has shipped we no longer need to keep these APIs in _NIOBeta.
We're going to do a two-stage removal: first we're going to move the
APIs to NIOCore and keep them in _NIOBeta with deprecations on them. In
a later release, we'll remove the APIs from _NIOBeta entirely.
Modifications:
- Move the TimeAmount + Duration APIs to NIOCore
- Deprecate the APIs in _NIOBeta.
Result:
We're on a path to remove _NIOBeta
Motivation
Some bootstraps may require the ability to remove
Channel options from the ChannelOption storage in
cases where setting a higher-level flag makes those
options unusable. This patch adds that functionality.
Modifications
Add ChannelOptions.Storage.remove
Result
Users can remove channel options from storage
Motivation:
Basic set of function that NonBlockingFileIO provides is enough to read and write files, but as we get more and more into fully asynchronous world we need other non-blocking file-related function.
Modifications:
- Adds support for getting file information using lstat
- Adds ability to create a symlink, delete symlink and read it's destination
- Adds ability to list directories
- Adds ability to rename and remove files
# Motivation
Using the `NIOAsyncSequenceProducer` requires a bunch of knowledge around when and how the delegate is getting called to ensure a correct back-pressure implementation. We should enhance the docs a bit more to surface some of the invariants that we expect.
# Modification
Extend the docs for `NIOAsyncSequenceProducer`.
# Result
Better docs.
Motivation:
When using a `Lock` instance handed by another library and calling `withLock` or `withLockVoid` on it,
the compiler will issue an incorrect fix-it warning that `withLock` (or `withLockVoid`) was renamed to `NIOLock`.
Modifications:
The warning results from annotating the extension that adds `withLock` and `withLockVoid` with `@available(*, deprecated, renamed: "NIOLock")`.
This removes the annotation and moves these two methods into the main declaration of `Lock`.
Result:
The fix-it warning no longer appears when using `withLock` or `withLockVoid`.
Motivation
It's good if we compile with the GM Xcode.
Modifications
Guard Duration usability to at least Swift 5.7.1.
Result
NIO compiles on GM Xcode again.
Note that this is technically an API break for those using Xcode 14
Betas. As those users can (and should) update to 14.1 Beta, and we don't
define API stability for Beta xcodes, this is acceptable.
# Motivation
Our `NIOAsyncSequenceProducer` is a unicast `AsyncSequence`; therefor, we must ensure that only a single iterator is every created. Our failure mode in the case another iterator is created is to `fatalError`. Currently, this `fatalError` is not produced if the sequence is in the `finished` state. This results in hard to debug behaviour since the user will get a basically useless iterator.
# Modification
Ensure that the `finished` state also keeps track of if an iterator was initialised and produces the correct `fatalError`.
# Result
We are now consistent in how many iterators can be created for every state.
Motivation
When we receive a HEAD response, it's possible that the response
contains a content-length. llhttp has a bug
(https://github.com/nodejs/llhttp/issues/202) that prevents it from
properly managing that issue, which causes us to incorrectly parse
responses.
Modifications
Forcibly set llhttp's content-length value to 0.
Result
Correctly handle HTTP framing around llhttp's issues.
Motivation:
If the "New issue" button is clicked on GitHub it takes you straight to
the NIO bug report template. This makes it slightly too easy to
accidentally file a security vulnerability in public.
Modifications:
Move the ISSUE_TEMPLATE.md to ISSUE_TEMPLATE/bug-report.md, doing so
means the "New issue" will take users to a page where they can select
from filing a bug report or reporting a security vulnerability.
Result:
It's clearer when filing an issue that security vulnerabilities are
handled differently.
Motivation:
SwiftNIO periodically drops support for older Swift versions. Now that
5.7 has been released, 5.4 will be dropped.
Modifications:
- Remove 5.4 specific Package.swift and docker-compose
- Update the 5.7 docker-compose to use the released 5.7 and move from
focal (2004) to jammy (2204)
- Remove unused swiftformat from Dockerfile
- Update tools version in syscall wrapper tests to 5.5
- Update docs
Results:
Minimum Swift version is 5.5
Motivation
HTTP headers are prevented from containing certain characters that can
potentially affect parsing or interpretation. Inadequately policing this
can lead to vulnerabilities in web applications, most notably HTTP
Response Splitting.
NIO was insufficiently policing the correctness of the header fields we
emit in HTTP/1.1. We've therefore added a new handler that is
automatically added to channel pipelines that will police the validity
of header fields.
For projects that are already running the validation themselves, this
can be easily disabled. Note that by default NIO does not validate
content length is correctly calculated, so applications can have their
framing fall out of sync unless they appropriately calculate this
themselves or use chunked transfer encoding.
Modifications
- Add thorough unit testing to confirm we will not emit invalid header
fields.
- Error if a user attempts to send an invalid header field.
Result
NIO applications are no longer vulnerable to response splitting by CRLF
injection by default.
# Motivation
We went through a lot of changes for the API of the `NIOAsyncWriter` and some doc comments suffered from this.
# Modification
Update doc comments for the `NIOAsyncWriter`.
Small fixup for the docs of the `NIOLockedValueBox`
# Result
More accurate docs
* Implement a `NIOAsyncWriter`
# Motivation
We previously added the `NIOAsyncProducer` to bridge between the NIO channel pipeline and the asynchronous world. However, we still need something to bridge writes from the asynchronous world back to the NIO channel pipeline.
# Modification
This PR adds a new `NIOAsyncWriter` type that allows us to asynchronously `yield` elements to it. On the other side, we can register a `NIOAsyncWriterDelegate` which will get informed about any written elements. Furthermore, the synchronous side can toggle the writability of the `AsyncWriter` which allows it to implement flow control.
A main goal of this type is to be as performant as possible. To achieve this I did the following things:
- Make everything generic and inlinable
- Use a class with a lock instead of an actor
- Provide methods to yield a sequence of things which allows users to reduce the amount of times the lock gets acquired.
# Result
We now have the means to bridge writes from the asynchronous world to the synchronous
* Remove the completion struct and incorporate code review comments
* Fixup some refactoring leftovers
* More code review comments
* Move to holding the lock around the delegate and moved the delegate into the state machine
* Comment fixups
* More doc fixes
* Call finish when the sink deinits
* Refactor the writer to only yield Deques and rename the delegate to NIOAsyncWriterSinkDelegate
* Review
* Fix some warnings
* Fix benchmark sendability
* Remove Failure generic parameter and allow sending of an error through the Sink
Motivation:
`LockedValueBox` can help programmers to not forget to actually take a
`Lock` when they should.
Modifications:
Add `LockedValue`.
Result:
Hopefully fewer concurrency bugs.
Co-authored-by: Cory Benfield <lukasa@apple.com>
Motivation:
We should better tolerate LLHTTP status codes we don't yet know about.
Modifications:
- Added support for the status codes that currently exist
- Add a fallback to the RAW case for the future.
Result:
Better management of LLHTTP status codes
Motivation:
The node.js HTTP parser library that we use has been unmaintained for some time. We should move to the maintained replacement, which is llhttp. This patch will update our dependency and bring us over to the new library, as well as make any changes we need.
Modifications:
This patch comes in 4 parts, each contained in a separate commit in the PR.
The first commit drops the existing http_parser code and updates some of the repo state for using llhttp.
The second commit rewrites the update script to bring in llhttp instead of http_parser.
The third runs the actual script. You can skip reviewing this except to sanity check the outcome.
The fourth commit updates the NIO code and the tests to get everything working.
In general the substance of the product modifications was minimal. The logic around keeping track of where we are in the buffer and how upgrades work has changed a bit, so that required some fiddling. I also had to add an error reporting path for the delegates to be able to throw specific errors that llhttp no longer checks for. Finally, I removed two tests that were a little overzealous and that llhttp does not police.
Result:
Back on the supported path.
Motivation:
We have downstream CI systems testing us that can't meet this
performance requirement. Given that the debug mode tests are only
checking correctness, not perf, we can widen the buffer.
Modifications:
Double the tolerance of testSystemCallWrapperPerformance in debug mode.
Result:
Downstream CI should be more reliable
* Call finish once the Source is deinited
# Motivation
We **MUST** call `finish()` when the `Source` deinits otherwise we can have a suspended continuation that never gets resumed.
# Modification
Introduce an internal class to both `Source`s and call `finish()` in their `deinit`s.
# Result
We are now resuming all continuations.
* Remove @unchecked
* Small changes for the `NIOAsyncSequenceProducer`
# Motivation
In the PR for the `NIOAsyncWriter`, a couple of comments around naming of `private` properties that needed to be `internal` due to inlinability and other smaller nits came up.
# Modification
This PR includes two things:
1. Fixing up of the small nits like using `_` or getting the imports inside the `#if` checks
2. Changing the public API of the `makeSequence` to be aligned across the throwing and non-throwing one.
# Result
Cleaner code and alinged APIs.
* Fix refactoring left-overs
Motivation
Testing versions of NIO code that involve interfacing with Swift
Concurrency is currently a difficult business. In particular,
EmbeddedChannel is not available in Swift concurrency, making it
difficult to write tests where you fully control the I/O.
To that end, we should provide a variation of EmbeddedChannel that makes
testing these things possible.
Modifications
Provide an implementation of NIOAsyncTestingChannel.
Results
Users can write tests confidently with async/await.
* Add throwing version of `NIOAsyncSequenceProducer`
# Motivation
We recently introduced a `NIOAsyncSequenceProducer` to bridge a stream of elements from the NIO world into the async world. The introduced type was a non-throwing `AsyncSequence`. To support all use-cases we also need to offer a throwing variant of the type.
# Modification
- Introduce a new `NIOThrowingAsyncSequenceProducer` that is identical to the `NIOAsyncSequenceProducer` except that it has a `Failure` generic parameter and that the `next()` method is throwing.
- Extract the `StateMachine` from both `AsyncSequenceProducer`s and unify them.
- There is one modification in behaviour: `didTerminate` is now only called after `nil` or the error has been consumed from the sequence.
# Result
We now have a throwing variant of the `NIOAsyncSequenceProducer`.
* Code review and fix CI
* Remove duplicated code
Co-authored-by: Cory Benfield <lukasa@apple.com>
* initial adoption of DocC based documentaiton
motivation: use DocC for docs
change:
* add DocC catalog to NIO target, as the "landing page"
* remove jazzy doc generation script
* add Package.swift with tools-version 5.6 so that DocC can be used, and add a dependency on the DocC plugin
* udpate soundness script
* * add packge.swift files for older vrsions
* adjust waning-as-error and enable-test-discovery to the different permutations
* fixup
* remove jazzy
Co-authored-by: Cory Benfield <lukasa@apple.com>
Motivation:
One of the `NIOAsyncSequenceProducer` extensions was missing a
availability requirements.
Modifications:
Add missing requirement.
Result:
Fewer bugs.
* Implement a back-pressure aware `AsyncSequence` source
# Motivation
We ran into multiple use-cases (https://github.com/apple/swift-nio/pull/2067, https://github.com/grpc/grpc-swift/blob/main/Sources/GRPC/AsyncAwaitSupport/PassthroughMessageSource.swift) already where we want to vend an `AsyncSequence` where elements are produced from the sync world while the consumer is in the async world. Furthermore, we need the `AsyncSequence` to properly support back-pressure.
Since we already identified that this is something fundamental for our ecosystem and that current `AsyncSequence` sources are not providing the proper semantics or performance, it would be great to find a single solution that we can use everywhere.
Before diving into the code, I think it is good to understand the goals of this `AsyncSequence`:
- The `AsyncSequence` should support a single unicast `Subscriber`
- The `AsyncSequence` should allow a pluggable back-pressure strategy
- The `AsyncSequence` should allow to yield a sequence of elements to avoid aquiring the lock for every element.
- We should make sure to do as few thread hops as possible to signal the producer to demand more elements.
# Modification
This PR introduces a new `AsyncSequence` called `NIOBackPressuredAsyncSequence`. The goal of that sequence to enable sync to async element streaming with back-pressure support.
# Result
We can now power our sync to async use-cases with this new `AsyncSequence`.
# Future work
There are couple of things left that I wanna land in a follow up PR:
1. An adaptive back-pressure strategy that grows and shrinks depending on the speed of consumption
2. A throwing version of this sequence
3. Potentially an async version that suspends on `yield()` and resumes when more elements should be demanded.
* Fix cancellation handling
* Review
* Add init helper
* Add return types to help the type checker
* Fix 5.7 CI
* Rename delegate method and update docs
* Review
* Switch to Deque, rename the type and change the behaviour of yielding an empty sequence
* Review comments from George
* Review comments by Konrad
* Review and add tests for the high low strategy
* Fix 5.4 tests
* Code review
Co-authored-by: Cory Benfield <lukasa@apple.com>
George Barnett