Motivation:
In a few instances, we have code that ignores a `()` argument in a
closure with this syntax:
{ (_: Void) in
whilst that's a bit odd, it shouldn't be wrong and Swift 4.0.x is also
totally happy with it. The latest Swift 4.1 candidates however are not
and also there's a better way of spelling this:
{ () in
and in some cases even just
{
Modifications:
Removed instances of `{ (_: Void) in`
Result:
Compiles with `swift-4.1-DEVELOPMENT-SNAPSHOT-2018-03-12-a`
Motivation:
Websockets is a major protocol in use on the web today, and is
particularly valuable in applications that use asynchronous I/O
as it allows servers to keep connections open for long periods of
time for fully-duplex communication.
Users of NIO should be able to build websocket clients and servers
without too much difficulty.
Modifications:
Provided a WebsocketFrameEncoder and Decoder that can serialize and
deserialize Websocket frames.
Result:
Easier use of websockets.
Motivation:
HTTP pipelining can be tricky to handle properly on the server side.
In particular, it's very easy to write out of order or inconsistently
mutate state. Users often need help to handle this appropriately.
Modifications:
Added a HTTPServerPipelineHandler that only lets one request through
at a time.
Result:
Better servers that are more able to handle HTTP pipelining
Motivation:
We should not close the server socket when accept fails with ECONNABORTED / EMFILE / ENFILE, ENOBUFS. ENOMEM as generally speaking these may be "recoverable" once some existing connections were closed for example. It's possible to implement a backoff strategy for these with a ChannelHandler.
Modifications:
- Special handle of ECONNABORTED / EMFILE / ENFILE, ENOBUFS. ENOMEM for ServerSocketChannel.
Result:
Not close servet socket for errors from which it is possible to recover.
Motivation:
Currently we only supporting adding channel handlers at the front
or the end of the pipeline. This is potentially a bit frustrating:
it becomes impossible to insert handlers relative to a specific
point in the pipeline, which limits users ability to code generic
pipeline modification operations.
Modifications:
Added two new methods to ChannelPipeline, specifically
add(handler:before:) and add(handler:after:). Also performed a
substantial internal refactoring of the ChannelPipeline handler
adding code to ensure that all operations go through the same
logical path, redefining add(handler:first:) in terms of adding
either before tail or after head.
Result:
Users will be able to insert channel handlers at any point in a
ChannelPipeline, so long as they are able to point at a handler
already in that pipeline.
Motivation:
Reduce memory footprint of BaseSocket and FileHandle
Modifications:
- made isOpen a computed var from 'descriptor >= 0'
- add withSocketpair to TestUtils
- use real pipe and socketpair file descriptors for tests
- BaseSocket and FileHandle init check precondition 'descriptor >= 0'
Result:
- isOpen no longer requires storage space
- close() sets descriptor to -1
- descriptor property changed from let to var
- tests will need to use valid file descriptors
Motivation:
We can cleanup the code by removing return keywords in closures sometimes.
Modifications:
Remove return when possible.
Result:
Cleaner code.
Motivation:
Right now addHandlers(first:) does not actually add handlers at the
front of the pipeline: it just adds them *backwards* at the end of
the pipeline. That's definitely not right.
Modifications:
Pass the first: flag into the call to add each specific handler to
ensure they're actually put at the front of the pipeline.
Result:
Handlers added to the front of the pipeline actually are.
Motivation:
Fix for issue #110, .description on IPv6 addresses
return a String with 0 codepoints in it, like so:
[IPv6]::1<NUL><NUL><NUL>....lots..
Modifications:
Use String(cString:) instead of decode, which doesn't
care about \0-cstr terminators.
Result:
Issue fixed, everything is awesome, everyone is happy.
Motivation:
It should be possible to have channels that are not registered
for any form of I/O without them getting stuck in that model
forever.
Modifications:
Remove the code that prevents channels registered for `.none` from
registering for anything else.
Result:
Channels can actually be registered for nothing without becoming
wedged open forever.
Motivation:
Sometimes getting read/getInteger to guess the type correctly can be
annoying and it's handy to just say `as: UInt8.self` or similar. In
other cases it's not as the type is clear from the environment. This now
enables both, just like we already have for `write/set(integer:)`
Modifications:
added an `as: T.Type = T.self` argument which is optional as it has a
default
Result:
easier to use `ByteBuffer` interfaces
Motivation:
We missed to correctly update the cached remote and local addresses for accepted Channels. Because of this localAddress and remoteAddr always returned nil.
Modifications:
- Update cached addresses when constructing SocketChannel from existing Socket.
- Add testcase
Result:
Fixes [#97].
Motivation:
The previous delayed upgrade test didn't really validate what it
claimed to be doing: it validated properties of the I/O, which made
it enormously flaky.
Modifications:
Change the tests to check for the presence/absence of the server
upgrade handler, which is a mark of the actual upgrade process.
Result:
Less flaky tests
Motivation:
When writing programs that use networks, particularly if you want to
perform multicast joins, it's extremely important to be able to query
network interfaces and get results.
Modifications:
Add support for querying interfaces using getifaddrs and wrap those query
results in a Swift class.
Result:
Users will be able to enumerate the interfaces on their machine.
Motivation:
Previously if we'd futureOnEL1.then { ... in futureInEL2 } we'd forget
to hop from EL1 to EL2 which is bad. Shout out to @tanner0101 for
reporting!
Modifications:
Hop from one EventLoop to another if needed.
Result:
thens between Futures of different EventLoops now correctly switch
EventLoops.
Motivation:
Sometimes a user needs to implement ChannelInboundHandler and ChannelOutboundHandler. We should remove some of the "boilerplate" code for doing so.
Modifications:
- Add ChannelDuplexHandler and use it.
Result:
Less boilerplate code.
Motivation:
The upgrade test would race with itself and occasionally fail.
Modifications:
Give enough time for all the work to settle before closing the
client channel.
Result:
No more flaky tests.
Motivation:
We'd go into an infinite loop when trying to increase the capacity of a
0 capacity ByteBuffer *doh*. Thanks @vlm for reporting and writing the
test.
Modifications:
made sure ensureAvailableCapacity returns at least 1 if we don't have
enough capacity available.
Result:
0 capacity ByteBuffers resize now.
Motivation:
FileHandle and BaseSocket should implement a common protocol as these are both "based" on file descriptors
Modifications:
- Add FileDescriptor protocol
- Implement it for FileHandler and BaseSocket
Result:
Better code structure
Motivation:
73a805c7f6 did some changes but missed to rename one variable. This broke the build on linux.
Modifications:
Rename fn to body.
Result:
Builds again on linux.
Motivation:
Sometimes we deviated from the style the Swift stdlib sets out for no
reason.
Modifications:
Fixed some stylistic deviations.
Result:
Looks more Swift-like.
Motivation:
At the moment EmbeddedChannel calls fireChannelRegistered() when connect(...) is called. This is not correct, as it should be done when register(...) is called.
Modifications:
- Correctly call `fireChannelRegistered(...)` when register(...) is called
- Simplify init of EmbeddedChannel
- Fix EmbeddedChannelTest
Result:
Correct invocation of methods.
Motivation:
ChannelLifecycleHandler should record .registered as lifecycle in channelRegistered and not .inactive
Modification:
- Correct record .registered
- Fix tests
Result:
Correctly record lifecycle state.
Motivation:
We no longer have flush promises, so the PWM no longer needs to
handle them.
Modifications:
Removed all flush promise handling from the PWM and associated
tests.
Result:
Less code, more clarity.
Motivation:
A substantial chunk of code complexity exists in the PDWM to manage
flush promises. This complexity is no longer justified now that we
don't bother with them.
Modifications:
Removed much of the complexity of the PDWM handling of flush promises.
Result:
Simpler, easier to understand code.
Motivation:
We used Range for WriteBufferWaterMark which did not do any validation of the low and high marks.
Modifications:
Make WriteBufferWaterMark a struct and validate low and high values
Result:
More correct impl for watermarks.
Motivation:
When the connect() call does not complete synchronously we were not
updating the peer IP address when it finally did complete. This meant
that any channel that didn't connect synchronously (in the real world,
almost all of them) would not correctly see the remote peer IP.
Modifications:
Update the cached addresses on delayed connect.
Result:
Connected channels will see the correct remote IP.
Motivation:
Currently the HTTPUpgradeHandler does not allow for the protocol
upgraders to return EventLoopFuture objects. This leads to an
awkward scenario: it's essentially required that an upgrader be able
to synchronously change the pipeline to its satisfaction and prepare for
more bytes.
This is not really possible: ChannelPipeline.add(handler:) returns a
Future, which means that it is possible for this operation, at least
in principle, to not execute synchronously. This reality puts the
HTTPUpgradeHandler at odds with the reality of channel pipelines, and
cannot stand.
Modifications:
Give the HTTPProtocolUpgrader protocol to have upgrade() return an
EventLoopFuture<Void>, and update the HTTPServerUpgradeHandler to
appropriately handle the returned future by buffering until the future
completes.
Result:
It is no longer necessary to synchronously change the channel
pipeline during protocol upgrade.
Motivation:
Right now the test gen script will rewrite every file. Not ideal!
Modifications:
Added the license header, and made the follow-on comment match that
style.
Result:
Test files will look the same when running this script as they do
now.
Motivation:
The HTTP upgrade handler should not require that we case-match with
the mandatory headers.
Modifications:
Treat all headers as lowercase.
Result:
Upgrade is easier.
Motivation:
guard !self.open else { ... } is a double negation and can be confusing,
guard self.isOpen else { ... } is much better
Modifications:
replaced all `var closed: Bool` with `var isOpen: Bool`
Result:
we're more positive
Motivation:
Out half-closure tests did not wait for the actual event to be received before trying to tear-down the Channel which could lead to test-failures if the event was received not fast enough. We need to ensure we wait until we receive the event before closing the Channel and assert the received events.
Modifications:
Ensure we wait for the events before we try to close the Channel and assert the received events.
Result:
Less flaky tests.
Motivation:
Previously we required the `ByteBuffer.set/write` methods to be a
`Collection` of `UInt8` but a `Sequence` is really enough.
Modifications:
Implemented `ByteBuffer.write/set` for `Sequence`s of `UInt8`
Result:
More stuff can be used with `ByteBuffer`
Motivation:
We have multiple reasons why flush promises weren't great, for example:
- writeAndFlush sounds like it's a write + flush optimisation but in
reality it was more expensive (2 extra promises)
- the semantics were never quite clear
- lots of implementation complexity, especially for the datagram channel
Modifications:
- removed the flush promises in the API
- this deliberately doesn't do the PendingWritesManager simplifications
that this unlocks
Result:
flush doesn't have a promise anymore.
Motivation:
Except for blocking methods (which shouldn't be used on the event loop),
there aren't that many error cases that we should just send through the
pipeline automatically. Best proof for that is that nothing in the code
base did that. Therefore these methods shouldn't be throwing.
If the user does indeed want to catch and fire an error through the
pipeline that's still super simple `ctx.fireErrorCaught(error)`.
Modifications:
removed `throws` from the `ChannelInboundHandler` methods.
Result:
better
Motivation:
ByteToMessageDecoder implementations can potentially have usage
patterns that never reclaim memory. This is problematic when used
for a long time, as they can be holding on to large chunks of memory
they don't need.
Modifications:
Gave ByteToMessageDecoder a new protocol method,
shouldReclaimBytes(buffer:), and a default implementation.
Result:
ByteToMessageDecoders will not be able to hold on to more than 2kB
of unreachable memory under any circumstances, and will often hold
less memory than that.
Motivation:
There were a couple of places in the Channel implementations that just
weren't thread-safe at all, namely:
- localAddress
- remoetAddress
- parent
those are now fixed. I also re-ordered the code so it should be easier
to maintain in the future (the `// MARK` markers were totally
incorrect).
Modifications:
- made Channel.{local,remote}Address return a future
- made Channel.parent a `let`
- unified more of `SocketChannel` and `DatagramSocketChannel`
- fixed the `// MARK`s by reordering code
- annotated methods/properties that are in the Channel API and need to
be thread-safe
Result:
slightly more thread-safety :)
Motivation:
We used a bool to signal if we should continue decoding or not. Using an enum is more self-documenting.
Modifications:
Add DecodingState and use.
Result:
Code is more self-documenting.
Motivation:
Don't make the system fall over under slowloris-style attacks.
Modifications:
Remove the quadratic copy loop from ByteToMessageDecoder.
Result:
NIO goes from 🚂 to 🚄
Motivation:
Foundation is problematic for a few reasons:
- its implementation is different on Linux and on macOS which means our
macOS tests might be inaccurate
- on macOS it uses ObjC Foundation which means the autorelease pool
might get populated
- it links the world on Linux which means we can't do static
binaries at all
Modifications:
removed the last bits of Foundation dependency
Result:
no Foundation dependency
Motivation:
For HTTP parsing, we used Foundation to trim whitespace which is not
needed.
Modifications:
Implemented whitespace trimming in Swift.
Result:
less Foundation
to not return a value
Motivation:
We recently had a bug where we had `EventLoopFuture<EventLoopFuture<()>>` which didn't make any sense. The compiler couldn't catch that problem because we just ignored a closure's argument like this:
future.then { _ in
...
}
which is dangerous. For closures that take an empty tuple, the `_ in`
isn't actually required and the others should state the type they want
to ignore.
And most whenComplete calls can be better (and often shorter) expressed
by other combinators.
Modifications:
remove pretty much all closures which just blanket ignore their
parameter.
Result:
- no closures which just ignore their parameter without at least stating
its type.
- rewrote all whenCompletes that actually used the value
Motivation:
We had a few return statements that could be removed and some places where trailing closure syntax could be used.
Modifications:
- Remove returns
- Use trailing closures
Result:
Cleaner code.
* Expose BlockingIOThreadPool
Motivation:
Sometimes we need to execute some blocking IO. For this we should expose the BlockingIOThreadPool that can be used.
Modifications:
- Factor out BlockingIOThreadPool
- Added tests
- Correctly start threadpool before execute NonBlockingIO tests.
Result:
Possible to do blocking IO.
* Corrys comment
* Correctly start pool before using it
Motivation:
After the previous change to the channel writes, the datagram and the
stream PWMs behaved differently and there was code duplication.
Modifications:
Removed different behaviour & reduced code duplication
Result:
Less code and hopefully less bugs.
Motivation:
Channel writes are a complex matter and it was even more
complex when `FileRegion` and `ByteBuffer` were completely
different kinds of objects. Now that they're more in line we can
simplify a lot of things.
Modifications:
Rewrote the inner layers of channel writes to make them more readable
and hopefully more correct.
Result:
hopefully fewer bugs.
Motivation:
Previously `FileRegion` was a special snow flake, a bit like
`ByteBuffer` but also totally different. That caused surprise and made
`PendingWritesManager` even harder. It was also used to manage the
lifetime of a file descriptor but only sort of.
Modifications:
We now have a `FileHandle` which is a one-to-one mapping with a file
descriptor and its lifetime must be managed appropriately.
Result:
hopefully less bugs and fd leaks.
Motivation:
Lots of our most important operations had redundant labels like
func write(data: NIOAny)
the `data: ` label doesn't add anything meaningful and therefore it
should be removed.
Modifications:
removed lots of redundant labels
Result:
less redundant labels
Motivation:
The code to filter out ChannelError.eof when reading from the socket and not call fireErrorCaught(...) was broken and so the error was propagated through the pipeline.
Modifications:
- Correctly filter out .eof
- Added testcase
Result:
Correct handling of .eof
- This fixes Bootstrap's ChannelOptionStore.applyAll to return a future rather
than synchronously iterating through all the options.
- This is particular important because when a server accepts a child, if the
child channel is on a different event loop then it is possible the
synchronous calls may deadlock (if the child's eventloop happens to be
scheduled with a similar accept sequence).
- I did not tackle also making getOption() async, which means the Channel API
is asymmetric at the moment. That should probably be addressed, potentialy
with synchronous wrappers for API compatibility.
- Fixes: <rdar://problem/37191923> [Omega] Worker tasks fail to close subtasks (many connections in CLOSE_WAIT state)
Motivation:
To ensure we not leave any unread data on the socket we should try to read one last time (if autoread is enabled).
Modifications:
- Depending on if autoread is enabled or not we will try to read one more time
- Guard against re-entry of flushNow0()
- Correctly handle SIGPIPE in all cases
- Add testcase
Result:
Drain data from socket on write error if needed.
We now use two (doubly) linked lists (inbound & outbound) for the
ChannelPipeline. Previously we used two singly linked lists and an
array. That would give us the worst of the two worlds: Linear time
modifications and complicated code.
Before this was printed out.
NIOOpenSSLTests/SSLCertificateTest.swift:105: warning: the use of `mktemp' is dangerous, better use `mkstemp' or `mkdtemp'
Motiviation:
We used to only allocate one buffer per read loop (by default we use multiple reads per loop) and record for the next loop iteration if we need to allocate a bigger buffer or not. This can lead to very slowely ramping up of number of bytes we read. We should better detect if we should allocate a new buffer because we used all the writable bytes to allow faster ramping up of numbre of bytes to read .
Modification:
- Change RecvByteBufferAllocator.record to return a Bool that wil lsignal if we used all the writable bytes of the last allocation and so may want to try to allocate a bigger buffer again
- Adjust tests.
Result:
Faster adjustment of the ByteBuffer capacity when reading data.
Motivation:
The default method we added for the MessageToByteEncoder has the incorrect parameter types and so is useless.
Modifications:
Fix type of parameter.
Result:
Default method match the protocol requirement.
Motivation:
We can allocate the buffer on the fly if needed, no need to add the constructor argument.
Modifications:
Just allocate the buffer in the methods itself that need it.
Result:
Easier to construct encoder and cleaner code.
Motivation:
Often people want to transfer large files over http which should be done using sendfile(...) for best performance.
Modifications:
- Add support for using FileRegion when sending body parts.
- Add tests
Result:
Be able to serve files without loading these into memory
motivation: generated linux tests dont have consitent sorting in different OSs
changes:
* change test generator to sort tests and imports in memory before writing to file
* update LinuxMain to the sorted version
* HTTP/1.1 Chunked Encoding
* fix MarkedCircularBuffer: if not marked do not try to move mark when removing first
* fix HTTPRequestDecoder: dispatch callouts after parsing is complete
* fix HTTPTest: feed pipeline the inbound data correctly
* Make HTTPHeaders iterate with originalcase'd names
Interface change as a consequence of this: now iterating headers
produces (name, value) tuples rather than (name, [values]).
* Add linux test wrapper
* Add case insensitive lookup test
* Update generate test headers
* Make HTTPHeadersIterator private
* Massage HTTPHeaders
Simple had it's own version that was the same aside from a couple of
minor tweaks.
* Default initializer for HTTPHeaders
* Rename HTTPRequest -> HTTPRequestPart
* HTTPResponse -> HTTPResponsePart
* Add HTTPVersion(major:,minor:)
* Allow access to HTTPVersion.major/minor
* Comments on HTTP response codes
* Fix tests (needed rename applied)
* Futures init'd with errors/results are fulfilled
They automatically succeed/fail when new callbacks are added, so
semantically they're already fulfilled, no?
Without this, EventLoop.newSucceedFuture 'leaks' and explodes in the
provided tests.
* Regenerate linux tests
* Add missing generated extension for linux CI
Johannes Weiß