Replace the non-portable `u_*_t` types with the C standard `u*_t` types
instead. Remove the Unix headers as the only headers really needed are
the C standard headers. Provide shims for deprecated `strings.h`
functions in terms of "modern" (C89+) functions from `string.h`. Move
an android only include into the implementation from the interface.
This enables building this library on Windows.
Co-Authored-By: George Barnett <gbrntt@gmail.com>
Co-authored-by: Johannes Weiss <johannesweiss@apple.com>
* prevent execution of tasks on an already shutdown EL
* fixup! prevent execution of tasks on an already shutdown EL
* fixup! prevent execution of tasks on an already shutdown EL
Co-authored-by: Johannes Weiss <johannesweiss@apple.com>
Motivation:
To read the bytes from timerfd, we use Glibc.read directly but we should use try! Posix.read
Modifications:
Using Posix instead of Glibc for read operation with try! to check for any error.
Result:
Fixes#1341
Co-authored-by: Johannes Weiss <johannesweiss@apple.com>
* removed decodeLast function from NIOChatServer
* updated decodeLast function from WebSocketFrameDecoder
* default implementation of decodeLast function
* removed implementation of decodeLast in WebSocketFrameDecoder
Motivation:
The macOS build was broken by #1424, also the Thread.swift file became
a bit of a mess with `#if os(...)/#else` everywhere.
Modifications:
- fix macOS build
- cleanup Thread by splitting it into ThreadPosix & ThreadWindows
Result:
- build working again
- cleaner code
This restructures `Thread` to have a more generic representation of a
thread handle. Using that it is possible to have an implementation of
the threading APIs for Windows.
Similarly, introduce an abstraction over the Thread Specific Key. This
allows us to abstract away the key type and the usage.
Co-authored-by: Johannes Weiss <johannesweiss@apple.com>
Add support for Windows to the `NIOConcurrencyHelpers`, replacing the
`pthreads` usage for Windows threading primitives.
Co-authored-by: Johannes Weiss <johannesweiss@apple.com>
Motivation:
The
do {
try someOperation()
XCTFail("should throw") // easy to forget
} catch error as SomethingError {
XCTAssertEqual(.something, error as? SomethingError)
} catch {
XCTFail("wrong error")
}
pattern is not only very long, it's also very error prone. If you forget
any of the XCTFails, you might not tests what it looks like
XCTAssertThrowsError(try someOperation) { error in
XCTAssertEqual(.something, error as? SomethingError)
}
is much safer and shorter.
Modifcations:
Do many of the above replaces.
Result:
Cleaner, shorter, and safer tests.
Motivation:
The latest Swift compilers have again added new (great) warnings which
some NIO tests tripped over.
Modifications:
- fix the warnings
- fix/improve some tests
Result:
better code, fewer warnings
Motivation:
list_topsorted_dependencies had multiple issues:
- it would list some executable targets (even in libs-only mode)
- it confused packages with modules
- it would also list dependencies from outside the current package
Modifications:
Address all of the above.
Result:
Script more useful, inside and outside of swift-nio.
Co-authored-by: David Evans <david.evans@blackpoint.co>
* Add offset writes and file size support to NonBlockingFileIO
Motivation:
NonBlockingFileIO currently only supports writes from 0 or appending,
if opened with O_APPEND. To facilitate random read/write file access
the API needs to evolve.
Modifications:
- Add toOffset function parameter to write for offset file write access
- Add changeFileSize function for growing or shrinking the file
- Add readFileSize function for asynchronously reading the file size
Result:
- Improve API for random access scenarios
- Provide method for reading and writing file size
* Conform to code review
* Formatting
* Update Sources/NIO/NonBlockingFileIO.swift
Co-Authored-By: Johannes Weiss <johannesweiss@apple.com>
* Update Sources/NIO/NonBlockingFileIO.swift
Co-Authored-By: Johannes Weiss <johannesweiss@apple.com>
* Modify offset handling to support partial writes.
Clean up stray comment.
* Fix precondition check within offset pwrite handling
* Improve guard for readability
* White space cleanup
* Single spacing issue
Co-authored-by: Johannes Weiss <johannesweiss@apple.com>
motivation: docs scripts is broken on selinux due to restriction on ssh-agent
changes: change the jazzy workspace to a directory accessible by the CI node so we can push from the agent instead of from the docker container
Motivation:
SSWG guidelines recommend that accepted projects list their maturity
level in their README with an appropriate badge.
Modifications:
- Add SSWG graduated badge to README
- Add that gRPC is a SSWG community project
Result:
- A badgier README
On LLP64 hosts, `Int` is mapped to `intptr_t` and `UInt` to `uintptr_t`.
As a result, we need to ensure that we map the atomic extensions to
those types for type conversions.
Add the types for `uintptr_t` and `intptr_t` so that they can be used to
implement `CNIOConcurrencyHelpers` on Windows.
Co-authored-by: Johannes Weiss <johannesweiss@apple.com>
Motivation:
Resolve#1409
Modifications:
Using withContiguousStorageIfAvailable if string.utf8 supports an internal representation in a form. Otherwise, fall back to use withUnsafeBufferPointer.
Result:
Save an allocation for each hash function update
On LLP64 platforms, `Int` is mapped to `intptr_t` and `UInt` is mapped
to `uintptr_t`. Use the newly minted `CNIOAtomics` operations to
support these semantics.
Co-authored-by: Cory Benfield <lukasa@apple.com>
Co-authored-by: Johannes Weiss <johannesweiss@apple.com>
Motivation:
The performance of firstIndex and lastIndex is
dramatically improved by addding custom
indexing methods as detailed in issue 1385.
Modifications:
Added methods to ByteBufferView:
_customIndexOfEquatableElement
for firstIndex
_customLastIndexOfEquatableElement
for lastIndex
Added a unit test for firstIndex and lastIndex
to ensure functional behaviour is unchanged.
Result:
firstIndex and lastIndex behave as before.
Performance is improved by approximately 20% for
the shortest searches.
When tested searching over 1024 byte buffer
with either no match or a match at the end
I observed approximately 15 times performance
Co-authored-by: Johannes Weiss <johannesweiss@apple.com>
Motivation:
Was talking about chunked encoding with @adtrevor, so I thought let's
actually capture it in a test.
Modifications:
- add a test
Result:
- more tests
Use a macro to define the interface for the typed atomics. This reduces
the boiler plate that is expanded by hand. More importantly, it makes
it easier to add new types in the future.
Co-authored-by: Cory Benfield <lukasa@apple.com>
This removes support for Unix Domain Sockets from NIO on Windows.
Although Windows 10.17063 introduced basic support for UDS, the support
is incomplete and has differences from the tradtional UDS sockets in
terms of behaviour. Excise the support initially to get NIO to build on
Windows.
Co-authored-by: Cory Benfield <lukasa@apple.com>
Motivation:
For compatibility with 2.13.1 and earlier, we're adding the
AcceptHandler after the serverChannelInitializer runs. If that's the
best solution is discussed in #1392.
Modifications:
- add AcceptHandler after the serverChannelInitializer
- give the AcceptHandler the name AcceptHandler
Result:
- better compatibility
Motivation:
Somewhere in github or Cocoapods, there seems to be a problem that if
you push a pod A and quickly after a pod B that depends on A with
exactly the version you just pushed, the validation sometimes fails
claiming that the right version of A is not available. The workaround is
to wait a bit and to call `pod repo update`.
The modifications in this PR alleviate this problem somewhat, it now
calls `pod repo update` by itself and should it still go wrong, there's
a new `-f` (from) option that can be used to skip over all pods until
the one that failed last. Example
scripts/build_podspecs.sh -u -f SwiftNIO 2.13.1
Modifications:
- update the pod repo often
- provide a skipping option to skip over earlier repos in case something
still goes wrong
Result:
Easier way to update the pods.
Motivation:
http-parser shipped a patche for node.js CVE-2019-15605, which allowed
HTTP request smuggling. This affected SwiftNIO as well, and so we need
to immediately ship an update to help protect affected users.
A CVE for SwiftNIO will follow, but as this patch is in the wild and
SwiftNIO is known to be affected we should not delay shipping this fix.
Modifications:
- Update http-parser.
- Add regression tests to validate this behaviour.
Result:
Close request smugging vector.
Motivation:
We recently grew a script to diff the output of `malloc-aggregation.d`,
we should document how this can be used when debugging allocation
regressions.
Modifications:
- Update the doc to include an example of using the script to debug an
allocation regression
- Fix a bug in the script where before/after were mixed up
- Fix a bug in the script where usage would print '()\n' instead of '\n'
Result:
- Better info on debugging allocations
Co-authored-by: Johannes Weiss <johannesweiss@apple.com>
Motivation:
Swift 5.2 is already available as developer snapshots and Xcode 11.4
beta got released which also includes Swift 5.2.
Modifications:
- add 5.2 support to the readme
- showcase SwiftPM's package syntax for 5.2
Result:
the future is here.
Motivation:
getInteger used the normal, generic implementation for UInt8s, however
they can be made considerably faster by using a specialised method.
ByteBufferIterator was also affected by this.
Modifications:
Use a specialised version for UInt8s for getInteger.
Result:
- ByteBufferView can be iterated about 20% faster.
Motivation:
This updates all vendored code and also lets github know what's vendored
code through `.gitattributes`.
Modifications:
- http_parser update
- attribution
Result:
- better github stats
- up to date software
Motivation:
EventLoopError does not conform to CustomStringConvertible. When these
errors appear in programs depending on NIO the source of the error is
non-obvious to the user.
EventLoopError.shutdown, for example, would appear in logs as
"shutdown" with no indication that it was the event loop being referred
to.
Modifications:
- Conform EventLoopError to CustomStringConvertible
Result:
- More helpful error descriptions for EventLoopError
Motivation:
Often we know ahead of time that we will need to write a certain number
of bytes into a buffer. It is useful to ensure the buffer has enough
capacity to store these bytes without causing unnecessary allocations.
However, `reserveCapacity()` is defined in terms of the capacity of the
buffer and it's can be easy to forget that this does not take into
account the writer index.
Modifications:
- Add a method which reserves enough capacity to write at least the
specified number of bytes.
- Update `writeWithUnsafeMutableBytes(minimumWritableBytes:body:)` to use the
new method.
Result:
It's easier for users to reserve enough writable capacity ahead of time.
resolves#1355 shaving off some bytes from IOError struct
Motivation:
It's important for frequently used errors to fit in existential containers because otherwise throwing them leads to allocations.
Modifications:
- Don't store `StaticString`s anymore which are 2 words + 1 bytes wide
Result:
Fewer allocations when IOErrors are thrown.
Motivation:
NIO has allocation counter tests, NIO also has dev/malloc-aggregation.d
which can give you aggregated stacks that allocated alongside how many
times they allocated. What's missing is a script that can diff two
aggregated stack traces.
Modifications:
Add dev/stacktrace-dtrace which allows aggregated stack trace diffing.
Result:
Easier to track performance regressions.
Motivation:
The channel initializers should see the the Channel after the options
have been applied so they can potentially override the options.
Modifications:
Set the options before the channel initialiser rather than after.
Result:
- More sense, more compatibility with Netty
- fixes#1114
Co-authored-by: George Barnett <gbrntt@gmail.com>
Co-authored-by: Cory Benfield <lukasa@apple.com>
motivation: more secured ci setup
changes:
* enable :z selinux flag on bind mounts so we can enable selinux on ci
* drop potentially exploitable capabilities from docker-compose
Motivation:
Calling `function(otherFunction)` allocates, but
`function({ otherFunction($0) }` does not.
See: SR-12115 and SR-12116.
Modifications:
Avoid passing functions directly; pass them in new closures instead.
Result:
Fewer allocations.
Motivation:
Clang modules seem to mess with stuff like _GNU_SOURCE as soon as
they're included. We do need _GNU_SOURCE (but don't like the
`#define __USE_GNU` hack).
Modifications:
Include the modularised headers last.
Result:
Alloc counter tests work without `#define __USE_GNU`.
Motivation:
As can be seen in #1358, we have a bunch of very slow tests.
Some of them are unnecessarily slow.
Modifications:
Make easy to make fast tests faster.
Result:
Faster test suite.
motivation: tsan is broken on ubuntu 18.04.use 16.04 so 5.2 build can pass
changes:
* change 5.2 docker setup to use 16.04 instead of 18.04
* add the ability to pipe arguments to the integration tests via the docker setup
* skip tests_02 integration tests suite since it broken on 5.2 (swift-pm issue?)
fake the new limits from #1362
Co-authored-by: tomer doron <tomer@apple.com>
Motivation:
The problem that manifested in #1363 was that when we told the EL to exit, it would sometimes exit straight away and sometimes go around its loop one more time and call whenReady, waiting for a wakeup. Previously, we assumed that it would always wait for a wakeup which isn't true.
Modifications:
Offer it a wakeup if it needs to but don't require it.
Result:
fixes#1363
## Motivation:
In SwiftNIO's codebase, most things were always testable just as well as user code is. With one big missing piece: We could never make the `Selector` do arbitrary things to the `Channel` implementations which lead to super expensive to (create, read, understand) integration tests like [`testWriteAndFlushFromReentrantFlushNowTriggeredOutOfWritabilityWhereOuterSaysAllWrittenAndInnerDoesNot `](5dc9083c6a/Tests/NIOTests/StreamChannelsTest.swift (L468-L760)).
## Modification
Add SAL, a 'syscall abstraction layer' which makes it possible to assert all syscalls that come through a `Socket` or the `Selector` by mocking them. It lets the NIO programmer play the kernel in these "SAL tests" which makes it possible to assert the exact syscalls (with their parameters) and decide the syscalls' returns.
## Result
Finally, we're able to write proper unit tests for scenarios where the `Channel` drove the `Selector` incorrectly.
Saleem Abdulrasool