Better docs for threading guarantees of futures (#197)

Sources/NIO/EventLoopFuture.swift

@@ -283,6 +283,53 @@ public struct EventLoopPromise<T> {
 /// * Using `EventLoopFuture<T>.async` or a similar wrapper function.
 /// * Using `.then()` on an existing future to create a new future for the next step in a series of operations.
 /// * Initializing an `EventLoopFuture` that already has a value or an error
+///
+/// ### Threading and Futures
+///
+/// One of the major performance advantages of NIO over something like Node.js or Python’s asyncio is that NIO will
+/// by default run multiple event loops at once, on different threads. As most network protocols do not require
+/// blocking operation, at least in their low level implementations, this provides enormous speedups on machines
+/// with many cores such as most modern servers.
+///
+/// However, it can present a challenge at higher levels of abstraction when coordination between those threads
+/// becomes necessary. This is usually the case whenever the events on one connection (that is, one `Channel`) depend
+/// on events on another one. As these `Channel`s may be scheduled on different event loops (and so different threads)
+/// care needs to be taken to ensure that communication between the two loops is done in a thread-safe manner that
+/// avoids concurrent mutation of shared state from multiple loops at once.
+///
+/// The main primitives NIO provides for this use are the `EventLoopPromise` and `EventLoopFuture`. As their names
+/// suggest, these two objects are aware of event loops, and so can help manage the safety and correctness of your
+/// programs. However, understanding the exact semantics of these objects is critical to ensuring the safety of your code.
+///
+/// ####  Callbacks
+///
+/// The most important principle of the `EventLoopPromise` and `EventLoopFuture` is this: all callbacks registered on
+/// an `EventLoopFuture` will execute on the thread corresponding to the event loop that created the `Future`,
+/// *regardless* of what thread succeeds or fails the corresponding `EventLoopPromise`.
+///
+/// This means that if *your code* created the `EventLoopPromise`, you can be extremely confident of what thread the
+/// callback will execute on: after all, you held the event loop in hand when you created the `EventLoopPromise`.
+/// However, if your code is handed an `EventLoopFuture` or `EventLoopPromise`, and you want to register callbacks
+/// on those objects, you cannot be confident that those callbacks will execute on the same `EventLoop` that your
+/// code does.
+///
+/// This presents a problem: how do you ensure thread-safety when registering callbacks on an arbitrary
+/// `EventLoopFuture`? The short answer is that when you are holding an `EventLoopFuture`, you can always obtain a
+/// new `EventLoopFuture` whose callbacks will execute on your event loop. You do this by calling
+/// `EventLoopFuture.hopTo(eventLoop:)`. This function returns a new `EventLoopFuture` whose callbacks are guaranteed
+/// to fire on the provided event loop. As an added bonus, `hopTo` will check whether the provided `EventLoopFuture`
+/// was already scheduled to dispatch on the event loop in question, and avoid doing any work if that was the case.
+///
+/// This means that for any `EventLoopFuture` that your code did not create itself (via
+/// `EventLoopPromise.futureResult`), use of `hopTo` is **strongly encouraged** to help guarantee thread-safety. It
+/// should only be elided when thread-safety is provably not needed.
+///
+/// The "thread affinity" of `EventLoopFuture`s is critical to writing safe, performant concurrent code without
+/// boilerplate. It allows you to avoid needing to write or use locks in your own code, instead using the natural
+/// synchronization of the `EventLoop` to manage your thread-safety. In general, if any of your `ChannelHandler`s
+/// or `EventLoopFuture` callbacks need to invoke a lock (either directly or in the form of `DispatchQueue`) this
+/// should be considered a code smell worth investigating: the `EventLoop`-based synchronization guarantees of
+/// `EventLoopFuture` should be sufficient to guarantee thread-safety.
 public final class EventLoopFuture<T> {
     // TODO: Provide a tracing facility.  It would be nice to be able to set '.debugTrace = true' on any EventLoopFuture or EventLoopPromise and have every subsequent chained EventLoopFuture report the success result or failure error.  That would simplify some debugging scenarios.
     fileprivate var value: EventLoopFutureValue<T>? {