More responsive shutdown logic during handshake and fixed crashes when forcing a shutdown

code polish

LICENSE.MPLv2

@@ -0,0 +1,373 @@
+Mozilla Public License Version 2.0
+==================================
+
+1. Definitions
+--------------
+
+1.1. "Contributor"
+    means each individual or legal entity that creates, contributes to
+    the creation of, or owns Covered Software.
+
+1.2. "Contributor Version"
+    means the combination of the Contributions of others (if any) used
+    by a Contributor and that particular Contributor's Contribution.
+
+1.3. "Contribution"
+    means Covered Software of a particular Contributor.
+
+1.4. "Covered Software"
+    means Source Code Form to which the initial Contributor has attached
+    the notice in Exhibit A, the Executable Form of such Source Code
+    Form, and Modifications of such Source Code Form, in each case
+    including portions thereof.
+
+1.5. "Incompatible With Secondary Licenses"
+    means
+
+    (a) that the initial Contributor has attached the notice described
+        in Exhibit B to the Covered Software; or
+
+    (b) that the Covered Software was made available under the terms of
+        version 1.1 or earlier of the License, but not also under the
+        terms of a Secondary License.
+
+1.6. "Executable Form"
+    means any form of the work other than Source Code Form.
+
+1.7. "Larger Work"
+    means a work that combines Covered Software with other material, in 
+    a separate file or files, that is not Covered Software.
+
+1.8. "License"
+    means this document.
+
+1.9. "Licensable"
+    means having the right to grant, to the maximum extent possible,
+    whether at the time of the initial grant or subsequently, any and
+    all of the rights conveyed by this License.
+
+1.10. "Modifications"
+    means any of the following:
+
+    (a) any file in Source Code Form that results from an addition to,
+        deletion from, or modification of the contents of Covered
+        Software; or
+
+    (b) any new file in Source Code Form that contains any Covered
+        Software.
+
+1.11. "Patent Claims" of a Contributor
+    means any patent claim(s), including without limitation, method,
+    process, and apparatus claims, in any patent Licensable by such
+    Contributor that would be infringed, but for the grant of the
+    License, by the making, using, selling, offering for sale, having
+    made, import, or transfer of either its Contributions or its
+    Contributor Version.
+
+1.12. "Secondary License"
+    means either the GNU General Public License, Version 2.0, the GNU
+    Lesser General Public License, Version 2.1, the GNU Affero General
+    Public License, Version 3.0, or any later versions of those
+    licenses.
+
+1.13. "Source Code Form"
+    means the form of the work preferred for making modifications.
+
+1.14. "You" (or "Your")
+    means an individual or a legal entity exercising rights under this
+    License. For legal entities, "You" includes any entity that
+    controls, is controlled by, or is under common control with You. For
+    purposes of this definition, "control" means (a) the power, direct
+    or indirect, to cause the direction or management of such entity,
+    whether by contract or otherwise, or (b) ownership of more than
+    fifty percent (50%) of the outstanding shares or beneficial
+    ownership of such entity.
+
+2. License Grants and Conditions
+--------------------------------
+
+2.1. Grants
+
+Each Contributor hereby grants You a world-wide, royalty-free,
+non-exclusive license:
+
+(a) under intellectual property rights (other than patent or trademark)
+    Licensable by such Contributor to use, reproduce, make available,
+    modify, display, perform, distribute, and otherwise exploit its
+    Contributions, either on an unmodified basis, with Modifications, or
+    as part of a Larger Work; and
+
+(b) under Patent Claims of such Contributor to make, use, sell, offer
+    for sale, have made, import, and otherwise transfer either its
+    Contributions or its Contributor Version.
+
+2.2. Effective Date
+
+The licenses granted in Section 2.1 with respect to any Contribution
+become effective for each Contribution on the date the Contributor first
+distributes such Contribution.
+
+2.3. Limitations on Grant Scope
+
+The licenses granted in this Section 2 are the only rights granted under
+this License. No additional rights or licenses will be implied from the
+distribution or licensing of Covered Software under this License.
+Notwithstanding Section 2.1(b) above, no patent license is granted by a
+Contributor:
+
+(a) for any code that a Contributor has removed from Covered Software;
+    or
+
+(b) for infringements caused by: (i) Your and any other third party's
+    modifications of Covered Software, or (ii) the combination of its
+    Contributions with other software (except as part of its Contributor
+    Version); or
+
+(c) under Patent Claims infringed by Covered Software in the absence of
+    its Contributions.
+
+This License does not grant any rights in the trademarks, service marks,
+or logos of any Contributor (except as may be necessary to comply with
+the notice requirements in Section 3.4).
+
+2.4. Subsequent Licenses
+
+No Contributor makes additional grants as a result of Your choice to
+distribute the Covered Software under a subsequent version of this
+License (see Section 10.2) or under the terms of a Secondary License (if
+permitted under the terms of Section 3.3).
+
+2.5. Representation
+
+Each Contributor represents that the Contributor believes its
+Contributions are its original creation(s) or it has sufficient rights
+to grant the rights to its Contributions conveyed by this License.
+
+2.6. Fair Use
+
+This License is not intended to limit any rights You have under
+applicable copyright doctrines of fair use, fair dealing, or other
+equivalents.
+
+2.7. Conditions
+
+Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted
+in Section 2.1.
+
+3. Responsibilities
+-------------------
+
+3.1. Distribution of Source Form
+
+All distribution of Covered Software in Source Code Form, including any
+Modifications that You create or to which You contribute, must be under
+the terms of this License. You must inform recipients that the Source
+Code Form of the Covered Software is governed by the terms of this
+License, and how they can obtain a copy of this License. You may not
+attempt to alter or restrict the recipients' rights in the Source Code
+Form.
+
+3.2. Distribution of Executable Form
+
+If You distribute Covered Software in Executable Form then:
+
+(a) such Covered Software must also be made available in Source Code
+    Form, as described in Section 3.1, and You must inform recipients of
+    the Executable Form how they can obtain a copy of such Source Code
+    Form by reasonable means in a timely manner, at a charge no more
+    than the cost of distribution to the recipient; and
+
+(b) You may distribute such Executable Form under the terms of this
+    License, or sublicense it under different terms, provided that the
+    license for the Executable Form does not attempt to limit or alter
+    the recipients' rights in the Source Code Form under this License.
+
+3.3. Distribution of a Larger Work
+
+You may create and distribute a Larger Work under terms of Your choice,
+provided that You also comply with the requirements of this License for
+the Covered Software. If the Larger Work is a combination of Covered
+Software with a work governed by one or more Secondary Licenses, and the
+Covered Software is not Incompatible With Secondary Licenses, this
+License permits You to additionally distribute such Covered Software
+under the terms of such Secondary License(s), so that the recipient of
+the Larger Work may, at their option, further distribute the Covered
+Software under the terms of either this License or such Secondary
+License(s).
+
+3.4. Notices
+
+You may not remove or alter the substance of any license notices
+(including copyright notices, patent notices, disclaimers of warranty,
+or limitations of liability) contained within the Source Code Form of
+the Covered Software, except that You may alter any license notices to
+the extent required to remedy known factual inaccuracies.
+
+3.5. Application of Additional Terms
+
+You may choose to offer, and to charge a fee for, warranty, support,
+indemnity or liability obligations to one or more recipients of Covered
+Software. However, You may do so only on Your own behalf, and not on
+behalf of any Contributor. You must make it absolutely clear that any
+such warranty, support, indemnity, or liability obligation is offered by
+You alone, and You hereby agree to indemnify every Contributor for any
+liability incurred by such Contributor as a result of warranty, support,
+indemnity or liability terms You offer. You may include additional
+disclaimers of warranty and limitations of liability specific to any
+jurisdiction.
+
+4. Inability to Comply Due to Statute or Regulation
+---------------------------------------------------
+
+If it is impossible for You to comply with any of the terms of this
+License with respect to some or all of the Covered Software due to
+statute, judicial order, or regulation then You must: (a) comply with
+the terms of this License to the maximum extent possible; and (b)
+describe the limitations and the code they affect. Such description must
+be placed in a text file included with all distributions of the Covered
+Software under this License. Except to the extent prohibited by statute
+or regulation, such description must be sufficiently detailed for a
+recipient of ordinary skill to be able to understand it.
+
+5. Termination
+--------------
+
+5.1. The rights granted under this License will terminate automatically
+if You fail to comply with any of its terms. However, if You become
+compliant, then the rights granted under this License from a particular
+Contributor are reinstated (a) provisionally, unless and until such
+Contributor explicitly and finally terminates Your grants, and (b) on an
+ongoing basis, if such Contributor fails to notify You of the
+non-compliance by some reasonable means prior to 60 days after You have
+come back into compliance. Moreover, Your grants from a particular
+Contributor are reinstated on an ongoing basis if such Contributor
+notifies You of the non-compliance by some reasonable means, this is the
+first time You have received notice of non-compliance with this License
+from such Contributor, and You become compliant prior to 30 days after
+Your receipt of the notice.
+
+5.2. If You initiate litigation against any entity by asserting a patent
+infringement claim (excluding declaratory judgment actions,
+counter-claims, and cross-claims) alleging that a Contributor Version
+directly or indirectly infringes any patent, then the rights granted to
+You by any and all Contributors for the Covered Software under Section
+2.1 of this License shall terminate.
+
+5.3. In the event of termination under Sections 5.1 or 5.2 above, all
+end user license agreements (excluding distributors and resellers) which
+have been validly granted by You or Your distributors under this License
+prior to termination shall survive termination.
+
+************************************************************************
+*                                                                      *
+*  6. Disclaimer of Warranty                                           *
+*  -------------------------                                           *
+*                                                                      *
+*  Covered Software is provided under this License on an "as is"       *
+*  basis, without warranty of any kind, either expressed, implied, or  *
+*  statutory, including, without limitation, warranties that the       *
+*  Covered Software is free of defects, merchantable, fit for a        *
+*  particular purpose or non-infringing. The entire risk as to the     *
+*  quality and performance of the Covered Software is with You.        *
+*  Should any Covered Software prove defective in any respect, You     *
+*  (not any Contributor) assume the cost of any necessary servicing,   *
+*  repair, or correction. This disclaimer of warranty constitutes an   *
+*  essential part of this License. No use of any Covered Software is   *
+*  authorized under this License except under this disclaimer.         *
+*                                                                      *
+************************************************************************
+
+************************************************************************
+*                                                                      *
+*  7. Limitation of Liability                                          *
+*  --------------------------                                          *
+*                                                                      *
+*  Under no circumstances and under no legal theory, whether tort      *
+*  (including negligence), contract, or otherwise, shall any           *
+*  Contributor, or anyone who distributes Covered Software as          *
+*  permitted above, be liable to You for any direct, indirect,         *
+*  special, incidental, or consequential damages of any character      *
+*  including, without limitation, damages for lost profits, loss of    *
+*  goodwill, work stoppage, computer failure or malfunction, or any    *
+*  and all other commercial damages or losses, even if such party      *
+*  shall have been informed of the possibility of such damages. This   *
+*  limitation of liability shall not apply to liability for death or   *
+*  personal injury resulting from such party's negligence to the       *
+*  extent applicable law prohibits such limitation. Some               *
+*  jurisdictions do not allow the exclusion or limitation of           *
+*  incidental or consequential damages, so this exclusion and          *
+*  limitation may not apply to You.                                    *
+*                                                                      *
+************************************************************************
+
+8. Litigation
+-------------
+
+Any litigation relating to this License may be brought only in the
+courts of a jurisdiction where the defendant maintains its principal
+place of business and such litigation shall be governed by laws of that
+jurisdiction, without reference to its conflict-of-law provisions.
+Nothing in this Section shall prevent a party's ability to bring
+cross-claims or counter-claims.
+
+9. Miscellaneous
+----------------
+
+This License represents the complete agreement concerning the subject
+matter hereof. If any provision of this License is held to be
+unenforceable, such provision shall be reformed only to the extent
+necessary to make it enforceable. Any law or regulation which provides
+that the language of a contract shall be construed against the drafter
+shall not be used to construe this License against a Contributor.
+
+10. Versions of the License
+---------------------------
+
+10.1. New Versions
+
+Mozilla Foundation is the license steward. Except as provided in Section
+10.3, no one other than the license steward has the right to modify or
+publish new versions of this License. Each version will be given a
+distinguishing version number.
+
+10.2. Effect of New Versions
+
+You may distribute the Covered Software under the terms of the version
+of the License under which You originally received the Covered Software,
+or under the terms of any subsequent version published by the license
+steward.
+
+10.3. Modified Versions
+
+If you create software not governed by this License, and you want to
+create a new license for such software, you may create and use a
+modified version of this License if you rename the license and remove
+any references to the name of the license steward (except to note that
+such modified license differs from this License).
+
+10.4. Distributing Source Code Form that is Incompatible With Secondary
+Licenses
+
+If You choose to distribute Source Code Form that is Incompatible With
+Secondary Licenses under the terms of this version of the License, the
+notice described in Exhibit B of this License must be attached.
+
+Exhibit A - Source Code Form License Notice
+-------------------------------------------
+
+  This Source Code Form is subject to the terms of the Mozilla Public
+  License, v. 2.0. If a copy of the MPL was not distributed with this
+  file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+If it is not possible or desirable to put the notice in a particular
+file, then You may include the notice in a location (such as a LICENSE
+file in a relevant directory) where a recipient would be likely to look
+for such a notice.
+
+You may add additional accurate notices of copyright ownership.
+
+Exhibit B - "Incompatible With Secondary Licenses" Notice
+---------------------------------------------------------
+
+  This Source Code Form is "Incompatible With Secondary Licenses", as
+  defined by the Mozilla Public License, v. 2.0.

README.md

@@ -1,89 +1,66 @@
 Network
 =======
 
-###### [![Dorkbox](https://badge.dorkbox.com/dorkbox.svg "Dorkbox")](https://git.dorkbox.com/dorkbox/Network) [![Github](https://badge.dorkbox.com/github.svg "Github")](https://github.com/dorkbox/Network) [![Gitlab](https://badge.dorkbox.com/gitlab.svg "Gitlab")](https://gitlab.com/dorkbox/Network) [![Bitbucket](https://badge.dorkbox.com/bitbucket.svg "Bitbucket")](https://bitbucket.org/dorkbox/Network)
+###### [![Dorkbox](https://badge.dorkbox.com/dorkbox.svg "Dorkbox")](https://git.dorkbox.com/dorkbox/Network) [![Github](https://badge.dorkbox.com/github.svg "Github")](https://github.com/dorkbox/Network) [![Gitlab](https://badge.dorkbox.com/gitlab.svg "Gitlab")](https://gitlab.com/dorkbox/Network)
 
 
-The Network project is an encrypted, high-performance, event-driven/reactive Network stack with DNS and RMI, using Netty, Kryo, KryoNet RMI, and LZ4 via TCP/UDP. 
+The Network project is an ~~encrypted~~, high-performance, event-driven/reactive Network stack with DNS and RMI, using Aeron, Kryo, KryoNet RMI, ~~encryption and LZ4 via UDP.~~ 
 
 These are the main features:
-* The connection between endpoints is AES256-GCM / EC curve25519. (WIP, this was updated for use with Aeron, which changes this)
-* The connection data is LZ4 compressed and byte-packed for small payload sizes. (WIP, this was updated for use with Aeron, which 
-  changes this)
-- The connection supports:
+
+~~* The connection between endpoints is AES256-GCM / EC curve25519. (WIP, this was updated for use with Aeron, which changes this)~~
+~~* The connection data is LZ4 compressed and byte-packed for small payload sizes. (WIP, this was updated for use with Aeron, which 
+  changes this)~~
+### The connection supports:
+ - Sending object (via the Kryo serialization framework)
+ - Sending arbitrarily large objects
  - Remote Method Invocation
    - Blocking
    - Non-Blocking
    - Void returns
    - Exceptions can be returned
    - Kotlin coroutine suspend functions
- - Sending data when Idle
+ - ~~Sending data when Idle~~
  - "Pinging" the remote end (for measuring round-trip time)
  - Firewall connections by IP+CIDR
- - Specify the connection type (nothing, compress, compress+encrypt)
+ - ~~Specify the connection type (nothing, compress, compress+encrypt)~~
  
+- The available transports is UDP
 
-- The available transports are TCP and UDP
-- There are simple wrapper classes for:
-  - Server
-  - Client
-  * MultiCast Broadcast client and server discovery (WIP, this was updated for use with Aeron, which changes this)
-  
-
-- Note: There is a maximum packet size for UDP, 508 bytes *to guarantee it's unfragmented*
-
-- This is for cross-platform use, specifically - linux 32/64, mac 64, and windows 32/64. Java 1.8+
+- This is for cross-platform use, specifically - linux 32/64, mac 64, and windows 32/64. Java 1.11+
+- This library is designed to be used with kotlin, specifically the use of coroutines.
     
 ``` java
-public static
-class AMessage {
-    public
-    AMessage() {
-    }
+val configurationServer = ServerConfiguration()
+configurationServer.settingsStore = Storage.Memory() // don't want to persist anything on disk!
+configurationServer.port = 2000
+configurationServer.enableIPv4 = true
+
+val server: Server<Connection> = Server(configurationServer)
+
+server.onMessage<String> { message ->
+    logger.error("Received message '$message'")
 }
 
-KryoCryptoSerializationManager.DEFAULT.register(AMessage.class);
+server.bind()
 
-Configuration configuration = new Configuration();
-configuration.tcpPort = tcpPort;
-configuration.host = host;
 
-final Server server = new Server(configuration);
-addEndPoint(server);
-server.bind(false);
 
-server.listeners()
-      .add(new Listener<AMessage>() {
-          @Override
-          public
-          void received(Connection connection, AMessage object) {
-              System.err.println("Server received message from client. Bouncing back.");
-              connection.send()
-                        .TCP(object);
-          }
-      });
+val configurationClient = ClientConfiguration()
+configurationClient.settingsStore = Storage.Memory() // don't want to persist anything on disk!
+configurationClient.port = 2000
 
-Client client = new Client(configuration);
-client.disableRemoteKeyValidation();
-addEndPoint(client);
-client.connect(5000);
+val client: Client<Connection> = Client(configurationClient)
 
-client.listeners()
-      .add(new Listener<AMessage>() {
-          @Override
-          public
-          void received(Connection connection, AMessage object) {
-              ClientSendTest.this.checkPassed.set(true);
-              System.err.println("Tada! It's been bounced back.");
-              server.stop();
-          }
-      });
+client.onConnect {
+    send("client test message")
+}
 
-client.send()
-      .TCP(new AMessage());
+client.connect()
 
 ```
 
+
 &nbsp; 
 &nbsp; 
 
@@ -95,7 +72,7 @@ Maven Info
     <dependency>
       <groupId>com.dorkbox</groupId>
       <artifactId>Network</artifactId>
-      <version>5.32</version>
+      <version>6.15</version>
     </dependency>
 </dependencies>
 ```
@@ -105,11 +82,11 @@ Gradle Info
 ```
 dependencies {
     ...
-    implementation("com.dorkbox:Network:5.32")
+    implementation("com.dorkbox:Network:6.15")
 }
 ```
 
 License
 ---------
-This project is © 2021 dorkbox llc, and is distributed under the terms of the Apache v2.0 License. See file "LICENSE" for further 
+This project is © 2023 dorkbox llc, and is distributed under the terms of the Apache v2.0 License. See file "LICENSE" for further 
 references.

build.gradle.kts

@@ -1,5 +1,5 @@
 /*
- * Copyright 2020 dorkbox, llc
+ * Copyright 2023 dorkbox, llc
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -14,8 +14,6 @@
  * limitations under the License.
  */
 
-import java.time.Instant
-
 ///////////////////////////////
 //////    PUBLISH TO SONATYPE / MAVEN CENTRAL
 ////// TESTING : (to local maven repo) <'publish and release' - 'publishToMavenLocal'>
@@ -25,19 +23,22 @@ import java.time.Instant
 gradle.startParameter.showStacktrace = ShowStacktrace.ALWAYS   // always show the stacktrace!
 
 plugins {
-    id("com.dorkbox.GradleUtils") version "2.17"
-    id("com.dorkbox.Licensing") version "2.12"
-    id("com.dorkbox.VersionUpdate") version "2.5"
-    id("com.dorkbox.GradlePublish") version "1.12"
+    id("com.dorkbox.GradleUtils") version "3.18"
+    id("com.dorkbox.Licensing") version "2.28"
+    id("com.dorkbox.VersionUpdate") version "2.8"
+    id("com.dorkbox.GradlePublish") version "1.22"
 
-    kotlin("jvm") version "1.6.10"
+    id("com.github.johnrengelman.shadow") version "8.1.1"
+
+    kotlin("jvm") version "1.9.0"
 }
 
+@Suppress("ConstPropertyName")
 object Extras {
     // set for the project
     const val description = "High-performance, event-driven/reactive network stack for Java 11+"
     const val group = "com.dorkbox"
-    const val version = "5.32"
+    const val version = "6.15"
 
     // set as project.ext
     const val name = "Network"
@@ -45,8 +46,6 @@ object Extras {
     const val vendor = "Dorkbox LLC"
     const val vendorUrl = "https://dorkbox.com"
     const val url = "https://git.dorkbox.com/dorkbox/Network"
-
-    val buildDate = Instant.now().toString()
 }
 
 ///////////////////////////////
@@ -60,31 +59,38 @@ GradleUtils.compileConfiguration(JavaVersion.VERSION_11) {
     // enable the use of inline classes. see https://kotlinlang.org/docs/reference/inline-classes.html
     freeCompilerArgs = listOf("-Xinline-classes")
 }
-//NOTE: we do not support JPMS yet, as there are some libraries missing support for it still
 
 
-// ratelimiter, "other" package
+//val kotlin = project.extensions.getByType(org.jetbrains.kotlin.gradle.dsl.KotlinJvmProjectExtension::class.java).sourceSets.getByName("main").kotlin
+//kotlin.apply {
+//    setSrcDirs(project.files("src"))
+//    include("**/*.kt") // want to include kotlin files for the source. 'setSrcDirs' resets includes...
+//}
+
+// TODO: driver name resolution: https://github.com/real-logic/aeron/wiki/Driver-Name-Resolution
+//  this keeps us from having to restart the media driver when a connection changes IP addresses
+
+// TODO: virtual threads in java21 for polling?
+
+// if we are sending a SMALL byte array, then we SEND IT DIRECTLY in a more optimized manner (because we can count size info!)
+//   other side has to be able to parse/know that this was sent directly as bytes. It could be game state data, or voice data, etc.
+// another idea is to be able to "send" a stream of bytes (this would also get chunked/etc!). if chunked, these are fixed byte sizes!
+// -- the first byte manage: byte/message/stream/etc, no-crypt, crypt, crypt+compress
+// - connection.inputStream() --> behaves as an input stream to remote endpoint --> connection.outputStream()
+//   -- open/close/flush/etc commands also go through
+//   -- this can be used to stream files/audio/etc VERY easily
+//   -- have a createInputStream(), which will cause the outputStream() on the remote end to be created.
+//     --- this remote outputStream is a file, raw??? this is setup by createInputStream() on the remote end
+// - state-machine for kryo class registrations?
+
+// ratelimiter, "other" package, send-on-idle
 // rest of unit tests
 // getConnectionUpgradeType
 // ability to send with a function callback (using RMI waiter type stuff for callbacks)
-// use conscrypt?!
 
 // java 14 is faster with aeron!
-// NOTE: now using aeron instead of netty
-// todo: remove BC! use conscrypt instead, or native java? (if possible. we are java 11 now, instead of 1.6)
 
 
-// also, NOT using bouncastle, but instead the google one
-// better SSL library
-// implementation("org.conscrypt:conscrypt-openjdk-uber:2.2.1")
-//    init {
-//            try {
-//                Security.insertProviderAt(Conscrypt.newProvider(), 1);
-//            }
-//            catch (e: Throwable) {
-//                e.printStackTrace();
-//            }
-//        }
 
 
 licensing {
@@ -134,66 +140,92 @@ tasks.jar.get().apply {
         attributes["Specification-Vendor"] = Extras.vendor
 
         attributes["Implementation-Title"] = "${Extras.group}.${Extras.id}"
-        attributes["Implementation-Version"] = Extras.buildDate
+        attributes["Implementation-Version"] = GradleUtils.now()
         attributes["Implementation-Vendor"] = Extras.vendor
     }
 }
 
+
+val shadowJar: com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar by tasks
+shadowJar.apply {
+    manifest.inheritFrom(tasks.jar.get().manifest)
+
+    manifest.attributes.apply {
+        put("Main-Class", "dorkboxTest.network.app.AeronClientServerForever")
+    }
+
+    mergeServiceFiles()
+
+    duplicatesStrategy = DuplicatesStrategy.INCLUDE
+
+    from(sourceSets.test.get().output)
+    configurations = listOf(project.configurations.testRuntimeClasspath.get())
+
+    archiveBaseName.set(project.name + "-all")
+}
+
+
 dependencies {
-    api("org.jetbrains.kotlinx:atomicfu:0.17.3")
-    api("org.jetbrains.kotlinx:kotlinx-coroutines-core:1.6.2")
+    api("org.jetbrains.kotlinx:atomicfu:0.23.0")
+    api("org.jetbrains.kotlinx:kotlinx-coroutines-core:1.7.3")
 
     // https://github.com/dorkbox
-    api("com.dorkbox:ByteUtilities:1.5")
-    api("com.dorkbox:Collections:1.1")
-    api("com.dorkbox:MinLog:2.4")
-    api("com.dorkbox:NetworkDNS:2.7.1")
-    api("com.dorkbox:NetworkUtils:2.18")
-    api("com.dorkbox:ObjectPool:4.0")
-    api("com.dorkbox:OS:1.0")
-    api("com.dorkbox:Serializers:2.7")
-    api("com.dorkbox:Storage:1.1")
+    api("com.dorkbox:ByteUtilities:2.1")
+    api("com.dorkbox:ClassUtils:1.3")
+    api("com.dorkbox:Collections:2.7")
+    api("com.dorkbox:HexUtilities:1.1")
+    api("com.dorkbox:JNA:1.4")
+    api("com.dorkbox:MinLog:2.7")
+    api("com.dorkbox:NetworkDNS:2.16")
+    api("com.dorkbox:NetworkUtils:2.23")
+    api("com.dorkbox:OS:1.11")
+    api("com.dorkbox:Serializers:2.9")
+    api("com.dorkbox:Storage:1.11")
     api("com.dorkbox:Updates:1.1")
-    api("com.dorkbox:Utilities:1.29")
+    api("com.dorkbox:Utilities:1.48")
+
+
+    // how we bypass using reflection/jpms to access fields for java17+
+    api("org.javassist:javassist:3.29.2-GA")
+
+
+    val jnaVersion = "5.13.0"
+    api("net.java.dev.jna:jna-jpms:${jnaVersion}")
+    api("net.java.dev.jna:jna-platform-jpms:${jnaVersion}")
 
 
-    // we include ALL of aeron, in case we need to debug aeron behavior
     // https://github.com/real-logic/aeron
-    val aeronVer = "1.38.1"
-    api("io.aeron:aeron-all:$aeronVer")
-//    api("io.aeron:aeron-client:$aeronVer")
-//    api("io.aeron:aeron-driver:$aeronVer")
+    val aeronVer = "1.42.1"
+    api("io.aeron:aeron-driver:$aeronVer")
+    // ALL of aeron, in case we need to debug aeron behavior
+//    api("io.aeron:aeron-all:$aeronVer")
+//    api("org.agrona:agrona:1.18.2") // sources for this aren't included in aeron-all!
 
     // https://github.com/EsotericSoftware/kryo
-    api("com.esotericsoftware:kryo:5.3.0") {
+    api("com.esotericsoftware:kryo:5.5.0") {
         exclude("com.esotericsoftware", "minlog") // we use our own minlog, that logs to SLF4j instead
     }
 
-    // https://github.com/jpountz/lz4-java
-//    implementation("net.jpountz.lz4:lz4:1.3.0")
 
-    // this is NOT the same thing as LMAX disruptor.
-    // This is just a slightly faster queue than LMAX. (LMAX is a fast queue + other things w/ a difficult DSL)
-    // https://github.com/conversant/disruptor_benchmark
-    // https://www.youtube.com/watch?v=jVMOgQgYzWU
-    //api("com.conversantmedia:disruptor:1.2.19")
+    // https://github.com/lz4/lz4-java
+    api("org.lz4:lz4-java:1.8.0")
 
-    // https://github.com/jhalterman/typetools
-    api("net.jodah:typetools:0.6.3")
 
     // Expiring Map (A high performance thread-safe map that expires entries)
     // https://github.com/jhalterman/expiringmap
-    api("net.jodah:expiringmap:0.5.10")
+    api("net.jodah:expiringmap:0.5.11")
 
     // https://github.com/MicroUtils/kotlin-logging
-    api("io.github.microutils:kotlin-logging:2.1.23")
-    api("org.slf4j:slf4j-api:1.8.0-beta4")
-
+//    api("io.github.microutils:kotlin-logging:3.0.5")
+    implementation("org.slf4j:slf4j-api:2.0.9")
 
 
 
     testImplementation("junit:junit:4.13.2")
-    testImplementation("ch.qos.logback:logback-classic:1.3.0-alpha4")
+    testImplementation("ch.qos.logback:logback-classic:1.4.5")
+    testImplementation("io.aeron:aeron-all:$aeronVer")
+
+    testImplementation("com.dorkbox:Config:2.1")
 }
 
 publishToSonatype {

gradle/wrapper/gradle-wrapper.properties

@@ -1,5 +1,6 @@
 distributionBase=GRADLE_USER_HOME
 distributionPath=wrapper/dists
-distributionUrl=https\://services.gradle.org/distributions/gradle-7.4-all.zip
+distributionUrl=https\://services.gradle.org/distributions/gradle-8.4-all.zip
+validateDistributionUrl=true
 zipStoreBase=GRADLE_USER_HOME
 zipStorePath=wrapper/dists

gradlew

@@ -55,7 +55,7 @@
 #       Darwin, MinGW, and NonStop.
 #
 #   (3) This script is generated from the Groovy template
-#       https://github.com/gradle/gradle/blob/master/subprojects/plugins/src/main/resources/org/gradle/api/internal/plugins/unixStartScript.txt
+#       https://github.com/gradle/gradle/blob/HEAD/subprojects/plugins/src/main/resources/org/gradle/api/internal/plugins/unixStartScript.txt
 #       within the Gradle project.
 #
 #       You can find Gradle at https://github.com/gradle/gradle/.
@@ -80,13 +80,11 @@ do
     esac
 done
 
-APP_HOME=$( cd "${APP_HOME:-./}" && pwd -P ) || exit
-
-APP_NAME="Gradle"
+# This is normally unused
+# shellcheck disable=SC2034
 APP_BASE_NAME=${0##*/}
-
-# Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
-DEFAULT_JVM_OPTS='"-Xmx64m" "-Xms64m"'
+# Discard cd standard output in case $CDPATH is set (https://github.com/gradle/gradle/issues/25036)
+APP_HOME=$( cd "${APP_HOME:-./}" > /dev/null && pwd -P ) || exit
 
 # Use the maximum available, or set MAX_FD != -1 to use that value.
 MAX_FD=maximum
@@ -133,22 +131,29 @@ location of your Java installation."
     fi
 else
     JAVACMD=java
-    which java >/dev/null 2>&1 || die "ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.
+    if ! command -v java >/dev/null 2>&1
+    then
+        die "ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.
 
 Please set the JAVA_HOME variable in your environment to match the
 location of your Java installation."
+    fi
 fi
 
 # Increase the maximum file descriptors if we can.
 if ! "$cygwin" && ! "$darwin" && ! "$nonstop" ; then
     case $MAX_FD in #(
       max*)
+        # In POSIX sh, ulimit -H is undefined. That's why the result is checked to see if it worked.
+        # shellcheck disable=SC3045
         MAX_FD=$( ulimit -H -n ) ||
             warn "Could not query maximum file descriptor limit"
     esac
     case $MAX_FD in  #(
       '' | soft) :;; #(
       *)
+        # In POSIX sh, ulimit -n is undefined. That's why the result is checked to see if it worked.
+        # shellcheck disable=SC3045
         ulimit -n "$MAX_FD" ||
             warn "Could not set maximum file descriptor limit to $MAX_FD"
     esac
@@ -193,6 +198,10 @@ if "$cygwin" || "$msys" ; then
     done
 fi
 
+
+# Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
+DEFAULT_JVM_OPTS='"-Xmx64m" "-Xms64m"'
+
 # Collect all arguments for the java command;
 #   * $DEFAULT_JVM_OPTS, $JAVA_OPTS, and $GRADLE_OPTS can contain fragments of
 #     shell script including quotes and variable substitutions, so put them in
@@ -205,6 +214,12 @@ set -- \
         org.gradle.wrapper.GradleWrapperMain \
         "$@"
 
+# Stop when "xargs" is not available.
+if ! command -v xargs >/dev/null 2>&1
+then
+    die "xargs is not available"
+fi
+
 # Use "xargs" to parse quoted args.
 #
 # With -n1 it outputs one arg per line, with the quotes and backslashes removed.

gradlew.bat

@@ -14,7 +14,7 @@
 @rem limitations under the License.
 @rem
 
-@if "%DEBUG%" == "" @echo off
+@if "%DEBUG%"=="" @echo off
 @rem ##########################################################################
 @rem
 @rem  Gradle startup script for Windows
@@ -25,7 +25,8 @@
 if "%OS%"=="Windows_NT" setlocal
 
 set DIRNAME=%~dp0
-if "%DIRNAME%" == "" set DIRNAME=.
+if "%DIRNAME%"=="" set DIRNAME=.
+@rem This is normally unused
 set APP_BASE_NAME=%~n0
 set APP_HOME=%DIRNAME%
 
@@ -40,7 +41,7 @@ if defined JAVA_HOME goto findJavaFromJavaHome
 
 set JAVA_EXE=java.exe
 %JAVA_EXE% -version >NUL 2>&1
-if "%ERRORLEVEL%" == "0" goto execute
+if %ERRORLEVEL% equ 0 goto execute
 
 echo.
 echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.
@@ -75,13 +76,15 @@ set CLASSPATH=%APP_HOME%\gradle\wrapper\gradle-wrapper.jar
 
 :end
 @rem End local scope for the variables with windows NT shell
-if "%ERRORLEVEL%"=="0" goto mainEnd
+if %ERRORLEVEL% equ 0 goto mainEnd
 
 :fail
 rem Set variable GRADLE_EXIT_CONSOLE if you need the _script_ return code instead of
 rem the _cmd.exe /c_ return code!
-if  not "" == "%GRADLE_EXIT_CONSOLE%" exit 1
-exit /b 1
+set EXIT_CODE=%ERRORLEVEL%
+if %EXIT_CODE% equ 0 set EXIT_CODE=1
+if not ""=="%GRADLE_EXIT_CONSOLE%" exit %EXIT_CODE%
+exit /b %EXIT_CODE%
 
 :mainEnd
 if "%OS%"=="Windows_NT" endlocal

not-fixed/CryptoEccNative.kt

@@ -1,293 +0,0 @@
-package dorkbox.network.other
-
-import java.math.BigInteger
-import java.security.GeneralSecurityException
-import java.security.KeyFactory
-import java.security.KeyPair
-import java.security.KeyPairGenerator
-import java.security.PrivateKey
-import java.security.SecureRandom
-import java.security.interfaces.ECPrivateKey
-import java.security.interfaces.ECPublicKey
-import java.security.spec.ECField
-import java.security.spec.ECFieldFp
-import java.security.spec.ECParameterSpec
-import java.security.spec.ECPoint
-import java.security.spec.ECPublicKeySpec
-import java.security.spec.EllipticCurve
-import java.security.spec.NamedParameterSpec
-import java.security.spec.PKCS8EncodedKeySpec
-import java.security.spec.X509EncodedKeySpec
-import javax.crypto.Cipher
-
-
-
-/**
- *
- */
-private object CryptoEccNative {
-    // see: https://openjdk.java.net/jeps/324
-
-    const val curve25519 = "curve25519"
-    const val default_curve = curve25519
-
-    const val macSize = 512
-    // on NIST vs 25519 vs Brainpool, see:
-    //  - http://ogryb.blogspot.de/2014/11/why-i-dont-trust-nist-p-256.html
-    //  - http://credelius.com/credelius/?p=97
-    //  - http://safecurves.cr.yp.to/
-    // we should be using 25519, because NIST and brainpool are "unsafe". Brainpool is "more random" than 25519, but is still not considered safe.
-
-    // more info about ECC from:
-    // http://www.johannes-bauer.com/compsci/ecc/?menuid=4
-    // http://stackoverflow.com/questions/7419183/problems-implementing-ecdh-on-android-using-bouncycastle
-    // http://tools.ietf.org/html/draft-jivsov-openpgp-ecc-06#page-4
-    // http://www.nsa.gov/ia/programs/suiteb_cryptography/
-    // https://github.com/nelenkov/ecdh-kx/blob/master/src/org/nick/ecdhkx/Crypto.java
-    // http://nelenkov.blogspot.com/2011/12/using-ecdh-on-android.html
-    // http://www.secg.org/collateral/sec1_final.pdf
-
-    // More info about 25519 key types (ed25519 and X25519)
-    // https://blog.filippo.io/using-ed25519-keys-for-encryption/
-
-
-    fun createKeyPair(secureRandom: SecureRandom): KeyPair {
-        val kpg: KeyPairGenerator = KeyPairGenerator.getInstance("XDH")
-        kpg.initialize(NamedParameterSpec.X25519, secureRandom)
-        return kpg.generateKeyPair()
-
-
-//        println("--- Public Key ---")
-//        val publicKey = kp.public
-//
-//        System.out.println(publicKey.algorithm) // XDH
-//        System.out.println(publicKey.format)    // X.509
-//
-//        // save this public key
-//        val pubKey = publicKey.encoded
-//
-//        println("---")
-//
-//        println("--- Private Key ---")
-//        val privateKey = kp.private
-//
-//        System.out.println(privateKey.algorithm);  // XDH
-//        System.out.println(privateKey.format);     // PKCS#8
-//
-//        // save this private key
-//        val priKey = privateKey.encoded
-
-
-//        val kf: KeyFactory = KeyFactory.getInstance("XDH");
-
-//        //BigInteger u = ...
-//        val pubSpec: XECPublicKeySpec = XECPublicKeySpec(paramSpec, u);
-//        val pubKey: PublicKey = kf.generatePublic(pubSpec);
-//        //
-//
-//        val ka: KeyAgreement = KeyAgreement.getInstance("XDH");
-//        ka.init(kp.private);
-        //ka.doPhase(pubKey, true);
-        //byte[] secret = ka.generateSecret();
-    }
-
-
-
-    private val FieldP_2: BigInteger = BigInteger.TWO // constant for scalar operations
-    private val FieldP_3: BigInteger = BigInteger.valueOf(3) // constant for scalar operations
-    private const val byteVal1 = 1.toByte()
-
-    @Throws(GeneralSecurityException::class)
-    fun getPublicKey(pk: ECPrivateKey): ECPublicKey? {
-        val params: ECParameterSpec = pk.params
-        val w: ECPoint = scalmultNew(params, params.generator, pk.s)
-
-        //final ECPoint w = scalmult(params.getCurve(), pk.getParams().getGenerator(), pk.getS());
-        val kg: KeyFactory = KeyFactory.getInstance("EC")
-        return kg.generatePublic(ECPublicKeySpec(w, params)) as ECPublicKey
-    }
-
-    private fun scalmultNew(params: ECParameterSpec, g: ECPoint, kin: BigInteger): ECPoint {
-        val curve = params.curve
-        val field = curve.field
-        if (field !is ECFieldFp) throw java.lang.UnsupportedOperationException(field::class.java.canonicalName)
-
-        val p = field.p
-        val a = curve.a
-        var R = ECPoint.POINT_INFINITY
-
-        // value only valid for curve secp256k1, code taken from https://www.secg.org/sec2-v2.pdf,
-        // see "Finally the order n of G and the cofactor are: n = "FF.."
-        val SECP256K1_Q = params.order
-        //BigInteger SECP256K1_Q = new BigInteger("00FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFEBAAEDCE6AF48A03BBFD25E8CD0364141",16);
-        var k = kin.mod(SECP256K1_Q) // uses this !
-        // BigInteger k = kin.mod(p); // do not use this ! wrong as per comment from President James Moveon Polk
-        val length = k.bitLength()
-        val binarray = ByteArray(length)
-
-        for (i in 0..length - 1) {
-            binarray[i] = k.mod(FieldP_2).byteValueExact()
-            k = k.shiftRight(1)
-        }
-        for (i in length - 1 downTo 0) {
-            R = doublePoint(p, a, R)
-            if (binarray[i] == byteVal1) R = addPoint(p, a, R, g)
-        }
-
-        return R
-    }
-
-    fun scalmultOrg(curve: EllipticCurve, g: ECPoint, kin: BigInteger): ECPoint {
-        val field: ECField = curve.getField()
-        if (field !is ECFieldFp) throw UnsupportedOperationException(field::class.java.canonicalName)
-        val p: BigInteger = (field as ECFieldFp).getP()
-        val a: BigInteger = curve.getA()
-        var R = ECPoint.POINT_INFINITY
-        // value only valid for curve secp256k1, code taken from https://www.secg.org/sec2-v2.pdf,
-        // see "Finally the order n of G and the cofactor are: n = "FF.."
-        val SECP256K1_Q = BigInteger("00FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFEBAAEDCE6AF48A03BBFD25E8CD0364141", 16)
-        var k = kin.mod(SECP256K1_Q) // uses this !
-
-        // wrong as per comment from President James Moveon Polk
-        // BigInteger k = kin.mod(p); // do not use this !
-        println(" SECP256K1_Q: $SECP256K1_Q")
-        println("           p: $p")
-
-        System.out.println("curve: " + curve.toString())
-        val length = k.bitLength()
-        val binarray = ByteArray(length)
-        for (i in 0..length - 1) {
-            binarray[i] = k.mod(FieldP_2).byteValueExact()
-            k = k.shiftRight(1)
-        }
-        for (i in length - 1 downTo 0) {
-            R = doublePoint(p, a, R)
-            if (binarray[i] == byteVal1) R = addPoint(p, a, R, g)
-        }
-        return R
-    }
-
-    // scalar operations for native java
-    // https://stackoverflow.com/a/42797410/8166854
-    // written by author: SkateScout
-    private fun doublePoint(p: BigInteger, a: BigInteger, R: ECPoint): ECPoint? {
-        if (R == ECPoint.POINT_INFINITY) return R
-
-        var slope = R.affineX.pow(2).multiply(FieldP_3)
-        slope = slope.add(a)
-        slope = slope.multiply(R.affineY.multiply(FieldP_2).modInverse(p))
-
-        val Xout = slope.pow(2).subtract(R.affineX.multiply(FieldP_2)).mod(p)
-        val Yout = R.affineY.negate().add(slope.multiply(R.affineX.subtract(Xout))).mod(p)
-
-        return ECPoint(Xout, Yout)
-    }
-
-    private fun addPoint(p: BigInteger, a: BigInteger, r: ECPoint, g: ECPoint): ECPoint? {
-        if (r == ECPoint.POINT_INFINITY) return g
-        if (g == ECPoint.POINT_INFINITY) return r
-
-        if (r == g || r == g) return doublePoint(p, a, r)
-
-        val gX = g.affineX
-        val sY = g.affineY
-        val rX = r.affineX
-        val rY = r.affineY
-
-        val slope = rY.subtract(sY).multiply(rX.subtract(gX).modInverse(p)).mod(p)
-        val Xout = slope.modPow(FieldP_2, p).subtract(rX).subtract(gX).mod(p)
-        var Yout = sY.negate().mod(p)
-        Yout = Yout.add(slope.multiply(gX.subtract(Xout))).mod(p)
-        return ECPoint(Xout, Yout)
-    }
-
-
-    private fun byteArrayToHexString(a: ByteArray): String {
-        val sb = StringBuilder(a.size * 2)
-        for (b in a) sb.append(String.format("%02X", b))
-        return sb.toString()
-    }
-
-    fun hexStringToByteArray(s: String): ByteArray {
-        val len = s.length
-        val data = ByteArray(len / 2)
-        var i = 0
-        while (i < len) {
-            data[i / 2] = ((Character.digit(s[i], 16) shl 4)
-                    + Character.digit(s[i + 1], 16)).toByte()
-            i += 2
-        }
-        return data
-    }
-
-    @Throws(GeneralSecurityException::class)
-    @JvmStatic
-    fun main(args: Array<String>) {
-        val cryptoText = "i23j4jh234kjh234kjh23lkjnfa9s8egfuypuh325"
-
-        // NOTE: THIS IS NOT 25519!!
-        println("Generate ECPublicKey from PrivateKey (String) for curve secp256k1 (final)")
-        println("Check keys with https://gobittest.appspot.com/Address")
-
-        // https://gobittest.appspot.com/Address
-        val privateKey = "D12D2FACA9AD92828D89683778CB8DFCCDBD6C9E92F6AB7D6065E8AACC1FF6D6"
-        val publicKeyExpected = "04661BA57FED0D115222E30FE7E9509325EE30E7E284D3641E6FB5E67368C2DB185ADA8EFC5DC43AF6BF474A41ED6237573DC4ED693D49102C42FFC88510500799"
-        println("\nprivatekey given : $privateKey")
-        println("publicKeyExpected: $publicKeyExpected")
-
-
-//        // routine with bouncy castle
-//        println("\nGenerate PublicKey from PrivateKey with BouncyCastle")
-//        val spec: ECNamedCurveParameterSpec = ECNamedCurveTable.getParameterSpec("secp256k1") // this ec curve is used for bitcoin operations
-//        val pointQ: org.bouncycastle.math.ec.ECPoint = spec.getG().multiply(BigInteger(1, ch.qos.logback.core.encoder.ByteArrayUtil.hexStringToByteArray(privateKey)))
-//        val publickKeyByte = pointQ.getEncoded(false)
-//        val publicKeyBc: String = byteArrayToHexString(publickKeyByte)
-//        println("publicKeyExpected: $publicKeyExpected")
-//        println("publicKey BC     : $publicKeyBc")
-//        println("publicKeys match : " + publicKeyBc.contentEquals(publicKeyExpected))
-
-        // regeneration of ECPublicKey with java native starts here
-        println("\nGenerate PublicKey from PrivateKey with Java native routines")
-        // the preset "303E.." only works for elliptic curve secp256k1
-        // see answer by user dave_thompson_085
-        // https://stackoverflow.com/questions/48832170/generate-ec-public-key-from-byte-array-private-key-in-native-java-7
-
-        val privateKeyFull = "303E020100301006072A8648CE3D020106052B8104000A042730250201010420" + privateKey
-        val privateKeyFullByte: ByteArray = hexStringToByteArray(privateKeyFull)
-
-
-        println("privateKey full  : $privateKeyFull")
-        val keyFactory = KeyFactory.getInstance("EC")
-        val privateKeyNative: PrivateKey = keyFactory.generatePrivate(PKCS8EncodedKeySpec(privateKeyFullByte))
-        val ecPrivateKeyNative = privateKeyNative as ECPrivateKey
-
-        val ecPublicKeyNative = getPublicKey(ecPrivateKeyNative)
-        val ecPublicKeyNativeByte = ecPublicKeyNative!!.encoded
-
-        val testPubKey = keyFactory.generatePublic(X509EncodedKeySpec(ecPublicKeyNativeByte)) as ECPublicKey
-        val equal = ecPublicKeyNativeByte.contentEquals(testPubKey.encoded)
-
-        val publicKeyNativeFull: String = byteArrayToHexString(ecPublicKeyNativeByte)
-        val publicKeyNativeHeader = publicKeyNativeFull.substring(0, 46)
-        val publicKeyNativeKey = publicKeyNativeFull.substring(46, 176)
-
-        println("ecPublicKeyFull  : $publicKeyNativeFull")
-        println("ecPublicKeyHeader: $publicKeyNativeHeader")
-        println("ecPublicKeyKey   : $publicKeyNativeKey")
-        println("publicKeyExpected: $publicKeyExpected")
-        println("publicKeys match : " + publicKeyNativeKey.contentEquals(publicKeyExpected))
-
-
-        // encrypt
-        val encryptCipher: Cipher = Cipher.getInstance("RSA")
-        encryptCipher.init(Cipher.ENCRYPT_MODE, ecPublicKeyNative)
-        val cipherText: ByteArray = encryptCipher.doFinal(cryptoText.toByteArray())
-
-        // decrypt
-        val decryptCipher = Cipher.getInstance("RSA");
-        decryptCipher.init(Cipher.DECRYPT_MODE, ecPrivateKeyNative);
-
-        val outputBytes = decryptCipher.doFinal(cipherText)
-        println("Crypto round passed: ${String(outputBytes) == cryptoText}")
-    }
-}

not-fixed/LargeResizeBufferTest.java

@@ -1,159 +0,0 @@
-/* Copyright (c) 2008, Nathan Sweet
- * All rights reserved.
- *
- * Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
- * - Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials provided with the distribution.
- * - Neither the name of Esoteric Software nor the names of its contributors may be used to endorse or promote products derived
- * from this software without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING,
- * BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT
- * SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
- * NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
- */
-
-import static org.junit.Assert.fail;
-
-import java.io.IOException;
-import java.security.SecureRandom;
-import java.util.concurrent.atomic.AtomicInteger;
-
-import org.junit.Test;
-
-import dorkbox.network.connection.Connection;
-import dorkbox.network.connection.Listener;
-import dorkbox.network.serialization.Serialization;
-import dorkbox.util.exceptions.SecurityException;
-import dorkbox.util.serialization.SerializationManager;
-
-public
-class LargeResizeBufferTest extends BaseTest {
-    private static final int OBJ_SIZE = 1024 * 100;
-
-    private volatile int finalCheckAmount = 0;
-    private volatile int serverCheck = -1;
-    private volatile int clientCheck = -1;
-
-    @Test
-    public
-    void manyLargeMessages() throws SecurityException, IOException {
-        final int messageCount = 1024;
-
-        Configuration configuration = new Configuration();
-        configuration.tcpPort = tcpPort;
-        configuration.udpPort = udpPort;
-        configuration.host = host;
-        register(configuration.serialization);
-
-        Server server = new Server(configuration);
-        addEndPoint(server);
-        server.bind(false);
-
-        server.listeners()
-              .add(new Listener.OnMessageReceived<Connection, LargeMessage>() {
-                  AtomicInteger received = new AtomicInteger();
-                  AtomicInteger receivedBytes = new AtomicInteger();
-
-                  @Override
-                  public
-                  void received(Connection connection, LargeMessage object) {
-                      // System.err.println("Server ack message: " + received.get());
-
-                      connection.send()
-                                .TCP(object);
-                      this.receivedBytes.addAndGet(object.bytes.length);
-
-                      if (this.received.incrementAndGet() == messageCount) {
-                          System.out.println("Server received all " + messageCount + " messages!");
-                          System.out.println("Server received and sent " + this.receivedBytes.get() + " bytes.");
-                          LargeResizeBufferTest.this.serverCheck = LargeResizeBufferTest.this.finalCheckAmount - this.receivedBytes.get();
-                          System.out.println("Server missed " + LargeResizeBufferTest.this.serverCheck + " bytes.");
-                          stopEndPoints();
-                      }
-                  }
-              });
-
-        Client client = new Client(configuration);
-        addEndPoint(client);
-
-        client.listeners()
-              .add(new Listener.OnMessageReceived<Connection, LargeMessage>() {
-                  AtomicInteger received = new AtomicInteger();
-                  AtomicInteger receivedBytes = new AtomicInteger();
-
-                  @Override
-                  public
-                  void received(Connection connection, LargeMessage object) {
-                      this.receivedBytes.addAndGet(object.bytes.length);
-
-                      int count = this.received.getAndIncrement();
-                      // System.out.println("Client received message: " + count);
-
-                      if (count == messageCount) {
-                          System.out.println("Client received all " + messageCount + " messages!");
-                          System.out.println("Client received and sent " + this.receivedBytes.get() + " bytes.");
-                          LargeResizeBufferTest.this.clientCheck = LargeResizeBufferTest.this.finalCheckAmount - this.receivedBytes.get();
-                          System.out.println("Client missed " + LargeResizeBufferTest.this.clientCheck + " bytes.");
-                      }
-                  }
-              });
-        client.connect(5000);
-
-        SecureRandom random = new SecureRandom();
-
-        System.err.println("  Client sending " + messageCount + " messages");
-        for (int i = 0; i < messageCount; i++) {
-            this.finalCheckAmount += OBJ_SIZE; // keep increasing size
-
-            byte[] b = new byte[OBJ_SIZE];
-            random.nextBytes(b);
-
-            // set some of the bytes to be all `244`, just so some compression can occur (to test that as well)
-            for (int j = 0; j < 400; j++) {
-                b[j] = (byte) 244;
-            }
-
-//            System.err.println("Sending " + b.length + " bytes");
-            client.send()
-                  .TCP(new LargeMessage(b));
-        }
-
-        System.err.println("Client has queued " + messageCount + " messages.");
-
-        waitForThreads();
-
-        if (this.clientCheck > 0) {
-            fail("Client missed " + this.clientCheck + " bytes.");
-        }
-
-        if (this.serverCheck > 0) {
-            fail("Server missed " + this.serverCheck + " bytes.");
-        }
-    }
-
-    private
-    void register(SerializationManager manager) {
-        manager.register(byte[].class);
-        manager.register(LargeMessage.class);
-    }
-
-    public static
-    class LargeMessage {
-        public byte[] bytes;
-
-        public
-        LargeMessage() {
-        }
-
-        public
-        LargeMessage(byte[] bytes) {
-            this.bytes = bytes;
-        }
-    }
-}

not-fixed/Misc.kt

@@ -1,197 +0,0 @@
-package dorkbox.network.other
-
-import kotlin.math.ceil
-
-/**
- *
- */
-object Misc {
-
-    private fun annotations() {
-        //    internal val classesWithRmiFields = IdentityMap<Class<*>, Array<Field>>()
-//        // get all classes that have fields with @Rmi field annotation.
-//        // THESE classes must be customized with our special RmiFieldSerializer serializer so that the @Rmi field is properly handled
-//
-//        // SPECIFICALLY, these fields must also be an IFACE for the field type!
-//
-//        // NOTE: The @Rmi field type will already have to be a registered type with kryo!
-//        // we can use this information on WHERE to scan for classes.
-//        val filesToScan = mutableSetOf<File>()
-//
-//        classesToRegister.forEach { registration ->
-//            val clazz = registration.clazz
-//
-//            // can't do anything if codeSource is null!
-//            val codeSource = clazz.protectionDomain.codeSource ?: return@forEach
-//            // file:/Users/home/java/libs/xyz-123.jar
-//            // file:/projects/classes
-//            val jarOrClassPath = codeSource.location.toString()
-//
-//            if (jarOrClassPath.endsWith(".jar")) {
-//                val fileName: String = URLDecoder.decode(jarOrClassPath.substring("file:".length), Charset.defaultCharset())
-//                filesToScan.add(File(fileName).absoluteFile)
-//            } else {
-//                val classPath: String = URLDecoder.decode(jarOrClassPath.substring("file:".length), Charset.defaultCharset())
-//                filesToScan.add(File(classPath).absoluteFile)
-//            }
-//        }
-//
-//        val toTypedArray = filesToScan.toTypedArray()
-//        if (logger.isTraceEnabled) {
-//            toTypedArray.forEach {
-//                logger.trace { "Adding location to annotation scanner: $it"}
-//            }
-//        }
-//
-//
-//
-//        // now scan these jars/directories
-//        val fieldsWithRmiAnnotation = AnnotationDetector.scanFiles(*toTypedArray)
-//            .forAnnotations(Rmi::class.java)
-//            .on(ElementType.FIELD)
-//            .collect { cursor -> Pair(cursor.type, cursor.field!!) }
-//
-//        // have to make sure that the field type is specified as an interface (and not an implementation)
-//        fieldsWithRmiAnnotation.forEach { pair ->
-//            require(pair.second.type.isInterface) { "@Rmi annotated fields must be an interface!" }
-//        }
-//
-//        if (fieldsWithRmiAnnotation.isNotEmpty()) {
-//            logger.info { "Verifying scanned classes containing @Rmi field annotations" }
-//        }
-//
-//        // have to put this in a map, so we can quickly lookup + get the fields later on.
-//        // NOTE: a single class can have MULTIPLE fields with @Rmi annotations!
-//        val rmiAnnotationMap = IdentityMap<Class<*>, MutableList<Field>>()
-//        fieldsWithRmiAnnotation.forEach {
-//            var fields = rmiAnnotationMap[it.first]
-//            if (fields == null) {
-//                fields = mutableListOf()
-//            }
-//
-//            fields.add(it.second)
-//            rmiAnnotationMap.put(it.first, fields)
-//        }
-//
-//        // now make it an array for fast lookup for the [parent class] -> [annotated fields]
-//        rmiAnnotationMap.forEach {
-//            classesWithRmiFields.put(it.key, it.value.toTypedArray())
-//        }
-//
-//        // this will set up the class registration information
-//        initKryo()
-//
-//        // now everything is REGISTERED, possibly with custom serializers, we have to go back and change them to use our RmiFieldSerializer
-//        fieldsWithRmiAnnotation.forEach FIELD_SCAN@{ pair ->
-//            // the parent class must be an IMPL. The reason is that THIS FIELD will be sent as a RMI object, and this can only
-//            // happen on objects that exist
-//
-//            // NOTE: it IS necessary for the rmi-client to be aware of the @Rmi annotation (because it also has to have the correct serialization)
-//
-//            // also, it is possible for the class that has the @Rmi field to be a NORMAL object (and not an RMI object)
-//            // this means we found the registration for the @Rmi field annotation
-//
-//            val parentRmiRegistration = classesToRegister.firstOrNull { it is ClassRegistrationForRmi && it.implClass == pair.first}
-//
-//
-//            // if we have a parent-class registration, this means we are the rmi-server
-//            //
-//            // AND BECAUSE OF THIS
-//            //
-//            // we must also have the field type registered as RMI
-//            if (parentRmiRegistration != null) {
-//                // rmi-server
-//
-//                // is the field type registered also?
-//                val fieldRmiRegistration = classesToRegister.firstOrNull { it.clazz == pair.second.type}
-//                require(fieldRmiRegistration is ClassRegistrationForRmi) { "${pair.second.type} is not registered for RMI! Unable to continue"}
-//
-//                logger.trace { "Found @Rmi field annotation '${pair.second.type}' in class '${pair.first}'" }
-//            } else {
-//                // rmi-client
-//
-//                // NOTE: rmi-server MUST have the field IMPL registered (ie: via RegisterRmi)
-//                //       rmi-client will have the serialization updated from the rmi-server during connection handshake
-//            }
-//        }
-    }
-
-
-    /**
-     * Split array into chunks, max of 256 chunks.
-     * byte[0] = chunk ID
-     * byte[1] = total chunks (0-255) (where 0->1, 2->3, 127->127 because this is indexed by a byte)
-     */
-    private fun divideArray(source: ByteArray, chunksize: Int): Array<ByteArray>? {
-        val fragments = ceil(source.size / chunksize.toDouble()).toInt()
-        if (fragments > 127) {
-            // cannot allow more than 127
-            return null
-        }
-
-        // pre-allocate the memory
-        val splitArray = Array(fragments) { ByteArray(chunksize + 2) }
-        var start = 0
-        for (i in splitArray.indices) {
-            var length = if (start + chunksize > source.size) {
-                source.size - start
-            } else {
-                chunksize
-            }
-            splitArray[i] = ByteArray(length + 2)
-            splitArray[i][0] = i.toByte() // index
-            splitArray[i][1] = fragments.toByte() // total number of fragments
-            System.arraycopy(source, start, splitArray[i], 2, length)
-            start += chunksize
-        }
-        return splitArray
-    }
-}
-
-//    fun initClassRegistration(channel: Channel, registration: Registration): Boolean {
-//        val details = serialization.getKryoRegistrationDetails()
-//        val length = details.size
-//        if (length > Serialization.CLASS_REGISTRATION_VALIDATION_FRAGMENT_SIZE) {
-//            // it is too large to send in a single packet
-//
-//            // child arrays have index 0 also as their 'index' and 1 is the total number of fragments
-//            val fragments = divideArray(details, Serialization.CLASS_REGISTRATION_VALIDATION_FRAGMENT_SIZE)
-//            if (fragments == null) {
-//                logger.error("Too many classes have been registered for Serialization. Please report this issue")
-//                return false
-//            }
-//            val allButLast = fragments.size - 1
-//            for (i in 0 until allButLast) {
-//                val fragment = fragments[i]
-//                val fragmentedRegistration = Registration.hello(registration.oneTimePad, config.settingsStore.getPublicKey())
-//                fragmentedRegistration.payload = fragment
-//
-//                // tell the server we are fragmented
-//                fragmentedRegistration.upgradeType = UpgradeType.FRAGMENTED
-//
-//                // tell the server we are upgraded (it will bounce back telling us to connect)
-//                fragmentedRegistration.upgraded = true
-//                channel.writeAndFlush(fragmentedRegistration)
-//            }
-//
-//            // now tell the server we are done with the fragments
-//            val fragmentedRegistration = Registration.hello(registration.oneTimePad, config.settingsStore.getPublicKey())
-//            fragmentedRegistration.payload = fragments[allButLast]
-//
-//            // tell the server we are fragmented
-//            fragmentedRegistration.upgradeType = UpgradeType.FRAGMENTED
-//
-//            // tell the server we are upgraded (it will bounce back telling us to connect)
-//            fragmentedRegistration.upgraded = true
-//            channel.writeAndFlush(fragmentedRegistration)
-//        } else {
-//            registration.payload = details
-//
-//            // tell the server we are upgraded (it will bounce back telling us to connect)
-//            registration.upgraded = true
-//            channel.writeAndFlush(registration)
-//        }
-//        return true
-//    }
-
-

not-fixed/MultipleThreadTest.java

@@ -1,242 +0,0 @@
-/* Copyright (c) 2008, Nathan Sweet
- * All rights reserved.
- *
- * Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
- * - Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials provided with the distribution.
- * - Neither the name of Esoteric Software nor the names of its contributors may be used to endorse or promote products derived
- * from this software without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING,
- * BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT
- * SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
- * NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
- */
-
-import static org.junit.Assert.assertEquals;
-
-import java.io.IOException;
-import java.util.ArrayList;
-import java.util.List;
-import java.util.Map;
-import java.util.concurrent.ConcurrentHashMap;
-import java.util.concurrent.atomic.AtomicInteger;
-
-import org.junit.Test;
-
-import dorkbox.network.connection.Connection;
-import dorkbox.network.connection.Listener;
-import dorkbox.network.connection.Listeners;
-import dorkbox.network.serialization.Serialization;
-import dorkbox.util.exceptions.SecurityException;
-
-public
-class MultipleThreadTest extends BaseTest {
-    private final Object lock = new Object();
-    private volatile boolean stillRunning = false;
-
-    private final Object finalRunLock = new Object();
-    private volatile boolean finalStillRunning = false;
-
-    private final int messageCount = 150;
-    private final int threadCount = 15;
-    private final int clientCount = 13;
-
-    private final List<Client> clients = new ArrayList<Client>(this.clientCount);
-
-    int perClientReceiveTotal = (this.messageCount * this.threadCount);
-    int serverReceiveTotal = perClientReceiveTotal * this.clientCount;
-
-    AtomicInteger sent = new AtomicInteger(0);
-    AtomicInteger totalClientReceived = new AtomicInteger(0);
-    AtomicInteger receivedServer = new AtomicInteger(1);
-
-    ConcurrentHashMap<Integer, DataClass> sentStringsToClientDebug = new ConcurrentHashMap<Integer, DataClass>();
-
-    @Test
-    public
-    void multipleThreads() throws SecurityException, IOException {
-        // our clients should receive messageCount * threadCount * clientCount TOTAL messages
-        final int totalClientReceivedCountExpected = this.clientCount * this.messageCount * this.threadCount;
-        final int totalServerReceivedCountExpected = this.clientCount * this.messageCount;
-
-        System.err.println("CLIENT RECEIVES: " + totalClientReceivedCountExpected);
-        System.err.println("SERVER RECEIVES: " + totalServerReceivedCountExpected);
-
-
-        Configuration configuration = new Configuration();
-        configuration.tcpPort = tcpPort;
-        configuration.host = host;
-        configuration.serialization.register(String[].class);
-        configuration.serialization.register(DataClass.class);
-
-
-        final Server server = new Server(configuration);
-        server.disableRemoteKeyValidation();
-
-        addEndPoint(server);
-        server.bind(false);
-
-
-        final Listeners listeners = server.listeners();
-        listeners.add(new Listener.OnConnected<Connection>() {
-
-            @Override
-            public
-            void connected(final Connection connection) {
-                System.err.println("Client connected to server.");
-
-                // kickoff however many threads we need, and send data to the client.
-                for (int i = 1; i <= MultipleThreadTest.this.threadCount; i++) {
-                    final int index = i;
-                    new Thread() {
-                        @Override
-                        public
-                        void run() {
-                            for (int i = 1; i <= MultipleThreadTest.this.messageCount; i++) {
-                                int incrementAndGet = MultipleThreadTest.this.sent.getAndIncrement();
-                                DataClass dataClass = new DataClass("Server -> client. Thread #" + index + "  message# " + incrementAndGet,
-                                                                    incrementAndGet);
-
-                                //System.err.println(dataClass.data);
-                                MultipleThreadTest.this.sentStringsToClientDebug.put(incrementAndGet, dataClass);
-                                connection.send()
-                                          .TCP(dataClass)
-                                          .flush();
-                            }
-
-                        }
-                    }.start();
-                }
-            }
-        });
-
-        listeners.add(new Listener.OnMessageReceived<Connection, DataClass>() {
-            @Override
-            public
-            void received(Connection connection, DataClass object) {
-                int incrementAndGet = MultipleThreadTest.this.receivedServer.getAndIncrement();
-                //System.err.println("server #" + incrementAndGet);
-
-
-                if (incrementAndGet % MultipleThreadTest.this.messageCount == 0) {
-                    System.err.println("Server receive DONE for client " + incrementAndGet);
-
-                    stillRunning = false;
-                    synchronized (MultipleThreadTest.this.lock) {
-                        MultipleThreadTest.this.lock.notifyAll();
-                    }
-                }
-
-                if (incrementAndGet == totalServerReceivedCountExpected) {
-                    System.err.println("Server DONE: " + incrementAndGet);
-
-                    finalStillRunning = false;
-                    synchronized (MultipleThreadTest.this.finalRunLock) {
-                        MultipleThreadTest.this.finalRunLock.notifyAll();
-                    }
-                }
-            }
-        });
-
-        // ----
-        finalStillRunning = true;
-        for (int i = 1; i <= this.clientCount; i++) {
-            final int index = i;
-
-            Client client = new Client(configuration);
-            this.clients.add(client);
-
-            addEndPoint(client);
-            client.listeners()
-                  .add(new Listener.OnMessageReceived<Connection, DataClass>() {
-                      final int clientIndex = index;
-                      final AtomicInteger received = new AtomicInteger(1);
-
-                      @Override
-                      public
-                      void received(Connection connection, DataClass object) {
-                          totalClientReceived.getAndIncrement();
-                          int clientLocalCounter = this.received.getAndIncrement();
-                          MultipleThreadTest.this.sentStringsToClientDebug.remove(object.index);
-
-                          //System.err.println(object.data);
-                          // we finished!!
-                          if (clientLocalCounter == perClientReceiveTotal) {
-                              //System.err.println("Client #" + clientIndex + " received " + clientLocalCounter + " Sending back " +
-                              //                   MultipleThreadTest.this.messageCount + " messages.");
-
-                              // now spam back messages!
-                              for (int i = 0; i < MultipleThreadTest.this.messageCount; i++) {
-                                  connection.send()
-                                            .TCP(new DataClass("Client #" + clientIndex + " -> Server  message " + i, index));
-                              }
-                          }
-                      }
-                  });
-
-
-            stillRunning = true;
-
-            client.connect(5000);
-
-            while (stillRunning) {
-                synchronized (this.lock) {
-                    try {
-                        this.lock.wait(5 * 1000); // 5 secs
-                    } catch (InterruptedException e) {
-                        e.printStackTrace();
-                    }
-                }
-            }
-        }
-
-        while (finalStillRunning) {
-            synchronized (this.finalRunLock) {
-                try {
-                    this.finalRunLock.wait(5 * 1000); // 5 secs
-                } catch (InterruptedException e) {
-                    e.printStackTrace();
-                }
-            }
-        }
-
-        // CLIENT will wait until it's done connecting, but SERVER is async.
-        // the ONLY way to safely work in the server is with LISTENERS. Everything else can FAIL, because of it's async nature.
-
-        if (!this.sentStringsToClientDebug.isEmpty()) {
-            System.err.println("MISSED DATA: " + this.sentStringsToClientDebug.size());
-            for (Map.Entry<Integer, DataClass> i : this.sentStringsToClientDebug.entrySet()) {
-                System.err.println(i.getKey() + " : " + i.getValue().data);
-            }
-        }
-
-        stopEndPoints();
-        assertEquals(totalClientReceivedCountExpected, totalClientReceived.get());
-
-        // offset by 1 since we start at 1
-        assertEquals(totalServerReceivedCountExpected, receivedServer.get()-1);
-    }
-
-
-    public static
-    class DataClass {
-        public String data;
-        public Integer index;
-
-        public
-        DataClass() {
-        }
-
-        public
-        DataClass(String data, Integer index) {
-            this.data = data;
-            this.index = index;
-        }
-    }
-}

not-fixed/PingPongLocalTest.java

@@ -1,326 +0,0 @@
-/* Copyright (c) 2008, Nathan Sweet
- * All rights reserved.
- *
- * Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
- * - Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials provided with the distribution.
- * - Neither the name of Esoteric Software nor the names of its contributors may be used to endorse or promote products derived
- * from this software without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING,
- * BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT
- * SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
- * NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
- */
-
-import static org.junit.Assert.fail;
-
-import java.io.IOException;
-import java.util.Arrays;
-import java.util.concurrent.atomic.AtomicInteger;
-
-import org.junit.Test;
-
-import dorkbox.network.connection.Connection;
-import dorkbox.network.connection.Listener;
-import dorkbox.network.connection.Listeners;
-import dorkbox.network.serialization.Serialization;
-import dorkbox.util.exceptions.SecurityException;
-import dorkbox.util.serialization.SerializationManager;
-
-public
-class PingPongLocalTest extends BaseTest {
-    int tries = 10000;
-    private volatile String fail;
-
-    @Test
-    public void pingPongLocal() throws SecurityException, IOException {
-        this.fail = "Data not received.";
-
-        final Data dataLOCAL = new Data();
-        populateData(dataLOCAL);
-
-        Configuration configuration = Configuration.localOnly();
-        register(configuration.serialization);
-
-
-        Server server = new Server(configuration);
-        addEndPoint(server);
-        server.bind(false);
-        final Listeners listeners = server.listeners();
-        listeners.add(new Listener.OnError<Connection>() {
-            @Override
-            public
-            void error(Connection connection, Throwable throwable) {
-                PingPongLocalTest.this.fail = "Error during processing. " + throwable;
-            }
-        });
-        listeners.add(new Listener.OnMessageReceived<Connection, Data>() {
-            @Override
-            public
-            void received(Connection connection, Data data) {
-                connection.id();
-                if (!data.equals(dataLOCAL)) {
-                    PingPongLocalTest.this.fail = "data is not equal on server.";
-                    throw new RuntimeException("Fail! " + PingPongLocalTest.this.fail);
-                }
-                connection.send()
-                          .TCP(data);
-            }
-        });
-
-            // ----
-
-        Client client = new Client(configuration);
-        addEndPoint(client);
-        final Listeners listeners1 = client.listeners();
-        listeners1.add(new Listener.OnConnected<Connection>() {
-            @Override
-            public
-            void connected(Connection connection) {
-                PingPongLocalTest.this.fail = null;
-                connection.send()
-                          .TCP(dataLOCAL);
-                // connection.sendUDP(dataUDP); // TCP and UDP are the same for a local channel.
-            }
-        });
-
-        listeners1.add(new Listener.OnError<Connection>() {
-            @Override
-            public
-            void error(Connection connection, Throwable throwable) {
-                PingPongLocalTest.this.fail = "Error during processing. " + throwable;
-                System.err.println(PingPongLocalTest.this.fail);
-            }
-        });
-
-        listeners1.add(new Listener.OnMessageReceived<Connection, Data>() {
-            AtomicInteger check = new AtomicInteger(0);
-
-            @Override
-            public
-            void received(Connection connection, Data data) {
-                if (!data.equals(dataLOCAL)) {
-                    PingPongLocalTest.this.fail = "data is not equal on client.";
-                    throw new RuntimeException("Fail! " + PingPongLocalTest.this.fail);
-                }
-
-                if (this.check.getAndIncrement() <= PingPongLocalTest.this.tries) {
-                    connection.send()
-                              .TCP(data);
-                }
-                else {
-                    System.err.println("Ran LOCAL " + PingPongLocalTest.this.tries + " times");
-                    stopEndPoints();
-                }
-            }
-        });
-
-        client.connect(5000);
-
-        waitForThreads();
-
-        if (this.fail != null) {
-            fail(this.fail);
-        }
-    }
-
-    private void populateData(Data data) {
-        StringBuilder buffer = new StringBuilder();
-        for (int i = 0; i < 3000; i++) {
-            buffer.append('a');
-        }
-        data.string = buffer.toString();
-
-        data.strings = new String[] {"abcdefghijklmnopqrstuvwxyz0123456789","",null,"!@#$","<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>"};
-        data.ints = new int[] {-1234567,1234567,-1,0,1,Integer.MAX_VALUE,Integer.MIN_VALUE};
-        data.shorts = new short[] {-12345,12345,-1,0,1,Short.MAX_VALUE,Short.MIN_VALUE};
-        data.floats = new float[] {0,-0,1,-1,123456,-123456,0.1f,0.2f,-0.3f,(float) Math.PI,Float.MAX_VALUE,
-                Float.MIN_VALUE};
-
-        data.doubles = new double[] {0,-0,1,-1,123456,-123456,0.1d,0.2d,-0.3d,Math.PI,Double.MAX_VALUE,Double.MIN_VALUE};
-        data.longs = new long[] {0,-0,1,-1,123456,-123456,99999999999l,-99999999999l,Long.MAX_VALUE,Long.MIN_VALUE};
-        data.bytes = new byte[] {-123,123,-1,0,1,Byte.MAX_VALUE,Byte.MIN_VALUE};
-        data.chars = new char[] {32345,12345,0,1,63,Character.MAX_VALUE,Character.MIN_VALUE};
-
-        data.booleans = new boolean[] {true,false};
-        data.Ints = new Integer[] {-1234567,1234567,-1,0,1,Integer.MAX_VALUE,Integer.MIN_VALUE};
-        data.Shorts = new Short[] {-12345,12345,-1,0,1,Short.MAX_VALUE,Short.MIN_VALUE};
-        data.Floats = new Float[] {0f,-0f,1f,-1f,123456f,-123456f,0.1f,0.2f,-0.3f,(float) Math.PI,Float.MAX_VALUE,
-                Float.MIN_VALUE};
-        data.Doubles = new Double[] {0d,-0d,1d,-1d,123456d,-123456d,0.1d,0.2d,-0.3d,Math.PI,Double.MAX_VALUE,
-                Double.MIN_VALUE};
-        data.Longs = new Long[] {0l,-0l,1l,-1l,123456l,-123456l,99999999999l,-99999999999l,Long.MAX_VALUE,
-                Long.MIN_VALUE};
-        data.Bytes = new Byte[] {-123,123,-1,0,1,Byte.MAX_VALUE,Byte.MIN_VALUE};
-        data.Chars = new Character[] {32345,12345,0,1,63,Character.MAX_VALUE,Character.MIN_VALUE};
-        data.Booleans = new Boolean[] {true,false};
-    }
-
-    private void register(SerializationManager manager) {
-        manager.register(int[].class);
-        manager.register(short[].class);
-        manager.register(float[].class);
-        manager.register(double[].class);
-        manager.register(long[].class);
-        manager.register(byte[].class);
-        manager.register(char[].class);
-        manager.register(boolean[].class);
-        manager.register(String[].class);
-        manager.register(Integer[].class);
-        manager.register(Short[].class);
-        manager.register(Float[].class);
-        manager.register(Double[].class);
-        manager.register(Long[].class);
-        manager.register(Byte[].class);
-        manager.register(Character[].class);
-        manager.register(Boolean[].class);
-        manager.register(Data.class);
-    }
-
-    static public class Data {
-        public String      string;
-
-        public String[]    strings;
-
-        public int[]       ints;
-
-        public short[]     shorts;
-
-        public float[]     floats;
-
-        public double[]    doubles;
-
-        public long[]      longs;
-
-        public byte[]      bytes;
-
-        public char[]      chars;
-
-        public boolean[]   booleans;
-
-        public Integer[]   Ints;
-
-        public Short[]     Shorts;
-
-        public Float[]     Floats;
-
-        public Double[]    Doubles;
-
-        public Long[]      Longs;
-
-        public Byte[]      Bytes;
-
-        public Character[] Chars;
-
-        public Boolean[]   Booleans;
-
-        @Override
-        public int hashCode() {
-            final int prime = 31;
-            int result = 1;
-            result = prime * result + Arrays.hashCode(this.Booleans);
-            result = prime * result + Arrays.hashCode(this.Bytes);
-            result = prime * result + Arrays.hashCode(this.Chars);
-            result = prime * result + Arrays.hashCode(this.Doubles);
-            result = prime * result + Arrays.hashCode(this.Floats);
-            result = prime * result + Arrays.hashCode(this.Ints);
-            result = prime * result + Arrays.hashCode(this.Longs);
-            result = prime * result + Arrays.hashCode(this.Shorts);
-            result = prime * result + Arrays.hashCode(this.booleans);
-            result = prime * result + Arrays.hashCode(this.bytes);
-            result = prime * result + Arrays.hashCode(this.chars);
-            result = prime * result + Arrays.hashCode(this.doubles);
-            result = prime * result + Arrays.hashCode(this.floats);
-            result = prime * result + Arrays.hashCode(this.ints);
-            result = prime * result + Arrays.hashCode(this.longs);
-            result = prime * result + Arrays.hashCode(this.shorts);
-            result = prime * result + (this.string == null ? 0 : this.string.hashCode());
-            result = prime * result + Arrays.hashCode(this.strings);
-            return result;
-        }
-
-        @Override
-        public boolean equals(Object obj) {
-            if (this == obj) {
-                return true;
-            }
-            if (obj == null) {
-                return false;
-            }
-            if (getClass() != obj.getClass()) {
-                return false;
-            }
-            Data other = (Data) obj;
-            if (!Arrays.equals(this.Booleans, other.Booleans)) {
-                return false;
-            }
-            if (!Arrays.equals(this.Bytes, other.Bytes)) {
-                return false;
-            }
-            if (!Arrays.equals(this.Chars, other.Chars)) {
-                return false;
-            }
-            if (!Arrays.equals(this.Doubles, other.Doubles)) {
-                return false;
-            }
-            if (!Arrays.equals(this.Floats, other.Floats)) {
-                return false;
-            }
-            if (!Arrays.equals(this.Ints, other.Ints)) {
-                return false;
-            }
-            if (!Arrays.equals(this.Longs, other.Longs)) {
-                return false;
-            }
-            if (!Arrays.equals(this.Shorts, other.Shorts)) {
-                return false;
-            }
-            if (!Arrays.equals(this.booleans, other.booleans)) {
-                return false;
-            }
-            if (!Arrays.equals(this.bytes, other.bytes)) {
-                return false;
-            }
-            if (!Arrays.equals(this.chars, other.chars)) {
-                return false;
-            }
-            if (!Arrays.equals(this.doubles, other.doubles)) {
-                return false;
-            }
-            if (!Arrays.equals(this.floats, other.floats)) {
-                return false;
-            }
-            if (!Arrays.equals(this.ints, other.ints)) {
-                return false;
-            }
-            if (!Arrays.equals(this.longs, other.longs)) {
-                return false;
-            }
-            if (!Arrays.equals(this.shorts, other.shorts)) {
-                return false;
-            }
-            if (this.string == null) {
-                if (other.string != null) {
-                    return false;
-                }
-            } else if (!this.string.equals(other.string)) {
-                return false;
-            }
-            if (!Arrays.equals(this.strings, other.strings)) {
-                return false;
-            }
-            return true;
-        }
-
-        @Override
-        public String toString() {
-            return "Data";
-        }
-    }
-}

not-fixed/PooledSerialization.kt

@@ -1,213 +0,0 @@
-/*
- * Copyright 2014 dorkbox, llc
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package dorkbox.network.other
-
-import com.conversantmedia.util.concurrent.MultithreadConcurrentQueue
-import com.esotericsoftware.kryo.Kryo
-import com.esotericsoftware.kryo.Serializer
-import com.esotericsoftware.minlog.Log
-import dorkbox.network.serialization.ClassRegistration
-import dorkbox.network.serialization.ClassRegistration0
-import dorkbox.network.serialization.ClassRegistration1
-import dorkbox.network.serialization.ClassRegistration2
-import dorkbox.network.serialization.ClassRegistration3
-import dorkbox.network.serialization.KryoExtra
-import dorkbox.util.serialization.SerializationDefaults
-import kotlinx.atomicfu.atomic
-
-class PooledSerialization {
-    companion object {
-        init {
-            Log.set(Log.LEVEL_ERROR)
-        }
-    }
-
-    private var initialized = atomic(false)
-    private val classesToRegister = mutableListOf<ClassRegistration>()
-
-    private var kryoPoolSize = 16
-    private val kryoInUse = atomic(0)
-
-    @Volatile
-    private var kryoPool = MultithreadConcurrentQueue<KryoExtra>(kryoPoolSize)
-
-    /**
-     * If you customize anything, you will want to register custom types before init() is called!
-     */
-    fun init() {
-        // NOTE: there are problems if our serializer is THE SAME serializer used by the network stack!
-        //  We are explicitly differet types to prevent that form happening
-
-        initialized.value = true
-    }
-
-    private fun initKryo(): KryoExtra {
-        val kryo = KryoExtra()
-
-        SerializationDefaults.register(kryo)
-
-        classesToRegister.forEach { registration ->
-            registration.register(kryo)
-        }
-
-        return kryo
-    }
-
-
-    /**
-     * Registers the class using the lowest, next available integer ID and the [default serializer][Kryo.getDefaultSerializer].
-     * If the class is already registered, the existing entry is updated with the new serializer.
-     *
-     *
-     * Registering a primitive also affects the corresponding primitive wrapper.
-     *
-     * Because the ID assigned is affected by the IDs registered before it, the order classes are registered is important when using this
-     * method.
-     *
-     * The order must be the same at deserialization as it was for serialization.
-     *
-     * This must happen before the creation of the client/server
-     */
-    fun <T> register(clazz: Class<T>): PooledSerialization {
-        require(!initialized.value) { "Serialization 'register(class)' cannot happen after initialization!" }
-
-        // The reason it must be an implementation, is because the reflection serializer DOES NOT WORK with field types, but rather
-        // with object types... EVEN IF THERE IS A SERIALIZER
-        require(!clazz.isInterface) { "Cannot register '${clazz}' with specified ID for serialization. It must be an implementation." }
-
-        classesToRegister.add(ClassRegistration3(clazz))
-        return this
-    }
-
-    /**
-     * Registers the class using the specified ID. If the ID is already in use by the same type, the old entry is overwritten. If the ID
-     * is already in use by a different type, an exception is thrown.
-     *
-     *
-     * Registering a primitive also affects the corresponding primitive wrapper.
-     *
-     * IDs must be the same at deserialization as they were for serialization.
-     *
-     * This must happen before the creation of the client/server
-     *
-     * @param id Must be >= 0. Smaller IDs are serialized more efficiently. IDs 0-8 are used by default for primitive types and String, but
-     * these IDs can be repurposed.
-     */
-    fun <T> register(clazz: Class<T>, id: Int): PooledSerialization {
-        require(!initialized.value) { "Serialization 'register(Class, int)' cannot happen after initialization!" }
-
-        // The reason it must be an implementation, is because the reflection serializer DOES NOT WORK with field types, but rather
-        // with object types... EVEN IF THERE IS A SERIALIZER
-        require(!clazz.isInterface) { "Cannot register '${clazz}' with specified ID for serialization. It must be an implementation." }
-
-        classesToRegister.add(ClassRegistration1(clazz, id))
-        return this
-    }
-
-    /**
-     * Registers the class using the lowest, next available integer ID and the specified serializer. If the class is already registered,
-     * the existing entry is updated with the new serializer.
-     *
-     *
-     * Registering a primitive also affects the corresponding primitive wrapper.
-     *
-     *
-     * Because the ID assigned is affected by the IDs registered before it, the order classes are registered is important when using this
-     * method. The order must be the same at deserialization as it was for serialization.
-     */
-    @Synchronized
-    fun <T> register(clazz: Class<T>, serializer: Serializer<T>): PooledSerialization {
-        require(!initialized.value) { "Serialization 'register(Class, Serializer)' cannot happen after initialization!" }
-
-        // The reason it must be an implementation, is because the reflection serializer DOES NOT WORK with field types, but rather
-        // with object types... EVEN IF THERE IS A SERIALIZER
-        require(!clazz.isInterface) { "Cannot register '${clazz.name}' with a serializer. It must be an implementation." }
-
-        classesToRegister.add(ClassRegistration0(clazz, serializer))
-        return this
-    }
-
-    /**
-     * Registers the class using the specified ID and serializer. If the ID is already in use by the same type, the old entry is
-     * overwritten. If the ID is already in use by a different type, an exception is thrown.
-     *
-     *
-     * Registering a primitive also affects the corresponding primitive wrapper.
-     *
-     *
-     * IDs must be the same at deserialization as they were for serialization.
-     *
-     * @param id Must be >= 0. Smaller IDs are serialized more efficiently. IDs 0-8 are used by default for primitive types and String, but
-     * these IDs can be repurposed.
-     */
-    @Synchronized
-    fun <T> register(clazz: Class<T>, serializer: Serializer<T>, id: Int): PooledSerialization {
-        require(!initialized.value) { "Serialization 'register(Class, Serializer, int)' cannot happen after initialization!" }
-
-        // The reason it must be an implementation, is because the reflection serializer DOES NOT WORK with field types, but rather
-        // with object types... EVEN IF THERE IS A SERIALIZER
-        require(!clazz.isInterface) { "Cannot register '${clazz.name}'. It must be an implementation." }
-
-        classesToRegister.add(ClassRegistration2(clazz, serializer, id))
-        return this
-    }
-
-    /**
-     * @return takes a kryo instance from the pool, or creates one if the pool was empty
-     */
-    fun takeKryo(): KryoExtra {
-        kryoInUse.getAndIncrement()
-
-        // ALWAYS get as many as needed. We recycle them (with an auto-growing pool) to prevent too many getting created
-        return kryoPool.poll() ?: initKryo()
-    }
-
-    /**
-     * Returns a kryo instance to the pool for re-use later on
-     */
-    fun returnKryo(kryo: KryoExtra) {
-        val kryoCount = kryoInUse.getAndDecrement()
-        if (kryoCount > kryoPoolSize) {
-            // this is CLEARLY a problem, as we have more kryos in use that our pool can support.
-            // This happens when we send messages REALLY fast.
-            //
-            // We fix this by increasing the size of the pool, so kryos aren't thrown away (and create a GC hit)
-
-            synchronized(kryoInUse) {
-                // we have a double check here on purpose. only 1 will work
-                if (kryoCount > kryoPoolSize) {
-                    val oldPool = kryoPool
-                    val oldSize = kryoPoolSize
-                    val newSize = kryoPoolSize * 2
-
-                    kryoPoolSize = newSize
-                    kryoPool = MultithreadConcurrentQueue<KryoExtra>(kryoPoolSize)
-
-
-                    // take all of the old kryos and put them in the new one
-                    val array = arrayOfNulls<KryoExtra>(oldSize)
-                    val count = oldPool.remove(array)
-
-                    for (i in 0 until count) {
-                        kryoPool.offer(array[i])
-                    }
-                }
-            }
-        }
-
-        kryoPool.offer(kryo)
-    }
-}

not-fixed/ReconnectTest.java

@@ -1,296 +0,0 @@
-/* Copyright (c) 2008, Nathan Sweet
- * All rights reserved.
- *
- * Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
- * - Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials provided with the distribution.
- * - Neither the name of Esoteric Software nor the names of its contributors may be used to endorse or promote products derived
- * from this software without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING,
- * BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT
- * SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
- * NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
- */
-
-import static org.junit.Assert.assertEquals;
-
-import java.io.IOException;
-import java.util.concurrent.CountDownLatch;
-import java.util.concurrent.TimeUnit;
-import java.util.concurrent.atomic.AtomicInteger;
-import java.util.concurrent.atomic.AtomicReference;
-
-import org.junit.Test;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import dorkbox.network.connection.Connection;
-import dorkbox.network.connection.Listener;
-import dorkbox.network.connection.Listeners;
-import dorkbox.util.exceptions.SecurityException;
-
-// NOTE: UDP is unreliable, EVEN ON LOOPBACK! So this can fail with UDP. TCP will never fail.
-public
-class ReconnectTest extends BaseTest {
-    private final AtomicInteger receivedCount = new AtomicInteger(0);
-
-    private static final Logger logger = LoggerFactory.getLogger(ReconnectTest.class.getSimpleName());
-
-    @Test
-    public
-    void socketReuseUDP() throws IOException, SecurityException {
-        socketReuse(false, true);
-    }
-
-    @Test
-    public
-    void socketReuseTCP() throws IOException, SecurityException {
-        socketReuse(true, false);
-    }
-
-    @Test
-    public
-    void socketReuseTCPUDP() throws IOException, SecurityException {
-        socketReuse(true, true);
-    }
-
-    private
-    void socketReuse(final boolean useTCP, final boolean useUDP) throws SecurityException, IOException {
-        receivedCount.set(0);
-
-        Configuration configuration = new Configuration();
-        configuration.host = host;
-
-        if (useTCP) {
-            configuration.tcpPort = tcpPort;
-        }
-
-        if (useUDP) {
-            configuration.udpPort = udpPort;
-        }
-
-        AtomicReference<CountDownLatch> latch = new AtomicReference<CountDownLatch>();
-
-
-        Server server = new Server(configuration);
-        addEndPoint(server);
-        final Listeners listeners = server.listeners();
-        listeners.add(new Listener.OnConnected<Connection>() {
-            @Override
-            public
-            void connected(Connection connection) {
-                if (useTCP) {
-                    connection.send()
-                              .TCP("-- TCP from server");
-                }
-                if (useUDP) {
-                    connection.send()
-                              .UDP("-- UDP from server");
-                }
-            }
-        });
-        listeners.add(new Listener.OnMessageReceived<Connection, String>() {
-            @Override
-            public
-            void received(Connection connection, String object) {
-                int incrementAndGet = ReconnectTest.this.receivedCount.incrementAndGet();
-                logger.error("----- <S " + connection + "> " + incrementAndGet + " : " + object);
-
-                latch.get().countDown();
-            }
-        });
-        server.bind(false);
-
-
-        // ----
-
-        Client client = new Client(configuration);
-        addEndPoint(client);
-        final Listeners listeners1 = client.listeners();
-        listeners1.add(new Listener.OnConnected<Connection>() {
-            @Override
-            public
-            void connected(Connection connection) {
-                if (useTCP) {
-                    connection.send()
-                              .TCP("-- TCP from client");
-                }
-                if (useUDP) {
-                    connection.send()
-                              .UDP("-- UDP from client");
-                }
-            }
-        });
-        listeners1.add(new Listener.OnMessageReceived<Connection, String>() {
-            @Override
-            public
-            void received(Connection connection, String object) {
-                int incrementAndGet = ReconnectTest.this.receivedCount.incrementAndGet();
-                logger.error("----- <C " + connection + "> " + incrementAndGet + " : " + object);
-
-                latch.get().countDown();
-            }
-        });
-
-
-
-        int latchCount = 2;
-        int count = 100;
-        int initialCount = 2;
-        if (useTCP && useUDP) {
-            initialCount += 2;
-            latchCount += 2;
-        }
-
-
-        try {
-            for (int i = 1; i < count + 1; i++) {
-                logger.error(".....");
-                latch.set(new CountDownLatch(latchCount));
-
-                try {
-                    client.connect(5000);
-                } catch (IOException e) {
-                    e.printStackTrace();
-                }
-
-                int retryCount = 20;
-                int lastRetryCount;
-
-                int target = i * initialCount;
-                boolean failed = false;
-
-                synchronized (receivedCount) {
-                    while (this.receivedCount.get() != target) {
-                        lastRetryCount = this.receivedCount.get();
-
-                        try {
-                            latch.get().await(1, TimeUnit.SECONDS);
-                        } catch (InterruptedException e) {
-                            e.printStackTrace();
-                        }
-
-                        // check to see if we changed at all...
-                        if (lastRetryCount == this.receivedCount.get()) {
-                            if (retryCount-- < 0) {
-                                logger.error("Aborting unit test... wrong count!");
-                                if (useUDP) {
-                                    // If TCP and UDP both fill the pipe, THERE WILL BE FRAGMENTATION and dropped UDP packets!
-                                    // it results in severe UDP packet loss and contention.
-                                    //
-                                    // http://www.isoc.org/INET97/proceedings/F3/F3_1.HTM
-                                    // also, a google search on just "INET97/proceedings/F3/F3_1.HTM" turns up interesting problems.
-                                    // Usually it's with ISPs.
-
-                                    logger.error("NOTE: UDP can fail, even on loopback! See: http://www.isoc.org/INET97/proceedings/F3/F3_1.HTM");
-                                }
-                                failed = true;
-                                break;
-                            }
-                        } else {
-                            retryCount = 20;
-                        }
-                    }
-                }
-
-                client.close();
-                logger.error(".....");
-
-                if (failed) {
-                    break;
-                }
-            }
-
-            int specified = count * initialCount;
-            int received = this.receivedCount.get();
-
-            if (specified != received) {
-                logger.error("NOTE: UDP can fail, even on loopback! See: http://www.isoc.org/INET97/proceedings/F3/F3_1.HTM");
-            }
-
-            assertEquals(specified, received);
-        } finally {
-            stopEndPoints();
-            waitForThreads(10);
-        }
-    }
-
-    @Test
-    public
-    void localReuse() throws SecurityException, IOException {
-        receivedCount.set(0);
-
-        Server server = new Server();
-        addEndPoint(server);
-        server.listeners()
-              .add(new Listener.OnConnected<Connection>() {
-                  @Override
-                  public
-                  void connected(Connection connection) {
-                      connection.send()
-                                .self("-- LOCAL from server");
-                  }
-              });
-        server.listeners()
-              .add(new Listener.OnMessageReceived<Connection, String>() {
-                  @Override
-                  public
-                  void received(Connection connection, String object) {
-                      int incrementAndGet = ReconnectTest.this.receivedCount.incrementAndGet();
-                      System.out.println("----- <S " + connection + "> " + incrementAndGet + " : " + object);
-                  }
-              });
-
-        // ----
-
-        Client client = new Client();
-        addEndPoint(client);
-        client.listeners()
-              .add(new Listener.OnConnected<Connection>() {
-                  @Override
-                  public
-                  void connected(Connection connection) {
-                      connection.send()
-                                .self("-- LOCAL from client");
-                  }
-              });
-
-        client.listeners()
-              .add(new Listener.OnMessageReceived<Connection, String>() {
-                  @Override
-                  public
-                  void received(Connection connection, String object) {
-                      int incrementAndGet = ReconnectTest.this.receivedCount.incrementAndGet();
-                      System.out.println("----- <C " + connection + "> " + incrementAndGet + " : " + object);
-                  }
-              });
-
-        server.bind(false);
-        int count = 10;
-        for (int i = 1; i < count + 1; i++) {
-            client.connect(5000);
-
-            int target = i * 2;
-            while (this.receivedCount.get() != target) {
-                System.out.println("----- Waiting...");
-                try {
-                    Thread.sleep(100);
-                } catch (InterruptedException ignored) {
-                }
-            }
-
-            client.close();
-        }
-
-        assertEquals(count * 2, this.receivedCount.get());
-
-        stopEndPoints();
-        waitForThreads(10);
-    }
-}

settings.gradle.kts

@@ -1,5 +1,5 @@
 /*
- * Copyright 2018 dorkbox, llc
+ * Copyright 2023 dorkbox, llc
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -13,3 +13,4 @@
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
+rootProject.name = "Network"

src/dorkbox/network/Server.kt

@@ -1,5 +1,5 @@
 /*
- * Copyright 2020 dorkbox, llc
+ * Copyright 2024 dorkbox, llc
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -15,25 +15,18 @@
  */
 package dorkbox.network
 
-import dorkbox.netUtil.IPv4
-import dorkbox.netUtil.IPv6
-import dorkbox.netUtil.Inet4
-import dorkbox.netUtil.Inet6
-import dorkbox.network.aeron.AeronDriver
-import dorkbox.network.aeron.AeronPoller
-import dorkbox.network.connection.Connection
-import dorkbox.network.connection.ConnectionParams
-import dorkbox.network.connection.EndPoint
-import dorkbox.network.connectionType.ConnectionRule
-import dorkbox.network.exceptions.AllocationException
+import dorkbox.hex.toHexString
+import dorkbox.network.aeron.*
+import dorkbox.network.connection.*
+import dorkbox.network.connection.IpInfo.Companion.IpListenType
+import dorkbox.network.connection.ListenerManager.Companion.cleanStackTrace
+import dorkbox.network.connection.buffer.BufferManager
 import dorkbox.network.exceptions.ServerException
 import dorkbox.network.handshake.ServerHandshake
 import dorkbox.network.handshake.ServerHandshakePollers
+import dorkbox.network.ipFilter.IpFilterRule
 import dorkbox.network.rmi.RmiSupportServer
-import kotlinx.atomicfu.atomic
-import kotlinx.coroutines.Job
-import kotlinx.coroutines.launch
-import kotlinx.coroutines.runBlocking
+import org.slf4j.LoggerFactory
 import java.net.InetAddress
 import java.util.concurrent.*
 
@@ -47,90 +40,41 @@ import java.util.concurrent.*
  * @param connectionFunc allows for custom connection implementations defined as a unit function
  * @param loggerName allows for a custom logger name for this endpoint (for when there are multiple endpoints)
  */
-open class Server<CONNECTION : Connection>(
-        config: ServerConfiguration = ServerConfiguration(),
-        connectionFunc: (connectionParameters: ConnectionParams<CONNECTION>) -> CONNECTION,
-        loggerName: String = Server::class.java.simpleName)
-    : EndPoint<CONNECTION>(config, connectionFunc, loggerName) {
-
-    /**
-     * The server can only be accessed in an ASYNC manner. This means that the server can only be used in RESPONSE to events. If you access the
-     * server OUTSIDE of events, you will get inaccurate information from the server (such as getConnections())
-     *
-     * To put it bluntly, ONLY have the server do work inside a listener!
-     *
-     * @param config these are the specific connection options
-     * @param loggerName allows for a custom logger name for this endpoint (for when there are multiple endpoints)
-     * @param connectionFunc allows for custom connection implementations defined as a unit function
-     */
-    constructor(config: ServerConfiguration,
-                loggerName: String,
-                connectionFunc: (connectionParameters: ConnectionParams<CONNECTION>) -> CONNECTION)
-            : this(config, connectionFunc, loggerName)
-
-
-    /**
-     * The server can only be accessed in an ASYNC manner. This means that the server can only be used in RESPONSE to events. If you access the
-     * server OUTSIDE of events, you will get inaccurate information from the server (such as getConnections())
-     *
-     * To put it bluntly, ONLY have the server do work inside of a listener!
-     *
-     * @param config these are the specific connection options
-     * @param connectionFunc allows for custom connection implementations defined as a unit function
-     */
-    constructor(config: ServerConfiguration,
-                connectionFunc: (connectionParameters: ConnectionParams<CONNECTION>) -> CONNECTION)
-            : this(config, connectionFunc, Server::class.java.simpleName)
-
-
-    /**
-     * The server can only be accessed in an ASYNC manner. This means that the server can only be used in RESPONSE to events. If you access the
-     * server OUTSIDE of events, you will get inaccurate information from the server (such as getConnections())
-     *
-     * To put it bluntly, ONLY have the server do work inside of a listener!
-     *
-     * @param config these are the specific connection options
-     * @param loggerName allows for a custom logger name for this endpoint (for when there are multiple endpoints)
-     */
-    constructor(config: ServerConfiguration,
-                loggerName: String = Server::class.java.simpleName)
-            : this(config,
-                   {
-                        @Suppress("UNCHECKED_CAST")
-                        Connection(it) as CONNECTION
-                   },
-                   loggerName)
-
-    /**
-     * The server can only be accessed in an ASYNC manner. This means that the server can only be used in RESPONSE to events. If you access the
-     * server OUTSIDE of events, you will get inaccurate information from the server (such as getConnections())
-     *
-     * To put it bluntly, ONLY have the server do work inside of a listener!
-     *
-     * @param config these are the specific connection options
-     */
-    constructor(config: ServerConfiguration)
-            : this(config,
-                   {
-                        @Suppress("UNCHECKED_CAST")
-                        Connection(it) as CONNECTION
-                   },
-                   Server::class.java.simpleName)
-
+open class Server<CONNECTION : Connection>(config: ServerConfiguration = ServerConfiguration(), loggerName: String = Server::class.java.simpleName)
+    : EndPoint<CONNECTION>(config, loggerName) {
 
     companion object {
         /**
          * Gets the version number.
          */
-        const val version = "5.32"
+        const val version = Configuration.version
+
+        /**
+         * Ensures that an endpoint (using the specified configuration) is NO LONGER running.
+         *
+         * NOTE: This method should only be used to check if a server is running for a DIFFERENT configuration than the currently running server
+         *
+         * By default, we will wait the [Configuration.connectionCloseTimeoutInSeconds] * 2 amount of time before returning.
+         *
+         * @return true if the media driver is STOPPED.
+         */
+        fun ensureStopped(configuration: ServerConfiguration): Boolean {
+            val timeout = TimeUnit.SECONDS.toMillis(configuration.connectionCloseTimeoutInSeconds.toLong() * 2)
+
+            val logger = LoggerFactory.getLogger(Server::class.java.simpleName)
+            return AeronDriver.ensureStopped(configuration.copy(), logger, timeout)
+        }
 
         /**
          * Checks to see if a server (using the specified configuration) is running.
          *
-         * This method should only be used to check if a server is running for a DIFFERENT configuration than the currently running server
+         * NOTE: This method should only be used to check if a server is running for a DIFFERENT configuration than the currently running server
+         *
+         * @return true if the media driver is active and running
          */
         fun isRunning(configuration: ServerConfiguration): Boolean {
-            return AeronDriver(configuration).isRunning()
+            val logger = LoggerFactory.getLogger(Server::class.java.simpleName)
+            return AeronDriver.isRunning(configuration.copy(), logger)
         }
 
         init {
@@ -144,261 +88,310 @@ open class Server<CONNECTION : Connection>(
      */
     val rmiGlobal = RmiSupportServer(logger, rmiGlobalSupport)
 
-    /**
-     * @return true if this server has successfully bound to an IP address and is running
-     */
-    private var bindAlreadyCalled = atomic(false)
+//    /**
+//     * Maintains a thread-safe collection of rules used to define the connection type with this server.
+//     */
+//    private val connectionRules = CopyOnWriteArrayList<ConnectionRule>()
 
     /**
-     * These are run in lock-step to shutdown/close the server. Afterwards, bind() can be called again
+     * the IP address information, if available.
      */
-    @Volatile
-    private var shutdownPollLatch = CountDownLatch(1)
+    internal val ipInfo = IpInfo(config)
 
     @Volatile
-    private var shutdownEventLatch = CountDownLatch(1)
+    internal lateinit var handshake: ServerHandshake<CONNECTION>
 
     /**
-     * Maintains a thread-safe collection of rules used to define the connection type with this server.
+     * Different connections (to the same client) can be "buffered", meaning that if they "go down" because of a network glitch -- the data
+     * being sent is not lost (it is buffered) and then re-sent once the new connection is established. References to the old connection
+     * will also redirect to the new connection.
      */
-    private val connectionRules = CopyOnWriteArrayList<ConnectionRule>()
+    internal val bufferedManager: BufferManager<CONNECTION>
 
-    /**
-     * true if the following network stacks are available for use
-     */
-    internal val canUseIPv4 = config.enableIPv4 && IPv4.isAvailable
-    internal val canUseIPv6 = config.enableIPv6 && IPv6.isAvailable
-
-
-    // localhost/loopback IP might not always be 127.0.0.1 or ::1
-    // We want to listen on BOTH IPv4 and IPv6 (config option lets us configure this)
-    internal val listenIPv4Address: InetAddress? =
-        if (canUseIPv4) {
-            when (config.listenIpAddress) {
-                "loopback", "localhost", "lo", "127.0.0.1", "::1" -> IPv4.LOCALHOST
-                "0", "::", "0.0.0.0", "*" -> {
-                    // this is the "wildcard" address. Windows has problems with this.
-                    IPv4.WILDCARD
-                }
-                else -> Inet4.toAddress(config.listenIpAddress) // Inet4Address.getAllByName(config.listenIpAddress)[0]
-            }
-        }
-        else {
-            null
-        }
-
-
-    internal val listenIPv6Address: InetAddress? =
-        if (canUseIPv6) {
-            when (config.listenIpAddress) {
-                "loopback", "localhost", "lo", "127.0.0.1", "::1" -> IPv6.LOCALHOST
-                "0", "::", "0.0.0.0", "*" -> {
-                    // this is the "wildcard" address. Windows has problems with this.
-                    IPv6.WILDCARD
-                }
-                else -> Inet6.toAddress(config.listenIpAddress)
-            }
-        }
-        else {
-            null
-        }
+    private val string0: String by lazy {
+        "EndPoint [Server: ${storage.publicKey.toHexString()}]"
+    }
 
     init {
-        // we are done with initial configuration, now finish serialization
-        serialization.finishInit(type)
+        bufferedManager = BufferManager(config, listenerManager, aeronDriver, config.bufferedConnectionTimeoutSeconds)
     }
 
     final override fun newException(message: String, cause: Throwable?): Throwable {
-        return ServerException(message, cause)
+        // +2 because we do not want to see the stack for the abstract `newException`
+        val serverException = ServerException(message, cause)
+        serverException.cleanStackTrace(2)
+        return serverException
     }
 
     /**
-     * Binds the server to AERON configuration
+     * Binds the server IPC only, using the previously set AERON configuration
+     */
+    fun bindIpc() {
+        if (!config.enableIpc) {
+            logger.warn("IPC explicitly requested, but not enabled. Enabling IPC...")
+            // we explicitly requested IPC, make sure it's enabled
+            config.contextDefined = false
+            config.enableIpc = true
+            config.contextDefined = true
+        }
+
+        if (config.enableIPv4) { logger.warn("IPv4 is enabled, but only IPC will be used.") }
+        if (config.enableIPv6) { logger.warn("IPv6 is enabled, but only IPC will be used.") }
+
+        internalBind(port1 = 0, port2 = 0, onlyBindIpc = true, runShutdownCheck = true)
+    }
+
+    /**
+     * Binds the server to UDP ports, using the previously set AERON configuration
+     *
+     * @param port1 this is the network port which will be listening for incoming connections
+     * @param port2 this is the network port that the server will use to work around NAT firewalls. By default, this is port1+1, but
+     *              can also be configured independently. This is required, and must be different from port1.
      */
     @Suppress("DuplicatedCode")
-    fun bind() {
-        // NOTE: it is critical to remember that Aeron DOES NOT like running from coroutines!
+    fun bind(port1: Int, port2: Int = port1+1) {
+        if (config.enableIPv4 || config.enableIPv6) {
+            require(port1 != port2) { "port1 cannot be the same as port2" }
+            require(port1 > 0) { "port1 must be > 0" }
+            require(port2 > 0) { "port2 must be > 0" }
+            require(port1 < 65535) { "port1 must be < 65535" }
+            require(port2 < 65535) { "port2 must be < 65535" }
+        }
 
-        if (bindAlreadyCalled.getAndSet(true)) {
-            logger.error { "Unable to bind when the server is already running!" }
+        internalBind(port1 = port1, port2 = port2, onlyBindIpc = false, runShutdownCheck = true)
+    }
+
+    @Suppress("DuplicatedCode")
+    private fun internalBind(port1: Int, port2: Int, onlyBindIpc: Boolean, runShutdownCheck: Boolean) {
+        // the lifecycle of a server is the ENDPOINT (measured via the network event poller)
+        if (endpointIsRunning.value) {
+            listenerManager.notifyError(ServerException("Unable to start, the server is already running!"))
+            return
+        }
+
+        if (runShutdownCheck && !waitForEndpointShutdown()) {
+            listenerManager.notifyError(ServerException("Unable to start the server!"))
             return
         }
 
         try {
             startDriver()
-        } catch (e: Exception) {
-            logger.error(e) { "Unable to start the network driver" }
+            initializeState()
+        }
+        catch (e: Exception) {
+            resetOnError()
+            listenerManager.notifyError(ServerException("Unable to start the server!", e))
             return
         }
 
-        shutdownPollLatch = CountDownLatch(1)
-        shutdownEventLatch = CountDownLatch(1)
+        this@Server.port1 = port1
+        this@Server.port2 = port2
 
         config as ServerConfiguration
 
-        val handshake = ServerHandshake(logger, config, listenerManager, aeronDriver)
-
         // we are done with initial configuration, now initialize aeron and the general state of this endpoint
 
-        // this forces the current thread to WAIT until poll system has started
-        val pollStartupLatch = CountDownLatch(1)
-
         val server = this@Server
-        val ipcPoller: AeronPoller = ServerHandshakePollers.ipc(aeronDriver, config, server, handshake)
+        handshake = ServerHandshake(config, listenerManager, aeronDriver, eventDispatch)
 
-        // if we are binding to WILDCARD, then we have to do something special if BOTH IPv4 and IPv6 are enabled!
-        val isWildcard = listenIPv4Address == IPv4.WILDCARD || listenIPv6Address == IPv6.WILDCARD
-        val ipv4Poller: AeronPoller
-        val ipv6Poller: AeronPoller
-
-        if (isWildcard) {
-            if (canUseIPv4 && canUseIPv6) {
-                // IPv6 will bind to IPv4 wildcard as well, so don't bind both!
-                ipv4Poller = ServerHandshakePollers.disabled("IPv4 Disabled")
-                ipv6Poller = ServerHandshakePollers.ip6Wildcard(aeronDriver, config, server, handshake)
-            } else {
-                // only 1 will be a real poller
-                ipv4Poller = ServerHandshakePollers.ip4(aeronDriver, config, server, handshake)
-                ipv6Poller = ServerHandshakePollers.ip6(aeronDriver, config, server, handshake)
-            }
+        val ipcPoller: AeronPoller = if (config.enableIpc || onlyBindIpc) {
+            ServerHandshakePollers.ipc(server, handshake)
         } else {
-            ipv4Poller = ServerHandshakePollers.ip4(aeronDriver, config, server, handshake)
-            ipv6Poller = ServerHandshakePollers.ip6(aeronDriver, config, server, handshake)
+            ServerHandshakePollers.disabled("IPC Disabled")
         }
 
 
-        val networkEventProcessor = Runnable {
-            pollStartupLatch.countDown()
+        val ipPoller = if (onlyBindIpc) {
+            ServerHandshakePollers.disabled("IPv4/6 Disabled")
+        } else {
+            when (ipInfo.ipType) {
+                // IPv6 will bind to IPv4 wildcard as well, so don't bind both!
+                IpListenType.IPWildcard   -> ServerHandshakePollers.ip6Wildcard(server, handshake)
+                IpListenType.IPv4Wildcard -> ServerHandshakePollers.ip4(server, handshake)
+                IpListenType.IPv6Wildcard -> ServerHandshakePollers.ip6(server, handshake)
+                IpListenType.IPv4 -> ServerHandshakePollers.ip4(server, handshake)
+                IpListenType.IPv6 -> ServerHandshakePollers.ip6(server, handshake)
+                IpListenType.IPC  -> ServerHandshakePollers.disabled("IPv4/6 Disabled")
+            }
+        }
 
-            val pollIdleStrategy = config.pollIdleStrategy.cloneToNormal()
-            try {
-                var pollCount: Int
 
-                while (!isShutdown()) {
-                    pollCount = 0
+        logger.info(ipcPoller.info)
+        logger.info(ipPoller.info)
 
+        // if we shutdown/close before the poller starts, we don't want to block forever
+        pollerClosedLatch = CountDownLatch(1)
+        networkEventPoller.submit(
+        action = object : EventActionOperator {
+            override fun invoke(): Int {
+                return if (!shutdownEventPoller) {
                     // NOTE: regarding fragment limit size. Repeated calls to '.poll' will reassemble a fragment.
                     //   `.poll(handler, 4)` == `.poll(handler, 2)` + `.poll(handler, 2)`
 
-                    // this checks to see if there are NEW clients on the handshake ports
-                    pollCount += ipv4Poller.poll()
-                    pollCount += ipv6Poller.poll()
-
-                    // this checks to see if there are NEW clients via IPC
-                    pollCount += ipcPoller.poll()
+                    // this checks to see if there are NEW clients to handshake with
+                    var pollCount = ipcPoller.poll() + ipPoller.poll()
 
                     // this manages existing clients (for cleanup + connection polling). This has a concurrent iterator,
                     // so we can modify this as we go
                     connections.forEach { connection ->
-                        if (!connection.isClosedViaAeron()) {
+                        if (connection.canPoll()) {
                             // Otherwise, poll the connection for messages
                             pollCount += connection.poll()
                         } else {
                             // If the connection has either been closed, or has expired, it needs to be cleaned-up/deleted.
-                            logger.debug { "[${connection.id}/${connection.streamId}] connection expired" }
+                            if (logger.isDebugEnabled) {
+                                logger.debug("[${connection}] connection expired (cleanup)")
+                            }
 
+                            // the connection MUST be removed in the same thread that is processing events (it will be removed again in close, and that is expected)
                             removeConnection(connection)
 
-                            // this will call removeConnection again, but that is ok
-                            // this is blocking, because the connection MUST be removed in the same thread that is processing events
+                            // we already removed the connection, we can call it again without side effects
                             connection.close()
-
-                            // have to manually notify the server-listenerManager that this connection was closed
-                            // if the connection was MANUALLY closed (via calling connection.close()), then the connection-listenermanager is
-                            // instantly notified and on cleanup, the server-listenermanager is called
-
-                            // this always has to be on event dispatch, otherwise we can have weird logic loops if we reconnect within a disconnect callback
-                            actionDispatch.launch {
-                                listenerManager.notifyDisconnect(connection)
-                            }
                         }
                     }
 
-                    // 0 means we idle. >0 means reset and don't idle (because there are likely more poll events)
-                    pollIdleStrategy.idle(pollCount)
+                    pollCount
+                } else {
+                    // remove ourselves from processing
+                    EventPoller.REMOVE
                 }
+            }
+        },
+        onClose = object : EventCloseOperator {
+            override fun invoke() {
+                val mustRestartDriverOnError = aeronDriver.internal.mustRestartDriverOnError
+                logger.debug("Server event dispatch closing...")
 
-                logger.debug { "Network event dispatch closing..." }
-
-                // we want to process **actual** close cleanup events on this thread as well, otherwise we will have threading problems
-                shutdownPollLatch.await()
-
-                // we have to manually cleanup the connections and call server-notifyDisconnect because otherwise this will never get called
-                val jobs = mutableListOf<Job>()
-
-                // we want to clear all the connections FIRST (since we are shutting down)
-                val cons = mutableListOf<CONNECTION>()
-                connections.forEach { cons.add(it) }
-                connections.clear()
-
-                cons.forEach { connection ->
-                    logger.info { "[${connection.id}/${connection.streamId}] Connection cleanup and close" }
-
-                    // make sure the connection is closed (close can only happen once, so a duplicate call does nothing!)
-                    connection.close()
-
-                    // have to manually notify the server-listenerManager that this connection was closed
-                    // if the connection was MANUALLY closed (via calling connection.close()), then the connection-listenermanager is
-                    // instantly notified and on cleanup, the server-listenermanager is called
-                    // NOTE: this must be the LAST thing happening!
-
-                    // this always has to be on event dispatch, otherwise we can have weird logic loops if we reconnect within a disconnect callback
-                    val job = actionDispatch.launch {
-                        listenerManager.notifyDisconnect(connection)
-                    }
-                    jobs.add(job)
-                }
-
-                // when we close a client or a server, we want to make sure that ALL notifications are finished.
-                // when it's just a connection getting closed, we don't care about this. We only care when it's "global" shutdown
-                runBlocking {
-                    jobs.forEach { it.join() }
-                }
-            } catch (e: Exception) {
-                logger.error(e) { "Unexpected error during server message polling!" }
-            } finally {
-                ipv4Poller.close()
-                ipv6Poller.close()
                 ipcPoller.close()
+                ipPoller.close()
 
                 // clear all the handshake info
                 handshake.clear()
 
-                try {
-                    // make sure that we have de-allocated all connection data
-                    handshake.checkForMemoryLeaks()
-                } catch (e: AllocationException) {
-                    logger.error(e) { "Error during server cleanup" }
+
+                // we only need to run shutdown methods if there was a network outage or D/C
+                if (!shutdownInProgress.value) {
+                    // this is because we restart automatically on driver errors
+                    this@Server.close(closeEverything = false, sendDisconnectMessage = true, releaseWaitingThreads = !mustRestartDriverOnError)
                 }
 
-                // finish closing -- this lets us make sure that we don't run into race conditions on the thread that calls close()
-                try {
-                    shutdownEventLatch.countDown()
-                } catch (ignored: Exception) {}
-            }
-        }
-        config.networkInterfaceEventDispatcher.submit(networkEventProcessor)
 
-        // wait for the polling thread to startup before letting bind() return
-        pollStartupLatch.await()
+                if (mustRestartDriverOnError) {
+                    logger.error("Critical driver error detected, restarting server.")
+
+                    eventDispatch.CLOSE.launch {
+                        waitForEndpointShutdown()
+
+                        // also wait for everyone else to shutdown!!
+                        aeronDriver.internal.endPointUsages.forEach {
+                            if (it !== this@Server) {
+                                it.waitForEndpointShutdown()
+                            }
+                        }
+
+
+                        // if we restart/reconnect too fast, errors from the previous run will still be present!
+                        aeronDriver.delayLingerTimeout()
+
+                        val p1 = this@Server.port1
+                        val p2 = this@Server.port2
+
+                        if (p1 == 0 && p2 == 0) {
+                            internalBind(port1 = 0, port2 = 0, onlyBindIpc = true, runShutdownCheck = false)
+                        } else {
+                            internalBind(port1 = p1, port2 = p2, onlyBindIpc = false, runShutdownCheck = false)
+                        }
+                    }
+                }
+
+                // we can now call bind again
+                endpointIsRunning.lazySet(false)
+                logger.debug("Closed the Network Event Poller task.")
+                pollerClosedLatch.countDown()
+            }
+        })
+    }
+
+
+//    /**
+//     * Adds an IP+subnet rule that defines what type of connection this IP+subnet should have.
+//     * - NOTHING : Nothing happens to the in/out bytes
+//     * - COMPRESS: The in/out bytes are compressed with LZ4-fast
+//     * - COMPRESS_AND_ENCRYPT: The in/out bytes are compressed (LZ4-fast) THEN encrypted (AES-256-GCM)
+//     *
+//     * If no rules are defined, then for LOOPBACK, it will always be `COMPRESS` and for everything else it will always be `COMPRESS_AND_ENCRYPT`.
+//     *
+//     * If rules are defined, then everything by default is `COMPRESS_AND_ENCRYPT`.
+//     *
+//     * The compression algorithm is LZ4-fast, so there is a small performance impact for a very large gain
+//     * Compress   :       6.210 micros/op;  629.0 MB/s (output: 55.4%)
+//     * Uncompress :       0.641 micros/op; 6097.9 MB/s
+//     */
+//    fun addConnectionRules(vararg rules: ConnectionRule) {
+//        connectionRules.addAll(listOf(*rules))
+//    }
+
+    /**
+     * Adds an IP+subnet rule that defines if that IP+subnet is allowed/denied connectivity to this server.
+     *
+     * By default, if there are no filter rules, then all connections are allowed to connect
+     * If there are filter rules - then ONLY connections for the filter that returns true are allowed to connect (all else are denied)
+     *
+     * If ANY filter rule that is applied returns true, then the connection is permitted
+     *
+     * This function will be called for **only** network clients (IPC client are excluded)
+     *
+     * @param ipFilterRule the IpFilterRule to determine if this connection will be allowed to connect
+     */
+    fun filter(ipFilterRule: IpFilterRule) {
+        listenerManager.filter(ipFilterRule)
     }
 
     /**
-     * Adds an IP+subnet rule that defines what type of connection this IP+subnet should have.
-     * - NOTHING : Nothing happens to the in/out bytes
-     * - COMPRESS: The in/out bytes are compressed with LZ4-fast
-     * - COMPRESS_AND_ENCRYPT: The in/out bytes are compressed (LZ4-fast) THEN encrypted (AES-256-GCM)
+     * Adds a function that will be called BEFORE a client/server "connects" with each other, and used to determine if a connection
+     * should be allowed
      *
-     * If no rules are defined, then for LOOPBACK, it will always be `COMPRESS` and for everything else it will always be `COMPRESS_AND_ENCRYPT`.
+     * By default, if there are no filter rules, then all connections are allowed to connect
+     * If there are filter rules - then ONLY connections for the filter that returns true are allowed to connect (all else are denied)
      *
-     * If rules are defined, then everything by default is `COMPRESS_AND_ENCRYPT`.
+     * It is the responsibility of the custom filter to write the error, if there is one
      *
-     * The compression algorithm is LZ4-fast, so there is a small performance impact for a very large gain
-     * Compress   :       6.210 micros/op;  629.0 MB/s (output: 55.4%)
-     * Uncompress :       0.641 micros/op; 6097.9 MB/s
+     * If the function returns TRUE, then the connection will continue to connect.
+     * If the function returns FALSE, then the other end of the connection will
+     *   receive a connection error
+     *
+     *
+     * If ANY filter rule that is applied returns true, then the connection is permitted
+     *
+     * This function will be called for **only** network clients (IPC client are excluded)
+     *
+     * @param function clientAddress: UDP connection address
+     *                       tagName: the connection tag name
      */
-    fun addConnectionRules(vararg rules: ConnectionRule) {
-        connectionRules.addAll(listOf(*rules))
+    fun filter(function: (clientAddress: InetAddress, tagName: String) -> Boolean)  {
+        listenerManager.filter(function)
+    }
+
+    /**
+     * Adds a function that will be called BEFORE a client/server "connects" with each other, and used to determine if buffered messages
+     * for a connection should be enabled
+     *
+     * By default, if there are no rules, then all connections will have buffered messages enabled
+     * If there are rules - then ONLY connections for the rule that returns true will have buffered messages enabled (all else are disabled)
+     *
+     * It is the responsibility of the custom filter to write the error, if there is one
+     *
+     * If the function returns TRUE, then the buffered messages for a connection are enabled.
+     * If the function returns FALSE, then the buffered messages for a connection is disabled.
+     *
+     * If ANY rule that is applied returns true, then the buffered messages for a connection are enabled
+     *
+     * @param function clientAddress: not-null when UDP connection, null when IPC connection
+     *                       tagName: the connection tag name
+     */
+    fun enableBufferedMessages(function: (clientAddress: InetAddress?, tagName: String) -> Boolean)  {
+        listenerManager.enableBufferedMessages(function)
     }
 
     /**
@@ -411,20 +404,36 @@ open class Server<CONNECTION : Connection>(
     }
 
     /**
-     * Closes the server and all it's connections. After a close, you may call 'bind' again.
+     * Will throw an exception if there are resources that are still in use
      */
-    final override fun close0() {
-        // when we call close, it will shutdown the polling mechanism, then wait for us to tell it to cleanup connections.
-        //
-        // Aeron + the Media Driver will have already been shutdown at this point.
-        if (bindAlreadyCalled.getAndSet(false)) {
-            // These are run in lock-step
-            shutdownPollLatch.countDown()
-            shutdownEventLatch.await()
-        }
+    fun checkForMemoryLeaks() {
+        AeronDriver.checkForMemoryLeaks()
+
+        // make sure that we have de-allocated all connection data
+        handshake.checkForMemoryLeaks()
     }
 
+    /**
+     * By default, if you call close() on the server, it will shut down all parts of the endpoint (listeners, driver, event polling, etc).
+     *
+     * @param closeEverything if true, all parts of the server will be closed (listeners, driver, event polling, etc)
+     */
+    fun close(closeEverything: Boolean = true) {
+        bufferedManager.close()
+        close(closeEverything = closeEverything, sendDisconnectMessage = true, releaseWaitingThreads = true)
+    }
 
+    override fun toString(): String {
+       return string0
+    }
+
+    fun <R> use(block: (Server<CONNECTION>) -> R): R {
+        return try {
+            block(this)
+        } finally {
+            close()
+        }
+    }
 
 //    /**
 //     * Only called by the server!

src/dorkbox/network/aeron/AeronContext.kt

@@ -1,36 +1,43 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
 package dorkbox.network.aeron
 
 import dorkbox.network.Configuration
-import dorkbox.util.NamedThreadFactory
+import dorkbox.network.exceptions.AeronDriverException
+import dorkbox.util.Sys
 import io.aeron.driver.MediaDriver
 import io.aeron.exceptions.DriverTimeoutException
-import mu.KLogger
+import org.slf4j.Logger
+import java.io.Closeable
 import java.io.File
-import java.util.concurrent.locks.*
-
-class AeronContext(
-    val config: Configuration,
-    val type: Class<*> = AeronDriver::class.java,
-    val logger: KLogger,
-    aeronErrorHandler: (error: Throwable) -> Unit
-) {
-    fun close() {
-        context.close()
-
-        // Destroys this thread group and all of its subgroups.
-        // This thread group must be empty, indicating that all threads that had been in this thread group have since stopped.
-        threadFactory.group.destroy()
-    }
+import java.util.concurrent.*
 
+/**
+ * Creates the Aeron Media Driver context
+ *
+ * @throws IllegalStateException if the configuration has already been used to create a context
+ * @throws IllegalArgumentException if the aeron media driver directory cannot be setup
+ */
+internal class AeronContext(config: Configuration.MediaDriverConfig, logger: Logger, aeronErrorHandler: (Throwable) -> Unit) : Closeable {
     companion object {
-        private fun create(
-            config: Configuration,
-            threadFactory: NamedThreadFactory,
-            aeronErrorHandler: (error: Throwable) -> Unit
-
-        ): MediaDriver.Context {
+        private fun create(config: Configuration.MediaDriverConfig, aeronErrorHandler: (Throwable) -> Unit): MediaDriver.Context {
             // LOW-LATENCY SETTINGS
-            // .termBufferSparseFile(false)
+            // MediaDriver.Context()
+            //             .termBufferSparseFile(false)
             //             .useWindowsHighResTimer(true)
             //             .threadingMode(ThreadingMode.DEDICATED)
             //             .conductorIdleStrategy(BusySpinIdleStrategy.INSTANCE)
@@ -42,53 +49,69 @@ class AeronContext(
             //    setProperty("aeron.socket.so_rcvbuf", "2097152");
             //    setProperty("aeron.rcv.initial.window.length", "2097152");
 
+            val threadFactory = Configuration.aeronThreadFactory
+
             // driver context must happen in the initializer, because we have a Server.isRunning() method that uses the mediaDriverContext (without bind)
             val mediaDriverContext = MediaDriver.Context()
+                .termBufferSparseFile(false) // files occupy the same space virtually AND physically!
+                .useWindowsHighResTimer(true)
+
+                // we assign our OWN ID! so we reserve everything.
                 .publicationReservedSessionIdLow(AeronDriver.RESERVED_SESSION_ID_LOW)
                 .publicationReservedSessionIdHigh(AeronDriver.RESERVED_SESSION_ID_HIGH)
+
                 .threadingMode(config.threadingMode)
                 .mtuLength(config.networkMtuSize)
+                .ipcMtuLength(config.ipcMtuSize)
 
                 .initialWindowLength(config.initialWindowLength)
-                .socketSndbufLength(config.sendBufferSize)
-                .socketRcvbufLength(config.receiveBufferSize)
 
-            mediaDriverContext
                 .conductorThreadFactory(threadFactory)
                 .receiverThreadFactory(threadFactory)
                 .senderThreadFactory(threadFactory)
                 .sharedNetworkThreadFactory(threadFactory)
                 .sharedThreadFactory(threadFactory)
 
-            mediaDriverContext.aeronDirectoryName(config.aeronDirectory!!.absolutePath)
 
-            if (mediaDriverContext.ipcTermBufferLength() != io.aeron.driver.Configuration.ipcTermBufferLength()) {
-                // default 64 megs each is HUGE
-                mediaDriverContext.ipcTermBufferLength(8 * 1024 * 1024)
+            if (config.sendBufferSize > 0) {
+                mediaDriverContext.socketSndbufLength(config.sendBufferSize)
             }
 
-            if (mediaDriverContext.publicationTermBufferLength() != io.aeron.driver.Configuration.termBufferLength()) {
-                // default 16 megs each is HUGE (we run out of space in production w/ lots of clients)
-                mediaDriverContext.publicationTermBufferLength(2 * 1024 * 1024)
+            if (config.receiveBufferSize > 0) {
+                mediaDriverContext.socketRcvbufLength(config.receiveBufferSize)
+            }
+
+            if (config.conductorIdleStrategy != null) {
+                mediaDriverContext.conductorIdleStrategy(config.conductorIdleStrategy)
+            }
+            if (config.sharedIdleStrategy != null) {
+                mediaDriverContext.sharedIdleStrategy(config.sharedIdleStrategy)
+            }
+            if (config.receiverIdleStrategy != null) {
+                mediaDriverContext.receiverIdleStrategy(config.receiverIdleStrategy)
+            }
+            if (config.senderIdleStrategy != null) {
+                mediaDriverContext.senderIdleStrategy(config.senderIdleStrategy)
+            }
+
+            mediaDriverContext.aeronDirectoryName(config.aeronDirectory!!.path)
+
+            if (config.ipcTermBufferLength > 0) {
+                mediaDriverContext.ipcTermBufferLength(config.ipcTermBufferLength)
+            }
+
+            if (config.publicationTermBufferLength > 0) {
+                mediaDriverContext.publicationTermBufferLength(config.publicationTermBufferLength)
             }
 
             // we DO NOT want to abort the JVM if there are errors.
             // this replaces the default handler with one that doesn't abort the JVM
-            mediaDriverContext.errorHandler { error ->
-                aeronErrorHandler(error)
-            }
+            mediaDriverContext.errorHandler(aeronErrorHandler)
 
             return mediaDriverContext
         }
     }
 
-    // this is the aeron conductor/network processor thread factory which manages the incoming messages from the network.
-    internal val threadFactory = NamedThreadFactory(
-        "Aeron",
-        ThreadGroup("${type.simpleName}-AeronDriver"), Thread.MAX_PRIORITY,
-        true)
-
-
     // the context is validated before the AeronDriver object is created
     val context: MediaDriver.Context
 
@@ -105,11 +128,17 @@ class AeronContext(
      *
      * @return the aeron context directory
      */
-    val driverDirectory: File
+    val directory: File
         get() {
             return context.aeronDirectory()
         }
 
+    fun deleteAeronDir(): Boolean {
+        // NOTE: We must be *super* careful trying to delete directories, because if we have multiple AERON/MEDIA DRIVERS connected to the
+        //   same directory, deleting the directory will cause any other aeron connection to fail! (which makes sense).
+        return directory.deleteRecursively()
+    }
+
     /**
      * Checks to see if an endpoint (using the specified configuration) is running.
      *
@@ -120,15 +149,20 @@ class AeronContext(
         return context.isDriverActive(context.driverTimeoutMs()) { }
     }
 
+    private fun isRunning(context: MediaDriver.Context): Boolean {
+        // if the media driver is running, it will be a quick connection. Usually 100ms or so
+        return try {
+            context.isDriverActive(context.driverTimeoutMs()) { }
+        } catch (e: Exception) {
+            false
+        }
+    }
 
-    /**
-     * Creates the Aeron Media Driver context
-     *
-     * @throws IllegalStateException if the configuration has already been used to create a context
-     * @throws IllegalArgumentException if the aeron media driver directory cannot be setup
-     */
-     init {
-        var context = create(config, threadFactory, aeronErrorHandler)
+    init {
+        // NOTE: if a DIFFERENT PROCESS is using the SAME driver location, THERE WILL BE POTENTIAL PROBLEMS!
+        //  ADDITIONALLY, the ONLY TIME we create a new aeron context is when it is the FIRST aeron context for a driver. Within the same
+        //  JVM, the aeron driver/context is SHARED.
+        val context = create(config, aeronErrorHandler)
 
         // this happens EXACTLY once. Must be BEFORE the "isRunning" check!
         context.concludeAeronDirectory()
@@ -139,57 +173,57 @@ class AeronContext(
         val driverTimeout = context.driverTimeoutMs()
 
         // sometimes when starting up, if a PREVIOUS run was corrupted (during startup, for example)
-        // we ONLY do this during the initial startup check because it will delete the directory, and we don't
-        // always want to do this.
-        var isRunning = try {
+        // we ONLY do this during the initial startup check because it will delete the directory, and we don't always want to do this.
+
+        val isRunning = try {
             context.isDriverActive(driverTimeout) { }
         } catch (e: DriverTimeoutException) {
             // we have to delete the directory, since it was corrupted, and we try again.
-            if (aeronDir.deleteRecursively()) {
+            if (!config.forceAllowSharedAeronDriver && aeronDir.deleteRecursively()) {
                 context.isDriverActive(driverTimeout) { }
+            } else if (config.forceAllowSharedAeronDriver) {
+                // we are expecting a shared directory. SOMETHING is screwed up!
+                throw AeronDriverException("Aeron was expected to be running, and the current location is corrupted. Not doing anything!", e)
             } else {
                 // unable to delete the directory
                 throw e
             }
         }
 
-        // this is incompatible with IPC, and will not be set if IPC is enabled
-        if (config.uniqueAeronDirectory && isRunning) {
-            val savedParent = aeronDir.parentFile
-            var retry = 0
-            val retryMax = 100
+        // only do this if we KNOW we are not running!
+        if (!isRunning) {
+            // NOTE: We must be *super* careful trying to delete directories, because if we have multiple AERON/MEDIA DRIVERS connected to the
+            //   same directory, deleting the directory will cause any other aeron connection to fail! (which makes sense).
+            // make sure it's clean!
+            aeronDir.deleteRecursively()
 
-            while (config.uniqueAeronDirectory && isRunning) {
-                if (retry++ > retryMax) {
-                    throw IllegalArgumentException("Unable to force unique aeron Directory. Tried $retryMax times and all tries were in use.")
-                }
+            // if we are not CURRENTLY running, then we should ALSO delete it when we are done!
+            context.dirDeleteOnShutdown()
+        } else if (!config.forceAllowSharedAeronDriver) {
+            // maybe it's a mistake because we restarted too quickly! A brief pause to fix this!
 
-                val randomNum = (1..retryMax).shuffled().first()
-                val newDir = savedParent.resolve("${aeronDir.name}_$randomNum")
+            val timeoutInNs = TimeUnit.SECONDS.toNanos(config.connectionCloseTimeoutInSeconds.toLong()) + context.publicationLingerTimeoutNs()
+            val timeoutInMs = TimeUnit.NANOSECONDS.toMillis(timeoutInNs)
+            logger.warn("Aeron is currently running, waiting ${Sys.getTimePrettyFull(timeoutInNs)} for it to close.")
 
-                context = create(config, threadFactory, aeronErrorHandler)
-                context.aeronDirectoryName(newDir.path)
-
-                // this happens EXACTLY once. Must be BEFORE the "isRunning" check!
-                context.concludeAeronDirectory()
-
-                isRunning = context.isDriverActive(driverTimeout) { }
+            // wait for it to close! wait longer.
+            val startTime = System.nanoTime()
+            while (isRunning(context) && System.nanoTime() - startTime < timeoutInNs) {
+                Thread.sleep(timeoutInMs)
             }
 
-            if (!isRunning) {
-                // NOTE: We must be *super* careful trying to delete directories, because if we have multiple AERON/MEDIA DRIVERS connected to the
-                //   same directory, deleting the directory will cause any other aeron connection to fail! (which makes sense).
-                // since we are forcing a unique directory, we should ALSO delete it when we are done!
-                context.dirDeleteOnShutdown()
-            }
+            require(!isRunning(context)) { "Aeron is currently running, and this is the first instance created by this JVM. " +
+                    "You must use `config.forceAllowSharedAeronDriver` to be able to re-use a shared aeron process at: $aeronDir" }
         }
 
-        logger.info { "Aeron directory: '${context.aeronDirectory()}'" }
-
         this.context = context
     }
 
     override fun toString(): String {
         return context.toString()
     }
+
+    override fun close() {
+        context.close()
+    }
 }

src/dorkbox/network/aeron/AeronPoller.kt

@@ -1,8 +1,23 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
 package dorkbox.network.aeron
 
 internal interface AeronPoller {
     fun poll(): Int
     fun close()
-
     val info: String
 }

src/dorkbox/network/aeron/BacklogStat.kt

@@ -1,3 +1,19 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
 /*
  * Copyright 2014-2022 Real Logic Limited.
  *
@@ -33,22 +49,19 @@ import java.util.*
  * [StreamStat] counters.
  *
  *
- * Each stream managed by the [io.aeron.driver.MediaDriver] will be sampled and printed out on [System.out].
- */
-class BacklogStat
-/**
- * Construct by using a [CountersReader] which can be obtained from [Aeron.countersReader].
+ * Each stream managed by the [io.aeron.driver.MediaDriver] will be sampled
  *
  * @param counters to read for tracking positions.
  */
-    (private val counters: CountersReader) {
+class BacklogStat(private val counters: CountersReader) {
     /**
      * Take a snapshot of all the backlog information and group by stream.
      *
      * @return a snapshot of all the backlog information and group by stream.
      */
     fun snapshot(): Map<StreamCompositeKey, StreamBacklog> {
-        val streams: MutableMap<StreamCompositeKey, StreamBacklog> = HashMap()
+        val streams = mutableMapOf<StreamCompositeKey, StreamBacklog>()
+
         counters.forEach { counterId: Int, typeId: Int, keyBuffer: DirectBuffer, _: String? ->
             if (typeId >= PublisherLimit.PUBLISHER_LIMIT_TYPE_ID && typeId <= ReceiverPos.RECEIVER_POS_TYPE_ID || typeId == SenderLimit.SENDER_LIMIT_TYPE_ID || typeId == PerImageIndicator.PER_IMAGE_TYPE_ID || typeId == PublisherPos.PUBLISHER_POS_TYPE_ID) {
                 val key = StreamCompositeKey(

src/dorkbox/network/aeron/CoroutineBackoffIdleStrategy.kt

@@ -1,352 +0,0 @@
-/*
- * Copyright 2014-2020 Real Logic Limited.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- * https://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package dorkbox.network.aeron
-
-import kotlinx.coroutines.delay
-import kotlinx.coroutines.yield
-import org.agrona.concurrent.BackoffIdleStrategy
-import org.agrona.hints.ThreadHints
-
-abstract class BackoffIdleStrategyPrePad {
-    var p000: Byte = 0
-    var p001: Byte = 0
-    var p002: Byte = 0
-    var p003: Byte = 0
-    var p004: Byte = 0
-    var p005: Byte = 0
-    var p006: Byte = 0
-    var p007: Byte = 0
-    var p008: Byte = 0
-    var p009: Byte = 0
-    var p010: Byte = 0
-    var p011: Byte = 0
-    var p012: Byte = 0
-    var p013: Byte = 0
-    var p014: Byte = 0
-    var p015: Byte = 0
-    var p016: Byte = 0
-    var p017: Byte = 0
-    var p018: Byte = 0
-    var p019: Byte = 0
-    var p020: Byte = 0
-    var p021: Byte = 0
-    var p022: Byte = 0
-    var p023: Byte = 0
-    var p024: Byte = 0
-    var p025: Byte = 0
-    var p026: Byte = 0
-    var p027: Byte = 0
-    var p028: Byte = 0
-    var p029: Byte = 0
-    var p030: Byte = 0
-    var p031: Byte = 0
-    var p032: Byte = 0
-    var p033: Byte = 0
-    var p034: Byte = 0
-    var p035: Byte = 0
-    var p036: Byte = 0
-    var p037: Byte = 0
-    var p038: Byte = 0
-    var p039: Byte = 0
-    var p040: Byte = 0
-    var p041: Byte = 0
-    var p042: Byte = 0
-    var p043: Byte = 0
-    var p044: Byte = 0
-    var p045: Byte = 0
-    var p046: Byte = 0
-    var p047: Byte = 0
-    var p048: Byte = 0
-    var p049: Byte = 0
-    var p050: Byte = 0
-    var p051: Byte = 0
-    var p052: Byte = 0
-    var p053: Byte = 0
-    var p054: Byte = 0
-    var p055: Byte = 0
-    var p056: Byte = 0
-    var p057: Byte = 0
-    var p058: Byte = 0
-    var p059: Byte = 0
-    var p060: Byte = 0
-    var p061: Byte = 0
-    var p062: Byte = 0
-    var p063: Byte = 0
-}
-
-abstract class BackoffIdleStrategyData(
-        protected val maxSpins: Long, protected val maxYields: Long, protected val minParkPeriodMs: Long, protected val maxParkPeriodMs: Long) : BackoffIdleStrategyPrePad() {
-
-    protected var state = 0 // NOT_IDLE
-    protected var spins: Long = 0
-    protected var yields: Long = 0
-    protected var parkPeriodMs: Long = 0
-}
-
-/**
- * Idling strategy for threads when they have no work to do.
- * <p>
- * Spin for maxSpins, then
- * [Coroutine.yield] for maxYields, then
- * [Coroutine.delay] on an exponential backoff to maxParkPeriodMs
- */
-@Suppress("unused")
-class CoroutineBackoffIdleStrategy : BackoffIdleStrategyData, CoroutineIdleStrategy {
-    var p064: Byte = 0
-    var p065: Byte = 0
-    var p066: Byte = 0
-    var p067: Byte = 0
-    var p068: Byte = 0
-    var p069: Byte = 0
-    var p070: Byte = 0
-    var p071: Byte = 0
-    var p072: Byte = 0
-    var p073: Byte = 0
-    var p074: Byte = 0
-    var p075: Byte = 0
-    var p076: Byte = 0
-    var p077: Byte = 0
-    var p078: Byte = 0
-    var p079: Byte = 0
-    var p080: Byte = 0
-    var p081: Byte = 0
-    var p082: Byte = 0
-    var p083: Byte = 0
-    var p084: Byte = 0
-    var p085: Byte = 0
-    var p086: Byte = 0
-    var p087: Byte = 0
-    var p088: Byte = 0
-    var p089: Byte = 0
-    var p090: Byte = 0
-    var p091: Byte = 0
-    var p092: Byte = 0
-    var p093: Byte = 0
-    var p094: Byte = 0
-    var p095: Byte = 0
-    var p096: Byte = 0
-    var p097: Byte = 0
-    var p098: Byte = 0
-    var p099: Byte = 0
-    var p100: Byte = 0
-    var p101: Byte = 0
-    var p102: Byte = 0
-    var p103: Byte = 0
-    var p104: Byte = 0
-    var p105: Byte = 0
-    var p106: Byte = 0
-    var p107: Byte = 0
-    var p108: Byte = 0
-    var p109: Byte = 0
-    var p110: Byte = 0
-    var p111: Byte = 0
-    var p112: Byte = 0
-    var p113: Byte = 0
-    var p114: Byte = 0
-    var p115: Byte = 0
-    var p116: Byte = 0
-    var p117: Byte = 0
-    var p118: Byte = 0
-    var p119: Byte = 0
-    var p120: Byte = 0
-    var p121: Byte = 0
-    var p122: Byte = 0
-    var p123: Byte = 0
-    var p124: Byte = 0
-    var p125: Byte = 0
-    var p126: Byte = 0
-    var p127: Byte = 0
-
-    companion object {
-        private const val NOT_IDLE = 0
-        private const val SPINNING = 1
-        private const val YIELDING = 2
-        private const val PARKING = 3
-
-        /**
-         * Name to be returned from [.alias].
-         */
-        const val ALIAS = "backoff"
-
-        /**
-         * Default number of times the strategy will spin without work before going to next state.
-         */
-        const val DEFAULT_MAX_SPINS = 10L
-
-        /**
-         * Default number of times the strategy will yield without work before going to next state.
-         */
-        const val DEFAULT_MAX_YIELDS = 5L
-
-        /**
-         * Default interval the strategy will park the thread on entering the park state in milliseconds.
-         */
-        const val DEFAULT_MIN_PARK_PERIOD_MS = 1L
-
-        /**
-         * Default interval the strategy will park the thread will expand interval to as a max in milliseconds.
-         */
-        const val DEFAULT_MAX_PARK_PERIOD_MS = 1000L
-    }
-
-    /**
-     * Default constructor using [.DEFAULT_MAX_SPINS], [.DEFAULT_MAX_YIELDS], [.DEFAULT_MIN_PARK_PERIOD_MS], and [.DEFAULT_MAX_PARK_PERIOD_MS].
-     */
-    constructor() : super(DEFAULT_MAX_SPINS, DEFAULT_MAX_YIELDS, DEFAULT_MIN_PARK_PERIOD_MS, DEFAULT_MAX_PARK_PERIOD_MS) {}
-
-    /**
-     * Create a set of state tracking idle behavior
-     * <p>
-     * @param maxSpins        to perform before moving to [Coroutine.yield]
-     * @param maxYields       to perform before moving to [Coroutine.delay]
-     * @param minParkPeriodMs to use when initiating parking
-     * @param maxParkPeriodMs to use for end duration when parking
-     */
-    constructor(maxSpins: Long, maxYields: Long, minParkPeriodMs: Long, maxParkPeriodMs: Long)
-            : super(maxSpins, maxYields, minParkPeriodMs, maxParkPeriodMs) {
-    }
-
-    /**
-     * Perform current idle action (e.g. nothing/yield/sleep). This method signature expects users to call into it on
-     * every work 'cycle'. The implementations may use the indication "workCount &gt; 0" to reset internal backoff
-     * state. This method works well with 'work' APIs which follow the following rules:
-     * <ul>
-     * <li>'work' returns a value larger than 0 when some work has been done</li>
-     * <li>'work' returns 0 when no work has been done</li>
-     * <li>'work' may return error codes which are less than 0, but which amount to no work has been done</li>
-     * </ul>
-     * <p>
-     * Callers are expected to follow this pattern:
-     *
-     * <pre>
-     * <code>
-     * while (isRunning)
-     * {
-     *     idleStrategy.idle(doWork());
-     * }
-     * </code>
-     * </pre>
-     *
-     * @param workCount performed in last duty cycle.
-     */
-    override suspend fun idle(workCount: Int) {
-        if (workCount > 0) {
-            reset()
-        } else {
-            idle()
-        }
-    }
-
-    /**
-     * Perform current idle action (e.g. nothing/yield/sleep). To be used in conjunction with
-     * {@link IdleStrategy#reset()} to clear internal state when idle period is over (or before it begins).
-     * Callers are expected to follow this pattern:
-     *
-     * <pre>
-     * <code>
-     * while (isRunning)
-     * {
-     *   if (!hasWork())
-     *   {
-     *     idleStrategy.reset();
-     *     while (!hasWork())
-     *     {
-     *       if (!isRunning)
-     *       {
-     *         return;
-     *       }
-     *       idleStrategy.idle();
-     *     }
-     *   }
-     *   doWork();
-     * }
-     * </code>
-     * </pre>
-     */
-    override suspend fun idle() {
-        when (state) {
-            NOT_IDLE -> {
-                state = SPINNING
-                spins++
-            }
-
-            SPINNING -> {
-                ThreadHints.onSpinWait()
-                if (++spins > maxSpins) {
-                    state = YIELDING
-                    yields = 0
-                }
-            }
-
-            YIELDING -> if (++yields > maxYields) {
-                state = PARKING
-                parkPeriodMs = minParkPeriodMs
-            } else {
-                yield()
-            }
-
-            PARKING -> {
-                delay(parkPeriodMs)
-                // double the delay until we get to MAX
-                parkPeriodMs = (parkPeriodMs shl 1).coerceAtMost(maxParkPeriodMs)
-            }
-        }
-    }
-
-
-    /**
-     * Reset the internal state in preparation for entering an idle state again.
-     */
-    override fun reset() {
-        spins = 0
-        yields = 0
-        parkPeriodMs = minParkPeriodMs
-        state = NOT_IDLE
-    }
-
-    /**
-     * Simple name by which the strategy can be identified.
-     *
-     * @return simple name by which the strategy can be identified.
-     */
-    override fun alias(): String {
-        return ALIAS
-    }
-
-    /**
-     * Creates a clone of this IdleStrategy
-     */
-    override fun clone(): CoroutineBackoffIdleStrategy {
-        return CoroutineBackoffIdleStrategy(maxSpins = maxSpins, maxYields = maxYields, minParkPeriodMs = minParkPeriodMs, maxParkPeriodMs = maxParkPeriodMs)
-    }
-
-    /**
-     * Creates a clone of this IdleStrategy
-     */
-    override fun cloneToNormal(): BackoffIdleStrategy {
-        return BackoffIdleStrategy(maxSpins, maxYields, minParkPeriodMs, maxParkPeriodMs)
-    }
-
-    override fun toString(): String {
-        return "BackoffIdleStrategy{" +
-                "alias=" + ALIAS +
-                ", maxSpins=" + maxSpins +
-                ", maxYields=" + maxYields +
-                ", minParkPeriodMs=" + minParkPeriodMs +
-                ", maxParkPeriodMs=" + maxParkPeriodMs +
-                '}'
-    }
-}

src/dorkbox/network/aeron/CoroutineIdleStrategy.kt

@@ -1,117 +0,0 @@
-/*
- * Copyright 2014-2020 Real Logic Limited.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- * https://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package dorkbox.network.aeron
-
-import org.agrona.concurrent.IdleStrategy
-
-/**
- * Idle strategy for use by threads when they do not have work to do.
- *
- *
- * **Note regarding implementor state**
- *
- *
- * Some implementations are known to be stateful, please note that you cannot safely assume implementations to be
- * stateless. Where implementations are stateful it is recommended that implementation state is padded to avoid false
- * sharing.
- *
- *
- * **Note regarding potential for TTSP(Time To Safe Point) issues**
- *
- *
- * If the caller spins in a 'counted' loop, and the implementation does not include a a safepoint poll this may cause a
- * TTSP (Time To SafePoint) problem. If this is the case for your application you can solve it by preventing the idle
- * method from being inlined by using a Hotspot compiler command as a JVM argument e.g:
- * `-XX:CompileCommand=dontinline,org.agrona.concurrent.NoOpIdleStrategy::idle`
- */
-interface CoroutineIdleStrategy {
-    /**
-     * Perform current idle action (e.g. nothing/yield/sleep). This method signature expects users to call into it on
-     * every work 'cycle'. The implementations may use the indication "workCount > 0" to reset internal backoff
-     * state. This method works well with 'work' APIs which follow the following rules:
-     * <ul>
-     * <li>'work' returns a value larger than 0 when some work has been done</li>
-     * <li>'work' returns 0 when no work has been done</li>
-     * <li>'work' may return error codes which are less than 0, but which amount to no work has been done</li>
-     * </ul>
-     * <p>
-     * Callers are expected to follow this pattern:
-     *
-     * <pre>
-     * <code>
-     * while (isRunning)
-     * {
-     *     idleStrategy.idle(doWork());
-     * }
-     * </code>
-     * </pre>
-     *
-     * @param workCount performed in last duty cycle.
-     */
-    suspend fun idle(workCount: Int)
-
-    /**
-     * Perform current idle action (e.g. nothing/yield/sleep). To be used in conjunction with
-     * {@link IdleStrategy#reset()} to clear internal state when idle period is over (or before it begins).
-     * Callers are expected to follow this pattern:
-     *
-     * <pre>
-     * <code>
-     * while (isRunning)
-     * {
-     *   if (!hasWork())
-     *   {
-     *     idleStrategy.reset();
-     *     while (!hasWork())
-     *     {
-     *       if (!isRunning)
-     *       {
-     *         return;
-     *       }
-     *       idleStrategy.idle();
-     *     }
-     *   }
-     *   doWork();
-     * }
-     * </code>
-     * </pre>
-     */
-    suspend fun idle()
-
-    /**
-     * Reset the internal state in preparation for entering an idle state again.
-     */
-    fun reset()
-
-    /**
-     * Simple name by which the strategy can be identified.
-     *
-     * @return simple name by which the strategy can be identified.
-     */
-    fun alias(): String {
-        return ""
-    }
-
-    /**
-     * Creates a clone of this IdleStrategy
-     */
-    fun clone(): CoroutineIdleStrategy
-
-    /**
-     * Creates a clone of this IdleStrategy
-     */
-    fun cloneToNormal(): IdleStrategy
-}

src/dorkbox/network/aeron/CoroutineSleepingMillisIdleStrategy.kt

@@ -1,108 +0,0 @@
-/*
- * Copyright 2014-2020 Real Logic Limited.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- * https://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package dorkbox.network.aeron
-
-import kotlinx.coroutines.delay
-import org.agrona.concurrent.SleepingMillisIdleStrategy
-
-/**
- * When idle this strategy is to sleep for a specified period time in milliseconds.
- *
- *
- * This class uses [Coroutine.delay] to idle.
- */
-class CoroutineSleepingMillisIdleStrategy : CoroutineIdleStrategy {
-    companion object {
-        /**
-         * Name to be returned from [.alias].
-         */
-        const val ALIAS = "sleep-ms"
-
-        /**
-         * Default sleep period when the default constructor is used.
-         */
-        const val DEFAULT_SLEEP_PERIOD_MS = 1L
-    }
-
-    private val sleepPeriodMs: Long
-
-    /**
-     * Default constructor that uses [.DEFAULT_SLEEP_PERIOD_MS].
-     */
-    constructor() {
-        sleepPeriodMs = DEFAULT_SLEEP_PERIOD_MS
-    }
-
-    /**
-     * Constructed a new strategy that will sleep for a given period when idle.
-     *
-     * @param sleepPeriodMs period in milliseconds for which the strategy will sleep when work count is 0.
-     */
-    constructor(sleepPeriodMs: Long) {
-        this.sleepPeriodMs = sleepPeriodMs
-    }
-
-    /**
-     * {@inheritDoc}
-     */
-    override suspend fun idle(workCount: Int) {
-        if (workCount > 0) {
-            return
-        }
-        delay(sleepPeriodMs)
-    }
-
-    /**
-     * {@inheritDoc}
-     */
-    override suspend fun idle() {
-        delay(sleepPeriodMs)
-    }
-
-    /**
-     * {@inheritDoc}
-     */
-    override fun reset() {}
-
-    /**
-     * {@inheritDoc}
-     */
-    override fun alias(): String {
-        return ALIAS
-    }
-
-    /**
-     * Creates a clone of this IdleStrategy
-     */
-    override fun clone(): CoroutineSleepingMillisIdleStrategy {
-        return CoroutineSleepingMillisIdleStrategy(sleepPeriodMs = sleepPeriodMs)
-    }
-
-    /**
-     * Creates a clone of this IdleStrategy
-     */
-    override fun cloneToNormal(): SleepingMillisIdleStrategy {
-        return SleepingMillisIdleStrategy(sleepPeriodMs)
-    }
-
-
-    override fun toString(): String {
-        return "SleepingMillisIdleStrategy{" +
-                "alias=" + ALIAS +
-                ", sleepPeriodMs=" + sleepPeriodMs +
-                '}'
-    }
-}

src/dorkbox/network/aeron/EventActionOperator.kt

@@ -1,5 +1,5 @@
 /*
- * Copyright 2020 dorkbox, llc
+ * Copyright 2023 dorkbox, llc
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -13,17 +13,9 @@
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
-@file:Suppress("DuplicatedCode")
 
-package dorkbox.network.aeron.mediaDriver
+package dorkbox.network.aeron
 
-import io.aeron.Subscription
-
-abstract class MediaDriverServer(val port: Int,
-                                 val streamId: Int,
-                                 val sessionId: Int,
-                                 val connectionTimeoutSec: Int, val
-                                 isReliable: Boolean) : MediaDriverConnection {
-
-    lateinit var subscription: Subscription
+internal interface EventActionOperator {
+    operator fun invoke(): Int
 }

src/dorkbox/network/aeron/EventCloseOperator.kt

@@ -0,0 +1,21 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package dorkbox.network.aeron
+
+internal interface EventCloseOperator {
+    operator fun invoke()
+}

src/dorkbox/network/aeron/EventPoller.kt

@@ -0,0 +1,241 @@
+/*
+ * Copyright 2024 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package dorkbox.network.aeron
+
+import dorkbox.bytes.ByteArrayWrapper
+import dorkbox.collections.ConcurrentIterator
+import dorkbox.network.Configuration
+import dorkbox.network.connection.EndPoint
+import dorkbox.util.NamedThreadFactory
+import kotlinx.atomicfu.atomic
+import org.agrona.concurrent.IdleStrategy
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+import java.util.concurrent.*
+import java.util.concurrent.locks.*
+import kotlin.concurrent.write
+
+/**
+ * there are threading issues if there are client(s) and server's within the same JVM, where we have thread starvation
+ *
+ * additionally, if we have MULTIPLE clients on the same machine, we are limited by the CPU core count. Ideally we want to share
+ * this among ALL clients within the same JVM so that we can support multiple clients/servers
+ */
+internal class EventPoller {
+
+    private class EventAction(val onAction: EventActionOperator, val onClose: EventCloseOperator)
+
+    companion object {
+        internal const val REMOVE = -1
+
+        val eventLogger = LoggerFactory.getLogger(EventPoller::class.java.simpleName)
+
+
+        private val pollExecutor = Executors.newSingleThreadExecutor(
+            NamedThreadFactory("Poll Dispatcher", Configuration.networkThreadGroup, true)
+        )
+    }
+
+    private var configured = false
+
+    private lateinit var pollStrategy: IdleStrategy
+
+    @Volatile
+    private var running = false
+    private var lock = ReentrantReadWriteLock()
+
+    // this is thread safe
+    private val pollEvents = ConcurrentIterator<EventAction>()
+    private val submitEvents = atomic(0)
+    private val configureEventsEndpoints = mutableSetOf<ByteArrayWrapper>()
+
+    @Volatile
+    private var shutdownLatch = CountDownLatch(0)
+
+
+    @Volatile
+    private var threadId = 0L
+
+
+    fun isDispatch(): Boolean {
+        // this only works because we are a single thread dispatch
+        return threadId == Thread.currentThread().id
+    }
+
+    fun configure(logger: Logger, config: Configuration, endPoint: EndPoint<*>) {
+        lock.write {
+            if (logger.isDebugEnabled) {
+                logger.debug("Initializing the Network Event Poller...")
+            }
+            configureEventsEndpoints.add(ByteArrayWrapper.wrap(endPoint.storage.publicKey))
+
+            if (!configured) {
+                if (logger.isTraceEnabled) {
+                    logger.trace("Configuring the Network Event Poller...")
+                }
+
+                running = true
+                configured = true
+                shutdownLatch = CountDownLatch(1)
+                pollStrategy = config.pollIdleStrategy
+
+                pollExecutor.submit {
+                    val pollIdleStrategy = pollStrategy
+                    var pollCount = 0
+                    threadId = Thread.currentThread().id // only ever 1 thread!!!
+
+                    pollIdleStrategy.reset()
+
+                    while (running) {
+                        pollEvents.forEachRemovable {
+                            try {
+                                // check to see if we should remove this event (when a client/server closes, it is removed)
+                                // once ALL endpoint are closed, this is shutdown.
+                                val poll = it.onAction()
+
+                                // <0 means we remove the event from processing
+                                // 0 means we idle
+                                // >0 means reset and don't idle (because there are likely more poll events)
+                                if (poll < 0) {
+                                    // remove our event, it is no longer valid
+                                    pollEvents.remove(this)
+                                    it.onClose() // shutting down
+                                } else if (poll > 0) {
+                                    pollCount += poll
+                                }
+                            } catch (e: Exception) {
+                                eventLogger.error("Unexpected error during Network Event Polling! Aborting event dispatch for it!", e)
+
+                                // remove our event, it is no longer valid
+                                pollEvents.remove(this)
+                                it.onClose() // shutting down
+                            }
+                        }
+
+                        pollIdleStrategy.idle(pollCount)
+                    }
+
+
+                    // now we have to REMOVE all poll events -- so that their remove logic will run.
+                    pollEvents.forEachRemovable {
+                        // remove our event, it is no longer valid
+                        pollEvents.remove(this)
+                        it.onClose() // shutting down
+                    }
+
+                    shutdownLatch.countDown()
+                }
+            } else {
+                // we don't want to use .equals, because that also compares STATE, which for us is going to be different because we are cloned!
+                // toString has the right info to compare types/config accurately
+                require(pollStrategy.toString() == config.pollIdleStrategy.toString()) {
+                    "The network event poll strategy is different between the multiple instances of network clients/servers. There **WILL BE** thread starvation, so this behavior is forbidden!"
+                }
+            }
+        }
+    }
+
+    /**
+     * Will cause the executing thread to wait until the event has been started
+     */
+    fun submit(action: EventActionOperator, onClose: EventCloseOperator) = lock.write {
+        submitEvents.getAndIncrement()
+
+        // this forces the current thread to WAIT until the network poll system has started
+        val pollStartupLatch = CountDownLatch(1)
+
+        pollEvents.add(EventAction(action, onClose))
+        pollEvents.add(EventAction(
+            object : EventActionOperator {
+                override fun invoke(): Int {
+                    pollStartupLatch.countDown()
+
+                    // remove ourselves
+                    return REMOVE
+                }
+            }
+            , object : EventCloseOperator {
+                override fun invoke() {}
+            }
+        ))
+
+        pollStartupLatch.await()
+
+        submitEvents.getAndDecrement()
+    }
+
+
+
+    /**
+     * Waits for all events to finish running
+     */
+    fun close(logger: Logger, endPoint: EndPoint<*>) {
+        // make sure that we close on the CLOSE dispatcher if we run on the poll dispatcher!
+        if (isDispatch()) {
+            endPoint.eventDispatch.CLOSE.launch {
+                close(logger, endPoint)
+            }
+            return
+        }
+
+        lock.write {
+            logger.debug("Requesting close for the Network Event Poller...")
+
+            // ONLY if there are no more poll-events do we ACTUALLY shut down.
+            // when an endpoint closes its polling, it will automatically be removed from this datastructure.
+            val publicKeyWrapped = ByteArrayWrapper.wrap(endPoint.storage.publicKey)
+            configureEventsEndpoints.removeIf { it == publicKeyWrapped }
+            val cEvents = configureEventsEndpoints.size
+
+            // these prevent us from closing too early
+            val pEvents = pollEvents.size()
+            val sEvents = submitEvents.value
+
+            if (running && sEvents == 0 && cEvents == 0) {
+                when (pEvents) {
+                    0 -> {
+                        logger.debug("Closing the Network Event Poller...")
+                        doClose(logger)
+                    }
+                    else -> {
+                        if (logger.isDebugEnabled) {
+                            logger.debug("Not closing the Network Event Poller... (isRunning=$running submitEvents=$sEvents configureEvents=${cEvents} pollEvents=$pEvents)")
+                        }
+                    }
+                }
+            } else if (logger.isDebugEnabled) {
+                logger.debug("Not closing the Network Event Poller... (isRunning=$running submitEvents=$sEvents configureEvents=${cEvents} pollEvents=$pEvents)")
+            }
+        }
+    }
+
+    private fun doClose(logger: Logger) {
+        val wasRunning = running
+
+        running = false
+        while (!shutdownLatch.await(500, TimeUnit.MILLISECONDS)) {
+            logger.error("Waiting for Network Event Poller to close. It should not take this long")
+        }
+        configured = false
+
+        if (wasRunning) {
+            pollExecutor.awaitTermination(200, TimeUnit.MILLISECONDS)
+        }
+
+        logger.debug("Closed Network Event Poller: wasRunning=$wasRunning")
+    }
+}

src/dorkbox/network/aeron/FixTransportPoller.kt

@@ -0,0 +1,126 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package dorkbox.network.aeron
+
+import dorkbox.jna.ClassUtils
+import dorkbox.os.OS
+import javassist.ClassPool
+import javassist.CtNewMethod
+
+
+
+object FixTransportPoller {
+    // allow access to sun.nio.ch.SelectorImpl without causing reflection or JPMS module issues
+    fun init() {
+        if (OS.javaVersion <= 11) {
+            // older versions of java don't need to worry about rewriting anything
+            return
+        }
+
+
+        try {
+            val pool = ClassPool.getDefault()
+
+            run {
+                val dynamicClass = pool.makeClass("sun.nio.ch.SelectorImplAccessory")
+                val method = CtNewMethod.make(
+                    ("public static java.lang.reflect.Field getKey(java.lang.String fieldName) { " +
+                            "java.lang.reflect.Field field = Class.forName(\"sun.nio.ch.SelectorImpl\").getDeclaredField( fieldName );" +
+                            "field.setAccessible( true );" +
+                            "return field;" +
+                        "}"), dynamicClass
+                )
+                dynamicClass.addMethod(method)
+
+                val dynamicClassBytes = dynamicClass.toBytecode()
+                ClassUtils.defineClass(null, dynamicClassBytes)
+            }
+
+            // have to trampoline off this to get around module access
+            run {
+                val dynamicClass = pool.makeClass("java.lang.SelectorImplAccessory")
+                val method = CtNewMethod.make(
+                    ("public static java.lang.reflect.Field getKey(java.lang.String fieldName) { " +
+                            "return sun.nio.ch.SelectorImplAccessory.getKey(fieldName);" +
+                            "}"), dynamicClass
+                )
+                dynamicClass.addMethod(method)
+
+                val dynamicClassBytes = dynamicClass.toBytecode()
+                ClassUtils.defineClass(null, dynamicClassBytes)
+            }
+
+            run {
+                val dynamicClass = pool.getCtClass("org.agrona.nio.TransportPoller")
+
+                // Get the static initializer
+                val staticInitializer = dynamicClass.classInitializer
+
+                // Remove the existing static initializer
+                dynamicClass.removeConstructor(staticInitializer)
+
+                val initializer = dynamicClass.makeClassInitializer()
+                initializer.insertAfter(
+                    "java.lang.System.err.println(\"updating TransportPoller!\");" +
+                       "java.lang.reflect.Field selectKeysField = null;\n" +
+                       "java.lang.reflect.Field publicSelectKeysField = null;\n" +
+                       "try {\n" +
+                       "    java.nio.channels.Selector selector = java.nio.channels.Selector.open();\n" +
+                       "    Throwable var3 = null;\n" + "\n" +
+                       "    try {\n" +
+                       "        Class clazz = Class.forName(\"sun.nio.ch.SelectorImpl\", false, ClassLoader.getSystemClassLoader());\n" +
+                       "        if (clazz.isAssignableFrom(selector.getClass())) {\n" +
+                       "            selectKeysField = java.lang.SelectorImplAccessory.getKey(\"selectedKeys\");\n" +
+                       "            publicSelectKeysField = java.lang.SelectorImplAccessory.getKey(\"publicSelectedKeys\");\n" +
+                       "        }\n" +
+                       "    } catch (Throwable var21) {\n" +
+                       "        var3 = var21;\n" +
+                       "        throw var21;\n" +
+                       "    } finally {\n" +
+                       "        if (selector != null) {\n" +
+                       "            if (var3 != null) {\n" +
+                       "                try {\n" +
+                       "                    selector.close();\n" +
+                       "                } catch (Throwable var20) {\n" +
+                       "                    var3.addSuppressed(var20);\n" +
+                       "                }\n" +
+                       "            } else {\n" +
+                       "                selector.close();\n" +
+                       "            }\n" +
+                       "       }\n" +
+                       "    }\n" +
+                       "} catch (Exception var23) {\n" +
+                       "    org.agrona.LangUtil.rethrowUnchecked(var23);\n" +
+                       "} finally {\n" +
+                       "    org.agrona.nio.TransportPoller.SELECTED_KEYS_FIELD = selectKeysField;\n" +
+                       "    org.agrona.nio.TransportPoller.PUBLIC_SELECTED_KEYS_FIELD = publicSelectKeysField;\n" +
+                       "}"
+                )
+
+
+                // perform pre-verification for the modified method
+                initializer.methodInfo.rebuildStackMapForME(pool)
+
+                val dynamicClassBytes = dynamicClass.toBytecode()
+                ClassUtils.defineClass(ClassLoader.getSystemClassLoader(), dynamicClassBytes)
+            }
+
+        } catch (e: Exception) {
+            throw RuntimeException("Could not fix Aeron TransportPoller", e)
+        }
+    }
+}

src/dorkbox/network/aeron/mediaDriver/ClientIpcDriver.kt

@@ -1,118 +0,0 @@
-/*
- * Copyright 2021 dorkbox, llc
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package dorkbox.network.aeron.mediaDriver
-
-import dorkbox.network.aeron.AeronDriver
-import dorkbox.network.aeron.mediaDriver.MediaDriverConnection.Companion.uri
-import dorkbox.network.connection.ListenerManager
-import dorkbox.network.exceptions.ClientRetryException
-import dorkbox.network.exceptions.ClientTimedOutException
-import mu.KLogger
-import java.lang.Thread.sleep
-import java.util.concurrent.*
-
-/**
- * For a client, the streamId specified here MUST be manually flipped because they are in the perspective of the SERVER
- * NOTE: IPC connection will ALWAYS have a timeout of 10 second to connect. This is IPC, it should connect fast
- */
-internal open class ClientIpcDriver(streamId: Int,
-                                    sessionId: Int,
-                                    localSessionId: Int) :
-    MediaDriverClient(
-        port = streamId,
-        streamId = streamId,
-        remoteSessionId = sessionId,
-        localSessionId = localSessionId,
-        connectionTimeoutSec = 10,
-        isReliable = true
-    ) {
-
-    var success: Boolean = false
-    override val type = "ipc"
-
-    override val subscriptionPort: Int = localSessionId
-
-    /**
-     * Set up the subscription + publication channels to the server
-     *
-     * @throws ClientRetryException if we need to retry to connect
-     * @throws ClientTimedOutException if we cannot connect to the server in the designated time
-     */
-    fun build(aeronDriver: AeronDriver, logger: KLogger) {
-        // Create a publication at the given address and port, using the given stream ID.
-        // Note: The Aeron.addPublication method will block until the Media Driver acknowledges the request or a timeout occurs.
-        val publicationUri = uri("ipc", remoteSessionId)
-
-        // Create a subscription at the given address and port, using the given stream ID.
-        val subscriptionUri = uri("ipc", 0)
-
-        if (logger.isTraceEnabled) {
-            logger.trace("IPC client pub URI: ${publicationUri.build()}")
-            logger.trace("IPC server sub URI: ${subscriptionUri.build()}")
-        }
-
-        var success = false
-
-        // NOTE: Handlers are called on the client conductor thread. The client conductor thread expects handlers to do safe
-        //  publication of any state to other threads and not be long running or re-entrant with the client.
-
-        // For publications, if we add them "too quickly" (faster than the 'linger' timeout), Aeron will throw exceptions.
-        //      ESPECIALLY if it is with the same streamID
-        // this check is in the "reconnect" logic
-
-        val publication = aeronDriver.addPublication(publicationUri, streamId)
-        val subscription = aeronDriver.addSubscription(subscriptionUri, localSessionId)
-
-
-        // always include the linger timeout, so we don't accidentally kill ourself by taking too long
-        val timoutInNanos = TimeUnit.SECONDS.toNanos(connectionTimeoutSec.toLong()) + aeronDriver.getLingerNs()
-        val startTime = System.nanoTime()
-
-        while (System.nanoTime() - startTime < timoutInNanos) {
-            if (publication.isConnected) {
-                success = true
-                break
-            }
-
-            sleep(500L)
-        }
-        if (!success) {
-            subscription.close()
-            publication.close()
-
-            val clientTimedOutException = ClientTimedOutException("Cannot create publication IPC connection to server")
-            ListenerManager.cleanAllStackTrace(clientTimedOutException)
-            throw clientTimedOutException
-        }
-
-        this.success = true
-        this.subscription = subscription
-        this.publication = publication
-    }
-
-    override val info : String by lazy {
-        if (sessionId != AeronDriver.RESERVED_SESSION_ID_INVALID) {
-            "[$sessionId] IPC connection established to [$streamId|$subscriptionPort]"
-        } else {
-            "Connecting handshake to IPC [$streamId|$subscriptionPort]"
-        }
-    }
-
-    override fun toString(): String {
-        return info
-    }
-}

src/dorkbox/network/aeron/mediaDriver/ClientUdpDriver.kt

@@ -1,146 +0,0 @@
-/*
- * Copyright 2021 dorkbox, llc
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package dorkbox.network.aeron.mediaDriver
-
-import dorkbox.netUtil.IPv4
-import dorkbox.netUtil.IPv6
-import dorkbox.network.aeron.AeronDriver
-import dorkbox.network.aeron.mediaDriver.MediaDriverConnection.Companion.uriEndpoint
-import dorkbox.network.connection.ListenerManager
-import dorkbox.network.exceptions.ClientRetryException
-import dorkbox.network.exceptions.ClientTimedOutException
-import mu.KLogger
-import java.lang.Thread.sleep
-import java.net.Inet4Address
-import java.net.Inet6Address
-import java.net.InetAddress
-import java.util.concurrent.*
-
-/**
- * For a client, the ports specified here MUST be manually flipped because they are in the perspective of the SERVER.
- * A connection timeout of 0, means to wait forever
- */
-internal class ClientUdpDriver(val address: InetAddress, val addressString: String,
-                               port: Int,
-                               streamId: Int,
-                               sessionId: Int,
-                               localSessionId: Int,
-                               connectionTimeoutSec: Int = 0,
-                               isReliable: Boolean) :
-    MediaDriverClient(port, streamId, sessionId, localSessionId, connectionTimeoutSec, isReliable) {
-
-    var success: Boolean = false
-    override val type: String by lazy {
-        if (address is Inet4Address) {
-            "IPv4"
-        } else {
-            "IPv6"
-        }
-    }
-
-    override val subscriptionPort: Int by lazy {
-        val addressesAndPorts = subscription.localSocketAddresses()
-        val first = addressesAndPorts.first()
-
-        // split
-        val splitPoint = first.lastIndexOf(':')
-        val port = first.substring(splitPoint+1)
-        port.toInt()
-    }
-
-    /**
-     * @throws ClientRetryException if we need to retry to connect
-     * @throws ClientTimedOutException if we cannot connect to the server in the designated time
-     */
-    @Suppress("DuplicatedCode")
-    fun build(aeronDriver: AeronDriver, logger: KLogger) {
-        var success = false
-
-        // NOTE: Handlers are called on the client conductor thread. The client conductor thread expects handlers to do safe
-        //  publication of any state to other threads and not be long running or re-entrant with the client.
-
-        // on close, the publication CAN linger (in case a client goes away, and then comes back)
-        // AERON_PUBLICATION_LINGER_TIMEOUT, 5s by default (this can also be set as a URI param)
-
-        // Create a publication at the given address and port, using the given stream ID.
-        // Note: The Aeron.addPublication method will block until the Media Driver acknowledges the request or a timeout occurs.
-        val publicationUri = uriEndpoint("udp", remoteSessionId, isReliable, address, addressString, port)
-        logger.trace("client pub URI: $type ${publicationUri.build()}")
-
-        // For publications, if we add them "too quickly" (faster than the 'linger' timeout), Aeron will throw exceptions.
-        //      ESPECIALLY if it is with the same streamID. This was noticed as a problem with IPC
-        val publication = aeronDriver.addPublication(publicationUri, streamId)
-
-
-        val localAddresses = publication.localSocketAddresses().first()
-        // split
-        val splitPoint = localAddresses.lastIndexOf(':')
-        val localAddressString = localAddresses.substring(0, splitPoint)
-
-
-        // the subscription here is WILDCARD
-        val localAddress = if (address is Inet6Address) {
-            IPv6.toAddress(localAddressString)!!
-        } else {
-            IPv4.toAddress(localAddressString)!!
-        }
-
-        // Create a subscription the given address and port, using the given stream ID.
-        val subscriptionUri = uriEndpoint("udp", localSessionId, isReliable, localAddress, localAddressString, 0)
-        logger.trace("client sub URI: $type ${subscriptionUri.build()}")
-
-        val subscription = aeronDriver.addSubscription(subscriptionUri, streamId)
-
-        // always include the linger timeout, so we don't accidentally kill ourself by taking too long
-        val timoutInNanos = TimeUnit.SECONDS.toNanos(connectionTimeoutSec.toLong()) + aeronDriver.getLingerNs()
-        val startTime = System.nanoTime()
-
-        while (System.nanoTime() - startTime < timoutInNanos) {
-            if (publication.isConnected) {
-                success = true
-                break
-            }
-
-            sleep(500L)
-        }
-
-        if (!success) {
-            subscription.close()
-            publication.close()
-
-            val ex = ClientTimedOutException("Cannot create publication to $type $addressString in $connectionTimeoutSec seconds")
-            ListenerManager.cleanAllStackTrace(ex)
-            throw ex
-        }
-
-        this.success = true
-        this.publication = publication
-        this.subscription = subscription
-    }
-
-    override val info: String by lazy {
-        if (sessionId != AeronDriver.RESERVED_SESSION_ID_INVALID) {
-            "$addressString [$port|$subscriptionPort] [$streamId|$sessionId] (reliable:$isReliable)"
-        } else {
-            "Connecting handshake to $addressString [$port|$subscriptionPort] [$streamId|*] (reliable:$isReliable)"
-        }
-    }
-
-    override fun toString(): String {
-        return info
-    }
-}

src/dorkbox/network/aeron/mediaDriver/MediaDriverClient.kt

@@ -1,34 +0,0 @@
-/*
- * Copyright 2020 dorkbox, llc
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-@file:Suppress("DuplicatedCode")
-
-package dorkbox.network.aeron.mediaDriver
-
-import io.aeron.Publication
-import io.aeron.Subscription
-
-abstract class MediaDriverClient(val port: Int,
-                                 val streamId: Int,
-                                 val remoteSessionId: Int,
-                                 val localSessionId: Int,
-                                 val connectionTimeoutSec: Int,
-                                 val isReliable: Boolean) : MediaDriverConnection {
-
-    lateinit var subscription: Subscription
-    lateinit var publication: Publication
-
-    abstract val subscriptionPort: Int
-}

src/dorkbox/network/aeron/mediaDriver/MediaDriverConnectInfo.kt

@@ -1,16 +0,0 @@
-package dorkbox.network.aeron.mediaDriver
-
-import io.aeron.Publication
-import io.aeron.Subscription
-import java.net.InetAddress
-
-data class MediaDriverConnectInfo(val subscription: Subscription,
-                                  val publication: Publication,
-                                  val subscriptionPort: Int,
-                                  val publicationPort: Int,
-                                  val streamId: Int,
-                                  val sessionId: Int,
-                                  val isReliable: Boolean,
-                                  val remoteAddress: InetAddress?,
-                                  val remoteAddressString: String,
-)

src/dorkbox/network/aeron/mediaDriver/MediaDriverConnection.kt

@@ -1,65 +0,0 @@
-/*
- * Copyright 2020 dorkbox, llc
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-@file:Suppress("DuplicatedCode")
-
-package dorkbox.network.aeron.mediaDriver
-
-import dorkbox.network.aeron.AeronDriver
-import io.aeron.ChannelUriStringBuilder
-import java.net.Inet4Address
-import java.net.InetAddress
-
-interface MediaDriverConnection {
-
-    val type: String
-
-    // We don't use 'suspend' for these, because we have to pump events from a NORMAL thread. If there are any suspend points, there is
-    // the potential for a live-lock due to coroutine scheduling
-    val info : String
-
-    companion object {
-        fun uri(type: String, sessionId: Int, isReliable: Boolean? = null): ChannelUriStringBuilder {
-            val builder = ChannelUriStringBuilder().media(type)
-            if (isReliable != null) {
-                builder.reliable(isReliable)
-            }
-
-            if (sessionId != AeronDriver.RESERVED_SESSION_ID_INVALID) {
-                builder.sessionId(sessionId)
-            }
-
-            return builder
-        }
-
-        fun uriEndpoint(type: String, sessionId: Int, isReliable: Boolean, address: InetAddress, addressString: String, port: Int): ChannelUriStringBuilder {
-            val builder = uri(type, sessionId, isReliable)
-
-            if (address is Inet4Address) {
-                builder.endpoint("$addressString:$port")
-            } else {
-                // IPv6 requires the address to be bracketed by [...]
-                if (addressString[0] == '[') {
-                    builder.endpoint("$addressString:$port")
-                } else {
-                    // there MUST be [] surrounding the IPv6 address for aeron to like it!
-                    builder.endpoint("[$addressString]:$port")
-                }
-            }
-
-            return builder
-        }
-    }
-}

src/dorkbox/network/aeron/mediaDriver/ServerIpcDriver.kt

@@ -1,63 +0,0 @@
-/*
- * Copyright 2021 dorkbox, llc
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package dorkbox.network.aeron.mediaDriver
-
-import dorkbox.network.aeron.AeronDriver
-import dorkbox.network.aeron.mediaDriver.MediaDriverConnection.Companion.uri
-import mu.KLogger
-
-/**
- * For a client, the streamId specified here MUST be manually flipped because they are in the perspective of the SERVER
- * NOTE: IPC connection will ALWAYS have a timeout of 10 second to connect. This is IPC, it should connect fast
- */
-internal open class ServerIpcDriver(streamId: Int,
-                                    sessionId: Int) :
-    MediaDriverServer(0, streamId, sessionId, 10, true) {
-
-
-    var success: Boolean = false
-    override val type = "ipc"
-
-    /**
-     * Setup the subscription + publication channels on the server.
-     *
-     * serverAddress is ignored for IPC
-     */
-     fun build(aeronDriver: AeronDriver, logger: KLogger) {
-        // Create a subscription at the given address and port, using the given stream ID.
-        val subscriptionUri = uri("ipc", sessionId)
-
-        if (logger.isTraceEnabled) {
-            logger.trace("IPC server sub URI: ${subscriptionUri.build()}")
-        }
-
-        success = true
-        subscription = aeronDriver.addSubscription(subscriptionUri, streamId)
-    }
-
-    override val info : String by lazy {
-        if (sessionId != AeronDriver.RESERVED_SESSION_ID_INVALID) {
-            "[$sessionId] IPC listening on [$streamId] [$sessionId]"
-        } else {
-            "Listening handshake on IPC [$streamId] [$sessionId]"
-        }
-    }
-
-    override fun toString(): String {
-        return info
-    }
-}

src/dorkbox/network/aeron/mediaDriver/ServerUdpDriver.kt

@@ -1,81 +0,0 @@
-/*
- * Copyright 2021 dorkbox, llc
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package dorkbox.network.aeron.mediaDriver
-
-import dorkbox.netUtil.IP
-import dorkbox.netUtil.IPv4
-import dorkbox.netUtil.IPv6
-import dorkbox.network.aeron.AeronDriver
-import dorkbox.network.aeron.mediaDriver.MediaDriverConnection.Companion.uriEndpoint
-import mu.KLogger
-import java.net.Inet4Address
-import java.net.InetAddress
-
-/**
- * For a client, the ports specified here MUST be manually flipped because they are in the perspective of the SERVER.
- * A connection timeout of 0, means to wait forever
- */
-internal open class ServerUdpDriver(val listenAddress: InetAddress,
-                                    port: Int,
-                                    streamId: Int,
-                                    sessionId: Int,
-                                    connectionTimeoutSec: Int,
-                                    isReliable: Boolean) :
-    MediaDriverServer(port, streamId, sessionId, connectionTimeoutSec, isReliable) {
-
-
-    var success: Boolean = false
-    override val type = "udp"
-
-    fun build(aeronDriver: AeronDriver, logger: KLogger) {
-        // Create a subscription at the given address and port, using the given stream ID.
-        val subscriptionUri = uriEndpoint("udp", sessionId, isReliable, listenAddress, IP.toString(listenAddress), port)
-
-        if (logger.isTraceEnabled) {
-            if (listenAddress is Inet4Address) {
-                logger.trace("IPV4 server sub URI: ${subscriptionUri.build()}")
-            } else {
-                logger.trace("IPV6 server sub URI: ${subscriptionUri.build()}")
-            }
-        }
-
-        this.success = true
-        this.subscription = aeronDriver.addSubscription(subscriptionUri, streamId)
-    }
-
-    override val info: String by lazy {
-        val address = if (listenAddress == IPv4.WILDCARD || listenAddress == IPv6.WILDCARD) {
-            if (listenAddress == IPv4.WILDCARD) {
-                listenAddress.hostAddress
-            } else {
-                IPv4.WILDCARD.hostAddress + "/" + listenAddress.hostAddress
-            }
-        } else {
-            IP.toString(listenAddress)
-        }
-
-        if (sessionId != AeronDriver.RESERVED_SESSION_ID_INVALID) {
-            "Listening on $address [$port] [$streamId|$sessionId] (reliable:$isReliable)"
-        } else {
-            "Listening handshake on $address [$port] [$streamId|*] (reliable:$isReliable)"
-        }
-    }
-
-    override fun toString(): String {
-        return info
-    }
-}

src/dorkbox/network/aeron/mediaDriver/UdpMediaDriverPairedConnection.kt

@@ -1,43 +0,0 @@
-/*
- * Copyright 2021 dorkbox, llc
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package dorkbox.network.aeron.mediaDriver
-
-import io.aeron.Publication
-import java.net.InetAddress
-
-/**
- * This represents the connection PAIR between a server<->client
- * A connection timeout of 0, means to wait forever
- */
-internal class UdpMediaDriverPairedConnection(
-    listenAddress: InetAddress,
-    val remoteAddress: InetAddress,
-    val remoteAddressString: String,
-    val publicationPort: Int,
-    subscriptionPort: Int,
-    streamId: Int,
-    sessionId: Int,
-    connectionTimeoutSec: Int,
-    isReliable: Boolean,
-    val publication: Publication
-) :
-    ServerUdpDriver(listenAddress, subscriptionPort, streamId, sessionId, connectionTimeoutSec, isReliable) {
-
-    override fun toString(): String {
-        return "$remoteAddressString [$port|$publicationPort] [$streamId|$sessionId] (reliable:$isReliable)"
-    }
-}

src/dorkbox/network/aeron/package-info.java

@@ -0,0 +1,17 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package dorkbox.network.aeron;

src/dorkbox/network/connection/Connection.kt

@@ -1,5 +1,5 @@
 /*
- * Copyright 2020 dorkbox, llc
+ * Copyright 2023 dorkbox, llc
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -15,61 +15,102 @@
  */
 package dorkbox.network.connection
 
-import dorkbox.network.handshake.ConnectionCounts
-import dorkbox.network.handshake.RandomId65kAllocator
+import dorkbox.network.Client
+import dorkbox.network.Server
+import dorkbox.network.aeron.AeronDriver.Companion.sessionIdAllocator
+import dorkbox.network.aeron.AeronDriver.Companion.streamIdAllocator
+import dorkbox.network.connection.buffer.BufferedMessages
+import dorkbox.network.connection.buffer.BufferedSession
 import dorkbox.network.ping.Ping
-import dorkbox.network.ping.PingManager
 import dorkbox.network.rmi.RmiSupportConnection
-import io.aeron.FragmentAssembler
-import io.aeron.Publication
-import io.aeron.Subscription
+import io.aeron.Image
+import io.aeron.logbuffer.FragmentHandler
 import io.aeron.logbuffer.Header
+import io.aeron.protocol.DataHeaderFlyweight
 import kotlinx.atomicfu.atomic
 import kotlinx.atomicfu.getAndUpdate
-import kotlinx.coroutines.runBlocking
 import org.agrona.DirectBuffer
-import java.lang.Thread.sleep
-import java.net.InetAddress
-import java.util.concurrent.*
+import javax.crypto.SecretKey
 
 /**
- * This connection is established once the registration information is validated, and the various connect/filter checks have passed
+ * This connection is established once the registration information is validated, and the various connect/filter checks have passed.
+ *
+ * Connections are also BUFFERED, meaning that if the connection between a client-server goes down because of a network glitch, then the
+ * data being sent is not lost (it is buffered) and then re-sent once a new connection has the same UUID within the timout period.
+ *
+ * References to the old connection will also redirect to the new connection.
  */
 open class Connection(connectionParameters: ConnectionParams<*>) {
-    private var messageHandler: FragmentAssembler
-    internal val subscription: Subscription
-    internal val publication: Publication
+    private val messageHandler: FragmentHandler
 
     /**
-     * The publication port (used by aeron) for this connection. This is from the perspective of the server!
+     * The specific connection details for this connection!
+     *
+     * NOTE: remember, the connection details are for the connection, but the toString() info is reversed for the client
+     *     (so that we can line-up client/server connection logs)
      */
-    private val subscriptionPort: Int
-    private val publicationPort: Int
+    val info = connectionParameters.connectionInfo
 
     /**
-     * the stream id of this connection. Can be 0 for IPC connections
+     * the endpoint associated with this connection
      */
-    val streamId: Int
+    internal val endPoint = connectionParameters.endPoint
+
+    internal val subscription = info.sub
+    internal val publication = info.pub
+    private lateinit var image: Image
+
+    // only accessed on a single thread!
+    private val connectionExpirationTimoutNanos = endPoint.config.connectionExpirationTimoutNanos
+    // the timeout starts from when the connection is first created, so that we don't get "instant" timeouts when the server rejects a connection
+    private var connectionTimeoutTimeNanos = System.nanoTime()
 
     /**
-     * the session id of this connection. This value is UNIQUE
+     * There can be concurrent writes to the network stack, at most 1 per connection. Each connection has its own logic on the remote endpoint,
+     * and can have its own back-pressure.
      */
-    val id: Int
+    internal val sendIdleStrategy = endPoint.config.sendIdleStrategy
 
     /**
-     * the remote address, as a string. Will be null for IPC connections
+     * This is the client UUID. This is useful determine if the same client is connecting multiple times to a server (instead of only using IP address)
      */
-    val remoteAddress: InetAddress?
+    val uuid = connectionParameters.publicKey
 
     /**
-     * the remote address, as a string. Will be "ipc" for IPC connections
+     * The unique session id of this connection, assigned by the server.
+     *
+     * Specifically this is the subscription session ID for the server
      */
-    val remoteAddressString: String
+    val id = if (endPoint::class.java == Client::class.java) {
+        info.sessionIdPub
+    } else {
+        info.sessionIdSub
+    }
+
+    /**
+     * The tag name for a connection permits an INCOMING client to define a custom string. The max length is 32
+     */
+    val tag = info.tagName
+
+    /**
+     * The remote address, as a string. Will be null for IPC connections
+     */
+    val remoteAddress = info.remoteAddress
+
+    /**
+     * The remote address, as a string. Will be "IPC" for IPC connections
+     */
+    val remoteAddressString = info.remoteAddressString
+
+    /**
+     * The remote port. Will be 0 for IPC connections
+     */
+    val remotePort = info.portPub
 
     /**
      * @return true if this connection is an IPC connection
      */
-    val isIpc = connectionParameters.connectionInfo.remoteAddress == null
+    val isIpc = info.isIpc
 
     /**
      * @return true if this connection is a network connection
@@ -77,9 +118,26 @@ open class Connection(connectionParameters: ConnectionParams<*>) {
     val isNetwork = !isIpc
 
     /**
-     * the endpoint associated with this connection
+     * used when the connection is buffered
      */
-    internal val endPoint = connectionParameters.endPoint
+    private val bufferedSession: BufferedSession
+
+    /**
+     * used to determine if this connection will have buffered messages enabled or not.
+     */
+    internal val enableBufferedMessages = connectionParameters.enableBufferedMessages
+
+    /**
+     * The largest size a SINGLE message via AERON can be. Because the maximum size we can send in a "single fragment" is the
+     * publication.maxPayloadLength() function (which is the MTU length less header). We could depend on Aeron for fragment reassembly,
+     * but that has a (very low) maximum reassembly size -- so we have our own mechanism for object fragmentation/assembly, which
+     * is (in reality) only limited by available ram.
+     */
+    internal val maxMessageSize = if (isNetwork) {
+        endPoint.config.networkMtuSize - DataHeaderFlyweight.HEADER_LENGTH
+    } else {
+        endPoint.config.ipcMtuSize - DataHeaderFlyweight.HEADER_LENGTH
+    }
 
 
     private val listenerManager = atomic<ListenerManager<Connection>?>(null)
@@ -87,105 +145,120 @@ open class Connection(connectionParameters: ConnectionParams<*>) {
 
     private val isClosed = atomic(false)
 
-    // enableNotifyDisconnect : we don't always want to enable notifications on disconnect
-    internal var closeAction: suspend () -> Unit = {}
-
-    // only accessed on a single thread!
-    private var connectionLastCheckTimeNanos = 0L
-    private var connectionTimeoutTimeNanos = 0L
-
-    // always offset by the linger amount, since we cannot act faster than the linger for adding/removing publications
-    private val connectionCheckIntervalNanos = connectionParameters.endPoint.config.connectionCheckIntervalNanos + endPoint.aeronDriver.getLingerNs()
-    private val connectionExpirationTimoutNanos = connectionParameters.endPoint.config.connectionExpirationTimoutNanos + endPoint.aeronDriver.getLingerNs()
-
 
     // while on the CLIENT, if the SERVER's ecc key has changed, the client will abort and show an error.
-    private val remoteKeyChanged = connectionParameters.publicKeyValidation == PublicKeyValidationState.TAMPERED
-
-    // The IV for AES-GCM must be 12 bytes, since it's 4 (salt) + 8 (external counter) + 4 (GCM counter)
-    // The 12 bytes IV is created during connection registration, and during the AES-GCM crypto, we override the last 8 with this
-    // counter, which is also transmitted as an optimized int. (which is why it starts at 0, so the transmitted bytes are small)
-//    private val aes_gcm_iv = atomic(0)
+    internal val remoteKeyChanged = connectionParameters.publicKeyValidation == PublicKeyValidationState.TAMPERED
 
     /**
      * Methods supporting Remote Method Invocation and Objects
      */
     val rmi: RmiSupportConnection<out Connection>
 
-    // a record of how many messages are in progress of being sent. When closing the connection, this number must be 0
-    private val messagesInProgress = atomic(0)
-
-    // we customize the toString() value for this connection, and it's just better to cache it's value (since it's a modestly complex string)
+    // we customize the toString() value for this connection, and it's just better to cache its value (since it's a modestly complex string)
     private val toString0: String
 
+    /**
+     * @return the AES key
+     */
+    internal val cryptoKey: SecretKey = connectionParameters.cryptoKey
+
+    // The IV for AES-GCM must be 12 bytes, since it's 4 (salt) + 4 (external counter) + 4 (GCM counter)
+    // The 12 bytes IV is created during connection registration, and during the AES-GCM crypto, we override the last 8 with this
+    // counter, which is also transmitted as an optimized int. (which is why it starts at 0, so the transmitted bytes are small)
+    internal val aes_gcm_iv = atomic(0)
+
+    // Used to track that this connection WILL be closed, but has not yet been closed.
+    @Volatile
+    internal var closeRequested = false
 
 
     init {
-        val connectionInfo = connectionParameters.connectionInfo
+        // NOTE: subscriptions (ie: reading from buffers, etc) are not thread safe!  Because it is ambiguous HOW EXACTLY they are unsafe,
+        //  we exclusively read from the DirectBuffer on a single thread.
 
-        id = connectionInfo.sessionId // NOTE: this is UNIQUE per server!
+        // NOTE: Handlers are called on the client conductor thread. The client conductor thread expects handlers to do safe
+        //  publication of any state to other threads and not be:
+        //   - long running
+        //   - re-entrant with the client
+        messageHandler = FragmentHandler { buffer: DirectBuffer, offset: Int, length: Int, header: Header ->
+            // Subscriptions are NOT multi-thread safe, so only processed on the thread that calls .poll()!
+            endPoint.dataReceive(buffer, offset, length, header, this@Connection)
+        }
 
-        subscription = connectionInfo.subscription
-        publication = connectionInfo.publication
-
-        // can only get this AFTER we have built the sub/pub
-        streamId = connectionInfo.streamId // NOTE: this is UNIQUE per server!
-        subscriptionPort = connectionInfo.subscriptionPort
-        publicationPort = connectionInfo.publicationPort
-
-        remoteAddress = connectionInfo.remoteAddress
-        remoteAddressString = connectionInfo.remoteAddressString
-
-        toString0 = "[${id}/${streamId}] $remoteAddressString [$publicationPort|$subscriptionPort]"
-
-        messageHandler = FragmentAssembler { buffer: DirectBuffer, offset: Int, length: Int, header: Header ->
-            // this is processed on the thread that calls "poll". Subscriptions are NOT multi-thread safe!
-
-            // NOTE: subscriptions (ie: reading from buffers, etc) are not thread safe!  Because it is ambiguous HOW EXACTLY they are unsafe,
-            //  we exclusively read from the DirectBuffer on a single thread.
-            endPoint.processMessage(buffer, offset, length, header, this@Connection)
+        bufferedSession = when (endPoint) {
+            is Server -> endPoint.bufferedManager.onConnect(this)
+            is Client -> endPoint.bufferedManager!!.onConnect(this)
+            else  -> throw RuntimeException("Unable to determine type, aborting!")
         }
 
         @Suppress("LeakingThis")
-        rmi = connectionParameters.endPoint.rmiConnectionSupport.getNewRmiSupport(this)
+        rmi = endPoint.rmiConnectionSupport.getNewRmiSupport(this)
+
+        // For toString() and logging
+        toString0 = info.getLogInfo(logger.isDebugEnabled)
     }
 
     /**
-     * @return true if the remote public key changed. This can be useful if specific actions are necessary when the key has changed.
+     * When this is called, we should always have a subscription image!
      */
-    fun hasRemoteKeyChanged(): Boolean {
-        return remoteKeyChanged
+    internal fun setImage() {
+        var triggered = false
+        while (subscription.hasNoImages()) {
+            triggered = true
+            Thread.sleep(50)
+        }
+
+        if (triggered) {
+            logger.error("Delay while configuring subscription!")
+        }
+
+        image = subscription.imageAtIndex(0)
     }
 
-//    /**
-//     * This is the per-message sequence number.
-//     *
-//     * The IV for AES-GCM must be 12 bytes, since it's 4 (salt) + 4 (external counter) + 4 (GCM counter)
-//     * The 12 bytes IV is created during connection registration, and during the AES-GCM crypto, we override the last 8 with this
-//     * counter, which is also transmitted as an optimized int. (which is why it starts at 0, so the transmitted bytes are small)
-//     */
-//    fun nextGcmSequence(): Long {
-//        return aes_gcm_iv.getAndIncrement()
-//    }
-//
-//    /**
-//     * @return the AES key. key=32 byte, iv=12 bytes (AES-GCM implementation).
-//     */
-//    fun cryptoKey(): SecretKey {
-//        TODO()
-////        return channelWrapper.cryptoKey()
-//    }
-
-
-
-
     /**
      * Polls the AERON media driver subscription channel for incoming messages
      */
     internal fun poll(): Int {
-        // NOTE: regarding fragment limit size. Repeated calls to '.poll' will reassemble a fragment.
-        //   `.poll(handler, 4)` == `.poll(handler, 2)` + `.poll(handler, 2)`
-        return subscription.poll(messageHandler, 1)
+        return image.poll(messageHandler, 1)
+    }
+
+
+
+    /**
+     * Safely sends objects to a destination, if `abortEarly` is true, there are no retries if sending the message fails.
+     *
+     * @return true if the message was successfully sent, false otherwise. Exceptions are caught and NOT rethrown!
+     */
+    internal fun send(message: Any, abortEarly: Boolean): Boolean {
+        if (logger.isTraceEnabled) {
+            // The handshake sessionId IS NOT globally unique
+            // don't automatically create the lambda when trace is disabled! Because this uses 'outside' scoped info, it's a new lambda each time!
+            if (logger.isTraceEnabled) {
+                logger.trace("[$toString0] send: ${message.javaClass.simpleName} : $message")
+            }
+        }
+
+        val success = endPoint.write(message, publication, sendIdleStrategy, this@Connection, maxMessageSize, abortEarly)
+
+        return if (!success && message !is DisconnectMessage) {
+            // queue up the messages, because we couldn't write them for whatever reason!
+            // NEVER QUEUE THE DISCONNECT MESSAGE!
+            bufferedSession.queueMessage(this@Connection, message, abortEarly)
+        } else {
+            success
+        }
+    }
+
+    private fun sendNoBuffer(message: Any): Boolean {
+        if (logger.isTraceEnabled) {
+            // The handshake sessionId IS NOT globally unique
+            // don't automatically create the lambda when trace is disabled! Because this uses 'outside' scoped info, it's a new lambda each time!
+            if (logger.isTraceEnabled) {
+                logger.trace("[$toString0] send: ${message.javaClass.simpleName} : $message")
+            }
+        }
+
+        return endPoint.write(message, publication, sendIdleStrategy, this@Connection, maxMessageSize, false)
     }
 
     /**
@@ -194,11 +267,20 @@ open class Connection(connectionParameters: ConnectionParams<*>) {
      * @return true if the message was successfully sent, false otherwise. Exceptions are caught and NOT rethrown!
      */
     fun send(message: Any): Boolean {
-        messagesInProgress.getAndIncrement()
-        val success = endPoint.send(message, publication, this)
-        messagesInProgress.getAndDecrement()
+        return send(message, false)
+    }
 
-        return success
+
+    /**
+     * Safely sends objects to a destination, where the callback is notified once the remote endpoint has received the message.
+     *
+     * This is to guarantee happens-before, and using this will depend upon APP+NETWORK latency, and is (by design) not as performant as
+     * sending a regular message!
+     *
+     * @return true if the message was successfully sent, false otherwise. Exceptions are caught and NOT rethrown!
+     */
+    fun send(message: Any, onSuccessCallback: Connection.() -> Unit): Boolean {
+        return sendSync(message, onSuccessCallback)
     }
 
     /**
@@ -206,17 +288,19 @@ open class Connection(connectionParameters: ConnectionParams<*>) {
      *
      * @return true if the message was successfully sent by aeron
      */
-    suspend fun ping(pingTimeoutSeconds: Int = PingManager.DEFAULT_TIMEOUT_SECONDS, function: suspend Ping.() -> Unit): Boolean {
-        return endPoint.ping(this, pingTimeoutSeconds, function)
+    fun ping(function: Ping.() -> Unit = {}): Boolean {
+        return sendPing(function)
     }
 
     /**
-     * A message in progress means that we have requested to to send an object over the network, but it hasn't finished sending over the network
+     * This is the per-message sequence number.
      *
-     * @return the number of messages in progress for this connection.
+     * The IV for AES-GCM must be 12 bytes, since it's 4 (salt) + 4 (external counter) + 4 (GCM counter)
+     * The 12 bytes IV is created during connection registration, and during the AES-GCM crypto, we override the last 8 with this
+     * counter, which is also transmitted as an optimized int. (which is why it starts at 0, so the transmitted bytes are small)
      */
-    fun messagesInProgress(): Int {
-        return messagesInProgress.value
+    internal fun nextGcmSequence(): Int {
+        return aes_gcm_iv.getAndIncrement()
     }
 
     /**
@@ -229,10 +313,10 @@ open class Connection(connectionParameters: ConnectionParams<*>) {
      * (via connection.addListener), meaning that ONLY that listener attached to
      * the connection is notified on that event (ie, admin type listeners)
      */
-    suspend fun onDisconnect(function: suspend Connection.() -> Unit) {
+    fun onDisconnect(function: Connection.() -> Unit) {
         // make sure we atomically create the listener manager, if necessary
         listenerManager.getAndUpdate { origManager ->
-            origManager ?: ListenerManager(logger)
+            origManager ?: ListenerManager(logger, endPoint.eventDispatch)
         }
 
         listenerManager.value!!.onDisconnect(function)
@@ -241,10 +325,10 @@ open class Connection(connectionParameters: ConnectionParams<*>) {
     /**
      * Adds a function that will be called only for this connection, when a client/server receives a message
      */
-    suspend fun <MESSAGE> onMessage(function: suspend Connection.(MESSAGE) -> Unit) {
+    fun <MESSAGE> onMessage(function: Connection.(MESSAGE) -> Unit) {
         // make sure we atomically create the listener manager, if necessary
         listenerManager.getAndUpdate { origManager ->
-            origManager ?: ListenerManager(logger)
+            origManager ?: ListenerManager(logger, endPoint.eventDispatch)
         }
 
         listenerManager.value!!.onMessage(function)
@@ -255,43 +339,67 @@ open class Connection(connectionParameters: ConnectionParams<*>) {
      *
      * This is ALWAYS called on a new dispatch
      */
-    internal suspend fun notifyOnMessage(message: Any): Boolean {
+    internal fun notifyOnMessage(message: Any): Boolean {
         return listenerManager.value?.notifyOnMessage(this, message) ?: false
     }
 
+    internal fun sendBufferedMessages() {
+        if (enableBufferedMessages) {
+            val bufferedMessage = BufferedMessages()
+            val numberDrained = bufferedSession.pendingMessagesQueue.drainTo(bufferedMessage.messages)
+
+            if (numberDrained > 0) {
+                // now send all buffered/pending messages
+                if (logger.isDebugEnabled) {
+                    logger.debug("Sending buffered messages: ${bufferedSession.pendingMessagesQueue.size}")
+                }
+
+                sendNoBuffer(bufferedMessage)
+            }
+        }
+    }
+
+    /**
+     * @return true if this connection has had close() called
+     */
+    fun isClosed(): Boolean {
+        return isClosed.value
+    }
+
+    /**
+     * Is this a "dirty" disconnect, meaning that it has timed out, but not been explicitly closed
+     */
+    internal fun isDirtyClose(): Boolean {
+        return !closeRequested && !isClosed() && isClosedWithTimeout()
+    }
+
+    /**
+     * Is this connection considered still safe for polling (or rather, has it been closed in an unusual way?)
+     */
+    internal fun canPoll(): Boolean {
+        return !closeRequested && !isClosed() && !isClosedWithTimeout()
+    }
+
     /**
      * We must account for network blips. The blips will be recovered by aeron, but we want to make sure that we are actually
      * disconnected for a set period of time before we start the close process for a connection
      *
      * @return `true` if this connection has been closed via aeron
      */
-    fun isClosedViaAeron(): Boolean {
+    internal fun isClosedWithTimeout(): Boolean {
         // we ONLY want to actually, legit check, 1 time every XXX ms.
         val now = System.nanoTime()
 
-        if (now - connectionLastCheckTimeNanos < connectionCheckIntervalNanos) {
-            // we haven't waited long enough for another check. always return false (true means we are closed)
-            return false
-        }
-        connectionLastCheckTimeNanos = now
-
         // as long as we are connected, we reset the state, so that if there is a network blip, we want to make sure that it is
-        // a network blip for a while, instead of just once or twice. (which can happen)
+        // a network blip for a while, instead of just once or twice. (which WILL happen)
         if (subscription.isConnected && publication.isConnected) {
             // reset connection timeout
-            connectionTimeoutTimeNanos = 0L
+            connectionTimeoutTimeNanos = now
 
             // we are still connected (true means we are closed)
             return false
         }
 
-        //
-        // aeron is not connected
-        //
-
-        if (connectionTimeoutTimeNanos == 0L) {
-            connectionTimeoutTimeNanos = now
-        }
 
         // make sure that our "isConnected" state lasts LONGER than the expiry timeout!
 
@@ -300,92 +408,154 @@ open class Connection(connectionParameters: ConnectionParams<*>) {
         return now - connectionTimeoutTimeNanos >= connectionExpirationTimoutNanos
     }
 
+
     /**
      * Closes the connection, and removes all connection specific listeners
      */
     fun close() {
-        close(enableRemove = true)
+        close(sendDisconnectMessage = true,
+              closeEverything = true)
     }
 
     /**
      * Closes the connection, and removes all connection specific listeners
      */
-    internal fun close(enableRemove: Boolean) {
+    internal fun close(sendDisconnectMessage: Boolean, closeEverything: Boolean) {
         // there are 2 ways to call close.
         //   MANUALLY
         //   When a connection is disconnected via a timeout/expire.
+
         // the compareAndSet is used to make sure that if we call close() MANUALLY, (and later) when the auto-cleanup/disconnect is called -- it doesn't
         // try to do it again.
+        closeRequested = true
 
+        // make sure that EVERYTHING before "close()" runs before we do.
+        // If there are multiple clients/servers sharing the same NetworkPoller -- then they will wait on each other!
+        val close = endPoint.eventDispatch.CLOSE
+        if (!close.isDispatch()) {
+            close.launch {
+                close(sendDisconnectMessage = sendDisconnectMessage, closeEverything = closeEverything)
+            }
+            return
+        }
+
+        closeImmediately(sendDisconnectMessage = sendDisconnectMessage, closeEverything = closeEverything)
+    }
+
+
+    // connection.close() -> this
+    // endpoint.close() -> connection.close() -> this
+    internal fun closeImmediately(sendDisconnectMessage: Boolean, closeEverything: Boolean) {
         // the server 'handshake' connection info is cleaned up with the disconnect via timeout/expire.
-        if (isClosed.compareAndSet(expect = false, update = true)) {
-            val aeronLogInfo = "${id}/${streamId}"
-            logger.debug {"[$aeronLogInfo] connection closing"}
+        if (!isClosed.compareAndSet(expect = false, update = true)) {
+            logger.debug("[$toString0] connection ignoring close request.")
+            return
+        }
 
-            subscription.close()
+        if (logger.isDebugEnabled) {
+            logger.debug("[$toString0] connection closing. sendDisconnectMessage=$sendDisconnectMessage, closeEverything=$closeEverything")
+        }
 
-            // send out a "close" message. MAYBE it gets to the remote endpoint, maybe not. If it DOES, then the remote endpoint starts
-            // the close process faster.
-            try {
-                endPoint.send(CloseMessage(), publication, this)
-            } catch (ignored: Exception) {
+        // make sure to save off the RMI objects for session management
+        if (!closeEverything) {
+            when (endPoint) {
+                is Server     -> endPoint.bufferedManager.onDisconnect(this)
+                is Client -> endPoint.bufferedManager!!.onDisconnect(this)
+                else  -> throw RuntimeException("Unable to determine type, aborting!")
             }
+        }
 
-
-
-            val timoutInNanos = TimeUnit.SECONDS.toNanos(endPoint.config.connectionCloseTimeoutInSeconds.toLong())
-            var closeTimeoutTime = System.nanoTime()
-
-            // we do not want to close until AFTER all publications have been sent. Calling this WITHOUT waiting will instantly stop everything
-            // we want a timeout-check, otherwise this will run forever
-            while (messagesInProgress.value != 0 && System.nanoTime() - closeTimeoutTime < timoutInNanos) {
-                sleep(50)
+        if (!closeEverything) {
+            when (endPoint) {
+                is Server -> endPoint.bufferedManager.onDisconnect(this)
+                is Client -> endPoint.bufferedManager!!.onDisconnect(this)
+                else  -> throw RuntimeException("Unable to determine type, aborting!")
             }
+        }
 
-            // on close, we want to make sure this file is DELETED!
-            val logFile = endPoint.aeronDriver.getMediaDriverPublicationFile(publication.registrationId())
-            publication.close()
+        // on close, we want to make sure this file is DELETED!
+        try {
+            // we might not be able to close this connection!!
+            endPoint.aeronDriver.close(subscription, toString0)
+        }
+        catch (e: Exception) {
+            endPoint.listenerManager.notifyError(e)
+        }
 
-
-            closeTimeoutTime = System.nanoTime()
-            while (logFile.exists() && System.nanoTime() - closeTimeoutTime < timoutInNanos) {
-                if (logFile.delete()) {
-                    break
+        // notify the remote endPoint that we are closing
+        // we send this AFTER we close our subscription (so that no more messages will be received, when the remote end ping-pong's this message back)
+        if (sendDisconnectMessage) {
+            if (publication.isConnected) {
+                if (logger.isDebugEnabled) {
+                    logger.debug("Sending disconnect message to ${endPoint.otherTypeName}")
                 }
-                sleep(100)
-            }
 
-            if (logFile.exists()) {
-                logger.error("[$aeronLogInfo] Unable to delete aeron publication log on close: $logFile")
-            }
+                // sometimes the remote end has already disconnected, THERE WILL BE ERRORS if this happens (but they are ok)
+                if (closeEverything) {
+                    send(DisconnectMessage.CLOSE_EVERYTHING, true)
+                } else {
+                    send(DisconnectMessage.CLOSE_SIMPLE, true)
+                }
 
-            if (enableRemove) {
-                endPoint.removeConnection(this)
+                // wait for .5 seconds to (help) make sure that the messages are sent before shutdown! This is not guaranteed!
+                if (logger.isDebugEnabled) {
+                    logger.debug("Waiting for disconnect message to send")
+                }
+                Thread.sleep(500L)
+            } else {
+                if (logger.isDebugEnabled) {
+                    logger.debug("Publication is not connected with ${endPoint.otherTypeName}, not sending disconnect message.")
+                }
             }
+        }
 
-            // NOTE: notifyDisconnect() is called inside closeAction()!!
+        // on close, we want to make sure this file is DELETED!
+        try {
+            // we might not be able to close this connection.
+            endPoint.aeronDriver.close(publication, toString0)
+        }
+        catch (e: Exception) {
+            endPoint.listenerManager.notifyError(e)
+        }
 
-            // This is set by the client/server so if there is a "connect()" call in the the disconnect callback, we can have proper
-            // lock-stop ordering for how disconnect and connect work with each-other
-            runBlocking {
-                closeAction()
+        // NOTE: any waiting RMI messages that are in-flight will terminate when they time-out (and then do nothing)
+        // if there are errors within the driver, we do not want to notify disconnect, as we will automatically reconnect.
+        endPoint.listenerManager.notifyDisconnect(this)
+
+        endPoint.removeConnection(this)
+
+
+        val connection = this
+        if (endPoint.isServer()) {
+            // clean up the resources associated with this connection when it's closed
+            if (logger.isDebugEnabled) {
+                logger.debug("[${connection}] freeing resources")
             }
-            logger.debug {"[$aeronLogInfo] connection closed"}
+            sessionIdAllocator.free(info.sessionIdPub)
+            sessionIdAllocator.free(info.sessionIdSub)
+
+            streamIdAllocator.free(info.streamIdPub)
+            streamIdAllocator.free(info.streamIdSub)
+
+            if (remoteAddress != null) {
+                // unique for UDP endpoints
+                (endPoint as Server).handshake.connectionsPerIpCounts.decrementSlow(remoteAddress)
+            }
+        }
+
+        if (logger.isDebugEnabled) {
+            logger.debug("[$toString0] connection closed")
         }
     }
 
-    // called in postCloseAction(), so we don't expose our internal listenerManager
-    internal suspend fun doNotifyDisconnect() {
+
+    // called in a ListenerManager.notifyDisconnect(), so we don't expose our internal listenerManager
+    internal fun notifyDisconnect() {
         val connectionSpecificListenerManager = listenerManager.value
-        connectionSpecificListenerManager?.notifyDisconnect(this@Connection)
+        connectionSpecificListenerManager?.directNotifyDisconnect(this@Connection)
     }
 
 
-    //
-    //
-    // Generic object methods
-    //
-    //
     override fun toString(): String {
         return toString0
     }
@@ -409,17 +579,93 @@ open class Connection(connectionParameters: ConnectionParams<*>) {
         return id == other1.id
     }
 
-    // cleans up the connection information
-    internal fun cleanup(connectionsPerIpCounts: ConnectionCounts, sessionIdAllocator: RandomId65kAllocator, streamIdAllocator: RandomId65kAllocator) {
-        sessionIdAllocator.free(id)
+    internal fun receiveSendSync(sendSync: SendSync) {
+        if (sendSync.message != null) {
+            // this is on the "remote end".
+            sendSync.message = null
 
-        if (isIpc) {
-            streamIdAllocator.free(publicationPort)
-            streamIdAllocator.free(subscriptionPort)
+            if (!send(sendSync)) {
+                logger.error("Error returning send-sync: $sendSync")
+            }
         } else {
-            // unique for UDP endpoints
-            connectionsPerIpCounts.decrementSlow(remoteAddress!!)
-            streamIdAllocator.free(streamId)
+            // this is on the "local end" when the response comes back
+            val responseId = sendSync.id
+
+            // process the ping message so that our ping callback does something
+
+            // this will be null if the ping took longer than XXX seconds and was cancelled
+            val result = EndPoint.responseManager.removeWaiterCallback<Connection.() -> Unit>(responseId, logger)
+            if (result != null) {
+                result(this)
+            } else {
+                logger.error("Unable to receive send-sync, there was no waiting response for $sendSync ($responseId)")
+            }
         }
     }
+
+
+    /**
+     * Safely sends objects to a destination, the callback is notified once the remote endpoint has received the message.
+     *
+     * This is to guarantee happens-before, and using this will depend upon APP+NETWORK latency, and is (by design) not as performant as
+     * sending a regular message!
+     *
+     * @return true if the message was successfully sent, false otherwise. Exceptions are caught and NOT rethrown!
+     */
+    private fun sendSync(message: Any, onSuccessCallback: Connection.() -> Unit): Boolean {
+        val id = EndPoint.responseManager.prepWithCallback(logger, onSuccessCallback)
+
+        val sendSync = SendSync()
+        sendSync.message = message
+        sendSync.id = id
+
+        // if there is no sync response EVER, it means that the connection is in a critically BAD state!
+        // eventually, all the ping/sync replies (or, in our case, the replies that have timed out) will
+        // become recycled.
+        // Is it a memory-leak? No, because the memory will **EVENTUALLY** get freed.
+
+        return send(sendSync, false)
+    }
+
+
+    internal fun receivePing(ping: Ping) {
+        if (ping.pongTime == 0L) {
+            // this is on the "remote end".
+            ping.pongTime = System.currentTimeMillis()
+
+            if (!send(ping)) {
+                logger.error("Error returning ping: $ping")
+            }
+        } else {
+            // this is on the "local end" when the response comes back
+            ping.finishedTime = System.currentTimeMillis()
+
+            val responseId = ping.packedId
+
+            // process the ping message so that our ping callback does something
+
+            // this will be null if the ping took longer than XXX seconds and was cancelled
+            val result = EndPoint.responseManager.removeWaiterCallback<Ping.() -> Unit>(responseId, logger)
+            if (result != null) {
+                result(ping)
+            } else {
+                logger.error("Unable to receive ping, there was no waiting response for $ping ($responseId)")
+            }
+        }
+    }
+
+    private fun sendPing(function: Ping.() -> Unit): Boolean {
+        val id = EndPoint.responseManager.prepWithCallback(logger, function)
+
+        val ping = Ping()
+        ping.packedId = id
+        ping.pingTime = System.currentTimeMillis()
+
+        // if there is no ping response EVER, it means that the connection is in a critically BAD state!
+        // eventually, all the ping replies (or, in our case, the RMI replies that have timed out) will
+        // become recycled.
+        // Is it a memory-leak? No, because the memory will **EVENTUALLY** get freed.
+
+        return send(ping)
+    }
 }

src/dorkbox/network/connection/ConnectionManager.kt

@@ -1,78 +0,0 @@
-/*
- * Copyright 2020 dorkbox, llc
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package dorkbox.network.connection
-
-import dorkbox.collections.ConcurrentEntry
-import dorkbox.collections.ConcurrentIterator
-import dorkbox.collections.ConcurrentIterator.headREF
-
-// .equals() compares the identity on purpose,this because we cannot create two separate objects that are somehow equal to each other.
-@Suppress("UNCHECKED_CAST")
-internal open class ConnectionManager<CONNECTION: Connection>() {
-
-    private val connections = ConcurrentIterator<CONNECTION>()
-
-
-    /**
-     * Invoked when aeron successfully connects to a remote address.
-     *
-     * @param connection the connection to add
-     */
-    fun add(connection: CONNECTION) {
-        connections.add(connection)
-    }
-
-    /**
-     * Removes a custom connection to the server.
-     *
-     * This should only be used in situations where there can be DIFFERENT types of connections (such as a 'web-based' connection) and
-     * you want *this* server instance to manage listeners + message dispatch
-     *
-     * @param connection the connection to remove
-     */
-    fun remove(connection: CONNECTION) {
-        connections.remove(connection)
-    }
-
-    /**
-     * Performs an action on each connection in the list.
-     */
-    inline fun forEach(function: (connection: CONNECTION) -> Unit) {
-        // access a snapshot (single-writer-principle)
-        val head = headREF.get(connections) as ConcurrentEntry<CONNECTION>?
-        var current: ConcurrentEntry<CONNECTION>? = head
-
-        var connection: CONNECTION
-        while (current != null) {
-            // Concurrent iteration...
-            connection = current.value
-            current = current.next()
-
-            function(connection)
-        }
-    }
-
-    fun connectionCount(): Int {
-        return connections.size()
-    }
-
-    /**
-     * Removes all connections. Does not call close or anything else on them
-     */
-    fun clear() {
-        connections.clear()
-    }
-}

src/dorkbox/network/connection/ConnectionParams.kt

@@ -1,5 +1,5 @@
 /*
- * Copyright 2020 dorkbox, llc
+ * Copyright 2023 dorkbox, llc
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -15,10 +15,14 @@
  */
 package dorkbox.network.connection
 
-import dorkbox.network.aeron.mediaDriver.MediaDriverConnectInfo
+import dorkbox.network.handshake.PubSub
+import javax.crypto.spec.SecretKeySpec
 
 data class ConnectionParams<CONNECTION : Connection>(
+    val publicKey: ByteArray,
     val endPoint: EndPoint<CONNECTION>,
-    val connectionInfo: MediaDriverConnectInfo,
-    val publicKeyValidation: PublicKeyValidationState
+    val connectionInfo: PubSub,
+    val publicKeyValidation: PublicKeyValidationState,
+    val enableBufferedMessages: Boolean,
+    val cryptoKey: SecretKeySpec
 )

src/dorkbox/network/connection/CryptoManagement.kt

@@ -1,5 +1,5 @@
 /*
- * Copyright 2020 dorkbox, llc
+ * Copyright 2023 dorkbox, llc
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -16,13 +16,13 @@
 package dorkbox.network.connection
 
 import dorkbox.bytes.Hash
-import dorkbox.bytes.toHexString
+import dorkbox.hex.toHexString
 import dorkbox.network.handshake.ClientConnectionInfo
 import dorkbox.network.serialization.AeronInput
 import dorkbox.network.serialization.AeronOutput
 import dorkbox.network.serialization.SettingsStore
 import dorkbox.util.entropy.Entropy
-import mu.KLogger
+import org.slf4j.Logger
 import java.math.BigInteger
 import java.net.InetAddress
 import java.security.KeyFactory
@@ -42,33 +42,39 @@ import javax.crypto.spec.SecretKeySpec
 /**
  * Management for all the crypto stuff used
  */
-internal class CryptoManagement(val logger: KLogger,
+internal class CryptoManagement(val logger: Logger,
                                 private val settingsStore: SettingsStore,
                                 type: Class<*>,
                                 private val enableRemoteSignatureValidation: Boolean) {
+    companion object {
+        private val X25519 = "X25519"
+        const val curve25519 = "curve25519"
+
+        const val GCM_IV_LENGTH_BYTES = 12 // 12 bytes for a 96-bit IV
+        const val GCM_TAG_LENGTH_BITS = 128
+        const val AES_ALGORITHM = "AES/GCM/NoPadding"
+
+        val NOCRYPT = SecretKeySpec(ByteArray(1), "NOCRYPT")
+        val secureRandom = SecureRandom()
+    }
+
 
-    private val X25519 = "X25519"
     private val X25519KeySpec = NamedParameterSpec(X25519)
 
     private val keyFactory = KeyFactory.getInstance(X25519)  // key size is 32 bytes (256 bits)
     private val keyAgreement = KeyAgreement.getInstance("XDH")
 
-    private val aesCipher = Cipher.getInstance("AES/GCM/NoPadding")
+    private val aesCipher = Cipher.getInstance(AES_ALGORITHM)
+
 
-    companion object {
-        const val curve25519 = "curve25519"
-        const val GCM_IV_LENGTH_BYTES = 12
-        const val GCM_TAG_LENGTH_BITS = 128
-    }
 
     val privateKey: XECPrivateKey
     val publicKey: XECPublicKey
 
+    // These are both 32 bytes long (256 bits)
     val privateKeyBytes: ByteArray
     val publicKeyBytes: ByteArray
 
-    val secureRandom = SecureRandom(settingsStore.getSalt())
-
     private val iv = ByteArray(GCM_IV_LENGTH_BYTES)
     val cryptOutput = AeronOutput()
     val cryptInput = AeronInput()
@@ -78,12 +84,17 @@ internal class CryptoManagement(val logger: KLogger,
             logger.warn("WARNING: Disabling remote key validation is a security risk!!")
         }
 
+        secureRandom.setSeed(settingsStore.salt)
+
         // initialize the private/public keys used for negotiating ECC handshakes
         // these are ONLY used for IP connections. LOCAL connections do not need a handshake!
-        var privateKeyBytes = settingsStore.getPrivateKey()
-        var publicKeyBytes = settingsStore.getPublicKey()
+        val privateKeyBytes: ByteArray
+        val publicKeyBytes: ByteArray
 
-        if (privateKeyBytes == null || publicKeyBytes == null) {
+        if (settingsStore.validKeys()) {
+            privateKeyBytes = settingsStore.privateKey
+            publicKeyBytes = settingsStore.publicKey
+        } else {
             try {
                 // seed our RNG based off of this and create our ECC keys
                 val seedBytes = Entropy["There are no ECC keys for the ${type.simpleName} yet"]
@@ -98,8 +109,8 @@ internal class CryptoManagement(val logger: KLogger,
                 privateKeyBytes = xdhPrivate.scalar
 
                 // save to properties file
-                settingsStore.savePrivateKey(privateKeyBytes)
-                settingsStore.savePublicKey(publicKeyBytes)
+                settingsStore.privateKey = privateKeyBytes
+                settingsStore.publicKey = publicKeyBytes
             } catch (e: Exception) {
                 val message = "Unable to initialize/generate ECC keys. FORCED SHUTDOWN."
                 logger.error(message, e)
@@ -107,14 +118,12 @@ internal class CryptoManagement(val logger: KLogger,
             }
         }
 
-        publicKeyBytes!!
-
         logger.info("ECC public key: ${publicKeyBytes.toHexString()}")
 
         this.publicKey = keyFactory.generatePublic(XECPublicKeySpec(X25519KeySpec, BigInteger(publicKeyBytes))) as XECPublicKey
         this.privateKey = keyFactory.generatePrivate(XECPrivateKeySpec(X25519KeySpec, privateKeyBytes)) as XECPrivateKey
 
-        this.privateKeyBytes = privateKeyBytes!!
+        this.privateKeyBytes = privateKeyBytes
         this.publicKeyBytes = publicKeyBytes
     }
 
@@ -167,10 +176,77 @@ internal class CryptoManagement(val logger: KLogger,
         return PublicKeyValidationState.VALID
     }
 
+    private fun makeInfo(serverPublicKeyBytes: ByteArray, secretKey: SecretKeySpec): ClientConnectionInfo {
+        val sessionIdPub = cryptInput.readInt()
+        val sessionIdSub = cryptInput.readInt()
+        val streamIdPub = cryptInput.readInt()
+        val streamIdSub = cryptInput.readInt()
+        val regDetailsSize = cryptInput.readInt()
+        val sessionTimeout = cryptInput.readLong()
+        val bufferedMessages = cryptInput.readBoolean()
+        val regDetails = cryptInput.readBytes(regDetailsSize)
+
+        // now save data off
+        return ClientConnectionInfo(
+            sessionIdPub = sessionIdPub,
+            sessionIdSub = sessionIdSub,
+            streamIdPub = streamIdPub,
+            streamIdSub = streamIdSub,
+            publicKey = serverPublicKeyBytes,
+            sessionTimeout = sessionTimeout,
+            bufferedMessages = bufferedMessages,
+            kryoRegistrationDetails = regDetails,
+            secretKey = secretKey)
+    }
+
+    // NOTE: ALWAYS CALLED ON THE SAME THREAD! (from the server, mutually exclusive calls to decrypt)
+    fun nocrypt(
+        sessionIdPub: Int,
+        sessionIdSub: Int,
+        streamIdPub: Int,
+        streamIdSub: Int,
+        sessionTimeout: Long,
+        bufferedMessages: Boolean,
+        kryoRegDetails: ByteArray
+    ): ByteArray {
+
+        return try {
+            // now create the byte array that holds all our data
+            cryptOutput.reset()
+            cryptOutput.writeInt(sessionIdPub)
+            cryptOutput.writeInt(sessionIdSub)
+            cryptOutput.writeInt(streamIdPub)
+            cryptOutput.writeInt(streamIdSub)
+            cryptOutput.writeInt(kryoRegDetails.size)
+            cryptOutput.writeLong(sessionTimeout)
+            cryptOutput.writeBoolean(bufferedMessages)
+            cryptOutput.writeBytes(kryoRegDetails)
+
+            cryptOutput.toBytes()
+        } catch (e: Exception) {
+            logger.error("Error during AES encrypt", e)
+            ByteArray(0)
+        }
+    }
+
+    // NOTE: ALWAYS CALLED ON THE SAME THREAD! (from the client, mutually exclusive calls to encrypt)
+    fun nocrypt(registrationData: ByteArray, serverPublicKeyBytes: ByteArray): ClientConnectionInfo? {
+        return try {
+            // The message was intended for this client. Try to parse it as one of the available message types.
+            // this message is NOT-ENCRYPTED!
+            cryptInput.buffer = registrationData
+
+            makeInfo(serverPublicKeyBytes, NOCRYPT)
+        } catch (e: Exception) {
+            logger.error("Error during IPC decrypt!", e)
+            null
+        }
+    }
+
     /**
      * Generate the AES key based on ECDH
      */
-    private fun generateAesKey(remotePublicKeyBytes: ByteArray, bytesA: ByteArray, bytesB: ByteArray): SecretKeySpec {
+    internal fun generateAesKey(remotePublicKeyBytes: ByteArray, bytesA: ByteArray, bytesB: ByteArray): SecretKeySpec {
         val clientPublicKey = keyFactory.generatePublic(XECPublicKeySpec(X25519KeySpec, BigInteger(remotePublicKeyBytes)))
         keyAgreement.init(privateKey)
         keyAgreement.doPhase(clientPublicKey, true)
@@ -187,25 +263,32 @@ internal class CryptoManagement(val logger: KLogger,
     }
 
     // NOTE: ALWAYS CALLED ON THE SAME THREAD! (from the server, mutually exclusive calls to decrypt)
-    fun encrypt(clientPublicKeyBytes: ByteArray,
-                subscriptionPort: Int,
-                connectionSessionId: Int,
-                connectionStreamId: Int,
-                kryoRegDetails: ByteArray): ByteArray {
+    fun encrypt(
+        cryptoSecretKey: SecretKeySpec,
+        sessionIdPub: Int,
+        sessionIdSub: Int,
+        streamIdPub: Int,
+        streamIdSub: Int,
+        sessionTimeout: Long,
+        bufferedMessages: Boolean,
+        kryoRegDetails: ByteArray
+    ): ByteArray {
 
         try {
-            val secretKeySpec = generateAesKey(clientPublicKeyBytes, clientPublicKeyBytes, publicKeyBytes)
             secureRandom.nextBytes(iv)
 
             val gcmParameterSpec = GCMParameterSpec(GCM_TAG_LENGTH_BITS, iv)
-            aesCipher.init(Cipher.ENCRYPT_MODE, secretKeySpec, gcmParameterSpec)
+            aesCipher.init(Cipher.ENCRYPT_MODE, cryptoSecretKey, gcmParameterSpec)
 
             // now create the byte array that holds all our data
             cryptOutput.reset()
-            cryptOutput.writeInt(connectionSessionId)
-            cryptOutput.writeInt(connectionStreamId)
-            cryptOutput.writeInt(subscriptionPort)
+            cryptOutput.writeInt(sessionIdPub)
+            cryptOutput.writeInt(sessionIdSub)
+            cryptOutput.writeInt(streamIdPub)
+            cryptOutput.writeInt(streamIdSub)
             cryptOutput.writeInt(kryoRegDetails.size)
+            cryptOutput.writeLong(sessionTimeout)
+            cryptOutput.writeBoolean(bufferedMessages)
             cryptOutput.writeBytes(kryoRegDetails)
 
             return iv + aesCipher.doFinal(cryptOutput.toBytes())
@@ -217,7 +300,7 @@ internal class CryptoManagement(val logger: KLogger,
 
     // NOTE: ALWAYS CALLED ON THE SAME THREAD! (from the client, mutually exclusive calls to encrypt)
     fun decrypt(registrationData: ByteArray, serverPublicKeyBytes: ByteArray): ClientConnectionInfo? {
-        try {
+        return try {
             val secretKeySpec = generateAesKey(serverPublicKeyBytes, publicKeyBytes, serverPublicKeyBytes)
 
             // now decrypt the data
@@ -226,21 +309,11 @@ internal class CryptoManagement(val logger: KLogger,
 
             cryptInput.buffer = aesCipher.doFinal(registrationData, GCM_IV_LENGTH_BYTES, registrationData.size - GCM_IV_LENGTH_BYTES)
 
-            val sessionId = cryptInput.readInt()
-            val streamId = cryptInput.readInt()
-            val subscriptionPort = cryptInput.readInt()
-            val regDetailsSize = cryptInput.readInt()
-            val regDetails = cryptInput.readBytes(regDetailsSize)
+            makeInfo(serverPublicKeyBytes, secretKeySpec)
 
-            // now read data off
-            return ClientConnectionInfo(sessionId = sessionId,
-                                        streamId = streamId,
-                                        port = subscriptionPort,
-                                        publicKey = serverPublicKeyBytes,
-                                        kryoRegistrationDetails = regDetails)
         } catch (e: Exception) {
             logger.error("Error during AES decrypt!", e)
-            return null
+            null
         }
     }
 

src/dorkbox/network/connection/DisconnectMessage.kt

@@ -1,5 +1,5 @@
 /*
- * Copyright 2020 dorkbox, llc
+ * Copyright 2023 dorkbox, llc
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -13,13 +13,12 @@
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
-package dorkbox.network.rmi.messages
 
-/**
- * @param rmiId which rmi object was deleted
- */
-data class ConnectionObjectDeleteResponse(val rmiId: Int) : RmiMessage {
-    override fun toString(): String {
-        return "ConnectionObjectDeleteResponse(id: $rmiId)"
+package dorkbox.network.connection
+
+class DisconnectMessage(val closeEverything: Boolean) {
+    companion object {
+        val CLOSE_SIMPLE = DisconnectMessage(false)
+        val CLOSE_EVERYTHING = DisconnectMessage(true)
     }
 }

src/dorkbox/network/connection/EventDispatcher.kt

@@ -0,0 +1,179 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package dorkbox.network.connection
+
+import dorkbox.network.Configuration
+import dorkbox.util.NamedThreadFactory
+import kotlinx.atomicfu.atomic
+import org.slf4j.LoggerFactory
+import java.util.concurrent.*
+
+/**
+ * Event logic throughout the network MUST be run on multiple threads! There are deadlock issues if it is only one, or if the client + server
+ * share an event dispatcher (multiple network restarts were required to check this)
+ *
+ * WARNING: The logic in this class will ONLY work in this class, as it relies on this specific behavior. Do not use it elsewhere!
+ */
+internal class EventDispatcher(val type: String) {
+    enum class EDType {
+        // CLOSE must be last!
+        HANDSHAKE, CONNECT, ERROR, CLOSE
+    }
+
+    internal class ED(private val dispatcher: EventDispatcher, private val type: EDType) {
+        fun launch(function: () -> Unit) {
+            dispatcher.launch(type, function)
+        }
+
+        fun isDispatch(): Boolean {
+            return dispatcher.isDispatch(type)
+        }
+
+        fun shutdownAndWait(timeout: Long, timeoutUnit: TimeUnit) {
+            dispatcher.shutdownAndWait(type, timeout, timeoutUnit)
+        }
+    }
+
+    companion object {
+        private val DEBUG_EVENTS = false
+        private val traceId = atomic(0)
+
+        private val typedEntries: Array<EDType>
+
+        init {
+            typedEntries = EDType.entries.toTypedArray()
+        }
+    }
+
+    private val logger = LoggerFactory.getLogger("$type Dispatch")
+
+    private val threadIds = EDType.entries.map { atomic(0L) }.toTypedArray()
+
+    private val executors = EDType.entries.map { event ->
+        // It CANNOT be the default dispatch because there will be thread starvation
+        // NOTE: THIS CANNOT CHANGE!! IT WILL BREAK EVERYTHING IF IT CHANGES!
+        Executors.newSingleThreadExecutor(
+            NamedThreadFactory(
+                namePrefix = "$type-${event.name}",
+                group = Configuration.networkThreadGroup,
+                threadPriority = Thread.NORM_PRIORITY,
+                daemon = true
+            ) { thread ->
+                // when a new thread is created, assign it to the array
+                threadIds[event.ordinal].lazySet(thread.id)
+            }
+        )
+    }.toTypedArray()
+
+    val HANDSHAKE: ED
+    val CONNECT: ED
+    val ERROR: ED
+    val CLOSE: ED
+
+
+    init {
+        executors.forEachIndexed { _, executor ->
+            executor.submit {
+                // this is to create a new thread only, so that the thread ID can be assigned
+            }
+        }
+
+        HANDSHAKE = ED(this, EDType.HANDSHAKE)
+        CONNECT = ED(this, EDType.CONNECT)
+        ERROR = ED(this, EDType.ERROR)
+        CLOSE = ED(this, EDType.CLOSE)
+    }
+
+
+    /**
+     * Shuts-down each event dispatcher executor, and waits for it to gracefully shutdown. Once shutdown, it cannot be restarted.
+     *
+     * @param timeout how long to wait
+     * @param timeoutUnit what the unit count is
+     */
+    fun shutdownAndWait(timeout: Long, timeoutUnit: TimeUnit) {
+        require(timeout > 0) { logger.error("The EventDispatcher shutdown timeout must be > 0!") }
+
+        HANDSHAKE.shutdownAndWait(timeout, timeoutUnit)
+        CONNECT.shutdownAndWait(timeout, timeoutUnit)
+        ERROR.shutdownAndWait(timeout, timeoutUnit)
+        CLOSE.shutdownAndWait(timeout, timeoutUnit)
+    }
+
+    /**
+     *  Checks if the current execution thread is running inside one of the event dispatchers.
+     */
+    fun isDispatch(): Boolean {
+        val threadId = Thread.currentThread().id
+
+        typedEntries.forEach { event ->
+            if (threadIds[event.ordinal].value == threadId) {
+                return true
+            }
+        }
+
+        return false
+    }
+
+    /**
+     *  Checks if the current execution thread is running inside one of the event dispatchers.
+     */
+    private fun isDispatch(type: EDType): Boolean {
+        val threadId = Thread.currentThread().id
+
+        return threadIds[type.ordinal].value == threadId
+    }
+
+    /**
+     * shuts-down the current execution thread and waits for it complete.
+     */
+    private fun shutdownAndWait(type: EDType, timeout: Long, timeoutUnit: TimeUnit) {
+        executors[type.ordinal].shutdown()
+        executors[type.ordinal].awaitTermination(timeout, timeoutUnit)
+    }
+
+    /**
+     * Each event type runs inside its own thread executor.
+     *
+     * We want EACH event type to run in its own executor... on its OWN thread, in order to prevent deadlocks
+     * This is because there are blocking dependencies: DISCONNECT -> CONNECT.
+     *
+     * If an event is RE-ENTRANT, then it will immediately execute!
+     */
+    private fun launch(event: EDType, function: () -> Unit) {
+        val eventId = event.ordinal
+
+        try {
+            if (DEBUG_EVENTS) {
+                val id = traceId.getAndIncrement()
+                executors[eventId].submit {
+                    if (logger.isDebugEnabled) {
+                        logger.debug("Starting $event : $id")
+                    }
+                    function()
+                    if (logger.isDebugEnabled) {
+                        logger.debug("Finished $event : $id")
+                    }
+                }
+            } else {
+                executors[eventId].submit(function)
+            }
+        } catch (e: Exception) {
+            logger.error("Error during event dispatch!", e)
+        }
+    }
+}

src/dorkbox/network/connection/IpInfo.kt

@@ -0,0 +1,231 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package dorkbox.network.connection
+
+import dorkbox.netUtil.IPv4
+import dorkbox.netUtil.IPv6
+import dorkbox.network.ServerConfiguration
+import dorkbox.network.connection.IpInfo.Companion.IpListenType.IPC
+import dorkbox.network.connection.IpInfo.Companion.IpListenType.IPWildcard
+import dorkbox.network.connection.IpInfo.Companion.IpListenType.IPv4Wildcard
+import dorkbox.network.connection.IpInfo.Companion.IpListenType.IPv6Wildcard
+import java.net.Inet4Address
+import java.net.Inet6Address
+import java.net.InetAddress
+
+internal class IpInfo(config: ServerConfiguration) {
+
+
+    companion object {
+        enum class IpListenType {
+            IPv4, IPv6, IPv4Wildcard, IPv6Wildcard, IPWildcard, IPC
+        }
+
+        fun isLocalhost(ipAddress: String): Boolean {
+            return when (ipAddress.lowercase()) {
+                "loopback", "localhost", "lo", "127.0.0.1", "::1" -> true
+                else -> false
+            }
+        }
+        fun isWildcard(ipAddress: String): Boolean {
+            return when (ipAddress) {
+                // this is the "wildcard" address. Windows has problems with this.
+                "0", "::", "0.0.0.0", "*" -> true
+                else -> false
+            }
+        }
+
+        fun isWildcard(ipAddress: InetAddress): Boolean {
+            return when (ipAddress) {
+                // this is the "wildcard" address. Windows has problems with this.
+                IPv4.WILDCARD, IPv6.WILDCARD -> true
+                else -> false
+            }
+        }
+
+        fun getWildcard(ipAddress: InetAddress, ipAddressString: String, shouldBeIpv4: Boolean): String {
+            return if (isWildcard(ipAddress)) {
+                if (shouldBeIpv4) {
+                    IPv4.WILDCARD_STRING
+                } else {
+                    IPv6.WILDCARD_STRING
+                }
+            } else {
+                ipAddressString
+            }
+        }
+
+        fun formatCommonAddress(ipAddress: String, isIpv4: Boolean, elseAction: () -> InetAddress?): InetAddress? {
+            return if (isLocalhost(ipAddress)) {
+                if (isIpv4) { IPv4.LOCALHOST } else { IPv6.LOCALHOST }
+            } else if (isWildcard(ipAddress)) {
+                if (isIpv4) { IPv4.WILDCARD } else { IPv6.WILDCARD }
+            } else if (IPv4.isValid(ipAddress)) {
+                IPv4.toAddress(ipAddress)!!
+            } else if (IPv6.isValid(ipAddress)) {
+                IPv6.toAddress(ipAddress)!!
+            } else {
+                elseAction()
+            }
+        }
+
+        fun formatCommonAddressString(ipAddress: String, isIpv4: Boolean, elseAction: () -> String = { ipAddress }): String {
+            return if (isLocalhost(ipAddress)) {
+                if (isIpv4) { IPv4.LOCALHOST_STRING } else { IPv6.LOCALHOST_STRING }
+            } else if (isWildcard(ipAddress)) {
+                if (isIpv4) { IPv4.WILDCARD_STRING } else { IPv6.WILDCARD_STRING }
+            } else if (IPv4.isValid(ipAddress)) {
+                ipAddress
+            } else if (IPv6.isValid(ipAddress)) {
+                ipAddress
+            } else {
+                elseAction()
+            }
+        }
+    }
+
+
+
+
+    val ipType: IpListenType
+
+
+    val listenAddress: InetAddress?
+    val listenAddressString: String
+    val formattedListenAddressString: String
+    val listenAddressStringPretty: String
+    val isReliable = config.isReliable
+
+    val isIpv4: Boolean
+
+    init {
+        val canUseIPv4 = config.enableIPv4 && IPv4.isAvailable
+        val canUseIPv6 = config.enableIPv6 && IPv6.isAvailable
+
+        // localhost/loopback IP might not always be 127.0.0.1 or ::1
+        // We want to listen on BOTH IPv4 and IPv6 (config option lets us configure this) we listen in IPv6 WILDCARD
+        var listenAddress: InetAddress?
+        var ip46Wildcard = false
+
+        when {
+            canUseIPv4 && canUseIPv6 ->  {
+                // if it's not a valid IP, the lambda will return null
+                listenAddress = formatCommonAddress(config.listenIpAddress, false) { null }
+
+                if (listenAddress == null) {
+                    listenAddress = formatCommonAddress(config.listenIpAddress, true) { null }
+                } else {
+                    ip46Wildcard = true
+                }
+            }
+            canUseIPv4 -> {
+                // if it's not a valid IP, the lambda will return null
+                listenAddress = formatCommonAddress(config.listenIpAddress, true) { null }
+            }
+            canUseIPv6 -> {
+                // if it's not a valid IP, the lambda will return null
+                listenAddress = formatCommonAddress(config.listenIpAddress, false) { null }
+            }
+            else -> {
+                listenAddress = null
+            }
+        }
+
+        this.listenAddress = listenAddress
+
+
+        isIpv4 = listenAddress is Inet4Address
+
+        // if we are IPv6 WILDCARD -- then our listen-address must ALSO be IPv6, even if our connection is via IPv4
+        when (listenAddress) {
+            IPv6.WILDCARD -> {
+                ipType = if (ip46Wildcard) {
+                    IPWildcard
+                } else {
+                    IPv6Wildcard
+                }
+
+                listenAddressString = IPv6.WILDCARD_STRING
+                formattedListenAddressString = if (listenAddressString[0] == '[') {
+                    listenAddressString
+                } else {
+                    // there MUST be [] surrounding the IPv6 address for aeron to like it!
+                    "[$listenAddressString]"
+                }
+            }
+            IPv4.WILDCARD -> {
+                ipType = IPv4Wildcard
+                listenAddressString = IPv4.WILDCARD_STRING
+                formattedListenAddressString = listenAddressString
+            }
+            is Inet6Address -> {
+                ipType = IpListenType.IPv6
+                listenAddressString = IPv6.toString(listenAddress)
+                formattedListenAddressString = if (listenAddressString[0] == '[') {
+                    listenAddressString
+                } else {
+                    // there MUST be [] surrounding the IPv6 address for aeron to like it!
+                    "[$listenAddressString]"
+                }
+            }
+            is Inet4Address -> {
+                ipType = IpListenType.IPv4
+                listenAddressString = IPv4.toString(listenAddress)
+                formattedListenAddressString = listenAddressString
+            }
+            else -> {
+                ipType = IPC
+                listenAddressString = EndPoint.IPC_NAME
+                formattedListenAddressString = listenAddressString
+            }
+        }
+
+        listenAddressStringPretty = when (listenAddress) {
+            IPv4.WILDCARD -> listenAddressString
+            IPv6.WILDCARD -> IPv4.WILDCARD.hostAddress + "/" + listenAddressString
+            else -> listenAddressString
+        }
+    }
+
+    /**
+     * if we are listening on :: (ipv6), and a connection via ipv4 arrives, aeron MUST publish on the IPv4 version
+     */
+    fun getAeronPubAddress(remoteIpv4: Boolean): String {
+        return if (remoteIpv4) {
+            when (ipType) {
+                IPWildcard -> IPv4.WILDCARD_STRING
+                else -> formattedListenAddressString
+            }
+        } else {
+            formattedListenAddressString
+        }
+    }
+
+    /**
+     * if we are listening on :: (ipv6), and a connection via ipv4 arrives, aeron MUST publish on the IPv6 version
+     */
+    fun getAeronSubAddress(remoteIpv4: Boolean): String {
+        return if (remoteIpv4) {
+            when (ipType) {
+                IPWildcard -> IPv4.WILDCARD_STRING
+                else -> formattedListenAddressString
+            }
+        } else {
+            formattedListenAddressString
+        }
+    }
+}

src/dorkbox/network/connection/ListenerManager.kt

@@ -1,5 +1,5 @@
 /*
- * Copyright 2020 dorkbox, llc
+ * Copyright 2023 dorkbox, llc
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -15,22 +15,21 @@
  */
 package dorkbox.network.connection
 
+import dorkbox.classUtil.ClassHelper
+import dorkbox.classUtil.ClassHierarchy
 import dorkbox.collections.IdentityMap
 import dorkbox.network.ipFilter.IpFilterRule
 import dorkbox.os.OS
-import dorkbox.util.classes.ClassHelper
-import dorkbox.util.classes.ClassHierarchy
-import kotlinx.atomicfu.atomic
-import kotlinx.coroutines.runBlocking
-import kotlinx.coroutines.sync.Mutex
-import kotlinx.coroutines.sync.withLock
-import mu.KLogger
 import net.jodah.typetools.TypeResolver
+import org.slf4j.Logger
+import java.net.InetAddress
+import java.util.concurrent.locks.*
+import kotlin.concurrent.write
 
 /**
  * Manages all of the different connect/disconnect/etc listeners
  */
-internal class ListenerManager<CONNECTION: Connection>(private val logger: KLogger) {
+internal class ListenerManager<CONNECTION: Connection>(private val logger: Logger, val eventDispatch: EventDispatcher) {
     companion object {
         /**
          * Specifies the load-factor for the IdentityMap used to manage keeping track of the number of connections + listeners
@@ -42,13 +41,13 @@ internal class ListenerManager<CONNECTION: Connection>(private val logger: KLogg
          *
          * Neither of these are useful in resolving exception handling from a users perspective, and only clutter the stacktrace.
          */
-        fun cleanStackTrace(throwable: Throwable, adjustedStartOfStack: Int = 0) {
+        fun Throwable.cleanStackTrace(adjustedStartOfStack: Int = 0): Throwable {
             // we never care about coroutine stacks, so filter then to start with.
-            val origStackTrace = throwable.stackTrace
+            val origStackTrace = this.stackTrace
             val size = origStackTrace.size
 
             if (size == 0) {
-                return
+                return this
             }
 
             val stackTrace = origStackTrace.filterNot {
@@ -85,15 +84,17 @@ internal class ListenerManager<CONNECTION: Connection>(private val logger: KLogg
             if (newEndIndex > 0) {
                 if (savedFirstStack != null) {
                     // we want to save the FIRST stack frame also, maybe
-                    throwable.stackTrace = savedFirstStack + stackTrace.copyOfRange(newStartIndex, newEndIndex)
+                    this.stackTrace = savedFirstStack + stackTrace.copyOfRange(newStartIndex, newEndIndex)
                 } else {
-                    throwable.stackTrace = stackTrace.copyOfRange(newStartIndex, newEndIndex)
+                    this.stackTrace = stackTrace.copyOfRange(newStartIndex, newEndIndex)
                 }
 
             } else {
                 // keep just one, since it's a stack frame INSIDE our network library, and we need that!
-                throwable.stackTrace = stackTrace.copyOfRange(0, 1)
+                this.stackTrace = stackTrace.copyOfRange(0, 1)
             }
+
+            return this
         }
 
         /**
@@ -101,9 +102,9 @@ internal class ListenerManager<CONNECTION: Connection>(private val logger: KLogg
          *
          * Neither of these are useful in resolving exception handling from a users perspective, and only clutter the stacktrace.
          */
-        fun cleanStackTraceInternal(throwable: Throwable) {
+        fun Throwable.cleanStackTraceInternal() {
             // NOTE: when we remove stuff, we ONLY want to remove the "tail" of the stacktrace, not ALL parts of the stacktrace
-            val stackTrace = throwable.stackTrace
+            val stackTrace = this.stackTrace
             val size = stackTrace.size
 
             if (size == 0) {
@@ -114,7 +115,7 @@ internal class ListenerManager<CONNECTION: Connection>(private val logger: KLogg
             val firstDorkboxIndex = stackTrace.indexOfFirst { it.className.startsWith("dorkbox.network.") }
             val lastDorkboxIndex = stackTrace.indexOfLast { it.className.startsWith("dorkbox.network.") }
 
-            throwable.stackTrace = stackTrace.filterIndexed { index, element ->
+            this.stackTrace = stackTrace.filterIndexed { index, element ->
                 val stackName = element.className
                 if (index <= firstDorkboxIndex && index >= lastDorkboxIndex) {
                     false
@@ -131,65 +132,83 @@ internal class ListenerManager<CONNECTION: Connection>(private val logger: KLogg
          *
          * We only want the error message, because we do something based on it (and the full stack trace is meaningless)
          */
-        fun cleanAllStackTrace(throwable: Throwable) {
-            val stackTrace = throwable.stackTrace
+        fun Throwable.cleanAllStackTrace(): Throwable{
+            val stackTrace = this.stackTrace
             val size = stackTrace.size
 
             if (size == 0) {
-                return
+                return this
             }
 
             // throw everything out
-            throwable.stackTrace = stackTrace.copyOfRange(0, 1)
+            this.stackTrace = stackTrace.copyOfRange(0, 1)
+            return this
+        }
+
+        internal inline fun <reified T: Any> add(thing: T, array: Array<T>): Array<T> {
+            val currentLength: Int = array.size
+
+            // add the new subscription to the END of the array
+            @Suppress("UNCHECKED_CAST")
+            val newMessageArray = array.copyOf(currentLength + 1) as Array<T>
+            newMessageArray[currentLength] = thing
+
+            return newMessageArray
+        }
+
+        internal inline fun <reified T: Any> remove(thing: T, array: Array<T>): Array<T> {
+            // remove the subscription form the array
+            // THIS IS IDENTITY CHECKS, NOT EQUALITY
+            return array.filter { it !== thing }.toTypedArray()
         }
     }
 
-    // initialize a emtpy arrays
-    private val onConnectFilterList = atomic(Array<(CONNECTION.() -> Boolean)>(0) { { true } })
-    private val onConnectFilterMutex = Mutex()
+    // initialize emtpy arrays
+    @Volatile
+    private var onConnectFilterList = Array<((InetAddress, String) -> Boolean)>(0) { { _, _ -> true } }
+    private val onConnectFilterLock = ReentrantReadWriteLock()
 
-    private val onInitList = atomic(Array<suspend (CONNECTION.() -> Unit)>(0) { { } })
-    private val onInitMutex = Mutex()
+    @Volatile
+    private var onConnectBufferedMessageFilterList = Array<((InetAddress?, String) -> Boolean)>(0) { { _, _ -> true } }
+    private val onConnectBufferedMessageFilterLock = ReentrantReadWriteLock()
 
-    private val onConnectList = atomic(Array<suspend (CONNECTION.() -> Unit)>(0) { { } })
-    private val onConnectMutex = Mutex()
+    @Volatile
+    private var onInitList = Array<(CONNECTION.() -> Unit)>(0) { { } }
+    private val onInitLock = ReentrantReadWriteLock()
 
-    private val onDisconnectList = atomic(Array<suspend CONNECTION.() -> Unit>(0) { { } })
-    private val onDisconnectMutex = Mutex()
+    @Volatile
+    private var onConnectList = Array<(CONNECTION.() -> Unit)>(0) { { } }
+    private val onConnectLock = ReentrantReadWriteLock()
 
-    private val onErrorList = atomic(Array<CONNECTION.(Throwable) -> Unit>(0) { {  } })
-    private val onErrorMutex = Mutex()
+    @Volatile
+    private var onDisconnectList = Array<CONNECTION.() -> Unit>(0) { { } }
+    private val onDisconnectLock = ReentrantReadWriteLock()
 
-    private val onErrorGlobalList = atomic(Array<Throwable.() -> Unit>(0) { { } })
-    private val onErrorGlobalMutex = Mutex()
+    @Volatile
+    private var onErrorList = Array<CONNECTION.(Throwable) -> Unit>(0) { {  } }
+    private val onErrorLock = ReentrantReadWriteLock()
 
-    private val onMessageMap = atomic(IdentityMap<Class<*>, Array<suspend CONNECTION.(Any) -> Unit>>(32, LOAD_FACTOR))
-    private val onMessageMutex = Mutex()
+    @Volatile
+    private var onErrorGlobalList = Array<Throwable.() -> Unit>(0) { { } }
+    private val onErrorGlobalLock = ReentrantReadWriteLock()
+
+    @Volatile
+    private var onMessageMap = IdentityMap<Class<*>, Array<CONNECTION.(Any) -> Unit>>(32, LOAD_FACTOR)
+    private val onMessageLock = ReentrantReadWriteLock()
 
     // used to keep a cache of class hierarchy for distributing messages
     private val classHierarchyCache = ClassHierarchy(LOAD_FACTOR)
 
-    private inline fun <reified T> add(thing: T, array: Array<T>): Array<T> {
-        val currentLength: Int = array.size
-
-        // add the new subscription to the array
-        @Suppress("UNCHECKED_CAST")
-        val newMessageArray = array.copyOf(currentLength + 1) as Array<T>
-        newMessageArray[currentLength] = thing
-
-        return newMessageArray
-    }
-
     /**
      * Adds an IP+subnet rule that defines if that IP+subnet is allowed or denied connectivity to this server.
      *
      * If there are no rules added, then all connections are allowed
      * If there are rules added, then a rule MUST be matched to be allowed
      */
-    suspend fun filter(ipFilterRule: IpFilterRule) {
-        filter {
-            // IPC will not filter, so this is OK to coerce to not-null
-            ipFilterRule.matches(remoteAddress!!)
+    fun filter(ipFilterRule: IpFilterRule) {
+        filter { clientAddress, _ ->
+            // IPC will not filter
+            ipFilterRule.matches(clientAddress)
         }
     }
 
@@ -198,18 +217,51 @@ internal class ListenerManager<CONNECTION: Connection>(private val logger: KLogg
      * Adds a function that will be called BEFORE a client/server "connects" with each other, and used to determine if a connection
      * should be allowed
      *
+     * By default, if there are no filter rules, then all connections are allowed to connect
+     * If there are filter rules - then ONLY connections for the filter that returns true are allowed to connect (all else are denied)
+     *
      * It is the responsibility of the custom filter to write the error, if there is one
      *
      * If the function returns TRUE, then the connection will continue to connect.
      * If the function returns FALSE, then the other end of the connection will
      *   receive a connection error
      *
-     * For a server, this function will be called for ALL clients.
+     *
+     * If ANY filter rule that is applied returns true, then the connection is permitted
+     *
+     * This function will be called for **only** network clients (IPC client are excluded)
+     *
+     * @param function clientAddress: UDP connection address
+     *                       tagName: the connection tag name
      */
-    suspend fun filter(function: CONNECTION.() -> Boolean) {
-        onConnectFilterMutex.withLock {
+    fun filter(function: (clientAddress: InetAddress, tagName: String) -> Boolean) {
+        onConnectFilterLock.write {
             // we have to follow the single-writer principle!
-            onConnectFilterList.lazySet(add(function, onConnectFilterList.value))
+            onConnectFilterList = add(function, onConnectFilterList)
+        }
+    }
+
+    /**
+     * Adds a function that will be called BEFORE a client/server "connects" with each other, and used to determine if buffered messages
+     * for a connection should be enabled
+     *
+     * By default, if there are no rules, then all connections will have buffered messages enabled
+     * If there are rules - then ONLY connections for the rule that returns true will have buffered messages enabled (all else are disabled)
+     *
+     * It is the responsibility of the custom filter to write the error, if there is one
+     *
+     * If the function returns TRUE, then the buffered messages for a connection are enabled.
+     * If the function returns FALSE, then the buffered messages for a connection is disabled.
+     *
+     * If ANY rule that is applied returns true, then the buffered messages for a connection are enabled
+     *
+     * @param function clientAddress: not-null when UDP connection, null when IPC connection
+     *                       tagName: the connection tag name
+     */
+    fun enableBufferedMessages(function: (clientAddress: InetAddress?, tagName: String) -> Boolean) {
+        onConnectBufferedMessageFilterLock.write {
+            // we have to follow the single-writer principle!
+            onConnectBufferedMessageFilterList = add(function, onConnectBufferedMessageFilterList)
         }
     }
 
@@ -219,10 +271,10 @@ internal class ListenerManager<CONNECTION: Connection>(private val logger: KLogg
      *
      * For a server, this function will be called for ALL client connections.
      */
-    suspend fun onInit(function: suspend CONNECTION.() -> Unit) {
-        onInitMutex.withLock {
+    fun onInit(function: CONNECTION.() -> Unit) {
+        onInitLock.write {
             // we have to follow the single-writer principle!
-            onInitList.lazySet(add(function, onInitList.value))
+            onInitList = add(function, onInitList)
         }
     }
 
@@ -230,10 +282,10 @@ internal class ListenerManager<CONNECTION: Connection>(private val logger: KLogg
      * Adds a function that will be called when a client/server connection first establishes a connection with the remote end.
      * 'onInit()' callbacks will execute for both the client and server before `onConnect()` will execute will "connects" with each other
      */
-    suspend fun onConnect(function: suspend CONNECTION.() -> Unit) {
-        onConnectMutex.withLock {
+    fun onConnect(function: CONNECTION.() -> Unit) {
+        onConnectLock.write {
             // we have to follow the single-writer principle!
-            onConnectList.lazySet(add(function, onConnectList.value))
+            onConnectList = add(function, onConnectList)
         }
     }
 
@@ -242,10 +294,10 @@ internal class ListenerManager<CONNECTION: Connection>(private val logger: KLogg
      *
      * Do not try to send messages! The connection will already be closed, resulting in an error if you attempt to do so.
      */
-    suspend fun onDisconnect(function: suspend CONNECTION.() -> Unit) {
-        onDisconnectMutex.withLock {
+    fun onDisconnect(function: CONNECTION.() -> Unit) {
+        onDisconnectLock.write {
             // we have to follow the single-writer principle!
-            onDisconnectList.lazySet(add(function, onDisconnectList.value))
+            onDisconnectList = add(function, onDisconnectList)
         }
     }
 
@@ -254,10 +306,10 @@ internal class ListenerManager<CONNECTION: Connection>(private val logger: KLogg
      *
      * The error is also sent to an error log before this method is called.
      */
-    suspend fun onError(function: CONNECTION.(Throwable) -> Unit) {
-        onErrorMutex.withLock {
+    fun onError(function: CONNECTION.(Throwable) -> Unit) {
+        onErrorLock.write {
             // we have to follow the single-writer principle!
-            onErrorList.lazySet(add(function, onErrorList.value))
+            onErrorList = add(function, onErrorList)
         }
     }
 
@@ -266,10 +318,10 @@ internal class ListenerManager<CONNECTION: Connection>(private val logger: KLogg
      *
      * The error is also sent to an error log before this method is called.
      */
-    suspend fun onError(function: Throwable.() -> Unit) {
-        onErrorGlobalMutex.withLock {
+    fun onError(function: Throwable.() -> Unit) {
+        onErrorGlobalLock.write {
             // we have to follow the single-writer principle!
-            onErrorGlobalList.lazySet(add(function, onErrorGlobalList.value))
+            onErrorGlobalList = add(function, onErrorGlobalList)
         }
     }
 
@@ -278,8 +330,8 @@ internal class ListenerManager<CONNECTION: Connection>(private val logger: KLogg
      *
      * This method should not block for long periods as other network activity will not be processed until it returns.
      */
-    suspend fun <MESSAGE> onMessage(function: suspend CONNECTION.(MESSAGE) -> Unit) {
-        onMessageMutex.withLock {
+    fun <MESSAGE> onMessage(function: CONNECTION.(MESSAGE) -> Unit) {
+        onMessageLock.write {
             // we have to follow the single-writer principle!
 
             // this is the connection generic parameter for the listener, works for lambda expressions as well
@@ -300,27 +352,27 @@ internal class ListenerManager<CONNECTION: Connection>(private val logger: KLogg
             }
 
             if (success) {
-                // NOTE: https://github.com/Kotlin/kotlinx.atomicfu
+                // https://github.com/Kotlin/kotlinx.atomicfu
                 // this is EXPLICITLY listed as a "Don't" via the documentation. The ****ONLY**** reason this is actually OK is because
                 // we are following the "single-writer principle", so only ONE THREAD can modify this at a time.
-                val tempMap = onMessageMap.value
+                val tempMap = onMessageMap
 
                 @Suppress("UNCHECKED_CAST")
-                val func = function as suspend (CONNECTION, Any) -> Unit
+                val func = function as (CONNECTION, Any) -> Unit
 
-                val newMessageArray: Array<suspend (CONNECTION, Any) -> Unit>
-                val onMessageArray: Array<suspend (CONNECTION, Any) -> Unit>? = tempMap.get(messageClass)
+                val newMessageArray: Array<(CONNECTION, Any) -> Unit>
+                val onMessageArray: Array<(CONNECTION, Any) -> Unit>? = tempMap[messageClass]
 
                 if (onMessageArray != null) {
                     newMessageArray = add(function, onMessageArray)
                 } else {
                     @Suppress("RemoveExplicitTypeArguments")
-                    newMessageArray = Array<suspend (CONNECTION, Any) -> Unit>(1) { { _, _ -> } }
+                    newMessageArray = Array<(CONNECTION, Any) -> Unit>(1) { { _, _ -> } }
                     newMessageArray[0] = func
                 }
 
-                tempMap.put(messageClass, newMessageArray)
-                onMessageMap.lazySet(tempMap)
+                tempMap.put(messageClass!!, newMessageArray)
+                onMessageMap = tempMap
             } else {
                 throw IllegalArgumentException("Unable to add incompatible types! Detected connection/message classes: $connectionClass, $messageClass")
             }
@@ -332,23 +384,18 @@ internal class ListenerManager<CONNECTION: Connection>(private val logger: KLogg
      *
      * It is the responsibility of the custom filter to write the error, if there is one
      *
-     * @return true if the connection will be allowed to connect. False if we should terminate this connection
+     * This is run directly on the thread that calls it!
+     *
+     * @return true if the client address is allowed to connect. False if we should terminate this connection
      */
-     fun notifyFilter(connection: CONNECTION): Boolean {
-        // remote address will NOT be null at this stage, but best to verify.
-        val remoteAddress = connection.remoteAddress
-        if (remoteAddress == null) {
-            logger.error("Connection ${connection.id}: Unable to attempt connection stages when no remote address is present")
-            return false
-        }
-
+    fun notifyFilter(clientAddress: InetAddress, clientTagName: String): Boolean {
         // by default, there is a SINGLE rule that will always exist, and will always ACCEPT ALL connections.
         // This is so the array types can be setup (the compiler needs SOMETHING there)
-        val arrayOfIpFilterRules = onConnectFilterList.value
+        val list = onConnectFilterList
 
         // if there is a rule, a connection must match for it to connect
-        arrayOfIpFilterRules.forEach {
-            if (it.invoke(connection)) {
+        list.forEach {
+            if (it.invoke(clientAddress, clientTagName)) {
                 return true
             }
         }
@@ -356,70 +403,135 @@ internal class ListenerManager<CONNECTION: Connection>(private val logger: KLogg
         // default if nothing matches
         // NO RULES ADDED -> ACCEPT
         //    RULES ADDED -> DENY
-        return arrayOfIpFilterRules.isEmpty()
+        return list.isEmpty()
+    }
+
+    /**
+     * Invoked just after a connection is created, but before it is connected.
+     *
+     * It is the responsibility of the custom filter to write the error, if there is one
+     *
+     * This is run directly on the thread that calls it!
+     *
+     * @return true if the connection will have buffered messages enabled. False if buffered messages for this connection should be disabled.
+     */
+    fun notifyEnableBufferedMessages(clientAddress: InetAddress?, clientTagName: String): Boolean {
+        // by default, there is a SINGLE rule that will always exist, and will always PERMIT buffered messages.
+        // This is so the array types can be setup (the compiler needs SOMETHING there)
+        val list = onConnectBufferedMessageFilterList
+
+        // if there is a rule, a connection must match for it to enable buffered messages
+        list.forEach {
+            if (it.invoke(clientAddress, clientTagName)) {
+                return true
+            }
+        }
+
+        // default if nothing matches
+        // NO RULES ADDED -> ALLOW Buffered Messages
+        //    RULES ADDED -> DISABLE Buffered Messages
+        return list.isEmpty()
     }
 
     /**
      * Invoked when a connection is first initialized, but BEFORE it's connected to the remote address.
+     *
+     * NOTE: This is run directly on the thread that calls it! Things that happen in event are TIME-CRITICAL, and must happen before connect happens.
+     * Because of this guarantee, init is immediately executed where connect is on a separate thread
      */
     fun notifyInit(connection: CONNECTION) {
-        runBlocking {
-            onInitList.value.forEach {
-                try {
-                    it(connection)
-                } catch (t: Throwable) {
-                    // NOTE: when we remove stuff, we ONLY want to remove the "tail" of the stacktrace, not ALL parts of the stacktrace
-                    cleanStackTrace(t)
-                    logger.error("Connection ${connection.id} error", t)
-                }
+        val list = onInitList
+        list.forEach {
+            try {
+                it(connection)
+            } catch (t: Throwable) {
+                // when we remove stuff, we ONLY want to remove the "tail" of the stacktrace, not ALL parts of the stacktrace
+                t.cleanStackTrace()
+                logger.error("Connection ${connection.id} error", t)
             }
         }
     }
 
     /**
      * Invoked when a connection is connected to a remote address.
+     *
+     * This is run on the EventDispatch!
      */
-    suspend fun notifyConnect(connection: CONNECTION) {
-        onConnectList.value.forEach {
-            try {
-                it(connection)
-            } catch (t: Throwable) {
-                // NOTE: when we remove stuff, we ONLY want to remove the "tail" of the stacktrace, not ALL parts of the stacktrace
-                cleanStackTrace(t)
-                logger.error("Connection ${connection.id} error", t)
+    fun notifyConnect(connection: CONNECTION) {
+        val list = onConnectList
+        if (list.isNotEmpty()) {
+            connection.endPoint.eventDispatch.CONNECT.launch {
+                list.forEach {
+                    try {
+                        it(connection)
+                    } catch (t: Throwable) {
+                        //  when we remove stuff, we ONLY want to remove the "tail" of the stacktrace, not ALL parts of the stacktrace
+                        t.cleanStackTrace()
+                        logger.error("Connection ${connection.id} error", t)
+                    }
+                }
             }
         }
     }
 
     /**
      * Invoked when a connection is disconnected to a remote address.
+     *
+     * This is exclusively called from a connection, when that connection is closed!
+     *
+     * This is run on the EventDispatch!
      */
-    suspend fun notifyDisconnect(connection: CONNECTION) {
-        onDisconnectList.value.forEach {
-            try {
-                it(connection)
-            } catch (t: Throwable) {
-                // NOTE: when we remove stuff, we ONLY want to remove the "tail" of the stacktrace, not ALL parts of the stacktrace
-                cleanStackTrace(t)
-                logger.error("Connection ${connection.id} error", t)
+    fun notifyDisconnect(connection: Connection) {
+        connection.notifyDisconnect()
+
+        @Suppress("UNCHECKED_CAST")
+        directNotifyDisconnect(connection as CONNECTION)
+    }
+
+    /**
+     * This is invoked by either a GLOBAL listener manager, or for a SPECIFIC CONNECTION listener manager.
+     */
+    fun directNotifyDisconnect(connection: CONNECTION) {
+        val list = onDisconnectList
+        if (list.isNotEmpty()) {
+            connection.endPoint.eventDispatch.CLOSE.launch {
+                list.forEach {
+                    try {
+                        it(connection)
+                    } catch (t: Throwable) {
+                        // when we remove stuff, we ONLY want to remove the "tail" of the stacktrace, not ALL parts of the stacktrace
+                        t.cleanStackTrace()
+                        logger.error("Connection ${connection.id} error", t)
+                    }
+                }
             }
         }
     }
 
+
     /**
      * Invoked when there is an error for a specific connection
      *
      * The error is also sent to an error log before notifying callbacks
+     *
+     * This is run on the EventDispatch!
      */
     fun notifyError(connection: CONNECTION, exception: Throwable) {
-        onErrorList.value.forEach {
-            try {
-                it(connection, exception)
-            } catch (t: Throwable) {
-                // NOTE: when we remove stuff, we ONLY want to remove the "tail" of the stacktrace, not ALL parts of the stacktrace
-                cleanStackTrace(t)
-                logger.error("Connection ${connection.id} error", t)
+        val list = onErrorList
+        if (list.isNotEmpty()) {
+            connection.endPoint.eventDispatch.ERROR.launch {
+                list.forEach {
+                    try {
+                        it(connection, exception)
+                    } catch (t: Throwable) {
+                        // when we remove stuff, we ONLY want to remove the "tail" of the stacktrace, not ALL parts of the stacktrace
+                        t.cleanStackTrace()
+                        logger.error("Connection ${connection.id} error", t)
+                    }
+                }
             }
+        } else {
+            logger.error("Error with connection $connection", exception)
         }
     }
 
@@ -428,15 +540,22 @@ internal class ListenerManager<CONNECTION: Connection>(private val logger: KLogg
      *
      * The error is also sent to an error log before notifying callbacks
      */
-    val notifyError: (exception: Throwable) -> Unit = { exception ->
-        onErrorGlobalList.value.forEach {
-            try {
-                it(exception)
-            } catch (t: Throwable) {
-                // NOTE: when we remove stuff, we ONLY want to remove the "tail" of the stacktrace, not ALL parts of the stacktrace
-                cleanStackTrace(t)
-                logger.error("Global error", t)
+    fun notifyError(exception: Throwable) {
+        val list = onErrorGlobalList
+        if (list.isNotEmpty()) {
+            eventDispatch.ERROR.launch {
+                list.forEach {
+                    try {
+                        it(exception)
+                    } catch (t: Throwable) {
+                        // when we remove stuff, we ONLY want to remove the "tail" of the stacktrace, not ALL parts of the stacktrace
+                        t.cleanStackTrace()
+                        logger.error("Global error", t)
+                    }
+                }
             }
+        } else {
+            logger.error("Global error", exception)
         }
     }
 
@@ -445,7 +564,7 @@ internal class ListenerManager<CONNECTION: Connection>(private val logger: KLogg
      *
      * @return true if there were listeners assigned for this message type
      */
-    suspend fun notifyOnMessage(connection: CONNECTION, message: Any): Boolean {
+    fun notifyOnMessage(connection: CONNECTION, message: Any): Boolean {
         val messageClass: Class<*> = message.javaClass
 
         // have to save the types + hierarchy (note: duplicates are OK, since they will just be overwritten)
@@ -465,10 +584,10 @@ internal class ListenerManager<CONNECTION: Connection>(private val logger: KLogg
 
         // cache the lookup
         //   we don't care about race conditions, since the object hierarchy will be ALREADY established at this exact moment
-        val tempMap = onMessageMap.value
+        val tempMap = onMessageMap
         var hasListeners = false
         hierarchy.forEach { clazz ->
-            val onMessageArray: Array<suspend (CONNECTION, Any) -> Unit>? = tempMap.get(clazz)
+            val onMessageArray: Array<(CONNECTION, Any) -> Unit>? = tempMap[clazz]
             if (onMessageArray != null) {
                 hasListeners = true
 
@@ -476,8 +595,6 @@ internal class ListenerManager<CONNECTION: Connection>(private val logger: KLogg
                     try {
                         func(connection, message)
                     } catch (t: Throwable) {
-                        cleanStackTrace(t)
-                        logger.error("Connection ${connection.id} error", t)
                         notifyError(connection, t)
                     }
                 }
@@ -486,4 +603,37 @@ internal class ListenerManager<CONNECTION: Connection>(private val logger: KLogg
 
         return hasListeners
     }
+
+    /**
+     * This will remove all listeners that have been registered!
+     */
+    fun close() {
+        // we have to follow the single-writer principle!
+        logger.debug("Closing the listener manager")
+
+        onConnectFilterLock.write {
+            onConnectFilterList = Array(0) { { _, _ -> true } }
+        }
+        onConnectBufferedMessageFilterLock.write {
+            onConnectBufferedMessageFilterList = Array(0) { { _, _ -> true } }
+        }
+        onInitLock.write {
+            onInitList = Array(0) { { } }
+        }
+        onConnectLock.write {
+            onConnectList = Array(0) { { } }
+        }
+        onDisconnectLock.write {
+            onDisconnectList = Array(0) { { } }
+        }
+        onErrorLock.write {
+            onErrorList = Array(0) { {  } }
+        }
+        onErrorGlobalLock.write {
+            onErrorGlobalList = Array(0) { { } }
+        }
+        onMessageLock.write {
+            onMessageMap = IdentityMap(32, LOAD_FACTOR)
+        }
+    }
 }

src/dorkbox/network/connection/Paired.kt

@@ -0,0 +1,22 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package dorkbox.network.connection
+
+class Paired<CONNECTION : Connection> {
+    lateinit var connection: CONNECTION
+    lateinit var message: Any
+}

src/dorkbox/network/connection/SendSync.kt

@@ -0,0 +1,46 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package dorkbox.network.connection
+
+import dorkbox.network.rmi.RmiUtils
+
+class SendSync {
+    var message: Any? = null
+
+    // used to notify the remote endpoint that the message has been processed
+    var id: Int = 0
+
+    override fun equals(other: Any?): Boolean {
+        if (this === other) return true
+        if (other !is SendSync) return false
+
+        if (message != other.message) return false
+        if (id != other.id) return false
+
+        return true
+    }
+
+    override fun hashCode(): Int {
+        var result = message?.hashCode() ?: 0
+        result = 31 * result + id
+        return result
+    }
+
+    override fun toString(): String {
+        return "SendSync ${RmiUtils.unpackUnsignedRight(id)}  (message=$message)"
+    }
+}

src/dorkbox/network/connection/buffer/BufferManager.kt

@@ -0,0 +1,130 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package dorkbox.network.connection.buffer
+
+import dorkbox.bytes.ByteArrayWrapper
+import dorkbox.collections.LockFreeHashMap
+import dorkbox.hex.toHexString
+import dorkbox.network.Configuration
+import dorkbox.network.aeron.AeronDriver
+import dorkbox.network.connection.Connection
+import dorkbox.network.connection.EndPoint
+import dorkbox.network.connection.ListenerManager
+import dorkbox.util.Sys
+import net.jodah.expiringmap.ExpirationPolicy
+import net.jodah.expiringmap.ExpiringMap
+import org.slf4j.LoggerFactory
+import java.util.concurrent.*
+
+internal open class BufferManager<CONNECTION: Connection>(
+    config: Configuration,
+    listenerManager: ListenerManager<CONNECTION>,
+    aeronDriver: AeronDriver,
+    sessionTimeout: Long
+) {
+
+    companion object {
+        private val logger = LoggerFactory.getLogger(BufferManager::class.java.simpleName)
+    }
+
+    private val sessions = LockFreeHashMap<ByteArrayWrapper, BufferedSession>()
+    private val expiringSessions: ExpiringMap<ByteArrayWrapper, BufferedSession>
+
+    init {
+        require(sessionTimeout >= 60) { "The buffered connection timeout 'bufferedConnectionTimeoutSeconds' must be greater than 60 seconds!" }
+
+        // ignore 0
+        val check = TimeUnit.SECONDS.toNanos(sessionTimeout)
+        val lingerNs = aeronDriver.lingerNs()
+        val required = TimeUnit.SECONDS.toNanos(config.connectionCloseTimeoutInSeconds.toLong())
+        require(check == 0L || check > required + lingerNs) {
+            "The session timeout (${Sys.getTimePretty(check)}) must be longer than the connection close timeout (${Sys.getTimePretty(required)}) + the aeron driver linger timeout (${Sys.getTimePretty(lingerNs)})!"
+        }
+
+        // connections are extremely difficult to diagnose when the connection timeout is short
+        val timeUnit = if (EndPoint.DEBUG_CONNECTIONS) { TimeUnit.HOURS } else { TimeUnit.SECONDS }
+
+        expiringSessions = ExpiringMap.builder()
+            .expiration(sessionTimeout, timeUnit)
+            .expirationPolicy(ExpirationPolicy.CREATED)
+            .expirationListener<ByteArrayWrapper, BufferedSession> { publicKeyWrapped, sessionConnection ->
+                // this blocks until it fully runs (which is ok. this is fast)
+                logger.debug("Connection session expired for: ${publicKeyWrapped.bytes.toHexString()}")
+
+                // this SESSION has expired, so we should call the onDisconnect for the underlying connection, in order to clean it up.
+                listenerManager.notifyDisconnect(sessionConnection.connection)
+            }
+            .build()
+    }
+
+    /**
+     * this must be called when a new connection is created
+     *
+     * @return true if this is a new session, false if it is an existing session
+     */
+    fun onConnect(connection: Connection): BufferedSession {
+        val publicKeyWrapped = ByteArrayWrapper.wrap(connection.uuid)
+
+        return synchronized(sessions) {
+            // always check if we are expiring first...
+            val expiring = expiringSessions.remove(publicKeyWrapped)
+            if (expiring != null) {
+                expiring.connection = connection
+                expiring
+            } else {
+                val existing = sessions[publicKeyWrapped]
+                if (existing != null) {
+                    // we must always set this session value!!
+                    existing.connection = connection
+                    existing
+                } else {
+                    val newSession = BufferedSession(connection)
+                    sessions[publicKeyWrapped] = newSession
+
+                    // we must always set this when the connection is created, and it must be inside the sync block!
+                    newSession
+                }
+            }
+        }
+    }
+
+    /**
+     * Always called when a connection is disconnected from the network
+     */
+    fun onDisconnect(connection: Connection) {
+        try {
+            val publicKeyWrapped = ByteArrayWrapper.wrap(connection.uuid)
+
+            synchronized(sessions) {
+                val sess = sessions.remove(publicKeyWrapped)
+                // we want to expire this session after XYZ time
+                expiringSessions[publicKeyWrapped] = sess
+            }
+        }
+        catch (e: Exception) {
+            logger.error("Unable to run session expire logic!", e)
+        }
+    }
+
+
+    fun close() {
+        synchronized(sessions) {
+            sessions.clear()
+            expiringSessions.clear()
+        }
+    }
+}

src/dorkbox/network/connection/buffer/BufferedMessages.kt

@@ -0,0 +1,21 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package dorkbox.network.connection.buffer
+
+class BufferedMessages {
+    var messages = arrayListOf<Any>()
+}

src/dorkbox/network/connection/buffer/BufferedSerializer.kt

@@ -0,0 +1,34 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package dorkbox.network.connection.buffer
+
+import com.esotericsoftware.kryo.Kryo
+import com.esotericsoftware.kryo.Serializer
+import com.esotericsoftware.kryo.io.Input
+import com.esotericsoftware.kryo.io.Output
+
+internal class BufferedSerializer: Serializer<BufferedMessages>() {
+    override fun write(kryo: Kryo, output: Output, messages: BufferedMessages) {
+        kryo.writeClassAndObject(output, messages.messages)
+    }
+
+    override fun read(kryo: Kryo, input: Input, type: Class<out BufferedMessages>): BufferedMessages {
+        val messages = BufferedMessages()
+        messages.messages = kryo.readClassAndObject(input) as ArrayList<Any>
+        return messages
+    }
+}

src/dorkbox/network/connection/buffer/BufferedSession.kt

@@ -0,0 +1,66 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package dorkbox.network.connection.buffer
+
+import dorkbox.network.connection.Connection
+import java.util.concurrent.*
+
+open class BufferedSession(@Volatile var connection: Connection) {
+    /**
+     * Only used when configured. Will re-send all missing messages to a connection when a connection re-connects.
+     */
+    val pendingMessagesQueue: LinkedTransferQueue<Any> = LinkedTransferQueue()
+
+    fun queueMessage(connection: Connection, message: Any, abortEarly: Boolean): Boolean {
+        if (this.connection != connection) {
+            connection.logger.trace("[{}] message received on old connection, resending", connection)
+
+            // we received a message on an OLD connection (which is no longer connected ---- BUT we have a NEW connection that is connected)
+            // this can happen on RMI object that are old
+            val success = this.connection.send(message, abortEarly)
+            if (success) {
+                connection.logger.trace("[{}] successfully resent message", connection)
+                return true
+            }
+        }
+
+        if (!connection.enableBufferedMessages) {
+            // nothing, since we emit logs during connection initialization that pending messages are DISABLED
+            return false
+        }
+
+        if (!abortEarly) {
+            // this was a "normal" send (instead of the disconnect message).
+            pendingMessagesQueue.put(message)
+            connection.logger.trace("[{}] queueing message", connection)
+        }
+        else if (connection.endPoint.aeronDriver.internal.mustRestartDriverOnError) {
+            // the only way we get errors, is if the connection is bad OR if we are sending so fast that the connection cannot keep up.
+
+            // don't restart/reconnect -- there was an internal network error
+            pendingMessagesQueue.put(message)
+            connection.logger.trace("[{}] queueing message", connection)
+        }
+        else if (!connection.isClosedWithTimeout()) {
+            // there was an issue - the connection should automatically reconnect
+            pendingMessagesQueue.put(message)
+            connection.logger.trace("[{}] queueing message", connection)
+        }
+
+        return false
+    }
+}

src/dorkbox/network/connection/buffer/package-info.java

@@ -0,0 +1,17 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package dorkbox.network.connection.buffer;

src/dorkbox/network/connection/package-info.java

@@ -0,0 +1,17 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package dorkbox.network.connection;

src/dorkbox/network/connection/streaming/AeronWriter.kt

@@ -0,0 +1,34 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package dorkbox.network.connection.streaming
+
+import dorkbox.network.serialization.AeronOutput
+import kotlinx.atomicfu.atomic
+
+class AeronWriter(val size: Int): StreamingWriter, AeronOutput(size) {
+    private val written = atomic(0)
+
+    override fun writeBytes(startPosition: Int, bytes: ByteArray) {
+        position = startPosition
+        writeBytes(bytes)
+        written.getAndAdd(bytes.size)
+    }
+
+    override fun isFinished(): Boolean {
+        return written.value == size
+    }
+}

src/dorkbox/network/connection/streaming/FileWriter.kt

@@ -0,0 +1,62 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package dorkbox.network.connection.streaming
+
+import kotlinx.atomicfu.atomic
+import java.io.File
+import java.io.FileOutputStream
+import java.io.RandomAccessFile
+
+class FileWriter(val size: Int, val file: File) : StreamingWriter, RandomAccessFile(file, "rw") {
+
+    private val written = atomic(0)
+
+    init {
+        // reserve space on disk!
+        val saveSize = size.coerceAtMost(4096)
+        var bytes = ByteArray(saveSize)
+        this.write(bytes)
+
+        if (saveSize < size) {
+            var remainingBytes = size - saveSize
+
+            while (remainingBytes > 0) {
+                if (saveSize > remainingBytes) {
+                    bytes = ByteArray(remainingBytes)
+                }
+                this.write(bytes)
+                remainingBytes = (remainingBytes - saveSize).coerceAtLeast(0)
+            }
+        }
+    }
+
+    override fun writeBytes(startPosition: Int, bytes: ByteArray) {
+        // the OS will synchronize writes to disk
+        this.seek(startPosition.toLong())
+        write(bytes)
+        written.addAndGet(bytes.size)
+    }
+
+    override fun isFinished(): Boolean {
+        return written.value == size
+    }
+
+    fun finishAndClose() {
+        fd.sync()
+        close()
+    }
+}

src/dorkbox/network/connection/streaming/StreamingControl.kt

@@ -1,5 +1,23 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
 package dorkbox.network.connection.streaming
 
-data class StreamingControl(val state: StreamingState, val streamId: Long,
-                            val totalSize: Long = 0L,
-                            val isFile: Boolean = false, val fileName: String = ""): StreamingMessage
+data class StreamingControl(val state: StreamingState,
+                            val isFile: Boolean,
+                            val streamId: Int,
+                            val totalSize: Long = 0L
+                            ): StreamingMessage

src/dorkbox/network/connection/streaming/StreamingControlSerializer.kt

@@ -1,5 +1,5 @@
 /*
- * Copyright 2020 dorkbox, llc
+ * Copyright 2023 dorkbox, llc
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -13,6 +13,7 @@
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
+
 package dorkbox.network.connection.streaming
 
 import com.esotericsoftware.kryo.Kryo
@@ -20,41 +21,21 @@ import com.esotericsoftware.kryo.Serializer
 import com.esotericsoftware.kryo.io.Input
 import com.esotericsoftware.kryo.io.Output
 
-class StreamingControlSerializer: Serializer<StreamingControl>() {
+internal class StreamingControlSerializer: Serializer<StreamingControl>() {
     override fun write(kryo: Kryo, output: Output, data: StreamingControl) {
         output.writeByte(data.state.ordinal)
-        output.writeVarLong(data.streamId, true)
-        output.writeVarLong(data.totalSize, true)
         output.writeBoolean(data.isFile)
-        if (data.isFile) {
-            output.writeString(data.fileName)
-        }
+        output.writeVarInt(data.streamId, true)
+        output.writeVarLong(data.totalSize, true)
     }
 
     override fun read(kryo: Kryo, input: Input, type: Class<out StreamingControl>): StreamingControl {
         val stateOrdinal = input.readByte().toInt()
-        val state = StreamingState.values().first { it.ordinal == stateOrdinal }
-        val streamId = input.readVarLong(true)
-        val totalSize = input.readVarLong(true)
         val isFile = input.readBoolean()
-        val fileName = if (isFile) {
-            input.readString()
-        } else {
-            ""
-        }
+        val state = StreamingState.entries.first { it.ordinal == stateOrdinal }
+        val streamId = input.readVarInt(true)
+        val totalSize = input.readVarLong(true)
 
-        return StreamingControl(state, streamId, totalSize, isFile, fileName)
-    }
-}
-
-class StreamingDataSerializer: Serializer<StreamingData>() {
-    override fun write(kryo: Kryo, output: Output, data: StreamingData) {
-        output.writeVarLong(data.streamId, true)
-    }
-
-    override fun read(kryo: Kryo, input: Input, type: Class<out StreamingData>): StreamingData {
-        val streamId = input.readVarLong(true)
-
-        return StreamingData(streamId)
+        return StreamingControl(state, isFile, streamId, totalSize)
     }
 }

src/dorkbox/network/connection/streaming/StreamingData.kt

@@ -1,9 +1,27 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
 package dorkbox.network.connection.streaming
 
-class StreamingData(val streamId: Long) : StreamingMessage {
+import dorkbox.bytes.xxHash32
 
-    // These are set just after we receive the message, and before we process it
-    @Transient var payload: ByteArray? = null
+class StreamingData(val streamId: Int) : StreamingMessage {
+
+    var payload: ByteArray? = null
+    var startPosition: Int = 0
     
     override fun equals(other: Any?): Boolean {
         if (this === other) return true
@@ -17,16 +35,19 @@ class StreamingData(val streamId: Long) : StreamingMessage {
             if (!payload.contentEquals(other.payload)) return false
         } else if (other.payload != null) return false
 
+        if (startPosition != other.startPosition) return false
+
         return true
     }
 
     override fun hashCode(): Int {
         var result = streamId.hashCode()
         result = 31 * result + (payload?.contentHashCode() ?: 0)
+        result = 31 * result + (startPosition)
         return result
     }
 
     override fun toString(): String {
-        return "StreamingData(streamId=$streamId)"
+        return "StreamingData(streamId=$streamId position=${startPosition}, xxHash=${payload?.xxHash32()})"
     }
 }

src/dorkbox/network/connection/streaming/StreamingDataSerializer.kt

@@ -0,0 +1,41 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package dorkbox.network.connection.streaming
+
+import com.esotericsoftware.kryo.Kryo
+import com.esotericsoftware.kryo.Serializer
+import com.esotericsoftware.kryo.io.Input
+import com.esotericsoftware.kryo.io.Output
+
+
+internal class StreamingDataSerializer: Serializer<StreamingData>() {
+    override fun write(kryo: Kryo, output: Output, data: StreamingData) {
+        output.writeVarInt(data.streamId, true)
+        // we re-use this data when streaming data to the remote endpoint, so we don't write out the payload here, we do it in another place
+    }
+
+    override fun read(kryo: Kryo, input: Input, type: Class<out StreamingData>): StreamingData {
+        val streamId = input.readVarInt(true)
+        val streamingData = StreamingData(streamId)
+
+        // we want to read out the start-position AND payload. It is not written by the serializer, but by the streaming manager
+        val startPosition = input.readVarInt(true)
+        val payloadSize = input.readVarInt(true)
+        streamingData.startPosition = startPosition
+        streamingData.payload = input.readBytes(payloadSize)
+        return streamingData
+    }
+}

src/dorkbox/network/connection/streaming/StreamingManager.kt

@@ -1,30 +1,55 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
 @file:Suppress("DuplicatedCode")
 
 package dorkbox.network.connection.streaming
 
+import com.esotericsoftware.kryo.io.Input
+import dorkbox.bytes.OptimizeUtilsByteArray
 import dorkbox.bytes.OptimizeUtilsByteBuf
-import dorkbox.collections.LockFreeHashMap
+import dorkbox.collections.LockFreeLongMap
+import dorkbox.network.Configuration
 import dorkbox.network.connection.Connection
+import dorkbox.network.connection.CryptoManagement
 import dorkbox.network.connection.EndPoint
-import dorkbox.network.connection.ListenerManager
+import dorkbox.network.connection.ListenerManager.Companion.cleanAllStackTrace
+import dorkbox.network.connection.ListenerManager.Companion.cleanStackTrace
 import dorkbox.network.serialization.AeronInput
 import dorkbox.network.serialization.AeronOutput
-import dorkbox.network.serialization.KryoExtra
+import dorkbox.network.serialization.KryoWriter
+import dorkbox.os.OS
+import dorkbox.util.Sys
 import io.aeron.Publication
-import kotlinx.coroutines.CoroutineScope
-import kotlinx.coroutines.launch
-import mu.KLogger
 import org.agrona.MutableDirectBuffer
-import java.security.SecureRandom
+import org.agrona.concurrent.IdleStrategy
+import org.agrona.concurrent.UnsafeBuffer
+import org.slf4j.Logger
+import java.io.File
+import java.io.FileInputStream
 
-internal class StreamingManager<CONNECTION : Connection>(private val logger: KLogger, private val actionDispatch: CoroutineScope) {
-    private val streamingDataTarget = LockFreeHashMap<Long, StreamingControl>()
-    private val streamingDataInMemory = LockFreeHashMap<Long, AeronOutput>()
+internal class StreamingManager<CONNECTION : Connection>(private val logger: Logger, val config: Configuration) {
 
     companion object {
-        val random = SecureRandom()
+        private const val KILOBYTE = 1024
+        private const val MEGABYTE = 1024 * KILOBYTE
+        private const val GIGABYTE = 1024 * MEGABYTE
+        private const val TERABYTE = 1024L * GIGABYTE
 
-        @Suppress("UNUSED_CHANGED_VALUE")
+        @Suppress("UNUSED_CHANGED_VALUE", "SameParameterValue")
         private fun writeVarInt(internalBuffer: MutableDirectBuffer, position: Int, value: Int, optimizePositive: Boolean): Int {
             var p = position
             var newValue = value
@@ -64,85 +89,171 @@ internal class StreamingManager<CONNECTION : Connection>(private val logger: KLo
     }
 
 
+    private val streamingDataTarget = LockFreeLongMap<StreamingControl>()
+    private val streamingDataInMemory = LockFreeLongMap<StreamingWriter>()
+
+
     /**
-    * Reassemble/figure out the internal message pieces
+     * What is the max stream size that can exist in memory when deciding if data blocks are in memory or temp-file on disk
+     */
+    private val maxStreamSizeInMemoryInBytes = config.maxStreamSizeInMemoryMB * MEGABYTE
+
+    fun getFile(connection: CONNECTION, endPoint: EndPoint<CONNECTION>, messageStreamId: Int): File {
+        // NOTE: the stream session ID is a combination of the connection ID + random ID (on the receiving side),
+        //      otherwise clients can abuse it and corrupt OTHER clients data!!
+        val streamId = (connection.id.toLong() shl 4) or messageStreamId.toLong()
+
+        val output = streamingDataInMemory[streamId]
+        return if (output is FileWriter) {
+            streamingDataInMemory.remove(streamId)
+            output.file
+        } else {
+            // something SUPER wrong!
+            // more critical error sending the message. we shouldn't retry or anything.
+            val errorMessage = "Error while reading file output, stream $streamId was of the wrong type!"
+
+            // either client or server. No other choices. We create an exception, because it's more useful!
+            throw endPoint.newException(errorMessage)
+        }
+    }
+
+    /**
+     * NOTE: MUST BE ON THE AERON THREAD!
+     *
+    * Reassemble/figure out the internal message pieces. Processed always on the same thread
     */
-    fun processControlMessage(message: StreamingControl, endPoint: EndPoint<CONNECTION>, connection: CONNECTION) {
-        val streamId = message.streamId
+    fun processControlMessage(
+        message: StreamingControl,
+        endPoint: EndPoint<CONNECTION>,
+        connection: CONNECTION
+    ) {
+        // NOTE: the stream session ID is a combination of the connection ID + random ID (on the receiving side),
+        //      otherwise clients can abuse it and corrupt OTHER clients data!!
+        val streamId = (connection.id.toLong() shl 4) or message.streamId.toLong()
 
         when (message.state) {
             StreamingState.START -> {
-                streamingDataTarget[streamId] = message
-                if (!message.isFile) {
-                    streamingDataInMemory[streamId] = AeronOutput()
+                // message.totalSize > maxInMemory  OR  if we are a file, then write to a temp file INSTEAD
+                if (message.isFile || message.totalSize > maxStreamSizeInMemoryInBytes) {
+                    var fileName = "${config.appId}_${streamId}_${connection.id}.tmp"
+
+                    var tempFileLocation = OS.TEMP_DIR.resolve(fileName)
+                    while (tempFileLocation.canRead()) {
+                        fileName = "${config.appId}_${streamId}_${connection.id}_${CryptoManagement.secureRandom.nextInt()}.tmp"
+                        tempFileLocation = OS.TEMP_DIR.resolve(fileName)
+                    }
+                    tempFileLocation.deleteOnExit()
+
+                    val prettySize = Sys.getSizePretty(message.totalSize)
+
+                    if (endPoint.logger.isInfoEnabled) {
+                        endPoint.logger.info("Saving $prettySize of streaming data [${streamId}] to: $tempFileLocation")
+                    }
+                    streamingDataInMemory[streamId] = FileWriter(message.totalSize.toInt(), tempFileLocation)
+                } else {
+                    if (endPoint.logger.isTraceEnabled) {
+                        endPoint.logger.trace("Saving streaming data [${streamId}] in memory")
+                    }
+                    // .toInt is safe because we know the total size is < than maxStreamSizeInMemoryInBytes
+                    streamingDataInMemory[streamId] = AeronWriter(message.totalSize.toInt())
                 }
+
+                // this must be last
+                streamingDataTarget[streamId] = message
             }
+
             StreamingState.FINISHED -> {
-                // get the data out and send messages!
-                if (!message.isFile) {
-                    val output = streamingDataInMemory.remove(streamId)
-                    if (output != null) {
-                        val kryo: KryoExtra<CONNECTION> = endPoint.serialization.takeKryo()
+                // NOTE: cannot be on a coroutine before kryo usage!
 
-                        try {
-                            val input = AeronInput(output.internalBuffer)
-                            val streamedMessage = kryo.read(input)
+                if (message.isFile) {
+                    // we do not do anything with this file yet! The serializer has to return this instance!
+                    val output = streamingDataInMemory[streamId]
 
-                            // NOTE: This MUST be on a new co-routine
-                            actionDispatch.launch {
-                                val listenerManager = endPoint.listenerManager
-
-                                try {
-                                    @Suppress("UNCHECKED_CAST")
-                                    var hasListeners = listenerManager.notifyOnMessage(connection, streamedMessage)
-
-                                    // each connection registers, and is polled INDEPENDENTLY for messages.
-                                    hasListeners = hasListeners or connection.notifyOnMessage(streamedMessage)
-
-                                    if (!hasListeners) {
-                                        logger.error("No message callbacks found for ${streamedMessage::class.java.name}")
-                                    }
-                                } catch (e: Exception) {
-                                    logger.error("Error processing message ${streamedMessage::class.java.name}", e)
-                                    listenerManager.notifyError(connection, e)
-                                }
-                            }
-                        } catch (e: Exception) {
-                            // something SUPER wrong!
-                            // more critical error sending the message. we shouldn't retry or anything.
-                            val errorMessage = "Error serializing message from received streaming content, stream $streamId"
-
-                            // either client or server. No other choices. We create an exception, because it's more useful!
-                            val exception = endPoint.newException(errorMessage, e)
-
-                            // +2 because we do not want to see the stack for the abstract `newException`
-                            // +3 more because we do not need to see the "internals" for sending messages. The important part of the stack trace is
-                            // where we see who is calling "send()"
-                            ListenerManager.cleanStackTrace(exception, 2)
-                            throw exception
-                        } finally {
-                            endPoint.serialization.returnKryo(kryo)
-                        }
+                    if (output is FileWriter) {
+                        output.finishAndClose()
+                        // we don't need to do anything else (no de-serialization into an object) because we are already our target object
+                        return
                     } else {
                         // something SUPER wrong!
                         // more critical error sending the message. we shouldn't retry or anything.
-                        val errorMessage = "Error while receiving streaming content, stream $streamId not available."
+                        val errorMessage = "Error while processing streaming content, stream $streamId was supposed to be a FileWriter."
 
                         // either client or server. No other choices. We create an exception, because it's more useful!
-                        val exception = endPoint.newException(errorMessage)
-
-                        // +2 because we do not want to see the stack for the abstract `newException`
-                        // +3 more because we do not need to see the "internals" for sending messages. The important part of the stack trace is
-                        // where we see who is calling "send()"
-                        ListenerManager.cleanStackTrace(exception, 2)
-                        throw exception
+                        throw endPoint.newException(errorMessage)
                     }
-                } else {
-                    // we are a file, so process accordingly
-
                 }
+
+                // get the data out and send messages!
+                val output = streamingDataInMemory.remove(streamId)
+
+                val input = when (output) {
+                    is AeronWriter -> {
+                        // the position can be wrong, especially if there are multiple threads setting the data
+                        output.setPosition(output.size)
+                        AeronInput(output.internalBuffer)
+                    }
+                    is FileWriter -> {
+                        // if we are too large to fit in memory while streaming, we store it on disk.
+                        output.finishAndClose()
+
+                        val fileInputStream = FileInputStream(output.file)
+                        Input(fileInputStream)
+                    }
+                    else -> {
+                        null
+                    }
+                }
+
+                val streamedMessage = if (input != null) {
+                        val kryo = endPoint.serialization.takeRead()
+                        try {
+                            kryo.read(connection, input)
+                        } catch (e: Exception) {
+                            // something SUPER wrong!
+                            // more critical error sending the message. we shouldn't retry or anything.
+                            val errorMessage = "Error deserializing message from received streaming content, stream $streamId"
+
+                            // either client or server. No other choices. We create an exception, because it's more useful!
+                            throw endPoint.newException(errorMessage, e)
+                        } finally {
+                            endPoint.serialization.putRead(kryo)
+                            if (output is FileWriter) {
+                                val fileName = "${config.appId}_${streamId}_${connection.id}.tmp"
+                                val tempFileLocation = OS.TEMP_DIR.resolve(fileName)
+                                tempFileLocation.delete()
+                            }
+                        }
+                    } else {
+                        null
+                    }
+
+                if (streamedMessage == null) {
+                    if (output is FileWriter) {
+                        val fileName = "${config.appId}_${streamId}_${connection.id}.tmp"
+                        val tempFileLocation = OS.TEMP_DIR.resolve(fileName)
+                        tempFileLocation.delete()
+                    }
+
+                    // something SUPER wrong!
+                    // more critical error sending the message. we shouldn't retry or anything.
+                    val errorMessage = "Error while processing streaming content, stream $streamId was null."
+
+                    // either client or server. No other choices. We create an exception, because it's more useful!
+                    throw endPoint.newException(errorMessage)
+                }
+
+
+                // this can be a regular message or an RMI message. Redispatch!
+                endPoint.processMessageFromChannel(connection, streamedMessage)
             }
             StreamingState.FAILED -> {
+                val output = streamingDataInMemory.remove(streamId)
+                if (output is FileWriter) {
+                    val fileName = "${config.appId}_${streamId}_${connection.id}.tmp"
+                    val tempFileLocation = OS.TEMP_DIR.resolve(fileName)
+                    tempFileLocation.delete()
+                }
+
                 // clear all state
                 // something SUPER wrong!
                 // more critical error sending the message. we shouldn't retry or anything.
@@ -151,13 +262,19 @@ internal class StreamingManager<CONNECTION : Connection>(private val logger: KLo
                 // either client or server. No other choices. We create an exception, because it's more useful!
                 val exception = endPoint.newException(errorMessage)
 
-                // +2 because we do not want to see the stack for the abstract `newException`
                 // +3 more because we do not need to see the "internals" for sending messages. The important part of the stack trace is
                 // where we see who is calling "send()"
-                ListenerManager.cleanStackTrace(exception, 2)
+                exception.cleanStackTrace(3)
                 throw exception
             }
             StreamingState.UNKNOWN ->  {
+                val output = streamingDataInMemory.remove(streamId)
+                if (output is FileWriter) {
+                    val fileName = "${config.appId}_${streamId}_${connection.id}.tmp"
+                    val tempFileLocation = OS.TEMP_DIR.resolve(fileName)
+                    tempFileLocation.delete()
+                }
+
                 // something SUPER wrong!
                 // more critical error sending the message. we shouldn't retry or anything.
                 val errorMessage = "Unknown failure while receiving streaming content for stream $streamId"
@@ -165,29 +282,29 @@ internal class StreamingManager<CONNECTION : Connection>(private val logger: KLo
                 // either client or server. No other choices. We create an exception, because it's more useful!
                 val exception = endPoint.newException(errorMessage)
 
-                // +2 because we do not want to see the stack for the abstract `newException`
                 // +3 more because we do not need to see the "internals" for sending messages. The important part of the stack trace is
                 // where we see who is calling "send()"
-                ListenerManager.cleanStackTrace(exception, 2)
+                exception.cleanStackTrace(3)
                 throw exception
             }
         }
     }
 
     /**
-     * NOTE: MUST BE ON THE AERON THREAD!
+     * NOTE: MUST BE ON THE AERON THREAD BECAUSE THIS MUST BE SINGLE THREADED!!!
      *
      * Reassemble/figure out the internal message pieces
      *
-     * NOTE sending a huge file can prevent other other network traffic from arriving until it's done!
+     * NOTE sending a huge file can cause other network traffic delays!
      */
-    fun processDataMessage(message: StreamingData, endPoint: EndPoint<CONNECTION>) {
+    fun processDataMessage(message: StreamingData, endPoint: EndPoint<CONNECTION>, connection: CONNECTION) {
         // the receiving data will ALWAYS come sequentially, but there might be OTHER streaming data received meanwhile.
-        val streamId = message.streamId
+        // NOTE: the stream session ID is a combination of the connection ID + random ID (on the receiving side)
+        val streamId = (connection.id.toLong() shl 4) or message.streamId.toLong()
 
-        val controlMessage = streamingDataTarget[streamId]
-        if (controlMessage != null) {
-            streamingDataInMemory.getOrPut(streamId) { AeronOutput() }.writeBytes(message.payload!!)
+        val dataWriter = streamingDataInMemory[streamId]
+        if (dataWriter != null) {
+            dataWriter.writeBytes(message.startPosition, message.payload!!)
         } else {
             // something SUPER wrong!
             // more critical error sending the message. we shouldn't retry or anything.
@@ -196,23 +313,25 @@ internal class StreamingManager<CONNECTION : Connection>(private val logger: KLo
             // either client or server. No other choices. We create an exception, because it's more useful!
             val exception = endPoint.newException(errorMessage)
 
-            // +2 because we do not want to see the stack for the abstract `newException`
             // +3 more because we do not need to see the "internals" for sending messages. The important part of the stack trace is
             // where we see who is calling "send()"
-            ListenerManager.cleanStackTrace(exception, 5)
+            exception.cleanStackTrace(3)
             throw exception
         }
     }
 
     private fun sendFailMessageAndThrow(
         e: Exception,
-        streamSessionId: Long,
+        streamSessionId: Int,
         publication: Publication,
         endPoint: EndPoint<CONNECTION>,
-        connection: CONNECTION
+        sendIdleStrategy: IdleStrategy,
+        connection: CONNECTION,
+        kryo: KryoWriter<CONNECTION>
     ) {
-        val failMessage = StreamingControl(StreamingState.FAILED, streamSessionId)
-        val failSent = endPoint.send(failMessage, publication, connection)
+        val failMessage = StreamingControl(StreamingState.FAILED, false, streamSessionId)
+
+        val failSent = endPoint.writeUnsafe(failMessage, publication, sendIdleStrategy, connection, kryo)
         if (!failSent) {
             // something SUPER wrong!
             // more critical error sending the message. we shouldn't retry or anything.
@@ -221,10 +340,9 @@ internal class StreamingManager<CONNECTION : Connection>(private val logger: KLo
             // either client or server. No other choices. We create an exception, because it's more useful!
             val exception = endPoint.newException(errorMessage)
 
-            // +2 because we do not want to see the stack for the abstract `newException`
             // +4 more because we do not need to see the "internals" for sending messages. The important part of the stack trace is
             // where we see who is calling "send()"
-            ListenerManager.cleanStackTrace(exception, 6)
+            exception.cleanStackTrace(4)
             throw exception
         } else {
             // send it up!
@@ -241,84 +359,103 @@ internal class StreamingManager<CONNECTION : Connection>(private val logger: KLo
      * We don't write max possible length per message, we write out MTU (payload) length (so aeron doesn't fragment the message).
      * The max possible length is WAY, WAY more than the max payload length.
      *
-     * @param internalBuffer this is the ORIGINAL object data that is to be "chunked" and sent across the wire
-     * @return true if ALL the message chunks were successfully sent by aeron, false otherwise. Exceptions are caught and rethrown!
+     * @param originalBuffer this is the ORIGINAL object data that is to be blocks sent across the wire
+     *
+     * @return true if ALL the message blocks were successfully sent by aeron, false otherwise. Exceptions are caught and rethrown!
      */
     fun send(
         publication: Publication,
-        internalBuffer: MutableDirectBuffer,
+        originalBuffer: MutableDirectBuffer,
+        maxMessageSize: Int,
         objectSize: Int,
         endPoint: EndPoint<CONNECTION>,
-        connection: CONNECTION): Boolean {
-
+        kryo: KryoWriter<CONNECTION>,
+        sendIdleStrategy: IdleStrategy,
+        connection: CONNECTION
+    ): Boolean {
         // NOTE: our max object size for IN-MEMORY messages is an INT. For file transfer it's a LONG (so everything here is cast to a long)
         var remainingPayload = objectSize
         var payloadSent = 0
 
-        val streamSessionId = random.nextLong()
+
+        // NOTE: the stream session ID is a combination of the connection ID + random ID (on the receiving side)
+        val streamSessionId = CryptoManagement.secureRandom.nextInt()
 
         // tell the other side how much data we are sending
-        val startMessage = StreamingControl(StreamingState.START, streamSessionId, objectSize.toLong())
-        val startSent = endPoint.send(startMessage, publication, connection)
+        val startMessage = StreamingControl(StreamingState.START, false, streamSessionId, remainingPayload.toLong())
+
+        val startSent = endPoint.writeUnsafe(startMessage, publication, sendIdleStrategy, connection, kryo)
         if (!startSent) {
             // more critical error sending the message. we shouldn't retry or anything.
-            val errorMessage = "[${publication.sessionId()}] Error starting streaming content."
+            val errorMessage = "[${publication.sessionId()}] Error starting streaming content (could not send data)."
 
             // either client or server. No other choices. We create an exception, because it's more useful!
             val exception = endPoint.newException(errorMessage)
 
-            // +2 because we do not want to see the stack for the abstract `newException`
             // +3 more because we do not need to see the "internals" for sending messages. The important part of the stack trace is
             // where we see who is calling "send()"
-            ListenerManager.cleanStackTrace(exception, 5)
+            exception.cleanStackTrace(3)
             throw exception
         }
 
 
-        val kryo: KryoExtra<CONNECTION> = endPoint.serialization.takeKryo()
-
-        // we do the FIRST chunk super-weird, because of the way we copy data around (we inject headers,
+        // we do the FIRST block super-weird, because of the way we copy data around (we inject headers,
         // so the first message is SUPER tiny and is a COPY, the rest are no-copy.
 
-        // This is REUSED to prevent garbage collection issues.
-        val chunkData = StreamingData(streamSessionId)
-
         // payload size is for a PRODUCER, and not SUBSCRIBER, so we have to include this amount every time.
-        // MINOR fragmentation by aeron is OK, since that will greatly speed up data transfer rates!
 
-        // the maxPayloadLength MUST ABSOLUTELY be less that the max size + header!
-        var sizeOfPayload = publication.maxMessageLength() - 200
+        var sizeOfBlockData = maxMessageSize
 
         val header: ByteArray
         val headerSize: Int
 
         try {
-            val objectBuffer = kryo.write(connection, chunkData)
+            // This is REUSED to prevent garbage collection issues.
+            val blockData = StreamingData(streamSessionId)
+            val objectBuffer = kryo.write(connection, blockData)
             headerSize = objectBuffer.position()
             header = ByteArray(headerSize)
 
-            // we have to account for the header + the MAX optimized int size
-            sizeOfPayload -= (headerSize + 5)
+            // we have to account for the header + the MAX optimized int size (position and data-length)
+            val dataSize = headerSize + 5 + 5
+            sizeOfBlockData -= dataSize
 
             // this size might be a LITTLE too big, but that's ok, since we only make this specific buffer once.
-            val chunkBuffer = AeronOutput(headerSize + sizeOfPayload)
+            val blockBuffer = AeronOutput(dataSize)
 
             // copy out our header info
             objectBuffer.internalBuffer.getBytes(0, header, 0, headerSize)
 
             // write out our header
-            chunkBuffer.writeBytes(header)
+            blockBuffer.writeBytes(header)
 
-            // write out the payload size using optimized data structures.
-            val varIntSize = chunkBuffer.writeVarInt(sizeOfPayload, true)
+            // write out the start-position (of the payload). First start-position is always 0
+            val positionIntSize = blockBuffer.writeVarInt(0, true)
+
+            // write out the payload size
+            val payloadIntSize = blockBuffer.writeVarInt(sizeOfBlockData, true)
 
             // write out the payload. Our resulting data written out is the ACTUAL MTU of aeron.
-            internalBuffer.getBytes(0, chunkBuffer.internalBuffer, headerSize + varIntSize, sizeOfPayload)
+            originalBuffer.getBytes(0, blockBuffer.internalBuffer, headerSize + positionIntSize + payloadIntSize, sizeOfBlockData)
 
-            remainingPayload -= sizeOfPayload
-            payloadSent += sizeOfPayload
+            remainingPayload -= sizeOfBlockData
+            payloadSent += sizeOfBlockData
+
+            // we reuse/recycle objects, so the payload size is not EXACTLY what is specified
+            val reusedPayloadSize = headerSize + positionIntSize + payloadIntSize + sizeOfBlockData
+
+            val success = endPoint.aeronDriver.send(
+                publication = publication,
+                internalBuffer = blockBuffer.internalBuffer,
+                bufferClaim = kryo.bufferClaim,
+                offset = 0,
+                objectSize = reusedPayloadSize,
+                sendIdleStrategy = sendIdleStrategy,
+                connection = connection,
+                abortEarly = false,
+                listenerManager = endPoint.listenerManager
+            )
 
-            val success = endPoint.sendData(publication, chunkBuffer.internalBuffer, 0, headerSize + varIntSize + sizeOfPayload, connection)
             if (!success) {
                 // something SUPER wrong!
                 // more critical error sending the message. we shouldn't retry or anything.
@@ -327,25 +464,22 @@ internal class StreamingManager<CONNECTION : Connection>(private val logger: KLo
                 // either client or server. No other choices. We create an exception, because it's more useful!
                 val exception = endPoint.newException(errorMessage)
 
-                // +2 because we do not want to see the stack for the abstract `newException`
                 // +3 more because we do not need to see the "internals" for sending messages. The important part of the stack trace is
                 // where we see who is calling "send()"
-                ListenerManager.cleanStackTrace(exception, 5)
+                exception.cleanStackTrace(3)
                 throw exception
             }
         } catch (e: Exception) {
-            sendFailMessageAndThrow(e, streamSessionId, publication, endPoint, connection)
+            sendFailMessageAndThrow(e, streamSessionId, publication, endPoint, sendIdleStrategy, connection, kryo)
             return false // doesn't actually get here because exceptions are thrown, but this makes the IDE happy.
-        } finally {
-            endPoint.serialization.returnKryo(kryo)
         }
 
-        // now send the chunks as fast as possible. Aeron will have us back-off if we send too quickly
+        // now send the block as fast as possible. Aeron will have us back-off if we send too quickly
         while (remainingPayload > 0) {
-            val amountToSend = if (remainingPayload < sizeOfPayload) {
+            val amountToSend = if (remainingPayload < sizeOfBlockData) {
                 remainingPayload
             } else {
-                sizeOfPayload
+                sizeOfBlockData
             }
 
             remainingPayload -= amountToSend
@@ -358,35 +492,56 @@ internal class StreamingManager<CONNECTION : Connection>(private val logger: KLo
             // fortunately, the way that serialization works, we can safely ADD data to the tail and then appropriately read it off
             // on the receiving end without worry.
 
+/// TODO: Compression/encryption??
+
             try {
-                val varIntSize = OptimizeUtilsByteBuf.intLength(sizeOfPayload, true)
-                val writeIndex = payloadSent - headerSize - varIntSize
+                val positionIntSize = OptimizeUtilsByteBuf.intLength(payloadSent, true)
+                val payloadIntSize = OptimizeUtilsByteBuf.intLength(amountToSend, true)
+                val writeIndex = payloadSent - headerSize - positionIntSize - payloadIntSize
 
                 // write out our header data (this will OVERWRITE previous data!)
-                internalBuffer.putBytes(writeIndex, header)
+                originalBuffer.putBytes(writeIndex, header)
 
-                // write out the payload size using optimized data structures.
-                writeVarInt(internalBuffer, writeIndex + headerSize, sizeOfPayload, true)
+                // write out the payload start position
+                writeVarInt(originalBuffer, writeIndex + headerSize, payloadSent, true)
+
+                // write out the payload size
+                writeVarInt(originalBuffer, writeIndex + headerSize + positionIntSize, amountToSend, true)
+
+                // we reuse/recycle objects, so the payload size is not EXACTLY what is specified
+                val reusedPayloadSize = headerSize + payloadIntSize + positionIntSize + amountToSend
 
                 // write out the payload
-                endPoint.sendData(publication, internalBuffer, writeIndex, headerSize + varIntSize + amountToSend, connection)
+                val success = endPoint.aeronDriver.send(
+                    publication = publication,
+                    internalBuffer = originalBuffer,
+                    bufferClaim = kryo.bufferClaim,
+                    offset = writeIndex,
+                    objectSize = reusedPayloadSize,
+                    sendIdleStrategy = sendIdleStrategy,
+                    connection = connection,
+                    abortEarly = false,
+                    listenerManager = endPoint.listenerManager
+                )
+
+                if (!success) {
+                    // critical errors have an exception. Normal "the connection is closed" do not.
+                    return false
+                }
 
                 payloadSent += amountToSend
             } catch (e: Exception) {
-                val failMessage = StreamingControl(StreamingState.FAILED, streamSessionId)
-                val failSent = endPoint.send(failMessage, publication, connection)
+                val failMessage = StreamingControl(StreamingState.FAILED, false, streamSessionId)
+
+                val failSent = endPoint.writeUnsafe(failMessage, publication, sendIdleStrategy, connection, kryo)
                 if (!failSent) {
                     // something SUPER wrong!
                     // more critical error sending the message. we shouldn't retry or anything.
-                    val errorMessage = "[${publication.sessionId()}] Abnormal failure while streaming content."
+                    val errorMessage = "[${publication.sessionId()}] Abnormal failure with exception while streaming content."
 
                     // either client or server. No other choices. We create an exception, because it's more useful!
-                    val exception = endPoint.newException(errorMessage)
-
-                    // +2 because we do not want to see the stack for the abstract `newException`
-                    // +3 more because we do not need to see the "internals" for sending messages. The important part of the stack trace is
-                    // where we see who is calling "send()"
-                    ListenerManager.cleanStackTrace(exception, 5)
+                    val exception = endPoint.newException(errorMessage, e)
+                    exception.cleanAllStackTrace()
                     throw exception
                 } else {
                     // send it up!
@@ -395,8 +550,249 @@ internal class StreamingManager<CONNECTION : Connection>(private val logger: KLo
             }
         }
 
-        // send the last chunk of data
-        val finishedMessage = StreamingControl(StreamingState.FINISHED, streamSessionId, payloadSent.toLong())
-        return endPoint.send(finishedMessage, publication, connection)
+        // send the last block of data
+        val finishedMessage = StreamingControl(StreamingState.FINISHED, false, streamSessionId, payloadSent.toLong())
+
+        return endPoint.writeUnsafe(finishedMessage, publication, sendIdleStrategy, connection, kryo)
+    }
+
+    /**
+     * This is called ONLY when a message is too large to send across the network in a single message (large data messages should
+     * be split into smaller ones anyways!)
+     *
+     * NOTE: this **MUST** stay on the same co-routine that calls "send". This cannot be re-dispatched onto a different coroutine!
+     *
+     * We don't write max possible length per message, we write out MTU (payload) length (so aeron doesn't fragment the message).
+     * The max possible length is WAY, WAY more than the max payload length.
+     *
+     * @param streamSessionId the stream session ID is a combination of the connection ID + random ID (on the receiving side)
+     *
+     * @return true if ALL the message blocks were successfully sent by aeron, false otherwise. Exceptions are caught and rethrown!
+     */
+    @Suppress("SameParameterValue")
+    fun sendFile(
+        file: File,
+        publication: Publication,
+        endPoint: EndPoint<CONNECTION>,
+        kryo: KryoWriter<CONNECTION>,
+        sendIdleStrategy: IdleStrategy,
+        connection: CONNECTION,
+        streamSessionId: Int
+    ): Boolean {
+        val maxMessageSize = connection.maxMessageSize.toLong()
+        val fileInputStream = file.inputStream()
+
+        // if the message is a file, we xfer the file AS a file, and leave it as a temp file (with a file reference to it) on the remote endpoint
+        // the temp file will be unique.
+
+        // NOTE: our max object size for IN-MEMORY messages is an INT. For file transfer it's a LONG (so everything here is cast to a long)
+        var remainingPayload = file.length()
+        var payloadSent = 0
+
+        // tell the other side how much data we are sending
+        val startMessage = StreamingControl(StreamingState.START, true, streamSessionId, remainingPayload)
+
+        val startSent = endPoint.writeUnsafe(startMessage, publication, sendIdleStrategy, connection, kryo)
+        if (!startSent) {
+            fileInputStream.close()
+
+            // more critical error sending the message. we shouldn't retry or anything.
+            val errorMessage = "[${publication.sessionId()}] Error starting streaming file."
+
+            // either client or server. No other choices. We create an exception, because it's more useful!
+            val exception = endPoint.newException(errorMessage)
+
+            // +3 more because we do not need to see the "internals" for sending messages. The important part of the stack trace is
+            // where we see who is calling "send()"
+            exception.cleanStackTrace(3)
+            throw exception
+        }
+
+
+
+        // we do the FIRST block super-weird, because of the way we copy data around (we inject headers),
+        // so the first message is SUPER tiny and is a COPY, the rest are no-copy.
+
+        // payload size is for a PRODUCER, and not SUBSCRIBER, so we have to include this amount every time.
+
+        // we don't know which is larger, the max message size or the file size!
+        var sizeOfBlockData = maxMessageSize.coerceAtMost(remainingPayload).toInt()
+
+        val headerSize: Int
+
+        val buffer: ByteArray
+        val blockBuffer: UnsafeBuffer
+
+        try {
+            // This is REUSED to prevent garbage collection issues.
+            val blockData = StreamingData(streamSessionId)
+            val objectBuffer = kryo.write(connection, blockData)
+            headerSize = objectBuffer.position()
+
+            // we have to account for the header + the MAX optimized int size (position and data-length)
+            val dataSize = headerSize + 5 + 5
+            sizeOfBlockData -= dataSize
+
+            // this size might be a LITTLE too big, but that's ok, since we only make this specific buffer once.
+            buffer = ByteArray(sizeOfBlockData + dataSize)
+            blockBuffer = UnsafeBuffer(buffer)
+
+            // copy out our header info (this skips the header object)
+            objectBuffer.internalBuffer.getBytes(0, buffer, 0, headerSize)
+
+            // write out the start-position (of the payload). First start-position is always 0
+            val positionIntSize = OptimizeUtilsByteArray.writeInt(buffer, 0, true, headerSize)
+
+            // write out the payload size
+            val payloadIntSize = OptimizeUtilsByteArray.writeInt(buffer, sizeOfBlockData, true, headerSize + positionIntSize)
+
+            // write out the payload. Our resulting data written out is the ACTUAL MTU of aeron.
+            val readBytes = fileInputStream.read(buffer, headerSize + positionIntSize + payloadIntSize, sizeOfBlockData)
+            if (readBytes != sizeOfBlockData) {
+                // something SUPER wrong!
+                // more critical error sending the message. we shouldn't retry or anything.
+                val errorMessage = "[${publication.sessionId()}] Abnormal failure while streaming file (read bytes was wrong! ${readBytes} - ${sizeOfBlockData}."
+
+                // either client or server. No other choices. We create an exception, because it's more useful!
+                val exception = endPoint.newException(errorMessage)
+
+                // +3 more because we do not need to see the "internals" for sending messages. The important part of the stack trace is
+                // where we see who is calling "send()"
+                exception.cleanStackTrace(3)
+                throw exception
+            }
+
+            remainingPayload -= sizeOfBlockData
+            payloadSent += sizeOfBlockData
+
+            // we reuse/recycle objects, so the payload size is not EXACTLY what is specified
+            val reusedPayloadSize = headerSize + positionIntSize + payloadIntSize + sizeOfBlockData
+
+            val success = endPoint.aeronDriver.send(
+                publication = publication,
+                internalBuffer = blockBuffer,
+                bufferClaim = kryo.bufferClaim,
+                offset = 0,
+                objectSize = reusedPayloadSize,
+                sendIdleStrategy = sendIdleStrategy,
+                connection = connection,
+                abortEarly = false,
+                listenerManager = endPoint.listenerManager
+            )
+
+            if (!success) {
+                // something SUPER wrong!
+                // more critical error sending the message. we shouldn't retry or anything.
+                val errorMessage = "[${publication.sessionId()}] Abnormal failure while streaming file."
+
+                // either client or server. No other choices. We create an exception, because it's more useful!
+                val exception = endPoint.newException(errorMessage)
+
+                // +3 more because we do not need to see the "internals" for sending messages. The important part of the stack trace is
+                // where we see who is calling "send()"
+                exception.cleanStackTrace(3)
+                throw exception
+            }
+        } catch (e: Exception) {
+            fileInputStream.close()
+
+            sendFailMessageAndThrow(e, streamSessionId, publication, endPoint, sendIdleStrategy, connection, kryo)
+            return false // doesn't actually get here because exceptions are thrown, but this makes the IDE happy.
+        }
+
+
+        val aeronDriver = endPoint.aeronDriver
+        val listenerManager = endPoint.listenerManager
+
+        // now send the block as fast as possible. Aeron will have us back-off if we send too quickly
+        while (remainingPayload > 0) {
+            val amountToSend = if (remainingPayload < sizeOfBlockData) {
+                remainingPayload.toInt()
+            } else {
+                sizeOfBlockData
+            }
+
+            remainingPayload -= amountToSend
+
+
+            // to properly do this, we have to be careful with the underlying protocol, in order to avoid copying the buffer multiple times.
+            // the data that will be sent is object data + buffer data. We are sending the SAME parent buffer, just at different spots and
+            // with different headers -- so we don't copy out the data repeatedly
+
+            // fortunately, the way that serialization works, we can safely ADD data to the tail and then appropriately read it off
+            // on the receiving end without worry.
+
+/// TODO: Compression/encryption??
+
+            try {
+                // write out the payload start position
+                val positionIntSize = OptimizeUtilsByteArray.writeInt(buffer, payloadSent, true, headerSize)
+                // write out the payload size
+                val payloadIntSize = OptimizeUtilsByteArray.writeInt(buffer, amountToSend, true, headerSize + positionIntSize)
+
+                // write out the payload. Our resulting data written out is the ACTUAL MTU of aeron.
+                val readBytes = fileInputStream.read(buffer, headerSize + positionIntSize + payloadIntSize, amountToSend)
+                if (readBytes != amountToSend) {
+                    // something SUPER wrong!
+                    // more critical error sending the message. we shouldn't retry or anything.
+                    val errorMessage = "[${publication.sessionId()}] Abnormal failure while streaming file (read bytes was wrong! ${readBytes} - ${amountToSend}."
+
+                    // either client or server. No other choices. We create an exception, because it's more useful!
+                    val exception = endPoint.newException(errorMessage)
+
+                    // +3 more because we do not need to see the "internals" for sending messages. The important part of the stack trace is
+                    // where we see who is calling "send()"
+                    exception.cleanStackTrace(3)
+                    throw exception
+                }
+
+                // we reuse/recycle objects, so the payload size is not EXACTLY what is specified
+                val reusedPayloadSize = headerSize + positionIntSize + payloadIntSize + amountToSend
+
+                // write out the payload
+                aeronDriver.send(
+                    publication = publication,
+                    internalBuffer = blockBuffer,
+                    bufferClaim = kryo.bufferClaim,
+                    offset = 0, // 0 because we are not reading the entire file at once
+                    objectSize = reusedPayloadSize,
+                    sendIdleStrategy = sendIdleStrategy,
+                    connection = connection,
+                    abortEarly = false,
+                    listenerManager = listenerManager
+                )
+
+                payloadSent += amountToSend
+            } catch (e: Exception) {
+                fileInputStream.close()
+
+                val failMessage = StreamingControl(StreamingState.FAILED, false, streamSessionId)
+
+                val failSent = endPoint.writeUnsafe(failMessage, publication, sendIdleStrategy, connection, kryo)
+                if (!failSent) {
+                    // something SUPER wrong!
+                    // more critical error sending the message. we shouldn't retry or anything.
+                    val errorMessage = "[${publication.sessionId()}] Abnormal failure while streaming content."
+
+                    // either client or server. No other choices. We create an exception, because it's more useful!
+                    val exception = endPoint.newException(errorMessage)
+
+                    // +3 more because we do not need to see the "internals" for sending messages. The important part of the stack trace is
+                    // where we see who is calling "send()"
+                    exception.cleanStackTrace(3)
+                    throw exception
+                } else {
+                    // send it up!
+                    throw e
+                }
+            }
+        }
+
+        fileInputStream.close()
+
+        // send the last block of data
+        val finishedMessage = StreamingControl(StreamingState.FINISHED, true, streamSessionId, payloadSent.toLong())
+
+        return endPoint.writeUnsafe(finishedMessage, publication, sendIdleStrategy, connection, kryo)
     }
 }

src/dorkbox/network/connection/streaming/StreamingWriter.kt

@@ -0,0 +1,22 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package dorkbox.network.connection.streaming
+
+interface StreamingWriter {
+    fun writeBytes(startPosition: Int, bytes: ByteArray)
+    fun isFinished(): Boolean
+}

src/dorkbox/network/connection/streaming/package-info.java

@@ -0,0 +1,17 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package dorkbox.network.connection.streaming;

src/dorkbox/network/connectionType/package-info.java

@@ -0,0 +1,17 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package dorkbox.network.connectionType;

src/dorkbox/network/coroutines/SuspendFunctionTrampoline.java

@@ -1,43 +0,0 @@
-/*
- * Copyright 2020 dorkbox, llc
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package dorkbox.network.coroutines;
-
-import org.jetbrains.annotations.NotNull;
-import org.jetbrains.annotations.Nullable;
-
-import kotlin.coroutines.Continuation;
-import kotlin.jvm.functions.Function1;
-
-/**
- * Class to access suspending invocation of methods from kotlin...
- *
- * ULTIMATELY, this is all java bytecode, and the bytecode signature here matches what kotlin expects. The generics type information is
- * discarded at compile time.
- */
-public
-class SuspendFunctionTrampoline {
-
-    /**
-     * trampoline so we can access suspend functions correctly using reflection
-     */
-    @SuppressWarnings("unchecked")
-    @Nullable
-    public static
-    Object invoke(@NotNull final Continuation<?> continuation, @NotNull final Object suspendFunction) throws Throwable {
-        Function1<? super Continuation<? super Object>, ?> suspendFunction1 = (Function1<? super Continuation<? super Object>, ?>) suspendFunction;
-        return suspendFunction1.invoke((Continuation<? super Object>) continuation);
-    }
-}

src/dorkbox/network/exceptions/AllocationException.kt

@@ -1,5 +1,5 @@
 /*
- * Copyright 2020 dorkbox, llc
+ * Copyright 2023 dorkbox, llc
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -16,6 +16,6 @@
 package dorkbox.network.exceptions
 
 /**
- * A session/stream could not be allocated.
+ * A session/stream/resource could not be allocated.
  */
 class AllocationException(message: String) : ServerException(message)

src/dorkbox/network/exceptions/ClientHandshakeException.kt

@@ -0,0 +1,43 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package dorkbox.network.exceptions
+
+/**
+ * The type of exceptions raised by the client handshake.
+ */
+open class ClientHandshakeException : ClientException {
+    /**
+     * Create an exception.
+     *
+     * @param message The message
+     */
+    constructor(message: String) : super(message)
+
+    /**
+     * Create an exception.
+     *
+     * @param cause The cause
+     */
+    constructor(cause: Throwable) : super(cause)
+
+    /**
+     * Create an exception.
+     *
+     * @param message The message
+     *  @param cause The cause
+     */
+    constructor(message: String, cause: Throwable?) : super(message, cause)
+}

src/dorkbox/network/exceptions/MessageDispatchException.kt

@@ -0,0 +1,43 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package dorkbox.network.exceptions
+
+/**
+ * The type of exceptions raised for errors when dispatching messages
+ */
+open class MessageDispatchException : Exception {
+    /**
+     * Create an exception.
+     *
+     * @param message The message
+     */
+    constructor(message: String) : super(message)
+
+    /**
+     * Create an exception.
+     *
+     * @param cause The cause
+     */
+    constructor(cause: Throwable) : super(cause)
+
+    /**
+     * Create an exception.
+     *
+     * @param message The message
+     *  @param cause The cause
+     */
+    constructor(message: String, cause: Throwable?) : super(message, cause)
+}

src/dorkbox/network/exceptions/PingException.kt

@@ -0,0 +1,43 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package dorkbox.network.exceptions
+
+/**
+ * The type of exceptions raised for ping errors
+ */
+open class PingException : Exception {
+    /**
+     * Create an exception.
+     *
+     * @param message The message
+     */
+    constructor(message: String) : super(message)
+
+    /**
+     * Create an exception.
+     *
+     * @param cause The cause
+     */
+    constructor(cause: Throwable) : super(cause)
+
+    /**
+     * Create an exception.
+     *
+     * @param message The message
+     *  @param cause The cause
+     */
+    constructor(message: String, cause: Throwable?) : super(message, cause)
+}

src/dorkbox/network/exceptions/RMIException.kt

@@ -0,0 +1,43 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package dorkbox.network.exceptions
+
+/**
+ * The type of exceptions raised for RMI errors
+ */
+open class RMIException : Exception {
+    /**
+     * Create an exception.
+     *
+     * @param message The message
+     */
+    constructor(message: String) : super(message)
+
+    /**
+     * Create an exception.
+     *
+     * @param cause The cause
+     */
+    constructor(cause: Throwable) : super(cause)
+
+    /**
+     * Create an exception.
+     *
+     * @param message The message
+     *  @param cause The cause
+     */
+    constructor(message: String, cause: Throwable?) : super(message, cause)
+}

src/dorkbox/network/exceptions/SendSyncException.kt

@@ -0,0 +1,43 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package dorkbox.network.exceptions
+
+/**
+ * The type of exceptions raised for send-sync errors
+ */
+open class SendSyncException : Exception {
+    /**
+     * Create an exception.
+     *
+     * @param message The message
+     */
+    constructor(message: String) : super(message)
+
+    /**
+     * Create an exception.
+     *
+     * @param cause The cause
+     */
+    constructor(cause: Throwable) : super(cause)
+
+    /**
+     * Create an exception.
+     *
+     * @param message The message
+     * @param cause The cause
+     */
+    constructor(message: String, cause: Throwable?) : super(message, cause)
+}

src/dorkbox/network/exceptions/SerializationException.kt

@@ -0,0 +1,43 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package dorkbox.network.exceptions
+
+/**
+ * The type of exceptions raised for a serialization error.
+ */
+open class SerializationException : Exception {
+    /**
+     * Create an exception.
+     *
+     * @param message The message
+     */
+    constructor(message: String) : super(message)
+
+    /**
+     * Create an exception.
+     *
+     * @param cause The cause
+     */
+    constructor(cause: Throwable) : super(cause)
+
+    /**
+     * Create an exception.
+     *
+     * @param message The message
+     *  @param cause The cause
+     */
+    constructor(message: String, cause: Throwable?) : super(message, cause)
+}

src/dorkbox/network/exceptions/ServerHandshakeException.kt

@@ -0,0 +1,43 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package dorkbox.network.exceptions
+
+/**
+ * The type of exceptions raised by the server handshake.
+ */
+open class ServerHandshakeException : ServerException {
+    /**
+     * Create an exception.
+     *
+     * @param message The message
+     */
+    constructor(message: String) : super(message)
+
+    /**
+     * Create an exception.
+     *
+     * @param cause The cause
+     */
+    constructor(cause: Throwable) : super(cause)
+
+    /**
+     * Create an exception.
+     *
+     * @param message The message
+     *  @param cause The cause
+     */
+    constructor(message: String, cause: Throwable?) : super(message, cause)
+}

src/dorkbox/network/exceptions/ServerTimedoutException.kt

@@ -0,0 +1,43 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package dorkbox.network.exceptions
+
+/**
+ * The type of exceptions raised by the server handshake when it times out.
+ */
+open class ServerTimedoutException : ServerException {
+    /**
+     * Create an exception.
+     *
+     * @param message The message
+     */
+    constructor(message: String) : super(message)
+
+    /**
+     * Create an exception.
+     *
+     * @param cause The cause
+     */
+    constructor(cause: Throwable) : super(cause)
+
+    /**
+     * Create an exception.
+     *
+     * @param message The message
+     *  @param cause The cause
+     */
+    constructor(message: String, cause: Throwable?) : super(message, cause)
+}

src/dorkbox/network/exceptions/StreamingException.kt

@@ -0,0 +1,43 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package dorkbox.network.exceptions
+
+/**
+ * The type of exceptions raised for a streaming error.
+ */
+open class StreamingException : Exception {
+    /**
+     * Create an exception.
+     *
+     * @param message The message
+     */
+    constructor(message: String) : super(message)
+
+    /**
+     * Create an exception.
+     *
+     * @param cause The cause
+     */
+    constructor(cause: Throwable) : super(cause)
+
+    /**
+     * Create an exception.
+     *
+     * @param message The message
+     *  @param cause The cause
+     */
+    constructor(message: String, cause: Throwable?) : super(message, cause)
+}

src/dorkbox/network/exceptions/TimeoutException.kt

@@ -1,5 +1,5 @@
 /*
- * Copyright 2020 dorkbox, llc
+ * Copyright 2023 dorkbox, llc
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -13,9 +13,9 @@
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
+
 package dorkbox.network.exceptions
 
-/**
- * A port could not be allocated.
- */
-class PortAllocationException(message: String) : ServerException(message)
+class TimeoutException: Exception() {
+
+}

src/dorkbox/network/exceptions/TransmitException.kt

@@ -0,0 +1,43 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package dorkbox.network.exceptions
+
+/**
+ * The type of exceptions raised when transmitting data.
+ */
+open class TransmitException : Exception {
+    /**
+     * Create an exception.
+     *
+     * @param message The message
+     */
+    constructor(message: String) : super(message)
+
+    /**
+     * Create an exception.
+     *
+     * @param cause The cause
+     */
+    constructor(cause: Throwable) : super(cause)
+
+    /**
+     * Create an exception.
+     *
+     * @param message The message
+     *  @param cause The cause
+     */
+    constructor(message: String, cause: Throwable?) : super(message, cause)
+}

src/dorkbox/network/exceptions/package-info.java

@@ -0,0 +1,17 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package dorkbox.network.exceptions;

src/dorkbox/network/handshake/ClientConnectionDriver.kt

@@ -0,0 +1,254 @@
+/*
+ * Copyright 2024 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package dorkbox.network.handshake
+
+import dorkbox.network.aeron.AeronDriver
+import dorkbox.network.aeron.AeronDriver.Companion.getLocalAddressString
+import dorkbox.network.aeron.AeronDriver.Companion.uri
+import dorkbox.network.aeron.controlEndpoint
+import dorkbox.network.aeron.endpoint
+import dorkbox.network.connection.EndPoint
+import dorkbox.network.exceptions.ClientRetryException
+import dorkbox.network.exceptions.ClientTimedOutException
+import io.aeron.CommonContext
+import kotlinx.atomicfu.AtomicBoolean
+import java.net.Inet4Address
+import java.net.InetAddress
+
+
+/**
+ * Set up the subscription + publication channels to the server
+ *
+ * Note: this class is NOT closed the traditional way! It's pub/sub objects are used by the connection (which is where they are closed)
+ *
+ * @throws ClientRetryException if we need to retry to connect
+ * @throws ClientTimedOutException if we cannot connect to the server in the designated time
+ */
+internal class ClientConnectionDriver(val connectionInfo: PubSub) {
+
+    companion object {
+        fun build(
+            shutdown: AtomicBoolean,
+            aeronDriver: AeronDriver,
+            handshakeTimeoutNs: Long,
+            handshakeConnection: ClientHandshakeDriver,
+            connectionInfo: ClientConnectionInfo,
+            port2Server: Int, // this is the port2 value from the server
+            tagName: String
+        ): ClientConnectionDriver {
+            val handshakePubSub = handshakeConnection.pubSub
+            val reliable = handshakePubSub.reliable
+
+            // flipped because we are connecting to these!
+            val sessionIdPub = connectionInfo.sessionIdSub
+            val sessionIdSub = connectionInfo.sessionIdPub
+            val streamIdPub = connectionInfo.streamIdSub
+            val streamIdSub = connectionInfo.streamIdPub
+
+            val isUsingIPC = handshakePubSub.isIpc
+
+            val logInfo: String
+
+            val pubSub: PubSub
+
+            if (isUsingIPC) {
+                // Create a subscription at the given address and port, using the given stream ID.
+                logInfo = "CONNECTION-IPC"
+
+                pubSub = buildIPC(
+                    shutdown = shutdown,
+                    aeronDriver = aeronDriver,
+                    handshakeTimeoutNs = handshakeTimeoutNs,
+                    sessionIdPub = sessionIdPub,
+                    sessionIdSub = sessionIdSub,
+                    streamIdPub = streamIdPub,
+                    streamIdSub = streamIdSub,
+                    reliable = reliable,
+                    tagName = tagName,
+                    logInfo = logInfo
+                )
+            }
+            else {
+                val remoteAddress = handshakePubSub.remoteAddress
+                val remoteAddressString = handshakePubSub.remoteAddressString
+                val portPub = handshakePubSub.portPub
+                val portSub = handshakePubSub.portSub
+
+                logInfo = if (remoteAddress is Inet4Address) {
+                    "CONNECTION-IPv4"
+                } else {
+                    "CONNECTION-IPv6"
+                }
+
+                pubSub = buildUDP(
+                    shutdown = shutdown,
+                    aeronDriver = aeronDriver,
+                    handshakeTimeoutNs = handshakeTimeoutNs,
+                    sessionIdPub = sessionIdPub,
+                    sessionIdSub = sessionIdSub,
+                    streamIdPub = streamIdPub,
+                    streamIdSub = streamIdSub,
+                    remoteAddress = remoteAddress!!,
+                    remoteAddressString = remoteAddressString,
+                    portPub = portPub,
+                    portSub = portSub,
+                    port2Server = port2Server,
+                    reliable = reliable,
+                    tagName = tagName,
+                    logInfo = logInfo
+                )
+            }
+
+            return ClientConnectionDriver(pubSub)
+        }
+
+        @Throws(ClientTimedOutException::class)
+        private fun buildIPC(
+            shutdown: AtomicBoolean,
+            aeronDriver: AeronDriver,
+            handshakeTimeoutNs: Long,
+            sessionIdPub: Int,
+            sessionIdSub: Int,
+            streamIdPub: Int,
+            streamIdSub: Int,
+            reliable: Boolean,
+            tagName: String,
+            logInfo: String
+        ): PubSub {
+            // on close, the publication CAN linger (in case a client goes away, and then comes back)
+            // AERON_PUBLICATION_LINGER_TIMEOUT, 5s by default (this can also be set as a URI param)
+
+            // Create a publication at the given address and port, using the given stream ID.
+            val publicationUri = uri(CommonContext.IPC_MEDIA, sessionIdPub, reliable)
+
+            // NOTE: Handlers are called on the client conductor thread. The client conductor thread expects handlers to do safe
+            //  publication of any state to other threads and not be long running or re-entrant with the client.
+
+
+            // can throw an exception! We catch it in the calling class
+            val publication = aeronDriver.addPublication(publicationUri, streamIdPub, logInfo, true)
+
+            // can throw an exception! We catch it in the calling class
+            // we actually have to wait for it to connect before we continue
+            aeronDriver.waitForConnection(shutdown, publication, handshakeTimeoutNs, logInfo) { cause ->
+                ClientTimedOutException("$logInfo publication cannot connect with server!", cause)
+            }
+
+
+            // Create a subscription at the given address and port, using the given stream ID.
+            val subscriptionUri = uri(CommonContext.IPC_MEDIA, sessionIdSub, reliable)
+            val subscription = aeronDriver.addSubscription(subscriptionUri, streamIdSub, logInfo, true)
+
+
+            // wait for the REMOTE end to also connect to us!
+            aeronDriver.waitForConnection(shutdown, subscription, handshakeTimeoutNs, logInfo) { cause ->
+                ClientTimedOutException("$logInfo subscription cannot connect with server!", cause)
+            }
+
+
+            return PubSub(
+                pub = publication,
+                sub = subscription,
+                sessionIdPub = sessionIdPub,
+                sessionIdSub = sessionIdSub,
+                streamIdPub = streamIdPub,
+                streamIdSub = streamIdSub,
+                reliable = reliable,
+                remoteAddress = null,
+                remoteAddressString = EndPoint.IPC_NAME,
+                portPub = 0,
+                portSub = 0,
+                tagName = tagName
+            )
+        }
+
+        @Throws(ClientTimedOutException::class)
+        private fun buildUDP(
+            shutdown: AtomicBoolean,
+            aeronDriver: AeronDriver,
+            handshakeTimeoutNs: Long,
+            sessionIdPub: Int,
+            sessionIdSub: Int,
+            streamIdPub: Int,
+            streamIdSub: Int,
+            remoteAddress: InetAddress,
+            remoteAddressString: String,
+            portPub: Int,
+            portSub: Int,
+            port2Server: Int, // this is the port2 value from the server
+            reliable: Boolean,
+            tagName: String,
+            logInfo: String,
+        ): PubSub {
+            val isRemoteIpv4 = remoteAddress is Inet4Address
+
+            // on close, the publication CAN linger (in case a client goes away, and then comes back)
+            // AERON_PUBLICATION_LINGER_TIMEOUT, 5s by default (this can also be set as a URI param)
+
+            // Create a publication at the given address and port, using the given stream ID.
+            val publicationUri = uri(CommonContext.UDP_MEDIA, sessionIdPub, reliable)
+                .endpoint(isRemoteIpv4, remoteAddressString, portPub)
+
+
+            // NOTE: Handlers are called on the client conductor thread. The client conductor thread expects handlers to do safe
+            //  publication of any state to other threads and not be long running or re-entrant with the client.
+
+            // can throw an exception! We catch it in the calling class
+            val publication = aeronDriver.addPublication(publicationUri, streamIdPub, logInfo, false)
+
+            // can throw an exception! We catch it in the calling class
+            // we actually have to wait for it to connect before we continue
+            aeronDriver.waitForConnection(shutdown, publication, handshakeTimeoutNs, logInfo) { cause ->
+                ClientTimedOutException("$logInfo publication cannot connect with server $remoteAddressString", cause)
+            }
+
+            // this will cause us to listen on the interface that connects with the remote address, instead of ALL interfaces.
+            val localAddressString = getLocalAddressString(publication, isRemoteIpv4)
+
+
+            // A control endpoint for the subscriptions will cause a periodic service management "heartbeat" to be sent to the
+            // remote endpoint publication, which permits the remote publication to send us data, thereby getting us around NAT
+            val subscriptionUri = uri(CommonContext.UDP_MEDIA, sessionIdSub, reliable)
+                .endpoint(isRemoteIpv4, localAddressString, portSub)
+                .controlEndpoint(isRemoteIpv4, remoteAddressString, port2Server)
+                .controlMode(CommonContext.MDC_CONTROL_MODE_DYNAMIC)
+
+            val subscription = aeronDriver.addSubscription(subscriptionUri, streamIdSub, logInfo, false)
+
+
+            // wait for the REMOTE end to also connect to us!
+            aeronDriver.waitForConnection(shutdown, subscription, handshakeTimeoutNs, logInfo) { cause ->
+                ClientTimedOutException("$logInfo subscription cannot connect with server!", cause)
+            }
+
+            return PubSub(
+                pub = publication,
+                sub = subscription,
+                sessionIdPub = sessionIdPub,
+                sessionIdSub = sessionIdSub,
+                streamIdPub = streamIdPub,
+                streamIdSub = streamIdSub,
+                reliable = reliable,
+                remoteAddress = remoteAddress,
+                remoteAddressString = remoteAddressString,
+                portPub = portPub,
+                portSub = portSub,
+                tagName = tagName
+            )
+        }
+    }
+}

src/dorkbox/network/handshake/ClientConnectionInfo.kt

@@ -1,5 +1,5 @@
 /*
- * Copyright 2020 dorkbox, llc
+ * Copyright 2023 dorkbox, llc
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -15,9 +15,16 @@
  */
 package dorkbox.network.handshake
 
-internal class ClientConnectionInfo(val port: Int = 0,
-                                    val sessionId: Int,
-                                    val streamId: Int = 0,
-                                    val publicKey: ByteArray = ByteArray(0),
-                                    val kryoRegistrationDetails: ByteArray) {
-}
+import javax.crypto.spec.SecretKeySpec
+
+internal class ClientConnectionInfo(
+    val sessionIdPub: Int = 0,
+    val sessionIdSub: Int = 0,
+    val streamIdPub: Int,
+    val streamIdSub: Int = 0,
+    val publicKey: ByteArray = ByteArray(0),
+    val sessionTimeout: Long,
+    val bufferedMessages: Boolean,
+    val kryoRegistrationDetails: ByteArray,
+    val secretKey: SecretKeySpec
+)

src/dorkbox/network/handshake/ClientHandshake.kt

@@ -1,5 +1,5 @@
 /*
- * Copyright 2020 dorkbox, llc
+ * Copyright 2024 dorkbox, llc
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -16,33 +16,32 @@
 package dorkbox.network.handshake
 
 import dorkbox.network.Client
-import dorkbox.network.aeron.mediaDriver.MediaDriverClient
 import dorkbox.network.connection.Connection
 import dorkbox.network.connection.CryptoManagement
-import dorkbox.network.connection.ListenerManager
-import dorkbox.network.exceptions.ClientRejectedException
-import dorkbox.network.exceptions.ClientTimedOutException
-import dorkbox.network.exceptions.ServerException
+import dorkbox.network.connection.EndPoint
+import dorkbox.network.connection.ListenerManager.Companion.cleanAllStackTrace
+import dorkbox.network.connection.ListenerManager.Companion.cleanStackTraceInternal
+import dorkbox.network.exceptions.*
+import dorkbox.util.Sys
 import io.aeron.FragmentAssembler
+import io.aeron.Image
 import io.aeron.logbuffer.FragmentHandler
 import io.aeron.logbuffer.Header
-import mu.KLogger
 import org.agrona.DirectBuffer
-import java.lang.Thread.sleep
-import java.util.concurrent.*
+import org.slf4j.Logger
 
 internal class ClientHandshake<CONNECTION: Connection>(
-    private val crypto: CryptoManagement,
-    private val endPoint: Client<CONNECTION>,
-    private val logger: KLogger
+    private val client: Client<CONNECTION>,
+    private val logger: Logger
 ) {
 
     // @Volatile is used BECAUSE suspension of coroutines can continue on a DIFFERENT thread. We want to make sure that thread visibility is
     // correct when this happens. There are no race-conditions to be wary of.
 
+    private val crypto = client.crypto
     private val handler: FragmentHandler
 
-    private val pollIdleStrategy = endPoint.config.pollIdleStrategy.cloneToNormal()
+    private val handshaker = client.handshaker
 
     // used to keep track and associate UDP/IPC handshakes between client/server
     @Volatile
@@ -61,28 +60,54 @@ internal class ClientHandshake<CONNECTION: Connection>(
     private var failedException: Exception? = null
 
     init {
-        // now we have a bi-directional connection with the server on the handshake "socket".
+        // NOTE: subscriptions (ie: reading from buffers, etc) are not thread safe!  Because it is ambiguous HOW EXACTLY they are unsafe,
+        //  we exclusively read from the DirectBuffer on a single thread.
+
+        // NOTE: Handlers are called on the client conductor thread. The client conductor thread expects handlers to do safe
+        //  publication of any state to other threads and not be:
+        //   - long running
+        //   - re-entrant with the client
         handler = FragmentAssembler { buffer: DirectBuffer, offset: Int, length: Int, header: Header ->
             // this is processed on the thread that calls "poll". Subscriptions are NOT multi-thread safe!
             val sessionId = header.sessionId()
             val streamId = header.streamId()
-            val aeronLogInfo = "$sessionId/$streamId"
 
-            val message = endPoint.readHandshakeMessage(buffer, offset, length, header, aeronLogInfo)
+            // note: this address will ALWAYS be an IP:PORT combo  OR  it will be aeron:ipc  (if IPC, it will be a different handler!)
+            val remoteIpAndPort = (header.context() as Image).sourceIdentity()
+
+            // split
+            val splitPoint = remoteIpAndPort.lastIndexOf(':')
+            val clientAddressString = remoteIpAndPort.substring(0, splitPoint)
+
+            val logInfo = "$sessionId/$streamId:$clientAddressString"
 
             failedException = null
             needToRetry = false
 
-            // it must be a registration message
-            if (message !is HandshakeMessage) {
-                failedException = ClientRejectedException("[$aeronLogInfo] cancelled handshake for unrecognized message: $message")
-                return@FragmentAssembler
-            }
+
+            // ugh, this is verbose -- but necessary
+            val message = try {
+                val msg = handshaker.readMessage(buffer, offset, length)
+
+                // VALIDATE:: a Registration object is the only acceptable message during the connection phase
+                if (msg !is HandshakeMessage) {
+                    throw ClientRejectedException("[$logInfo] Connection not allowed! unrecognized message: $msg") .apply { cleanAllStackTrace() }
+                } else if (logger.isTraceEnabled) {
+                    logger.trace("[$logInfo] (${msg.connectKey}) received HS: $msg")
+                }
+                msg
+            } catch (e: Exception) {
+                client.listenerManager.notifyError(ClientHandshakeException("[$logInfo] Error de-serializing handshake message!!", e))
+                null
+            } ?: return@FragmentAssembler
+
+
 
             // this is an error message
             if (message.state == HandshakeMessage.INVALID) {
-                val cause = ServerException(message.errorMessage ?: "Unknown").apply { stackTrace = stackTrace.copyOfRange(0, 1) }
-                failedException = ClientRejectedException("[$aeronLogInfo} - ${message.connectKey}] cancelled handshake", cause)
+                val cause = ServerException(message.errorMessage ?: "Unknown").apply { stackTrace = emptyArray() }
+                failedException = ClientRejectedException("[$logInfo}] (${message.connectKey}) cancelled handshake", cause)
+                    .apply { cleanAllStackTrace() }
                 return@FragmentAssembler
             }
 
@@ -94,7 +119,7 @@ internal class ClientHandshake<CONNECTION: Connection>(
             }
 
             if (connectKey != message.connectKey) {
-                logger.error("[$aeronLogInfo - $connectKey] ignored handshake for ${message.connectKey} (Was for another client)")
+                logger.error("[$logInfo] ($connectKey) ignored handshake for ${message.connectKey} (Was for another client)")
                 return@FragmentAssembler
             }
 
@@ -110,28 +135,20 @@ internal class ClientHandshake<CONNECTION: Connection>(
                     if (registrationData != null && serverPublicKeyBytes != null) {
                         connectionHelloInfo = crypto.decrypt(registrationData, serverPublicKeyBytes)
                     } else {
-                        failedException = ClientRejectedException("[$aeronLogInfo} - ${message.connectKey}] canceled handshake for message without registration and/or public key info")
+                        failedException = ClientRejectedException("[$logInfo}] (${message.connectKey}) canceled handshake for message without registration and/or public key info")
+                            .apply { cleanAllStackTrace() }
                     }
                 }
                 HandshakeMessage.HELLO_ACK_IPC -> {
                     // The message was intended for this client. Try to parse it as one of the available message types.
                     // this message is NOT-ENCRYPTED!
-                    val cryptInput = crypto.cryptInput
+                    val serverPublicKeyBytes = message.publicKey
 
-                    if (registrationData != null) {
-                        cryptInput.buffer = registrationData
-
-                        val sessId = cryptInput.readInt()
-                        val streamPubId = cryptInput.readInt()
-                        val regDetailsSize = cryptInput.readInt()
-                        val regDetails = cryptInput.readBytes(regDetailsSize)
-
-                        // now read data off
-                        connectionHelloInfo = ClientConnectionInfo(sessionId = sessId,
-                                                                   port = streamPubId,
-                                                                   kryoRegistrationDetails = regDetails)
+                    if (registrationData != null && serverPublicKeyBytes != null) {
+                        connectionHelloInfo = crypto.nocrypt(registrationData, serverPublicKeyBytes)
                     } else {
-                        failedException = ClientRejectedException("[$aeronLogInfo - ${message.connectKey}] canceled handshake for message without registration data")
+                        failedException = ClientRejectedException("[$logInfo}] (${message.connectKey}) canceled handshake for message without registration and/or public key info")
+                                .apply { cleanAllStackTrace() }
                     }
                 }
                 HandshakeMessage.DONE_ACK -> {
@@ -139,7 +156,8 @@ internal class ClientHandshake<CONNECTION: Connection>(
                 }
                 else -> {
                     val stateString = HandshakeMessage.toStateString(message.state)
-                    failedException = ClientRejectedException("[$aeronLogInfo - ${message.connectKey}] cancelled handshake for message that is $stateString")
+                    failedException = ClientRejectedException("[$logInfo] (${message.connectKey}) cancelled handshake for message that is $stateString")
+                        .apply { cleanAllStackTrace() }
                 }
             }
         }
@@ -149,76 +167,75 @@ internal class ClientHandshake<CONNECTION: Connection>(
      * Make sure that NON-ZERO is returned
      */
     private fun getSafeConnectKey(): Long {
-        var key = endPoint.crypto.secureRandom.nextLong()
+        var key = CryptoManagement.secureRandom.nextLong()
         while (key == 0L) {
-            key = endPoint.crypto.secureRandom.nextLong()
+            key = CryptoManagement.secureRandom.nextLong()
         }
 
         return key
     }
 
     // called from the connect thread
-    fun hello(handshakeConnection: MediaDriverClient, connectionTimeoutSec: Int) : ClientConnectionInfo {
-        failedException = null
+    // when exceptions are thrown, the handshake pub/sub will be closed
+    fun hello(
+        tagName: String,
+        endPoint: EndPoint<CONNECTION>,
+        handshakeConnection: ClientHandshakeDriver,
+        handshakeTimeoutNs: Long
+    ) : ClientConnectionInfo {
+        val pubSub = handshakeConnection.pubSub
+
+        // is our pub still connected??
+        if (!pubSub.pub.isConnected) {
+            throw ClientException("Handshake publication is not connected, and it is expected to be connected!")
+        }
+
+        // always make sure that we reset the state when we start (in the event of reconnects)
+        reset()
         connectKey = getSafeConnectKey()
-        val publicKey = endPoint.storage.getPublicKey()!!
-
-        val aeronLogInfo = "${handshakeConnection.remoteSessionId}/${handshakeConnection.streamId}"
-
-        // Send the one-time pad to the server.
-        val publication = handshakeConnection.publication
-        val subscription = handshakeConnection.subscription
 
         try {
-            endPoint.writeHandshakeMessage(publication, aeronLogInfo,
-                                           HandshakeMessage.helloFromClient(connectKey, publicKey,
-                                                                            handshakeConnection.localSessionId,
-                                                                            handshakeConnection.subscriptionPort,
-                                                                            handshakeConnection.subscription.streamId()))
+            // Send the one-time pad to the server.
+            handshaker.writeMessage(pubSub.pub, handshakeConnection.details,
+                                    HandshakeMessage.helloFromClient(
+                                        connectKey = connectKey,
+                                        publicKey = client.storage.publicKey,
+                                        streamIdSub = pubSub.streamIdSub,
+                                        portSub = pubSub.portSub,
+                                        tagName = tagName
+                                    ))
         } catch (e: Exception) {
-            subscription.close()
-            publication.close()
-
-            logger.error("[$aeronLogInfo] Handshake error!", e)
-            throw e
+            handshakeConnection.close(endPoint)
+            throw TransmitException("$handshakeConnection Handshake message error!", e)
         }
 
         // block until we receive the connection information from the server
-        var pollCount: Int
-        pollIdleStrategy.reset()
 
-        val timoutInNanos = TimeUnit.SECONDS.toNanos(connectionTimeoutSec.toLong()) + endPoint.aeronDriver.getLingerNs()
         val startTime = System.nanoTime()
-        while (System.nanoTime() - startTime < timoutInNanos) {
+        while (System.nanoTime() - startTime < handshakeTimeoutNs) {
             // NOTE: regarding fragment limit size. Repeated calls to '.poll' will reassemble a fragment.
             //   `.poll(handler, 4)` == `.poll(handler, 2)` + `.poll(handler, 2)`
-            pollCount = subscription.poll(handler, 1)
+            pubSub.sub.poll(handler, 1)
 
-            if (failedException != null || connectionHelloInfo != null) {
+            if (endPoint.isShutdown() || failedException != null || connectionHelloInfo != null) {
                 break
             }
 
-            // 0 means we idle. >0 means reset and don't idle (because there are likely more)
-            pollIdleStrategy.idle(pollCount)
+            Thread.sleep(100)
         }
 
         val failedEx = failedException
         if (failedEx != null) {
-            // no longer necessary to hold this connection open (if not a failure, we close the handshake after the DONE message)
-            subscription.close()
-            publication.close()
+            handshakeConnection.close(endPoint)
 
-            ListenerManager.cleanStackTraceInternal(failedEx)
+            failedEx.cleanStackTraceInternal()
             throw failedEx
         }
 
         if (connectionHelloInfo == null) {
-            // no longer necessary to hold this connection open (if not a failure, we close the handshake after the DONE message)
-            subscription.close()
-            publication.close()
+            handshakeConnection.close(endPoint)
 
-            val exception = ClientTimedOutException("[$aeronLogInfo] Waiting for registration response from server")
-            ListenerManager.cleanStackTraceInternal(exception)
+            val exception = ClientTimedOutException("$handshakeConnection Waiting for registration response from server for more than ${Sys.getTimePrettyFull(handshakeTimeoutNs)}")
             throw exception
         }
 
@@ -226,39 +243,47 @@ internal class ClientHandshake<CONNECTION: Connection>(
     }
 
     // called from the connect thread
-    fun done(handshakeConnection: MediaDriverClient, connectionTimeoutSec: Int) {
-        val registrationMessage = HandshakeMessage.doneFromClient(connectKey,
-                                                                  handshakeConnection.subscriptionPort,
-                                                                  handshakeConnection.subscription.streamId())
+    // when exceptions are thrown, the handshake pub/sub will be closed
+    fun done(
+        endPoint: EndPoint<CONNECTION>,
+        handshakeConnection: ClientHandshakeDriver,
+        clientConnection: ClientConnectionDriver,
+        handshakeTimeoutNs: Long,
+        logInfo: String
+    ) {
+        val pubSub = clientConnection.connectionInfo
+        val handshakePubSub = handshakeConnection.pubSub
 
-        val aeronLogInfo = "${handshakeConnection.remoteSessionId}/${handshakeConnection.streamId}"
+        // is our pub still connected??
+        if (!pubSub.pub.isConnected) {
+            throw ClientException("Handshake publication is not connected, and it is expected to be connected!")
+        }
 
         // Send the done message to the server.
         try {
-            endPoint.writeHandshakeMessage(handshakeConnection.publication, aeronLogInfo, registrationMessage)
+            handshaker.writeMessage(handshakeConnection.pubSub.pub, logInfo,
+                                    HandshakeMessage.doneFromClient(
+                                        connectKey = connectKey,
+                                        sessionIdSub = handshakePubSub.sessionIdSub,
+                                        streamIdSub = handshakePubSub.streamIdSub
+                                    ))
         } catch (e: Exception) {
-            handshakeConnection.subscription.close()
-            handshakeConnection.publication.close()
-            throw e
+            handshakeConnection.close(endPoint)
+            throw TransmitException("$handshakeConnection Handshake message error!", e)
         }
 
-        // block until we receive the connection information from the server
 
         failedException = null
-        pollIdleStrategy.reset()
+        connectionDone = false
 
-        var pollCount: Int
-
-        val subscription = handshakeConnection.subscription
-
-        val timoutInNanos = TimeUnit.SECONDS.toNanos(connectionTimeoutSec.toLong())
+        // block until we receive the connection information from the server
         var startTime = System.nanoTime()
-        while (System.nanoTime() - startTime < timoutInNanos) {
+        while (System.nanoTime() - startTime < handshakeTimeoutNs) {
             // NOTE: regarding fragment limit size. Repeated calls to '.poll' will reassemble a fragment.
             //   `.poll(handler, 4)` == `.poll(handler, 2)` + `.poll(handler, 2)`
-            pollCount = subscription.poll(handler, 1)
+            handshakePubSub.sub.poll(handler, 1)
 
-            if (failedException != null || connectionDone) {
+            if (endPoint.isShutdown() || failedException != null || connectionDone) {
                 break
             }
 
@@ -269,20 +294,21 @@ internal class ClientHandshake<CONNECTION: Connection>(
                 startTime = System.nanoTime()
             }
 
-            sleep(100L)
-
-            // 0 means we idle. >0 means reset and don't idle (because there are likely more)
-            pollIdleStrategy.idle(pollCount)
+            Thread.sleep(100)
         }
 
         val failedEx = failedException
         if (failedEx != null) {
+            handshakeConnection.close(endPoint)
+
             throw failedEx
         }
 
         if (!connectionDone) {
-            val exception = ClientTimedOutException("Waiting for registration response from server")
-            ListenerManager.cleanStackTraceInternal(exception)
+            // since this failed, close everything
+            handshakeConnection.close(endPoint)
+
+            val exception = ClientTimedOutException("Timed out waiting for registration response from server: ${Sys.getTimePrettyFull(handshakeTimeoutNs)}")
             throw exception
         }
     }

src/dorkbox/network/handshake/ClientHandshakeDriver.kt

@@ -0,0 +1,402 @@
+/*
+ * Copyright 2024 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package dorkbox.network.handshake
+
+import dorkbox.network.aeron.AeronDriver
+import dorkbox.network.aeron.AeronDriver.Companion.getLocalAddressString
+import dorkbox.network.aeron.AeronDriver.Companion.streamIdAllocator
+import dorkbox.network.aeron.AeronDriver.Companion.uri
+import dorkbox.network.aeron.AeronDriver.Companion.uriHandshake
+import dorkbox.network.aeron.controlEndpoint
+import dorkbox.network.aeron.endpoint
+import dorkbox.network.connection.CryptoManagement
+import dorkbox.network.connection.EndPoint
+import dorkbox.network.connection.ListenerManager.Companion.cleanAllStackTrace
+import dorkbox.network.connection.ListenerManager.Companion.cleanStackTraceInternal
+import dorkbox.network.exceptions.ClientException
+import dorkbox.network.exceptions.ClientRetryException
+import dorkbox.network.exceptions.ClientTimedOutException
+import dorkbox.util.Sys
+import io.aeron.CommonContext
+import io.aeron.Subscription
+import kotlinx.atomicfu.AtomicBoolean
+import org.slf4j.Logger
+import java.net.Inet4Address
+import java.net.InetAddress
+import java.util.*
+
+
+
+/**
+ * Set up the subscription + publication channels to the server
+ *
+ * @throws ClientRetryException if we need to retry to connect
+ * @throws ClientTimedOutException if we cannot connect to the server in the designated time
+ */
+internal class ClientHandshakeDriver(
+    val aeronDriver: AeronDriver,
+    val pubSub: PubSub,
+    private val logInfo: String,
+    val details: String
+) {
+    companion object {
+        fun build(
+            endpoint: EndPoint<*>,
+            aeronDriver: AeronDriver,
+            autoChangeToIpc: Boolean,
+            remoteAddress: InetAddress?,
+            remoteAddressString: String,
+            remotePort1: Int,
+            remotePort2: Int,
+            clientListenPort: Int,
+            handshakeTimeoutNs: Long,
+            connectionTimoutInNs: Long,
+            reliable: Boolean,
+            tagName: String,
+            logger: Logger
+        ): ClientHandshakeDriver {
+            logger.trace("Starting client handshake")
+
+            var isUsingIPC = false
+
+            if (autoChangeToIpc) {
+                if (remoteAddress == null) {
+                    logger.info("IPC enabled")
+                } else {
+                    logger.warn("IPC for loopback enabled and aeron is already running. Auto-changing network connection from '$remoteAddressString' -> IPC")
+                }
+                isUsingIPC = true
+            }
+
+
+            var logInfo = ""
+
+            var details = ""
+
+            // this must be unique otherwise we CANNOT connect to the server!
+            val sessionIdPub = CryptoManagement.secureRandom.nextInt()
+
+            // with IPC, the aeron driver MUST be shared, so having a UNIQUE sessionIdPub/Sub is unnecessary.
+//          sessionIdPub = sessionIdAllocator.allocate()
+//          sessionIdSub = sessionIdAllocator.allocate()
+            // streamIdPub is assigned by ipc/udp directly
+            var streamIdPub: Int
+            val streamIdSub = streamIdAllocator.allocate() // sub stream ID so the server can comm back to the client
+
+            var pubSub: PubSub? = null
+
+            val timeoutInfo = if (connectionTimoutInNs > 0L) {
+                "[Handshake: ${Sys.getTimePrettyFull(handshakeTimeoutNs)}, Max connection attempt: ${Sys.getTimePrettyFull(connectionTimoutInNs)}]"
+            } else {
+                "[Handshake: ${Sys.getTimePrettyFull(handshakeTimeoutNs)}, Max connection attempt: Unlimited]"
+            }
+
+            val config = endpoint.config
+            val shutdown = endpoint.shutdown
+
+            if (isUsingIPC) {
+                streamIdPub = config.ipcId
+
+                logInfo = "HANDSHAKE-IPC"
+                details = logInfo
+
+
+                logger.info("Client connecting via IPC. $timeoutInfo")
+
+                try {
+                    pubSub = buildIPC(
+                        shutdown = shutdown,
+                        aeronDriver = aeronDriver,
+                        handshakeTimeoutNs = handshakeTimeoutNs,
+                        sessionIdPub = sessionIdPub,
+                        streamIdPub = streamIdPub,
+                        streamIdSub = streamIdSub,
+                        reliable = reliable,
+                        tagName = tagName,
+                        logInfo = logInfo
+                    )
+                } catch (exception: Exception) {
+                    logger.error("Error initializing IPC connection", exception)
+
+                    // MAYBE the server doesn't have IPC enabled? If no, we need to connect via network instead
+                    isUsingIPC = false
+
+                    // we will retry!
+                    if (remoteAddress == null) {
+                        // the exception will HARD KILL the client, make sure aeron driver is closed.
+                        aeronDriver.close()
+
+                        // if we specified that we MUST use IPC, then we have to throw the exception, because there is no IPC
+                        val clientException = ClientException("Unable to connect via IPC to server. No address specified so fallback is unavailable", exception)
+                        clientException.cleanStackTraceInternal()
+                        throw clientException
+                    }
+                }
+            }
+
+            if (!isUsingIPC) {
+                if (remoteAddress == null) {
+                    val clientException = ClientException("Unable to connect via UDP to server. No address specified!")
+                    clientException.cleanStackTraceInternal()
+                    throw clientException
+                }
+
+                logInfo = if (remoteAddress is Inet4Address) {
+                    "HANDSHAKE-IPv4"
+                } else {
+                    "HANDSHAKE-IPv6"
+                }
+
+                streamIdPub = config.udpId
+
+
+                if (remoteAddress is Inet4Address) {
+                    logger.info("Client connecting to IPv4 $remoteAddressString. $timeoutInfo")
+                } else {
+                    logger.info("Client connecting to IPv6 $remoteAddressString. $timeoutInfo")
+                }
+
+                pubSub = buildUDP(
+                    shutdown = shutdown,
+                    aeronDriver = aeronDriver,
+                    handshakeTimeoutNs = handshakeTimeoutNs,
+                    remoteAddress = remoteAddress,
+                    remoteAddressString = remoteAddressString,
+                    portPub = remotePort1,
+                    portSub = clientListenPort,
+                    port2Server = remotePort2,
+                    sessionIdPub = sessionIdPub,
+                    streamIdPub = streamIdPub,
+                    reliable = reliable,
+                    streamIdSub = streamIdSub,
+                    tagName = tagName,
+                    logInfo = logInfo
+                )
+
+
+                // we have to figure out what our sub port info is, otherwise the server cannot connect back!
+                val subscriptionAddress = try {
+                    getLocalAddressString(pubSub.sub)
+                } catch (e: Exception) {
+                    throw ClientRetryException("$logInfo subscription is not properly created!", e)
+                }
+
+                details = if (subscriptionAddress == remoteAddressString) {
+                    logInfo
+                } else {
+                    "$logInfo $subscriptionAddress -> $remoteAddressString"
+                }
+            }
+
+            return ClientHandshakeDriver(aeronDriver, pubSub!!, logInfo, details)
+        }
+
+        @Throws(ClientTimedOutException::class)
+        private fun buildIPC(
+            shutdown: AtomicBoolean,
+            aeronDriver: AeronDriver,
+            handshakeTimeoutNs: Long,
+            sessionIdPub: Int,
+            streamIdPub: Int,
+            streamIdSub: Int,
+            reliable: Boolean,
+            tagName: String,
+            logInfo: String,
+        ): PubSub {
+            // Create a publication at the given address and port, using the given stream ID.
+            // Note: The Aeron.addPublication method will block until the Media Driver acknowledges the request or a timeout occurs.
+            val publicationUri = uri(CommonContext.IPC_MEDIA, sessionIdPub, reliable)
+
+            // NOTE: Handlers are called on the client conductor thread. The client conductor thread expects handlers to do safe
+            //  publication of any state to other threads and not be long running or re-entrant with the client.
+
+            // For publications, if we add them "too quickly" (faster than the 'linger' timeout), Aeron will throw exceptions.
+            //      ESPECIALLY if it is with the same streamID
+            // this check is in the "reconnect" logic
+
+            // can throw an exception! We catch it in the calling class
+            val publication = aeronDriver.addPublication(publicationUri, streamIdPub, logInfo, true)
+
+            // can throw an exception! We catch it in the calling class
+            // we actually have to wait for it to connect before we continue
+            aeronDriver.waitForConnection(shutdown, publication, handshakeTimeoutNs, logInfo) { cause ->
+                ClientTimedOutException("$logInfo publication cannot connect with server in ${Sys.getTimePrettyFull(handshakeTimeoutNs)}", cause)
+            }
+
+            // Create a subscription at the given address and port, using the given stream ID.
+            val subscriptionUri = uriHandshake(CommonContext.IPC_MEDIA, reliable)
+            val subscription = aeronDriver.addSubscription(subscriptionUri, streamIdSub, logInfo, true)
+
+            return PubSub(
+                pub = publication,
+                sub = subscription,
+                sessionIdPub = sessionIdPub,
+                sessionIdSub = 0,
+                streamIdPub = streamIdPub,
+                streamIdSub = streamIdSub,
+                reliable = reliable,
+                remoteAddress = null,
+                remoteAddressString = EndPoint.IPC_NAME,
+                portPub = 0,
+                portSub = 0,
+                tagName = tagName
+            )
+        }
+
+        @Throws(ClientTimedOutException::class)
+        private fun buildUDP(
+            shutdown: AtomicBoolean,
+            aeronDriver: AeronDriver,
+            handshakeTimeoutNs: Long,
+            remoteAddress: InetAddress,
+            remoteAddressString: String,
+            portPub: Int, // this is the port1 value from the server
+            portSub: Int,
+            port2Server: Int, // this is the port2 value from the server
+            sessionIdPub: Int,
+            streamIdPub: Int,
+            reliable: Boolean,
+            streamIdSub: Int,
+            tagName: String,
+            logInfo: String,
+        ): PubSub {
+            @Suppress("NAME_SHADOWING")
+            var portSub = portSub
+
+            // on close, the publication CAN linger (in case a client goes away, and then comes back)
+            // AERON_PUBLICATION_LINGER_TIMEOUT, 5s by default (this can also be set as a URI param)
+
+            val isRemoteIpv4 = remoteAddress is Inet4Address
+
+            // Create a publication at the given address and port, using the given stream ID.
+            // ANY sessionID for the publication will work, because the SERVER doesn't have it defined
+            val publicationUri = uri(CommonContext.UDP_MEDIA, sessionIdPub, reliable)
+                .endpoint(isRemoteIpv4, remoteAddressString, portPub)
+
+
+            // For publications, if we add them "too quickly" (faster than the 'linger' timeout), Aeron will throw exceptions.
+            //      ESPECIALLY if it is with the same streamID. This was noticed as a problem with IPC
+
+            // NOTE: Handlers are called on the client conductor thread. The client conductor thread expects handlers to do safe
+            //  publication of any state to other threads and not be long running or re-entrant with the client.
+
+
+            // can throw an exception! We catch it in the calling class
+            val publication = aeronDriver.addPublication(publicationUri, streamIdPub, logInfo, false)
+
+            // can throw an exception! We catch it in the calling class
+            // we actually have to wait for it to connect before we continue
+            aeronDriver.waitForConnection(shutdown, publication, handshakeTimeoutNs, logInfo) { cause ->
+                streamIdAllocator.free(streamIdSub) // we don't continue, so close this as well
+                ClientTimedOutException("$logInfo publication cannot connect with server in ${Sys.getTimePrettyFull(handshakeTimeoutNs)}", cause)
+            }
+
+
+            // this will cause us to listen on the interface that connects with the remote address, instead of ALL interfaces.
+            val localAddressString = getLocalAddressString(publication, isRemoteIpv4)
+
+
+            // Create a subscription the given address and port, using the given stream ID.
+            var subscription: Subscription? = null
+
+            if (portSub > -1) {
+                // this means we have EXPLICITLY defined a port, we must try to use it
+
+                // A control endpoint for the subscriptions will cause a periodic service management "heartbeat" to be sent to the
+                // remote endpoint publication, which permits the remote publication to send us data, thereby getting us around NAT
+                val subscriptionUri = uriHandshake(CommonContext.UDP_MEDIA, reliable)
+                    .endpoint(isRemoteIpv4, localAddressString, portSub)
+                    .controlEndpoint(isRemoteIpv4, remoteAddressString, port2Server)
+                    .controlMode(CommonContext.MDC_CONTROL_MODE_DYNAMIC)
+
+                subscription = aeronDriver.addSubscription(subscriptionUri, streamIdSub, logInfo, false)
+            } else {
+                // randomly select what port should be used
+                var retryCount = 100
+                val random = CryptoManagement.secureRandom
+                val isSameMachine = remoteAddress.isLoopbackAddress || remoteAddress == EndPoint.lanAddress
+
+                portSub = random.nextInt(Short.MAX_VALUE-1025) + 1025
+                while (subscription == null && retryCount-- > 0) {
+                    // find a random port to bind to if we are loopback OR if we are the same IP address (not loopback, but to ourselves)
+                    if (isSameMachine) {
+                        // range from 1025-65534
+                        portSub = random.nextInt(Short.MAX_VALUE-1025) + 1025
+                    }
+
+                    try {
+                        // A control endpoint for the subscriptions will cause a periodic service management "heartbeat" to be sent to the
+                        // remote endpoint publication, which permits the remote publication to send us data, thereby getting us around NAT
+                        val subscriptionUri = uriHandshake(CommonContext.UDP_MEDIA, reliable)
+                            .endpoint(isRemoteIpv4, localAddressString, portSub)
+                            .controlEndpoint(isRemoteIpv4, remoteAddressString, port2Server)
+                            .controlMode(CommonContext.MDC_CONTROL_MODE_DYNAMIC)
+
+                        subscription = aeronDriver.addSubscription(subscriptionUri, streamIdSub, logInfo, false)
+                    } catch (ignored: Exception) {
+                        // whoops keep retrying!!
+                    }
+                }
+            }
+
+            if (subscription == null) {
+                val ex = ClientTimedOutException("Cannot create subscription port $logInfo. All attempted ports are invalid")
+                ex.cleanAllStackTrace()
+                throw ex
+            }
+
+            return PubSub(
+                pub = publication,
+                sub = subscription,
+                sessionIdPub = sessionIdPub,
+                sessionIdSub = 0,
+                streamIdPub = streamIdPub,
+                streamIdSub = streamIdSub,
+                reliable = reliable,
+                remoteAddress = remoteAddress,
+                remoteAddressString = remoteAddressString,
+                portPub = portPub,
+                portSub = portSub,
+                tagName = tagName
+            )
+        }
+    }
+
+    fun close(endpoint: EndPoint<*>) {
+        // only the subs are allocated on the client!
+//        sessionIdAllocator.free(pubSub.sessionIdPub)
+//        sessionIdAllocator.free(sessionIdSub)
+//        streamIdAllocator.free(streamIdPub)
+        streamIdAllocator.free(pubSub.streamIdSub)
+
+        // on close, we want to make sure this file is DELETED!
+
+        // we might not be able to close these connections.
+        try {
+            aeronDriver.close(pubSub.sub, logInfo)
+        }
+        catch (e: Exception) {
+            endpoint.listenerManager.notifyError(e)
+        }
+        try {
+            aeronDriver.close(pubSub.pub, logInfo)
+        }
+        catch (e: Exception) {
+            endpoint.listenerManager.notifyError(e)
+        }
+    }
+}

src/dorkbox/network/handshake/ConnectionCounts.kt

@@ -1,3 +1,19 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
 package dorkbox.network.handshake
 
 import org.agrona.collections.Object2IntHashMap
@@ -9,18 +25,22 @@ import java.net.InetAddress
 internal class ConnectionCounts {
     private val connectionsPerIpCounts = Object2IntHashMap<InetAddress>(-1)
 
+    @Synchronized
     fun get(inetAddress: InetAddress): Int {
         return connectionsPerIpCounts.getOrPut(inetAddress) { 0 }
     }
 
+    @Synchronized
     fun increment(inetAddress: InetAddress, currentCount: Int) {
         connectionsPerIpCounts[inetAddress] = currentCount + 1
     }
 
+    @Synchronized
     fun decrement(inetAddress: InetAddress, currentCount: Int) {
         connectionsPerIpCounts[inetAddress] = currentCount - 1
     }
 
+    @Synchronized
     fun decrementSlow(inetAddress: InetAddress) {
         if (connectionsPerIpCounts.containsKey(inetAddress)) {
             val defaultVal = connectionsPerIpCounts.getValue(inetAddress)
@@ -28,11 +48,13 @@ internal class ConnectionCounts {
         }
     }
 
+    @Synchronized
     fun isEmpty(): Boolean {
         return connectionsPerIpCounts.isEmpty()
     }
 
+    @Synchronized
     override fun toString(): String {
-        return connectionsPerIpCounts.entries.joinToString()
+        return connectionsPerIpCounts.entries.map { it.key }.joinToString()
     }
 }

src/dorkbox/network/handshake/HandshakeMessage.kt

@@ -1,5 +1,5 @@
 /*
- * Copyright 2020 dorkbox, llc
+ * Copyright 2023 dorkbox, llc
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -32,18 +32,16 @@ internal class HandshakeMessage private constructor() {
     // -1 means there is an error
     var state = INVALID
 
+    // used to name a connection (via the client)
+    var tag: String = ""
+
     var errorMessage: String? = null
 
-    var subscriptionPort = 0
+    var port = 0
     var streamId = 0
     var sessionId = 0
 
 
-    // by default, this will be a reliable connection. When the client connects to the server, the client will specify if the new connection
-    // is a reliable/unreliable connection when setting up the MediaDriverConnection
-    val isReliable = true
-
-
     // the client sends its registration data to the server to make sure that the registered classes are the same between the client/server
     var registrationData: ByteArray? = null
 
@@ -56,14 +54,15 @@ internal class HandshakeMessage private constructor() {
         const val DONE = 3
         const val DONE_ACK = 4
 
-        fun helloFromClient(connectKey: Long, publicKey: ByteArray, sessionId: Int, subscriptionPort: Int, streamId: Int): HandshakeMessage {
+        fun helloFromClient(connectKey: Long, publicKey: ByteArray, streamIdSub: Int, portSub: Int, tagName: String): HandshakeMessage {
             val hello = HandshakeMessage()
             hello.state = HELLO
             hello.connectKey = connectKey // this is 'bounced back' by the server, so the client knows if it's the correct connection message
             hello.publicKey = publicKey
-            hello.sessionId = sessionId
-            hello.subscriptionPort = subscriptionPort
-            hello.streamId = streamId
+            hello.sessionId = 0 // not used by the server, since it connects in a different way!
+            hello.streamId = streamIdSub
+            hello.port = portSub
+            hello.tag = tagName
             return hello
         }
 
@@ -81,12 +80,12 @@ internal class HandshakeMessage private constructor() {
             return hello
         }
 
-        fun doneFromClient(connectKey: Long, subscriptionPort: Int, streamId: Int): HandshakeMessage {
+        fun doneFromClient(connectKey: Long, sessionIdSub: Int, streamIdSub: Int): HandshakeMessage {
             val hello = HandshakeMessage()
             hello.state = DONE
             hello.connectKey = connectKey // THIS MUST NEVER CHANGE! (the server/client expect this)
-            hello.subscriptionPort = subscriptionPort
-            hello.streamId = streamId
+            hello.sessionId = sessionIdSub
+            hello.streamId = streamIdSub
             return hello
         }
 
@@ -134,6 +133,12 @@ internal class HandshakeMessage private constructor() {
             ", Error: $errorMessage"
         }
 
-        return "HandshakeMessage($stateStr$errorMsg)"
+        val connectInfo = if (connectKey != 0L) {
+            ", key=$connectKey"
+        } else {
+            ""
+        }
+
+        return "HandshakeMessage($tag :: $stateStr$errorMsg sessionId=$sessionId, streamId=$streamId, port=$port${connectInfo})"
     }
 }

src/dorkbox/network/handshake/Handshaker.kt

@@ -0,0 +1,117 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package dorkbox.network.handshake
+
+import dorkbox.network.Configuration
+import dorkbox.network.aeron.AeronDriver
+import dorkbox.network.connection.Connection
+import dorkbox.network.connection.ListenerManager
+import dorkbox.network.connection.ListenerManager.Companion.cleanStackTrace
+import dorkbox.network.exceptions.ClientException
+import dorkbox.network.exceptions.ServerException
+import dorkbox.network.serialization.KryoReader
+import dorkbox.network.serialization.KryoWriter
+import dorkbox.network.serialization.Serialization
+import io.aeron.Publication
+import io.aeron.logbuffer.FrameDescriptor
+import org.agrona.DirectBuffer
+import org.agrona.concurrent.IdleStrategy
+import org.slf4j.Logger
+
+internal class Handshaker<CONNECTION : Connection>(
+    private val logger: Logger,
+    config: Configuration,
+    serialization: Serialization<CONNECTION>,
+    private val listenerManager: ListenerManager<CONNECTION>,
+    val aeronDriver: AeronDriver,
+    val newException: (String, Throwable?) -> Throwable
+) {
+    private val handshakeReadKryo: KryoReader<CONNECTION>
+    private val handshakeWriteKryo: KryoWriter<CONNECTION>
+    private val handshakeSendIdleStrategy: IdleStrategy
+
+    init {
+        val maxMessageSize = FrameDescriptor.computeMaxMessageLength(config.publicationTermBufferLength)
+
+        // All registration MUST happen in-order of when the register(*) method was called, otherwise there are problems.
+
+        handshakeReadKryo = KryoReader(maxMessageSize)
+        handshakeWriteKryo = KryoWriter(maxMessageSize)
+
+        serialization.newHandshakeKryo(handshakeReadKryo)
+        serialization.newHandshakeKryo(handshakeWriteKryo)
+
+        handshakeSendIdleStrategy = config.sendIdleStrategy
+    }
+
+    /**
+     * NOTE: this **MUST** stay on the same co-routine that calls "send". This cannot be re-dispatched onto a different coroutine!
+     *       CANNOT be called in action dispatch. ALWAYS ON SAME THREAD
+     *       Server -> will be network polling thread
+     *       Client -> will be thread that calls `connect()`
+     *
+     * @return true if the message was successfully sent by aeron
+     */
+    @Suppress("DuplicatedCode")
+    internal fun writeMessage(publication: Publication, logInfo: String, message: HandshakeMessage): Boolean {
+        // The handshake sessionId IS NOT globally unique
+        if (logger.isTraceEnabled) {
+            logger.trace("[$logInfo] (${message.connectKey}) send HS: $message")
+        }
+
+        try {
+            val buffer = handshakeWriteKryo.write(message)
+
+            return aeronDriver.send(publication, buffer, logInfo, listenerManager, handshakeSendIdleStrategy)
+        } catch (e: Exception) {
+            // if the driver is closed due to a network disconnect or a remote-client termination, we also must close the connection.
+            if (aeronDriver.internal.mustRestartDriverOnError) {
+                // we had a HARD network crash/disconnect, we close the driver and then reconnect automatically
+                //NOTE: notifyDisconnect IS NOT CALLED!
+            }
+            else if (e is ClientException || e is ServerException) {
+                throw e
+            }
+            else {
+                val exception = newException("[$logInfo] Error serializing handshake message $message", e)
+                exception.cleanStackTrace(2) // 2 because we do not want to see the stack for the abstract `newException`
+                listenerManager.notifyError(exception)
+                throw exception
+            }
+
+            return false
+        } finally {
+            handshakeSendIdleStrategy.reset()
+        }
+    }
+
+    /**
+     * NOTE: CANNOT be called in action dispatch. ALWAYS ON SAME THREAD
+     *
+     * THROWS EXCEPTION IF INVALID READS!
+     *
+     * @param buffer The buffer
+     * @param offset The offset from the start of the buffer
+     * @param length The number of bytes to extract
+     *
+     * @return the message
+     */
+    internal fun readMessage(buffer: DirectBuffer, offset: Int, length: Int): Any? {
+        // NOTE: This ABSOLUTELY MUST be done on the same thread! This cannot be done on a new one, because the buffer could change!
+       return handshakeReadKryo.read(buffer, offset, length)
+    }
+}

src/dorkbox/network/handshake/PortAllocator.kt

@@ -1,120 +0,0 @@
-/*
- * Copyright 2020 dorkbox, llc
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package dorkbox.network.handshake
-
-import org.agrona.collections.IntArrayList
-
-/**
- * An allocator for port numbers.
- *
- * The allocator accepts a base number `p` and a maximum count `n | n > 0`, and will allocate
- * up to `n` numbers, in a random order, in the range `[p, p + n - 1`.
- *
- * @param basePort The base port
- * @param numberOfPortsToAllocate The maximum number of ports that will be allocated
- *
- * @throws IllegalArgumentException If the port range is not valid
- */
-class PortAllocator(basePort: Int, numberOfPortsToAllocate: Int) {
-    private val minPort: Int
-    private val maxPort: Int
-
-    private val portShuffleReset: Int
-    private var portShuffleCount: Int
-    private val freePorts: IntArrayList
-
-    init {
-        if (basePort !in 1..65535) {
-            throw IllegalArgumentException("Base port $basePort must be in the range [1, 65535]")
-        }
-
-        minPort = basePort
-        maxPort = Math.max(basePort+1, basePort + (numberOfPortsToAllocate - 1))
-
-        if (maxPort !in (basePort + 1)..65535) {
-            throw IllegalArgumentException("Uppermost port $maxPort must be in the range [$basePort, 65535]")
-        }
-
-        // every time we add 25% of ports back (via 'free'), reshuffle the ports
-        portShuffleReset = numberOfPortsToAllocate/4
-        portShuffleCount = portShuffleReset
-
-        freePorts = IntArrayList()
-
-        for (port in basePort..maxPort) {
-            freePorts.addInt(port)
-        }
-
-        freePorts.shuffle()
-    }
-
-    /**
-     * Allocate `count` number of ports.
-     *
-     * @param count The number of ports that will be allocated
-     *
-     * @return An array of allocated ports
-     *
-     * @throws PortAllocationException If there are fewer than `count` ports available to allocate
-     */
-    fun allocate(count: Int): IntArray {
-        if (freePorts.size < count) {
-            throw IllegalArgumentException("Too few ports available to allocate $count ports")
-        }
-
-        // reshuffle the ports once we need to re-allocate a new port
-        if (portShuffleCount <= 0) {
-            portShuffleCount = portShuffleReset
-            freePorts.shuffle()
-        }
-
-        val result = IntArray(count)
-        for (index in 0 until count) {
-            val lastValue = freePorts.size - 1
-            val removed = freePorts.removeAt(lastValue)
-            result[index] = removed
-        }
-
-        return result
-    }
-
-    /**
-     * Frees the given ports. Has no effect if the given port is outside of the range considered by the allocator.
-     *
-     * @param ports The array of ports to free
-     */
-    fun free(ports: IntArray) {
-        ports.forEach {
-            free(it)
-        }
-    }
-
-    /**
-     * Free a given port.
-     * <p>
-     * Has no effect if the given port is outside of the range considered by the allocator.
-     *
-     * @param port The port
-     */
-    fun free(port: Int) {
-        if (port in minPort..maxPort) {
-            // add at the end (so we don't have unnecessary array resizes)
-            freePorts.addInt(freePorts.size, port)
-
-            portShuffleCount--
-        }
-    }
-}

src/dorkbox/network/handshake/PubSub.kt

@@ -0,0 +1,72 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package dorkbox.network.handshake
+
+import dorkbox.network.connection.EndPoint
+import io.aeron.Publication
+import io.aeron.Subscription
+import java.net.Inet4Address
+import java.net.InetAddress
+
+data class PubSub(
+    val pub: Publication,
+    val sub: Subscription,
+    val sessionIdPub: Int,
+    val sessionIdSub: Int,
+    val streamIdPub: Int,
+    val streamIdSub: Int,
+    val reliable: Boolean,
+    val remoteAddress: InetAddress?,
+    val remoteAddressString: String,
+    val portPub: Int,
+    val portSub: Int,
+    val tagName: String  // will either be "", or will be "[tag_name]"
+) {
+    val isIpc get() = remoteAddress == null
+
+    fun getLogInfo(extraDetails: Boolean): String {
+        return if (isIpc) {
+            val prefix = if (tagName.isNotEmpty()) {
+                EndPoint.IPC_NAME + " ($tagName)"
+            } else {
+                EndPoint.IPC_NAME
+            }
+
+            if (extraDetails) {
+                "$prefix sessionID: p=${sessionIdPub} s=${sessionIdSub}, streamID: p=${streamIdPub} s=${streamIdSub}, reg: p=${pub.registrationId()} s=${sub.registrationId()}"
+            } else {
+                prefix
+            }
+        } else {
+            var prefix = if (remoteAddress is Inet4Address) {
+                "IPv4 $remoteAddressString"
+            } else {
+                "IPv6 $remoteAddressString"
+            }
+
+            if (tagName.isNotEmpty()) {
+                prefix += " ($tagName)"
+            }
+
+            if (extraDetails) {
+                "$prefix sessionID: p=${sessionIdPub} s=${sessionIdSub}, streamID: p=${streamIdPub} s=${streamIdSub}, port: p=${portPub} s=${portSub}, reg: p=${pub.registrationId()} s=${sub.registrationId()}"
+            } else {
+                prefix
+            }
+        }
+    }
+}

src/dorkbox/network/handshake/RandomId65kAllocator.kt

@@ -1,5 +1,5 @@
 /*
- * Copyright 2020 dorkbox, llc
+ * Copyright 2023 dorkbox, llc
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -15,11 +15,12 @@
  */
 package dorkbox.network.handshake
 
+import dorkbox.network.connection.CryptoManagement
 import dorkbox.network.exceptions.AllocationException
 import dorkbox.objectPool.ObjectPool
 import dorkbox.objectPool.Pool
 import kotlinx.atomicfu.atomic
-import java.security.SecureRandom
+import org.slf4j.LoggerFactory
 
 /**
  * An allocator for random IDs, the maximum number of IDs is an unsigned short (65535).
@@ -32,28 +33,33 @@ import java.security.SecureRandom
  * @param min The minimum ID (inclusive)
  * @param max The maximum ID (exclusive)
  */
-class RandomId65kAllocator(private val min: Int = Integer.MIN_VALUE, max: Int = Integer.MAX_VALUE) {
+class RandomId65kAllocator(private val min: Int, max: Int) {
+    constructor(size: Int): this(1, size + 1)
+
+    companion object {
+        private val logger = LoggerFactory.getLogger("RandomId65k")
+    }
+
 
     private val cache: Pool<Int>
     private val maxAssignments: Int
     private val assigned = atomic(0)
 
-
     init {
         // IllegalArgumentException
-        require(max >= min) {
-            "Maximum value $max must be >= minimum value $min"
-        }
+        require(max >= min) { "Maximum value $max must be >= minimum value $min" }
 
-        maxAssignments = (max - min).coerceIn(1, Short.MAX_VALUE * 2)
+        val max65k = Short.MAX_VALUE * 2
+        maxAssignments = (max - min).coerceIn(1, max65k)
 
         // create a shuffled list of ID's. This operation is ONLY performed ONE TIME per endpoint!
+        // Boxing the Ints here is OK, because they are boxed in the cache as well (so it doesn't matter).
         val ids = ArrayList<Int>(maxAssignments)
-        for (id in min..(min + maxAssignments - 1)) {
+        for (id in min until min + maxAssignments) {
             ids.add(id)
         }
 
-        ids.shuffle(SecureRandom())
+        ids.shuffle(CryptoManagement.secureRandom)
 
         // populate the array of randomly assigned ID's.
         cache = ObjectPool.blocking(ids)
@@ -71,8 +77,12 @@ class RandomId65kAllocator(private val min: Int = Integer.MIN_VALUE, max: Int =
             throw AllocationException("No IDs left to allocate")
         }
 
-        assigned.getAndIncrement()
-        return cache.take()
+        val count = assigned.incrementAndGet()
+        val id = cache.take()
+        if (logger.isTraceEnabled) {
+            logger.trace("Allocating $id (total $count)")
+        }
+        return id
     }
 
     /**
@@ -83,13 +93,16 @@ class RandomId65kAllocator(private val min: Int = Integer.MIN_VALUE, max: Int =
     fun free(id: Int) {
         val assigned = assigned.decrementAndGet()
         if (assigned < 0) {
-            throw AllocationException("Unequal allocate/free method calls.")
+            throw AllocationException("Unequal allocate/free method calls attempting to free [$id] (too many 'free' calls).")
+        }
+        if (logger.isTraceEnabled) {
+            logger.trace("Freeing $id")
         }
         cache.put(id)
     }
 
-    fun isEmpty(): Boolean {
-        return assigned.value == 0
+    fun counts(): Int {
+        return assigned.value
     }
 
     override fun toString(): String {

src/dorkbox/network/handshake/ServerConnectionDriver.kt

@@ -0,0 +1,181 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package dorkbox.network.handshake
+
+import dorkbox.network.aeron.AeronDriver
+import dorkbox.network.aeron.AeronDriver.Companion.uri
+import dorkbox.network.connection.EndPoint
+import dorkbox.network.connection.IpInfo
+import io.aeron.CommonContext
+import java.net.Inet4Address
+import java.net.InetAddress
+
+/**
+ * Set up the subscription + publication channels back to the client
+ *
+ * Note: this class is NOT closed the traditional way! It's pub/sub objects are used by the connection (which is where they are closed)
+ *
+ * This represents the connection PAIR between a server<->client
+ */
+internal class ServerConnectionDriver(val pubSub: PubSub) {
+    companion object {
+        fun build(isIpc: Boolean,
+                  aeronDriver: AeronDriver,
+                  sessionIdPub: Int, sessionIdSub: Int,
+                  streamIdPub: Int, streamIdSub: Int,
+
+                  ipInfo: IpInfo,
+                  remoteAddress: InetAddress?,
+                  remoteAddressString: String,
+                  portPubMdc: Int, portPub: Int, portSub: Int,
+                  reliable: Boolean,
+                  tagName: String,
+                  logInfo: String): ServerConnectionDriver {
+
+            val pubSub: PubSub
+
+            if (isIpc) {
+                pubSub = buildIPC(
+                        aeronDriver = aeronDriver,
+                        sessionIdPub = sessionIdPub,
+                        sessionIdSub = sessionIdSub,
+                        streamIdPub = streamIdPub,
+                        streamIdSub = streamIdSub,
+                        reliable = reliable,
+                        tagName = tagName,
+                        logInfo = logInfo
+                    )
+            } else {
+                pubSub = buildUdp(
+                    aeronDriver = aeronDriver,
+                    ipInfo = ipInfo,
+                    sessionIdPub = sessionIdPub,
+                    sessionIdSub = sessionIdSub,
+                    streamIdPub = streamIdPub,
+                    streamIdSub = streamIdSub,
+                    remoteAddress = remoteAddress!!,
+                    remoteAddressString = remoteAddressString,
+                    portPubMdc = portPubMdc,
+                    portPub = portPub,
+                    portSub = portSub,
+                    reliable = reliable,
+                    tagName = tagName,
+                    logInfo = logInfo
+                )
+            }
+
+            return ServerConnectionDriver(pubSub)
+        }
+
+        private fun buildIPC(
+            aeronDriver: AeronDriver,
+            sessionIdPub: Int, sessionIdSub: Int,
+            streamIdPub: Int, streamIdSub: Int,
+            reliable: Boolean,
+            tagName: String,
+            logInfo: String
+        ): PubSub {
+            // on close, the publication CAN linger (in case a client goes away, and then comes back)
+            // AERON_PUBLICATION_LINGER_TIMEOUT, 5s by default (this can also be set as a URI param)
+
+            // create a new publication for the connection (since the handshake ALWAYS closes the current publication)
+            val publicationUri = uri(CommonContext.IPC_MEDIA, sessionIdPub, reliable)
+
+            // NOTE: Handlers are called on the client conductor thread. The client conductor thread expects handlers to do safe
+            //  publication of any state to other threads and not be long running or re-entrant with the client.
+            val publication = aeronDriver.addPublication(publicationUri, streamIdPub, logInfo, true)
+
+            // Create a subscription at the given address and port, using the given stream ID.
+            val subscriptionUri = uri(CommonContext.IPC_MEDIA, sessionIdSub, reliable)
+            val subscription = aeronDriver.addSubscription(subscriptionUri, streamIdSub, logInfo, true)
+
+            return PubSub(
+                pub = publication,
+                sub = subscription,
+                sessionIdPub = sessionIdPub,
+                sessionIdSub = sessionIdSub,
+                streamIdPub = streamIdPub,
+                streamIdSub = streamIdSub,
+                reliable = reliable,
+                remoteAddress = null,
+                remoteAddressString = EndPoint.IPC_NAME,
+                portPub = 0,
+                portSub = 0,
+                tagName = tagName
+            )
+        }
+
+        private fun buildUdp(
+            aeronDriver: AeronDriver,
+            ipInfo: IpInfo,
+            sessionIdPub: Int, sessionIdSub: Int,
+            streamIdPub: Int, streamIdSub: Int,
+            remoteAddress: InetAddress, remoteAddressString: String,
+            portPubMdc: Int, // this is the MDC port - used to dynamically discover the portPub value (but we manually save this info)
+            portPub: Int,
+            portSub: Int,
+            reliable: Boolean,
+            tagName: String,
+            logInfo: String
+        ): PubSub {
+            // on close, the publication CAN linger (in case a client goes away, and then comes back)
+            // AERON_PUBLICATION_LINGER_TIMEOUT, 5s by default (this can also be set as a URI param)
+
+            // connection timeout of 0 doesn't matter. it is not used by the server
+            // the client address WILL BE either IPv4 or IPv6
+            val isRemoteIpv4 = remoteAddress is Inet4Address
+
+            // create a new publication for the connection (since the handshake ALWAYS closes the current publication)
+
+            // we explicitly have the publisher "connect to itself", because we are using MDC to work around NAT
+
+            // A control endpoint for the subscriptions will cause a periodic service management "heartbeat" to be sent to the
+            // remote endpoint publication, which permits the remote publication to send us data, thereby getting us around NAT
+            val publicationUri = uri(CommonContext.UDP_MEDIA, sessionIdPub, reliable)
+                .controlEndpoint(ipInfo.getAeronPubAddress(isRemoteIpv4) + ":" + portPubMdc) // this is the control port! (listens to status messages and NAK from client)
+
+
+            // NOTE: Handlers are called on the client conductor thread. The client conductor thread expects handlers to do safe
+            //  publication of any state to other threads and not be long running or re-entrant with the client.
+            val publication = aeronDriver.addPublication(publicationUri, streamIdPub, logInfo, false)
+
+            // if we are IPv6 WILDCARD -- then our subscription must ALSO be IPv6, even if our connection is via IPv4
+
+            // Create a subscription at the given address and port, using the given stream ID.
+            val subscriptionUri = uri(CommonContext.UDP_MEDIA, sessionIdSub, reliable)
+                .endpoint(ipInfo.formattedListenAddressString + ":" + portSub)
+
+
+            val subscription = aeronDriver.addSubscription(subscriptionUri, streamIdSub, logInfo, false)
+
+            return PubSub(
+                pub = publication,
+                sub = subscription,
+                sessionIdPub = sessionIdPub,
+                sessionIdSub = sessionIdSub,
+                streamIdPub = streamIdPub,
+                streamIdSub = streamIdSub,
+                reliable = reliable,
+                remoteAddress = remoteAddress,
+                remoteAddressString = remoteAddressString,
+                portPub = portPub,
+                portSub = portSub,
+                tagName = tagName
+            )
+        }
+    }
+}

src/dorkbox/network/handshake/ServerHandshake.kt

@@ -1,5 +1,5 @@
 /*
- * Copyright 2020 dorkbox, llc
+ * Copyright 2023 dorkbox, llc
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -18,24 +18,20 @@ package dorkbox.network.handshake
 import dorkbox.network.Server
 import dorkbox.network.ServerConfiguration
 import dorkbox.network.aeron.AeronDriver
-import dorkbox.network.aeron.mediaDriver.MediaDriverConnectInfo
-import dorkbox.network.aeron.mediaDriver.MediaDriverConnection
-import dorkbox.network.aeron.mediaDriver.ServerIpcDriver
-import dorkbox.network.aeron.mediaDriver.UdpMediaDriverPairedConnection
-import dorkbox.network.connection.Connection
-import dorkbox.network.connection.ConnectionParams
-import dorkbox.network.connection.ListenerManager
-import dorkbox.network.connection.PublicKeyValidationState
+import dorkbox.network.aeron.AeronDriver.Companion.sessionIdAllocator
+import dorkbox.network.aeron.AeronDriver.Companion.streamIdAllocator
+import dorkbox.network.connection.*
 import dorkbox.network.exceptions.AllocationException
+import dorkbox.network.exceptions.ServerHandshakeException
+import dorkbox.network.exceptions.ServerTimedoutException
+import dorkbox.network.exceptions.TransmitException
 import io.aeron.Publication
-import kotlinx.coroutines.CoroutineScope
-import kotlinx.coroutines.launch
-import kotlinx.coroutines.runBlocking
-import mu.KLogger
 import net.jodah.expiringmap.ExpirationPolicy
 import net.jodah.expiringmap.ExpiringMap
+import org.slf4j.Logger
 import java.net.Inet4Address
 import java.net.InetAddress
+import java.util.*
 import java.util.concurrent.*
 
 
@@ -46,45 +42,62 @@ import java.util.concurrent.*
  */
 @Suppress("DuplicatedCode", "JoinDeclarationAndAssignment")
 internal class ServerHandshake<CONNECTION : Connection>(
-    private val logger: KLogger,
     private val config: ServerConfiguration,
     private val listenerManager: ListenerManager<CONNECTION>,
-    aeronDriver: AeronDriver
+    private val aeronDriver: AeronDriver,
+    private val eventDispatch: EventDispatcher
 ) {
 
     // note: the expire time here is a LITTLE longer than the expire time in the client, this way we can adjust for network lag if it's close
     private val pendingConnections = ExpiringMap.builder()
-        // we MUST include the publication linger timeout, otherwise we might encounter problems that are NOT REALLY problems
-        .expiration(TimeUnit.SECONDS.toNanos(config.connectionCloseTimeoutInSeconds.toLong() * 2) + aeronDriver.getLingerNs(), TimeUnit.NANOSECONDS)
+        .apply {
+            // connections are extremely difficult to diagnose when the connection timeout is short
+            val timeUnit = if (EndPoint.DEBUG_CONNECTIONS) { TimeUnit.HOURS } else { TimeUnit.NANOSECONDS }
+
+            // we MUST include the publication linger timeout, otherwise we might encounter problems that are NOT REALLY problems
+            this.expiration(TimeUnit.SECONDS.toNanos(config.connectionCloseTimeoutInSeconds.toLong() * 2) + aeronDriver.lingerNs(), timeUnit)
+        }
         .expirationPolicy(ExpirationPolicy.CREATED)
         .expirationListener<Long, CONNECTION> { clientConnectKey, connection ->
             // this blocks until it fully runs (which is ok. this is fast)
-            logger.error { "[${clientConnectKey} Connection (${connection.id}) Timed out waiting for registration response from client" }
+            listenerManager.notifyError(ServerTimedoutException("[${clientConnectKey} Connection (${connection.id}) Timed out waiting for registration response from client"))
             connection.close()
         }
         .build<Long, CONNECTION>()
 
 
-    private val connectionsPerIpCounts = ConnectionCounts()
+    internal val connectionsPerIpCounts = ConnectionCounts()
 
-    // guarantee that session/stream ID's will ALWAYS be unique! (there can NEVER be a collision!)
-    private val sessionIdAllocator = RandomId65kAllocator(AeronDriver.RESERVED_SESSION_ID_LOW, AeronDriver.RESERVED_SESSION_ID_HIGH)
-    private val streamIdAllocator = RandomId65kAllocator(1, Integer.MAX_VALUE)
+    /**
+     * how long does the initial handshake take to connect
+     */
+    internal var handshakeTimeoutNs: Long
 
+    init {
+        // we MUST include the publication linger timeout, otherwise we might encounter problems that are NOT REALLY problems
+        var handshakeTimeoutNs = TimeUnit.SECONDS.toNanos(config.connectionCloseTimeoutInSeconds.toLong()) + aeronDriver.publicationConnectionTimeoutNs() + aeronDriver.lingerNs()
+
+        if (EndPoint.DEBUG_CONNECTIONS) {
+            // connections are extremely difficult to diagnose when the connection timeout is short
+            handshakeTimeoutNs = TimeUnit.HOURS.toNanos(1)
+        }
+
+        this.handshakeTimeoutNs = handshakeTimeoutNs
+    }
 
     /**
      * @return true if we should continue parsing the incoming message, false if we should abort (as we are DONE processing data)
      */
     // note: CANNOT be called in action dispatch. ALWAYS ON SAME THREAD. ONLY RESPONSES ARE ON ACTION DISPATCH!
-    private fun validateMessageTypeAndDoPending(
+    fun validateMessageTypeAndDoPending(
         server: Server<CONNECTION>,
-        actionDispatch: CoroutineScope,
+        handshaker: Handshaker<CONNECTION>,
         handshakePublication: Publication,
         message: HandshakeMessage,
-        connectionString: String,
-        aeronLogInfo: String,
-        logger: KLogger
+        logInfo: String,
+        logger: Logger
     ): Boolean {
+
         // check to see if this sessionId is ALREADY in use by another connection!
         // this can happen if there are multiple connections from the SAME ip address (ie: localhost)
         if (message.state == HandshakeMessage.HELLO) {
@@ -92,15 +105,17 @@ internal class ServerHandshake<CONNECTION : Connection>(
 
             val existingConnection = pendingConnections[message.connectKey]
             if (existingConnection != null) {
-                val existingAeronLogInfo = "${existingConnection.id}/${existingConnection.streamId}"
+                // Server is the "source", client mirrors the server
 
                 // WHOOPS! tell the client that it needs to retry, since a DIFFERENT client has a handshake in progress with the same sessionId
-                logger.error { "[$existingAeronLogInfo - ${message.connectKey}] Connection from $connectionString had an in-use session ID! Telling client to retry." }
+                listenerManager.notifyError(ServerHandshakeException("[$existingConnection] (${message.connectKey}) Connection had an in-use session ID! Telling client to retry."))
 
                 try {
-                    server.writeHandshakeMessage(handshakePublication, aeronLogInfo, HandshakeMessage.retry("Handshake already in progress for sessionID!"))
+                    handshaker.writeMessage(handshakePublication,
+                                            logInfo,
+                                            HandshakeMessage.retry("Handshake already in progress for sessionID!"))
                 } catch (e: Error) {
-                    logger.error(e) { "[$aeronLogInfo - $existingAeronLogInfo] Handshake error!" }
+                    listenerManager.notifyError(ServerHandshakeException("[$existingConnection] Handshake error", e))
                 }
                 return false
             }
@@ -108,48 +123,47 @@ internal class ServerHandshake<CONNECTION : Connection>(
 
         // check to see if this is a pending connection
         if (message.state == HandshakeMessage.DONE) {
-            val existingConnection = pendingConnections.remove(message.connectKey)
-            if (existingConnection == null) {
-                logger.error { "[$aeronLogInfo - ${message.connectKey}] Error! Pending connection from client $connectionString was null, and cannot complete handshake!" }
+            val newConnection = pendingConnections.remove(message.connectKey)
+            if (newConnection == null) {
+                listenerManager.notifyError(ServerHandshakeException("[?????] (${message.connectKey}) Error! Pending connection from client was null, and cannot complete handshake!"))
                 return true
-            } else {
-                val existingAeronLogInfo = "${existingConnection.id}/${existingConnection.streamId}"
-
-                logger.debug { "[$aeronLogInfo - $existingAeronLogInfo - ${message.connectKey}] Connection from $connectionString done with handshake." }
-
-                // called on connection.close()
-                existingConnection.closeAction = {
-                    // clean up the resources associated with this connection when it's closed
-                    logger.debug { "[$existingAeronLogInfo] freeing resources" }
-                    existingConnection.cleanup(connectionsPerIpCounts, sessionIdAllocator, streamIdAllocator)
-
-                    // this always has to be on event dispatch, otherwise we can have weird logic loops if we reconnect within a disconnect callback
-                    actionDispatch.launch {
-                        existingConnection.doNotifyDisconnect()
-                    }
-                }
-
-                // before we finish creating the connection, we initialize it (in case there needs to be logic that happens-before `onConnect` calls occur
-                listenerManager.notifyInit(existingConnection)
-
-                // this enables the connection to start polling for messages
-                server.addConnection(existingConnection)
-
-                // now tell the client we are done
-                try {
-                    server.writeHandshakeMessage(handshakePublication, aeronLogInfo,
-                                                 HandshakeMessage.doneToClient(message.connectKey))
-
-                    // this always has to be on event dispatch, otherwise we can have weird logic loops if we reconnect within a disconnect callback
-                    actionDispatch.launch {
-                        listenerManager.notifyConnect(existingConnection)
-                    }
-                } catch (e: Exception) {
-                    logger.error(e) { "$aeronLogInfo - $existingAeronLogInfo - Handshake error!" }
-                }
-
-                return false
             }
+
+            val connectionType = if (newConnection.enableBufferedMessages) {
+                "Buffered connection"
+            } else {
+                "Connection"
+            }
+
+            // Server is the "source", client mirrors the server
+            if (logger.isTraceEnabled) {
+                logger.trace("[${newConnection}] (${message.connectKey}) $connectionType (${newConnection.id}) done with handshake.")
+            } else if (logger.isDebugEnabled) {
+                logger.debug("[${newConnection}] $connectionType (${newConnection.id}) done with handshake.")
+            }
+
+            newConnection.setImage()
+
+            // before we finish creating the connection, we initialize it (in case there needs to be logic that happens-before `onConnect` calls
+            listenerManager.notifyInit(newConnection)
+
+            // this enables the connection to start polling for messages
+            server.addConnection(newConnection)
+
+            // now tell the client we are done
+            try {
+                handshaker.writeMessage(handshakePublication,
+                                        logInfo,
+                                        HandshakeMessage.doneToClient(message.connectKey))
+
+                listenerManager.notifyConnect(newConnection)
+
+                newConnection.sendBufferedMessages()
+            } catch (e: Exception) {
+                listenerManager.notifyError(newConnection, TransmitException("[$newConnection] Handshake error", e))
+            }
+
+            return false
         }
 
         return true
@@ -161,24 +175,23 @@ internal class ServerHandshake<CONNECTION : Connection>(
     // note: CANNOT be called in action dispatch. ALWAYS ON SAME THREAD
     private fun validateUdpConnectionInfo(
         server: Server<CONNECTION>,
+        handshaker: Handshaker<CONNECTION>,
         handshakePublication: Publication,
         config: ServerConfiguration,
-        clientAddressString: String,
         clientAddress: InetAddress,
-        aeronLogInfo: String,
-        logger: KLogger
+        logInfo: String
     ): Boolean {
 
         try {
             // VALIDATE:: Check to see if there are already too many clients connected.
-            if (server.connections.connectionCount() >= config.maxClientCount) {
-                logger.error("[$aeronLogInfo] Connection from $clientAddressString not allowed! Server is full. Max allowed is ${config.maxClientCount}")
+            if (server.connections.size() >= config.maxClientCount) {
+                listenerManager.notifyError(ServerHandshakeException("[$logInfo] Connection not allowed! Server is full. Max allowed is ${config.maxClientCount}"))
 
                 try {
-                    server.writeHandshakeMessage(handshakePublication, aeronLogInfo,
-                                                 HandshakeMessage.error("Server is full"))
+                    handshaker.writeMessage(handshakePublication, logInfo,
+                                            HandshakeMessage.error("Server is full"))
                 } catch (e: Exception) {
-                    logger.error(e) { "[$aeronLogInfo] Handshake error!" }
+                    listenerManager.notifyError(TransmitException("[$logInfo] Handshake error", e))
                 }
                 return false
             }
@@ -186,29 +199,29 @@ internal class ServerHandshake<CONNECTION : Connection>(
 
             // VALIDATE:: we are now connected to the client and are going to create a new connection.
             val currentCountForIp = connectionsPerIpCounts.get(clientAddress)
-            if (currentCountForIp >= config.maxConnectionsPerIpAddress) {
+            if (config.maxConnectionsPerIpAddress in 1..currentCountForIp) {
                 // decrement it now, since we aren't going to permit this connection (take the extra decrement hit on failure, instead of always)
                 connectionsPerIpCounts.decrement(clientAddress, currentCountForIp)
 
-                logger.error { "[$aeronLogInfo] Too many connections for IP address $clientAddressString. Max allowed is ${config.maxConnectionsPerIpAddress}" }
+                listenerManager.notifyError(ServerHandshakeException("[$logInfo] Too many connections for IP address. Max allowed is ${config.maxConnectionsPerIpAddress}"))
 
                 try {
-                    server.writeHandshakeMessage(handshakePublication, aeronLogInfo,
-                                                 HandshakeMessage.error("Too many connections for IP address"))
+                    handshaker.writeMessage(handshakePublication, logInfo,
+                                            HandshakeMessage.error("Too many connections for IP address"))
                 } catch (e: Exception) {
-                    logger.error(e) { "[$aeronLogInfo] Handshake error!" }
+                    listenerManager.notifyError(TransmitException("[$logInfo] Handshake error", e))
                 }
                 return false
             }
             connectionsPerIpCounts.increment(clientAddress, currentCountForIp)
         } catch (e: Exception) {
-            logger.error(e) { "[$aeronLogInfo] Could not validate client message" }
+            listenerManager.notifyError(ServerHandshakeException("[$logInfo] Handshake error, Could not validate client message", e))
 
             try {
-                server.writeHandshakeMessage(handshakePublication, aeronLogInfo,
-                                             HandshakeMessage.error("Invalid connection"))
+                handshaker.writeMessage(handshakePublication, logInfo,
+                                        HandshakeMessage.error("Invalid connection"))
             } catch (e: Exception) {
-                logger.error(e) { "[$aeronLogInfo] Handshake error!" }
+                listenerManager.notifyError(TransmitException("[$logInfo] Handshake error", e))
             }
         }
 
@@ -217,34 +230,28 @@ internal class ServerHandshake<CONNECTION : Connection>(
 
 
     /**
-     * @return true if the handshake poller is to close the publication, false will keep the publication (as we are DONE processing data)
+     * NOTE: This must not be called on the main thread because it is blocking!
+     *
+     * @return true if the connection was SUCCESS. False if the handshake poller should immediately close the publication
      */
     fun processIpcHandshakeMessageServer(
         server: Server<CONNECTION>,
-        handshakePublication: Publication,
-        message: HandshakeMessage,
+        handshaker: Handshaker<CONNECTION>,
         aeronDriver: AeronDriver,
-        aeronLogInfo: String,
-        connectionFunc: (connectionParameters: ConnectionParams<CONNECTION>) -> CONNECTION,
-        logger: KLogger
-    ) {
-
-        val connectionString = "IPC"
-
-        if (!validateMessageTypeAndDoPending(
-                server,
-                server.actionDispatch,
-                handshakePublication,
-                message,
-                connectionString,
-                aeronLogInfo,
-                logger
-            )) {
-            return
-        }
-
+        handshakePublication: Publication,
+        publicKey: ByteArray,
+        message: HandshakeMessage,
+        logInfo: String,
+        logger: Logger
+    ): Boolean {
         val serialization = config.serialization
 
+        val clientTagName = message.tag
+        if (clientTagName.length > 32) {
+            listenerManager.notifyError(ServerHandshakeException("[$logInfo] Connection not allowed! Invalid tag name."))
+            return false
+        }
+
         /////
         /////
         ///// DONE WITH VALIDATION
@@ -253,87 +260,134 @@ internal class ServerHandshake<CONNECTION : Connection>(
 
 
         // allocate session/stream id's
-        val connectionSessionId: Int
+        val connectionSessionIdPub: Int
         try {
-            connectionSessionId = sessionIdAllocator.allocate()
+            connectionSessionIdPub = sessionIdAllocator.allocate()
         } catch (e: AllocationException) {
-            logger.error { "[$aeronLogInfo] Connection from $connectionString not allowed! Unable to allocate a session ID for the client connection!" }
+            listenerManager.notifyError(ServerHandshakeException("[$logInfo] Connection not allowed! Unable to allocate a session pub ID for the client connection!", e))
 
             try {
-                server.writeHandshakeMessage(handshakePublication, aeronLogInfo,
-                                             HandshakeMessage.error("Connection error!"))
+                handshaker.writeMessage(handshakePublication, logInfo,
+                                        HandshakeMessage.error("Connection error!"))
             } catch (e: Exception) {
-                logger.error(e) { "[$aeronLogInfo] Handshake error!" }
+                listenerManager.notifyError(TransmitException("[$logInfo] Handshake error", e))
             }
-            return
+            return false
         }
 
-
-        val connectionStreamPubId: Int
+        val connectionSessionIdSub: Int
         try {
-            connectionStreamPubId = streamIdAllocator.allocate()
+            connectionSessionIdSub = sessionIdAllocator.allocate()
         } catch (e: AllocationException) {
             // have to unwind actions!
-            sessionIdAllocator.free(connectionSessionId)
+            sessionIdAllocator.free(connectionSessionIdPub)
 
-            logger.error { "[$aeronLogInfo] Connection from $connectionString not allowed! Unable to allocate a stream ID for the client connection!" }
+            listenerManager.notifyError(ServerHandshakeException("[$logInfo] Connection not allowed! Unable to allocate a session sub ID for the client connection!", e))
 
             try {
-                server.writeHandshakeMessage(handshakePublication, aeronLogInfo,
-                                             HandshakeMessage.error("Connection error!"))
+                handshaker.writeMessage(handshakePublication, logInfo,
+                                        HandshakeMessage.error("Connection error!"))
             } catch (e: Exception) {
-                logger.error(e) { "[$aeronLogInfo] Handshake error!" }
+                listenerManager.notifyError(TransmitException("[$logInfo] Handshake error", e))
             }
-            return
+            return false
         }
 
-        val connectionStreamSubId: Int
+
+        val connectionStreamIdPub: Int
         try {
-            connectionStreamSubId = streamIdAllocator.allocate()
+            connectionStreamIdPub = streamIdAllocator.allocate()
         } catch (e: AllocationException) {
             // have to unwind actions!
-            sessionIdAllocator.free(connectionSessionId)
-            sessionIdAllocator.free(connectionStreamPubId)
+            sessionIdAllocator.free(connectionSessionIdPub)
+            sessionIdAllocator.free(connectionSessionIdSub)
 
-            logger.error { "[$aeronLogInfo] Connection from $connectionString not allowed! Unable to allocate a stream ID for the client connection!" }
+            listenerManager.notifyError(ServerHandshakeException("[$logInfo] Connection not allowed! Unable to allocate a stream publication ID for the client connection!", e))
 
             try {
-                server.writeHandshakeMessage(handshakePublication, aeronLogInfo,
-                                             HandshakeMessage.error("Connection error!"))
+                handshaker.writeMessage(handshakePublication, logInfo,
+                                        HandshakeMessage.error("Connection error!"))
             } catch (e: Exception) {
-                logger.error(e) { "[$aeronLogInfo] Handshake error!" }
+                listenerManager.notifyError(TransmitException("[$logInfo] Handshake error", e))
             }
-            return
+            return false
         }
 
+        val connectionStreamIdSub: Int
+        try {
+            connectionStreamIdSub = streamIdAllocator.allocate()
+        } catch (e: AllocationException) {
+            // have to unwind actions!
+            sessionIdAllocator.free(connectionSessionIdPub)
+            sessionIdAllocator.free(connectionSessionIdSub)
+            streamIdAllocator.free(connectionStreamIdPub)
+
+            listenerManager.notifyError(ServerHandshakeException("[$logInfo] Connection not allowed! Unable to allocate a stream subscription ID for the client connection!", e))
+
+            try {
+                handshaker.writeMessage(handshakePublication, logInfo,
+                                        HandshakeMessage.error("Connection error!"))
+            } catch (e: Exception) {
+                listenerManager.notifyError(TransmitException("[$logInfo] Handshake error", e))
+            }
+            return false
+        }
+
+
 
         // create a new connection. The session ID is encrypted.
+        var newConnection: CONNECTION? = null
         try {
-            // Create a subscription at the given address and port, using the given stream ID.
-            val driver = ServerIpcDriver(streamId = connectionStreamSubId,
-                                         sessionId = connectionSessionId)
-            driver.build(aeronDriver, logger)
+            // Create a pub/sub at the given address and port, using the given stream ID.
+            // NOTE: This must not be called on the main thread because it is blocking!
+            val newConnectionDriver = ServerConnectionDriver.build(
+                aeronDriver = aeronDriver,
+                ipInfo = server.ipInfo,
+                isIpc = true,
+                tagName = clientTagName,
+                logInfo = EndPoint.IPC_NAME,
 
-            // create a new publication for the connection (since the handshake ALWAYS closes the current publication)
-            val publicationUri = MediaDriverConnection.uri("ipc", handshakePublication.sessionId())
-            val clientPublication = aeronDriver.addPublication(publicationUri, message.subscriptionPort)
-
-            val clientConnection = MediaDriverConnectInfo(
-                publication = clientPublication,
-                subscription = driver.subscription,
-                subscriptionPort = driver.streamId,
-                publicationPort = message.subscriptionPort,
-                streamId = 0, // this is because with IPC, we have stream sub/pub (which are replaced as port sub/pub)
-                sessionId = driver.sessionId,
-                isReliable = driver.isReliable,
                 remoteAddress = null,
-                remoteAddressString = "ipc"
+                remoteAddressString = "",
+                sessionIdPub = connectionSessionIdPub,
+                sessionIdSub = connectionSessionIdSub,
+                streamIdPub = connectionStreamIdPub,
+                streamIdSub = connectionStreamIdSub,
+                portPubMdc = 0,
+                portPub = 0,
+                portSub = 0,
+                reliable = true
             )
 
 
-            logger.info { "[$aeronLogInfo] Creating new IPC connection from $driver" }
+            val enableBufferedMessagesForConnection = listenerManager.notifyEnableBufferedMessages(null, clientTagName)
+            val connectionType = if (enableBufferedMessagesForConnection) {
+                "buffered connection"
+            } else {
+                "connection"
+            }
+
+            val connectionTypeCaps = connectionType.replaceFirstChar { if (it.isLowerCase()) it.titlecase(Locale.getDefault()) else it.toString() }
+
+
+            val logInfo = newConnectionDriver.pubSub.getLogInfo(logger.isDebugEnabled)
+            if (logger.isDebugEnabled) {
+                logger.debug("Creating new $connectionType to $logInfo")
+            } else {
+                logger.info("Creating new $connectionType to $logInfo")
+            }
+
+            newConnection = server.newConnection(ConnectionParams(
+                publicKey = publicKey,
+                endPoint = server,
+                connectionInfo = newConnectionDriver.pubSub,
+                publicKeyValidation = PublicKeyValidationState.VALID,
+                enableBufferedMessages = enableBufferedMessagesForConnection,
+                cryptoKey = CryptoManagement.NOCRYPT // we don't use encryption for IPC connections
+            ))
+
+            server.bufferedManager.onConnect(newConnection)
 
-            val connection = connectionFunc(ConnectionParams(server, clientConnection, PublicKeyValidationState.VALID))
 
             // VALIDATE:: are we allowed to connect to this server (now that we have the initial server information)
             // NOTE: all IPC client connections are, by default, always allowed to connect, because they are running on the same machine
@@ -344,82 +398,112 @@ internal class ServerHandshake<CONNECTION : Connection>(
             ///////////////
 
 
-
             // The one-time pad is used to encrypt the session ID, so that ONLY the correct client knows what it is!
             val successMessage = HandshakeMessage.helloAckIpcToClient(message.connectKey)
 
 
-            // if necessary, we also send the kryo RMI id's that are registered as RMI on this endpoint, but maybe not on the other endpoint
+            // Also send the RMI registration data to the client (so the client doesn't register anything)
 
-            // now create the encrypted payload, using ECDH
-            val cryptOutput = server.crypto.cryptOutput
-            cryptOutput.reset()
-            cryptOutput.writeInt(connectionSessionId)
-            cryptOutput.writeInt(connectionStreamSubId)
-
-            val regDetails = serialization.getKryoRegistrationDetails()
-            cryptOutput.writeInt(regDetails.size)
-            cryptOutput.writeBytes(regDetails)
-
-            successMessage.registrationData = cryptOutput.toBytes()
+            // now create the encrypted payload, using no crypto
+            successMessage.registrationData = server.crypto.nocrypt(
+                sessionIdPub = connectionSessionIdPub,
+                sessionIdSub = connectionSessionIdSub,
+                streamIdPub = connectionStreamIdPub,
+                streamIdSub = connectionStreamIdSub,
+                sessionTimeout = config.bufferedConnectionTimeoutSeconds,
+                bufferedMessages = enableBufferedMessagesForConnection,
+                kryoRegDetails = serialization.getKryoRegistrationDetails()
+            )
 
             successMessage.publicKey = server.crypto.publicKeyBytes
 
             // before we notify connect, we have to wait for the client to tell us that they can receive data
-            pendingConnections[message.connectKey] = connection
+            pendingConnections[message.connectKey] = newConnection
+
+            if (logger.isTraceEnabled) {
+                logger.trace("[$logInfo] (${message.connectKey}) $connectionType (${newConnection.id}) responding to handshake hello.")
+            } else if (logger.isDebugEnabled) {
+                logger.debug("[$logInfo] $connectionTypeCaps (${newConnection.id}) responding to handshake hello.")
+            }
 
             // this tells the client all the info to connect.
-            server.writeHandshakeMessage(handshakePublication, aeronLogInfo, successMessage) // exception is already caught!
+            handshaker.writeMessage(handshakePublication, logInfo, successMessage) // exception is already caught!
         } catch (e: Exception) {
             // have to unwind actions!
-            sessionIdAllocator.free(connectionSessionId)
-            streamIdAllocator.free(connectionStreamPubId)
+            sessionIdAllocator.free(connectionSessionIdPub)
+            sessionIdAllocator.free(connectionSessionIdSub)
+            streamIdAllocator.free(connectionStreamIdSub)
+            streamIdAllocator.free(connectionStreamIdPub)
 
-            logger.error(e) { "[$aeronLogInfo] Connection handshake from $connectionString crashed! Message $message" }
+            listenerManager.notifyError(ServerHandshakeException("[$logInfo] (${message.connectKey}) Connection (${newConnection?.id}) handshake crashed! Message $message", e))
+
+            return false
         }
+
+        return true
     }
 
     /**
-     * note: CANNOT be called in action dispatch. ALWAYS ON SAME THREAD
-     * @return true if the handshake poller is to close the publication, false will keep the publication
+     * NOTE: This must not be called on the main thread because it is blocking!
+     *
+     * @return true if the connection was SUCCESS. False if the handshake poller should immediately close the publication
      */
     fun processUdpHandshakeMessageServer(
         server: Server<CONNECTION>,
+        handshaker: Handshaker<CONNECTION>,
         handshakePublication: Publication,
+        publicKey: ByteArray,
         clientAddress: InetAddress,
         clientAddressString: String,
+        portSub: Int,
+        portPub: Int,
+        mdcPortPub: Int,
         isReliable: Boolean,
         message: HandshakeMessage,
-        aeronDriver: AeronDriver,
-        aeronLogInfo: String,
-        isIpv6Wildcard: Boolean,
-        connectionFunc: (connectionParameters: ConnectionParams<CONNECTION>) -> CONNECTION,
-        logger: KLogger
-    ) {
-        // Manage the Handshake state
-        if (!validateMessageTypeAndDoPending(
-                server, server.actionDispatch, handshakePublication, message,
-                clientAddressString, aeronLogInfo, logger))
-        {
-            return
-        }
+        logInfo: String,
+        logger: Logger
+    ): Boolean {
+        val serialization = config.serialization
 
+        // UDP ONLY
         val clientPublicKeyBytes = message.publicKey
         val validateRemoteAddress: PublicKeyValidationState
-        val serialization = config.serialization
 
         // VALIDATE:: check to see if the remote connection's public key has changed!
         validateRemoteAddress = server.crypto.validateRemoteAddress(clientAddress, clientAddressString, clientPublicKeyBytes)
         if (validateRemoteAddress == PublicKeyValidationState.INVALID) {
-            logger.error { "[$aeronLogInfo] Connection from $clientAddressString not allowed! Public key mismatch." }
-            return
+            listenerManager.notifyError(ServerHandshakeException("[$logInfo] Connection not allowed! Public key mismatch."))
+            return false
         }
 
+        clientPublicKeyBytes!!
 
-        if (!clientAddress.isLoopbackAddress &&
-            !validateUdpConnectionInfo(server, handshakePublication, config, clientAddressString, clientAddress, aeronLogInfo, logger)) {
+        val isSelfMachine = clientAddress.isLoopbackAddress || clientAddress == EndPoint.lanAddress
+
+        if (!isSelfMachine &&
+            !validateUdpConnectionInfo(server, handshaker, handshakePublication, config, clientAddress, logInfo)) {
             // we do not want to limit the loopback addresses!
-            return
+            return false
+        }
+
+        val clientTagName = message.tag
+        if (clientTagName.length > 32) {
+            listenerManager.notifyError(ServerHandshakeException("[$logInfo] Connection not allowed! Invalid tag name."))
+            return false
+        }
+
+        // VALIDATE:: are we allowed to connect to this server (now that we have the initial server information)
+        val permitConnection = listenerManager.notifyFilter(clientAddress, clientTagName)
+        if (!permitConnection) {
+            listenerManager.notifyError(ServerHandshakeException("[$logInfo] Connection was not permitted!"))
+
+            try {
+                handshaker.writeMessage(handshakePublication, logInfo,
+                                        HandshakeMessage.error("Connection was not permitted!"))
+            } catch (e: Exception) {
+                listenerManager.notifyError(TransmitException("[$logInfo] Handshake error", e))
+            }
+            return false
         }
 
 
@@ -431,121 +515,152 @@ internal class ServerHandshake<CONNECTION : Connection>(
 
 
         // allocate session/stream id's
-        val connectionSessionId: Int
+        val connectionSessionIdPub: Int
         try {
-            connectionSessionId = sessionIdAllocator.allocate()
+            connectionSessionIdPub = sessionIdAllocator.allocate()
         } catch (e: AllocationException) {
             // have to unwind actions!
             connectionsPerIpCounts.decrementSlow(clientAddress)
 
-            logger.error { "[$aeronLogInfo] Connection from $clientAddressString not allowed! Unable to allocate a session ID for the client connection!" }
+            listenerManager.notifyError(ServerHandshakeException("[$logInfo] Connection not allowed! Unable to allocate a session ID for the client connection!"))
 
             try {
-                server.writeHandshakeMessage(handshakePublication, aeronLogInfo,
-                                             HandshakeMessage.error("Connection error!"))
+                handshaker.writeMessage(handshakePublication, logInfo,
+                                        HandshakeMessage.error("Connection error!"))
             } catch (e: Exception) {
-                logger.error(e) { "[$aeronLogInfo] Handshake error!" }
+                listenerManager.notifyError(TransmitException("[$logInfo] Handshake error", e))
             }
-            return
+            return false
         }
 
 
-        val connectionStreamId: Int
+        val connectionSessionIdSub: Int
         try {
-            connectionStreamId = streamIdAllocator.allocate()
+            connectionSessionIdSub = sessionIdAllocator.allocate()
         } catch (e: AllocationException) {
             // have to unwind actions!
             connectionsPerIpCounts.decrementSlow(clientAddress)
-            sessionIdAllocator.free(connectionSessionId)
+            sessionIdAllocator.free(connectionSessionIdPub)
 
-            logger.error { "[$aeronLogInfo] Connection from $clientAddressString not allowed! Unable to allocate a stream ID for the client connection!" }
+            listenerManager.notifyError(ServerHandshakeException("[$logInfo] Connection not allowed! Unable to allocate a session ID for the client connection!"))
 
             try {
-                server.writeHandshakeMessage(handshakePublication, aeronLogInfo,
-                                             HandshakeMessage.error("Connection error!"))
+                handshaker.writeMessage(handshakePublication, logInfo,
+                                        HandshakeMessage.error("Connection error!"))
             } catch (e: Exception) {
-                logger.error(e) { "[$aeronLogInfo] Handshake error!" }
+                listenerManager.notifyError(TransmitException("[$logInfo] Handshake error", e))
             }
-            return
+            return false
         }
 
-        // the pub/sub do not necessarily have to be the same. They can be ANY port
-        val publicationPort = message.subscriptionPort
-        val subscriptionPort = config.port
 
+        val connectionStreamIdPub: Int
+        try {
+            connectionStreamIdPub = streamIdAllocator.allocate()
+        } catch (e: AllocationException) {
+            // have to unwind actions!
+            connectionsPerIpCounts.decrementSlow(clientAddress)
+            sessionIdAllocator.free(connectionSessionIdPub)
+            sessionIdAllocator.free(connectionSessionIdSub)
+
+            listenerManager.notifyError(ServerHandshakeException("[$logInfo] Connection not allowed! Unable to allocate a stream ID for the client connection!"))
+
+            try {
+                handshaker.writeMessage(handshakePublication, logInfo,
+                                        HandshakeMessage.error("Connection error!"))
+            } catch (e: Exception) {
+                listenerManager.notifyError(TransmitException("[$logInfo] Handshake error", e))
+            }
+            return false
+        }
+
+        val connectionStreamIdSub: Int
+        try {
+            connectionStreamIdSub = streamIdAllocator.allocate()
+        } catch (e: AllocationException) {
+            // have to unwind actions!
+            connectionsPerIpCounts.decrementSlow(clientAddress)
+            sessionIdAllocator.free(connectionSessionIdPub)
+            sessionIdAllocator.free(connectionSessionIdSub)
+            streamIdAllocator.free(connectionStreamIdPub)
+
+            listenerManager.notifyError(ServerHandshakeException("[$logInfo] Connection not allowed! Unable to allocate a stream ID for the client connection!"))
+
+            try {
+                handshaker.writeMessage(handshakePublication, logInfo,
+                                        HandshakeMessage.error("Connection error!"))
+            } catch (e: Exception) {
+                listenerManager.notifyError(TransmitException("[$logInfo] Handshake error", e))
+            }
+            return false
+        }
+
+
+        val logType = if (clientAddress is Inet4Address) {
+            "IPv4"
+        } else {
+            "IPv6"
+        }
 
         // create a new connection. The session ID is encrypted.
-        var connection: CONNECTION? = null
+        var newConnection: CONNECTION? = null
         try {
-            // connection timeout of 0 doesn't matter. it is not used by the server
-            // the client address WILL BE either IPv4 or IPv6
-            val listenAddress = if (clientAddress is Inet4Address && !isIpv6Wildcard) {
-                server.listenIPv4Address!!
-            } else {
-                // wildcard is SPECIAL, in that if we bind wildcard, it will ALSO bind to IPv4, so we can't bind both!
-                server.listenIPv6Address!!
-            }
+            // Create a pub/sub at the given address and port, using the given stream ID.
+            // NOTE: This must not be called on the main thread because it is blocking!
+            val newConnectionDriver = ServerConnectionDriver.build(
+                ipInfo = server.ipInfo,
+                aeronDriver = aeronDriver,
+                isIpc = false,
+                logInfo = logType,
 
-            // create a new publication for the connection (since the handshake ALWAYS closes the current publication)
-            val publicationUri = MediaDriverConnection.uriEndpoint("udp", message.sessionId, isReliable, clientAddress, clientAddressString, message.subscriptionPort)
-            val clientPublication = aeronDriver.addPublication(publicationUri, message.streamId)
-
-            val driver = UdpMediaDriverPairedConnection(
-                listenAddress,
-                clientAddress,
-                clientAddressString,
-                publicationPort,
-                subscriptionPort,
-                connectionStreamId,
-                connectionSessionId,
-                0,
-                isReliable,
-                clientPublication
-            )
-
-            driver.build(aeronDriver, logger)
-            logger.info { "[$aeronLogInfo] Creating new connection from $driver" }
-
-            val clientConnection = MediaDriverConnectInfo(
-                publication = driver.publication,
-                subscription = driver.subscription,
-                subscriptionPort = driver.port,
-                publicationPort = publicationPort,
-                streamId = driver.streamId,
-                sessionId = driver.sessionId,
-                isReliable = driver.isReliable,
                 remoteAddress = clientAddress,
-                remoteAddressString = clientAddressString
+                remoteAddressString = clientAddressString,
+                sessionIdPub = connectionSessionIdPub,
+                sessionIdSub = connectionSessionIdSub,
+                streamIdPub = connectionStreamIdPub,
+                streamIdSub = connectionStreamIdSub,
+                portPubMdc = mdcPortPub,
+                portPub = portPub,
+                portSub = portSub,
+                tagName = clientTagName,
+                reliable = isReliable
             )
 
-            connection = connectionFunc(ConnectionParams(server, clientConnection, validateRemoteAddress))
+            val cryptoSecretKey = server.crypto.generateAesKey(clientPublicKeyBytes, clientPublicKeyBytes, server.crypto.publicKeyBytes)
 
-            // VALIDATE:: are we allowed to connect to this server (now that we have the initial server information)
-            val permitConnection = listenerManager.notifyFilter(connection)
-            if (!permitConnection) {
-                // have to unwind actions!
-                connectionsPerIpCounts.decrementSlow(clientAddress)
-                sessionIdAllocator.free(connectionSessionId)
-                streamIdAllocator.free(connectionStreamId)
-                connection.close()
 
-                logger.error { "[$aeronLogInfo] Connection $clientAddressString was not permitted!" }
-
-                try {
-                    server.writeHandshakeMessage(handshakePublication, aeronLogInfo,
-                                                 HandshakeMessage.error("Connection was not permitted!"))
-                } catch (e: Exception) {
-                    logger.error(e) { "[$aeronLogInfo] Handshake error!" }
-                }
-                return
+            val enableBufferedMessagesForConnection = listenerManager.notifyEnableBufferedMessages(clientAddress, clientTagName)
+            val connectionType = if (enableBufferedMessagesForConnection) {
+                "buffered connection"
+            } else {
+                "connection"
             }
 
+            val connectionTypeCaps = connectionType.replaceFirstChar { if (it.isLowerCase()) it.titlecase(Locale.getDefault()) else it.toString() }
+
+            val logInfo = newConnectionDriver.pubSub.getLogInfo(logger.isDebugEnabled)
+            if (logger.isDebugEnabled) {
+                logger.debug("Creating new $connectionType to $logInfo")
+            } else {
+                logger.info("Creating new $connectionType to $logInfo")
+            }
+
+            newConnection = server.newConnection(ConnectionParams(
+                publicKey = publicKey,
+                endPoint = server,
+                connectionInfo = newConnectionDriver.pubSub,
+                publicKeyValidation = validateRemoteAddress,
+                enableBufferedMessages = enableBufferedMessagesForConnection,
+                cryptoKey = cryptoSecretKey
+            ))
+
+            server.bufferedManager.onConnect(newConnection)
+
 
             ///////////////
             ///  HANDSHAKE
             ///////////////
 
-
             // The one-time pad is used to encrypt the session ID, so that ONLY the correct client knows what it is!
             val successMessage = HandshakeMessage.helloAckToClient(message.connectKey)
 
@@ -553,29 +668,43 @@ internal class ServerHandshake<CONNECTION : Connection>(
             // Also send the RMI registration data to the client (so the client doesn't register anything)
 
             // now create the encrypted payload, using ECDH
-            successMessage.registrationData = server.crypto.encrypt(clientPublicKeyBytes!!,
-                                                                    subscriptionPort,
-                                                                    connectionSessionId,
-                                                                    connectionStreamId,
-                                                                    serialization.getKryoRegistrationDetails())
+            successMessage.registrationData = server.crypto.encrypt(
+                cryptoSecretKey = cryptoSecretKey,
+                sessionIdPub = connectionSessionIdPub,
+                sessionIdSub = connectionSessionIdSub,
+                streamIdPub = connectionStreamIdPub,
+                streamIdSub = connectionStreamIdSub,
+                sessionTimeout = config.bufferedConnectionTimeoutSeconds,
+                bufferedMessages =  enableBufferedMessagesForConnection,
+                kryoRegDetails = serialization.getKryoRegistrationDetails()
+            )
 
             successMessage.publicKey = server.crypto.publicKeyBytes
 
             // before we notify connect, we have to wait for the client to tell us that they can receive data
-            pendingConnections[message.connectKey] = connection
+            pendingConnections[message.connectKey] = newConnection
 
-            logger.debug { "[$aeronLogInfo - ${message.connectKey}] Connection (${connection.streamId}/${connection.id}) responding to handshake hello." }
+            if (logger.isTraceEnabled) {
+                logger.trace("[$logInfo] $connectionTypeCaps (${newConnection.id}) responding to handshake hello.")
+            } else if (logger.isDebugEnabled) {
+                logger.debug("[$logInfo] $connectionTypeCaps (${newConnection.id}) responding to handshake hello.")
+            }
 
             // this tells the client all the info to connect.
-            server.writeHandshakeMessage(handshakePublication, aeronLogInfo, successMessage) // exception is already caught
+            handshaker.writeMessage(handshakePublication, logInfo, successMessage) // exception is already caught
         } catch (e: Exception) {
             // have to unwind actions!
             connectionsPerIpCounts.decrementSlow(clientAddress)
-            sessionIdAllocator.free(connectionSessionId)
-            streamIdAllocator.free(connectionStreamId)
+            sessionIdAllocator.free(connectionSessionIdPub)
+            sessionIdAllocator.free(connectionSessionIdSub)
+            streamIdAllocator.free(connectionStreamIdPub)
+            streamIdAllocator.free(connectionStreamIdSub)
 
-            logger.error(e) { "[$aeronLogInfo - ${message.connectKey}] Connection (${connection?.streamId}/${connection?.id}) handshake from $clientAddressString crashed! Message $message" }
+            listenerManager.notifyError(ServerHandshakeException("[$logInfo] (${message.connectKey}) Connection (${newConnection?.id}) handshake crashed! Message $message", e))
+            return false
         }
+
+        return true
     }
 
     /**
@@ -584,14 +713,11 @@ internal class ServerHandshake<CONNECTION : Connection>(
      * note: CANNOT be called in action dispatch. ALWAYS ON SAME THREAD
      */
     fun checkForMemoryLeaks() {
-        val noAllocations = connectionsPerIpCounts.isEmpty() && sessionIdAllocator.isEmpty() && streamIdAllocator.isEmpty()
+        val noAllocations = connectionsPerIpCounts.isEmpty()
 
         if (!noAllocations) {
-            throw AllocationException("Unequal allocate/free method calls for validation. \n" +
-                                      "connectionsPerIpCounts: '$connectionsPerIpCounts' \n" +
-                                      "sessionIdAllocator: $sessionIdAllocator \n" +
-                                      "streamIdAllocator: $streamIdAllocator")
-
+            throw AllocationException("Unequal allocate/free method calls for IP validation. \n" +
+                                      "connectionsPerIpCounts: '$connectionsPerIpCounts'")
         }
     }
 
@@ -601,12 +727,17 @@ internal class ServerHandshake<CONNECTION : Connection>(
      * note: CANNOT be called in action dispatch. ALWAYS ON SAME THREAD
      */
     fun clear() {
-        runBlocking {
-            pendingConnections.forEach { (_, v) ->
-                v.close()
-            }
+        val connections = pendingConnections
+        val latch = CountDownLatch(connections.size)
 
-            pendingConnections.clear()
+        eventDispatch.CLOSE.launch {
+            connections.forEach { (_, v) ->
+                v.close()
+                latch.countDown()
+            }
         }
+
+        latch.await(config.connectionCloseTimeoutInSeconds.toLong() * connections.size, TimeUnit.MILLISECONDS)
+        connections.clear()
     }
 }

src/dorkbox/network/handshake/ServerHandshakeDriver.kt

@@ -0,0 +1,84 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package dorkbox.network.handshake
+
+import dorkbox.network.aeron.AeronDriver
+import dorkbox.network.aeron.AeronDriver.Companion.uriHandshake
+import dorkbox.network.connection.EndPoint
+import dorkbox.network.connection.IpInfo
+import io.aeron.ChannelUriStringBuilder
+import io.aeron.CommonContext
+import io.aeron.Subscription
+
+/**
+ * For a client, the ports specified here MUST be manually flipped because they are in the perspective of the SERVER.
+ * A connection timeout of 0, means to wait forever
+ */
+internal class ServerHandshakeDriver(
+    private val aeronDriver: AeronDriver,
+    val subscription: Subscription,
+    val info: String,
+    private val logInfo: String)
+{
+    companion object {
+        fun build(
+            aeronDriver: AeronDriver,
+            isIpc: Boolean,
+            ipInfo: IpInfo,
+            port: Int,
+            streamIdSub: Int, sessionIdSub: Int,
+            logInfo: String
+        ): ServerHandshakeDriver {
+
+            val info: String
+            val subscriptionUri: ChannelUriStringBuilder
+
+            if (isIpc) {
+                subscriptionUri = uriHandshake(CommonContext.IPC_MEDIA, true)
+                info = "$logInfo [$sessionIdSub|$streamIdSub]"
+            } else {
+                // are we ipv4 or ipv6 or ipv6wildcard?
+                subscriptionUri = uriHandshake(CommonContext.UDP_MEDIA, ipInfo.isReliable)
+                    .endpoint(ipInfo.getAeronPubAddress(ipInfo.isIpv4) + ":" + port)
+
+                info = "$logInfo ${ipInfo.listenAddressStringPretty}:$port [$sessionIdSub|$streamIdSub] (reliable:${ipInfo.isReliable})"
+            }
+
+            val subscription = aeronDriver.addSubscription(subscriptionUri, streamIdSub, logInfo, isIpc)
+            return ServerHandshakeDriver(aeronDriver, subscription, info, logInfo)
+        }
+    }
+
+    fun close(endPoint: EndPoint<*>) {
+        try {
+            // we might not be able to close this connection.
+            aeronDriver.close(subscription, logInfo)
+        }
+        catch (e: Exception) {
+            endPoint.listenerManager.notifyError(e)
+        }
+    }
+
+    fun unsafeClose() {
+        // we might not be able to close this connection.
+        aeronDriver.close(subscription, logInfo)
+    }
+
+    override fun toString(): String {
+        return info
+    }
+}

src/dorkbox/network/handshake/ServerHandshakePollers.kt

@@ -1,22 +1,49 @@
+/*
+ * Copyright 2024 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
 @file:Suppress("MemberVisibilityCanBePrivate", "DuplicatedCode")
 
 package dorkbox.network.handshake
 
 import dorkbox.netUtil.IP
+import dorkbox.network.Configuration
 import dorkbox.network.Server
 import dorkbox.network.ServerConfiguration
 import dorkbox.network.aeron.AeronDriver
+import dorkbox.network.aeron.AeronDriver.Companion.uriHandshake
 import dorkbox.network.aeron.AeronPoller
-import dorkbox.network.aeron.mediaDriver.MediaDriverConnection
-import dorkbox.network.aeron.mediaDriver.ServerIpcDriver
-import dorkbox.network.aeron.mediaDriver.ServerUdpDriver
 import dorkbox.network.connection.Connection
-import dorkbox.network.connection.ConnectionParams
+import dorkbox.network.connection.IpInfo
+import dorkbox.network.exceptions.ServerException
+import dorkbox.network.exceptions.ServerHandshakeException
+import dorkbox.network.exceptions.ServerTimedoutException
+import dorkbox.util.NamedThreadFactory
+import dorkbox.util.Sys
+import io.aeron.CommonContext
 import io.aeron.FragmentAssembler
 import io.aeron.Image
+import io.aeron.Publication
+import io.aeron.logbuffer.FragmentHandler
 import io.aeron.logbuffer.Header
-import mu.KLogger
+import net.jodah.expiringmap.ExpirationPolicy
+import net.jodah.expiringmap.ExpiringMap
 import org.agrona.DirectBuffer
+import org.slf4j.Logger
+import java.net.Inet4Address
+import java.util.concurrent.*
 
 internal object ServerHandshakePollers {
     fun disabled(serverInfo: String): AeronPoller {
@@ -27,286 +54,699 @@ internal object ServerHandshakePollers {
         }
     }
 
-    private fun <CONNECTION : Connection> ipcProcessing(
-        logger: KLogger,
-        server: Server<CONNECTION>, aeronDriver: AeronDriver,
-        header: Header, buffer: DirectBuffer, offset: Int, length: Int,
-        handshake: ServerHandshake<CONNECTION>, connectionFunc: (connectionParameters: ConnectionParams<CONNECTION>) -> CONNECTION
-    ) {
-        // this is processed on the thread that calls "poll". Subscriptions are NOT multi-thread safe!
+    class IpcProc<CONNECTION : Connection>(
+        val logger: Logger,
+        val server: Server<CONNECTION>,
+        val driver: AeronDriver,
+        val handshake: ServerHandshake<CONNECTION>
+    ): FragmentHandler {
 
-        // The sessionId is unique within a Subscription and unique across all Publication's from a sourceIdentity.
-        // for the handshake, the sessionId IS NOT GLOBALLY UNIQUE
-        val sessionId = header.sessionId()
-        val streamId = header.streamId()
-        val aeronLogInfo = "$sessionId/$streamId"
+        private val isReliable = server.config.isReliable
+        private val handshaker = server.handshaker
+        private val handshakeTimeoutNs = handshake.handshakeTimeoutNs
+        private val shutdownInProgress = server.shutdownInProgress
+        private val shutdown = server.shutdown
 
-        val message = server.readHandshakeMessage(buffer, offset, length, header, aeronLogInfo)
+        // note: the expire time here is a LITTLE longer than the expire time in the client, this way we can adjust for network lag if it's close
+        private val publications = ExpiringMap.builder()
+            .apply {
+                this.expiration(handshakeTimeoutNs, TimeUnit.NANOSECONDS)
+            }
+            .expirationPolicy(ExpirationPolicy.CREATED)
+            .expirationListener<Long, Publication> { connectKey, publication ->
+                try {
+                    // we might not be able to close this connection.
+                    driver.close(publication, "Server IPC Handshake ($connectKey)")
+                }
+                catch (e: Exception) {
+                    server.listenerManager.notifyError(e)
+                }
 
-        // VALIDATE:: a Registration object is the only acceptable message during the connection phase
-        if (message !is HandshakeMessage) {
-            logger.error { "[$aeronLogInfo] Connection from IPC not allowed! Invalid connection request" }
-        } else {
-            // we create a NEW publication for the handshake, which connects directly to the client handshake subscription
-            val publicationUri = MediaDriverConnection.uri("ipc", message.sessionId)
-            val publication = aeronDriver.addPublication(publicationUri, message.subscriptionPort)
+            }
+            .build<Long, Publication>()
 
-            handshake.processIpcHandshakeMessageServer(
-                server, publication, message,
-                aeronDriver, aeronLogInfo,
-                connectionFunc, logger
-            )
+        override fun onFragment(buffer: DirectBuffer, offset: Int, length: Int, header: Header) {
+            // this is processed on the thread that calls "poll". Subscriptions are NOT multi-thread safe!
 
-            publication.close()
-        }
-    }
+            // The sessionId is unique within a Subscription and unique across all Publication's from a sourceIdentity.
+            // for the handshake, the sessionId IS NOT GLOBALLY UNIQUE
+            val sessionId = header.sessionId()
+            val streamId = header.streamId()
+            val image = header.context() as Image
 
-    private fun <CONNECTION : Connection> ipProcessing(
-        logger: KLogger,
-        server: Server<CONNECTION>, isReliable: Boolean, aeronDriver: AeronDriver, isIpv6Wildcard: Boolean,
-        header: Header, buffer: DirectBuffer, offset: Int, length: Int,
-        handshake: ServerHandshake<CONNECTION>, connectionFunc: (connectionParameters: ConnectionParams<CONNECTION>) -> CONNECTION
-    ) {
-        // this is processed on the thread that calls "poll". Subscriptions are NOT multi-thread safe!
+            val logInfo = "$sessionId/$streamId : IPC" // Server is the "source", client mirrors the server
 
-        // The sessionId is unique within a Subscription and unique across all Publication's from a sourceIdentity.
-        // for the handshake, the sessionId IS NOT GLOBALLY UNIQUE
-        val sessionId = header.sessionId()
-        val streamId = header.streamId()
-        val aeronLogInfo = "$sessionId/$streamId"
-
-        // note: this address will ALWAYS be an IP:PORT combo  OR  it will be aeron:ipc  (if IPC, it will be a different handler!)
-        val remoteIpAndPort = (header.context() as Image).sourceIdentity()
-
-        // split
-        val splitPoint = remoteIpAndPort.lastIndexOf(':')
-        val clientAddressString = remoteIpAndPort.substring(0, splitPoint)
-
-        val message = server.readHandshakeMessage(buffer, offset, length, header, aeronLogInfo)
-
-        // VALIDATE:: a Registration object is the only acceptable message during the connection phase
-        if (message !is HandshakeMessage) {
-            logger.error { "[$aeronLogInfo] Connection from $clientAddressString not allowed! Invalid connection request" }
-        } else {
-            // this should never be null, because we are feeding it a valid IP address from aeron
-            val clientAddress = IP.toAddress(clientAddressString)
-            if (clientAddress == null) {
-                logger.error { "[$aeronLogInfo] Connection from $clientAddressString not allowed! Invalid IP address!" }
+            if (shutdownInProgress.value) {
+                driver.deleteLogFile(image)
+                server.listenerManager.notifyError(ServerHandshakeException("[$logInfo] server is shutting down. Aborting new connection attempts."))
                 return
             }
 
-            // we create a NEW publication for the handshake, which connects directly to the client handshake subscription
-            val publicationUri = MediaDriverConnection.uriEndpoint("udp", message.sessionId, isReliable, clientAddress, clientAddressString, message.subscriptionPort)
-            val publication = aeronDriver.addPublication(publicationUri, message.streamId)
+            // ugh, this is verbose -- but necessary
+            val message = try {
+                val msg = handshaker.readMessage(buffer, offset, length)
 
-            handshake.processUdpHandshakeMessageServer(
-                server, publication, clientAddress, clientAddressString, isReliable, message,
-                aeronDriver, aeronLogInfo, isIpv6Wildcard,
-                connectionFunc, logger
-            )
+                // VALIDATE:: a Registration object is the only acceptable message during the connection phase
+                if (msg !is HandshakeMessage) {
+                    throw ServerHandshakeException("[$logInfo] Connection not allowed! unrecognized message: $msg")
+                } else if (logger.isTraceEnabled) {
+                    logger.trace("[$logInfo] (${msg.connectKey}) received HS: $msg")
+                }
+                msg
+            } catch (e: Exception) {
+                server.listenerManager.notifyError(ServerHandshakeException("[$logInfo] Error de-serializing handshake message!!", e))
+                null
+            }
 
-            publication.close()
+
+            if (message == null) {
+                // we should immediately remove the logbuffer for this! Aeron will **EVENTUALLY** remove the logbuffer, but if errors
+                // and connections occur too quickly (within the cleanup/linger period), we can run out of memory!
+                driver.deleteLogFile(image)
+                return
+            }
+
+
+            // NOTE: This MUST to happen in separates thread so that we can take as long as we need when creating publications and handshaking,
+            //  because under load -- this will REGULARLY timeout! Under no circumstance can this happen in the main processing thread!!
+            server.eventDispatch.HANDSHAKE.launch {
+                // we have read all the data, now dispatch it.
+                // HandshakeMessage.HELLO
+                // HandshakeMessage.DONE
+                val messageState = message.state
+                val connectKey = message.connectKey
+
+
+                if (messageState == HandshakeMessage.HELLO) {
+                    // we create a NEW publication for the handshake, which connects directly to the client handshake subscription
+
+                    val publicationUri = uriHandshake(CommonContext.IPC_MEDIA, isReliable)
+
+                    // this will always connect to the CLIENT handshake subscription!
+                    val publication = try {
+                        driver.addPublication(publicationUri, message.streamId, logInfo, true)
+                    }
+                    catch (e: Exception) {
+                        // we should immediately remove the logbuffer for this! Aeron will **EVENTUALLY** remove the logbuffer, but if errors
+                        // and connections occur too quickly (within the cleanup/linger period), we can run out of memory!
+                        driver.deleteLogFile(image)
+
+                        server.listenerManager.notifyError(ServerHandshakeException("[$logInfo] Cannot create IPC publication back to client remote process", e))
+                        return@launch
+                    }
+
+                    try {
+                        // we actually have to wait for it to connect before we continue
+                        driver.waitForConnection(shutdown, publication, handshakeTimeoutNs, logInfo) { cause ->
+                            ServerTimedoutException("$logInfo publication cannot connect with client in ${Sys.getTimePrettyFull(handshakeTimeoutNs)}", cause)
+                        }
+                    }
+                    catch (e: Exception) {
+                        // we should immediately remove the logbuffer for this! Aeron will **EVENTUALLY** remove the logbuffer, but if errors
+                        // and connections occur too quickly (within the cleanup/linger period), we can run out of memory!
+                        driver.deleteLogFile(image)
+
+                        server.listenerManager.notifyError(ServerHandshakeException("[$logInfo] Cannot create IPC publication back to client remote process", e))
+                        return@launch
+                    }
+
+
+                    try {
+                        val success = handshake.processIpcHandshakeMessageServer(
+                            server = server,
+                            handshaker = handshaker,
+                            aeronDriver = driver,
+                            handshakePublication = publication,
+                            publicKey = message.publicKey!!,
+                            message = message,
+                            logInfo = logInfo,
+                            logger = logger
+                        )
+
+                        if (success) {
+                            publications[connectKey] = publication
+                        }
+                        else {
+                            try {
+                                // we might not be able to close this connection.
+                                driver.close(publication, logInfo)
+                            }
+                            catch (e: Exception) {
+                                server.listenerManager.notifyError(e)
+                            }
+                        }
+                    }
+                    catch (e: Exception) {
+                        // we should immediately remove the logbuffer for this! Aeron will **EVENTUALLY** remove the logbuffer, but if errors
+                        // and connections occur too quickly (within the cleanup/linger period), we can run out of memory!
+                        driver.deleteLogFile(image)
+
+                        try {
+                            // we might not be able to close this connection.
+                            driver.close(publication, logInfo)
+                        }
+                        catch (e: Exception) {
+                            server.listenerManager.notifyError(e)
+                        }
+
+                        server.listenerManager.notifyError(ServerHandshakeException("[$logInfo] Error processing IPC handshake", e))
+                    }
+                } else {
+                    // HandshakeMessage.DONE
+
+                    val publication = publications.remove(connectKey)
+                    if (publication == null) {
+                        // we should immediately remove the logbuffer for this! Aeron will **EVENTUALLY** remove the logbuffer, but if errors
+                        // and connections occur too quickly (within the cleanup/linger period), we can run out of memory!
+                        driver.deleteLogFile(image)
+
+                        server.listenerManager.notifyError(ServerHandshakeException("[$logInfo] No publication back to IPC"))
+                        return@launch
+                    }
+
+                    try {
+                        handshake.validateMessageTypeAndDoPending(
+                            server = server,
+                            handshaker = handshaker,
+                            handshakePublication = publication,
+                            message = message,
+                            logInfo = logInfo,
+                            logger = logger
+                        )
+                    } catch (e: Exception) {
+                        server.listenerManager.notifyError(ServerHandshakeException("[$logInfo] Error processing IPC handshake", e))
+                    }
+
+                    // we should immediately remove the logbuffer for this! Aeron will **EVENTUALLY** remove the logbuffer, but if errors
+                    // and connections occur too quickly (within the cleanup/linger period), we can run out of memory!
+                    driver.deleteLogFile(image)
+
+                    try {
+                        // we might not be able to close this connection.
+                        driver.close(publication, logInfo)
+                    }
+                    catch (e: Exception) {
+                        server.listenerManager.notifyError(e)
+                    }
+                }
+            }
+        }
+
+        fun close() {
+            publications.forEach { (connectKey, publication) ->
+                AeronDriver.sessionIdAllocator.free(publication.sessionId())
+                try {
+                    // we might not be able to close this connection.
+                    driver.close(publication, "Server Handshake ($connectKey)")
+                }
+                catch (e: Exception) {
+                    server.listenerManager.notifyError(e)
+                }
+            }
+            publications.clear()
         }
     }
 
+    class UdpProc<CONNECTION : Connection>(
+        val logger: Logger,
+        val server: Server<CONNECTION>,
+        val driver: AeronDriver,
+        val handshake: ServerHandshake<CONNECTION>,
+        val isReliable: Boolean
+    ): FragmentHandler {
+        companion object {
+            init {
+                ExpiringMap.setThreadFactory(NamedThreadFactory("ExpiringMap-Eviction", Configuration.networkThreadGroup, true))
+            }
+        }
+
+        private val ipInfo = server.ipInfo
+        private val handshaker = server.handshaker
+        private val handshakeTimeoutNs = handshake.handshakeTimeoutNs
+        private val shutdownInProgress = server.shutdownInProgress
+        private val shutdown = server.shutdown
+
+        private val serverPortSub = server.port1
+        // MDC 'dynamic control mode' means that the server will to listen for status messages and NAK (from the client) on a port.
+        private val mdcPortPub = server.port2
+
+        // note: the expire time here is a LITTLE longer than the expire time in the client, this way we can adjust for network lag if it's close
+        private val publications = ExpiringMap.builder()
+            .apply {
+
+                this.expiration(handshakeTimeoutNs, TimeUnit.NANOSECONDS)
+            }
+            .expirationPolicy(ExpirationPolicy.CREATED)
+            .expirationListener<Long, Publication> { connectKey, publication ->
+                try {
+                    // we might not be able to close this connection.
+                    driver.close(publication, "Server UDP Handshake ($connectKey)")
+                }
+                catch (e: Exception) {
+                    server.listenerManager.notifyError(e)
+                }
+            }
+            .build<Long, Publication>()
 
 
+        override fun onFragment(buffer: DirectBuffer, offset: Int, length: Int, header: Header) {
+            // The sessionId is unique within a Subscription and unique across all Publication's from a sourceIdentity.
+            // for the handshake, the sessionId IS NOT GLOBALLY UNIQUE
 
-    fun <CONNECTION : Connection> ipc(
-        aeronDriver: AeronDriver, config: ServerConfiguration, server: Server<CONNECTION>, handshake: ServerHandshake<CONNECTION>
-    ): AeronPoller
-    {
+
+            // this is processed on the thread that calls "poll". Subscriptions are NOT multi-thread safe!
+
+            // The sessionId is unique within a Subscription and unique across all Publication's from a sourceIdentity.
+            // for the handshake, the sessionId IS NOT GLOBALLY UNIQUE
+            val sessionId = header.sessionId()
+            val streamId = header.streamId()
+            val image = header.context() as Image
+
+            // note: this address will ALWAYS be an IP:PORT combo  OR  it will be aeron:ipc  (if IPC, it will be a different handler!)
+            val remoteIpAndPort = image.sourceIdentity()
+
+            // split
+            val splitPoint = remoteIpAndPort.lastIndexOf(':')
+            var clientAddressString = remoteIpAndPort.substring(0, splitPoint)
+
+            // this should never be null, because we are feeding it a valid IP address from aeron
+            val clientAddress = IP.toAddress(clientAddressString)
+            if (clientAddress == null) {
+                // we should immediately remove the logbuffer for this! Aeron will **EVENTUALLY** remove the logbuffer, but if errors
+                // and connections occur too quickly (within the cleanup/linger period), we can run out of memory!
+                driver.deleteLogFile(image)
+
+
+                // Server is the "source", client mirrors the server
+                server.listenerManager.notifyError(ServerHandshakeException("[$sessionId/$streamId] Connection from $clientAddressString not allowed! Invalid IP address!"))
+                return
+            }
+
+            val isRemoteIpv4 = clientAddress is Inet4Address
+            if (!isRemoteIpv4) {
+                // this is necessary to clean up the address when adding it to aeron, since different formats mess it up
+                clientAddressString = IP.toString(clientAddress)
+
+                if (ipInfo.ipType == IpInfo.Companion.IpListenType.IPv4Wildcard) {
+                    // we should immediately remove the logbuffer for this! Aeron will **EVENTUALLY** remove the logbuffer, but if errors
+                    // and connections occur too quickly (within the cleanup/linger period), we can run out of memory!
+                    driver.deleteLogFile(image)
+
+                    // we DO NOT want to listen to IPv4 traffic, but we received IPv4 traffic!
+                    server.listenerManager.notifyError(ServerHandshakeException("[$sessionId/$streamId] Connection from $clientAddressString not allowed! IPv4 connections not permitted!"))
+                    return
+                }
+            }
+
+            val logInfo = "$sessionId/$streamId:$clientAddressString"
+
+
+            if (shutdownInProgress.value) {
+                driver.deleteLogFile(image)
+                server.listenerManager.notifyError(ServerHandshakeException("[$logInfo] server is shutting down. Aborting new connection attempts."))
+                return
+            }
+
+            // ugh, this is verbose -- but necessary
+            val message = try {
+                val msg = handshaker.readMessage(buffer, offset, length)
+
+                // VALIDATE:: a Registration object is the only acceptable message during the connection phase
+                if (msg !is HandshakeMessage) {
+                    throw ServerHandshakeException("[$logInfo] Connection not allowed! unrecognized message: $msg")
+                } else if (logger.isTraceEnabled) {
+                    logger.trace("[$logInfo] (${msg.connectKey}) received HS: $msg")
+                }
+                msg
+            } catch (e: Exception) {
+                server.listenerManager.notifyError(ServerHandshakeException("[$logInfo] Error de-serializing handshake message!!", e))
+                null
+            }
+
+            if (message == null) {
+                // we should immediately remove the logbuffer for this! Aeron will **EVENTUALLY** remove the logbuffer, but if errors
+                // and connections occur too quickly (within the cleanup/linger period), we can run out of memory!
+                driver.deleteLogFile(image)
+                return
+            }
+
+            // NOTE: This MUST to happen in separates thread so that we can take as long as we need when creating publications and handshaking,
+            //  because under load -- this will REGULARLY timeout! Under no circumstance can this happen in the main processing thread!!
+            server.eventDispatch.HANDSHAKE.launch {
+                // HandshakeMessage.HELLO
+                // HandshakeMessage.DONE
+                val messageState = message.state
+                val connectKey = message.connectKey
+
+                if (messageState == HandshakeMessage.HELLO) {
+                    // we create a NEW publication for the handshake, which connects directly to the client handshake subscription
+
+                    // we explicitly have the publisher "connect to itself", because we are using MDC to work around NAT.
+                    // It will "auto-connect" to the correct client port (negotiated by the MDC client subscription negotiating on the
+                    // control port of the server)
+                    val publicationUri = uriHandshake(CommonContext.UDP_MEDIA, isReliable)
+                        .controlEndpoint(ipInfo.getAeronPubAddress(isRemoteIpv4) + ":" + mdcPortPub)
+
+
+                    // this will always connect to the CLIENT handshake subscription!
+                    val publication = try {
+                        driver.addPublication(publicationUri, message.streamId, logInfo, false)
+                    } catch (e: Exception) {
+                        // we should immediately remove the logbuffer for this! Aeron will **EVENTUALLY** remove the logbuffer, but if errors
+                        // and connections occur too quickly (within the cleanup/linger period), we can run out of memory!
+                        driver.deleteLogFile(image)
+
+                        server.listenerManager.notifyError(ServerHandshakeException("[$logInfo] Cannot create publication back to $clientAddressString", e))
+                        return@launch
+                    }
+
+                    try {
+                        // we actually have to wait for it to connect before we continue.
+                        //
+                        driver.waitForConnection(shutdown, publication, handshakeTimeoutNs, logInfo) { cause ->
+                            ServerTimedoutException("$logInfo publication cannot connect with client in ${Sys.getTimePrettyFull(handshakeTimeoutNs)}", cause)
+                        }
+                    } catch (e: Exception) {
+                        // we should immediately remove the logbuffer for this! Aeron will **EVENTUALLY** remove the logbuffer, but if errors
+                        // and connections occur too quickly (within the cleanup/linger period), we can run out of memory!
+                        driver.deleteLogFile(image)
+
+                        server.listenerManager.notifyError(ServerHandshakeException("[$logInfo] Cannot create publication back to $clientAddressString", e))
+                        return@launch
+                    }
+
+                    try {
+                        val success = handshake.processUdpHandshakeMessageServer(
+                            server = server,
+                            handshaker = handshaker,
+                            handshakePublication = publication,
+                            publicKey = message.publicKey!!,
+                            clientAddress = clientAddress,
+                            clientAddressString = clientAddressString,
+                            portPub = message.port,
+                            portSub = serverPortSub,
+                            mdcPortPub = mdcPortPub,
+                            isReliable = isReliable,
+                            message = message,
+                            logInfo = logInfo,
+                            logger = logger
+                        )
+
+                        if (success) {
+                            publications[connectKey] = publication
+                        } else {
+                            // we should immediately remove the logbuffer for this! Aeron will **EVENTUALLY** remove the logbuffer, but if errors
+                            // and connections occur too quickly (within the cleanup/linger period), we can run out of memory!
+                            driver.deleteLogFile(image)
+
+                            try {
+                                // we might not be able to close this connection.
+                                driver.close(publication, logInfo)
+                            }
+                            catch (e: Exception) {
+                                server.listenerManager.notifyError(e)
+                            }
+                        }
+                    } catch (e: Exception) {
+                        // we should immediately remove the logbuffer for this! Aeron will **EVENTUALLY** remove the logbuffer, but if errors
+                        // and connections occur too quickly (within the cleanup/linger period), we can run out of memory!
+                        driver.deleteLogFile(image)
+
+                        try {
+                            // we might not be able to close this connection.
+                            driver.close(publication, logInfo)
+                        }
+                        catch (e: Exception) {
+                            driver.close(publication, logInfo)
+                        }
+
+                        server.listenerManager.notifyError(ServerHandshakeException("[$logInfo] Error processing IPC handshake", e))
+                    }
+                } else {
+                    // HandshakeMessage.DONE
+
+                    val publication = publications.remove(connectKey)
+
+                    if (publication == null) {
+                        // we should immediately remove the logbuffer for this! Aeron will **EVENTUALLY** remove the logbuffer, but if errors
+                        // and connections occur too quickly (within the cleanup/linger period), we can run out of memory!
+                        driver.deleteLogFile(image)
+
+                        server.listenerManager.notifyError(ServerHandshakeException("[$logInfo] No publication back to $clientAddressString"))
+                        return@launch
+                    }
+
+                    try {
+                        handshake.validateMessageTypeAndDoPending(
+                            server = server,
+                            handshaker = handshaker,
+                            handshakePublication = publication,
+                            message = message,
+                            logInfo = logInfo,
+                            logger = logger
+                        )
+                    } catch (e: Exception) {
+                        server.listenerManager.notifyError(ServerHandshakeException("[$logInfo] Error processing IPC handshake", e))
+                    }
+
+                    try {
+                        // we might not be able to close this connection.
+                        driver.close(publication, logInfo)
+                    }
+                    catch (e: Exception) {
+                        server.listenerManager.notifyError(e)
+                    }
+
+                    // we should immediately remove the logbuffer for this! Aeron will **EVENTUALLY** remove the logbuffer, but if errors
+                    // and connections occur too quickly (within the cleanup/linger period), we can run out of memory!
+                    driver.deleteLogFile(image)
+                }
+            }
+        }
+
+        fun close() {
+            publications.forEach { (connectKey, publication) ->
+                AeronDriver.sessionIdAllocator.free(publication.sessionId())
+
+                try {
+                    // we might not be able to close this connection.
+                    driver.close(publication, "Server Handshake ($connectKey)")
+                }
+                catch (e: Exception) {
+                    server.listenerManager.notifyError(e)
+                }
+            }
+            publications.clear()
+        }
+    }
+
+    fun <CONNECTION : Connection> ipc(server: Server<CONNECTION>, handshake: ServerHandshake<CONNECTION>): AeronPoller {
         val logger = server.logger
-        val connectionFunc = server.connectionFunc
+        val config = server.config as ServerConfiguration
 
-        val poller = if (config.enableIpc) {
-            val driver = ServerIpcDriver(
-                streamId = config.ipcId,
-                sessionId = AeronDriver.RESERVED_SESSION_ID_INVALID
+        val poller = try {
+            val driver = ServerHandshakeDriver.build(
+                aeronDriver = server.aeronDriver,
+                isIpc = true,
+                port = 0,
+                ipInfo = server.ipInfo,
+                streamIdSub = config.ipcId,
+                sessionIdSub = AeronDriver.RESERVED_SESSION_ID_INVALID,
+                logInfo = "HANDSHAKE-IPC"
             )
-            driver.build(aeronDriver, logger)
-
-            val subscription = driver.subscription
 
             object : AeronPoller {
-                val handler = FragmentAssembler { buffer: DirectBuffer, offset: Int, length: Int, header: Header ->
-                    ipcProcessing(logger, server, aeronDriver, header, buffer, offset, length, handshake, connectionFunc)
-                }
+                // NOTE: Handlers are called on the client conductor thread. The client conductor thread expects handlers to do safe
+                //  publication of any state to other threads and not be:
+                //   - long running
+                //   - re-entrant with the client
+                val subscription = driver.subscription
+
+                val delegate = IpcProc(logger, server, server.aeronDriver, handshake)
+                val handler = FragmentAssembler(delegate)
 
                 override fun poll(): Int {
                     return subscription.poll(handler, 1)
                 }
 
                 override fun close() {
-                    subscription.close()
+                    delegate.close()
+                    handler.clear()
+                    try {
+                        driver.unsafeClose()
+                    }
+                    catch (ignored: Exception) {
+                        // we are already shutting down, ignore
+                    }
+                    logger.info("Closed IPC poller")
                 }
 
-                override val info = driver.info
+                override val info = "IPC ${driver.info}"
             }
-        } else {
+        } catch (e: Exception) {
+            server.listenerManager.notifyError(ServerException("Unable to create IPC listener.", e))
             disabled("IPC Disabled")
         }
 
-        logger.info { poller.info }
         return poller
     }
 
 
 
-    fun <CONNECTION : Connection> ip4(
-        aeronDriver: AeronDriver, config: ServerConfiguration, server: Server<CONNECTION>, handshake: ServerHandshake<CONNECTION>
-    ): AeronPoller
-    {
+    fun <CONNECTION : Connection> ip4(server: Server<CONNECTION>, handshake: ServerHandshake<CONNECTION>): AeronPoller {
         val logger = server.logger
-        val connectionFunc = server.connectionFunc
+        val config = server.config
         val isReliable = config.isReliable
 
-        val poller = if (server.canUseIPv4) {
-            val driver = ServerUdpDriver(
-                listenAddress = server.listenIPv4Address!!,
-                port = config.port,
-                streamId = AeronDriver.UDP_HANDSHAKE_STREAM_ID,
-                sessionId = AeronDriver.RESERVED_SESSION_ID_INVALID,
-                connectionTimeoutSec = config.connectionCloseTimeoutInSeconds,
-                isReliable = isReliable
+        val poller = try {
+            val driver = ServerHandshakeDriver.build(
+                aeronDriver = server.aeronDriver,
+                isIpc = false,
+                ipInfo = server.ipInfo,
+                port = server.port1,
+                streamIdSub = config.udpId,
+                sessionIdSub = 9,
+                logInfo = "HANDSHAKE-IPv4"
             )
 
-            driver.build(aeronDriver, logger)
-
-            val subscription = driver.subscription
-
             object : AeronPoller {
-                /**
-                 * Note:
-                 * Reassembly has been shown to be minimal impact to latency. But not totally negligible. If the lowest latency is
-                 * desired, then limiting message sizes to MTU size is a good practice.
-                 *
-                 * There is a maximum length allowed for messages which is the min of 1/8th a term length or 16MB.
-                 * Messages larger than this should chunked using an application level chunking protocol. Chunking has better recovery
-                 * properties from failure and streams with mechanical sympathy.
-                 */
-                val handler = FragmentAssembler { buffer: DirectBuffer, offset: Int, length: Int, header: Header ->
-                    ipProcessing(logger, server, isReliable, aeronDriver, false, header, buffer, offset, length, handshake, connectionFunc)
-                }
+                // NOTE: Handlers are called on the client conductor thread. The client conductor thread expects handlers to do safe
+                //  publication of any state to other threads and not be:
+                //   - long running
+                //   - re-entrant with the client
+                val subscription = driver.subscription
+
+                val delegate = UdpProc(logger, server, server.aeronDriver, handshake, isReliable)
+                val handler = FragmentAssembler(delegate)
 
                 override fun poll(): Int {
                     return subscription.poll(handler, 1)
                 }
 
                 override fun close() {
-                    subscription.close()
+                    delegate.close()
+                    handler.clear()
+                    try {
+                        driver.unsafeClose()
+                    }
+                    catch (ignored: Exception) {
+                        // we are already shutting down, ignore
+                    }
+                    logger.info("Closed IPv4 poller")
                 }
 
                 override val info = "IPv4 ${driver.info}"
             }
-        } else {
+        } catch (e: Exception) {
+            server.listenerManager.notifyError(ServerException("Unable to create IPv4 listener.", e))
             disabled("IPv4 Disabled")
         }
 
-        logger.info { poller.info }
         return poller
     }
 
-    fun <CONNECTION : Connection> ip6(
-        aeronDriver: AeronDriver, config: ServerConfiguration, server: Server<CONNECTION>, handshake: ServerHandshake<CONNECTION>
-    ): AeronPoller
-    {
+    fun <CONNECTION : Connection> ip6(server: Server<CONNECTION>, handshake: ServerHandshake<CONNECTION>): AeronPoller {
         val logger = server.logger
-        val connectionFunc = server.connectionFunc
+        val config = server.config
         val isReliable = config.isReliable
 
-        val poller = if (server.canUseIPv6) {
-            val driver = ServerUdpDriver(
-                listenAddress = server.listenIPv6Address!!,
-                port = config.port,
-                streamId = AeronDriver.UDP_HANDSHAKE_STREAM_ID,
-                sessionId = AeronDriver.RESERVED_SESSION_ID_INVALID,
-                connectionTimeoutSec = config.connectionCloseTimeoutInSeconds,
-                isReliable = isReliable
+        val poller =  try {
+            val driver = ServerHandshakeDriver.build(
+                aeronDriver = server.aeronDriver,
+                isIpc = false,
+                ipInfo = server.ipInfo,
+                port = server.port1,
+                streamIdSub = config.udpId,
+                sessionIdSub = 0,
+                logInfo = "HANDSHAKE-IPv6"
             )
 
-            driver.build(aeronDriver, logger)
-
-            val subscription = driver.subscription
-
             object : AeronPoller {
-                /**
-                 * Note:
-                 * Reassembly has been shown to be minimal impact to latency. But not totally negligible. If the lowest latency is
-                 * desired, then limiting message sizes to MTU size is a good practice.
-                 *
-                 * There is a maximum length allowed for messages which is the min of 1/8th a term length or 16MB.
-                 * Messages larger than this should chunked using an application level chunking protocol. Chunking has better recovery
-                 * properties from failure and streams with mechanical sympathy.
-                 */
-                val handler = FragmentAssembler { buffer: DirectBuffer, offset: Int, length: Int, header: Header ->
-                    ipProcessing(logger, server, isReliable, aeronDriver, false, header, buffer, offset, length, handshake, connectionFunc)
-                }
+                // NOTE: Handlers are called on the client conductor thread. The client conductor thread expects handlers to do safe
+                //  publication of any state to other threads and not be:
+                //   - long running
+                //   - re-entrant with the client
+                val subscription = driver.subscription
+
+                val delegate = UdpProc(logger, server, server.aeronDriver, handshake, isReliable)
+                val handler = FragmentAssembler(delegate)
 
                 override fun poll(): Int {
                     return subscription.poll(handler, 1)
                 }
 
                 override fun close() {
-                    subscription.close()
+                    delegate.close()
+                    handler.clear()
+                    try {
+                        driver.unsafeClose()
+                    }
+                    catch (ignored: Exception) {
+                        // we are already shutting down, ignore
+                    }
+                    logger.info("Closed IPv4 poller")
                 }
 
                 override val info = "IPv6 ${driver.info}"
             }
-        } else {
+        } catch (e: Exception) {
+            server.listenerManager.notifyError(ServerException("Unable to create IPv6 listener."))
             disabled("IPv6 Disabled")
         }
 
-        logger.info { poller.info }
         return poller
     }
 
-    fun <CONNECTION : Connection> ip6Wildcard(
-        aeronDriver: AeronDriver, config: ServerConfiguration, server: Server<CONNECTION>, handshake: ServerHandshake<CONNECTION>
-    ): AeronPoller {
+    fun <CONNECTION : Connection> ip6Wildcard(server: Server<CONNECTION>, handshake: ServerHandshake<CONNECTION>): AeronPoller {
+
         val logger = server.logger
-        val connectionFunc = server.connectionFunc
+        val config = server.config
         val isReliable = config.isReliable
 
-        val driver = ServerUdpDriver(
-            listenAddress = server.listenIPv6Address!!,
-            port = config.port,
-            streamId = AeronDriver.UDP_HANDSHAKE_STREAM_ID,
-            sessionId = AeronDriver.RESERVED_SESSION_ID_INVALID,
-            connectionTimeoutSec = config.connectionCloseTimeoutInSeconds,
-            isReliable = isReliable
-        )
+        val poller = try {
+            val driver = ServerHandshakeDriver.build(
+                aeronDriver = server.aeronDriver,
+                isIpc = false,
+                ipInfo = server.ipInfo,
+                port = server.port1,
+                streamIdSub = config.udpId,
+                sessionIdSub = 0,
+                logInfo = "HANDSHAKE-IPv4+6"
+            )
 
-        driver.build(aeronDriver, logger)
+            object : AeronPoller {
+                // NOTE: Handlers are called on the client conductor thread. The client conductor thread expects handlers to do safe
+                //  publication of any state to other threads and not be:
+                //   - long running
+                //   - re-entrant with the client
+                val subscription = driver.subscription
 
-        val subscription = driver.subscription
+                val delegate = UdpProc(logger, server, server.aeronDriver, handshake, isReliable)
+                val handler = FragmentAssembler(delegate)
 
-        val poller = object : AeronPoller {
-            /**
-             * Note:
-             * Reassembly has been shown to be minimal impact to latency. But not totally negligible. If the lowest latency is
-             * desired, then limiting message sizes to MTU size is a good practice.
-             *
-             * There is a maximum length allowed for messages which is the min of 1/8th a term length or 16MB.
-             * Messages larger than this should chunked using an application level chunking protocol. Chunking has better recovery
-             * properties from failure and streams with mechanical sympathy.
-             */
-            val handler = FragmentAssembler { buffer: DirectBuffer, offset: Int, length: Int, header: Header ->
-                ipProcessing(logger, server, isReliable, aeronDriver, true, header, buffer, offset, length, handshake, connectionFunc)
+                override fun poll(): Int {
+                    return subscription.poll(handler, 1)
+                }
+
+                override fun close() {
+                    delegate.close()
+                    handler.clear()
+                    try {
+                        driver.unsafeClose()
+                    }
+                    catch (ignored: Exception) {
+                        // we are already shutting down, ignore
+                    }
+                    logger.info("Closed IPv4+6 poller")
+                }
+
+                override val info = "IPv4+6 ${driver.info}"
             }
-
-            override fun poll(): Int {
-                return subscription.poll(handler, 1)
-            }
-
-            override fun close() {
-                subscription.close()
-            }
-
-            override val info = "IPv4+6 ${driver.info}"
+        } catch (e: Exception) {
+            server.listenerManager.notifyError(ServerException("Unable to create IPv4+6 listeners.", e))
+            disabled("IPv4+6 Disabled")
         }
 
-        logger.info { poller.info }
         return poller
     }
 }

src/dorkbox/network/handshake/package-info.java

@@ -0,0 +1,17 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package dorkbox.network.handshake;

src/dorkbox/network/ipFilter/IpSubnetFilterRule.kt

@@ -26,7 +26,7 @@ import java.net.InetAddress
  *
  * Supports both IPv4 and IPv6.
  */
-internal class IpSubnetFilterRule : IpFilterRule {
+class IpSubnetFilterRule : IpFilterRule {
     private val filterRule: IpFilterRule
 
     constructor(ipAddress: String, cidrPrefix: Int) {

src/dorkbox/network/ipFilter/package-info.java

@@ -0,0 +1,17 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package dorkbox.network.ipFilter;

src/dorkbox/network/package-info.java

@@ -0,0 +1,17 @@
+/*
+ * Copyright 2023 dorkbox, llc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package dorkbox.network;

src/dorkbox/network/ping/Ping.kt

@@ -1,5 +1,5 @@
 /*
- * Copyright 2020 dorkbox, llc
+ * Copyright 2023 dorkbox, llc
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -21,7 +21,6 @@ class Ping {
     var packedId = 0
 
     // ping/pong times are the LOWER 8 bytes of a long, which gives us 65 seconds. This is the same as the max value timeout (a short) so this is acceptable
-
     var pingTime = 0L
     var pongTime = 0L