Create automated tests for establishing connection via hole-punch signalling message

[joshuakarp]

Specification

In order to establish a unidirectional connection between two nodes (a 'client' and a 'server'), both nodes need to send each other hole-punching packets (that is, we require a relay message to be passed to the 'server' node to send relay packets back to the 'client' node). It's not sufficient enough for a node to simply be in its node table to establish a connection - the node needs to be aware that a connection is desired.

For a brand new keynode in the Polykey network, it will rely on the seed node/s to establish connections to other nodes in this manner. (However, note that once it becomes aware of other nodes in the network through Kademlia, it can send relay messages to these other nodes, with the aim that the relay message will eventually converge on the target node by "hopping" between nodes.)

This scenario of 3 keynodes needs to be incorporated into the NodeConnection testing suite (currently we only have automated tests for a direct connection between two nodes).

For example, for a node A to connect to node B, we require another node Seed to facilitate the connection. This process is depicted in the following diagram:

           ┌────────┐
    ┌──────►  Seed  ├──────┐
    │      └────────┘      │
    │1                    2│
┌───┴───┐       1      ┌───▼───┐
│       ├──────────────►       │
│   A   │              │   B   │
│       ◄──────────────┤       │
└───────┘       3      └───────┘

We make the assumption that direct, unidirectional connections are already established from A to Seed, and from Seed to B. We also assume that A already knows B's host and port (but B may not know of A's host and port, or even A's node ID). Then, it follows the process:

1. A sends the relay message to Seed (via its direct connection), and simultaneously begins sending hole-punching packets to B's IP
2. Seed receives the relay message, noting the target node ID. It relays this hole-punching message to B (via its direct connection)
3. B receives the hole-punching message, and begins sending hole-punching packets back to A.

Assuming that A is still sending hole-punching packets when B begins sending their own back to A, then a unidirectional connection is established from A to B.

The above process was previously manually verified by a collection of 3 scratchpad test files:

test-triple-client.ts:

import type { Host, Port, TLSConfig } from './src/network/types';
import type { NodeId, NodeAddress, NodeBucket } from './src/nodes/types';
import type { PrivateKeyPem, CertificatePemChain } from './src/keys/types';
import { ForwardProxy, ReverseProxy, utils as networkUtils } from './src/network';
import { NodeManager } from './src/nodes';
import { KeyManager, utils as keysUtils } from './src/keys';

import os from 'os';
import path from 'path';
import fs from 'fs';
import Logger, { LogLevel, StreamHandler } from '@matrixai/logger';


async function main() {

  const authToken = 'AUTH';

  const fwd = new ForwardProxy({
    authToken: authToken
  });

  // Keys and cert generation
  //const keyPair = await keysUtils.generateDeterministicKeyPair(4098, 'a');
  const keyPair = await keysUtils.generateKeyPair(4098);
  const keyPairPem = keysUtils.keyPairToPem(keyPair);
  const cert = keysUtils.generateCertificate(
    keyPair.publicKey,
    keyPair.privateKey,
    keyPair.privateKey,
    12332432423
  );
  const certPem = keysUtils.certToPem(cert);
  const nodeId = networkUtils.certNodeId(cert);
  const tlsConfig = {
    keyPrivatePem: keyPairPem.privateKey as PrivateKeyPem,
    certChainPem: certPem as CertificatePemChain
  } as TLSConfig;
  // --------------------------------------------------

  await fwd.start({
    proxyHost: '127.0.0.1' as Host,
    proxyPort: 44709 as Port,
    egressHost: '127.0.0.1' as Host,
    egressPort: 31111 as Port,
    tlsConfig: tlsConfig
  });

  // --------------------------------------------------
  // Setup
  const logger = new Logger('NodeClientTest', LogLevel.WARN, [
    new StreamHandler(),
  ]);
  let dataDir: string;
  let keysPath: string;
  let nodesPath: string;

  dataDir = await fs.promises.mkdtemp(path.join(os.tmpdir(), 'TripleTestClient'));
  keysPath = path.join(dataDir, 'keys');
  nodesPath = path.join(dataDir, 'nodes');

  let keyManager = new KeyManager({
    keysPath: keysPath,
    fs: fs,
    logger: logger
  });

  // Only needed to pass into nodeManager constructor
  const revProxy = new ReverseProxy({
    logger: logger
  });

  let nodeManager = new NodeManager({
    nodesPath: nodesPath,
    keyManager: keyManager,
    fwdProxy: fwd,
    revProxy: revProxy,
    fs: fs,
    logger: logger
  });

  let brokers: NodeBucket = {};
  // Set the console output of "BROKER NODE ID:" here (from test-triple-broker.ts)
  const brokerNodeId = 'h2xWGAVWDo67c3RB3vnfvASjL7HqdR+vcrSSmX09JbA=' as NodeId;
  brokers[brokerNodeId] = {
    address: { ip: '127.2.0.1' as Host, port: 2222 as Port },
    lastUpdated: new Date()
  };
  await keyManager.start({ password: 'password' });
  await nodeManager.start({
    nodeId: nodeId,
    brokerNodes: brokers
  });
  // --------------------------------------------------

  console.log("CLIENT NODE ID:", nodeId);
  // Client has no known nodes in its database (apart from what's discovered from
  // the initial calibration with brokers)

  const buckets = await nodeManager.getAllBuckets();
  console.log(`KNOWN NODES:`);
  buckets.forEach((bucket) => {
    for (const nodeId of Object.keys(bucket) as Array<NodeId>) {
      console.log(`${nodeId} -> { ${bucket[nodeId].address.ip}:${bucket[nodeId].address.port} }`);
    }
  });

  // We want to find 'NodeId3'.
  console.log("ATTEMPTING TO LOCATE: NodeId3");
  console.log("Calling nodeManager.getClosestGlobalNodes('NodeId3')");
  const found = await nodeManager.getClosestGlobalNodes('NodeId3' as NodeId);
  console.log(`KNOWN NODES:`);
  
  const newBuckets = await nodeManager.getAllBuckets();
  newBuckets.forEach((bucket) => {
    for (const nodeId of Object.keys(bucket) as Array<NodeId>) {
      console.log(`${nodeId} -> { ${bucket[nodeId].address.ip}:${bucket[nodeId].address.port} }`);
    }
  });
  // console.log(`FOUND NodeId3?: ${found}`)
}

main();

test-triple-broker.ts:

import type { Host, Port, TLSConfig } from './src/network/types';
import type { PrivateKeyPem, CertificatePemChain } from './src/keys/types';
import ReverseProxy from './src/network/ReverseProxy';
import { ForwardProxy, utils as networkUtils } from './src/network';
import { KeyManager, utils as keysUtils } from './src/keys';
import { VaultManager } from './src/vaults';
import { NodeManager } from './src/nodes';
import type { NodeId, NodeAddress } from './src/nodes/types';
import GRPCServer from './src/grpc/GRPCServer';
import { AgentService, createAgentService } from './src/agent';
import { GitBackend } from './src/git';
import os from 'os';
import path from 'path';
import fs from 'fs';
import Logger, { LogLevel, StreamHandler } from '@matrixai/logger';
import { NodeIdMessage } from './src/proto/js/Agent_pb';

async function main() {

  const authToken = 'AUTH';

  const fwd = new ForwardProxy({
    authToken: authToken
  });

  // Keys and cert generation
  //const keyPair = await keysUtils.generateDeterministicKeyPair(4098, 'b');
  const keyPair = await keysUtils.generateKeyPair(4098);
  const keyPairPem = keysUtils.keyPairToPem(keyPair);
  const cert = keysUtils.generateCertificate(
    keyPair.publicKey,
    keyPair.privateKey,
    keyPair.privateKey,
    12332432423
  );
  const certPem = keysUtils.certToPem(cert);
  const nodeId = networkUtils.certNodeId(cert);

  const tlsConfig = {
    keyPrivatePem: keyPairPem.privateKey as PrivateKeyPem,
    certChainPem: certPem as CertificatePemChain
  } as TLSConfig;

  // --------------------------------------------------

  await fwd.start({
    proxyHost: '127.2.0.1' as Host,
    proxyPort: 32221 as Port,
    egressHost: '127.2.0.1' as Host,
    egressPort: 32222 as Port,
    tlsConfig: tlsConfig
  });

  // --------------------------------------------------
  // Setup
  const logger = new Logger('NodeClientTest', LogLevel.WARN, [
    new StreamHandler(),
  ]);
  let dataDir: string;
  let keysPath: string;
  let vaultsPath: string;
  let nodesPath: string;

  let keyManager: KeyManager;
  let vaultManager: VaultManager;
  let nodeManager: NodeManager;

  dataDir = await fs.promises.mkdtemp(path.join(os.tmpdir(), 'TripleTestBroker'));
  keysPath = path.join(dataDir, 'keys');
  vaultsPath = path.join(dataDir, 'vaults');
  nodesPath = path.join(dataDir, 'nodes');

  keyManager = new KeyManager({
    keysPath: keysPath,
    fs: fs,
    logger: logger
  });
  
  vaultManager = new VaultManager({
    vaultsPath: vaultsPath,
    keyManager: keyManager,
    fs: fs,
    logger: logger,
  });

  const rev = new ReverseProxy({
    logger: logger
  });

  nodeManager = new NodeManager({
    nodesPath: nodesPath,
    keyManager: keyManager,
    fwdProxy: fwd,
    revProxy: rev,
    fs: fs,
    logger: logger
  });

  const gitBackend = new GitBackend({
    getVault: vaultManager.getVault.bind(vaultManager),
    getVaultID: vaultManager.getVaultIds.bind(vaultManager),
    getVaultNames: vaultManager.listVaults.bind(vaultManager),
    logger: logger,
  });

  await keyManager.start({ password: 'password' });
  await vaultManager.start({});
  await nodeManager.start({ nodeId: nodeId });
  // --------------------------------------------------

  console.log("BROKER NODE ID:", nodeId);
  // Broker has one node in its database (the 'server' node)
  // Set the console output of "SERVER NODE ID:" here (from test-triple-server.ts)
  const serverNodeId = 'VUF66MkpoATLR9qu21ttS6rQdUbGcsL7gms/PhivD5Y=' as NodeId;
  await nodeManager.setNode(
    serverNodeId,
    {ip: '127.3.0.1', port: 33333} as NodeAddress
  );

  await nodeManager.createConnectionToNode(
    serverNodeId,
    {ip: '127.3.0.1', port: 33333} as NodeAddress
  );

  const buckets = await nodeManager.getAllBuckets();
  console.log(`KNOWN NODES:`);
  buckets.forEach((bucket) => {
    for (const nodeId of Object.keys(bucket) as Array<NodeId>) {
      console.log(`${nodeId} -> { ${bucket[nodeId].address.ip}:${bucket[nodeId].address.port} }`);
    }
  });

  const agentService = createAgentService({
    keyManager: keyManager,
    vaultManager: vaultManager,
    nodeManager: nodeManager,
    git: gitBackend
  });

  const server = new GRPCServer({
    services: [
      [ AgentService, agentService ]
    ]
  });

  await server.start({
    host: '127.2.0.1' as Host,
  });

  console.log('GRPC HOST', server.getHost());
  console.log('GRPC PORT', server.getPort());

  // ingress host and port
  // upstream host and port is the GRPC server
  await rev.start({
    ingressHost: '127.2.0.1' as Host,
    ingressPort: 2222 as Port,
    serverHost: '127.2.0.1' as Host,
    serverPort: server.getPort(),
    tlsConfig: tlsConfig
  });

  console.log('INGRESS HOST', rev.getIngressHost());
  console.log('INGRESS PORT', rev.getIngressPort());
}

main();

test-triple-server.ts:

import type { Host, Port, TLSConfig } from './src/network/types';
import type { PrivateKeyPem, CertificatePemChain } from './src/keys/types';
import ReverseProxy from './src/network/ReverseProxy';
import { ForwardProxy, utils as networkUtils } from './src/network';
import { KeyManager, utils as keysUtils } from './src/keys';
import { VaultManager } from './src/vaults';
import { NodeManager } from './src/nodes';
import type { NodeId, NodeAddress } from './src/nodes/types';
import GRPCServer from './src/grpc/GRPCServer';
import { AgentService, createAgentService } from './src/agent';
import { GitBackend } from './src/git';
import os from 'os';
import path from 'path';
import fs from 'fs';
import Logger, { LogLevel, StreamHandler } from '@matrixai/logger';
import { NodeIdMessage } from './src/proto/js/Agent_pb';

async function main () {

  const rev = new ReverseProxy();

  // Keys and cert generation
  //const keyPair = await keysUtils.generateDeterministicKeyPair(4098, 'c');
  const keyPair = await keysUtils.generateKeyPair(4098);
  const keyPairPem = keysUtils.keyPairToPem(keyPair);
  const cert = keysUtils.generateCertificate(
    keyPair.publicKey,
    keyPair.privateKey,
    keyPair.privateKey,
    12332432423
  );
  // can be used to get node ID
  const certPem = keysUtils.certToPem(cert);
  const nodeId = networkUtils.certNodeId(cert);
  console.log('SERVER NODE ID:', nodeId);
  
  // --------------------------------------------------
  // Setup
  const logger = new Logger('NodeServerTest', LogLevel.WARN, [
    new StreamHandler(),
  ]);
  let dataDir: string;
  let keysPath: string;
  let vaultsPath: string;
  let nodesPath: string;

  let keyManager: KeyManager;
  let vaultManager: VaultManager;
  let nodeManager: NodeManager;

  dataDir = await fs.promises.mkdtemp(path.join(os.tmpdir(), 'polykey-test-'));
  keysPath = path.join(dataDir, 'keys');
  vaultsPath = path.join(dataDir, 'vaults');
  nodesPath = path.join(dataDir, 'nodes');

  keyManager = new KeyManager({
    keysPath,
    fs: fs,
    logger: logger,
  });

  vaultManager = new VaultManager({
    vaultsPath: vaultsPath,
    keyManager: keyManager,
    fs: fs,
    logger: logger,
  });

  // Only needed to pass into nodeManager constructor - won't be forwarding calls
  // so no need to start
  const fwdProxy = new ForwardProxy({
    authToken: '',
    logger: logger,
  });

  nodeManager = new NodeManager({
    nodesPath: nodesPath,
    keyManager: keyManager,
    fwdProxy: fwdProxy,
    revProxy: rev,
    fs: fs,
    logger: logger
  });

  const gitBackend = new GitBackend({
    getVault: vaultManager.getVault.bind(vaultManager),
    getVaultID: vaultManager.getVaultIds.bind(vaultManager),
    getVaultNames: vaultManager.listVaults.bind(vaultManager),
    logger: logger,
  });

  await keyManager.start({ password: 'password' });
  await vaultManager.start({});
  await nodeManager.start({ nodeId: nodeId });
  // --------------------------------------------------

  const agentService = createAgentService({
    keyManager: keyManager,
    vaultManager: vaultManager,
    nodeManager: nodeManager,
    git: gitBackend
  });

  const server = new GRPCServer({
    services: [
      [ AgentService, agentService ]
    ]
  });

  await server.start({
    host: '127.3.0.1' as Host,
  });

  console.log('GRPC HOST', server.getHost());
  console.log('GRPC PORT', server.getPort());

  const tlsConfig = {
    keyPrivatePem: keyPairPem.privateKey as PrivateKeyPem,
    certChainPem: certPem as CertificatePemChain
  } as TLSConfig;

  // ingress host and port
  // upstream host and port is the GRPC server
  await rev.start({
    ingressHost: '127.3.0.1' as Host,
    ingressPort: 33333 as Port,
    serverHost: '127.3.0.1' as Host,
    serverPort: server.getPort(),
    tlsConfig: tlsConfig
  });

  console.log('INGRESS HOST', rev.getIngressHost());
  console.log('INGRESS PORT', rev.getIngressPort());

  // Server has some known node 'NodeId3'
  await nodeManager.setNode(
    'NodeId3' as NodeId,
    { ip: '3.3.3.3', port: 3333 } as NodeAddress
  );

  const buckets = await nodeManager.getAllBuckets();
  console.log(`KNOWN NODES:`);
  buckets.forEach((bucket) => {
    for (const nodeId of Object.keys(bucket) as Array<NodeId>) {
      console.log(`${nodeId} -> { ${bucket[nodeId].address.ip}:${bucket[nodeId].address.port} }`);
    }
  });

}

main();

Additional context

• some original discussion of this from an old nodes MR https://gitlab.com/MatrixAI/Engineering/Polykey/js-polykey/-/merge_requests/178#note_561046259
• some further discussion of mocking 3 nodes in some scratchpad test files https://gitlab.com/MatrixAI/Engineering/Polykey/js-polykey/-/merge_requests/178#note_577565530

Tasks

1. Implement a test that mocks 2 other nodes (a "seed" and a "target node" to connect to) to test the above process.

[joshuakarp]

This can likely be resolved once we deploy the testnet in #194, and look into multinode testing.

[CMCDragonkai]

@joshuakarp

Should spec out this into a Development issue cause it will involve the development of new tests.

We can make this deterministic by mocking/simulating the seed nodes. And as we are finishing up #194, this should enable us to automate the seed node deployment during these tests.

These tests are likely to be expensive to run alot of, and therefore impacts #264.

[joshuakarp]

Fleshed out the specification for this issue.

[CMCDragonkai]

Seems like planning here should be incorporated into #159. @joshuakarp will rely on you to flesh out the specification for both issues.

[joshuakarp]

Note that we had a previous send hole punch message test in NodeConnection.test.ts removed here #283 (comment) in ffca4d9. It was doing more of a "2 keynode" test:

           ┌────────┐
    ┌──────►  Seed  │ (failure at this point)
    │      └────────┘      
    │1                    
┌───┴───┐
│       │
│   A   │              
│       │
└───────┘       

where we relayed a hole-punch message to a target, but the target couldn't relay it any further.

Keeping it here for posterity (with the InvalidNodeId error fixed), but note that it wasn't really testing anything, other than the fact that the message was relayed:

  test('sends hole punch message to connected target (expected to be broker, to relay further)', async () => {
    const conn = await NodeConnection.createNodeConnection({
      targetNodeId: targetNodeId,
      targetHost: targetHost,
      targetPort: targetPort,
      forwardProxy: clientFwdProxy,
      keyManager: clientKeyManager,
      logger: logger,
    });
    await serverRevProxy.openConnection(sourceHost, sourcePort);

    const egressAddress = networkUtils.buildAddress(
      clientFwdProxy.egressHost as Host,
      clientFwdProxy.egressPort as Port,
    );
    const signature = await clientKeyManager.signWithRootKeyPair(
      Buffer.from(egressAddress),
    );

    const dummyNode = makeNodeId(
      'vi3et1hrpv2m2lrplcm7cu913kr45v51cak54vm68anlbvuf83ra0',
    );
    // The dummyNode differs from the node ID of the connected target,
    // indicating that this relay message is intended for another node.
    // Expected to throw an error, as the connection to 1.1.1.1:11111 would not
    // exist on the server's side. A broker is expected to have this pre-existing
    // connection.
    await expect(
      async () =>
        await conn.sendHolePunchMessage(
          sourceNodeId,
          dummyNode,
          egressAddress,
          signature,
        ),
    ).rejects.toThrow(grpcErrors.ErrorGRPCClientCall);

    await conn.stop();
    await serverRevProxy.closeConnection(
      clientFwdProxy.egressHost,
      clientFwdProxy.egressPort,
    );
    await conn.destroy();
  });

[joshuakarp]

We can potentially use the global pkAgent setup in tests/ as the mocked seed node (somewhat similar to the pk agent start tests). This would mean though, that these tests would need to be at the CLI level, as a large-scale integration test.

Alternatively, we just run another keynode, making it a 3-keynode test.

It's worthwhile as a reminder too that this test wouldn't actually be testing the "hole-punching" functionality, as all the nodes would be running on localhost, and inherently this requires no hole-punching to establish a connection.

[CMCDragonkai]

In the past we have simulated firewalls/NATs on the OS to test this. This used NixOS modules to do so before. If we want the test to be as portable as possible, in terms of being able to run in CI/CD, it might work by using docker in docker, and so we can use setup network namespaces inside the dind system that we are using in #292 to build the docker image and testing it. However I'm not sure what the full capabilities are. I think for now we can test the entire behaviour spec by mocking and other things and then do a manual real world tests.

[joshuakarp]

As a result of #310, we should have general connection behaviour tested appropriately. As far as I'm aware though, there's no simulation of firewalls/NATs to test the hole-punching mechanism that were added in #310. Perhaps @tegefaulkes can confirm this too.

As a result, this can come after the testnet deployment (most likely arising from manual testing with the deployed seed nodes).

[joshuakarp]

Remember any automated tests (for nodesHolePunchMessage, etc) from agent service should be separated out (as per #310 (comment)).

[tegefaulkes]

No tests were added for NAT and hole punching testing. They seemed out of scope for #310 and as testing goes they are very involved and tricky tests to create.

[CMCDragonkai]

These tests must be written outside or separately from src/tests. This way npm test does not run the NAT traversal testing. This is because NAT traversal testing may require a real network (when going to the seed nodes) or require OS simulation of NAT. A couple solutions here:

1. Create a separate tests-nat directory - disadvantage here is that you lose all your existing jest context and utilities, but you have to configure it again
2. Use https://jestjs.io/docs/cli#--testpathignorepatternsregexarray if the we use something like tests/nat as a subdirectory - this is advantageous for re-using all the same jest context, but just means we have to configure jest to ignore by default these tests, which maybe done in package.json or jest.config.js.

It's best to continue using our jest tooling for these tests, but if we need to use OS simulation, then the jest tests may need to be executing shell commands which then encapsulate scripts that run inside a network namespaces.

[CMCDragonkai]

For testing the hole punching relay message, this can be done inside the normal tests/nodes. It doesn't actually need to be done in our "external NAT" testing.

However to test the hole punching relay message for the testnet.polykey.io, then YES it does require to be done in the external NAT testing. So to solve this issue, we need both new tests into tests/nodes, and also tests that go into the "external NAT" testing because using testnet.polykey.io would use cloud AWS and involving going over the real network.

[CMCDragonkai]

Let's clarify our terminology. This is signalling message, not a relay message. Otherwise it confuses with NAT relaying.

[CMCDragonkai]

With #381 merged, this is already done.