ci: merge staging to master #474

MatrixAI-Bot · 2022-10-05T07:09:38Z

This is an automatic PR generated by the pipeline CI/CD. This will be automatically fast-forward merged if successful.

Tasks

- Enable automated testnet connection tests
~~[ ] - Investigate random crashes~~ - unnecessary since CLI tests will go to Polykey-CLI
- Testnet nodes should never crash regardless of what connecting nodes are doing

MatrixAI-Bot · 2022-10-05T07:09:39Z

Pipeline Attempt on 658670966 for 06df14a

https://gitlab.com/MatrixAI/open-source/Polykey/-/pipelines/658670966

ghost · 2022-10-05T07:10:20Z

👇 Click on the image for a new way to code review

Legend

CMCDragonkai · 2022-10-05T08:48:53Z

@tegefaulkes the Nat tests failed here. There's a failure that is not just a timeout.

tegefaulkes · 2022-10-06T02:18:50Z

Looking at it now. The initial problem here is that the pings are failing after the default timeout of 20 seconds.

  console.time
    ping1: 22561 ms

      at Object.<anonymous> (tests/nat/DMZ.test.ts:213:15)

  console.log
    {"success":false,"message":"No response received"}

      at Object.<anonymous> (tests/nat/DMZ.test.ts:214:15)

  console.log
    {"type":"ErrorCLINodePingFailed","data":{"message":"No response received","timestamp":"2022-10-06T02:14:59.406Z","data":{},"stack":"ErrorCLINodePingFailed: No response received\n    at /home/faulkes/matrixcode/polykey/js-polykey/src/bin/nodes/CommandPing.ts:72:19\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at async CommandPing.<anonymous> (/home/faulkes/matrixcode/polykey/js-polykey/src/bin/CommandPolykey.ts:81:7)\n    at async CommandPolykey.parseAsync (/home/faulkes/matrixcode/polykey/js-polykey/node_modules/commander/lib/command.js:923:5)\n    at async main (/home/faulkes/matrixcode/polykey/js-polykey/src/bin/polykey.ts:54:5)","description":"Node was not online or not found.","exitCode":1}}

      at Object.<anonymous> (tests/nat/DMZ.test.ts:215:15)

So it's not related to the bug from yesterday. I need to look at the bin command closer to know why the ping is failing. But right now it's consistent with failing to connect.

tegefaulkes · 2022-10-06T03:05:56Z

It's odd, the core problem is that the connection is timing out with UTP_ETIMEDOUT. So it's UTP timing out here after 20 seconds. I'm not sure what change caused this, I need to review the the last merge for that.

Another odd thing is that setting a shorter timeout of '10000' doesn't change the outcome, I'd expect it to abort with a different timeout error but it doesn't. So there is a bug in the code here somewhere. But that relates to the nodes domain cancellability i'm currently working on so it should be addressed there.

tegefaulkes · 2022-10-06T03:19:21Z

The only changes was updating the network domain with cancellability. The network and nodes tests are passing so the connection logic is still working, including pingNode. So why is it only failing in the NAT testing? Is it the network namespacing? because we're using alternate IP addresses? Because it's in a separate process?

CMCDragonkai · 2022-10-06T03:38:39Z

Does the test work locally? Maybe you do need to fix the bug in the nodes domain first.

tegefaulkes · 2022-10-06T03:46:54Z

The tests are failing locally. the problem doesn't seem to be nodes domain specifically. So far as I can tell the ForwardConnection is just failing to connect and timing out, but only in the NAT tests. The problem does overlap enough that I'll need to fix it for the nodes PR. I may encounter a fix as part of that work.

I'll come back to this, I need to think on it for a little bit.

MatrixAI-Bot · 2022-10-07T09:28:32Z

Pipeline Attempt on 660898712 for e428206

https://gitlab.com/MatrixAI/open-source/Polykey/-/pipelines/660898712

MatrixAI-Bot · 2022-10-18T03:50:15Z

Pipeline Attempt on 669366315 for d0ffca0

https://gitlab.com/MatrixAI/open-source/Polykey/-/pipelines/669366315

MatrixAI-Bot · 2022-10-25T05:12:57Z

Pipeline Attempt on 675968580 for 0421f73

https://gitlab.com/MatrixAI/open-source/Polykey/-/pipelines/675968580

MatrixAI-Bot · 2022-10-26T07:26:59Z

Pipeline Attempt on 677224551 for b381e29

https://gitlab.com/MatrixAI/open-source/Polykey/-/pipelines/677224551

MatrixAI-Bot · 2022-10-27T06:57:44Z

Pipeline Attempt on 678321421 for bb9f39b

https://gitlab.com/MatrixAI/open-source/Polykey/-/pipelines/678321421

MatrixAI-Bot · 2022-10-27T08:01:11Z

Pipeline Attempt on 678384368 for 04737bc

https://gitlab.com/MatrixAI/open-source/Polykey/-/pipelines/678384368

CMCDragonkai · 2022-10-31T05:31:07Z

Make sure any logs left over should be following the rest of the code in their logging structure.

MatrixAI-Bot · 2022-11-08T08:16:36Z

Pipeline Attempt on 688345460 for 1890801

https://gitlab.com/MatrixAI/open-source/Polykey/-/pipelines/688345460

tegefaulkes · 2022-11-09T01:52:42Z

There seems to be a bug causing the seed nodes to crash when receiving certain connections.

{
    "type": "TypeError",
    "data": {
        "message": "Cannot read properties of null (reading 'finishShutdown')",
        "stack": "TypeError: Cannot read properties of null (reading 'finishShutdown')\n    at JSStreamSocket.finishShutdown (node:internal/js_stream_socket:160:12)\n    at node:internal/js_stream_socket:147:14\n    at processTicksAndRejections (node:internal/process/task_queues:78:11)\n    at runNextTicks (node:internal/process/task_queues:65:3)\n    at processImmediate (node:internal/timers:437:9)"
    }
}

It is internal to some node implementation. It may be related to this issue. nodejs/node#35695.

For now I'm going to disable the testnet connection tests due to the seed nodes instability.

MatrixAI-Bot · 2022-11-09T02:24:33Z

Pipeline Attempt on 689345933 for 2e972e7

https://gitlab.com/MatrixAI/open-source/Polykey/-/pipelines/689345933

MatrixAI-Bot · 2022-11-09T02:32:25Z

Pipeline Attempt on 689348586 for 6a18b6b

https://gitlab.com/MatrixAI/open-source/Polykey/-/pipelines/689348586

CMCDragonkai · 2022-11-09T02:42:55Z

There seems to be a bug causing the seed nodes to crash when receiving certain connections.
{
    "type": "TypeError",
    "data": {
        "message": "Cannot read properties of null (reading 'finishShutdown')",
        "stack": "TypeError: Cannot read properties of null (reading 'finishShutdown')\n    at JSStreamSocket.finishShutdown (node:internal/js_stream_socket:160:12)\n    at node:internal/js_stream_socket:147:14\n    at processTicksAndRejections (node:internal/process/task_queues:78:11)\n    at runNextTicks (node:internal/process/task_queues:65:3)\n    at processImmediate (node:internal/timers:437:9)"
    }
}
It is internal to some node implementation. It may be related to this issue. nodejs/node#35695.

For now I'm going to disable the testnet connection tests due to the seed nodes instability.

I think we saw that exception before? Is this is what is causing the random failures? Or is the random failures coming elsewhere?

Are you sure, it's HTTP2? And not the UDP, uTP or elsewhere?

CMCDragonkai · 2022-11-09T02:43:12Z

Also we should try to replicate that, and use rr to record the execution.

tegefaulkes · 2022-11-09T03:10:21Z

Yeah, I think we've seen this one before. I'm not sure exactly where it's happening but I found that issue after a quick search.

Triggering this bug might be a little tricky. Right now it seems to be caused when I connect to a seed node with the testnetConnection.test.ts test.

MatrixAI-Bot · 2022-11-09T03:35:25Z

Pipeline Attempt on 689376099 for ab8d759

https://gitlab.com/MatrixAI/open-source/Polykey/-/pipelines/689376099

CMCDragonkai · 2022-11-09T03:51:45Z

In order to make it easier for us to see if our manual test is succeeding, we need a pk nodes connections command that lists out all the active node connections. This means we don't have to keep looking over the logs to see if we managed to connect to the testnet.

If this succeeds and we don't crash the testnet, then testnet connection tests should succeed automatically too. We just need to ensure we are doing the same thing.

However our networking system is getting too complex and it is resulting in hard to figure out bugs. We may need to start on a networking refactoring soon after we merge in the feature-crypto.

CMCDragonkai · 2022-11-09T03:52:36Z

We can also add a pk network connections which shows active proxy connections. This can be compared to pk nodes connections.

CMCDragonkai · 2022-11-09T04:15:59Z

We'll look into network refactoring after feature-crypto (this should eliminate the situations where we have random failures). That will also need to fix MatrixAI/js-mdns#1.

Also try to rule out this https://aws.amazon.com/premiumsupport/knowledge-center/ecs-resolve-outofmemory-errors/ happening. It seems it never gets past 50% memory usage in the service.

Remove the old polykey-testnet service and update the cloudwatch dashboard accordingly. We are now using 1 service per NodeId.

[ci skip]

Turns out it wasn't needed. I ran all the tests and there were no errors relating to it. [ci skip]

[ci skip]

`js-quic` integration and Agent migration

MatrixAI-Bot · 2023-08-01T05:46:40Z

Pipeline Attempt on 951440760 for fc82a3d

https://gitlab.com/MatrixAI/open-source/Polykey/-/pipelines/951440760

CMCDragonkai · 2023-08-01T06:28:16Z

Once #534 is merged, this is expected to pass and merge into master. However CI deployment to testnet and mainnet will be moved to Polykey-CLI repository.

MatrixAI-Bot · 2023-08-03T07:40:35Z

Pipeline Attempt on 954294586 for f7708f6

https://gitlab.com/MatrixAI/open-source/Polykey/-/pipelines/954294586

MatrixAI-Bot · 2023-08-03T07:55:42Z

Pipeline Succeeded on 954294586 for f7708f6

https://gitlab.com/MatrixAI/open-source/Polykey/-/pipelines/954294586

MatrixAI-Bot self-assigned this Oct 5, 2022

tegefaulkes mentioned this pull request Oct 10, 2022

Fix failing NAT simulation tests #478

Closed

3 tasks

CMCDragonkai mentioned this pull request Oct 25, 2022

fix: unable to establish connection to seed node #489

Closed

2 tasks

CMCDragonkai mentioned this pull request Nov 6, 2022

Updating Crypto to use WebCrypto API and to replace RSA with ECC #446

Merged

75 tasks

tegefaulkes added 25 commits July 21, 2023 12:46

fix: fixed imports

adb8a31

[ci skip]

fix: fixing up import

b312bb8

[ci skip]

fix: removing placeholder agent handler

b74cadb

[ci skip]

fix: fixing up errors relating to missing connection certificates

bbc6647

[ci skip]

fix: removing echo handler

4aef5db

[ci skip]

fix: reverted changes to Sigchain.ts

b1b2db9

Turns out it wasn't needed. I ran all the tests and there were no errors relating to it. [ci skip]

dep: removed utp-native

d649dbe

[ci skip]

build: fixed pkg command

c56d1e7

[ci skip]

fix: updated Polykeyclient client manifest to be hard coded

408afe8

[ci skip]

fix: fixed index import for RPC handlers

f971650

[ci skip]

fix: renamed nodesChainDataGet to nodesClaimsGet

abfb115

[ci skip]

fix: fixed handler inputs

972ef96

[ci skip]

fix: parameterized process vertex timeout and other fixes

8f07a75

[ci skip]

fix: fixed network error name

4b16010

[ci skip]

fix: added comment to RPC response and result message types

43599f6

[ci skip]

fix: fixed some types

1ca6ea5

[ci skip]

fix: replaced NoData type with ObjectEmpty

d36a9ac

[ci skip]

fix: fixed mentions of GRPC

4f545d2

[ci skip]

fix: fixing usage of Host and Port types to be internal

c68c955

[ci skip]

fix: fixed keepAliveIntervalTime usage

c5da3d4

[ci skip]

docs: added doc-blocks describing RPC manifests

3427f5b

[ci skip]

fix: moved nodeIdFromMeta to agent domain

d72393e

[ci skip]

fix: updating config names

e3db77b

[ci skip]

fix: fixing up RPC handler imports

703ec5d

[ci skip]

Merge pull request #525 from MatrixAI/feature-agent_migration

fc82a3d

`js-quic` integration and Agent migration

MatrixAI-Bot merged commit fc82a3d into master Aug 3, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: merge staging to master #474

ci: merge staging to master #474

MatrixAI-Bot commented Oct 5, 2022 •

edited by CMCDragonkai

Loading

MatrixAI-Bot commented Oct 5, 2022

ghost commented Oct 5, 2022 •

edited by ghost

Loading

Legend

CMCDragonkai commented Oct 5, 2022

tegefaulkes commented Oct 6, 2022

tegefaulkes commented Oct 6, 2022

tegefaulkes commented Oct 6, 2022

CMCDragonkai commented Oct 6, 2022

tegefaulkes commented Oct 6, 2022 •

edited

Loading

MatrixAI-Bot commented Oct 7, 2022

MatrixAI-Bot commented Oct 18, 2022

MatrixAI-Bot commented Oct 25, 2022

MatrixAI-Bot commented Oct 26, 2022

MatrixAI-Bot commented Oct 27, 2022

MatrixAI-Bot commented Oct 27, 2022

CMCDragonkai commented Oct 31, 2022

MatrixAI-Bot commented Nov 8, 2022

tegefaulkes commented Nov 9, 2022

MatrixAI-Bot commented Nov 9, 2022

MatrixAI-Bot commented Nov 9, 2022

CMCDragonkai commented Nov 9, 2022

CMCDragonkai commented Nov 9, 2022

tegefaulkes commented Nov 9, 2022

MatrixAI-Bot commented Nov 9, 2022

CMCDragonkai commented Nov 9, 2022

CMCDragonkai commented Nov 9, 2022

CMCDragonkai commented Nov 9, 2022

MatrixAI-Bot commented Aug 1, 2023

CMCDragonkai commented Aug 1, 2023

MatrixAI-Bot commented Aug 3, 2023

MatrixAI-Bot commented Aug 3, 2023

ci: merge staging to master #474

ci: merge staging to master #474

Conversation

MatrixAI-Bot commented Oct 5, 2022 • edited by CMCDragonkai Loading

Tasks

MatrixAI-Bot commented Oct 5, 2022

ghost commented Oct 5, 2022 • edited by ghost Loading

Legend

CMCDragonkai commented Oct 5, 2022

tegefaulkes commented Oct 6, 2022

tegefaulkes commented Oct 6, 2022

tegefaulkes commented Oct 6, 2022

CMCDragonkai commented Oct 6, 2022

tegefaulkes commented Oct 6, 2022 • edited Loading

MatrixAI-Bot commented Oct 7, 2022

MatrixAI-Bot commented Oct 18, 2022

MatrixAI-Bot commented Oct 25, 2022

MatrixAI-Bot commented Oct 26, 2022

MatrixAI-Bot commented Oct 27, 2022

MatrixAI-Bot commented Oct 27, 2022

CMCDragonkai commented Oct 31, 2022

MatrixAI-Bot commented Nov 8, 2022

tegefaulkes commented Nov 9, 2022

MatrixAI-Bot commented Nov 9, 2022

MatrixAI-Bot commented Nov 9, 2022

CMCDragonkai commented Nov 9, 2022

CMCDragonkai commented Nov 9, 2022

tegefaulkes commented Nov 9, 2022

MatrixAI-Bot commented Nov 9, 2022

CMCDragonkai commented Nov 9, 2022

CMCDragonkai commented Nov 9, 2022

CMCDragonkai commented Nov 9, 2022

MatrixAI-Bot commented Aug 1, 2023

CMCDragonkai commented Aug 1, 2023

MatrixAI-Bot commented Aug 3, 2023

MatrixAI-Bot commented Aug 3, 2023

MatrixAI-Bot commented Oct 5, 2022 •

edited by CMCDragonkai

Loading

ghost commented Oct 5, 2022 •

edited by ghost

Loading

tegefaulkes commented Oct 6, 2022 •

edited

Loading