404 Error on consenter replace

[luca-simonetti]

What happened?

When I run the command

kubectl hlf channel consenter replace --config=ordservice.yaml \
    --orderer="ord-node1.ordnode" \
    --user=admin --channel=trading \
    --mspid=OrdererMSP --output=replace_orderers_consenter.pb

(from the documentation

I get the following error:

Error: Orderer Server Status Code: (404) NOT_FOUND. Description: error status from ordering service

which doesn't mean a thing

What did you expect to happen?

I don't actually know what to expect.

How can we reproduce it (as minimally and precisely as possible)?

We have a 2 org with 1 peer each and 1 ord node (as the base configuration is explained here...)

Anything else we need to know?

No response

Kubernetes version

v1.28.11-gke.1019001

[luca-simonetti]

Can this be due to some blocks missing in the folder? Some of them could be missing in our staging environment because we wanted to clear some space.

[gokulNeoSoft]

hi @luca-simonetti if you find solution please post the answer here, iam also facing exact error in this step..thanks

[luca-simonetti]

as soon as you run the renew commando on the ordnode the ordnode suddendly starts saying a 404 NOT_FOUND error. The documentation says after that an IMPORTANT note about the fact that also channels and consenters are supposed to receive the new certificates. Probably the documentation should be more clear and explicit about that.

[dviejokfs]

@luca-simonetti @gokulNeoSoft You need to make sure that the operator is upgraded to 1.10 and the peer/orderer versions are at least 2.4.9 before renewing the certificate

[gokulNeoSoft]

@dviejokfs is there way to increase expiration date when creating the network ?

[luca-simonetti]

the 404 is due to the fact that the consenter cannot find the channel after the certificates are expired.
To fix that you need to reissue certificates using fabric-ca-client:
https://hyperledger-fabric.readthedocs.io/en/latest/certs_management.html#renew-orderer-certificates

beware that you need to have fabric-ca to version >=1.5.1 or else it won't work
https://hyperledger-fabric.readthedocs.io/en/latest/certs_management.html#renewal-of-expired-certificates

so, summing up:

1. if your certificates are not expired, you need to have operator 1.10, peer 2.4.9 (there's no "official" support if you use external k8s-builder, you need to build your own custom image [1] ), orderer 2.4.9. Follow the guide in this repo [2], and after that remember to update certificates in the Follower CRD if they're in status FAILED (basically they are using the old TLS and need the new one)
2. if your certificates are expired, the renew part needs to be done manually. This means downloading current certificates (both TLS and client). You can see what certificates are expired in the peer/orderer log when you start the pod. When you download them, follow this to download a fresh public key. They're stored in secrets: ending in -idcert and -tls. To be sure you're doing the right thing just use openssl x509 -in mycert.crt -text --noout on your mycert.crt to check what is what. You need to refresh all certificates that have expired.

[1] https://github.com/kfsoftware/externalbuilder nothing fancy, really. Just clone this project create your own folder with your version. We find that images/fabric-peer/2.4.1/k8scc.yaml file is perfect for the config. We tried some different combination but with no luck, so we stick to the base one.
[2] https://hyperledger.github.io/bevel-operator-fabric/docs/operator-guide/renew-certificates/