Provider upgrade does not see old leases

[88plug]

bdl.computer (akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp) is showing multiple active leases before upgrade:

Found a lease with akash12rc45a8x0j4jdl5u2fcsytmtlqt25d4mff58vq
Found a lease with akash17ndnqvz4kx6aaf4l4tp6zsny0xkwqyl69egxw2
Found a lease with akash1xe342jdk25wugvrhlcsl65csr2z48etxp32mpt
Found a lease with akash1p83uq6sr8nqrv68cce7yjndrleswy7j55e3y29
Found a lease with akash1sum0e5ncl2d0kdeudy6w6x99ah5dy4u35x0wmf
Found a lease with akash1c9x3x96gdvkn3pq4u3qfnvv9gyzacvaymqqhpm
Found a lease with akash1w5uscl3jje6ydgq8gyw93qlmmgh9t3xjmamq7s
Found a lease with akash130est2p07yrfddsnmgyu4ejq6h5nadac0u7qhg
Found a lease with akash1m94c5jla8z0dm98ccn0gxhcehfke4pry3jqfum

Followed upgrade instructions and provider does not see any of the active leases before the upgrade.

I noticed that all the Manifest/ProviderHost/ProviderLeasedIP were not restored - they are blank after upgrade. Uninstalling old provider as per the instructions does not keep the manifests using the helm.sh/resource-policy=keep it appears they are deleted regardless when running helm -n akash-services uninstall akash-provider

I had to run kubectl apply -f manifests-backup.yaml and kubectl apply -f providerhosts-backup.yaml and pop the provider pod to see the old leases. But then even more trouble:

I[2022-11-27|03:16:34.578] using in cluster kube config                 cmp=provider
I[2022-11-27|03:16:34.690] using in cluster kube config                 cmp=provider
D[2022-11-27|03:16:34.695] service being found via autodetection        cmp=provider service=hostname-operator
I[2022-11-27|03:16:34.696] dns discovery success                        cmp=provider cmp=service-discovery-agent addrs="[{Target:akash-hostname-operator.akash-services.svc.cluster.local. Port:8188 Priority:0 Weight:100}]" portName=status service-name=akash-hostname-operator namespace=akash-services
D[2022-11-27|03:16:34.696] satisfying pending requests                  cmp=provider cmp=service-discovery-agent qty=1
I[2022-11-27|03:16:34.705] check result                                 cmp=provider operator=hostname status=200
I[2022-11-27|03:16:34.705] ready                                        cmp=provider cmp=waiter waitable="<*operatorclients.hostnameOperatorClient 0xc0006846c0>"
I[2022-11-27|03:16:34.705] all waitables ready                          cmp=provider cmp=waiter
I[2022-11-27|03:16:34.707] starting with existing reservations          module=provider-cluster cmp=provider cmp=service cmp=inventory-service qty=13
D[2022-11-27|03:16:34.710] found existing hostname                      module=provider-cluster cmp=provider cmp=service hostname=62jl13jhf5cj3anie7gs84b958.ingress.bdl.computer id=akash12rc45a8x0j4jdl5u2fcsytmtlqt25d4mff58vq/8371090/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp
D[2022-11-27|03:16:34.710] found existing hostname                      module=provider-cluster cmp=provider cmp=service hostname=blocklessghost.akash.pro id=akash1w5uscl3jje6ydgq8gyw93qlmmgh9t3xjmamq7s/8436789/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp
D[2022-11-27|03:16:34.710] found existing hostname                      module=provider-cluster cmp=provider cmp=service hostname=hlj6e8ntcdb4l7avv9aide98a4.ingress.bdl.computer id=akash17ndnqvz4kx6aaf4l4tp6zsny0xkwqyl69egxw2/8510193/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp
D[2022-11-27|03:16:34.710] found existing hostname                      module=provider-cluster cmp=provider cmp=service hostname=jsrltmfl8teo9a1t01rcqoaf50.ingress.bdl.computer id=akash18u6zew2sg9kfwlk4rurh3m8vxlylp9jmzr25ah/8525559/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp
D[2022-11-27|03:16:34.710] found existing hostname                      module=provider-cluster cmp=provider cmp=service hostname=n8ps10lhihckhbrqnspm3vqm68.ingress.bdl.computer id=akash1w5uscl3jje6ydgq8gyw93qlmmgh9t3xjmamq7s/8436789/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp
D[2022-11-27|03:16:34.710] found existing hostname                      module=provider-cluster cmp=provider cmp=service hostname=op5mo9gn5dbprfuigng0gqbv18.ingress.bdl.computer id=akash1xe342jdk25wugvrhlcsl65csr2z48etxp32mpt/8286524/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp
D[2022-11-27|03:16:34.710] found existing hostname                      module=provider-cluster cmp=provider cmp=service hostname=re6sr3kt11akd4ake9tnpne19c.ingress.bdl.computer id=akash1sum0e5ncl2d0kdeudy6w6x99ah5dy4u35x0wmf/8523185/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp
E[2022-11-27|03:16:34.752] node in poor condition                       cmp=provider client=kube node=k3s-wildponyexpress condition=MemoryPressure status=Unknown
E[2022-11-27|03:16:34.752] node in poor condition                       cmp=provider client=kube node=k3s-wildponyexpress condition=DiskPressure status=Unknown
E[2022-11-27|03:16:34.752] node in poor condition                       cmp=provider client=kube node=k3s-wildponyexpress condition=PIDPressure status=Unknown
D[2022-11-27|03:16:34.859] inventory ready                              module=provider-cluster cmp=provider cmp=service cmp=inventory-service
D[2022-11-27|03:16:34.859] cluster resources                            module=provider-cluster cmp=provider cmp=service cmp=inventory-service dump="{\"nodes\":[{\"name\":\"k3s-rs2\",\"allocatable\":{\"cpu\":32000,\"memory\":50618888192,\"storage_ephemeral\":192163573200},\"available\":{\"cpu\":30890,\"memory\":49962479616,\"storage_ephemeral\":191626702288}},{\"name\":\"akash-rpc-node\",\"allocatable\":{\"cpu\":24000,\"memory\":16766521344,\"storage_ephemeral\":264429877246},\"available\":{\"cpu\":23690,\"memory\":16500183040,\"storage_ephemeral\":264429877246}},{\"name\":\"k3s-c27\",\"allocatable\":{\"cpu\":16000,\"memory\":42168496128,\"storage_ephemeral\":14499801896},\"available\":{\"cpu\":15890,\"memory\":42048958464,\"storage_ephemeral\":14499801896}},{\"name\":\"k3s-ct21\",\"allocatable\":{\"cpu\":40000,\"memory\":101349371904,\"storage_ephemeral\":276607029646},\"available\":{\"cpu\":37890,\"memory\":94787383296,\"storage_ephemeral\":276607029646}},{\"name\":\"k3s-ct22\",\"allocatable\":{\"cpu\":40000,\"memory\":50615308288,\"storage_ephemeral\":14499801896},\"available\":{\"cpu\":38890,\"memory\":49958899712,\"storage_ephemeral\":12926937896}},{\"name\":\"k3s-rs4\",\"allocatable\":{\"cpu\":32000,\"memory\":50618884096,\"storage_ephemeral\":14499801896},\"available\":{\"cpu\":31890,\"memory\":50499346432,\"storage_ephemeral\":14499801896}},{\"name\":\"k3s-ct30\",\"allocatable\":{\"cpu\":12000,\"memory\":8334454784,\"storage_ephemeral\":14499801896},\"available\":{\"cpu\":11890,\"memory\":8214917120,\"storage_ephemeral\":14499801896}},{\"name\":\"k3s-node12\",\"allocatable\":{\"cpu\":128000,\"memory\":101226201088,\"storage_ephemeral\":4031878577677},\"available\":{\"cpu\":127790,\"memory\":100972445696,\"storage_ephemeral\":4031878577677}},{\"name\":\"k3s-rs3\",\"allocatable\":{\"cpu\":32000,\"memory\":18859696128,\"storage_ephemeral\":14499801896},\"available\":{\"cpu\":30890,\"memory\":18203287552,\"storage_ephemeral\":13426060072}},{\"name\":\"k3s-wildgamer\",\"allocatable\":{\"cpu\":32000,\"memory\":10396827648,\"storage_ephemeral\":14499801896},\"available\":{\"cpu\":30890,\"memory\":9740419072,\"storage_ephemeral\":13962930984}},{\"name\":\"k3s-chia1\",\"allocatable\":{\"cpu\":12000,\"memory\":2038071296,\"storage_ephemeral\":14499801896},\"available\":{\"cpu\":11880,\"memory\":1884979200,\"storage_ephemeral\":14499801896}},{\"name\":\"k3s-wildfan\",\"allocatable\":{\"cpu\":32000,\"memory\":29429399552,\"storage_ephemeral\":14499801896},\"available\":{\"cpu\":31890,\"memory\":29309861888,\"storage_ephemeral\":14499801896}},{\"name\":\"k3s-wilditx\",\"allocatable\":{\"cpu\":32000,\"memory\":10429444096,\"storage_ephemeral\":421090118326},\"available\":{\"cpu\":31890,\"memory\":10309906432,\"storage_ephemeral\":421090118326}},{\"name\":\"k3s-wildpony\",\"allocatable\":{\"cpu\":32000,\"memory\":33658916864,\"storage_ephemeral\":14499801896},\"available\":{\"cpu\":31890,\"memory\":33539379200,\"storage_ephemeral\":14499801896}},{\"name\":\"k3s-wildprox\",\"allocatable\":{\"cpu\":12000,\"memory\":8336101376,\"storage_ephemeral\":14499801896},\"available\":{\"cpu\":11890,\"memory\":8216563712,\"storage_ephemeral\":14499801896}},{\"name\":\"k3s-ct10\",\"allocatable\":{\"cpu\":12000,\"memory\":29435994112,\"storage_ephemeral\":14499801896},\"available\":{\"cpu\":11890,\"memory\":29316456448,\"storage_ephemeral\":14499801896}},{\"name\":\"k3s-ct26\",\"allocatable\":{\"cpu\":24000,\"memory\":8310390784,\"storage_ephemeral\":14499801896},\"available\":{\"cpu\":23890,\"memory\":8190853120,\"storage_ephemeral\":14499801896}},{\"name\":\"k3s-ct28\",\"allocatable\":{\"cpu\":12000,\"memory\":50625253376,\"storage_ephemeral\":14499801896},\"available\":{\"cpu\":11890,\"memory\":50505715712,\"storage_ephemeral\":14499801896}},{\"name\":\"k3s-ct29\",\"allocatable\":{\"cpu\":24000,\"memory\":52734746624,\"storage_ephemeral\":14499801896},\"available\":{\"cpu\":23890,\"memory\":52615208960,\"storage_ephemeral\":14499801896}},{\"name\":\"k3s-wildtaichi\",\"allocatable\":{\"cpu\":32000,\"memory\":29397020672,\"storage_ephemeral\":14499801896},\"available\":{\"cpu\":31890,\"memory\":29277483008,\"storage_ephemeral\":14499801896}},{\"name\":\"k3s-chia4\",\"allocatable\":{\"cpu\":24000,\"memory\":8342425600,\"storage_ephemeral\":14499801896},\"available\":{\"cpu\":23890,\"memory\":8222887936,\"storage_ephemeral\":14499801896}},{\"name\":\"k3s-ct20\",\"allocatable\":{\"cpu\":40000,\"memory\":101349376000,\"storage_ephemeral\":276607029646},\"available\":{\"cpu\":35890,\"memory\":92639903744,\"storage_ephemeral\":169232847246}},{\"name\":\"k3s-ct23\",\"allocatable\":{\"cpu\":40000,\"memory\":50615308288,\"storage_ephemeral\":14499801896},\"available\":{\"cpu\":39390,\"memory\":50361552896,\"storage_ephemeral\":14499801896}},{\"name\":\"k3s-node11\",\"allocatable\":{\"cpu\":128000,\"memory\":101223075840,\"storage_ephemeral\":4095691946254},\"available\":{\"cpu\":123790,\"memory\":96271699968,\"storage_ephemeral\":4074217109774}},{\"name\":\"k3s-chia2\",\"allocatable\":{\"cpu\":24000,\"memory\":4112039936,\"storage_ephemeral\":14499801896},\"available\":{\"cpu\":23890,\"memory\":3992502272,\"storage_ephemeral\":14499801896}},{\"name\":\"k3s-ct25\",\"allocatable\":{\"cpu\":24000,\"memory\":20993912832,\"storage_ephemeral\":14499801896},\"available\":{\"cpu\":22890,\"memory\":20337504256,\"storage_ephemeral\":13426060072}},{\"name\":\"k3s-rs1\",\"allocatable\":{\"cpu\":32000,\"memory\":8340492288,\"storage_ephemeral\":421090118326},\"available\":{\"cpu\":31890,\"memory\":8220954624,\"storage_ephemeral\":421090118326}},{\"name\":\"k3s-wildatxm\",\"allocatable\":{\"cpu\":24000,\"memory\":29431472128,\"storage_ephemeral\":195002651086},\"available\":{\"cpu\":22890,\"memory\":28775063552,\"storage_ephemeral\":193928909262}},{\"name\":\"k3s-wildbro\",\"allocatable\":{\"cpu\":32000,\"memory\":16738611200,\"storage_ephemeral\":14499801896},\"available\":{\"cpu\":31890,\"memory\":16619073536,\"storage_ephemeral\":14499801896}},{\"name\":\"k3s-wildsfo\",\"allocatable\":{\"cpu\":36000,\"memory\":67394703360,\"storage_ephemeral\":195002651086},\"available\":{\"cpu\":34890,\"memory\":65127682048,\"storage_ephemeral\":186412716494}}],\"total_allocatable\":{\"cpu\":1016000,\"memory\":1113891405824,\"storage_ephemeral\":10659559610413},\"total_available\":{\"cpu\":994790,\"memory\":1084623552512,\"storage_ephemeral\":10516252825645}}\n"
I[2022-11-27|03:16:35.048] hostnames withheld                           module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1m94c5jla8z0dm98ccn0gxhcehfke4pry3jqfum/8449748/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=argoPlacement cnt=0
I[2022-11-27|03:16:35.055] hostnames withheld                           module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1p83uq6sr8nqrv68cce7yjndrleswy7j55e3y29/8418358/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=argoPlacement cnt=0
E[2022-11-27|03:16:35.056] lease not active, not deploying              module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1fpaarnccrmrz5vfwqyyjd70x4uq7l5vnasay5r/8381375/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=westcoast
E[2022-11-27|03:16:35.057] execution error                              module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1fpaarnccrmrz5vfwqyyjd70x4uq7l5vnasay5r/8381375/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=westcoast state=deploy-active err="inactive Lease: akash1fpaarnccrmrz5vfwqyyjd70x4uq7l5vnasay5r/8381375/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp"
D[2022-11-27|03:16:35.058] purged ips                                   module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1fpaarnccrmrz5vfwqyyjd70x4uq7l5vnasay5r/8381375/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=westcoast
D[2022-11-27|03:16:35.059] purged hostnames                             module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1fpaarnccrmrz5vfwqyyjd70x4uq7l5vnasay5r/8381375/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=westcoast
E[2022-11-27|03:16:35.061] lease teardown failed                        module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1fpaarnccrmrz5vfwqyyjd70x4uq7l5vnasay5r/8381375/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=westcoast err="namespaces \"u4fctraths8fvqb2l74h3ercdhvfbktrktsb9i7gabe92\" not found"
I[2022-11-27|03:16:35.068] hostnames withheld                           module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1sum0e5ncl2d0kdeudy6w6x99ah5dy4u35x0wmf/8523185/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=akash cnt=0
I[2022-11-27|03:16:35.076] hostnames withheld                           module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash12rc45a8x0j4jdl5u2fcsytmtlqt25d4mff58vq/8371090/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=akash cnt=0
D[2022-11-27|03:16:35.077] deploy complete                              module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1m94c5jla8z0dm98ccn0gxhcehfke4pry3jqfum/8449748/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=argoPlacement
D[2022-11-27|03:16:35.081] deploy complete                              module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1p83uq6sr8nqrv68cce7yjndrleswy7j55e3y29/8418358/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=argoPlacement
I[2022-11-27|03:16:35.084] hostnames withheld                           module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash130est2p07yrfddsnmgyu4ejq6h5nadac0u7qhg/8449829/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=argoPlacement cnt=0
E[2022-11-27|03:16:35.088] node in poor condition                       cmp=provider client=kube node=k3s-wildponyexpress condition=MemoryPressure status=Unknown
E[2022-11-27|03:16:35.088] node in poor condition                       cmp=provider client=kube node=k3s-wildponyexpress condition=DiskPressure status=Unknown
E[2022-11-27|03:16:35.088] node in poor condition                       cmp=provider client=kube node=k3s-wildponyexpress condition=PIDPressure status=Unknown
I[2022-11-27|03:16:35.100] hostnames withheld                           module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1c9x3x96gdvkn3pq4u3qfnvv9gyzacvaymqqhpm/8449775/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=argoPlacement cnt=0
I[2022-11-27|03:16:35.102] hostnames withheld                           module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash17ndnqvz4kx6aaf4l4tp6zsny0xkwqyl69egxw2/8510193/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=dcloud cnt=0
I[2022-11-27|03:16:35.102] declaring hostname                           cmp=provider client=kube lease=akash1sum0e5ncl2d0kdeudy6w6x99ah5dy4u35x0wmf/8523185/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp service-name=app external-port=80 host=re6sr3kt11akd4ake9tnpne19c.ingress.bdl.computer
I[2022-11-27|03:16:35.104] declaring hostname                           cmp=provider client=kube lease=akash12rc45a8x0j4jdl5u2fcsytmtlqt25d4mff58vq/8371090/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp service-name=tronwallet external-port=80 host=62jl13jhf5cj3anie7gs84b958.ingress.bdl.computer
D[2022-11-27|03:16:35.105] deploy complete                              module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1sum0e5ncl2d0kdeudy6w6x99ah5dy4u35x0wmf/8523185/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=akash
D[2022-11-27|03:16:35.106] deploy complete                              module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash12rc45a8x0j4jdl5u2fcsytmtlqt25d4mff58vq/8371090/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=akash
I[2022-11-27|03:16:35.109] hostnames withheld                           module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1xe342jdk25wugvrhlcsl65csr2z48etxp32mpt/8286524/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=akash cnt=0
E[2022-11-27|03:16:35.111] lease not active, not deploying              module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash18u6zew2sg9kfwlk4rurh3m8vxlylp9jmzr25ah/8525559/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=akash
E[2022-11-27|03:16:35.111] execution error                              module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash18u6zew2sg9kfwlk4rurh3m8vxlylp9jmzr25ah/8525559/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=akash state=deploy-active err="inactive Lease: akash18u6zew2sg9kfwlk4rurh3m8vxlylp9jmzr25ah/8525559/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp"
E[2022-11-27|03:16:35.146] lease not active, not deploying              module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash18u6zew2sg9kfwlk4rurh3m8vxlylp9jmzr25ah/8476291/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=akash
E[2022-11-27|03:16:35.146] execution error                              module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash18u6zew2sg9kfwlk4rurh3m8vxlylp9jmzr25ah/8476291/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=akash state=deploy-active err="inactive Lease: akash18u6zew2sg9kfwlk4rurh3m8vxlylp9jmzr25ah/8476291/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp"
E[2022-11-27|03:16:35.146] lease not active, not deploying              module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1kccah0qqt0fsmve74900ha7en44c3qwsv72att/8042485/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=pkt-miner
D[2022-11-27|03:16:35.146] purged ips                                   module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash18u6zew2sg9kfwlk4rurh3m8vxlylp9jmzr25ah/8525559/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=akash
E[2022-11-27|03:16:35.146] execution error                              module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1kccah0qqt0fsmve74900ha7en44c3qwsv72att/8042485/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=pkt-miner state=deploy-active err="inactive Lease: akash1kccah0qqt0fsmve74900ha7en44c3qwsv72att/8042485/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp"
D[2022-11-27|03:16:35.146] purged hostnames                             module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash18u6zew2sg9kfwlk4rurh3m8vxlylp9jmzr25ah/8525559/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=akash
D[2022-11-27|03:16:35.148] purged ips                                   module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash18u6zew2sg9kfwlk4rurh3m8vxlylp9jmzr25ah/8476291/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=akash
I[2022-11-27|03:16:35.148] hostnames withheld                           module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1w5uscl3jje6ydgq8gyw93qlmmgh9t3xjmamq7s/8436789/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=akash cnt=0
E[2022-11-27|03:16:35.149] lease teardown failed                        module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash18u6zew2sg9kfwlk4rurh3m8vxlylp9jmzr25ah/8525559/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=akash err="namespaces \"jeaesjdj77ipj44ujmmgqdv36h37oofbs99v48eb7be3u\" not found"
D[2022-11-27|03:16:35.150] purged ips                                   module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1kccah0qqt0fsmve74900ha7en44c3qwsv72att/8042485/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=pkt-miner
D[2022-11-27|03:16:35.151] deploy complete                              module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash130est2p07yrfddsnmgyu4ejq6h5nadac0u7qhg/8449829/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=argoPlacement
D[2022-11-27|03:16:35.151] purged hostnames                             module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1kccah0qqt0fsmve74900ha7en44c3qwsv72att/8042485/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=pkt-miner
D[2022-11-27|03:16:35.151] purged hostnames                             module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash18u6zew2sg9kfwlk4rurh3m8vxlylp9jmzr25ah/8476291/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=akash
E[2022-11-27|03:16:35.151] lease teardown failed                        module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash18u6zew2sg9kfwlk4rurh3m8vxlylp9jmzr25ah/8476291/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=akash err="namespaces \"6r3h1ci17kj2d2n48f4nftkdu62o7ji1nh98d61jek3v4\" not found"
E[2022-11-27|03:16:35.152] lease teardown failed                        module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1kccah0qqt0fsmve74900ha7en44c3qwsv72att/8042485/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=pkt-miner err="namespaces \"jo9uinc9s7q0l8r1ca8d12nolg92m9r7dm4mh7e79v7pq\" not found"
E[2022-11-27|03:16:35.247] teardown lease: unable to delete manifest    cmp=provider client=kube ns=u4fctraths8fvqb2l74h3ercdhvfbktrktsb9i7gabe92 error="manifests.akash.network \"u4fctraths8fvqb2l74h3ercdhvfbktrktsb9i7gabe92\" not found"
E[2022-11-27|03:16:35.247] lease teardown failed                        module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1fpaarnccrmrz5vfwqyyjd70x4uq7l5vnasay5r/8381375/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=westcoast err="namespaces \"u4fctraths8fvqb2l74h3ercdhvfbktrktsb9i7gabe92\" not found"
D[2022-11-27|03:16:35.250] deploy complete                              module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1c9x3x96gdvkn3pq4u3qfnvv9gyzacvaymqqhpm/8449775/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=argoPlacement
E[2022-11-27|03:16:35.251] teardown lease: unable to delete manifest    cmp=provider client=kube ns=jeaesjdj77ipj44ujmmgqdv36h37oofbs99v48eb7be3u error="manifests.akash.network \"jeaesjdj77ipj44ujmmgqdv36h37oofbs99v48eb7be3u\" not found"
E[2022-11-27|03:16:35.251] lease teardown failed                        module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash18u6zew2sg9kfwlk4rurh3m8vxlylp9jmzr25ah/8525559/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=akash err="namespaces \"jeaesjdj77ipj44ujmmgqdv36h37oofbs99v48eb7be3u\" not found"
E[2022-11-27|03:16:35.253] teardown lease: unable to delete manifest    cmp=provider client=kube ns=6r3h1ci17kj2d2n48f4nftkdu62o7ji1nh98d61jek3v4 error="manifests.akash.network \"6r3h1ci17kj2d2n48f4nftkdu62o7ji1nh98d61jek3v4\" not found"
E[2022-11-27|03:16:35.253] lease teardown failed                        module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash18u6zew2sg9kfwlk4rurh3m8vxlylp9jmzr25ah/8476291/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=akash err="namespaces \"6r3h1ci17kj2d2n48f4nftkdu62o7ji1nh98d61jek3v4\" not found"
I[2022-11-27|03:16:35.254] declaring hostname                           cmp=provider client=kube lease=akash17ndnqvz4kx6aaf4l4tp6zsny0xkwqyl69egxw2/8510193/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp service-name=node external-port=80 host=hlj6e8ntcdb4l7avv9aide98a4.ingress.bdl.computer
E[2022-11-27|03:16:35.254] teardown lease: unable to delete manifest    cmp=provider client=kube ns=jo9uinc9s7q0l8r1ca8d12nolg92m9r7dm4mh7e79v7pq error="manifests.akash.network \"jo9uinc9s7q0l8r1ca8d12nolg92m9r7dm4mh7e79v7pq\" not found"
E[2022-11-27|03:16:35.254] lease teardown failed                        module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1kccah0qqt0fsmve74900ha7en44c3qwsv72att/8042485/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=pkt-miner err="namespaces \"jo9uinc9s7q0l8r1ca8d12nolg92m9r7dm4mh7e79v7pq\" not found"
D[2022-11-27|03:16:35.256] deploy complete                              module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash17ndnqvz4kx6aaf4l4tp6zsny0xkwqyl69egxw2/8510193/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=dcloud
I[2022-11-27|03:16:35.259] declaring hostname                           cmp=provider client=kube lease=akash1xe342jdk25wugvrhlcsl65csr2z48etxp32mpt/8286524/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp service-name=tetris external-port=80 host=op5mo9gn5dbprfuigng0gqbv18.ingress.bdl.computer
I[2022-11-27|03:16:35.260] declaring hostname                           cmp=provider client=kube lease=akash1w5uscl3jje6ydgq8gyw93qlmmgh9t3xjmamq7s/8436789/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp service-name=ghost external-port=80 host=n8ps10lhihckhbrqnspm3vqm68.ingress.bdl.computer
D[2022-11-27|03:16:35.261] deploy complete                              module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1xe342jdk25wugvrhlcsl65csr2z48etxp32mpt/8286524/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=akash
I[2022-11-27|03:16:35.263] declaring hostname                           cmp=provider client=kube lease=akash1w5uscl3jje6ydgq8gyw93qlmmgh9t3xjmamq7s/8436789/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp service-name=ghost external-port=80 host=blocklessghost.akash.pro
D[2022-11-27|03:16:35.265] deploy complete                              module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1w5uscl3jje6ydgq8gyw93qlmmgh9t3xjmamq7s/8436789/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=akash
E[2022-11-27|03:16:35.277] adjust inventory for pending reservation     module=provider-cluster cmp=provider cmp=service cmp=inventory-service error="insufficient capacity"
E[2022-11-27|03:16:35.451] teardown lease: unable to delete manifest    cmp=provider client=kube ns=u4fctraths8fvqb2l74h3ercdhvfbktrktsb9i7gabe92 error="manifests.akash.network \"u4fctraths8fvqb2l74h3ercdhvfbktrktsb9i7gabe92\" not found"
E[2022-11-27|03:16:35.451] lease teardown failed                        module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1fpaarnccrmrz5vfwqyyjd70x4uq7l5vnasay5r/8381375/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=westcoast err="namespaces \"u4fctraths8fvqb2l74h3ercdhvfbktrktsb9i7gabe92\" not found"
E[2022-11-27|03:16:35.454] teardown lease: unable to delete manifest    cmp=provider client=kube ns=jeaesjdj77ipj44ujmmgqdv36h37oofbs99v48eb7be3u error="manifests.akash.network \"jeaesjdj77ipj44ujmmgqdv36h37oofbs99v48eb7be3u\" not found"
E[2022-11-27|03:16:35.454] lease teardown failed                        module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash18u6zew2sg9kfwlk4rurh3m8vxlylp9jmzr25ah/8525559/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=akash err="namespaces \"jeaesjdj77ipj44ujmmgqdv36h37oofbs99v48eb7be3u\" not found"
E[2022-11-27|03:16:35.456] teardown lease: unable to delete manifest    cmp=provider client=kube ns=6r3h1ci17kj2d2n48f4nftkdu62o7ji1nh98d61jek3v4 error="manifests.akash.network \"6r3h1ci17kj2d2n48f4nftkdu62o7ji1nh98d61jek3v4\" not found"
E[2022-11-27|03:16:35.456] lease teardown failed                        module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash18u6zew2sg9kfwlk4rurh3m8vxlylp9jmzr25ah/8476291/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=akash err="namespaces \"6r3h1ci17kj2d2n48f4nftkdu62o7ji1nh98d61jek3v4\" not found"
E[2022-11-27|03:16:35.456] teardown lease: unable to delete manifest    cmp=provider client=kube ns=jo9uinc9s7q0l8r1ca8d12nolg92m9r7dm4mh7e79v7pq error="manifests.akash.network \"jo9uinc9s7q0l8r1ca8d12nolg92m9r7dm4mh7e79v7pq\" not found"
E[2022-11-27|03:16:35.456] lease teardown failed                        module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1kccah0qqt0fsmve74900ha7en44c3qwsv72att/8042485/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=pkt-miner err="namespaces \"jo9uinc9s7q0l8r1ca8d12nolg92m9r7dm4mh7e79v7pq\" not found"
I[2022-11-27|03:16:35.746] found orders                                 module=bidengine-service cmp=provider count=11
D[2022-11-27|03:16:35.746] creating catchup order                       module=bidengine-service cmp=provider order=order/akash1qqzwc5d7hynl67nsmn9jukvwqp3vzdl6j2t7lk/2528509/1/1
D[2022-11-27|03:16:35.746] creating catchup order                       module=bidengine-service cmp=provider order=order/akash1qqmhmrx9sd75kkfyyg9uncm326l8kclc5tckcu/5545245/1/1
D[2022-11-27|03:16:35.746] creating catchup order                       module=bidengine-service cmp=provider order=order/akash1qpygf7nphng80k6w3j9ljqy3ksnkk2gdcmwe55/4185046/1/1
D[2022-11-27|03:16:35.746] creating catchup order                       module=bidengine-service cmp=provider order=order/akash1qtj2uhj324wxg0083j5fkld7nv54dg8auksut6/2252060/1/1
D[2022-11-27|03:16:35.746] creating catchup order                       module=bidengine-service cmp=provider order=order/akash1q0m0kz83qwpuc5ss39y8sf25mq85a43ffjfd3v/8654676/1/1
D[2022-11-27|03:16:35.746] creating catchup order                       module=bidengine-service cmp=provider order=order/akash1qsrzvymyw06z4egr9w9w84jtxcy0na3rrtexc4/6445173/1/1
D[2022-11-27|03:16:35.746] creating catchup order                       module=bidengine-service cmp=provider order=order/akash1qk5pw9g348gl76e2m2esuh0j4wnvqjus9ts3c6/2249866/1/1
D[2022-11-27|03:16:35.746] creating catchup order                       module=bidengine-service cmp=provider order=order/akash1qch2ewwv4mm54ucn9z8y7ym43svg9gzk4g45pk/8274414/1/1
D[2022-11-27|03:16:35.746] creating catchup order                       module=bidengine-service cmp=provider order=order/akash1q6y7dnh2mlluh5w8v03asxqthe3m354x289asr/2/1/2
D[2022-11-27|03:16:35.746] creating catchup order                       module=bidengine-service cmp=provider order=order/akash1q6y7dnh2mlluh5w8v03asxqthe3m354x289asr/3/1/1
D[2022-11-27|03:16:35.747] creating catchup order                       module=bidengine-service cmp=provider order=order/akash1q6y7dnh2mlluh5w8v03asxqthe3m354x289asr/4/1/1
I[2022-11-27|03:16:35.846] found existing bid                           module=bidengine-order cmp=provider
I[2022-11-27|03:16:35.846] found existing bid                           module=bidengine-order cmp=provider
E[2022-11-27|03:16:35.846] bid in unexpected state                      module=bidengine-order cmp=provider bid-state=closed
I[2022-11-27|03:16:35.846] fetched provider attributes                  module=bidengine-service cmp=provider provider=akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp
E[2022-11-27|03:16:35.846] bid in unexpected state                      module=bidengine-order cmp=provider bid-state=closed
I[2022-11-27|03:16:35.846] group fetched                                module=bidengine-order cmp=provider order=akash1qqzwc5d7hynl67nsmn9jukvwqp3vzdl6j2t7lk/2528509/1/1
I[2022-11-27|03:16:35.846] shutting down                                module=bidengine-order cmp=provider order=akash1q6y7dnh2mlluh5w8v03asxqthe3m354x289asr/2/1/2
I[2022-11-27|03:16:35.846] found existing bid                           module=bidengine-order cmp=provider
I[2022-11-27|03:16:35.846] shutting down                                module=bidengine-order cmp=provider order=akash1q0m0kz83qwpuc5ss39y8sf25mq85a43ffjfd3v/8654676/1/1
E[2022-11-27|03:16:35.846] bid in unexpected state                      module=bidengine-order cmp=provider bid-state=closed
I[2022-11-27|03:16:35.846] found existing bid                           module=bidengine-order cmp=provider
I[2022-11-27|03:16:35.846] group fetched                                module=bidengine-order cmp=provider order=akash1q6y7dnh2mlluh5w8v03asxqthe3m354x289asr/3/1/1
I[2022-11-27|03:16:35.846] shutting down                                module=bidengine-order cmp=provider order=akash1qqmhmrx9sd75kkfyyg9uncm326l8kclc5tckcu/5545245/1/1
E[2022-11-27|03:16:35.846] bid in unexpected state                      module=bidengine-order cmp=provider bid-state=closed
I[2022-11-27|03:16:35.846] shutting down                                module=bidengine-order cmp=provider order=akash1qk5pw9g348gl76e2m2esuh0j4wnvqjus9ts3c6/2249866/1/1
I[2022-11-27|03:16:35.846] fetching provider auditor attributes         module=bidengine-service cmp=provider auditor=akash1qqzwc5d7hynl67nsmn9jukvwqp3vzdl6j2t7lk provider=akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp
D[2022-11-27|03:16:35.846] unable to fulfill: incompatible provider attributes module=bidengine-order cmp=provider order=akash1q6y7dnh2mlluh5w8v03asxqthe3m354x289asr/3/1/1
D[2022-11-27|03:16:35.846] declined to bid                              module=bidengine-order cmp=provider order=akash1q6y7dnh2mlluh5w8v03asxqthe3m354x289asr/3/1/1
I[2022-11-27|03:16:35.846] shutting down                                module=bidengine-order cmp=provider order=akash1q6y7dnh2mlluh5w8v03asxqthe3m354x289asr/3/1/1
I[2022-11-27|03:16:35.849] found existing bid                           module=bidengine-order cmp=provider
E[2022-11-27|03:16:35.849] bid in unexpected state                      module=bidengine-order cmp=provider bid-state=closed
I[2022-11-27|03:16:35.849] shutting down                                module=bidengine-order cmp=provider order=akash1qpygf7nphng80k6w3j9ljqy3ksnkk2gdcmwe55/4185046/1/1
I[2022-11-27|03:16:35.851] found existing bid                           module=bidengine-order cmp=provider
E[2022-11-27|03:16:35.851] bid in unexpected state                      module=bidengine-order cmp=provider bid-state=closed
I[2022-11-27|03:16:35.851] shutting down                                module=bidengine-order cmp=provider order=akash1qch2ewwv4mm54ucn9z8y7ym43svg9gzk4g45pk/8274414/1/1
I[2022-11-27|03:16:35.854] group fetched                                module=bidengine-order cmp=provider order=akash1q6y7dnh2mlluh5w8v03asxqthe3m354x289asr/4/1/1
D[2022-11-27|03:16:35.854] unable to fulfill: incompatible provider attributes module=bidengine-order cmp=provider order=akash1q6y7dnh2mlluh5w8v03asxqthe3m354x289asr/4/1/1
D[2022-11-27|03:16:35.854] declined to bid                              module=bidengine-order cmp=provider order=akash1q6y7dnh2mlluh5w8v03asxqthe3m354x289asr/4/1/1
I[2022-11-27|03:16:35.854] shutting down                                module=bidengine-order cmp=provider order=akash1q6y7dnh2mlluh5w8v03asxqthe3m354x289asr/4/1/1
E[2022-11-27|03:16:35.856] teardown lease: unable to delete manifest    cmp=provider client=kube ns=u4fctraths8fvqb2l74h3ercdhvfbktrktsb9i7gabe92 error="manifests.akash.network \"u4fctraths8fvqb2l74h3ercdhvfbktrktsb9i7gabe92\" not found"
E[2022-11-27|03:16:35.856] lease teardown failed                        module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1fpaarnccrmrz5vfwqyyjd70x4uq7l5vnasay5r/8381375/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=westcoast err="namespaces \"u4fctraths8fvqb2l74h3ercdhvfbktrktsb9i7gabe92\" not found"
I[2022-11-27|03:16:35.856] found existing bid                           module=bidengine-order cmp=provider
E[2022-11-27|03:16:35.856] bid in unexpected state                      module=bidengine-order cmp=provider bid-state=closed
I[2022-11-27|03:16:35.856] shutting down                                module=bidengine-order cmp=provider order=akash1qtj2uhj324wxg0083j5fkld7nv54dg8auksut6/2252060/1/1
E[2022-11-27|03:16:35.857] teardown lease: unable to delete manifest    cmp=provider client=kube ns=jeaesjdj77ipj44ujmmgqdv36h37oofbs99v48eb7be3u error="manifests.akash.network \"jeaesjdj77ipj44ujmmgqdv36h37oofbs99v48eb7be3u\" not found"
E[2022-11-27|03:16:35.857] lease teardown failed                        module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash18u6zew2sg9kfwlk4rurh3m8vxlylp9jmzr25ah/8525559/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=akash err="namespaces \"jeaesjdj77ipj44ujmmgqdv36h37oofbs99v48eb7be3u\" not found"
E[2022-11-27|03:16:35.858] teardown lease: unable to delete manifest    cmp=provider client=kube ns=6r3h1ci17kj2d2n48f4nftkdu62o7ji1nh98d61jek3v4 error="manifests.akash.network \"6r3h1ci17kj2d2n48f4nftkdu62o7ji1nh98d61jek3v4\" not found"
E[2022-11-27|03:16:35.858] lease teardown failed                        module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash18u6zew2sg9kfwlk4rurh3m8vxlylp9jmzr25ah/8476291/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=akash err="namespaces \"6r3h1ci17kj2d2n48f4nftkdu62o7ji1nh98d61jek3v4\" not found"
E[2022-11-27|03:16:35.859] teardown lease: unable to delete manifest    cmp=provider client=kube ns=jo9uinc9s7q0l8r1ca8d12nolg92m9r7dm4mh7e79v7pq error="manifests.akash.network \"jo9uinc9s7q0l8r1ca8d12nolg92m9r7dm4mh7e79v7pq\" not found"
E[2022-11-27|03:16:35.859] lease teardown failed                        module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1kccah0qqt0fsmve74900ha7en44c3qwsv72att/8042485/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=pkt-miner err="namespaces \"jo9uinc9s7q0l8r1ca8d12nolg92m9r7dm4mh7e79v7pq\" not found"
I[2022-11-27|03:16:35.859] found existing bid                           module=bidengine-order cmp=provider
E[2022-11-27|03:16:35.859] bid in unexpected state                      module=bidengine-order cmp=provider bid-state=closed
I[2022-11-27|03:16:35.859] shutting down                                module=bidengine-order cmp=provider order=akash1qsrzvymyw06z4egr9w9w84jtxcy0na3rrtexc4/6445173/1/1
D[2022-11-27|03:16:35.861] attribute signature requirements not met     module=bidengine-order cmp=provider order=akash1qqzwc5d7hynl67nsmn9jukvwqp3vzdl6j2t7lk/2528509/1/1
D[2022-11-27|03:16:35.861] declined to bid                              module=bidengine-order cmp=provider order=akash1qqzwc5d7hynl67nsmn9jukvwqp3vzdl6j2t7lk/2528509/1/1
I[2022-11-27|03:16:35.861] shutting down                                module=bidengine-order cmp=provider order=akash1qqzwc5d7hynl67nsmn9jukvwqp3vzdl6j2t7lk/2528509/1/1
E[2022-11-27|03:16:36.660] teardown lease: unable to delete manifest    cmp=provider client=kube ns=u4fctraths8fvqb2l74h3ercdhvfbktrktsb9i7gabe92 error="manifests.akash.network \"u4fctraths8fvqb2l74h3ercdhvfbktrktsb9i7gabe92\" not found"
E[2022-11-27|03:16:36.660] lease teardown failed                        module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1fpaarnccrmrz5vfwqyyjd70x4uq7l5vnasay5r/8381375/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=westcoast err="namespaces \"u4fctraths8fvqb2l74h3ercdhvfbktrktsb9i7gabe92\" not found"
E[2022-11-27|03:16:36.660] teardown lease: unable to delete manifest    cmp=provider client=kube ns=jeaesjdj77ipj44ujmmgqdv36h37oofbs99v48eb7be3u error="manifests.akash.network \"jeaesjdj77ipj44ujmmgqdv36h37oofbs99v48eb7be3u\" not found"
E[2022-11-27|03:16:36.660] lease teardown failed                        module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash18u6zew2sg9kfwlk4rurh3m8vxlylp9jmzr25ah/8525559/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=akash err="namespaces \"jeaesjdj77ipj44ujmmgqdv36h37oofbs99v48eb7be3u\" not found"
E[2022-11-27|03:16:36.661] teardown lease: unable to delete manifest    cmp=provider client=kube ns=6r3h1ci17kj2d2n48f4nftkdu62o7ji1nh98d61jek3v4 error="manifests.akash.network \"6r3h1ci17kj2d2n48f4nftkdu62o7ji1nh98d61jek3v4\" not found"
E[2022-11-27|03:16:36.661] lease teardown failed                        module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash18u6zew2sg9kfwlk4rurh3m8vxlylp9jmzr25ah/8476291/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=akash err="namespaces \"6r3h1ci17kj2d2n48f4nftkdu62o7ji1nh98d61jek3v4\" not found"
E[2022-11-27|03:16:36.662] teardown lease: unable to delete manifest    cmp=provider client=kube ns=jo9uinc9s7q0l8r1ca8d12nolg92m9r7dm4mh7e79v7pq error="manifests.akash.network \"jo9uinc9s7q0l8r1ca8d12nolg92m9r7dm4mh7e79v7pq\" not found"
E[2022-11-27|03:16:36.662] lease teardown failed                        module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1kccah0qqt0fsmve74900ha7en44c3qwsv72att/8042485/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=pkt-miner err="namespaces \"jo9uinc9s7q0l8r1ca8d12nolg92m9r7dm4mh7e79v7pq\" not found"
2022-11-27 03:16:36.780742 I | http: TLS handshake error from 10.0.21.151:44846: EOF
D[2022-11-27|03:16:37.863] lease is out of funds                        module=provider-service cmp=provider cmp=balance-checker lease=akash1kccah0qqt0fsmve74900ha7en44c3qwsv72att/8042485/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp
D[2022-11-27|03:16:37.863] sending withdraw                             module=provider-service cmp=provider cmp=balance-checker lease=akash1kccah0qqt0fsmve74900ha7en44c3qwsv72att/8042485/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp
E[2022-11-27|03:16:37.874] failed to do lease withdrawal                module=provider-service cmp=provider cmp=balance-checker err="rpc error: code = Unknown desc = rpc error: code = Unknown desc = failed to execute message; message index: 0: payment closed [cosmos/[email protected]/baseapp/baseapp.go:781] With gas wanted: '0' and gas used: '48650' : unknown request" LeaseID=akash1kccah0qqt0fsmve74900ha7en44c3qwsv72att/8042485/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp
D[2022-11-27|03:16:38.155] lease is out of funds                        module=provider-service cmp=provider cmp=balance-checker lease=akash18u6zew2sg9kfwlk4rurh3m8vxlylp9jmzr25ah/8476291/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp
D[2022-11-27|03:16:38.155] sending withdraw                             module=provider-service cmp=provider cmp=balance-checker lease=akash18u6zew2sg9kfwlk4rurh3m8vxlylp9jmzr25ah/8476291/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp
E[2022-11-27|03:16:38.165] failed to do lease withdrawal                module=provider-service cmp=provider cmp=balance-checker err="rpc error: code = Unknown desc = rpc error: code = Unknown desc = failed to execute message; message index: 0: payment closed [cosmos/[email protected]/baseapp/baseapp.go:781] With gas wanted: '0' and gas used: '48638' : unknown request" LeaseID=akash18u6zew2sg9kfwlk4rurh3m8vxlylp9jmzr25ah/8476291/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp
E[2022-11-27|03:16:38.263] teardown lease: unable to delete manifest    cmp=provider client=kube ns=jeaesjdj77ipj44ujmmgqdv36h37oofbs99v48eb7be3u error="manifests.akash.network \"jeaesjdj77ipj44ujmmgqdv36h37oofbs99v48eb7be3u\" not found"
E[2022-11-27|03:16:38.263] lease teardown failed                        module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash18u6zew2sg9kfwlk4rurh3m8vxlylp9jmzr25ah/8525559/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=akash err="namespaces \"jeaesjdj77ipj44ujmmgqdv36h37oofbs99v48eb7be3u\" not found"
E[2022-11-27|03:16:38.263] teardown lease: unable to delete manifest    cmp=provider client=kube ns=6r3h1ci17kj2d2n48f4nftkdu62o7ji1nh98d61jek3v4 error="manifests.akash.network \"6r3h1ci17kj2d2n48f4nftkdu62o7ji1nh98d61jek3v4\" not found"
E[2022-11-27|03:16:38.263] lease teardown failed                        module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash18u6zew2sg9kfwlk4rurh3m8vxlylp9jmzr25ah/8476291/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=akash err="namespaces \"6r3h1ci17kj2d2n48f4nftkdu62o7ji1nh98d61jek3v4\" not found"
E[2022-11-27|03:16:38.263] teardown lease: unable to delete manifest    cmp=provider client=kube ns=u4fctraths8fvqb2l74h3ercdhvfbktrktsb9i7gabe92 error="manifests.akash.network \"u4fctraths8fvqb2l74h3ercdhvfbktrktsb9i7gabe92\" not found"
E[2022-11-27|03:16:38.263] lease teardown failed                        module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1fpaarnccrmrz5vfwqyyjd70x4uq7l5vnasay5r/8381375/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=westcoast err="namespaces \"u4fctraths8fvqb2l74h3ercdhvfbktrktsb9i7gabe92\" not found"
E[2022-11-27|03:16:38.265] teardown lease: unable to delete manifest    cmp=provider client=kube ns=jo9uinc9s7q0l8r1ca8d12nolg92m9r7dm4mh7e79v7pq error="manifests.akash.network \"jo9uinc9s7q0l8r1ca8d12nolg92m9r7dm4mh7e79v7pq\" not found"
E[2022-11-27|03:16:38.265] lease teardown failed                        module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1kccah0qqt0fsmve74900ha7en44c3qwsv72att/8042485/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=pkt-miner err="namespaces \"jo9uinc9s7q0l8r1ca8d12nolg92m9r7dm4mh7e79v7pq\" not found"
D[2022-11-27|03:16:40.078] running check                                module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1xe342jdk25wugvrhlcsl65csr2z48etxp32mpt/8286524/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=akash cmp=deployment-monitor attempt=1
I[2022-11-27|03:16:40.085] check result                                 module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1xe342jdk25wugvrhlcsl65csr2z48etxp32mpt/8286524/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=akash cmp=deployment-monitor ok=true attempt=1
D[2022-11-27|03:16:40.085] reservation status update                    module=provider-cluster cmp=provider cmp=service cmp=inventory-service order=akash1xe342jdk25wugvrhlcsl65csr2z48etxp32mpt/8286524/1/1 resource-group=akash allocated=true
E[2022-11-27|03:16:40.097] node in poor condition                       cmp=provider client=kube node=k3s-wildponyexpress condition=MemoryPressure status=Unknown
E[2022-11-27|03:16:40.097] node in poor condition                       cmp=provider client=kube node=k3s-wildponyexpress condition=DiskPressure status=Unknown
E[2022-11-27|03:16:40.097] node in poor condition                       cmp=provider client=kube node=k3s-wildponyexpress condition=PIDPressure status=Unknown
D[2022-11-27|03:16:40.753] lease is out of funds                        module=provider-service cmp=provider cmp=balance-checker lease=akash18u6zew2sg9kfwlk4rurh3m8vxlylp9jmzr25ah/8525559/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp
D[2022-11-27|03:16:40.753] sending withdraw                             module=provider-service cmp=provider cmp=balance-checker lease=akash18u6zew2sg9kfwlk4rurh3m8vxlylp9jmzr25ah/8525559/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp
E[2022-11-27|03:16:40.762] failed to do lease withdrawal                module=provider-service cmp=provider cmp=balance-checker err="rpc error: code = Unknown desc = rpc error: code = Unknown desc = failed to execute message; message index: 0: payment closed [cosmos/[email protected]/baseapp/baseapp.go:781] With gas wanted: '0' and gas used: '48644' : unknown request" LeaseID=akash18u6zew2sg9kfwlk4rurh3m8vxlylp9jmzr25ah/8525559/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp
E[2022-11-27|03:16:41.268] teardown lease: unable to delete manifest    cmp=provider client=kube ns=jo9uinc9s7q0l8r1ca8d12nolg92m9r7dm4mh7e79v7pq error="manifests.akash.network \"jo9uinc9s7q0l8r1ca8d12nolg92m9r7dm4mh7e79v7pq\" not found"
E[2022-11-27|03:16:41.268] lease teardown failed                        module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1kccah0qqt0fsmve74900ha7en44c3qwsv72att/8042485/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=pkt-miner err="namespaces \"jo9uinc9s7q0l8r1ca8d12nolg92m9r7dm4mh7e79v7pq\" not found"
E[2022-11-27|03:16:41.268] teardown lease: unable to delete manifest    cmp=provider client=kube ns=6r3h1ci17kj2d2n48f4nftkdu62o7ji1nh98d61jek3v4 error="manifests.akash.network \"6r3h1ci17kj2d2n48f4nftkdu62o7ji1nh98d61jek3v4\" not found"
E[2022-11-27|03:16:41.268] lease teardown failed                        module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash18u6zew2sg9kfwlk4rurh3m8vxlylp9jmzr25ah/8476291/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=akash err="namespaces \"6r3h1ci17kj2d2n48f4nftkdu62o7ji1nh98d61jek3v4\" not found"
E[2022-11-27|03:16:41.268] teardown lease: unable to delete manifest    cmp=provider client=kube ns=jeaesjdj77ipj44ujmmgqdv36h37oofbs99v48eb7be3u error="manifests.akash.network \"jeaesjdj77ipj44ujmmgqdv36h37oofbs99v48eb7be3u\" not found"
E[2022-11-27|03:16:41.268] lease teardown failed                        module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash18u6zew2sg9kfwlk4rurh3m8vxlylp9jmzr25ah/8525559/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=akash err="namespaces \"jeaesjdj77ipj44ujmmgqdv36h37oofbs99v48eb7be3u\" not found"
E[2022-11-27|03:16:41.268] teardown lease: unable to delete manifest    cmp=provider client=kube ns=u4fctraths8fvqb2l74h3ercdhvfbktrktsb9i7gabe92 error="manifests.akash.network \"u4fctraths8fvqb2l74h3ercdhvfbktrktsb9i7gabe92\" not found"
E[2022-11-27|03:16:41.268] lease teardown failed                        module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1fpaarnccrmrz5vfwqyyjd70x4uq7l5vnasay5r/8381375/1/1/akash19yhu3jgw8h0320av98h8n5qczje3pj3u9u2amp manifest-group=westcoast err="namespaces \"u4fctraths8fvqb2l74h3ercdhvfbktrktsb9i7gabe92\" not found"