Configuration rework

[rustyrussell]

To make room for a future config command to dynamically set configuration variables, I had to delve into our configuration subsystem. It took far longer than I'd expected, but here is the result.

Concretely:

1. listconfigs produces a dict, in several cases giving multiple values (!), and also has a plugins and important-plugins sub-objects. This is deprecated in favor of a configs sub-object: each value in here is a dict, including source, and type information.
2. listconfigs now shows plugin options as you'd expect, including dynamic ones! It also hands the config-set options to restarted plugins, a long requested feature.

I apologize for the scope of this: configuration is hard and I tried not to break anything.

Fixes: #5294

Example output:

{
  "id": "pytest:listconfigs#2",
  "result": {
    "configs": {
      "lightning-dir": {
        "value_str": "/tmp/ltests-_9t7jvko/test_listconfigs_1/lightning-2/",
        "source": "cmdline"
      },
      "network": {
        "value_str": "regtest",
        "source": "cmdline"
      },
      "testnet": {
        "set": false,
        "source": "default"
      },
      "signet": {
        "set": false,
        "source": "default"
      },
      "mainnet": {
        "set": false,
        "source": "default"
      },
      "regtest": {
        "set": false,
        "source": "default"
      },
      "allow-deprecated-apis": {
        "value_bool": false,
        "source": "cmdline"
      },
      "rpc-file": {
        "value_str": "lightning-rpc",
        "source": "default"
      },
      "plugin": {
        "values_str": [],
        "sources": []
      },
      "plugin-dir": {
        "values_str": [],
        "sources": []
      },
      "clear-plugins": {
        "set": false,
        "source": "default"
      },
      "disable-plugin": {
        "values_str": [],
        "sources": []
      },
      "important-plugin": {
        "values_str": [],
        "sources": []
      },
      "always-use-proxy": {
        "value_bool": false,
        "source": "default"
      },
      "daemon": {
        "set": false,
        "source": "default"
      },
      "large-channels": {
        "set": true,
        "source": "cmdline"
      },
      "experimental-dual-fund": {
        "set": false,
        "source": "default"
      },
      "experimental-onion-messages": {
        "set": false,
        "source": "default"
      },
      "experimental-offers": {
        "set": false,
        "source": "default"
      },
      "experimental-shutdown-wrong-funding": {
        "set": false,
        "source": "default"
      },
      "experimental-peer-storage": {
        "set": false,
        "source": "default"
      },
      "experimental-quiesce": {
        "set": false,
        "source": "default"
      },
      "announce-addr-dns": {
        "value_bool": false,
        "source": "default"
      },
      "rgb": {
        "value_str": "022d22",
        "source": "default"
      },
      "alias": {
        "value_str": "SILENTARTIST-v23.05-47-g99a2a78",
        "source": "default"
      },
      "pid-file": {
        "value_str": "/tmp/ltests-_9t7jvko/test_listconfigs_1/lightning-2/lightningd-regtest.pid",
        "source": "default"
      },
      "ignore-fee-limits": {
        "value_bool": false,
        "source": "cmdline"
      },
      "watchtime-blocks": {
        "value_int": 5,
        "source": "cmdline"
      },
      "max-locktime-blocks": {
        "value_int": 2016,
        "source": "default"
      },
      "funding-confirms": {
        "value_int": 1,
        "source": "default"
      },
      "require-confirmed-inputs": {
        "value_bool": false,
        "source": "default"
      },
      "cltv-delta": {
        "value_int": 6,
        "source": "cmdline"
      },
      "cltv-final": {
        "value_int": 5,
        "source": "cmdline"
      },
      "commit-time": {
        "value_int": 10,
        "source": "default"
      },
      "fee-base": {
        "value_int": 1,
        "source": "default"
      },
      "rescan": {
        "value_int": 1,
        "source": "cmdline"
      },
      "fee-per-satoshi": {
        "value_int": 10,
        "source": "default"
      },
      "htlc-minimum-msat": {
        "value_msat": "0msat",
        "source": "default"
      },
      "htlc-maximum-msat": {
        "value_msat": "18446744073709551615msat",
        "source": "default"
      },
      "max-concurrent-htlcs": {
        "value_int": 483,
        "source": "default"
      },
      "max-dust-htlc-exposure-msat": {
        "value_msat": "50000000msat",
        "source": "default"
      },
      "min-capacity-sat": {
        "value_int": 10000,
        "source": "default"
      },
      "addr": {
        "values_str": [
          "127.0.0.1:45091"
        ],
        "sources": [
          "cmdline"
        ]
      },
      "bind-addr": {
        "values_str": [],
        "sources": []
      },
      "announce-addr": {
        "values_str": [],
        "sources": []
      },
      "announce-addr-discovered": {
        "value_str": "auto",
        "source": "default"
      },
      "announce-addr-discovered-port": {
        "value_int": 19846,
        "source": "default"
      },
      "offline": {
        "set": false,
        "source": "default"
      },
      "autolisten": {
        "value_bool": false,
        "source": "default"
      },
      "accept-htlc-tlv-type": {
        "values_int": [],
        "sources": []
      },
      "disable-dns": {
        "set": true,
        "source": "cmdline"
      },
      "encrypted-hsm": {
        "set": false,
        "source": "default"
      },
      "rpc-file-mode": {
        "value_str": "0600",
        "source": "default"
      },
      "commit-fee": {
        "value_int": 100,
        "source": "default"
      },
      "subdaemon": {
        "values_str": [],
        "sources": []
      },
      "experimental-upgrade-protocol": {
        "set": false,
        "source": "default"
      },
      "log-level": {
        "value_str": "debug",
        "source": "cmdline"
      },
      "log-timestamps": {
        "value_bool": true,
        "source": "default"
      },
      "log-prefix": {
        "value_str": "lightning1-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
        "source": "cmdline"
      },
      "log-file": {
        "values_str": [
          "-",
          "/tmp/ltests-_9t7jvko/test_listconfigs_1/lightning-2/log"
        ],
        "sources": [
          "cmdline",
          "cmdline"
        ]
      },
      "dev-no-reconnect": {
        "set": true,
        "source": "cmdline"
      },
      "dev-fail-on-subdaemon-fail": {
        "set": true,
        "source": "cmdline"
      },
      "dev-bitcoind-poll": {
        "value_int": 1,
        "source": "cmdline"
      },
      "dev-fast-gossip": {
        "set": true,
        "source": "cmdline"
      },
      "bitcoin-datadir": {
        "value_str": "/tmp/ltests-_9t7jvko/test_listconfigs_1/lightning-2/",
        "source": "cmdline",
        "plugin": "/home/rusty/devel/cvs/lightning/plugins/bcli"
      },
      "bitcoin-rpcuser": {
        "value_str": "rpcuser",
        "source": "cmdline",
        "plugin": "/home/rusty/devel/cvs/lightning/plugins/bcli"
      },
      "bitcoin-rpcpassword": {
        "value_str": "rpcpass",
        "source": "cmdline",
        "plugin": "/home/rusty/devel/cvs/lightning/plugins/bcli"
      },
      "bitcoin-rpcport": {
        "value_int": 33779,
        "source": "cmdline",
        "plugin": "/home/rusty/devel/cvs/lightning/plugins/bcli"
      },
      "disable-mpp": {
        "set": false,
        "source": "default",
        "plugin": "/home/rusty/devel/cvs/lightning/plugins/pay"
      }
    }
  }
}

[SimonVrouwe]

If you're not in a hurry, then I would love to delve into this and test it (give me 10days).

One question/idea about listconfigs: Would it be possible to have the descriptions (pulled from the schemas, or from a plugin) somehow added to the result ?

[SimonVrouwe]

to dynamically set configuration variables

Does it mean we can switch from --regtest to --mainnet without a restart ? That sounds horrible and impossible.

listconfigs now shows plugin options as you'd expect

Not sure what you mean, but when I run it with --allow-deprecated-apis=false then it shows all plugin options flat inside the configs object and the only ordering being alphabetical without any ordering, was this the intention? edit

IMHO being able to see to which plugin an option belongs to makes everything more understandable. Maybe the configs object can have a plugin array again (containing path and options array ) ?

[SimonVrouwe]

And finally a little off-topic, it would be nice if users can see more option's default values, it makes everything less obscure. Many build-in plugins based on libplugin don't supply default values and for example to find the funder-* options values you have to look into the source.

[rustyrussell]

If you're not in a hurry, then I would love to delve into this and test it (give me 10days).

One question/idea about listconfigs: Would it be possible to have the descriptions (pulled from the schemas, or from a plugin) somehow added to the result ?

[ Oops, I realized you are looking at the old PR. I hated this one, so I've reworked the format... Fixing tests now! ]

Well, since the format is now a dict, it's possible. It's a bit weird though, since it's not really dynamic.

[rustyrussell]

to dynamically set configuration variables

Does it mean we can switch from --regtest to --mainnet without a restart ? That sounds horrible and impossible.

Yeah, we won't be doing that 🤣

It would be on selected options, which would explicitly opt-in.

listconfigs now shows plugin options as you'd expect

Not sure what you mean, but when I run it with --allow-deprecated-apis=false then it shows all plugin options flat inside the configs object and the only ordering being alphabetical without any ordering, was this the intention? edit

IMHO being able to see to which plugin an option belongs to makes everything more understandable. Maybe the configs object can have a plugin array again (containing path and options array ) ?

That was redundant with plugins list, however I like the idea of showing what plugin the options is associated with. Now we have a dict per option, this is easy to add.

Testing now...

[rustyrussell]

Pushed updated, rebased version, and included more details in PR description!

[SimonVrouwe]

Ok looks good, descriptions can easily be added now when desired (I think so).

For convenience I did it on this branch, see the last two commits.

Perhaps default values can also added to the descriptions, which should be easy for plugin options.
For build-in options this is a bit harder because those defaults are lost when a user configures them and some default values are transient, i.e. they depend on the network choice.

[rustyrussell]

The problem now is that our descriptions are in several places: the man pages, and the code. We should settle on one place though.

The man page is more chatty, which is useful, but probably overkill for the code? Perhaps we should get one line from the code, and paste in the rest of the chatter for the man page. Anyway, I think this is a follow-on from this PR, which I'd like to merge soon.

[rustyrussell]

Trivial rebase to make --dev-allowdustreserve dev-only in its own commit, rather than a driveby...

[rustyrussell]

Rebased on master to resolve simple conflicts.

[rustyrussell]

Rebase to fix trivial error for liquid tests (assuming network=regtest in msg)

[cdecker]

Am I the only one who dislikes aliased options? --network=regtest == --regtest 😢

[rustyrussell]

I'm split. I hate the redundancy, but love the brevity...