[omdb] Add `omdb instance info` command #6610

hawkw · 2024-09-19T17:41:20Z

Presently, omdb can be used to query the database for instance records
using the omdb instances command. However, this command does not
output all the database state associated with an instance. Instead, it
displays the instance's name, UUID, effective state, active Propolis
UUID, sled UUID, and sled serial (the last three only if the instance is
running).

This is some of the most useful information about a customer instance,
but it's not all of the information we have about it. Debugging issues
related to instance state management may require other information from
the instance record, as well as from the instance's active VMM record,
and its target VMM and active migration records (if it is migrating).
Adding all of these additional details to the omdb db instances
command is a bit of a non-starter, though. Since that command outputs a
table of all instances, with one line per instance, the output is
already much wider than 80 columns, and requires a pretty wide
terminal to be readable --- once the table has line wrapped, it's much
harder to interpret. Therefore, I've added an omdb db instance info
command that outputs a complete dump of the state of a single
instance. Since this command is no longer constrained to putting all
salient details about the instance on a single line in a table, we can
show every field in the instance's DB representation without having to
worry about wrapping.

Output from omdb db instance info looks like this (yes, I spent
entirely too long tweaking the formatting 😅):

root@oxz_switch1:~# /var/tmp/omdb-eliza-test-12 db instance info 3e9f7bcb-1ae8-4708-af4f-4b6452af3d1f
note: database URL not specified.  Will search DNS.
note: (override with --db-url or OMDB_DB_URL)
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using database URL postgresql://root@[fd00:1122:3344:109::3]:32221,[fd00:1122:3344:105::3]:32221,[fd00:1122:3344:10b::3]:32221,[fd00:1122:3344:107::3]:32221,[fd00:1122:3344:108::3]:32221/omicron?sslmode=disable
note: database schema version matches expected (98.0.0)

INSTANCE

                  ID: 3e9f7bcb-1ae8-4708-af4f-4b6452af3d1f
                name: rocky9-0
         description: application instance
          created at: 2024-08-13 16:22:18.303110 UTC
    last modified at: 2024-08-13 16:22:18.303110 UTC

CONFIGURATION

               vCPUs: 32
              memory: 96 GiB
            hostname: node0
 auto-restart policy: None

RUNTIME STATE

               state: Vmm
     last updated at: 2024-09-19T00:32:23.871767Z (generation 16)
       active VMM ID: Some(e1a87a49-7ab7-40ff-965e-4d55cc6db394)
       target VMM ID: None
        migration ID: None
        updater lock: UNLOCKED at generation: 16

   active VMM record:
     Vmm {
         id: e1a87a49-7ab7-40ff-965e-4d55cc6db394,
         time_created: 2024-09-19T00:32:23.816650Z,
         time_deleted: None,
         instance_id: 3e9f7bcb-1ae8-4708-af4f-4b6452af3d1f,
         sled_id: f15774c1-b8e5-434f-a493-ec43f96cba06,
         propolis_ip: V6(
             Ipv6Network {
                 addr: fd00:1122:3344:105::1:580,
                 prefix: 128,
             },
         ),
         propolis_port: SqlU16(
             12400,
         ),
         runtime: VmmRuntimeState {
             time_state_updated: 2024-09-19T00:32:34.197708Z,
             gen: Generation(
                 Generation(
                     4,
                 ),
             ),
             state: Running,
         },
     }
root@oxz_switch1:~#

I've also added omdb db instance list (or instance ls) as a subcommand of
omdb db instance, but these are simply aliased to the current db instances
command. It just seemed nice to have additional verbs.

leftwo · 2024-09-19T17:51:54Z

This is great, I've wanted instance info, but not taken the time to write it.

Should I expect it to work for destroyed instances (granted some fields will be empty)?

root@oxz_switch1:~# /var/tmp/omdb-eliza-test-12 db --include-deleted --fetch-limit 999123 instance info fff916c4-814a-48a0-a04a-f887e6f81a9e
note: database URL not specified.  Will search DNS.
note: (override with --db-url or OMDB_DB_URL)
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using database URL postgresql://root@[fd00:1122:3344:109::3]:32221,[fd00:1122:3344:105::3]:32221,[fd00:1122:3344:10b::3]:32221,[fd00:1122:3344:107::3]:32221,[fd00:1122:3344:108::3]:32221/omicron?sslmode=disable
note: database schema version matches expected (98.0.0)
Error: failed to fetch details for instance fff916c4-814a-48a0-a04a-f887e6f81a9e

Caused by:
    Object (of type ById(fff916c4-814a-48a0-a04a-f887e6f81a9e)) not found: instance

but:

root@oxz_switch1:~# /var/tmp/omdb-eliza-test-12 db --include-deleted --fetch-limit 999123 instance list | grep fff916c4-814a-48a0-a04a-f887e6f81a9e
note: database URL not specified.  Will search DNS.
note: (override with --db-url or OMDB_DB_URL)
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using database URL postgresql://root@[fd00:1122:3344:109::3]:32221,[fd00:1122:3344:105::3]:32221,[fd00:1122:3344:10b::3]:32221,[fd00:1122:3344:107::3]:32221,[fd00:1122:3344:108::3]:32221/omicron?sslmode=disable
note: database schema version matches expected (98.0.0)
fff916c4-814a-48a0-a04a-f887e6f81a9e app-13                                                          destroyed <no active Propolis>                 <not on any sled>                    -

dev-tools/omdb/src/bin/omdb/db.rs

gjcolombo

I like @leftwo's idea about considering displaying soft-deleted instances as such. (I might do that with an ad hoc query in OMDB, though, similar to what cmd_db_instances does, so as not to introduce an "ignore time_deleted" flag into the datastore.) Otherwise this LGTM with a couple small comments.

gjcolombo · 2024-09-19T21:13:54Z

dev-tools/omdb/src/bin/omdb/db.rs

+    // /!\ WARNING /!\
+    // This does mean that anyone who adds new fields to the
+    // `nexus_db_model::Instance` type will want to make sure to update this
+    // code as well. Unfortunately, we can't just destructure the struct here to
+    // make sure this code breaks, since the `identity` field isn't public.
+    // So...just don't forget to do that, I  guess.


Should we have a similar warning over in nexus_db_model?

dev-tools/omdb/tests/usage_errors.out

dev-tools/omdb/src/bin/omdb/db.rs

Co-authored-by: Greg Colombo <[email protected]>

hawkw · 2024-09-19T21:59:15Z

Regarding soft-deleted instances...the reason we currently don't report them here is because I had wanted to reuse the instance_fetch_all query from nexus-db-queries, as it's quite complex (largely because it joins with the vmm table twice, for the active and target VMM records). Because it's so hairy, I didn't really want to have an ad-hoc query in omdb that duplicated the whole thing. But, upon further consideration, it would probably be fine to use ad-hoc queries here; the reason instance_fetch_all is Like That is because it needs to fetch all those records in a single atomic query for consistency reasons. We don't actually have that constraint here, and it would probably be fine to just have small ad-hoc queries that fetch the instance and then separately go and get its associated VMMs and mgiration...

hawkw · 2024-09-20T18:18:51Z

Okay, as of e3013e7, we now also handle deleted instances. For example:

root@oxz_switch1:~# /var/tmp/omdb-eliza-test-14 db instance show fff916c4-814a-48a0-a04a-f887e6f81a9e
note: database URL not specified.  Will search DNS.
note: (override with --db-url or OMDB_DB_URL)
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using database URL postgresql://root@[fd00:1122:3344:109::3]:32221,[fd00:1122:3344:105::3]:32221,[fd00:1122:3344:10b::3]:32221,[fd00:1122:3344:107::3]:32221,[fd00:1122:3344:108::3]:32221/omicron?sslmode=disable
note: database schema version matches expected (98.0.0)

 (i) instance has been deleted

INSTANCE

                  ID: fff916c4-814a-48a0-a04a-f887e6f81a9e
          project ID: fe0da422-5c48-4b52-8010-f2fc401f090f
                name: app-13
         description: application instance
          created at: 2024-06-22 00:31:43.889412 UTC
    last modified at: 2024-06-22 00:31:43.889412 UTC
          deleted at: 2024-06-22 00:43:59.156149 UTC

CONFIGURATION

               vCPUs: 1
              memory: 2 GiB
            hostname: node13
 auto-restart policy: None

RUNTIME STATE

               state: Destroyed
     last updated at: 2024-06-22T00:43:49.440274Z (generation 5)
       active VMM ID: None
       target VMM ID: None
        migration ID: None
        updater lock: UNLOCKED at generation: 1

hawkw · 2024-09-20T19:08:52Z

Some last, unnecessary formatting tweaks:

oot@oxz_switch1:~# /var/tmp/omdb-eliza-test-15 db instance show fff916c4-814a-48a0-a04a-f887e6f81a9e
note: database URL not specified.  Will search DNS.
note: (override with --db-url or OMDB_DB_URL)
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using database URL postgresql://root@[fd00:1122:3344:109::3]:32221,[fd00:1122:3344:105::3]:32221,[fd00:1122:3344:10b::3]:32221,[fd00:1122:3344:107::3]:32221,[fd00:1122:3344:108::3]:32221/omicron?sslmode=disable
note: database schema version matches expected (98.0.0)

INSTANCE
                     ID: fff916c4-814a-48a0-a04a-f887e6f81a9e
             project ID: fe0da422-5c48-4b52-8010-f2fc401f090f
                   name: app-13
            description: application instance
             created at: 2024-06-22 00:31:43.889412 UTC
       last modified at: 2024-06-22 00:31:43.889412 UTC
/!\          deleted at: 2024-06-22 00:43:59.156149 UTC

CONFIGURATION
                  vCPUs: 1
                 memory: 2 GiB
               hostname: node13
    auto-restart policy: None

RUNTIME STATE
            nexus state: Destroyed
(i)  external API state: Destroyed
        last updated at: 2024-06-22T00:43:49.440274Z (generation 5)
          active VMM ID: None
          target VMM ID: None
           migration ID: None
           updater lock: UNLOCKED at generation: 1
root@oxz_switch1:~# /var/tmp/omdb-eliza-test-15 db instance show 480c6167-dac2-4df7-808d-c9e776c1c312
note: database URL not specified.  Will search DNS.
note: (override with --db-url or OMDB_DB_URL)
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using database URL postgresql://root@[fd00:1122:3344:109::3]:32221,[fd00:1122:3344:105::3]:32221,[fd00:1122:3344:10b::3]:32221,[fd00:1122:3344:107::3]:32221,[fd00:1122:3344:108::3]:32221/omicron?sslmode=disable
note: database schema version matches expected (98.0.0)

INSTANCE
                     ID: 480c6167-dac2-4df7-808d-c9e776c1c312
             project ID: fe0da422-5c48-4b52-8010-f2fc401f090f
                   name: mysql-3
            description: MySQL Instance
             created at: 2024-09-16 00:43:10.264333 UTC
       last modified at: 2024-09-16 00:43:10.264333 UTC

CONFIGURATION
                  vCPUs: 4
                 memory: 16 GiB
               hostname: mysql-3
    auto-restart policy: None

RUNTIME STATE
            nexus state: Vmm
(i)  external API state: Stopping
        last updated at: 2024-09-19T23:15:50.698207Z (generation 5)
          active VMM ID: Some(51babee3-0ee5-4f67-bcbb-2189a39479a3)
          target VMM ID: None
           migration ID: None
           updater lock: UNLOCKED at generation: 4

      active VMM record:
          Vmm {
              id: 51babee3-0ee5-4f67-bcbb-2189a39479a3,
              time_created: 2024-09-19T23:15:50.647454Z,
              time_deleted: None,
              instance_id: 480c6167-dac2-4df7-808d-c9e776c1c312,
              sled_id: 71def415-55ad-46b4-ba88-3ca55d7fb287,
              propolis_ip: V6(
                  Ipv6Network {
                      addr: fd00:1122:3344:10b::1:4be,
                      prefix: 128,
                  },
              ),
              propolis_port: SqlU16(
                  12400,
              ),
              runtime: VmmRuntimeState {
                  time_state_updated: 2024-09-19T23:18:03.015643Z,
                  gen: Generation(
                      Generation(
                          4,
                      ),
                  ),
                  state: Stopping,
              },
          }
root@oxz_switch1:~#

I think this is pretty good now if either of y'all want to review it again.

leftwo

Thanks for showing the deleted instances too

hawkw added 7 commits September 18, 2024 13:03

[omdb] initial sketch of omdb db instance info

8b1d3e1

nicer instance formatting

6a034ea

that looks good enough

71cfc32

add updater lock state

9b5efc7

rm unused imports

15437aa

make output a little bit more like other OMDB commands

ce4886a

expectorate update

76401d5

hawkw requested review from leftwo and gjcolombo September 19, 2024 17:41

clippy placation

950e9b6

leftwo reviewed Sep 19, 2024

View reviewed changes

dev-tools/omdb/src/bin/omdb/db.rs Outdated Show resolved Hide resolved

hawkw added 2 commits September 19, 2024 11:21

clippy placation the correct way

ea00595

remove incorrect help text

02b53eb

gjcolombo approved these changes Sep 19, 2024

View reviewed changes

Update dev-tools/omdb/src/bin/omdb/db.rs

c1915a7

Co-authored-by: Greg Colombo <[email protected]>

hawkw added 3 commits September 20, 2024 11:06

also show deleted instances

e3013e7

show complete migration history

fa225ae

nicer condition formatting

f21e6cb

more goofy formatting tweaks

61f4f58

hawkw requested a review from gjcolombo September 20, 2024 19:08

fancy headers (i need to stop messing with this)

c7cf62f

leftwo approved these changes Sep 20, 2024

View reviewed changes

update stuff again

f4797f4

hawkw enabled auto-merge (squash) September 20, 2024 21:12

hawkw merged commit b99552c into main Sep 20, 2024
16 checks passed

hawkw deleted the eliza/omdb-instance-show branch September 20, 2024 22:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[omdb] Add `omdb instance info` command #6610

[omdb] Add `omdb instance info` command #6610

hawkw commented Sep 19, 2024 •

edited

Loading

leftwo commented Sep 19, 2024

gjcolombo left a comment

gjcolombo Sep 19, 2024

hawkw commented Sep 19, 2024

hawkw commented Sep 20, 2024

hawkw commented Sep 20, 2024

leftwo left a comment

[omdb] Add omdb instance info command #6610

[omdb] Add omdb instance info command #6610

Conversation

hawkw commented Sep 19, 2024 • edited Loading

leftwo commented Sep 19, 2024

gjcolombo left a comment

Choose a reason for hiding this comment

gjcolombo Sep 19, 2024

Choose a reason for hiding this comment

hawkw commented Sep 19, 2024

hawkw commented Sep 20, 2024

hawkw commented Sep 20, 2024

leftwo left a comment

Choose a reason for hiding this comment

[omdb] Add `omdb instance info` command #6610

[omdb] Add `omdb instance info` command #6610

hawkw commented Sep 19, 2024 •

edited

Loading