Deleted messages not being shown

[albionb96]

After starting multiple producers/consumers which publish with different subjects on the same jetstream I do not see the number of deleted messages in nats-box. As you can see from the following screenshot the stream size is ca. 600k messages, which is ok as expected and also the number of total sent messages is more than 43,6Mil which is also correct. But I would expect too se under deleted messages not 0 but something around 43Mil, which I don't see. When I started another java project with multiple producers/consumers I was able to see the number of deleted messages but this time not any more.

Should this parameter in nats-box show the number of deleted messages or am I wrong, because I couldn't find something in this regard?

[ripienaar]

Imagine you have a stream with messages 1,2,3,4,5,6,7,8,9,10 the stream removes messages due to limits so you have 5,6,7,8,9,10 this would not be reported as deleted messages.

If however you now delete using the delete API message 8, you would have 1 deleted message. If you deleted message 5 you would still have 1 deleted message reported.

So its about the definition of deleted messages - they are gaps in the stream if you had 5,6,7,|gap|,10 the gap is 2 big so deleted is 2.

[ripienaar]

If you consumed all first 1000 you wi t see any deleted as there will be no gap

[neilalexander]

The deletes are showing how many messages were deleted between the first and last sequence. If your delete caused the first or last sequence to change because it was the first message, it won't be counted as a delete within that range, because the range itself changed.

[ripienaar]

$ nats s add DELETED --subjects 'js.in.deleted.>' --defaults
$ for i in 1 2 3 4 5
do
nats req 'js.in.deleted.1' '' --count 100
nats req 'js.in.deleted.2' '' --count 100
done
$ nats s purge DELETED --subject js.in.deleted.1

So we make a stream, put in messages such that they are in groups such that when I purge the js.in.deleted.1 subject I have gaps, now stream info showsL:

State:

              Messages: 500
                 Bytes: 22 KiB
        First Sequence: 101 @ 2024-08-29 19:20:27
         Last Sequence: 1,000 @ 2024-08-29 19:20:33
      Deleted Messages: 400
      Active Consumers: 0
    Number of Subjects: 1

And we can see the gaps:

$ nats s gaps DELETED -f
500 / 500 [====================================================================] 100%

╭──────────────────────────────╮
│ 4 gaps found in Stream DELET │
│              ED              │
├───────────────┬──────────────┤
│ First Message │ Last Message │
├───────────────┼──────────────┤
│ 201           │ 300          │
│ 401           │ 500          │
│ 601           │ 700          │
│ 801           │ 900          │
╰───────────────┴──────────────╯

[sourabhaggrawal]

one of my stream is WQ with only one subject, no TTL & only one consumer reads from it, no explicit call to delete api.
Now that there is only one consumer reading from it, I am hoping it will read sequentially and nats server keep deleting the messages when ack'd and FirstSequence should be moving forward accordingly.
Yet I see FirstSequence is left far behind and only few messages are left in stream. I have not deleted/purged message or stream or subject anywhere. I still don't understand why it is showing Deleted now. However there are some redelivery of messages that might have happened

[ripienaar]

Probably some was not asked before max delivered.

[sourabhaggrawal]

Also when I run gaps command on stream, it throws me this error

? Really detect gaps in stream STREAM_1 with 277 messages and 1.1 MiB bytes Yes
panic: runtime error: index out of range [0] with length 0

goroutine 50 [running]:
github.com/gosuri/uiprogress.(*Bar).Bytes(0xc000312300)
	/go/amd64/pkg/mod/github.com/gosuri/[email protected]/bar.go:195 +0x40b
github.com/gosuri/uiprogress.(*Bar).String(...)
	/go/amd64/pkg/mod/github.com/gosuri/[email protected]/bar.go:214
github.com/gosuri/uiprogress.(*Progress).print(0xc00013e420)
	/go/amd64/pkg/mod/github.com/gosuri/[email protected]/progress.go:127 +0x8f
github.com/gosuri/uiprogress.(*Progress).Listen(0xc00013e420)
	/go/amd64/pkg/mod/github.com/gosuri/[email protected]/progress.go:114 +0x25
created by github.com/gosuri/uiprogress.(*Progress).Start in goroutine 1
	/go/amd64/pkg/mod/github.com/gosuri/[email protected]/progress.go:134 +0x4f

[neilalexander]

Showing the interior deletes is useful for us because interior deletes are one of the first things we look for when there are performance issues on a stream. It probably makes sense for it to be named as such — it might not be universally descriptive, but at least it's not misleading in the way Deleted Messages is.

I can understand the desire to show the overall count of deletes too, but keep in mind that the first sequence of a stream isn't guaranteed to be 1, it can be set to whatever you want when the stream is created, so calculating from last seq and available messages alone isn't necessarily going to do the right thing.

[sourabhaggrawal]

Showing the interior deletes is useful for us because interior deletes are one of the first things we look for when there are performance issues on a stream. It probably makes sense for it to be named as such — it might not be universally descriptive, but at least it's not misleading in the way Deleted Messages is.

I agree with you Deleted Message in misleading and something to be done with it.

I can understand the desire to show the overall count of deletes too, but keep in mind that the first sequence of a stream isn't guaranteed to be 1, it can be set to whatever you want when the stream is created, so calculating from last seq and available messages alone isn't necessarily going to do the right thing.

I did not understand this part, can you set the number of sequence to start with while creating the stream ? Are you referring to source and mirror stream here ?

[neilalexander]

can you set the number of sequence to start with while creating the stream ?

Yes:

$ nats stream create test --defaults --subjects test --first-sequence=100
Stream test was created

Information for Stream test created 2024-08-30 15:07:02

              Subjects: test
              Replicas: 1
               Storage: File
        First Sequence: 100

Options:

             Retention: Limits
       Acknowledgments: true
        Discard Policy: Old
      Duplicate Window: 2m0s
            Direct Get: true
     Allows Msg Delete: true
          Allows Purge: true
        Allows Rollups: false

Limits:

      Maximum Messages: unlimited
   Maximum Per Subject: unlimited
         Maximum Bytes: unlimited
           Maximum Age: unlimited
  Maximum Message Size: unlimited
     Maximum Consumers: unlimited

State:

              Messages: 0
                 Bytes: 0 B
        First Sequence: 100
         Last Sequence: 99
      Active Consumers: 0

[ripienaar]

to be fair setting the start seq is a special case, but this really just isn't that difficult to figure out deleted messages.

           Messages: 1,000
      Last Sequence: 8,689,458 @ 2024-08-30 14:13:06

so like 8,689,458 - 1000 messages were removed.

Seems to be a huge deal over simple math to me

[sourabhaggrawal]

can you set the number of sequence to start with while creating the stream ?

Yes:

$ nats stream create test --defaults --subjects test --first-sequence=100
Stream test was created

Information for Stream test created 2024-08-30 15:07:02

              Subjects: test
              Replicas: 1
               Storage: File
        First Sequence: 100

Options:

             Retention: Limits
       Acknowledgments: true
        Discard Policy: Old
      Duplicate Window: 2m0s
            Direct Get: true
     Allows Msg Delete: true
          Allows Purge: true
        Allows Rollups: false

Limits:

      Maximum Messages: unlimited
   Maximum Per Subject: unlimited
         Maximum Bytes: unlimited
           Maximum Age: unlimited
  Maximum Message Size: unlimited
     Maximum Consumers: unlimited

State:

              Messages: 0
                 Bytes: 0 B
        First Sequence: 100
         Last Sequence: 99
      Active Consumers: 0

This is interesting, what is the use case of this ?