This project aims to create a library for reading and writing mosquitto databases and perhaps provide some documentation about the mosquitto db internals.
- Reading (partially)
- Writing
The database always begins with a DBHeader, following with chunk headers and their data in between.
| File Layout |
|---|
| DbHeader |
| Chunk Header0 |
| Chunk Data0 |
| Chunk Header1 |
| Chunk Data1 |
| Chunk HeaderN |
| Chunk DataN |
| ..... |
The database file layout looks like it has been stable and not changed anything over versions. Most differences are slight changes to how chunks looks like.
- Configuration Chunk
- Message Store
- Client Message
- Retain
- Subscription
- Client
This piece contains a database file magic, a CRC and a Version. These fields should always be first and always be present in a mosquitto database.
| Name | Type | Notes |
|---|---|---|
| Magic | [15]byte | Always "\x00\xB5\x00mosquitto db" |
| CRC | uint32 | I don't see this crc used anywhere |
| Version | uint32 | Version of database. At time of writing 6 is the latest |
This header contains a chunk type id, and the chunk length. Having the chunk length enables skipping unknown / unwanted chunks.
| Name | Type | Notes |
|---|---|---|
| Type | uint32 | One of the Chunk type IDs |
| Length | uint32 | Length of the chunk |
| Name | Type | Notes |
|---|---|---|
| Type | uint16 | One of the Chunk type IDs |
| Length | uint32 | Length of the chunk |
I'm guessing this chunk always comes right after the DBHeader, and only one of this chunk is ever present in the file.
This chunk contains information about last allocated StoreID, whether the broker shut down properly and the size of a StoreID.
| Name | Type | Notes |
|---|---|---|
| LastStoreID | uint64 | Varies in size? |
| Shutdown | uint8 | this field is 1 if mosquitto was shut down properly |
| StoreIDSize | uint8 | Size of a StoreID |
| Name | Type | Notes |
|---|---|---|
| Shutdown | uint8 | this field is 1 if mosquitto was shut down properly |
| StoreIDSize | uint8 | Size of a StoreID |
| LastStoreID | uint64 | Varies in size? |
This chunk contains one stored message.
| Name | Type | Notes |
|---|---|---|
| StoreID | uint64 | Message store ID |
| ExpiryTime | int64 | Timestamp when this message is expired |
| PayloadLen | uint32 | Payload length |
| SourceMid | uint16 | Source message id |
| SourceIDLen | uint16 | Source client id length |
| SourceUsernameLen | uint16 | Source username length |
| TopicLen | uint16 | Topic length |
| SourcePort | uint16 | Source port |
| QoS | uint8 | Quality of service |
| Retain | uint8 | is 1 if this message is retained |
| SourceID | string | contains the source client id. size comes from SourceIDLen field |
| SourceUsername | string | contains the source username. size comes from SourceUsernameLen |
| Topic | string | Message topic. size comes from TopicLen field |
| Payload | []byte | payload data. size comes from PayloadLen field |
| Properties | - | this field covers the rest of the chunk length. This version of the library skips this field |
| Name | Type | Notes |
|---|---|---|
| StoreID | uint64 | Message store ID |
| SourceIDLen | uint16 | Source client id length |
| SourceID | string | contains the source client id. size comes from SourceIDLen field |
| SourceUsernameLen | uint16 | V4 only! Source username length |
| SourceUsername | string | V4 only! Source username. size comes from SourceUsernameLen field |
| SourcePort | uint16 | V4 only! Source port |
| SourceMid | uint16 | Source message id |
| MID | uint16 | This field is never used and is not stored anywhere when read |
| TopicLen | uint16 | Topic length |
| Topic | string | Message topic. size comes from TopicLen field |
| QoS | uint8 | Quality of service |
| Retain | uint8 | is 1 if this message is retained |
| PayloadLen | uint32 | Payload length |
| Payload | []byte | payload data. size comes from PayloadLen field |
| Name | Type | Notes |
|---|---|---|
| StoreID | uint64 | Message store ID |
| SourceIDLen | uint16 | Source client id length |
| SourceID | string | contains the source client id. size comes from SourceIDLen field |
| SourceMid | uint16 | Source message id |
| MID | uint16 | This field is never used and is not stored anywhere when read |
| TopicLen | uint16 | Topic length |
| Topic | string | Message topic. size comes from TopicLen field |
| QoS | uint8 | Quality of service |
| Retain | uint8 | is 1 if this message is retained |
| PayloadLen | uint32 | Payload length |
| Payload | []byte | payload data. size comes from PayloadLen field |
| Name | Type | Notes |
|---|---|---|
| StoreID | uint64 | Message store ID |
| MID | uint16 | Message ID |
| IDLen | uint16 | Client id length |
| QoS | uint8 | Quality of Service |
| State | uint8 | Some internal message state for mosquitto. TODO: enumerate |
| RetainDup | uint8 | Retention: (flags & 0xF0 >> 4) Dup: (flags & 0x0F) |
| Direction | uint8 | Direction of message. TODO: enumerate |
| ClientID | string | ClientID for message. size comes from IDLen field |
| Name | Type | Notes |
|---|---|---|
| IDLen | uint16 | Client id length |
| ClientID | string | ClientID for message. size comes from IDLen field |
| StoreID | uint64 | Message store ID |
| MID | uint16 | Message ID |
| QoS | uint8 | Quality of Service |
| Retain | uint8 | Retain flag |
| Direction | uint8 | Direction of message. TODO: enumerate |
| State | uint8 | Some internal message state for mosquitto. TODO: enumerate |
| Dup | uint8 | Duplicate flag |
This chunk contains an ID for a stored message, that is retained.
I'm unsure why this chunk is necessary. Both the Stored Message chunk, and the Client message chunk contains a retain-flag.
The StoreID seems to refer to a Message in a store chunk.
| Name | Type | Notes |
|---|---|---|
| StoreID | uint64 | Message store ID |
| Name | Type | Notes |
|---|---|---|
| Identifier | uint32 | mqttv5 subscription identifier |
| IDLen | uint16 | Length of client id |
| TopicLen | uint16 | Length of topic |
| QoS | uint8 | Quality of service |
| Options | uint8 | Subscription options |
| ClientID | string | Client id for subscription. size comes from IDLen field |
| Topic | string | Subscription topic. size comes from TopicLen field |
| Name | Type | Notes |
|---|---|---|
| IDLen | uint16 | Length of client id |
| ClientID | string | Client id for subscription. size comes from IDLen field |
| TopicLen | uint16 | Length of topic |
| Topic | string | Subscription topic. size comes from TopicLen field |
| QoS | uint8 | Quality of service |
This chunk keeps information about one client.
| Name | Type | Notes |
|---|---|---|
| SessionExpiryTime | int64 | Timestamp when the client is considered expired |
| SessionExpiryInterval | uint32 | Used for mqttv5 clients, set by "Session Expiry Interval" property |
| LastMID | uint16 | Last MessageID for client |
| IDLen | uint16 | Length of client id |
| ListenerPort | uint16 | Listener port. new in v6 |
| UsernameLen | uint16 | Username length. new in v6 |
| Padding | uint32 | Quote from mosquitto source: "tail: 4 byte padding, because 64bit member forces multiple of 8 for struct size" |
| ClientID | string | Client id. size comes from IDLen field |
| Username | string | Username of client. size comes from UsernameLen field. new in v6 |
| Name | Type | Notes |
|---|---|---|
| SessionExpiryTime | int64 | Timestamp when the client is considered expired |
| SessionExpiryInterval | uint32 | Used for mqttv5 clients, set by "Session Expiry Interval" property |
| LastMID | uint16 | Last MessageID for client |
| IDLen | uint16 | Length of client id |
| ClientID | string | Client id. size comes from IDLen field |
| Name | Type | Notes |
|---|---|---|
| IDLen | uint16 | Length of client id |
| ClientID | string | Client id. size comes from IDLen field |
| Time | time_t | Presumably the time this message was sent. This field appears not to be saved anywhere and just read then ignored. |
| Name | Type | Notes |
|---|---|---|
| IDLen | uint16 | Length of client id |
| ClientID | string | Client id. size comes from IDLen field |