Document PubSub changes from go-ipfs v0.11

[TheDiscordian]

This describes the issue very well: ipfs/kubo#8612

Caused by: ipfs/kubo#8343

Relevant areas: https://docs.ipfs.io/reference/http/api/#api-v0-pubsub-pub && https://docs.ipfs.io/reference/http/api/#api-v0-pubsub-sub (Edit: This actually doesn't effect sub, details below!)

Basically, we used to accept regular strings for pubsub topics over the HTTP API, now we require Base64URL encoding prefixed with a 'u' (Edit: As outlined below, this isn't entirely true, Base64URL is the default/recommended encoding, but technically any multibase should work https://github.com/multiformats/multibase, which is where the u comes from). Noting this down in an issue so I don't forget, I spun my wheels on this for a while not realizing this change happened, this is probably a good first issue though.

Also not sure how other APIs have been effected. I believe the CLI handles this for you for example.

[Annamarie2019]

We made a change and it's causing bugs. We used to accept regular strings for pubsub topics over the HTTP API, now we require Base64URL encoding prefixed with a 'u'.
These docs need to be fixed: https://docs.ipfs.io/reference/http/api/#api-v0-pubsub-pub && https://docs.ipfs.io/reference/http/api/#api-v0-pubsub-sub
Needs analysis as to whether other APIs are affected.

[TMoMoreau]

Fix for the areas specified by @TheDiscordian at PR #1046

[Annamarie2019]

@TMoMoreau or @TheDiscordian Sorry, I'm new to Base64URL encoding and can't find any discussion of it in respect to a prefix. (46 seems to mean u, but that doesn't help me any). I don't know how to prefix the u. Is there a . or an _ ? Is this doc correct (I bolded what I changed):

Arguments

• arg [string]: Topic to publish to. Required: yes. Requires Base64URL encoding, prefixed with u.

Request Body

Argument data is of file type. This endpoint expects one or several files (depending on the command) in the body of the request as umultipart/form-data.

As for affecting other APIs, publish is the only one that writes and has a Request Body, so it seemed to me that that's the only one to change. Is that correct thinking? (To read existing stuff, you'd have to call it by whatever it's named.)

[TheDiscordian]

@Annamarie2019 The u comes from multibase encoding: https://github.com/multiformats/multibase. So technically prefixes other than u are possible for other bases. I tested out your theory that this only effects publish and not subscribe, you're totally correct there.

Thanks so much for investigating!

Edit: Also sorry for not updating this issue when I realised other prefixes are possible! Totally slipped my mind.

[Annamarie2019]

@TheDiscordian @TMoMoreau

For reference, this is what the article says now:

---

/api/v0/pubsub/pub

Publish data to a given pubsub topic.

Arguments
arg [string]: Topic to publish to. Required: yes.

Request Body
Argument data is of file type. This endpoint expects one or several files (depending on the command) in the body of the request as 'multipart/form-data'.

---

I'll display my ignorance here, in case it helps the cause, but I don't understand:

• "of file type"... Do we mean, "Argument data is the endpoint, which expects one or more files"...or are we asking for a file type (.txt, .mp4...)?
• "depending on the command"....I thought we were defining the command: pub.... Does it mean "sub-command" or "argument"? (such as " ipfs multibase encode - Encode data into multibase string " ...from entering ipfs multibase u --help in cli)
• "as" .... meaning "such as" (introducing an example) or "in the format of"
• the original says "as multipart/form-data Multibase format is <base-encoding-character><base-encoded-data>".....does multipart/form-data need some explanation? (See What does multipart/form-data mean? and HTML | enctype Attribute ) Would the accepted format be umultipart/form-data or possibly /u/multiformat/form-data or other? How do you format multiple files...comma delimited? space? How do you format multiple files?
  Other than u, are other bases possible? under what conditions?

[TheDiscordian]

data/body is out of scope of this issue (hasn't been changed). arg is what this issue is about, and the formats other than u (Base64URL) are defined here: https://github.com/multiformats/multibase, as I outlined above.

Sorry, I'm genuinely not understanding the disconnect here. @TMoMoreau should be able to update his PR with this new info though.

Edit: BTW some of the confusion you've outlined really might be worth new issues to help with clarity.

[TMoMoreau]

Updated PR to document the changes that this issue specified. #1065

[lidel]

Related: ipfs/go-ipfs-cmds#224

[Annamarie2019]

Thanks, @TheDiscordian I read so much stuff, that i forgot where I started: "we used to accept regular strings for pubsub topics...now we require....." (Duh!) So how's this for the update:

Arguments
arg [string]: Topic to publish to. Required: yes. Requires multibase encoding. The default and recommended encoding is u for base64url, but you can use any base encoding character. See the Multibase Table at multibase.

[Annamarie2019]

@TMoMoreau Thanks. Your explanation in #1065 is great, but if we have it up by the argument, maybe after the curl example, all we need to say is, something like:

Arguments
arg [string]: Topic to publish to. Required: yes. Requires multibase encoding. The default and recommended encoding is u for base64url, but you can use any base encoding character. See the Multibase Table at multibase.

Request Body...

curl -X POST -F file=@myfile "http://127.0.0.1:5001/api/v0/pubsub/pub?arg=u<topic>"
Note the u prefix before the topic name, satisfying the argument requirement to be base-encoded.

cc: @TheDiscordian

[Annamarie2019]

@TMoMoreau @TheDiscordian Should we add the u prefix before the topics and link to this explanation for https://docs.ipfs.io/reference/cli/#ipfs-pubsub in the subcommands (maybe tick marks around those commands as well)?

[TheDiscordian]

@lidel First, reading the source, both pub and sub expect multibase, yes? If so, is this bandaid okay (until maybe a new type can be brewed up):

https://github.com/ipfs/go-ipfs/blob/a61c53f87fc6b5341c8d0a0b088cb16dd59f8cc8/core/commands/pubsub.go#L173 "Topic to publish to." -> "Topic to publish to (multibase encoded)."
https://github.com/ipfs/go-ipfs/blob/a61c53f87fc6b5341c8d0a0b088cb16dd59f8cc8/core/commands/pubsub.go#L78 "Name of topic to subscribe to." -> "Name of topic to subscribe to (multibase encoded)."

I specifically want to know if sub requires the multibase encoding because IIRC when I tested it out, it didn't need it, but it's possible (likely even) that I didn't test thoroughly enough.

If so @TMoMoreau can make a PR with those changes. And hopefully put this issue to bed until a better issue spawns.

CC @johnnymatthews

[lidel]

iirc all ipfs pubsub * commands got fixed to use multibase when topic is passed as arg (ipfs/kubo#8183 (review)) so this impacts both pub and sub.

The proposed bandaid is fine until ipfs/go-ipfs-cmds#224 is addressed 👍

There is a small caveat tho: ipfs command line client will do the multibase conversion for you automatically. So on the CLI you use regular strings, and the conversion happens behind the scenes before the wire format is sent over HTTP RPC.

Due to this, we need to clarify that in a way that looks acceptable on both HTTP RPC API reference page, and when user reads ipfs pubsub pub --help (unfortunately, the same text is displayed in both places).

Pubsub commands are marked as experimental, so we don't need to stress too much about this, does not have to be pretty.

Maybe (multibase encoded when sent over HTTP RPC) ?

[BigLep]

Closing this since ipfs/kubo#8933 was merged

[lidel]

Remaining work documented in #1146