Merge branch 'master' of linonetwo/ipfs-browser-gateway

[linonetwo]

service-worker-gateway

A IPFS gateway fully running on a Service Worker

Development

This is a "create-react-app" app, so just:

yarn && yarn start

Drawback

1. The first visit is way slower than traditional HTTP page. Though http://ipfs.io/ipfs is slow too.
2. Can't promise long-term cache, compared to a gateway running on a server, who can pin a file for a longer time.
3. Large folders like QmRoYXgYMfcP7YQR4sCuSeJy9afgA5XDJ78JzWntpRhmcu may destroy service worker (maybe due to my using files.get), so it's more suitable to just load HTML pages in this way.
4. Don't work with AJAX likes fetch API.
5. Loading dependencies by importScripts synchronously are slow (10s).

Road Map

• Bundle all dependencies.
• Fix Stream

Reference

• Discussion
• JS-IPFS-Gateway
• readable-stream in SW

[vasco-santos]

Hello @linonetwo

We appreciate your contribution to the service-worker-gateway.

At first, the codebase related to the service-worker implementation should be in the src directory, as its implementation is the purpose of the repo. Therefore, the example should be focused on the service worker usage.

Moreover, we prefer to provide more minimalist and barebone examples of the usage of our repos. Essentially, we intend to focus on the example instead of adding framework related code, which will turn the example more complex. As a result, we are looking for examples using standard JS with a simple web server, such as http-server. In addition, the IPFS organization prefer npm over yarn.

Regarding this example, I tried to run it but unfortunately, I was not able to fetch any file (had errors of undefined Readable Stream, as well as the Response not being returned).

I saw that you have some work regarding the different views of que fetch (file, host site, dir list), which you can contribute to the new repo created for this project. I will be working on the gateway in the next days, feel free to join me in the service-worker-gateway.

Other note @linonetwo , it is a good practice to squash all your commits in a single one before creating a PR.

[linonetwo]

Hi, @vasco-santos

@diasdavid Guide me to read JS-IPFS-Gateway, so I remove my previous POC (which loads a single website and using cache API), and port IPFS-Gateway here.

But IPFS-Gateway was using a web server, which has a different API. So I must have done something wrong when changing web server API to new Response()

[vasco-santos]

This should be the entry point of the service worker implementation. Not an example.

[linonetwo]

Do you mean packing service worker by webpack? I couldn't find a way to do so, so I just put all of them in ./public

[vasco-santos]

No, the "app" should be an example, which uses the module implemented in the src directory.

[linonetwo]

But service worker needs a webpage to install, so app.js is a simple webpage now. It doesn't depend on many modules.

[vasco-santos]

It is correct that the service worker needs a webpage to be installed in the browser.

I will try to be more clear about this repo. The aim here is to provide a module that can be used by anyone in their web pages.

For instance, if the ipfs.io was running this module, ipfs.io/ipfs/hash would use the ipfs node running in the service worker.

[linonetwo]

I got your idea. So there could be some very simple code inside ./src, that handles the returning of new Response(). (After js-ipfs gateway code being migrated.)

Then some webpack config to bundle ./src to ./dist or so.

I misunderstood the purpose of this repo, sorry about that. So let this PR be a POC. If this approach(using code from js-ipfs gateway) turns out to work, we can close it.

[linonetwo]

Who will handle the migration of JS-IPFS-Gateway? If there is a repo created, I can give a hand.

[linonetwo]

I'll squash my commits in future PRs, thanks for pointing that out. I didn't know that before.

[vasco-santos]

I will create several Issues in this REPO for being addressed.

[linonetwo]

It's pretty late in China 😪 If you think the code here is helpful, I can recheck all new Response() tomorrow morning to figure whether I'm not returning some of them.

[vasco-santos]

I will add the initial implementation of the service worker based on the ipfs-service-worker, as soon as possible. After this, we can discuss how to update this initial implementation for supporting the same outputs as the daemon gateway. I think you may have code that may be useful, but this PR will be outdated when adding this initial implementation.

The next steps after having the IPFS node working inside the service worker will be:

• migrate part of the js-ipfs gateway code to a new repo, containing, for example, the HTML templates, that will be used for js-ipfs, as well as the service-worker-gateway, among others;
• create a simple example of the service-worker usage.

You can help in those steps then 😄

[linonetwo]

OK, glad to help, let's accelerate this.

Do you think we can run IPNS inside service worker?

[vasco-santos]

IPNS will start being developed once the DHT Endeavour is finished.

As the IPNS will be part of the ipfs node methods, it will be possible to use it with a service worker as well!

[daviddias]

@vasco-santos DHT should not be a blocker. IPNS can be developed and tested locally + over PubSub. DHT is just another way to propagate the IPNS Records.

[linonetwo]

Forgot to return this Response

[linonetwo]

My god, https://wzrd.in/standalone/pull-stream is broken, I can't simply import it in sw. Sometimes I could.

[linonetwo]

Response should be able to receive a ReadableStream according to w3c/ServiceWorker sw-readablestreams and MDN/Response:

return new Response(ipfs.files.catReadableStream(data.multihash), {
  ...headerOK,
  headers: { 'Content-Type': mimeTypes.contentType(mimeType) },
})

But now Response() seems to stringify the ReadableStream it received, resulting in a [object Object] send to the web page.

It's strange. Maybe stream returned by ipfs.files.catReadableStream is incapable with Response?

-- Update --
Seems not, catReadableStream is using pull-stream-to-stream and returning a Stream instance. So I tried a plain old stream:

let count = 0;
const stream = new streams.Readable({
  read(size) {
    this.push('foo');
    if (count === 5) this.push(null);
    count++;
  },
});

respondWith(new Response(stream, headerOK));

It is not working too. Response is just a [object Object].

[linonetwo]

I found that, Response() in service worker only receives ReadableStream from ReadableStream API, but not stream.Readable from stream browserified. Their API is slightly different, like ReadableStream don't have stream.on('data', cb).

So in service worker, if we want to use catReadableStream directly, there should be a way to replace toStream https://github.com/ipfs/js-ipfs/blob/master/src/core/components/files.js#L295 with an implementation that uses ReadableStream Web API.
Another way is pipe stream.Readable from catReadableStream to a ReadableStream from ReadableStream API. I will try it later.

[vasco-santos]

@linonetwo I added an initial implementation of the service worker, based on the ipfs-service-worker code and it's previously PRs.

Sooner, I will create several Issues, dividing the next steps for this module in their own scopes. Then, I would appreciate that you can provide your research (for example regarding the Readable Stream) in the specific context and we will discuss the possible approaches in there.

Therefore, I will close this PR and we will focus on smaller tasks. Your codebase will still be available in this PR for being consulted.

[linonetwo]

OK, let's discuss in issues tomorrow.