Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OPDS endpoint for categories #518

Closed
rgaudin opened this issue Aug 15, 2019 · 17 comments · Fixed by #492
Closed

OPDS endpoint for categories #518

rgaudin opened this issue Aug 15, 2019 · 17 comments · Fixed by #492
Assignees
@veloman-yunkan
Labels
enhancement kiwix-serve
Milestone
10.0.0

Comments

@rgaudin
Copy link
Member

rgaudin commented Aug 15, 2019

In order to improve ZIM files filtering on readers, we need to maintain a centralized list of Categories to which ZIM files are attached to. This has been mentionned in kiwix-lib#130 and kiwix-desktop#193.

The Categories list will start with the followings and should support internationalizations (en translations bellow).

  • gutenberg: Gutenberg
  • phet: PhET
  • psiram: Psiram
  • stack_exchange: Stack Exchange
  • ted: TED
  • vikidia: Vikidia
  • wikibooks: Wikibooks
  • wikinews: Wikinews
  • wikipedia: Wikipedia
  • wikiquote: Wikiquote
  • wikisource: Wikisource
  • wikispecies: Wikispecies
  • wikiversity: Wikiversity
  • wikivoyage: Wikivoyage
  • wiktionary: Wiktionary
@dattaz
Copy link
Contributor

dattaz commented Sep 6, 2019

I suggest youtube : Youtube

@rgaudin
Copy link
Member Author

rgaudin commented Sep 6, 2019

Is Youtube a category in itself ? To me it’s a scraper that gives access to content and we could have content-specific categories later-on.

But maybe we want to add another tag to represent that it’s mostly video content (different to videos:yes which only express a capability) ?

mgautierfr referenced this issue in kiwix/kiwix-desktop Jan 30, 2020
@mgautierfr
Revamp categories handling.
744cbae 
Move categories list in `static_content.cpp`
We use two "string" to handle each category. The first one is the
tag to use to do the opds request. The second one is the one displayed to
the user. The later is translatable.
The list is taken from issue kiwix/kiwix-tools#317

Fix #193
mgautierfr referenced this issue in kiwix/kiwix-desktop Feb 5, 2020
@mgautierfr
Revamp categories handling.
6eb690b 
Move categories list in `static_content.cpp`
We use two "string" to handle each category. The first one is the
tag to use to do the opds request. The second one is the one displayed to
the user. The later is translatable.
The list is taken from issue kiwix/kiwix-tools#317

Fix #193
mgautierfr referenced this issue in kiwix/kiwix-desktop Apr 7, 2020
@mgautierfr
Revamp categories handling.
180b65e 
Move categories list in `static_content.cpp`
We use two "string" to handle each category. The first one is the
tag to use to do the opds request. The second one is the one displayed to
the user. The later is translatable.
The list is taken from issue kiwix/kiwix-tools#317

Fix #193
mgautierfr referenced this issue in kiwix/kiwix-desktop Apr 7, 2020
@mgautierfr
Revamp categories handling.
e7a7ae1 
Move categories list in `static_content.cpp`
We use two "string" to handle each category. The first one is the
tag to use to do the opds request. The second one is the one displayed to
the user. The later is translatable.
The list is taken from issue kiwix/kiwix-tools#317

Fix #193
mgautierfr referenced this issue in kiwix/kiwix-desktop Apr 8, 2020
@mgautierfr
Revamp categories handling.
d775b2e 
Move categories list in `static_content.cpp`
We use two "string" to handle each category. The first one is the
tag to use to do the opds request. The second one is the one displayed to
the user. The later is translatable.
The list is taken from issue kiwix/kiwix-tools#317

Fix #193
@kelson42 kelson42 assigned veloman-yunkan and unassigned mgautierfr Mar 6, 2021
@kelson42
Copy link
Collaborator

kelson42 commented Mar 6, 2021 •
edited
Loading

An other point is probably the translation of the labels. The API end point should allow to specify a language. See kiwix/kiwix-tools#59 to store translations.

@kelson42
Copy link
Collaborator

kelson42 commented Mar 6, 2021

Regarding the storage if these categories, I believe for now we can harcode them in the libkiwix. Each time the kiwix-serve would be build with a new linkiwix, then new categories would be made available.

@veloman-yunkan
Copy link
Collaborator

What format should be used for the response of the categories endpoint? XML? Something like the following?

<categories>
  <category>wikipedia</category>
  <category>ted</category>
  ...
</categories>

@kelson42
Copy link
Collaborator

kelson42 commented Mar 7, 2021 •
edited
Loading

@veloman-yunkan I believe this has to be more complexe because the label can be different from the Category ID. On the top of that we should be ready to provide localised versions of categories, see kiwix/kiwix-tools#357 and kiwix/kiwix-tools#358.

Maybe something like this:

<categories>
  <category id="wikipedia">
    <label lang="en">Wikipedia</label>
    <label lang="fr">Wikipédia</label>
  </category>
  <category>ted</category>
  ...
</categories>

@mgautierfr
Copy link
Member

We should use the opds specification as far as possible.

  • It is possible to have several acquisition feeds (to list book) and navigation feeds (to list feeds) in a opds catalog (https://specs.opds.io/opds-1.2#2-opds-catalog-feed-documents)
    We may/should use a acquisition feed per category and list categories/feeds in a navigation feeds.
  • We can use facets to search/list book. As you can see in the example here (https://specs.opds.io/opds-1.2#4-facets), it is possible to regroup facets (and such declare which facets are categories) and provide a title different from the id (for translation).

@veloman-yunkan
Copy link
Collaborator

We should use the opds specification as far as possible.

  • It is possible to have several acquisition feeds (to list book) and navigation feeds (to list feeds) in a opds catalog (https://specs.opds.io/opds-1.2#2-opds-catalog-feed-documents)
    We may/should use a acquisition feed per category and list categories/feeds in a navigation feeds.

  • We can use facets to search/list book. As you can see in the example here (https://specs.opds.io/opds-1.2#4-facets), it is possible to regroup facets (and such declare which facets are categories) and provide a title different from the id (for translation).

@mgautierfr Thanks for the information. I have the following questions (cc @kelson42):

  1. It seems that currently we are using the older (1.0) OPDS spec back from 2010 (at least that is what our feed's opening tag states). The older spec supports navigation feeds but not facets. Can we upgrade to OPDS 1.2?

  2. Our catalog root is an acquisition feed. We will have to change it to navigation feed. Is that OK?

@kelson42
Copy link
Collaborator

I have no string opinion on this, will let you discuss and make a decision with @mgautierfr.

@kelson42
Copy link
Collaborator

@mgautierfr I have the feeling you should give your informed feedback here and maybe even make the decision.

@veloman-yunkan
Copy link
Collaborator

@mgautierfr Just a reminder, that this is blocked pending feedback from you

@mgautierfr
Copy link
Member

It seems that currently we are using the older (1.0) OPDS spec back from 2010 (at least that is what our feed's opening tag states). The older spec supports navigation feeds but not facets. Can we upgrade to OPDS 1.2?

Yes, we can upgrade to OPDS 1.2

Our catalog root is an acquisition feed. We will have to change it to navigation feed. Is that OK?

No. It is somehow explains in #209 (but not explicitly).

The ideas is to create another feed in parallel of the root.xml feed.
This way we don't break user of root.xml

Which name to used is still open.
It is somehow part of the whole project. Better understand the opds standard and design a proper api (set of url) to handle this. But we will probably keep root.xml as it is now.

@veloman-yunkan
Copy link
Collaborator

I propose to create the new OPDS API under /catalog/v2/. Current API endpoints under /catalog/ (root.xml, searchdescription.xml and search) will remain absolutely unaffected. /catalog/v2/root.xml will be an OPDS 1.2 navigation feed.

@kelson42
Copy link
Collaborator

@mgautierfr Any feedbakc here?

@mgautierfr
Copy link
Member

Sorry for the delay. I agree for /catalog/v2/.

@kelson42 kelson42 transferred this issue from kiwix/kiwix-tools May 13, 2021
@kelson42 kelson42 added this to the 10.0.0 milestone May 13, 2021
@veloman-yunkan veloman-yunkan mentioned this issue May 13, 2021
Merged
@kelson42
Copy link
Collaborator

kelson42 commented Jun 5, 2021

Discussing about this feed with @Popolechien and @rgaudin, it appears that the feed should probably not offer categories for which there is no ZIM available. @veloman-yunkan This is already the case in your PR? Should I open a new ticket?

@veloman-yunkan
Copy link
Collaborator

@kelson42 I don't think that we need another ticket for that.

@macgills macgills mentioned this issue Jul 13, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@veloman-yunkan veloman-yunkan

Labels
enhancement kiwix-serve
Projects
None yet
Milestone
10.0.0
Development

Successfully merging a pull request may close this issue.

Opds categories feed kiwix/libkiwix
5 participants
@rgaudin @mgautierfr @kelson42 @dattaz @veloman-yunkan