Legacy Extensions Database #303
Comments
From Legacy Add-on Support on Firefox ESR | Mozilla Add-ons Blog (2017-10-03):
RelatedJustOff/ca-archive: Catalog of classic Firefox add-ons ... |
|
@grahamperrin - I'm moreso inquiring on the devs plan regarding extensions. Not sure if I overlooked something in the provided links, but they just appear to be stand-alone attempts at salvaging the extensions database. Palemoon has a few integral extensions that appear to be self-hosted: https://addons.palemoon.org/extensions/privacy-and-security/ Will Waterfox follow suit with something similar? |
|
AMO source code is available, but it's tied with Mozilla's servers (see here). I'd love to have it running myself but am unsure of the effort required in changed the accounts system. If anyone has extensive django experience and can take a look, that would be really helpful. Still focusing on polishing v56 and the new website. |
|
@MrAlex94, Am I correct in understanding the hurdle is the user authentication for a tie-in to Mozilla's back end? Why not a custom solution & have devs (or users) upload the extensions to the Waterfox's database themselves; make a public note as to whether it was community provided (not supported, but available) or developer added (supported)? With the antics Mozilla has been pulling as of late, is there any way to break away from the Mozilla dependency & have Waterfox be a stand alone browser? |
|
Well, it's more that setting it up there's no way for a user to create an account. And a custom solution needs to be adaptable, to make sure any future features (let's say Waterfox Sync etc) work with the created accounts.
It already is. You can remove any Mozilla parts and it will still work.
Not sure I understand this part. Doesn't matter as we'd have our own fork of AMO essentially. |
Do you have a list of what's a per-requisite?
The extension downloads & updates still rely on mozilla.org; once they delete the legacy extensions next year, everything dies, correct?
Dependent on their services, (or am I overlooking something)? That's why it's an API rather than a 'scrape' of the extensions database. After they purge all legacy, people will only have access to webextensions. |
|
Firefox accounts is open source you just have to set it up, remove any data collection from the project, then have Waterfox accounts server this will allow you to have a Waterfox sync server away from Mozilla with same telemetry code removed. This will also alleviate any issues with a legacy extensions database as can use your Waterfox account to login. The Mozilla addons sites open source, you can pull a fork before this ugly new one, then just a matter of getting devs, people etc to summit addons. Later implement a community moderation so bad addons are removed fast leaving good ones to really grow. Mozilla addons was toxic but this system will work better. Resource wise having a donate button on Waterfox Addons website maybe some non intrusive adds approved first by Waterfox community will help it grow. addons.waterfox.org would work good |
|
@PandaCodex That's the gist of what I've been trying to suggest, I'm just not being clear enough :-) If I understand you correctly, this "api" offered by Mozilla isn't actually an api in the traditional sense where you hook into their database to obtain content? Non-obtrusive ads and donate options for this service are more than acceptable solutions to the overhead something like this will undoubtedly generate. |
|
Maybe store the addons decentralized, like with ipfs |
|
Somebody should scrape addons.mozila.org - addons included - without that 90% of old addons will be lost forever in June. Necessary but probably illegal ... |
|
Cross reference Add-ons website for Waterfox : waterfox |
|
@ilu33 The only thing illegal about it is destruction of original works slated for June. 😠 I get it, it's their museum, they can toss all of the original masterpieces into a bonfire because you've got them on display there. Granted, this isn't a museum, it's a private company that has ToS that allow them to burn all of the original masterpieces to make room for the gallery of lost potential and crippling restriction. Prudes. 🍳 😞 |
|
Mirroring the AMO site in its present state and keeping it as static content is really urgent. Quite some addon authors are already pulling their code, so this should be done as quick as possible. Isn't someone around who has the technical knowledge to do so? It shouldn't need too much space. Login system and any fancy stuff still has time but the addons themselves will get lost if nobody does anything about it. There used to be a mirror in Belgium but they deleted everything some years ago (2012?) - it was 34 GB at the time. So mirroring at least was legal. I can't find any mirror still operating. |
|
The most important step before anything else is definitely to just have a
static copy of the AMO database. I just have to figure out a way to do it
by June and without causing any troubles.
…On Wed, 7 Feb 2018 at 00:41, ilu33 ***@***.***> wrote:
Mirroring the AMO site in its present state and keeping it as static
content is really urgent. Quite some addon authors are already pulling
their code, so this should be done as quick as possible. Isn't someone
around who has the technical knowledge to do so? It shouldn't need too much
space. Login system and any fancy stuff still has time but the addons
themselves will get lost if nobody does anything about it. There used to be
a mirror in Belgium but they deleted everything some years ago (2012?) - it
was 34 GB at the time. So mirroring at least was legal. I can't find any
mirror still operating.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#303 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AEgoWKkhlsNB-kZNcRgm06Dyt8ghWdKZks5tSOM2gaJpZM4Q4bw1>
.
|
|
belnet.be deleted their mirror in 2014. I have contacted them last week but so far haven't received an answer. Just fyi. |
|
Does anyone have an estimate of the amount of space this mirror will take? If it is not outrages I can very likely just download everything and store it somewhere safe. It should not be difficult and it should be legal to use the API do to so based on this comment: https://blog.mozilla.org/addons/2017/10/03/legacy-add-on-support-on-firefox-esr/#comment-224382. |
|
I was mistaken in my last comment, the belgian mirror was about 34 TB some time in 2014, but it was ftp.mozilla.org and I checked it and I don't think the addon site is even part of it. So no, I can't say. |
|
@JustOff created a extension (https://github.com/JustOff/ca-archive) that indexes the classic add-ons. Inside that extension is a easy to work with sqlite3 database, the total size can be found by using The url's for downloadeding the extensions can be easily extracted from the database, I will very likely just attempt to download all the extensions over the next week or so. |
|
The latest published version of CA Archive contains a list of all XUL extensions for Firefox from the AMO release channel as of December 12, 2017 and I plan to update this database further. I also have a full local copy of all these files with a total size of 28,226,401,643 bytes ordered in a two-level directory tree based on their digital id. If someone has the ability to provide a public mirror for this data, it can be easily connected to my add-on when Mozilla finally removes these extensions from their servers. However, there is no point in hurrying, there is still a lot of time until August 2018. |
|
@JustOff, I will happily host it. I'll boot up a bare metal server for the AMO server and then put the archives up on the CDN I use. On a related note, how easy do you think it would be to import all the archives into an AMO server? |
|
@MrAlex94 I don't know if this helps but have you heard about serverless hosting? |
|
Maybe host the add-ons on something like zeronet |
|
Please backup and host the files as of Dez 12 and please don't update later ... an increasing number of addons get replaced with non-functional versions, trying for webext compatibility without changing names. For example take xmarks sync addon: the new version 4.5.0.8 from Dez 4 offered for FF >= 48 is not only disfunctional (as you can see from 3 pages of 1star comments for a once 5stars addon) but also the feature to use your own sync server (webdav space) got removed. I already wondered why sync wasn't working anymore - luckily I could downgrade because old versions are still there. Edit: And the latest version has a new maintainer - that smells fishy. Palemoon forums have some more examples. |
|
@MrAlex94 : Please be careful with hosting, not all addons are published under free licenses. For example xmarks is not. It pains me but it must be said: Authors granted a distribution license only to Mozilla.org (https://developer.mozilla.org/de/Add-ons/AMO/Policy/Agreement). Downloading is allowed but hosting needs a license. I don't wish you to run into trouble. Edit: I don't know about just mirroring. But the license issue has to be adressed. Maybe a temporary solution would be to copy the website (which is under free license except trademark and design, https://www.mozilla.org/en-US/foundation/licensing/) with the code of the free addons and only link to unfree addons on AMO. And then ask the community to nag their favourite addon authors to consent to hosting for waterfox. @JustOff : (edited) I see your database include entries for licenses. During the publishing process the developer can choose a license from a dropdown list (https://mdn.mozillademos.org/files/15798/Submit_add_on_describe%20add_on.png) but it seems that also individual entries are possible :(. MPL, GPL, MIT and BSD allow distribution. The category "Multiple" and "Custom" probably have to be reviewed manually (f.e. HTTPS Everywhere, which, if you check, is free of course - Edit: Actually Its the only occurrence of "Multiple"). Edit: See my issue 5 on your project. |
|
@ilu33 you can checkout the database for yourself, just download the extension at https://github.com/JustOff/ca-archive/releases/download/1.1.0/ca-archive-1.1.0.xpi and unpack it. Inside is a sqlite3 database with a fairly straightforward schema. |
|
Yeah, I just did and looked at it. I heavily edited my earlier comments afterwards. Regarding the data that should be scraped from AMO website I think the user comments/reviews are very valuable. It seems you cannot access them from @JustOff 's adddon. Reviews regularly include valuable data about the trustworthyness of an addon which you otherwise can't figure out easily. For example the change in ownership of stylish in Jan 17 which started intensive datamining within a very popular addon. You wouldn't know that without reviews. Just note, my comments are in no way meant to criticize anybodies efforts, I just try to be as helpful as I can. I'm no dev. |
Just FYI: Most of that information can be scrapped from https://web.archive.org if push come to shove, they have an API that makes discovering what is there fairly easy and it possible to download the entire site at any given point in time. Trying to access addons.mozilla.org via the archives web's interface is not working very well right now but the data should be there. If we can capture it via the AMO API now, that would be better of course, but I have not looked if that is possible. |
|
Oh I forgot, sorry |
|
@MrAlex94 I just seen the new Waterfox website and noticed the buymeacoffee link. I have purchased a few coffees. Thanks for your work. 👍 |
|
From Discontinuing support for beta versions | Mozilla Add-ons Blog (2018-02-28):
– and support has ended. (I was aware of the countdown, but forgot about it when I pasted a beta URL to https://www.reddit.com/r/waterfox/comments/888u0u/-/dwjitrg/.) Do we (subscribers to this issue) have a reasonably recent 'snapshot' of the database that included beta versions of add-ons? Side noteIf I recall correctly, Mozilla's staging area sometimes used an outdated base so I wondered whether https://addons.allizom.org/firefox/addon/lastpass-password-manager/versions/beta would find anything. It does not. |
|
Yes, I’ve got a copy of the database, will need to upload it somewhere.
Been busy but it’s on the priority list.
…On Sat, 31 Mar 2018 at 08:41, Graham Perrin ***@***.***> wrote:
From Discontinuing support for beta versions | Mozilla Add-ons Blog
<https://blog.mozilla.org/addons/2018/02/28/discontinuing-support-for-beta-versions/>
(2018-02-28):
… discontinuing support for this feature in the next month. …
– and support has ended.
(I was aware of the countdown, but forgot about it when I pasted a beta
URL to https://www.reddit.com/r/waterfox/comments/888u0u/-/dwjitrg/.)
Do we (subscribers to this issue) have a reasonably recent 'snapshot' of
the database that included beta versions of add-ons?
Side note
If I recall correctly, Mozilla's staging area sometimes used an outdated
base so I wondered whether
https://addons.allizom.org/firefox/addon/lastpass-password-manager/versions/beta
would find anything. It does not.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#303 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AEgoWAuYRlXXGc6STJnOO2fvpBBlWMhgks5tj3mRgaJpZM4Q4bw1>
.
|
|
👍 thanks! Not chasing, just wondered. There might be edge cases, e.g. legacy extensions that were removed by authors before the copy was made, but nothing to worry about. |
|
From https://redd.it/8qsnqx (2018-06-13):
|
For what it's worth, I don't expect to know (or guess) the address(es) until after the release of a companion version of Waterfox. |
|
Deadline in one month, incoming: https://blog.mozilla.org/addons/2018/08/21/timeline-for-disabling-legacy-firefox-add-ons/ |
|
It could be just me but, what about the NPAPI Plugins? I know that the ESR, Pale Moon and Waterfox still supports them as well but, the ESR will be stopping this eventually. I'm hoping Waterfox will continue their support afterwards. ~Ibuprophen |
|
Briefly (on vacation, two weeks): recently in IRC for AMO I was advised that `.xpi` files will remain available after legacy content disappears from the front end.
Postscript: correction below.
|
|
In
The reply (2012-08-13):
From blog post commentary (2018-08-27)
@jvillalobos please, can we clarify:
NB just the files, not front end listings. I'm not attempting to sway any decision, simply seeking clarification on what may have been a misunderstanding in IRC. (Sorry for not seeking this sooner. I was on vacation for a few weeks.) Again, TIA |
|
The files will be disabled and won't be available anymore. That's because we're disabling versions, not listings. So, a listing with some legacy files and some WebExtensions files will have its legacy files disabled. A listing will all legacy files will no longer be available, only because all of its files are disabled. |
|
@jvillalobos thanks. Thanks also to Alex for yesterday's blog post about Waterfox 56.2.3. Relevant point:
Side note: (pre-integration) version 1.1.4 of the Classic Add-ons Archive extension (from the releases area of https://github.com/JustOff/ca-archive) does not use the Waterfox CDN when added to 56.2.2 or 56.2.3. |
|
1.1.5b1 also still points to AMO, making it pointless IMO. It should be easy to add an user editable option to specify the root domain to be referenced, then all links would use that. Although I still think using an extension is a bad idea. |
That beta served a specific purpose.
Where, now is the better alternative? |
|
|
Is this issue still help wanted, or can it be closed? @angela-d does the most recent release of Classic Add-ons Archive work for you? |
|
@grahamperrin Sorry, I missed this earlier, I haven't been logging into Github too often, lately. Yes, ca-archive-1.2.1.xpi works on Waterfox for me. My only gripe would be that it's opening in a new window, rather than a new tab.. so it ignores the preferred tab/window settings. Extremely awesome, otherwise!! |
That's normal, it avoids the incompatibility with multi-process Waterfox. To gain compatibility, switch to single-process mode:
|
Forgive me if this has been asked & answered already; nothing came up in the queries I tried:
Once Firefox ESR has transitioned to WebExtensions, what's to come of the "Legacy" add-ons in Mozilla's database?
Judging by the recent shady actions of Mozilla and even some extension devs, in favor of google; is one to believe Mozilla will keep the legacy add-ons in their database, so users can continue to use them in Waterfox?
Will Waterfox be creating its own extension repository, so it doesn't rely on Mozilla anymore?
The text was updated successfully, but these errors were encountered: