Is Anki too hard to build? (Or, make life easier for packagers.) #1378
Comments
|
I'm really curious to see whether Anki is truly a cross-platform app now. |
|
Bazel solves a lot of issues that the old Makefile-based solution had. It's a fairly recent technology, and I expect over time packagers will get more used to working with it as it becomes more widely adopted. The savings in build/test time for daily development alone are considerable, and I'm afraid I do not think it makes sense to prioritise simplicity for distro maintainers over the needs of developers who work on the code each day. If you're looking for an easy way to install Anki without compiling it yourself, installing via pip is quite simple, and it can work with your system-installed version of Qt. |
I don't think you understand the implications of this. As OP mentioned, distributions are already dropping Anki from their repos. Arch Linux is the first, and I'm sure not the last. This means Anki will get a lot less users. Or, distros keep it but remain at an older version. Then users will miss out on all updates made and will only get security and crash fixes. Both situations are not something you should want as a developer. You should want your users to get the latest and greatest. Wanting to move away from Makefiles makes sense, but there are other build systems out there that can be used instead rather than Bazel that are much more distribution and developer friendly at the same time. Think of CMake and Meson for example. Especially the latter is getting a lot of interest nowadays, have you considered it? Note again that basically every distribution out there (except for FreeBSD) is shipping an outdated Anki right now, and I'm afraid that won't change as long as Bazel remains in use. |
I'm nobody but a newbie Arch user, and I just found out how to do that:
You can now run the installed binary. You can also create a
|
I'm not sure why you're acting like it's impossible to get Anki if it's not available on the user's distro repository. The packaged version for Linux worked just fine on all the distros I have tried. It has a simple installation and update process. |
|
I'm not saying or acting like it's impossible, but you should realize that most users will get applications from their system repositories. Note I said "less users", not "no users". Yes Pip packages will work for a few users which is nice, but in the end most people will want to do either In the end I'm just a passer-by and you should do whatever you guys think is best for Anki, as developers you always have the final say in everything of course, I'm just trying to warn you about the implications of using a build system like Bazel when clearly distributions are having lots of problems with it.
Of course it's a target you probably shouldn't care about, but you clearly haven't tried it on a Musl system like Alpine Linux/postmarketOS 😉 |
You mean this? https://github.com/ankitects/anki/releases/download/2.1.48/anki-2.1.48-linux.tar.bz2 It seems that this version uses bundled python and qt5? and bundled Qt5 will need a patch to use Fcitx5 input method. Plus, removed out of distros repos mean no sync for packages for mirrors. It really hurts if your country censors GitHub, etc. Oh, I forgot to mention that using the bundled package will cause a lot of problems like the Fcitx one and waste system resources, too. |
|
@PureTryOut , just FYI, @Vitorvlv is just another passer-by like you and me. I think he don't represent the dev of Anki, but discussion on the problem is welcomed anyway. |
|
I just tried to build anki-git from AUR. The build process seems scary (java + python + node + rust) but it does work and took 4m30s on our 40-core server. |
|
On my Gentoo system I can build it with In order to get things work what I have to do is basically remove all bundled shared library and bunch of hack with Indeed, for such a complex piece of software with multiple languages, there are few solutions besides bazel. But I definitely hope the build/release process can depend less on fetching random binary packages online. It can also be very time-wasting (like fetching entire rust toolchain even if I have a prefectly working one on my machine). |
|
I've been a Linux user for over 10 years, so I may not be completely representative. That said, as a user literally all I've ever had to do to install Anki on any Linux over that time period is download the Linux package/tarball from the website, extract it, and run Even though I'm currently using an Arch-based distro, which has one of the better user package repos out there, I have quite a few pieces of software that need some other means of installation than either the official repos or the AUR (pip/npm, snap, Steam, downloads of archives like Anki, etc.). There are just a lot of ways to install software on Linux right now, and you kind of have to deal with that. As far as Anki development goes, doing a full build from source is a bit of a pain in my experience, but I'll take @dae's word for it that it's more helpful for the folks that spend the most time on it than it is annoying for me. Also, @lolilolicon, I don't know how long you've been using Anki, but over those same 10 years most distros' packages of Anki have always been out of date, some of them horrendously so (e.g., so old they're incapable of syncing with the current version of AnkiWeb). Maybe the new build system has made things a bit more difficult, but it's not by any means like we went from everything being neatly up-to-date to everything being a mess with that single change. |
I have to add that in some countries like China and Vietnam, pulling source / binary / tarballs from GitHub or Amazon AWS, etc IS difficult if not impossible. You have to use obscure ways to proxy through governments firewall (besides, it might be illegal). It's harder to setup proxies for so many tools trying to pull toolchains / source code / submodules from everywhere, the speed will be horrible and this produces extra financial costs for simply networking. This is true no matter which platform you try to download / build and install to, except iOS. Linux distros' mirror servers hosted by universities and companies in the country have always been a better way to circumvent this kind of dilemma and it's a loss for Anki's (potentially or already?) big amount of users in China who need not to be a tech-savvy, and probably just a future writer / doctor / historian and simply want to use a reasonable open source language learning tool to learn EN, JP, RU, ES languages. I bet not all of them are rich and free enough to buy an iPhone and AnkiMobile to utilize Apple's possibly not blocked CDN. Most of them are just honest in-school students and don't even have a stable income yet. Shutting down another window for them to learn just makes me feel more horrible. I don't know which side (dev of Anki or packagers of distros or even the government which obviously won't give a shit about all of us) could do a better job to make the software more reachable for these teenagers, I only know that as long as PyPi still have a mirror in their country , it's possible to install all dependencies with package managers and install wheels of aqt and anki with pip. And all I can do is consider at least write up an AUR PKGBUILD for automating this or some textual instructions to make the idea clearer for them. |
ropery
changed the title
|
Replying to @dae
Thanks for the explanation. I hope it gets easier for everyone.
Understandable. However, it doesn't have to be either/or. I think it's hard to overstate the importance of the official distro repos for a Linux user. Bazel or not, there must be space for improving the build process for both upstream developers and downstream distro packagers.
It's good to have an alternative, but as @OceanS2000 demonstrates, even this doesn't always work. (And it's not just his/her "exotic" setup; other issues may arise, e.g. as reported by @escape0707 here. Installing an oversized binary bundle when the source is available is, in any case, a somewhat iffy last resort.) With so many distros falling hopelessly behind, let's face it: Anki has a packaging problem.
|
|
Replying to @sobjornstad
Happy for you, but this is irrelevant. By "hard to build" I mean specifically for distro packagers; personal ad hoc success isn't a proper argument.
Call me a neatnik, but I never install packages system-wide without a package manager. What's more, for regular open-source software like Anki, nobody should expect to have to install it without a package manager.
All the more reason to raise this discussion, don't you think?
I never said Bazel changed things for the worse. Looking at the build script for anki 2.1.35-3 in Arch Linux, which required 5 patches plus 3 additional source tarballs, it's clear that anki hadn't been easy to build even before Bazel. Imagine maintaining a build script that required substantial effort to get right, and then have the whole thing scrapped, only to be met with a brand new build system that looks like it will take even more effort on your part. In fact, my conjecture (not based on any evidence) is that perhaps Bazel was adopted to make Anki easier to build for packages, developers having realized Anki had been too difficult to build for packagers to catch up? After all, adoption by distros is crucial for Anki's wider adoption. (@PureTryOut's words sound harsh, and I'm not so confident Bazel is the problem, but he is correct in emphasizing the importance of the official distribution channels for a Linux user.) In any case, "it's always been pretty bad" is no argument for not making things better now. |
To be clear, I believe my case is very exotic here: nowadays nearly every Linux distro is using systemd unless you are running Gentoo or Slackware. Even on Gentoo, it is rare to set When I configure my system this way, I'm kind of prepared for such thing to happen and it is easy (for me) to make the binary release work on my machine.
To my knowledge, Bazel might be the best solution here. Anki uses Python, Rust and Typescript. And all 'packager-friendly' build tools (pip, cargo) for them are only designed for mono language repo. Other than Bazel, I can only come up with custom written Makefile or build scripts in python, which is arguably more difficult to maintain and more fragile when build environment changes. The problem for Bazel, as it is designed by Google, is usually meant to be used in a corporate's monorepo, where all code including all dependencies is nicely sitting at the same place. This means the usual way to deal with dependencies in Bazel is to just fetch it. However, nearly all distribution packaging guideline requires no network connection during build and no vendor dependencies. But it doesn't seem to be what Bazel designers worry about. I really appreciate the FreeBSD developers' hard work here. Indeed, as you can see nearly entire Bazel build definition is rewritten, and it is quite involving. But to me it is a good starting point for other distros' packagers including me. As for advice for improvement, I suggest maybe some of FreeBSD's patch to the build system can be adopted upstream. I need some more time to read through the FreeBSD package but I believe Bazel can be made better than Makefile solution. |
|
Just out of curiosity, can someone explain to me what is so hard about running a
and building Anki? Maybe I'm missing something here, but I'm under the impression that the drama has been cranked up to Over 9000! levels for no real reason. If this were about possible licensing issues, or the need to use Java, or the fact that Bazel is a derivative of a proprietary Google tool, or even because the previous build process was a mess and people are so fed up that they don't want to bother anymore (just a hypothetical scenario, I have no personal experience with building Anki before Bazel was introduced); I'd certainly understand it. But Bazel being "difficult for package maintainers to understand and use"? Really now? To the point that nearly all distros carry a terribly outdated Anki package? And some like Arch are now even dropping it from their repos altogether because "no one could understand how it works"? Again, it really seems like I'm missing the elephant in the room here, so I'd appreciate it if someone could kindly point me to it. Also, regarding this:
How is this a problem when most all distros distribute their packages in pre-built binary form? I can understand it for the few outliers (in terms of package management) like Gentoo, but why should Average Joe care how a package was built before it was given to them to install via their distro's package manager? Finally:
Weird. In my case the build only takes a couple of minutes on my 12 core desktop (Ryzen 3600). And how is the build process scary? Because it uses more than one language? Or because it uses multiple files to orchestrate the build process instead of a single 10k LoC makefile? |
|
@Nocifer I saw your comments on AUR and I agree most of them. But just a note here:
I believe he mentioned packaging guideline which means the regulations a packager writing the build script should conform to. I guess this is because package builder should be able to check authority and integrity independently and apply patches like disabling auto-update or something when needed. About anything else? I'm actually too inexperienced to packaging and writing PKGBUILD for |
|
On Thu, Sep 23, 2021 at 06:29:59AM -0700, Alexander Michalopoulos wrote:
> However, nearly all distribution packaging guideline requires no network connection during build and no vendor dependencies. But it doesn't seem to be what Bazel designers worry about.
How is this a problem when most all distros distribute their packages in pre-built binary form? I can understand it for the few outliers (in terms of package management) like Gentoo, but why should Average Joe care how a package was built before it was given to them to install via their distro's package manager?
This is a problem about reproducibility (and security). To be able to
reproduce a package, nothing should be changed over time. So random
numbers with fixed seeds (e.g. Python) and fixed timestamp (there is an
environment variable that many tools respect). Mostly importantly, no
Internet, because data from the Internet can change over time.
Finally:
> I just tried to build [anki-git from AUR](https://aur.archlinux.org/packages/anki-git/). The build process seems scary (java + python + node + rust) but it does work and took 4m30s on our 40-core server.
Weird. In my case the build only takes a couple of minutes on my 12 core desktop (Ryzen 3600). And how is the build process scary? Because it uses more than one language? Or because it uses multiple files to orchestrate the build process instead of a single 10k LoC makefile?
Not only more than one language, but there are four languages (two of
them I don't feel good about). And I don't know what's going on under
the hood. Quite a few files in ~/.cache are touched. Perhaps it has
downloaded a lot of data from around the world.
The output is mystery to me: what is //:dist and other // things? What
are those linux-sandbox and ex processes I saw in htop? Where did they
come from? Just built or from the Internet?
To be short, it's beyond my understanding, and goes into the world of
magic.
…--
Best regards,
lilydjwg
|
Because most distros try not to use binary release build by upstream. Instead, they build packages from source release on their own. For example, here is the build farm used by Ubuntu. So, if package maintainers of distro cannot build the package cleanly, they just have no package to put into the package manager! As for why distros doing so, there are a few reasons:
Firstly, it will not work on my computer. anki % ./scripts/runopt
INFO: Analyzed target //qt:runanki (0 packages loaded, 3 targets configured).
INFO: Found 1 target...
Target //qt:runanki up-to-date:
bazel-bin/qt/runanki
INFO: Elapsed time: 0.300s, Critical Path: 0.05s
INFO: 4 processes: 4 internal.
INFO: Build completed successfully, 4 total actions
INFO: Build completed successfully, 4 total actions
/home/<username redacted>/.cache/bazel/_bazel_<username redacted>/c121bf9bd26c8131616486fc51130231/execroot/ankidesktop/bazel-out/k8-opt/bin/qt/runanki.runfiles/pypi__39__stringcase_1_2_0/stringcase.py:247: DeprecationWarning: invalid escape sequence \W
return re.sub("\W+", "", string)
Traceback (most recent call last):
<traceback redacted due to containing home path>
from PyQt5.QtNetwork import QLocalServer, QLocalSocket, QNetworkProxy
ImportError: libgssapi_krb5.so.2: cannot open shared object file: No such file or directoryAnd it is because it try to fetch dependencies online instead of using locally installed one: anki % cd /home/<username redacted>/.cache/bazel/_bazel_<username redacted>/c121bf9bd26c8131616486fc51130231/execroot/ankidesktop/bazel-out/k8-opt/bin/qt/runanki.runfiles
runanki.runfiles % ls -alh pyqt5
total 8.0K
drwxr-xr-x 1 <username redacted> 204 Sep 24 00:24 .
drwxr-xr-x 1 <username redacted> 1.4K Sep 24 00:24 ..
drwxr-xr-x 1 <username redacted> 2.7K Sep 24 00:24 PyQt5
drwxr-xr-x 1 <username redacted> 106 Sep 24 00:24 PyQt5-5.15.2.dist-info
drwxr-xr-x 1 <username redacted> 100 Sep 24 00:24 PyQt5_sip-12.8.1.dist-info
drwxr-xr-x 1 <username redacted> 74 Sep 24 00:24 PyQtWebEngine-5.15.2.dist-info
lrwxrwxrwx 1 <username redacted> 94 Sep 24 00:24 py.typed -> /home/<username redacted>/.cache/bazel/_bazel_<username redacted>/c121bf9bd26c8131616486fc51130231/external/pyqt5/py.typed
lrwxrwxrwx 1 <username redacted> 97 Sep 24 00:24 __init__.py -> /home/<username redacted>/.cache/bazel/_bazel_<username redacted>/c121bf9bd26c8131616486fc51130231/external/pyqt5/__init__.py
runanki.runfiles % rm -rf pyqt5/*
runanki.runfiles % cd pyqt5
pyqt5 % ln -s /usr/lib/python3.9/site-packages/PyQt* .
pyqt5 % cd ../ankidesktop/qt
qt % ls
aqt bazelfixes.py runanki runanki.py
qt % ./runanki
... Now it works fineThis is why packaging guidelines are against fetching any binary dependency online. And in order to make the build adhere to these guidelines the build definition has to be patched.
Yes. It is mainly because (1) it is mostly designed for corporate monorepo and more importantly (2) there is few open source software using Bazel. To my knowledge, Anki is the first and only open source application designed for daily drive that using Bazel. So most distro maintainers out there just have no experience about it.
It is mostly dependent on your network connection speed. On my computer the most time is consumed by fetching rust toolchain (a ~250 MB download). If your connection is bad then it can take up to 10 mins(!) for me... Once everything is downloaded the actually build process is fast. |
Yes. To my knowledge the following are fetched and put in
Also, all build output are put into
These are Bazel's notation for build targets. Instead of using plain text labels as Makefile Bazel uses this url-like notation to cope with more complex project structure.
They are part of Bazel. Basically Bazel uses its own sandbox to enforce dependency visibility rules so during build it can only see files exported in It is a good feature for development as basically it gives you a |
|
@escape0707 Anyway, it's a valid concern and a reason to push against such a build process, but IMHO hardly a reason for keeping a package outdated for a year and a half or even dropping it completely from the repos. That's a tad too extreme.
Yes, I'm beginning to get it now that lack of strict reproducibility is the main gripe people have with Anki's use of Babel. But again, as I just said to @escape0707, dropping a package from the repos because it's not reproducible sounds a bit extreme, especially when 1) full system reproducibility is still a future goal and not an established norm, and 2) we're talking about an educational (aka casual) application like Anki and not a system critical tool, so a bit of leeway would be advisable.
I never said that distros use or should use the ready made binary releases from upstream, I said that distros distribute binary packages and that Average Joe isn't affected by what goes on behind the scenes in order for these binary packages to be built. Barring the concept of reproducibility, if a packager manages to build a proper binary package and distribute it to the users, then this package is frozen in time and the users will never be affected by such things as online fetching during build, etc. So in fact, dropping
This is a very valid point/concern against systems like Bazel, but at the risk of repeating myself too many times, that's not really a reason to drop a package from the repos, especially a casual educational application like Anki. Remember that the point of contest here is not really whether Bazel is a good tool or even whether Anki is making good use of it, but whether dropping
I'd assume that it not working on your computer has something to do with you using Gentoo and having customized libraries, as you described in your previous post. And as you also said, having a heavily customized Gentoo system means you know to expect such incompatibilities to occur from time to time. Still, trying to fetch even common system components like PyQt5 and the like is not good practice in my books, and now I begin to see why Anki is getting all this flak. It's one thing to fetch stuff like rust or npm dependencies, and another thing to fetch system components that can produce incompatibilities even in your typical binary distro.
Lack of experience does not mean the tool is hard to use, it just means that we lack the experience to properly use it and we must first learn how it works. The debatable point here is whether Bazel brings enough to the table to warrant package managers spending their valuable time to do so. Also, just for the record, there is at least Mozc which has also switched to using Bazel since a few months ago, and it's actually the application that first made me aware of Bazel in the first place.
Nah, that's can't be it. It's more probable that the person I was replying to made a typo and simply meant "4-core", not "40-core".
Now this is the question I really wanted to ask in order to make up my mind on whether it's Bazel or Anki's devs at fault here: does Bazel force us by design to use bundled resources instead of local, system-wide ones? If that is so, then indeed Bazel is an unsuitable tool for building Linux distro packages, and Anki's at fault for choosing to use it. If not, then it's only Anki's fault for not creating a build process that respects Linux packaging standards. So either way, it seems Anki is at fault :P PS: I looked a bit more into reproducibility and, given that this is the number 1 gripe most people seem to have with Bazel, you gotta love the irony that the one and only Platinum sponsor of the Reproducible Builds project is Google's Open Source Security Team (Bazel is Google software). |
Uh well, at least I can confirm that lilydjwg is a very experienced packager and Arch user who is a main manager of ArchlinuxCN community and the unofficial repo. I think by referring to the packaging machine's power she means at least fetching all the resources and build them is a painful problem for people with a less powerful rig, just the same point as you supported. Also, I don't think this will be the main reason for unofficial repos to refuse to package and distribute it, it's mainly security and packager's effort which are put into concern.
As of Though, I wish these packages could reside in the official repos, too. |
|
Replying to @Nocifer
You're going off-topic. The point of contest here is definitely not whether @kyrias made the right decision dropping the package. (I myself commend his disciplined approach.)
This discussion is specifically about making the packager's job reasonably easy. Be reminded that Arch Linux Trusted Users (and packagers of most distros) are volunteers. If the software makes unreasonable demands on their time & effort, it only makes sense to drop what is unmaintainable; the AUR exists to ensure the quality of the official repos, while giving users wider options and a place to collaborate. The big question with Bazel is whether Anki can reconcile the Bazel way of doing things with distro packagers' requirements of independent verification and scrutability of the build process, etc. |
It is 40-core, although there might be a concurrent build job to at most reduce the available CPU power to half. It's also in a Europe datacenter with good network connectivity. I didn't know that it fetched rust toolchain (the PKGBUILD requires the rust package). This would take a lot of time for mainland Chinese users if they don't setup a proxy / mirror (and since it's like a virtualenv thing, I don't know if setting these up in it is easy or not). Also, would it pick up npm / pip mirror settings for those users? |
|
Firstly, for some perspective: less than 3% of Anki users are on a Linux machine. For the other 97% of users out there on a Windows or Mac machine, binary packages are the norm. The comments about reproducibility in Bazel are somewhat off the mark. The dependencies Anki pulls in are pinned to specific versions, and the majority of them are checksummed, meaning changes to the remote files will result in the build aborting with a warning. The pinning is important - they are versions we have tested and know will work, and it is not uncommon for a slightly older or slightly newer version of a dependency to break things. The dependencies are downloaded during build because that is the most convenient - to do otherwise, you'd need to install all the required dependencies yourself, and any version you accidentally got wrong could result in things breaking. Having a packager do that for you makes things easier, but even they don't always get it right, and there have been many Linux-specific issues over the years caused by distros switching out a different version without testing it properly, such as https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=980344 There are no technical reasons preventing packages from overriding the dependencies if they wish. Bazel provides tools to override arbitrary dependencies with other versions in other locations. So this is not a case of packagers not being able package the app - it's just that it's more work than they want to do. That's not to imply they are being lazy - the reality is that modern software development practices are somewhat at odds with the way Linux distros have historically swapped out dependencies with their own, and you can see this playing out with other projects as well. I don't really think it's realistic to expect developers to stand still for the sake of packagers - the onus is really on the distributions to keep up with the times. It may take them some time, but they'll likely get there in the end, and I believe Debian already have a group looking into improved tooling for packaging of Bazel projects. |
Ah, sorry, I didn't even realize that the "40-core person" was @lilydjwg. I guess that if it were a typo she would have said so in her reply to me, so it's probably like @OceanS2000 said, that the stated time includes resolving dependencies over the network etc.
I didn't mean to imply that, it's just that while skimming over the comments it hit me as weird, so I commented on it for the sake of discussion.
Yeah, again, I only mentioned that for the sake of discussion. I'm not involved enough with the Arch community to know the whys and hows of what package gets a place in the repos and what gets left out. I mean sure, Bazel notwithstanding I too wish that the UT packages were included in the repos (would save me the bother) but it's not really that important, especially compared to other missing packages like e.g. lib32-gst-plugins-{bad,ugly}. @lolilolicon
Well, I thought the point of contest was whether Bazel as a tool is too convoluted to the point that package maintainers are not able to understand and use it and thus are forced to drop packages that make use of it, like in the case of Anki. That struck me as very weird (because using Bazel is not that hard) so I questioned it. Now, since it turns out that @kyrias (on a side note, I thought the maintainer was @demize?) dropped the package due to other issues like lack of reproducibility, security concerns, the online fetching of dependencies, etc, then that's another matter and I'm not about to debate the decision, not the least because I don't really care about it, but also because I don't have any delusions that I'm knowledgeable enough to debate it. As for the rest of what you said, it seems to me that my first hunch about "bad blood" was correct and I've inadvertently stumbled upon a long-standing internal affair/debate of the Arch community, and most people involved have already established some rather strong opinions on the matter, and I really have no desire to debate policies or established processes, so I'll kindly excuse myself out. @lilydjwg |
Glad to see there are ongoing actions on this: https://salsa.debian.org/bazel-team/meta I think the original goal of the discussion for sure cannot be meet now as it takes time for bazel to get integrated into Linux packaging tool sets, meanwhile Anki's development need this build system right now to move forward faster and easier. So for now, the measures that can be taken are:
|
|
Well, if now we can not make Bazel easy to work with, why not try to at least get something usable? This is my take: OceanS2000@e4c820c @dae Does you guys feel comfortable about releasing a source tarball containing all auto-generated source using this build rule along with pre-built binaries? I think at least it can provide something to work on before everyone adopts Bazel :) The following is a full description about my commit: This fork contains an additional build rule for a 'source distribution tarball'. The original repo uses Bazel to build, which has been causing some headcache for The idea is, a 'source distribution tarball' is created when releasing by bazel -c opt '//src-dist:src-dist'It will generate a tarball includes:
And then the software can be built without hassle of Bazel relatively. Why prebuilt TypeScript part?While we love all-source builds, vendoring some javascript code seems fine for How to continue the build process, then?You will need two snapshot from https://github.com/ankitects/anki-desktop-ftl Extract them into the same working directory, and then build the Rust library: work % ls
bazel-dist core-i18n desktop-ftl
work % BAZEL=1 \
PYO3_PYTHON=/usr/bin/python3.9 \
PROTOC=/usr/bin/protoc \
BUILDINFO=$PWD/bazel-dist/buildinfo.txt \
PROTO_TOP=$PWD/bazel-dist/proto/.top_level \
BACKEND_PROTO=$PWD/bazel-dist/rslib/backend.proto \
RSLIB_FTL_ROOT=$PWD/core-i18n/l10n.toml \
EXTRA_FTL_ROOT=$PWD/desktop-ftl/l10n.toml \
cargo build --manifest-path $PWD/bazel-dist/pylib/rsbridge/Cargo.toml --verbose --release
work % find . -name 'librsbridge.so'
./bazel-dist/target/release/librsbridge.so
./bazel-dist/target/release/deps/librsbridge.so
work % find . -name 'strings.json'
./bazel-dist/target/release/build/anki_i18n-15e40669e4befd1b/out/strings.json
work % mkdir usr-share-anki
work % mv bazel-dist/pylib/anki usr-share-anki
work % mv bazel-dist/qt/aqt usr-share-anki
work % cp ./bazel-dist/target/release/librsbridge.so usr-share-anki/anki/_backend/rsbridge.so # The name is significant!
work % cp ./bazel-dist/target/release/build/anki_i18n-15e40669e4befd1b/out/strings.json usr-share-anki/anki/_backendGenerate the python file from work % export PYTHONPATH=usr-share-anki
work % cd usr-share-anki/anki/_backend
_backend % python3 genbackend.py generated.py
_backend % python3 genfluent.py strings.json fluent.pyNow (hopefully) you can try it by To finish the build process, a launcher can be generated from #!/usr/bin/env python3
import sys
sys.path.append("@PREFIX@/share/anki")
import aqt
aqt.run() |
Now, this is not too fair comparison, is it.
Yes, Steam may not be the best place to search for this kind of data, but it says a lot. |
|
@Bracket-H: This does not refer to building Anki for different Platforms, but is a problem that refers to the direct installation of Anki, so I suggest you open a separate bug ticket. Apart from that, it would be helpul giving more information, e.g. what version you were trying to install, what system you are installing on, how you installed it (specific steps) ect. (In case you are interested in solving this issue rather than just complaining about it. ;) |
|
I got it to build from source. It created almost 4 gigabytes of bazel cache files in my home dir, that's almost as much as the entire system software package size of my distro. This leads me to believe that this project is now moribund. I can smell the soy latte and estrogen pills clearly now. |
|
The "unbloated" original version can be seen here, source code is 1.4 MB and pure Python code, the binary is 115 MB: https://github.com/ankitects/anki/releases/tag/2.1.15 Talking about solutions, it would be nice to keep that version compatible with current Linux distributions by backporting critical updates from time to time to that branch. There don't seem to be many anyways, as the 2.1.15 version is being used in distributions for years now, but sometimes there are updates to Python or Qt that break that version and need a small update. It doesn't help to become too emotional here. We rather give constructive ideas or better code ourselves. If there is no Anki version that is compatible with true open source Linux distributions, I am thinking of creating a simple Anki version that can be cross-compiled to all platforms including mobile, iOS and Android. |
I take great exception to the scare quotes around the unbloated word. I agree that it wasn't the tidiest code base, and that Python has a tendency to under-estimate the size of hidden dependencies. The reason is inconsistently applying them. For example: the word "solution" should be surrounded by at least three sets of scare quotes. The word "small" in that same parapgraph — probably fine with a single pair. As for getting emotional… this is kinda pointless. For one, I don't think that telling people that they're bad is allowing them to save face and mend their ways. To change back from Bazel would be admitting to all the unflattering epithets flung your way. If I made a terrible decision and was told off in an issue, I would not be very amicable about reverting it either, if people were as vocal there as they are here. As for being constructive: hard forking anki into a pure Python version and a version with Rust and Typescript add-ons is IMO the best option. Ideally, you should also split the code-base: making a common library that can interact with the Web, and handles the business logic, (including the card canvas) is enough. The pure Python package that can then be used by platform-specific implementations is going to help. You the core developers only care about the core library, the platforms don't care about each other. Sure you get 5-6 different code-bases all of which need to synchronise with the core, but these codebases are probably 5-6 times smaller, and definitely much easier to read/fix/maintain. If the core library still uses Bazel, but is available on PyPi, the distribution maintainers will probably make an exception. The development of the Linux binaries can (should) be offloaded onto the community. We'd be happy to take over the development, and make Anki conform to the Qt HIG. Win-win.
Good thinking. Since you've already done the hard part of creating a plugin system, I would imagine that stripping it down as far as it would go, and modularising it, wouldn't be hard. Plus it would solve the problems that you swept under the rug by using Bazel. But I don't think it's possible at this point. From the looks of it, the problems that you were having were due to massive technical debt and the fact that Python is an incredibly powerful tool with its barrel aimed directly at the developer's face. It's hard to fix. In an ideal world, you'd focus on fixing the technical debt, and get rid of Bazel, consequently getting Anki re-instated as a regular package. I don't think it's feasible, though, so I would settle for an alternative, which is what I mentioned in the previous section. |
|
You made me smile with the quotes, I share this kind of humor. :-D I am not the developer of Anki though and have never contributed any code. As far as I know it's mostly a one-man show and I just acknowledge and appreciate all the work that went into it from Damian. But I see some (a little) responsibility from a developer to not let people of a community down that contributes so much to Open Source, including what Anki is based on, especially as it doesn't seem like much extra work to do so. Those people are using Linux distributions that have security standards that don't allow certain things like pulling source code remotely during compilation. We need to have an Anki version that can be compiled offline. We want to install from the repository of our distribution (or "App Store", however you want to call it). One could just fork the 2.1.15 branch and keep that compatible with the latest Python and Qt version. That would be the most efficient way to do. It's actually so easy and little work that I don't understand why it is not done this way already by the developer, as a branch within the main repository. There is not much to be done to keep it up-to-date. Still waiting for an explanation from the developer at this point, or his opinion on this, if this is a possible way of keeping Anki in the Linux distributions. |
Then you do it.
You understand it's mostly a one man show and yet it's asking for a lts version of the software? This require people spending time to keep it up to date, however 'easy' it might be. What other one man show projects that you know about provide both a regular release and lts one? |
Sure. I am a Python developer but have no experience with Qt. If that is what needs to be done to keep Anki in the Linux repositories, I would give it a try though. But it needs Damien's approval to open a 2.1.15.x branch. I don't think a fork would be accepted in Linux repositories so easily, would it? Perhaps someone else can give an informed assessment of this.
Yes, because for someone who is involved in the project and knows the code, this seems to be not much work, given the small changes that need to be done to keep the code up-to-date with the latest Python and Qt versions. But I might be wrong. Damien could say more about it. |
I am, a Python/rust developer with experience in writing one-man projects and using PySide2 (Qt, but a different library). This should clue you in on why I'm not too eager to jump head-first and volunteer as a developer to maintain the hard fork.
I think that a better option would be to completely decouple the pure python version. Damien (from the looks of it) probably has his hands full. Moreover, I think that eventually Damien will come to his senses and abandon Bazel, which in turn probably means that his package will be reinstated in the official repositories. If the projects are independent — less confusion for end users.
It's not a trivial amount of work. Nothing compared to instrumenting a new build system, but still you should probably expect to dedicate something like 1-2 hours a week just for maintenance, and 3-4 hours for actively developing it. More if you're not experienced with Qt, and significantly more if you want to maintain a Qt 6 version. |
|
I'm a bit saddened by some of the comments that have been posted here. To me at least, "free & open source" means "here's this software I and others wrote that you may find useful. It's free to use, and here's the source code so that you can modify it if you wish". Constructive feedback is welcome, but rude and demanding comments can really make one wonder why they bother sometimes. Bazel is not without its faults, but in terms of making Anki easier for users to build, I think it does its job pretty well. Something like 80% of Anki users are on Windows, and Bazel has made building Anki on Windows much easier than it used to be. Even on Linux, building is simpler. Yes, it downloads a bunch of stuff that you may or may not have system versions of already, but it means you don't have to go through the trouble of installing all the correct dependencies yourself, and it will build on many distros that do not have the correct dependencies available in their repos. And for people who just want to run Anki and have no interest in building it from source, there are packaged binaries and Python wheels available, that are far smaller to download and run. Earlier in the thread, I pointed out that less than 3% of Anki users are on Linux (taken from the number of syncers, not number of downloads). While I'm sure most Linux users would prefer the convenience of a distro package if one were available, the majority of them seem to be able to make do with the packaged version that is provided, and I would hope that a reasonable person could see how suggestions like switching back to a "pure" Python codebase, undoing the last two years of hard work, is not really a reasonable proposition. Likewise for resurrecting a two year old release and maintaining it in parallel - I'm pressed for time as it is, and I don't think it's a particularly good idea, as add-ons and the collection format have moved on already. The convenience of package managers is not lost on me - I have been using Linux for approximately 25 years, and used it as my primary desktop for about 15 of those years. But as has been mentioned in a previous reply, the "switch out all dependencies with our own" approach can introduce problems, and there were numerous occasions where Anki lagged behind or was broken in distros due to mismatched or missing dependencies in the past, even before Rust or Bazel were introduced into the codebase. Ultimately, I still think that this is a problem distros need to solve, either by adjusting their policies and/or improving their tooling. In the mean time, while I don't think suggestions like rewriting the build process or app are really practical, if there are specific parts of the build process where a flag such as the existing PYTHON_SITE_PACKAGES would make switching out dependencies easier, odds are a clean PR that implemented it would be accepted. In any case, I think this discussion has probably run its course. If you have something new to add that is not just rehashing the above, please do so over on the user forums. Thank you to all of you who contributed constructive feedback, and I hope you all have a happy start to the new year. |
That is half the packages in the git repo. But yeah the big ones are not python. Sadly your instructions imply I should first install anki-git but are not clear. I have been burnt too man times by anki. |
|
Stopping by to say that Bazel essentially rules out Anki entirely for me, which is quite sad as I made good use of it while learning Japanese. Bazel is a utter nightmare, and with its involvement I cannot use, contribute to, or package Anki. As far as I'm concerned the project is not useful until it is moved to a better system. |
|
FYI the [archlinuxcn] repo has packaged anki-git and in the meantime I have wiped bazel's cache at least two times to resolve compile errors, and I've just done that one more time because it's been failing recently again. |
|
I'm giving up bazel's cache and will clear its cache each build. Besides the poisonous cache, it's not quite obvious how to remove the cache since some directories are read only. Here's the command I use: find bazel -type d -exec chmod u+w {} +
rm -rf bazel |
|
Looks like bazel is out, ninja is in. #2202 |
|
🎉 |
|
Triumph of reason |
|
And here we are: Due to Anki's security issue of its build process (can't be compiled offline with reasonable afford), now Anki has been kicked out of the popular Linux distribution Fedora. |
|
@criatle808 Do you have a link for that? |
|
Yes. Here is a little collection of sources:
Anki has been kicked out with the start of the current Fedora version 38. |
|
@criatle808 Was there any discussion regarding the migration from bazel to ninja build system (#2202)? The original problem (bazel) started 2 years ago, and the migration from bazel happened only half a year ago. Perhaps the new setup will be easier to package? Regarding the general "downloading the dependencies during the build process via pip, npm and rustup" - as a developer that regularly uses three of these tools I don't see any easy way to avoid doing that. Many Python libraries are packaged as RPMs, allowing the RPM to depend on them directly. However NPM libs and Rust crates are usually not packaged, and making a separate package for each lib and crate will be a daunting undertaking since NPM and Rust dependency trees are often even bigger than Pythons'. I understand that it would be nice for package maintainers if everybody was using zero dependencies and/or coded in plain C, however that's not a sustainable assumption. |
|
I'm not a package maintainer myself, but there are many other packages in Fedora which are using Rust and Python, with a huge amount of dependencies, and building them for Fedora is obviously possible. So Anki's issue of being built easily enough is related to some specific decisions made by the developer, as far as I know introducing several other programming languages into the package (before it was almost only Python), and using Bazel. There was a discussion about it, but as long as there is no Anki package that is easily enough to be built offline, there is no Anki in Fedora. (And several other Linux distributions kick it out as well for the same reason, by the way.) In addition, Damien @dae, the main developer, has stated (also in this thread) that he won't give the requirements of those Linux distributions a high priority, and a distribution normally only accepts packages that it's user's can rely on to be available in the long run. As long as @dae doesn't state that he will consider the requirements of those distributions, they probably won't accept Anki. |
|
Can you give an example of some package that uses Rust with many dependencies? Bazel went out of the window, so that problem is gone now. |
|
I know there are are Python packages with a huge amount of dependencies in Fedora. And there are Rust packages, but I don't know about their dependencies. The essential point though is: Decisions about the technology stack of Anki lead to it being removed from Fedora. And as long as there is no stable version of Anki with a compilation process that is acceptable by Fedora, with the reasonable expectation of being stable in the long run (starting with the developer taking it important), it won't get back in. |
|
With the new new new build system, Anki is feasible to build fully offline with a moderate amount of patching. @euank packaged it for Nixpkgs: https://github.com/NixOS/nixpkgs/blob/c8f6370f7daf435d51d137dcbd80c7ebad1f21f2/pkgs/games/anki/default.nix |
|
That is basically good news, but now that Anki has been kicked out of Fedora, I have little hope it will make it back in soon. Distributions also take reliability of package availability important. Anki's developer not considering Linux distributions' requirements as important, he might take decisions in the future that ignore the same or other requirements of said distributions. Therefore, Linux distributions can't rely on the stability of Anki as long as the developer doesn't commit to their requirements. |
|
Update: Anki 24.04 includes support for offline builds: 42cc2c9 thanks to the FreeBSD package maintainer. |
I don’t think the package will return any time soon. And not for technical reasons. Any packager that sees this entire string of discussions is liable to think that no lessons had been learned. And indeed to hammer home the fact that package maintenance is an important aspect of software distribution, I wouldn’t be surprised if this package were made an example of. The steam deck came out. Anki is installable via flatpak. There is a reasonable argument to be made, that it is not worth the effort to try and allow anki into the official repos, given that it is both flatpak and pip installable. |
Our package maintainer decided to drop anki (having been stuck at 2.1.35 for months) from the official Arch Linux repos to the AUR, with the reasoning:
If you look at the anki packages in different distros, the majority are badly out of date.
The only exception is FreeBSD games/anki, whose maintainer is particularly tenacious; but if you look at the build files, and the packager's comments, it appears awfully tedious.
Edit: In the case of NixOS, they provide two versions of the package: the default build-from-source one stuck at 2.1.15 and anki-bin at 2.1.47 (= no building & uses official binaries & larger installation size).
So, is Anki too hard to build? Are packagers held back because of it?
The text was updated successfully, but these errors were encountered: