Add folder to specified tarball

[procount]

The tarball field is used to specify an external URL to download an image from. Often these external images do not have the default filename of partition.tar.xz. If tarball is absent, the image is assumed to be local and have the same name as the partition (e.g. root.tar.xz). But it could equally specify a different local image filename rather than the default, but in this case the complete path must be used (and therefore known).

This PR adds the image folder to the tarball if one does not already exist, so that alternative image filenames can be specified and found in their correct location (Useful if you want to have a local copy of one of these images in your SDcard /os folder without having to change the name to the default)

[lurch]

IIRC the local-tarball names are actually specified by the names in partitions.json, rather than being hardcoded to root.tar.xz - so there's already a way to use non-default names without having to rename files ;-)
See e.g. http://downloads.raspberrypi.org/data_partition/ and http://downloads.raspberrypi.org/openelec_pi2/

[procount]

Maybe I phrased it badly.
I didn't mean that root.tar.gz is hardcoded (that was just an example) Partitions.json specifies the partition names (label=), and that is used to find the image name by appending .tar.gz if tarball is absent. But in some cases, you might not want the image name to match label name, so by adding a tarball field you can specify a different filename to the partition label. But this must be an absolute path at the moment in order to work (just like a URL is an absolute path). But with this PR you can specify just an image name and NOOBS will prepend the image()->folder path just like it does for an absent tarball field.

(it is a small enhancement to give greater flexibility.)

[lurch]

so by adding a tarball field you can specify a different filename to the partition label. But this must be an absolute path at the moment in order to work

Oh, I'd never seen that feature actually used ;-) Does it make sense to be able to specify absolute local filepaths? It kinda breaks the idea of each OS being self-contained in its own os subdirectory doesn't it? Perhaps tarball should only specify an absolute URL, or a local (non-directory) filename?

(it is a small enhancement to give greater flexibility.)

Just out of curiosity, what flexibility does this give you that editing the label in partitions.json doesn't give you?

[procount]

If you look at LibreElec, the tarballs are as follows:
"tarballs": [
"http://releases.libreelec.tv/noobs/LibreELEC_RPi/LibreELEC_RPi_System.tar.xz",
"http://releases.libreelec.tv/noobs/LibreELEC_RPi/LibreELEC_RPi_Storage.tar.xz"
]
And in partitions_.json the labels match:
{
"label": "LibreELEC_RPi_System",
...
},
{
"label": "LibreELEC_RPi_Storage",
...
}
So yes, these partition names will be self-contained and rather unique, but also quite long.
You're right in that this feature isn't currently used and maybe the intention was that these 2 fields must always match, but they don't have to.

When I re-implemented Arch, I used labels of boot and root but the tarball is http://archlinuxarm.org/os/ArchLinuxARM-rpi-latest.tar.gz . I didn't fancy that as the name of a partition. (see https://github.com/procount/pinn-os/blob/master/os/Arch1/partitions.json) In fact there I have also included the tarball in os.json for the case where it is installed locally for offline use.

[procount]

Does it make sense to be able to specify absolute local filepaths?

I don't want to specify absolute local paths - the point of this PR is so that I don't have to. :)
It allows a local (non-directory) filename to be specified (but it doesn't exclude an absolute path)

And NOOBS already takes care of conflicting partition names for different OSes in case there is a clash (by appending an index number)

[XECDesign]

As far as I can tell, this change isn't backwards compatible, so distros taking advantage of this wouldn't work with older versions of NOOBS, which isn't ideal. Given that we only deal with distros which support NOOBS, I'm not sure that this is a good idea. Or did I get this wrong?

[procount]

You got it wrong 😉
It is totally backwards compatible.
It adds a small enhancement.
I obviously haven't explained it well enough as Lurch was also confused.
Let me try again, IIRC!

Remember there are 2 ways to install an OS: from the internet, or from a local source (SD card, USB etc).

1. When installing from the internet, it uses os_list_v3.json to identify the partition labels (via partitions.json url) and the tarballs to download (through the tarball entries). In this case, the names of the tarballs can be anything you like.
   In the case of LibreELEC, the tarball names are like: "http://releases.libreelec.tv/noobs/LibreELEC_RPi/LibreELEC_RPi_System.tar.xz".

Because, generally, the tarball name matches the partition name, LibreELEC partitions.json specifies the label name as "LibreELEC_RPi_System", which we realise from #460 is an invalid label name. For a remote install, we could change the label name to be valid in partitions.json and it will still install remotely because the tarball specifies the URL exactly.

2. For a local install, we have downloaded all the meta files and the tarballs to our local folder ready for installation. Now, if we had changed the partition label in partitions.json to be valid, a local install will no longer work because a matching tarball cannot be found. We would have to change the tarball name as well. To overcome this, we could add a tarball field to os.json to specify the original name of the file. But here's the problem: specifying a tarball in os.json only allows a URL or an absolute pathname. But I don't know the absolute path to the local tarball - it could be on SD or USB. What I want to do is to specify just the correct filename to use and let NOOBS figure out the absolute path to the file (i.e. where all the other metafiles are stored for this OS).

So this PR recognises a relative path in the local tarball field of os.json, and adds the absolute path to it so the file can be found. Existing tarballs that have absolute paths or URLs are unaffected. It can make it easier to download OSes for local installation.

This modification is included in PINN and works fine with all existing RPF OSes, whether installed remotely or locally. You may not need to use it at the moment since #460 has been merged, but it would allow you to manually shorten overly long partition labels like in LibreELEC and keep the existing overly long tarball filenames. It just provides more options.
HTH.

[XECDesign]

And what happens if you use current NOOBS with a PINN OS which takes advantage of this feature? Won't it fail to find the tarball?

[procount]

Correct - (only if you did a straight download of the /os from the repo and tried to install it locally). I think the only OS I modified in this way was Arch.
(That's a forward compatibility issue)

[XECDesign]

Right, that was what I meant when I said it wasn't backwards compatible. If we merged this, we wouldn't be able to use it.

[procount]

Because a user with an old version of NOOBS wouldn't be able to install a newer OS that used this feature, without upgrading NOOBS?

[XECDesign]

That is my understanding, yes. Given than NOOBS doesn't have an 'upgrade' feature, the limitation is that we can only break things if we really have to.

[lurch]

Yup, when I was working on NOOBS I tried my best to keep it both forwards and backwards compatible, i.e. older versions of NOOBS could still download & install the latest OSes (where applicable) from the internet (which is why there's separate os_list.json, os_list_v2.json and os_list_v3.json files), and newer versions of NOOBS could still download and install any old OSes (whether remote or local). 🙂

Unless I'm still misunderstanding it, this PR doesn't "fix" anything that couldn't be "fixed" by simply renaming the local tarballs you've downloaded? (which would seem to me to be the simpler option)

[procount]

Which is why I've closed it.

BTW - I looked at os_list.json, os_list_v2.json and os_list_v3.json and there seems to be some odd differences, like e.g. release dates and use of Pi Compute module. I guess they are there for a reason....

[lurch]

there seems to be some odd differences ... I guess they are there for a reason....

Yeah - it was all quite a while ago now, but IIRC some older versions of NOOBS aren't compatible with some of the newer OSes (I recall RiscOS being a bit problematic due to its non-standard partition handling) so the older versions of NOOBS are only compatible with certain versions of RiscOS, and they use os_list.json. Newer versions of NOOBS are compatible with both older and newer versions of RiscOS, and they use os_list_v2.json. And when a new OS is added to the server, there's then a script which updates both os_list.json and os_list_v2.json which knows which versions of the OSes are compatible with which versions of NOOBS, and so puts the newest version of each OS that that NOOBS-version is compatible with into the appropriate os_list file. If that makes sense... ;-)
e.g. if we had the following (made-up version numbers, because I can't remember what the real version-numbers are) :

• RiscOS v7.1 -> compatible with NOOBS v3.4 and v4.5
• RiscOS v7.2 -> compatible with NOOBS v3.4 and v4.5
• RiscOS v8.0 -> compatible with NOOBS v4.5
• RiscOS v8.1 -> compatible with NOOBS v4.5
• ProCountOS v1.5 -> compatible with NOOBS v4.5

Then NOOBS v3.4 would download os_list.json which would point to RiscOS v7.2, and NOOBS v4.5 would download os_list_v2.json which would point to both RiscOS v8.1 and ProCountOS v1.5.
That way, no matter which (possibly old) version of NOOBS you're running, it'll only ever show you compatible OSes 😃 At least that's the theory, but things might have changed since then...

And the whole process was repeated again with os_list_v3.json, which I think might have been created when Windows IoT support was added?

[XECDesign]

Yeah, os_list_v3.json was added for windows.

There may be all kind of unintentional inconsistencies with various versions of os_list. The script that automatically updates them is broken and I've never set time aside to fix it.