[FEATURE]: Add the ability to copy files to an image

[TangoOversway]

Is your feature request related to a problem? Please describe.

I often need to copy multiple files to a new Pi system, such as .zshrc, some Python and shell scripts, and sometimes configuration files. It would be easier if those could be copied over after an OS image was created on the media. This wouldn't work with some images, but with many, it would mean not having to copy files with SCP and altering settings that aren't included or are not editable in Imager.

Describe the solution you would like to see implemented

Currently there are 3 tabs under OS Customization: General, Services, and Options. There would be a 4th tab, "Files," where a list of files to copy over to the new image would be selectable and each file (or directory) would be in an editable list. Files and directories to be copied could be selected by a file dialog or by typing the path in manually.

Describe alternatives you've considered

I've used a copy script for ssh, but that requires me to have the new system on my LAN when I start it up means the image, after booting, still needs files copied to it. This would make sure the image has any special configuration files, scripts, or other files on it and ready to be used on first boot.

Version

1.8.5 (Default)

[lurch]

The problem with this is that (AFAIK) Windows and Mac have no way of writing to ext4 filesystems.

[TangoOversway]

If the fs can't be written to, how are things like the wifi info, default user and password, ssh keys, and so on, written to the image?

[lurch]

This is (indirectly) a duplicate of #554 (and similar requests).

[TangoOversway]

Thank you. I didn't know about firstboot.sh. The files I want to copy are small, mostly configuration files, and, in some cases, files that have entries I want to add to other configuration files. If I modify firstboot.sh similarly to the way mentioned in that discussion, I can also write my own script for firstboot.sh to run and write a script for my workstation. My script can modify firstboot.sh to look for my script. I can copy all my files into my own directory in the boot partition, have my script do what I want with them, self-delete everything, then return control to firstboot.sh so it can self-delete.

That basically does what I want to do - or most of it.

A couple questions, if you don't mind, before this gets closed out:

1. Under what user does firstboot.sh run? I would think under root, considering some of the things it might have to do.
2. Would there be an issue running a Python script from firstboot.sh?

[tdewey-rpi]

Feature request rejected.

Unfortunately I classify this under the same bracket as supporting arbitrary script execution as part of first boot / OS customisation. The principal blocking issues are:

1. I cannot conceive of a way to educate new users how to make educated choices in such a scheme
2. There is significant scope for abuse - tutorial sites recommending you just copy and paste this totally-not-a-rootkit, for example.
3. This mechanism would not be OS portable - short of having a complete description of the end OS to try and provide some indication of which shell profile scripts to allow, what python modules to allow, etc, I can't see a way to make this robust.

That said, there are mechanisms you can drive manually to get your desired effect. You could, for example, manually edit firstboot.sh ahead of your first boot. If you're using a cloud-init based system, you could also include a cloud-init configuration file: https://cloudinit.readthedocs.io/en/latest/reference/examples.html#yaml-examples

To answer your specific questions:

1. I'd expect it to run under root.
2. I wouldn't recommend that unless it was a heavily constrained script. One particularly important note is that firstboot.sh runs ahead of network start up, so any assets you'd want to use must be part of your base image or added to the FAT partition ahead of first boot.

I'm now going to close this issue as 'Won't Fix' - in line with the rejection. Discussion can continue in this state.

[TangoOversway]

Okay, I get your reasons for not doing this. Naturally I was thinking in terms of, "Gee, I really want this," but once you included your reasoning, I thought, "Yeah, from the developer's POV, that would be a nightmare and you can be sure SOMEONE will put up a "totally-not-a-rootkit" example and everyone will complain to you when they fall for that one.

My expectation was it would run under root, as well, but I wanted to check. That's one of those things that, if you don't check, you find there's some constraint you never thought of!

I'm thinking of using a Python script because I can code much faster in Python than shell scripting. (Shell scripting drives me bonkers - don't know why I have so many issues with it!) Currently what I'm thinking is that the files I want to copy over are all small, like text configuration files. In some cases, like with fstab, I have a few Samba mounts on my LAN I want to include, so it'd be a case of tacking them onto the end of the existing fstab file. I figured I could write a script, to run on my Mac, that would create an extra directory in the boot partition with my script and the files it will copy. It'd do the copying, creating any extra directories it needs, and, when done, would self-delete it all. So it'd be rather simple. Thanks for the warning about the network. not being available.

Also, thanks for the information, instead of just closing without looking into the reasoning behind the request - I now have a way to do what I want without needing this feature.

[lurch]

...and of course the other way of doing this would be to use RPi Imager to enable SSH access on the image you're writing, and then after the image has booted up for the first time, use a combination of scp / ssh on your Mac to copy the files / run the scripts you need to on the Pi 😉 (and at this point the network would be available).

[tdewey-rpi]

Okay, I get your reasons for not doing this. Naturally I was thinking in terms of, "Gee, I really want this," but once you included your reasoning, I thought, "Yeah, from the developer's POV, that would be a nightmare and you can be sure SOMEONE will put up a "totally-not-a-rootkit" example and everyone will complain to you when they fall for that one.

Thanks for your understanding. From a personal PoV, I would really like this feature - along with a few other 'power user' features - but Raspberry Pi Imager is probably the wrong tool to include these in, as it's the one we send people toward for the first out of box experience.

My expectation was it would run under root, as well, but I wanted to check. That's one of those things that, if you don't check, you find there's some constraint you never thought of!

Indeed it is!

I'm thinking of using a Python script because I can code much faster in Python than shell scripting. (Shell scripting drives me bonkers - don't know why I have so many issues with it!) Currently what I'm thinking is that the files I want to copy over are all small, like text configuration files. In some cases, like with fstab, I have a few Samba mounts on my LAN I want to include, so it'd be a case of tacking them onto the end of the existing fstab file. I figured I could write a script, to run on my Mac, that would create an extra directory in the boot partition with my script and the files it will copy. It'd do the copying, creating any extra directories it needs, and, when done, would self-delete it all. So it'd be rather simple. Thanks for the warning about the network. not being available.

Makes a lot of sense to me - always go with the language you're most comfortable in. Watch out for breakage across versions, however, as this sort of modification mechanism isn't something that carries a support commitment from the packages reading the config (generally, at least). As such, I'd encourage you to consider adding precondition checking to make sure you don't wind up in an undefined state.

Also, thanks for the information, instead of just closing without looking into the reasoning behind the request - I now have a way to do what I want without needing this feature.

My general policy for handling feature requests for Raspberry Pi Imager is to be as transparent as possible and offer as much rationale for any decision I'd make. It's fairer to the original poster, and fairer on someone going through older issues to find out why rpi-imager behaves how it does.

Glad you're unblocked.

[TangoOversway]

Thanks for your understanding. From a personal PoV, I would really like this feature - along with a few other 'power user' features - but Raspberry Pi Imager is probably the wrong tool to include these in, as it's the one we send people toward for the first out of box experience.

Would it be worth considering a 2nd tool, one not recommended for beginners, that could edit the image? Yeah, I know - another tool to maintain. Just passing the thought along. I think, for me, being able to do what I mentioned will take care of what I needed. I've also gone through raspi-config. I really wish it had a batch mode where an advanced user could control all the settings with keyword arguments, like, "vnc=enabled ssh=enabled" and so on. It's frustrating to have to run it on each new image and change 4 or 5 settings. One thing I'm looking at, with my script, is to handle some of that. For instance, create the symlinks to turn Weyland on or off and turn VNC on or off. At one point I was going through that script and working on extracting each function and writing a script that could read a text file with all the settings I wanted, so it would set them for me.

Watch out for breakage across versions, however, as this sort of modification mechanism isn't something that carries a support commitment from the packages reading the config (generally, at least). As such, I'd encourage you to consider adding precondition checking to make sure you don't wind up in an undefined state.

Always! That's why I try to make anything I write like that as non-intrusive into existing code as possible. This would only depend on the self-deletion line. I'd have to check for it and check if it's indented, in case it gets put into a conditional block.

My general policy for handling feature requests for Raspberry Pi Imager is to be as transparent as possible and offer as much rationale for any decision I'd make. It's fairer to the original poster, and fairer on someone going through older issues to find out why rpi-imager behaves how it does.

I agree with your reasoning and it's much appreciated!

[lurch]

Would it be worth considering a 2nd tool, one not recommended for beginners, that could edit the image? Yeah, I know - another tool to maintain.

If you want to build your own entirely-customised image, then have a look at https://github.com/RPi-Distro/pi-gen

I've also gone through raspi-config. I really wish it had a batch mode where an advanced user could control all the settings with keyword arguments

It does! See https://www.raspberrypi.com/documentation/computers/configuration.html#the-raspi-config-command-line-interface

[TangoOversway]

If you want to build your own entirely-customised image, then have a look at https://github.com/RPi-Distro/pi-gen

I'm not sure I need to go that far, but thank you! It's the kind of thing I probably won't use or think about until 6 months down the line when I'm thinking, "I REALLY need that!"

It does! See https://www.raspberrypi.com/documentation/computers/configuration.html#the-raspi-config-command-line-interface

Awesome! I don't know how I missed that when looking through the code - other than, as I said, that I have always had issues with shell scripts (have no idea why!). I can use that in the script system I've mentioned wanting to use. With that, and my script added in with firstboot, that will mean I can get everything the way I want without having to make changes on every image after they're running!

...and of course the other way of doing this would be to use RPi Imager to enable SSH access on the image you're writing, and then after the image has booted up for the first time, use a combination of scp / ssh on your Mac to copy the files / run the scripts you need to on the Pi 😉 (and at this point the network would be available).

I missed this reply earlier - don't know why. Yes, I've considered doing that with ssh after I get a system up and running. It's an option, but I like the idea that I can do it all at prep time. Part of that is because there are times I prep an SD card and it's a while before I can install it. I tend to "front load" things like this so I don't forget them later or if I have someone helping me, I don't have to worry about whether they know what to do.