Opinions wanted on new Plugin feature

[gitbls]

Here's a new plugin feature that popped up. Your thoughts appreciated.

A new switch -xxx has sdm run the plugins (also specified on that command line) in the context of the currently running system.

I think it could be useful for adding or updating plugins into the runing system and dev/test on a new plugins. I think there are benefits, but unclear how useful you guys would find this.

For instance, you decide to add the plugin clockfake to your system.

sudo sdm --xxx --plugin clockfake to install clockfake. It seems to be even more useful for a service you want to add to multiple different servers.

In the plugin dev/test scenario, useful? Def need to be careful about what you actually do to the running system!

Also, this would only work on systems that have a) sdm installed and (maybe) b) were built with sdm (or have a /etc/sdm/cparams file in place that's "good enough"

Appreciate your thoughts. Thx

[dshaw619]

I can see some possible benefits of such a capability. In particular if one wanted to add something new to multiple systems without going through complete rebuilds.*** BUT, if there is anything wrong in the plugin, you can't just fix it and reapply as the system is now in a different state than the plugin might be expecting.

I imagine you'd have to manually back out whatever changes the plugin made (e.g. copy/create files, modify permissions, install apps, remove systemd services, etc.) to get back to the previous running state and then run the fixed plugin. Guess this is pretty much what one does without sdm. If the plugin without any modifications could be added to future --customize or --burn operations**, it would be better than having manually performed the changes to the running system and then running an untested plugin at some point in the future when you next did a --customize and --burn.***

When I started this reply I was expecting to conclude that I probably wouldn't use it, but after thinking it through, I think I would find it very beneficial.

** This assumes the same plugin could run on both the Linux Mint system that I use to configure and burn uSD cards and on the running RPis.

*** Just realized I learned my lesson (the first, but not last, time) about making changes in the target system rather than in the original source almost 60 years ago. (:-)

[dshaw619]

Yes, Mint is based on Debian. I understand supporting sdm on various platforms can present many difficulties (and as I mentioned in a previous discussion, would completely understand if you decided to not support anything but Raspberry Pi OS).

None of my RPis have monitors; I primarily connect to them with RealVNC from the Mint laptop and occasionally a Windows 11 laptop in another room. I've been running sdm (and previously RPi Imager) on the Mint laptop to customize uSD cards for the RPis.

If the plugin used to add something to a running RPi was not able to be used with a subsequent --customize or --burn on the Mint system, there probably wouldn't be much point in using it for me. Might as well just make the changes manually or with an ordinary bash script; then create and test** an sdm plugin to perform the same update/addition in --customize or --burn.

My usage pattern may not be typical; hopefully you'll get additional input from other users.

** My early lesson: DEC PDP-5, Calcomp 565 drum plotter, assembly language program to plot quadratic equations. While debugging, made hand assembled patches in octal through the front panel switches. After the program was "working", made the "same changes" to the assembly language source (paper tape through ASR-33 Teletype at 10 char/sec), assembled, punched and loaded binary paper tape, then watched the program fail a demo spectacularly (drum plotter rolled around in ways never seen before (:-) ).

[gitbls]

Difficulties and a HUGE test matrix! Daunting to say the least.

Since most of the plugins that one would test on the system running sdm (Mint in your case) are going to be written by YOU, what are your thoughts on how to prevent them from borking your Mint system?

One thought I had was adding a way for a plugin to check that it's running on a Pi so that it wouldn't do the wrong thing on a non-RasPiOS system. But that's easy enough for you to check today without doing anything in sdm. It does, however, leave the responsibility for ensuring that your system doesn't get toasted with the plugin author (you in this case).

I haven't heard about the PDP-5 for years! I had hands-on a PDP-10 in college, which led me to take a job in Maynard. But by the time I got there the PDP-5 was ancient history.

[dshaw619]

I may not have been clear enough. I'm not so much worried about borking my host system (Mint in my case, but applies similarly to any host as is pointed out in the docs somewhere talking about the different phases/environments to be aware of when writing plugins; I probably should pay more attention to this; 1stcall's suggestion of Docker sounds valuable).

My concern was just about the possibility of a plugin script acting differently for some reason on the running target RPi that it might later when incorporated into a --customize or --burn command run on the host system (I'm envisioning just adding a --plugin switch with the same plugin file and parameters as previously used to modify the target RPi). I guess, by definition, such a plugin would be functioning as Phase 1 or post-install (and would be coded as such) so maybe it isn't likely to be an issue.

---

Did you work at DEC? What years? I managed four 2060s in the early '80s and made multiple trips (don't remember if Maynard or Marlborough) for briefings re. Jupiter.

[gitbls]

Maybe you're not worried about that, but as the person who will hear the "sdm toasted my system!" messages, I am. 🤣

There's always the risk that, with different distros between host OS and RasPiOS, things might not work quite the same for plugins that "do" stuff to the system. OTOH if the plugin doesn't do anything to the system, it's probably not all that useful.

I realized that I hadn't mentioned it, but I was actually thinking about these plugins as something you'd want to run on your running RasPiOS system. It also hasn't been mentioned here that it would be useful to run a new plugin on an already-customized IMG, using a broad definition of the phrase "running system".

Here are a couple of actual things I think you might want to use live plugins for:

• You want to update your system by running an additional or updated plugin. Simple example: You want to install VNC on your system using the VNC plugin, but it wasn't originally customized into the system. Or, similarly, you want to do an update customize of an IMG to add VNC to the already existing customized IMG.

• When you're building a new plugin, and want to do "light testing" of it without doing a full customize. I have sat through far too many lite customizations, waiting for even a few things to be set, just to test my plugin, and find I duffed it with something simple. Being able to quickly / easily run the pulgin without even a minimal customize would be a huge win for me, anyhow.

PS: All of my Pis are headless as well. 99% of what I do on these systems can be done in an xterm, so I simply run xterms on those systems and display the output on my desktop (Debian or Windows). My test Pi has a monitor on it, but I typically only use it when booting up a test system. The rest of the time I use those xterms. Since all of my systems are fully-equipped by sdm, I literally wait for a new system to boot, pop an xterm, and I'm ready to go.

In case you're interested in how I do my xterms (which could actually be ANY X11 app), see https://github.com/gitbls/rxapp. It' provides a single-line command to start a remote X11 app and display it locally. I tried using menus to start them, but keeping the menus updated got old quickly. 🙄 (I use rxapp to fire up an xterm on a Pi at my friend's house. So cool!)

[dshaw619]

rxapp looks very interesting. I'll have to read more about it.

[1stcall]

I think I would on try this from within a Docker development container.

[gitbls]

Indeed, that would provide isolation from borking the host system.

[gitbls]

Here's my updated thinking on this.

• A new switch --runonly will be added. --runonly takes a value, at the current time, the only value is plugins.
• The sd-image-file argument can be either a device or an IMG file
• If /full/path/to/plugin is specified, the plugin will first be copied into the device/IMG. Updated plugins in /usr/local/sdm will NOT be copied in. That can be done by using the full path to the desired plugin: /usr/local/sdm/plugins/theplugin
• sdm will run all 3 phases of the plugins specified with --plugin switches on the sd-image-file

What about the running system? Still thinking about the real value of that, so starting without that.

This is similar in operation to --aptmaint, which does one specific thing on an IMG or device.

In the case of --runonly, the one thing is run the plugins

Thoughts?

[dshaw619]

Aha!!! I misunderstood where you intended this to run. From "in the context of the currently running system" and "adding or updating plugins into the runing system" I thought it was to run on the target RPi. I see now that the command and plugin would run on the host system, not the target RPi.

I'll admit I get confused (and will probably find difficult to keep straight in the long term when I go away from sdm for a while) with all the variations of what plugin is copied to/from where and which version will get run. For that reason, I always maintain a master copy that I update and reference by full path name. I find implicit sources of things to be confusing at best. (I realize this may be influenced by my habit of running sdm customize or burn from a script so once I've added a plugin (or other) switch with a full path reference, it's just there the next time I run it. If one only needs a couple of very familiar switches and runs the command directly from the command line, it might be easier to have the plugins in a directory where sdm knows to look for them.)

You might consider a different name for the switch. --runonly could refer to running any number of things. Maybe --plugins-only?

I can see how using this would help with debugging plugins (especially as I'm not particularly adept with bash).

[gitbls]

Hmm...we seem to have a difference of definitions. Here's my thinking... It's not important what system the plugin RUNS on, what's important is what system (or IMG or disk) the plugin OPERATES on.

When I originally mentioned this, I was thinking it would be useful to be able to run a plugin on a target Pi (disk created by sdm of course) to modify the running system.

Somehow we collectively generalized this into operating on host systems of any flavor. That seemed like it might be interesting, for certain classes of plugins and plugin operations, until I thought about the bazillion ways one could (accidentally) screw the host OS, especially if it's NOT RasPiOS.

So, I have for now backed off to operating on RasPiOS IMGs and disks, while running on some host OS that sdm can run on. This is effectively the same as being able to customize said IMG/disk on some host OS, but only runs the specified plugins.

I was inspired to think about this as I was debugging some annoying bug in a plugin. Being able to quickly and easily run the plugin being tested satisfies my annoyance at having to wait for ... pretty much anything 🤣

As far as --runonly plugin, I want to do it that way in case there's a case to be made for only running other things in the future. Having a generalized switch (--runonly) with specific keyword enablers is more properly scalable than adding a new switch for each feature you might want to enable in this manner. An admittedly bad example: --runonly plugins,xyzzy would run plugins and the xyzzy feature (whatever that might mean. Think, for example, ssh, enabling/disabling services, or pretty much anything else that sdm does.

Lastly, you mentioned " If one only needs a couple of very familiar switches and runs the command directly from the command line, it might be easier to have the plugins in a directory where sdm knows to look for them.". Maybe I'm mis-understanding you, but sdm already has two directories for this: /usr/local/sdm/plugins for sdm-supplied plugins and /usr/local/sdm/local-plugins for YOUR plugins. Are you suggesting a third directory? How would that work?

[dshaw619]

Sorry if I've confused the discussion. I wasn't thinking of the plugin operating on (i.e. modifying) the host system intentionally. My initial concern was about the possibility that a plugin might be run (executed) on the target RPi for testing/minimizing turnaround time, then somehow working differently (e.g. file system references) when run (executed) on the host system (in my case, Linux Mint) as part of a --customize or --burn of an image or uSD card (i.e. incorporating the plugin into my standard build after the plugin has been tested).

I concur that you should only worry about sdm operating on RPiOS images and disks. I grateful that you have been willing and able to support running sdm on Linux Mint.

re. --runonly: understand.

No, I am NOT suggesting a third directory for plugins. I prefer to refer to all my plugins (and various configuration files) by full path name in my customize and burn scripts. I was only meaning to say that I understood why someone who runs sdm from the command line with minimal switches might find it convenient to use the existing /usr/local/sdm/local-plugins structure.

[gitbls]

I don't think you confused anything. You just floated another idea that gave me more to think about. I asked for inputs, and appreciated your perspective!

I'm pretty happy with the way the cross-architecture support (e.g., sdm running on Linux x64 to customize a RasPiOS IMG) turned out. It's quite seamless, and QEMU performance is pretty darn good these days. Of course, when the Pi5 comes out, I may be less impressed!