Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[PoC] Import AutoYaST profiles #1029

Merged
merged 25 commits into from
Feb 7, 2024
Merged
Show file tree
Hide file tree
Changes from 24 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
4c01cb8
[service] Add a script to import an AutoYaST profile
imobachgs Feb 23, 2023
6c2a7a4
[service] Do not import but convert to an Agama profile
imobachgs Feb 24, 2023
7ab66de
[service] Move the AutoYaST importer to lib and add tests
imobachgs Jan 31, 2024
2d5fc26
[service] Add a D-Bus method to convert an AutoYaST profile
imobachgs Jan 31, 2024
470145f
[rust] Refactor the code to download a profile
imobachgs Jan 31, 2024
d3938d5
[service] Add support for pre-script to the AutoYaST converter
imobachgs Feb 2, 2024
28ff598
[service] Add an agama-autoyast executable
imobachgs Feb 2, 2024
9fb1f78
[rust] Use agama-autoyast to process AutoYaST profiles
imobachgs Feb 2, 2024
f2e7cad
[service] Add dependency on yast2-schema
imobachgs Feb 2, 2024
48df260
[service] Replace Yast2::Popup with an Agama specific class
imobachgs Feb 6, 2024
a105391
[service] Do not crash when software/products is not an array
imobachgs Feb 6, 2024
be386a5
[doc] Document the AutoYaST support
imobachgs Feb 6, 2024
65f9b1d
[service] Temporarily disable some RuboCop rules
imobachgs Feb 7, 2024
45399ad
[service] Fix tests to not write in /tmp/profile
imobachgs Feb 7, 2024
3116c9c
[service] Imports AutoYaST root settings
imobachgs Feb 7, 2024
b7dab3c
[auto] Skip AutoYaST XML validation
imobachgs Feb 7, 2024
e0e395a
[service] Add a comment
imobachgs Feb 7, 2024
3dc6b71
[service] Imports AutoYaST software patterns
imobachgs Feb 7, 2024
74117b2
[service] Extract the users logic from the AutoYaST converter
imobachgs Feb 7, 2024
24019f4
[service] Removed unneeded expect
imobachgs Feb 7, 2024
aa71027
[doc] Update the AutoYaST support document
imobachgs Feb 7, 2024
62a9781
[service] Drop the ConvertProfile D-Bus method
imobachgs Feb 7, 2024
329be93
[service] Mock the profile validation
imobachgs Feb 7, 2024
a872ca1
Update changes files
imobachgs Feb 7, 2024
39caf46
Apply suggestions from code review
imobachgs Feb 7, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions autoinstallation/scripts/auto.sh
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
#!/usr/bin/sh
set -ex

# Temporarily skip the AutoYaST XML validation
export YAST_SKIP_XML_VALIDATION=1

if [ -z "$1" ]
then
url=$(awk -F 'agama.auto=' '{sub(/ .*$/, "", $2); print $2}' < /proc/cmdline)
Expand Down
219 changes: 219 additions & 0 deletions doc/autoyast.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,219 @@
# AutoYaST Support

Agama offers a mechanism to perform [unattended installations](../autoinstallation/). However, we
would like AutoYaST users to be able to use their AutoYaST profiles in Agama. This document
describes how Agama could support, to some extent, such profiles.

Bear in mind that this document is just a draft and our plans could change once we start working
on the implementation.

## What to support

We want to point out that Agama and AutoYaST have different features. Agama is focused on the
installation and delegates further configuration to other tools. From this point of view, it is
clear that many of the sections you can find in an AutoYaST profile will not have an Agama
counterpart.

Nevertheless, we want to cover:

* Dynamic profiles, including rules/classes, ERB templates, pre-installation scripts and even "ask
lists". See [Dynamic profiles](#dynamic-profiles).
* Compatibility (partial or full) for the following sections: `networking`, `partitioning`,
`language`, `timezone`, `keyboard`, `software`, `scripts`, `users`, `iscsi-client`, `proxy` and
`suse_register`. See [Supported sections](#supported-sections).

We still need to decide how to handle other sections like `firewall`, `bootloader`, `report`,
`general` or even some elements from `security` or `kdump`.

Finally, we plan to "ignore" many other sections (e.g., all *-server elements) and sysconfig-like
elements. See [Unsupported sections](#unsupported-sections).

## Dynamic profiles

Many AutoYaST users rely on its dynamic capabilities to build adaptable profiles that they can use
to install different systems. For that reason, we need Agama to support these features:

* [Rules and classes][rules-classes].
* [Embedded Ruby (ERB)][erb].
* [Pre-installation scripts][pre-scripts].
* [Ask lists]().

The most realistic way to support those features in the mid-term is to use the AutoYaST code with
some adaptations. The [import-autoyast-profiles branch][autoyast-branch] contains a proof-of-concept
that supports rules/classes, ERB and pre-installation scripts. If you are interested, you can give
it a try:

```
cd service
sudo bundle exec bin/agama-autoyast \
file:///$PWD/test/fixtures/profiles/invalid.xml /tmp/output
cat /tmp/output/autoinst.json
```

You can even use the `agama-cli`:

```
cd rust
cargo build
sudo PATH=$PWD/../service/bin:$PATH ./target/debug/agama profile download \
file:///$PWD/../service/test/fixtures/profiles/pre-scripts.xml
```

About "ask lists", there might need more work. Fortunately, the code to [parse][ask-list-reader] and
[run][ask-list-runner] the process are there but we need to adapt the [user
interface][ask-list-dialog], which is not trivial.

[rules-classes]: https://doc.opensuse.org/documentation/leap/autoyast/html/book-autoyast/rulesandclass.html
[erb]: https://doc.opensuse.org/documentation/leap/autoyast/html/book-autoyast/erb-templates.html
[pre-scripts]: https://doc.opensuse.org/documentation/leap/autoyast/html/book-autoyast/cha-configuration-installation-options.html#pre-install-scripts
[ask-lists]: https://doc.opensuse.org/documentation/leap/autoyast/html/book-autoyast/cha-configuration-installation-options.html#CreateProfile-Ask
[autoyast-branch]: https://github.com/openSUSE/agama/tree/import-autoyast-profiles
[ask-list-reader]: https://github.com/yast/yast-autoinstallation/blob/c2dc34560df4ba890688a0c84caec94cc2718f14/src/lib/autoinstall/ask/profile_reader.rb#L29
[ask-list-runner]: https://github.com/yast/yast-autoinstallation/blob/c2dc34560df4ba890688a0c84caec94cc2718f14/src/lib/autoinstall/ask/runner.rb#L50
[ask-list-dialog]: https://github.com/yast/yast-autoinstallation/blob/c2dc34560df4ba890688a0c84caec94cc2718f14/src/lib/autoinstall/ask/dialog.rb#L23

## Supported sections

### `dasd` and `iscsi-client`

Support for iSCSI and DASD devices is missing in Agama profiles. Let's work on that when adding the
`partitioning` section equivalent.

### `general`

AutoYaST `general` section contains a set of elements that, for some reason, did not find a better
place. Most of those options will be ignored by Agama (e.g., `cio_ignore`, `mode`, `proposals`,
etc.). However, we might need to add support for a handful of them.

Agama should process the `ask-list` section (see [Supported sections](#supported-sections)),
`signature-handling` (to deal with packages signatures) and, most probably, `storage` too (e.g.,
affects the proposal).

### `groups` and `users`

Regarding users, Agama only allows defining the first user and setting the root authentication
mechanism (password and/or SSH public key). However, AutoYaST allows to specify a list of users and
groups plus some authentication settings. We have at least two options here:

* Extract the root authentication data from the profile and try to infer which is the first user.
This behavior is already implemented.
* Import these sections as given because they are handled by the YaST code in Agama.

### `keyboard`, `language` and `timezone`

These sections are rather simple, but we need to do some mapping between AutoYaST and Agama values.
Additionally, the `hwclock` element is not present in Agama.

### `networking`

The `networking` section in AutoYaST is composed of several sections: `dns`, `interfaces`,
`net-udev`, `routing` and `s390-devices`. Additionally, other elements like `ipv6` or
`keep_install_network` might need some level of support.

At this point, Agama only supports defining a list of connections that could correspond with the
AutoYaST interfaces list. We might need to extend Agama to support `dns`, `net-udev`, etc.

About `keep_install_network` and `setup_before_proposal`, we should not implement them to keep
things simple.

### `partitioning`

By far, the most complex part of an AutoYaST profile. We can import the AutoYaST `partitioning`
section as is because the partitioning is handled by the same code in Agama and AutoyaST.
imobachgs marked this conversation as resolved.
Show resolved Hide resolved

However, we must implement a mechanism to convert to/from both profile types.

### `proxy`

To use a proxy in Agama, you set the `proxy` in the [kernel's command line][cmdline]. In AutoYaST,
you can specify the proxy in the profile apart from the command line.

Although we need to support the same use case, we should avoid introducing a `proxy` section unless
it is strictly required.

[cmdline]: https://github.com/openSUSE/agama/blob/a105391949a914ae57719c80a610c642fb581924/service/lib/agama/proxy_setup.rb#L31

### `report`

The AutoYaST `report` section defines which kind of messages to report (errors, warnings,
information and yes/no messages) and whether the installation should stop on any of them. Agama does
not have an equivalent mechanism. Moreover, it is arguable whether it is a good idea to base on the
type of message to stop the installation. A more fine-grained control over the situations that
should stop the installation would be better. As an example, consider the `signature-handling`
section.

### `scripts`

The only way to use scripts in Agama is to write your own autoinstallation script. Unlike AutoYaST,
you cannot embed the script within the Jsonnet-based profile. This is relevant from the
implementation point of view because we might need to extract AutoYaST scripts and put them in some
place for Agama to run them.

Apart from that, AutoYaST considers five kind of scripts: `pre`, `post-partitioning`, `chroot`,
`post`, and `init`. The last two are expected to run after the first boot, where Agama is not
present anymore.

If we want to support `post` or `init` scripts, we need to copy them to the installed system and run
them through a systemd service.

### `software`

The `software` section is composed of several lists:

* A list of products to install, although a single value is expected.
* A list of patterns to install, a list of patterns to install in the 2nd stage and a list of
patterns to remove.
* A list of packages to install, a list of packages to install in the 2nd stage and a list of
packages to remove.

Additionally, it is possible to force the installation of a specific kernel (`kernel`), perform
an online update at the end of the installation (`do_online_update`) or enable/disable the
installation of recommended packages (`install_recommended`).

Only the product and the list of products or patterns are available for Agama. We might consider
adding support for the packages list and the `install_recommended` setting, although none are in the
web UI.

### `suse_register`

Basic support for registering in the SUSE Customer Center is already in place, although
there is no way to select the list of add-ons.

It is arguable whether we should offer a `install_updates` element instead of just installing them
(which is the use case for not installing them?).

About the `slp_discoverty` element, Agama does not support [SLP] at all?

[SLP]: https://documentation.suse.com/sles/15-SP5/single-html/SLES-administration/#cha-slp

## Unsupported sections

* `FCoE`
* `add-on`
* `audit-laf`
* `auth-client`
* `configuration_management`
* `deploy_image`
* `dhcp-server`
* `dns-server`
* `files`
* `firstboot`
* `ftp-server`
* `groups`
* `host`
* `http-server`
* `mail`
* `nfs`
* `nfs_server`
* `nis`
* `nis_server`
* `ntp-client`
* `printer`
* `samba-client`
* `services-manager`
* `sound`
* `squid`
* `ssh_import`
* `sysconfig`
* `tftp-server`
* `upgrade`
1 change: 1 addition & 0 deletions rust/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

11 changes: 9 additions & 2 deletions rust/agama-cli/src/profile.rs
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
use agama_lib::profile::{download, ProfileEvaluator, ProfileValidator, ValidationResult};
use agama_lib::profile::{ProfileEvaluator, ProfileReader, ProfileValidator, ValidationResult};
use anyhow::Context;
use clap::Subcommand;
use std::path::Path;
Expand All @@ -15,6 +15,13 @@ pub enum ProfileCommands {
Evaluate { path: String },
}

fn download(url: &str) -> anyhow::Result<()> {
let reader = ProfileReader::new(url)?;
let contents = reader.read()?;
print!("{}", contents);
Ok(())
}

fn validate(path: String) -> anyhow::Result<()> {
let validator = ProfileValidator::default_schema()?;
let path = Path::new(&path);
Expand Down Expand Up @@ -45,7 +52,7 @@ fn evaluate(path: String) -> anyhow::Result<()> {

pub fn run(subcommand: ProfileCommands) -> anyhow::Result<()> {
match subcommand {
ProfileCommands::Download { url } => Ok(download(&url)?),
ProfileCommands::Download { url } => download(&url),
ProfileCommands::Validate { path } => validate(path),
ProfileCommands::Evaluate { path } => evaluate(path),
}
Expand Down
1 change: 1 addition & 0 deletions rust/agama-lib/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -19,4 +19,5 @@ tempfile = "3.4.0"
thiserror = "1.0.39"
tokio = { version = "1.33.0", features = ["macros", "rt-multi-thread"] }
tokio-stream = "0.1.14"
url = "2.5.0"
zbus = { version = "3", default-features = false, features = ["tokio"] }
75 changes: 52 additions & 23 deletions rust/agama-lib/src/profile.rs
Original file line number Diff line number Diff line change
Expand Up @@ -4,29 +4,58 @@ use curl::easy::Easy;
use jsonschema::JSONSchema;
use log::info;
use serde_json;
use std::{
fs, io,
io::{stdout, Write},
path::Path,
process::Command,
};
use tempfile::tempdir;

/// Downloads a file and writes it to the stdout()
///
/// TODO: move this code to a struct
/// TODO: add support for YaST-specific URLs
/// TODO: do not write to stdout, but to something implementing the Write trait
/// TODO: retry the download if it fails
pub fn download(url: &str) -> Result<(), ProfileError> {
let mut easy = Easy::new();
easy.url(url)?;
easy.write_function(|data| {
stdout().write_all(data).unwrap();
Ok(data.len())
})?;
easy.perform()?;
Ok(())
use std::{fs, io, io::Write, path::Path, process::Command};
use tempfile::{tempdir, TempDir};
use url::Url;

/// Downloads a profile for a given location.
pub struct ProfileReader {
url: Url,
}

impl ProfileReader {
pub fn new(url: &str) -> anyhow::Result<Self> {
let url = Url::parse(url)?;
Ok(Self { url })
}

pub fn read(&self) -> anyhow::Result<String> {
let path = self.url.path();
if path.ends_with(".xml") || path.ends_with(".erb") || path.ends_with('/') {
self.read_from_autoyast()
} else {
self.read_from_url()
}
}

fn read_from_url(&self) -> anyhow::Result<String> {
let mut buf = Vec::new();
{
let mut handle = Easy::new();
handle.url(self.url.as_str())?;

let mut transfer = handle.transfer();
transfer.write_function(|data| {
buf.extend(data);
Ok(data.len())
})?;
transfer.perform().unwrap();
}
Ok(String::from_utf8(buf)?)
}

fn read_from_autoyast(&self) -> anyhow::Result<String> {
const TMP_DIR_PREFIX: &str = "autoyast";
const AUTOINST_JSON: &str = "autoinst.json";

let tmp_dir = TempDir::with_prefix(TMP_DIR_PREFIX)?;
Command::new("agama-autoyast")
.args([self.url.as_str(), &tmp_dir.path().to_string_lossy()])
.status()?;

let autoinst_json = tmp_dir.path().join(AUTOINST_JSON);
Ok(fs::read_to_string(autoinst_json)?)
}
}

#[derive(Debug)]
Expand Down
6 changes: 6 additions & 0 deletions rust/package/agama-cli.changes
Original file line number Diff line number Diff line change
@@ -1,3 +1,9 @@
-------------------------------------------------------------------
Wed Feb 7 11:49:02 UTC 2024 - Imobach Gonzalez Sosa <[email protected]>

- Add preliminary support to import AutoYaST profiles
(gh#openSUSE/agama#1029).

-------------------------------------------------------------------
Mon Jan 29 15:53:56 UTC 2024 - Imobach Gonzalez Sosa <[email protected]>

Expand Down
2 changes: 1 addition & 1 deletion service/agama.gemspec
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ Gem::Specification.new do |spec|
spec.homepage = "https://github.com/openSUSE/agama"
spec.license = "GPL-2.0-only"
spec.files = Dir["lib/**/*.rb", "bin/*", "share/*", "conf.d/*"]
spec.executables = ["agamactl", "agama-proxy-setup"]
spec.executables = ["agamactl", "agama-proxy-setup", "agama-autoyast"]
spec.metadata = { "rubygems_mfa_required" => "true" }

spec.required_ruby_version = ">= 2.5.0"
Expand Down
Loading
Loading