Change Questions D-Bus API to fit new proposal #637

jreidinger · 2023-06-21T20:14:11Z

Problem

Questions does not allow yet automatic answering of question.

Solution

This is the first phase, which adapt API on backend and frontend side. Support for answers file is not part of this PR

Testing

Tested manually

TODO

adapt web UI
testing of changes using web UI

coveralls · 2023-06-21T20:19:23Z

coverage: 72.215% (-0.2%) from 72.385% when pulling 765135d on adapt_quesstions_api into dccd029 on master.

imobachgs

So if I got it correctly, you are wrapping the questions objects (e.g., GenericQuestion) into other structs (e.g., GenericQuestionObject) which implements support for answering the questions, right?

About answering the questions, I see you decided to go with a trait. I am not sure how other kinds of questions would be answered yet (I guess you need to work more on that part).

In general, the approach looks good to me.

jreidinger · 2023-06-22T13:50:36Z

@imobachgs well, idea is to have object GenericQuestion that can be used also on client side basically as question data holder. Then there is GenericQuestionObject that is DBus server API to be provided. Answering questions is defined in New method that creates question or in its specialized create_luks variant. For different variants I am not sure if we need specialized answers as all of them should define GenericQuestion and together with data holding all needed, it should be enough to answer question.

For answering idea is to have that trait AnsweringStrategy which each strategy needs to implement and we have list of those configured based on dbus calls and ask them if they know answer or not. so it can look like [FileAnswers, DefaultAnswers] and first it ask if file contain answer and if not then it ask for default answer.

mvidner · 2023-06-22T14:04:15Z

I wanted to point out that if the D-Bus client needs to change, then we should change the number in the API, to Questions2, to make it obvious.
But then it seems that so far it is not needed, only the Rust implementation is changing, right?

jreidinger · 2023-06-22T21:11:09Z

More and more I am thinking about how to implement answer strategy from a file, the more I see problematic our approach with specific questions ( for now just Luks Activation ).

With current approach it needs:

the question creator needs to use specific method on backend
all UI needs to handle that specialized question
questions API itself needs to know about it, so with every new specialized type it breaks API
answer file needs to know how to answer such specialized questions
in general any answering strategy needs to be aware of all specialized questions
if we do audit/trace for questions during installation as mentioned in research document it also needs to know about all specialized question types.
for that audit/trace we will need data filtering and it against needs to about any additional attributes from specialized question

So what I propose is:

remove LuksActivation interface
identify luks activation question just by its class string
have data map rw, so frontend can write to data
use data for password or any possible future additional data we need for more specific questions

with this approach the need to know about question type is:

common New method can be used for password
all UIs needs to know about that specialized question class and act accordingly to that class
questions API does not know about it and acts only as data holder
answer file does not need to know about specialized answer, we just need to define generic way to set data
answering strategy will be generic to provide answer + data
audit/trace just track generic data, no issue with specialized questions
data filtering also just needs to know which data + answer to filter. No need for specialized fields.

So how it will look on frontend side for LuksActivation:

signal for new question appear ( no need for workaround for multiple interfaces )
class is detected as luks activation so specialized dialog appears
if decrypt is requested then password is written to data map and answer set to decrypt

note: number of attempts is also in data map.
Another advantage that now it will be also easier to show more specialized dialogs if needed on frontend side like for GPG key we can add link to webpage explaining GPG or any other improvement as we know class of question. So basically now each question is specialized via its class that also allows in future something like "Do not ask me again" on frontend and automatic answer question of same class with answer already provided by user.

So to short it up, with current implementation polymorphism does not work at all, as every component that touch questions needs to know about all specialized types.

@imobachgs @joseivanlopez how do you see it?

imobachgs · 2023-06-23T10:21:11Z

Hi all,

We have been discussing this problem and, although we did not come to a conclusion, we have some ideas.

On the one hand, as @jreidinger noticed, the current API resembles a hierarchy: you have a Generic question and specific ones (e.g., LuksActivation) for those that do not fit into the Generic concept. The danger is that you might end up with many specific types (e.g., GpgKeyImport), so keeping the clients up-to-date might be problematic.

I do not like that much passing hashes through D-Bus because:

It is hard to keep the documentation up-to-date, as you cannot rely on introspection.
You can inadvertently break the API (adding, removing or changing some key). Having introspection data could help in those situations.

Having said that, if it is our best bet, I will not oppose it.

However, we wonder whether we could come up with a better (and more generic) model. Instead of having Generic, LuksActivation (that looks like a hierarchy), we could use composable API. For instance, having interfaces like Question, WithPassword, WithOptions, etc. Does it make sense? Does it bring any benefit?

mvidner · 2023-06-23T12:42:20Z

So what I propose is:

remove LuksActivation interface

I understand (1) but not the subsequent points, please show examples

have data map rw, so frontend can write to data

Not sure what you mean, but if you mean a hash over D-Bus, it would have to be one in parameter and another out parameter as D-Bus cannot do inout parameters or "modify argument objects".

mvidner · 2023-06-23T12:47:45Z

Regarding various kinds/classes of questions:

Currently we have a specialized LuksQuestion that the frontend must handle specially. If we change that to have only one QuestionNG with a complex hash argument, the frontend will still have to handle it specially, right? It will be just obfuscated in the API.

Unless we make the API so generic that it becomes a Dialog Interpreter, with Label, Hbox, Vbox, InputField, Button... 🤣

I maybe missing @jreidinger 's point, Josef please give examples.

jreidinger · 2023-06-23T13:45:45Z

@mvidner:

So what I propose is:

remove LuksActivation interface

I understand (1) but not the subsequent points, please show examples

Ok, to clarify Question has now in this PR several properties including data map type and class string type.
so that class string can be used on frontend side to identify that it is luks activation. String can look like "storage.luks.activation". And in its data property it can hold that the password value and also attempt value. So basically as Imo mention we will handle additional properties via hash entries instead of own interface that has it as additional properties.

have data map rw, so frontend can write to data

Not sure what you mean, but if you mean a hash over D-Bus, it would have to be one in parameter and another out parameter as D-Bus cannot do inout parameters or "modify argument objects".

As mention it is property. So only change is that property will be rw and not ro.

Regarding various kinds/classes of questions:

Currently we have a specialized LuksQuestion that the frontend must handle specially. If we change that to have only one QuestionNG with a complex hash argument, the frontend will still have to handle it specially, right? It will be just obfuscated in the API.

Well, frontend will have to handle it specially. That is true. But it will be simple interface and some other parts does not need to handle it specially like matching against answer or capturing question. Also now it needs specialized constructor in Questions object new_luks_activation. So what we discuss and what I mainly do not like that it makes extension with new questions type quite hard as it means that too many places needs to be adapted. And it can grow significantly. Basically what we discuss is that now we have "inheritance" in API as object with LuksActivation is also Generic Question, but polymorphism does not work well for it ( I kind of expect that it is also Dbus limitation ).

That is why we discuss on call about changing it to real "composition" that works much better with DBus ( at least to my limited knowledge ) that can be generic enough to allow easy extension in future and still kind of limit amount of questions that needs special handling. So that is what @imobachgs writes that we can have interfaces like BasicQuestion, WithPassword and so on. For LuksActivation it will mean that it no longer be specialized class and instead it will be BasicQuestion and WithPassword interfaces that together provide final question with all its data. And it should also limit amount of handling on frontend side with more generic composition.

Unless we make the API so generic that it becomes a Dialog Interpreter, with Label, Hbox, Vbox, InputField, Button... rofl

I maybe missing @jreidinger 's point, Josef please give examples.

mvidner · 2023-06-23T14:00:04Z

Ok, to clarify Question has now in this PR several properties including data map type and class string type.

That's Questions::new_question

As mention it is property. So only change is that property will be rw and not ro.

Not part of the PR yet, I don't see such #[dbus_interface(property)] in impl GenericQuestionObject right?

jreidinger · 2023-06-23T15:19:00Z

Ok, to clarify Question has now in this PR several properties including data map type and class string type.

That's Questions::new_question

yeap and it returns object path, so no need for any data as data will be filled as part of answer.

As mention it is property. So only change is that property will be rw and not ro.

Not part of the PR yet, I don't see such #[dbus_interface(property)] in impl GenericQuestionObject right?

yeap, not yet part of PR, but I can easily add it. Question is what to do with that LuksQuestion. Maybe what I can do now to unblock this PR is to just rename luks question to WithPassword and it move creation of question to caller and not ruby. And then lets see if we need to change it in future.

BTW to capture it also here. Agreement is to not bump version of API until we release some stable agama version as API is still kind of flexible and we are only user now.

mvidner · 2023-06-26T07:22:54Z

to unblock this PR

Oh just go ahead, I don't want to block anything

Agreement is to not bump version of API until we release some stable agama version

OK fine

jreidinger · 2023-06-28T09:05:50Z

I would say that dbus API should be final now. Implementation is still something that can change. E.g. I do not like much how mixins are implemented and if more types or wilder combination of mixins can happen, then we will probably need something like

struct Question {
  base : GenericQuestion,
  with_password: Option<WithPassword>,
  another_mixin: Option<AnotherMixin>
}

that allows variation of mixins and its combination and have methods on top of it that acts differently based on available mixins.

mvidner · 2023-06-28T10:36:39Z

rust/agama-dbus-server/src/questions.rs

@@ -16,6 +16,16 @@ impl GenericQuestionObject {
        self.0.id
    }

+    #[dbus_interface(property)]
+    pub fn class(&self) -> &str {


Please update also doc/dbus/org.opensuse.Agama.Questions1.Generic.doc.xml

will do, my plan is to update doc and test when everything works together as I am not sure if we not need to modify more.

mvidner · 2023-06-28T10:38:49Z

rust/agama-dbus-server/src/questions.rs

@@ -58,22 +69,19 @@ impl LuksQuestionObject {
    pub fn set_password(&mut self, value: &str) -> () {
        self.0.password = value.to_string();
    }
-
-    #[dbus_interface(property)]
-    pub fn attempt(&self) -> u8 {


Is it intentional to discard attempt?

yeap, as I think it is not generic password specific stuff. So my plan now is to move it do data as ro data and if we need it more often, we can add WithAttempt mixin for questions that needs to capture it.

mvidner · 2023-06-28T10:45:42Z

rust/agama-dbus-server/src/questions.rs

 }

 trait AnswerStrategy {
-    /// TODO: find way to be able to answer any type of question, not just generic
+    /// Provides answer for generic question


Suggested change

/// Provides answer for generic question

/// Provides answer for generic question.

/// Returns one of the question options.

What does the None return value mean?

None means in both cases that Answering strategy does not provide it. As FileStrategy will be another implementation of this trait and if answer is not specified there, it should return None.

mvidner · 2023-06-28T10:46:33Z

rust/agama-dbus-server/src/questions.rs

    fn answer(&self, question: &GenericQuestion) -> Option<String>;
+    /// Provides answer and password for base question with password
+    fn answer_with_password(&self, question: &WithPassword) -> (Option<String>, Option<String>);


again I don't understand why the returns are optional

Also, is their optionality independent?

yes and yes. None means that strategy does not provide it and it is up to question to decide what it means. E.g. Default strategy for password should return something like (Some(self.default_answer), None) which means use default answer and does not provide password.

mvidner · 2023-06-28T10:49:01Z

rust/agama-dbus-server/src/questions.rs

@@ -224,6 +239,19 @@ impl Questions {
            }
        }
    }
+
+        /// tries to provide answer to question using answer strategies


Describe what happens when multiple strategies do (not) provide an answer. Is there a short circuit? What if answers conflict?
I shouldn't need to read the implementation to know this.

I will document it.

mvidner · 2023-06-28T10:50:11Z

rust/agama-lib/src/questions.rs

@@ -41,48 +41,20 @@ impl GenericQuestion {
    }
 }

-/// Specialized question for Luks partition activation
+/// Composition for questions which include password.
+/// TODO: research a bit how ideally do mixins in rust


This TODO has been resolved by this code already?
Or add pointers to ideas and examples to comments/commit message.

I will write it before comment in this PR. I will try to document the idea

imobachgs

In general, it looks good to me. I only have a few questions that can be addressed in the following PR.

imobachgs · 2023-07-14T10:26:14Z

rust/agama-lib/src/questions.rs

+        }
+    }
+
+    pub fn object_path(&self) -> String {


Questions are coupled with D-Bus, right? I wonder whether this struct should live in agama-lib or directly in agama-dbus-server. After all, any other component should use the D-Bus API to deal with questions.

@imobachgs reason why it is is in agama-lib and not in agama-dbus-server is that object path is useful also for client side. So my general rule is if it is useful for server and client then have it in lib, so both CLI and dbus server can use it. If it is useful only for one of it, then it can live there.

OK, it is fine. I tend to move the stuff to agama-lib only when I use it from other parts, but if you find it useful, it is OK.

imobachgs · 2023-07-14T10:29:19Z

rust/agama-lib/src/questions.rs

+
+impl WithPassword {
+    pub fn new(base: GenericQuestion) -> Self {
+        info!("Adding to question with password interface.");


np: Most probably I would move the logging to the D-Bus interfaces objects. If you want to write some unit tests here, you will get a log message.

imobachgs · 2023-07-14T10:30:27Z

rust/agama-dbus-server/src/questions.rs

-    /// Confirmed answer. If empty then not answered yet.
-    answer: String,
-}
+struct GenericQuestionObject(questions::GenericQuestion);


Nice use of the newtype pattern. Actually, I wonder whether the object_path function could be here.

as written above I think it can be useful also for client implementation e.g. when you want to remove question, you should also provide object_path

and yeah, about newtype pattern, I hear about it multiple times from you and here I think it fits really well.

imobachgs · 2023-07-14T10:34:49Z

rust/agama-dbus-server/src/questions.rs

+    BaseWithPassword,
+}
+
+trait AnswerStrategy {


I would expect a description of what this trait does and how is expected to be used. It can be in a different PR, as we agreed offline, but please do not forget about it.

Moreover, perhaps it deserves its own strategies module.

yeap, own module can happen easily with adding FileStrategy to get it from file. I just do not want to over-complicate it now

OK, no problem. We can move later.

imobachgs · 2023-07-14T10:37:17Z

rust/agama-dbus-server/src/questions.rs

            .await?;
-        self.questions.insert(id, QuestionType::Generic);
+        self.questions.insert(id, QuestionType::Base);
        Ok(object_path)
    }

    /// creates new specialized luks activation question without answer and password


Suggested change

/// creates new specialized luks activation question without answer and password

/// creates new question to ask for a password with no default answer

I think it is not right. There have to be always default strategy. I just want to mention here that created question has empty answer and password property.

OK, then I did not understand the comment 😄

Rust zbus unfortunately outputs the provided interfaces in a random order in the introspection data, so we have to sort them before comparing

setup-service in particular

they are semantically the same, just with interfaces sorted by name

…/cleanup() had to manually re-add the interfaces missing on x86: <interface name="org.opensuse.Agama.Storage1.DASD.Manager"> <interface name="org.opensuse.Agama.Storage1.ZFCP.Manager">

@Result

The @Result output argument is unfortunately unnamed in zbus(Rust) introspection so we have to document it in plain text

jreidinger · 2023-07-14T19:23:47Z

doc/dbus/bus/seed.sh

+}
+
+# "dot dot name"
+# "slash slash name"


👍 for explanation of variable

jreidinger · 2023-07-14T19:27:51Z

doc/dbus/org.opensuse.Agama.Questions1.doc.xml

    <!--
      UseDefaultAnswer:

      Switches questions to be automatically answered by default answer.
      After this method call each follow up question is immediatelly answered.
      Useful for doing non interactive installation.
    -->
-    <method name="UseDefaultAnswer" />
+    <method name="UseDefaultAnswer">
+    </method>


I am just curious why this change is needed? It is due to output of introspect method?

Yes, introspection has the whitespace text node. We know it can be ignored because we know the schema/DTD but the checker is too simple for that.

imobachgs

Just some typos. Otherwise, LGTM. Great job! Thanks!

imobachgs · 2023-07-17T09:24:05Z

rust/agama-dbus-server/src/questions.rs

+    /// and try to find the first strategy that provides answer. When
+    /// answer is provided, it returns immediately.
+    fn fill_answer(&self, question: &mut GenericQuestion) {
+        for strategy in self.answer_strategies.iter() {


Just a suggestion for the future (not even for this PR), but I would say that a version using combinators would be easier to read. Something like (not tested):

let answer = self .answer_strategies .iter() .find_map(|s| s.answer(question)); // Note: I did not use `and_then` because it expects the closure to return an Option, and that's not the case. if let Some(value) = answer { question.answer = value; }

rust/package/agama-cli.changes

service/package/rubygem-agama.changes

web/package/cockpit-agama.changes

mvidner

Thank you! LGTM now

Co-authored-by: Imobach González Sosa <[email protected]>

Change dbus API to fit new proposal

9b25a65

imobachgs reviewed Jun 22, 2023

View reviewed changes

mvidner changed the title ~~Change dbus API to fit new proposal~~ Change Questions D-Bus API to fit new proposal Jun 22, 2023

change API to use more mixin approach

7e56e5f

mvidner reviewed Jun 28, 2023

View reviewed changes

jreidinger added 12 commits June 29, 2023 00:20

adapt ruby service to new questions API

21c64d6

adapt UI and fix service

f889ae2

Merge remote-tracking branch 'origin/master' into adapt_quesstions_api

41099ac

fix file name

a25b96e

fix merge errors

7101ec9

fix type of attempts

c9ec48b

add more logging for easier debugging

cfc68a8

more logging

c6e63a3

more logging

e907132

fix passing number of attempts

fb7c2ab

more logging to reveal why password is not passed

c38139a

more logging to debug answering

7f161da

jreidinger added 6 commits July 13, 2023 16:21

remove test for question which become dummy data holder

0a27a51

move adaptation for new iface

eda57af

more test adaptation and rubocop fixes

36a7723

more test adaptation and rubocop fixes

74462a0

Merge remote-tracking branch 'origin/master' into adapt_quesstions_api

5a1853b

document questions dbus interface

65d1547

imobachgs reviewed Jul 14, 2023

View reviewed changes

mvidner added 7 commits July 14, 2023 15:38

doc/dbus: sort by interface when seeding

31cd63a

Rust zbus unfortunately outputs the provided interfaces in a random order in the introspection data, so we have to sort them before comparing

setup.sh: exit as soon as a step fails

0438dbf

setup-service in particular

doc/dbus/bus: adjusted the introspected files after seed.sh/cleanup()

594e21e

they are semantically the same, just with interfaces sorted by name

doc/dbus/bus: Storage1: adjusted the introspected files after seed.sh…

f95e44b

…/cleanup() had to manually re-add the interfaces missing on x86: <interface name="org.opensuse.Agama.Storage1.DASD.Manager"> <interface name="org.opensuse.Agama.Storage1.ZFCP.Manager">

Merge branch 'doc-sort-interfaces' into adapt_quesstions_api

236f25b

doc/dbus/bus: update seeded Questions API

721c5ee

doc/dbus: Questions1.*.doc.xml now pass make check

f6429d7

The @Result output argument is unfortunately unnamed in zbus(Rust) introspection so we have to document it in plain text

mvidner force-pushed the adapt_quesstions_api branch from af76190 to f6429d7 Compare July 14, 2023 14:16

jreidinger commented Jul 14, 2023

View reviewed changes

doc/dbus/bus/seed.sh

}

# "dot dot name"

# "slash slash name"

Copy link

Contributor Author

jreidinger Jul 14, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 for explanation of variable

jreidinger commented Jul 14, 2023

View reviewed changes

jreidinger marked this pull request as ready for review July 17, 2023 05:58

jreidinger and others added 5 commits July 17, 2023 10:54

improve documentation

960024d

fix formatting

071ebd6

apply clippy fixes

cd79884

Fix typos

a0f1765

Update changes

54705ac

imobachgs approved these changes Jul 17, 2023

View reviewed changes

mvidner approved these changes Jul 17, 2023

View reviewed changes

jreidinger and others added 2 commits July 17, 2023 11:58

Apply suggestions from code review

c337aac

Co-authored-by: Imobach González Sosa <[email protected]>

fix clippy output

765135d

jreidinger merged commit c49c48d into master Jul 17, 2023
14 checks passed

jreidinger deleted the adapt_quesstions_api branch July 17, 2023 10:50

imobachgs mentioned this pull request Aug 2, 2023

Release version 3 #683

Merged

	/// Provides answer for generic question
	/// Provides answer for generic question.
	/// Returns one of the question options.

	/// creates new specialized luks activation question without answer and password
	/// creates new question to ask for a password with no default answer

Change Questions D-Bus API to fit new proposal #637

Change Questions D-Bus API to fit new proposal #637

Conversation

jreidinger commented Jun 21, 2023 • edited Loading

Problem

Solution

Testing

TODO

coveralls commented Jun 21, 2023 • edited Loading

imobachgs left a comment

Choose a reason for hiding this comment

jreidinger commented Jun 22, 2023

mvidner commented Jun 22, 2023

jreidinger commented Jun 22, 2023 • edited Loading

imobachgs commented Jun 23, 2023

mvidner commented Jun 23, 2023

mvidner commented Jun 23, 2023

jreidinger commented Jun 23, 2023

mvidner commented Jun 23, 2023

jreidinger commented Jun 23, 2023

mvidner commented Jun 26, 2023

jreidinger commented Jun 28, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

imobachgs left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreidinger Jul 14, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

imobachgs left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mvidner left a comment

Choose a reason for hiding this comment

jreidinger commented Jun 21, 2023 •

edited

Loading

coveralls commented Jun 21, 2023 •

edited

Loading

jreidinger commented Jun 22, 2023 •

edited

Loading

jreidinger Jul 14, 2023 •

edited

Loading