appstores #0

[JarbasAl]

This PR adds the concept of appstore to msm

Currently Pling and Mycroft Marketplace are implemented, all this is in a subpackage

from msm.appstores import pling, mycroft_marketplace
from msm import appstores
from msm import MycroftSkillsManager

msm = MycroftSkillsManager()


print("Searching pling appstore for hivemind")
for s in pling.search("hivemind", msm=msm):
    print(s.name)  # pling appstore only

print("Searching mycroft marketplace for jokes")
for s in mycroft_marketplace.search("jokes", msm=msm):
    print(s.name)  # mycroft appstore only

print("Searching everywhere for node red")
for s in appstores.search("node red", min_conf=0.4, msm=msm):
    print(s.name)  # All appstores

print("Listing pling appstore skills")
for s in pling.list_skills(msm=msm):
    print(s.name)

print("Listing mycroft marketplace skills")
for s in mycroft_marketplace.list_skills(msm=msm):
    print(s.name)

Discussion :

msm can be made to automatically check both appstores, this would be backwards compatible and automatically extend the command line usage and installer skill

this will avoid the skills submission guidelines and raises the question of what is a valid appstore, could i just spin up my own and host malware? feedback welcome on what is the correct next step

[JarbasAl]

My next step would be to create a new skill to install skills from the pling appstore, this would probably cause intent conflicts with installer skill. A preffered solution would be native integration into msm + skill installer skill

[AIIX]

This PR adds the concept of appstore to msm

Currently Pling and Mycroft Marketplace are implemented, all this is in a subpackage

from msm.appstores import pling, mycroft_marketplace
from msm import appstores
from msm import MycroftSkillsManager

msm = MycroftSkillsManager()


print("Searching pling appstore for hivemind")
for s in pling.search("hivemind", msm=msm):
    print(s.name)  # pling appstore only

print("Searching mycroft marketplace for jokes")
for s in mycroft_marketplace.search("jokes", msm=msm):
    print(s.name)  # mycroft appstore only

print("Searching everywhere for node red")
for s in appstores.search("node red", min_conf=0.4, msm=msm):
    print(s.name)  # All appstores

print("Listing pling appstore skills")
for s in pling.list_skills(msm=msm):
    print(s.name)

print("Listing mycroft marketplace skills")
for s in mycroft_marketplace.list_skills(msm=msm):
    print(s.name)

Discussion :

msm can be made to automatically check both appstores, this would be backwards compatible and automatically extend the command line usage and installer skill

this will avoid the skills submission guidelines and raises the question of what is a valid appstore, could i just spin up my own and host malware? feedback welcome on what is the correct next step

With my KDE Community/Bigscreen dev hat on:

Integration with existing libre services and app hosting providers would be nice, since it allows opening up the platform to more of the open source community and allows contributions from outside the current community bringing in more developers, supporters

Pling has publishing policy https://www.pling.com/terms/publishing, which is moderated by the open desktop team https://www.opendesktop.org/ a libre services provider which has been an around in the foss space since 2001 and is used by major desktops to provide community themes, plugins, wallpapers etc. https://store.kde.org/ and https://www.gnome-look.org/

With my Mycroft Skills Management member hat on:

I would suggest evaluating any appstore by reputation in the open source space (Major foss players support, communities, etc) and time its been in service, and the policies the appstore operates on (Moderation, Open Licenses, etc)

On the skills submission guidelines not sure how this can be enforced on external app stores, but there could be some concept of tested skill / untested skills not sure since skills hosted on an external app store do not anyways show up on marketplace, for now it would just be nice to have installation support within MSM for an external store like pling.

[AIIX]

A question that crossed my mind, how would msm handle installation if a skill exist in both the marketplace and an appstore in the appstores.search ? This also affects handling of installation for a particular platform, for example the youtube-skill, should use master branch for mark-2/desktop type devices and new-ui branch for bigscreen

[ChanceNCounter]

I would suggest evaluating any appstore by reputation in the open source space (Major foss players support, communities, etc) and time its been in service, and the policies the appstore operates on (Moderation, Open Licenses, etc)

On the skills submission guidelines not sure how this can be enforced on external app stores, but there could be some concept of tested skill / untested skills not sure since skills hosted on an external app store do not anyways show up on marketplace, for now it would just be nice to have installation support within MSM for an external store like pling.

Mac requires users to change a system setting to allow apps from non-Apple sources, and I seem to recall running a terminal command to bust out of the garden entirely (unsigned programs.)

This has the dual effect of requiring users to demonstrate a basic level of competence before they can escape the garden, and making it extremely clear that users are leaving the safety of Apple's embrace.

I know the walled garden has a bad reputation, but that's because Apple wants you to stay in there. Mycroft's gonna be more concerned about making sure you leave at your own risk, so treating it the way Mac does would be fine.

If that notion were extended to an approved_sources.list, implementations like Bigscreen could just add their repos to that list by default.

[AIIX]

I would suggest evaluating any appstore by reputation in the open source space (Major foss players support, communities, etc) and time its been in service, and the policies the appstore operates on (Moderation, Open Licenses, etc)

On the skills submission guidelines not sure how this can be enforced on external app stores, but there could be some concept of tested skill / untested skills not sure since skills hosted on an external app store do not anyways show up on marketplace, for now it would just be nice to have installation support within MSM for an external store like pling.

Mac requires users to change a system setting to allow apps from non-Apple sources, and I seem to recall running a terminal command to bust out of the garden entirely (unsigned programs.)

This has the dual effect of requiring users to demonstrate a basic level of competence before they can escape the garden, and making it extremely clear that users are leaving the safety of Apple's embrace.

I know the walled garden has a bad reputation, but that's because Apple wants you to stay in there. Mycroft's gonna be more concerned about making sure you leave at your own risk, so treating it the way Mac does would be fine.

If that notion were extended to an approved_sources.list, implementations like Bigscreen could just add their repos to that list by default.

Rather than running a command it would be better to have a configuration key for msm in mycroft.conf along the lines of SupportedAppstores: ["Marketplace", "Pling"] so platforms/distributions shipping mycroft are free to configure what marketplace, appstore they want to support, and users who personally want to access skills on Pling for example can enable it by simply adding it to an overriding mycroft.conf at user level / system level, this would be more inline with Linux mechanism of adding/supporting community/user repositories.

[forslund]

I like the configurable marketplace idea that @AIIX proposes, that way no separate installer skill would be needed for systems with other platforms. It could be embedded in the platform selection as well setting the "bigscreen" platform would enable the pling appstore.

Regarding security, currently msm allows anyone to host a skills repo with altered skills. So I'm not too worried about allowing third-party appstores as long as they are vetted by the team and that the stores to use are selectable so a Mycroft distribution can specify which store/stores to use.

[JarbasAl]

Since we are giving the power to the users, why not support random github repos? msm does support it but its clunky to use, and in the way its integrated into core does not really allow using alternative repos except in command line

@AIIX made a good point about branches, i would like to see the skill.json approach from pling adopted in more places, the pling methods above already provide more metadata than github, searching skills by tag/category would be an obvious enhancement to msm itself

If we do some changes also in pling side we can probably come up with a pretty decent platform support. I would like to be able to select a branch depending on platform, requirements etc are an obvious point of conflict

I'd also like to keep any extra code changes in follow up PRs and keep this one only for the basic pling api, it provides a SkillEntry object that is expected by the rest of the codebase so its easy to build on, theres lots of nuance here i dont have the availability to take care of in this PR, that way its cleaner and other members can contribute

[forslund]

Isn't it just a config value that needs to be changed?

I agree to keep that sort of changes to a separate PR is best. Leaving this with the base implementation 👍

[PureTryOut]

I wonder, could this be used to add support for distribution package managers as well? I'd like to be able to install packages from my distribution using MyCroft, also on interfaces like Plasma Bigscreen.
Especially in the case of Alpine Linux where skills are preferably installed via the system package manager, it'd be nice to have it supported.

Note that this would need an escalation of privileges, as only system administrators (basically root) are allowed to use the package manager on most systems. Not sure how to handle that.

[JarbasAl]

Isn't it just a config value that needs to be changed?

I agree to keep that sort of changes to a separate PR is best. Leaving this with the base implementation +1

I was thinking about supporting several app stores simultaneously, id like msm to check both pling and marketplace, maybe add a repo of my own in the mix. Essentially expanding the msm.list method

[AIIX]

I wonder, could this be used to add support for distribution package managers as well? I'd like to be able to install packages from my distribution using MyCroft, also on interfaces like Plasma Bigscreen.
Especially in the case of Alpine Linux where skills are preferably installed via the system package manager, it'd be nice to have it supported.

Note that this would need an escalation of privileges, as only system administrators (basically root) are allowed to use the package manager on most systems. Not sure how to handle that.

From bigscreen context: Bigscreen has a Graphical Installer for skills from Pling, The installer is basically a wrapper around Mycroft Skills Manager which handles the skill installation and python dependencies installation part, MSM via Pako (https://github.com/MycroftAI/pako) is supposedly able to handle distribution package installation using the data from the manifest.yml but it doesn't work for the bigscreen since Pako seems to support only a cli Interface, the Graphical Installer works around this issue by using polkit to show a graphical user interface to enter password and in the background execute Pako manually with privileges to install system dependencies.

Overall i would like to be able to simply wrap the installer around MSM without having to do the extra hacky stuff for Pako and also some extra skill installation steps we handle like the ".desktop" file and "icon" file installation/uninstall for bigscreen but I guess since this is bigscreen specific so it can be discussed in another issue.

[JarbasAl]

Overall i would like to be able to simply wrap the installer around MSM without having to do the extra hacky stuff for Pako and also some extra skill installation steps we handle like the ".desktop" file and "icon" file installation/uninstall for bigscreen but I guess since this is bigscreen specific so it can be discussed in another issue.

Bigscreen is the first alternative appstore, this is the perfect time to discuss scalability and moving forward, i think bigscreen is doing some things right, i like the .json file approach. If we can standardize this that would be cool. I think its very relevant to this discussion, its important to have a good template with clear specs, ideally we want skills to just work everywhere, we do not want devs having to make changes to be "right" for all app stores, at some point conflicts are inevitable and we dont really want "appstore wars" which is a PITA for devs (but arguably can bring advantages for users)

[JarbasAl]

I wonder, could this be used to add support for distribution package managers as well? I'd like to be able to install packages from my distribution using MyCroft, also on interfaces like Plasma Bigscreen.
Especially in the case of Alpine Linux where skills are preferably installed via the system package manager, it'd be nice to have it supported.

Note that this would need an escalation of privileges, as only system administrators (basically root) are allowed to use the package manager on most systems. Not sure how to handle that.

This sounds like a task for https://github.com/MycroftAI/pako

Privileges are hard, specially across lots of distros with different opinions, this is one more thing we should focus dev resources into a single place instead of leaving it up to skill devs (like requirements.sh used to do)

Ideally pako would evolve to a point where it can be used in all distros (from msm) and devs/appstores would not need to worry about that

2 good test cases would be support for bigscreen and alpine linux

EDIT: if skills are coming from the package manager it sounds like we need a new appstore integration for package managers, maybe this can use pako or maybe not, right now msm is a bit biased towards github. Essentially we need a way to create SkillEntry objects, in this case i assume the skills themselves would be a requirement handled by pako on install step

[ChanceNCounter]

What if we relied on distro packages to add support for a particular package manager?

apt install mycroft-msm-ubuntu
pamac install mycroft-msm-manjaro
yum also-exists-i-think
etc.

This would allow MSM to remain focused on installing Mycroft things from familiar sources, while acknowledging that it's quite sensible for skills to come from your distro's repositories.

It gets tricky when you think about that distro package notion. @PureTryOut makes a great point: a bunch of us are concerned with Linux integration, and he's already put skills in the Alpine repo. Furthermore, I'd love to be able to install system packages by voice!

But how much of that belongs in MSM, and how much in a distro control skill? MSM already has to invoke your system package manager for package requirements, yeah, but arbitrary packages from the nutso-long package lists?

It seems strange for MSM to support installing skills from system repos, but not to support other packages. On the other hand, it seems equally strange for Mycroft to install system packages through a discrete companion program, rather than through a skill.

[JarbasAl]

@ChanceNCounter until STT improves a lot installing packages by voice is a no go, optimistically 90% will be wrong, on the other hand a vetted source (configured app store) for a specific voice searchable thing is reasonable, so if packages are skill requirements thats ok and necessary

Challenge, make a distro control skill and prove me wrong, cli usage doesnt count

[ChanceNCounter]

@JarbasAl You already implemented ask_selection() =P and it does a pretty good job finding desktop applications from cached names. I suspect it's the opposite problem, where fuzzy matching turns the right input into too many similar possibilities. For instance, woe to the person who says, "Hey, Mycroft, launch LibreOffice," and woe to the person who tries to install a package that starts, libs<> or libg<>.

[JarbasAl]

anything blocking this? does not impact existing code in the wild. I want to build some functionality on top of this

[krisgesling]

Hey, the big questions for me are:

• What requirements would we have for 3rd party app stores to be included? What criteria must they fulfil?
• Does Pling actually review the code being submitted? I just submitted a Skill and its immediately available so I'm presuming it defaults to be available unless it is flagged as malicious? Is this actively reviewed or is it up to the open source community to flag things?
  • Given the level of system access that Skills currently have on most installations this seems quite dangerous. Even though we aren't publishing them on the Marketplace, if we are making them easily installed by a supported method, then that is pretty equivalent in my mind. We may not be legally liable for what happens, but we are morally liable given that a lot of users don't think twice before installing things.
• How would this be used by end-users? Only via the commandline or by voice if they have a Pling Installer Skill?
• I could see this working where users have to explicitly add 3rd party stores in the same way you have to add package repositories in your OS. Theoretically only people who have a certain level of understanding and technical ability will do so.

Overall, to me this seems like a pretty major process change, so I'd want to know how it will work as a whole before dropping in bits and pieces.

[JarbasAl]

using numbers to answer those points so its easier

1 - requirements is up to mycroft to decide, personally i think there should be no requirements, this is all disabled by default, and msm supports random github repos already, so i can set up evilSkillMarket but not pling?
2 - in pling things are approved and removed only if flagged, i would argue your moral obligation is to provide sane defaults only, not block users from installing what they want on their systems, as a side note you see my skills in pling but not in mycroft marketplace because its such a PITA to submit skills to mycroft
3 - this specific PR would only be used by directly writing code, next steps (for end users functionality) would be allowing the cli to parse pling urls, and eventually a pling voice installer skill. if we can use msm to install random github repos already why not pling?
4 - the idea would be that users needs to manually edit the .conf to enable a app store, i dont really see this as a end user thing but rather a enclosure thing, bigscreen use pling and would set this up on the enclosure .conf on their distro, instead of duplicating work

in here i'm talking about pling becauses it already exists, but you could consider a marketplace for adult content for example, i intend to make a demo marketplace anyone could fork and use (kinda like how mycroft does it), i see these being enabled by adding the url explicitly to the .conf or as msm arg in command line

[ChanceNCounter]

I think the chief difference between MSM's "vulnerability" to unvetted code and Pling's is that Pling has a GUI, and is therefore accessible to orders of magnitude more users. "Follow this simple tutorial to unlock your Mycroft" is much easier and more permanent than the GitHub route.

Ungoverned software extensions are time bombs. The recent Nano Defender misery is a perfect reminder. The developer sold an incredibly popular browser extension for a rumored $2000. An open-source extension. Sold control of the Chrome and Edge store accounts...

...and the new developers promptly added code to the release versions (but not the repositories) that phoned home. They started stealing session keys, and within a couple days were selling Instragram likes off of infected users' accounts.

And that's the problem in a nutshell. Even users who do a reasonable job vetting third-party software can still get burned.

It's true that this matters less when you have to modify config files just to access the third-party store, but I assume that once Pling (especially Pling) is integrated, it's a one-way trip. Sooner or later, a skill will follow to add and remove stores. Then it'll get a GUI. And that's fine, that's good, but we've gotta approach this on the assumption that any code which goes on Pling now will someday be seen and installed by thousands of random users.

8-year-olds think they're digitally invincible, they think malware is obnoxious at worst, and they think app store reviews are truthful. I don't wanna make it easier for somebody's kid to install EvilSkill!

I think either third-party stores need BDFLs, or enabling other stores and repos needs to scream and scream. "You're leaving the garden! It's not safe out there! Nothing has been vetted! Stay away from quicksand! Don't do online banking on this device!"

Otherwise, that guy who "publicly disclosed" that the dev build is not locked down, that guy will turn out to have had a valid point when code is sideloaded by a different route.

[JarbasAl]

@ChanceNCounter https://github.com/MycroftAI/mycroft-core/blob/dev/mycroft/configuration/mycroft.conf#L104

i can literally set up a random github appstore, and install random repos from cli.

why is pling a lesser "app store provider" than others?

[ChanceNCounter]

It's not, it's The Bestest App Store =P but it's also much, much more accessible to Users ™️

[JarbasAl]

MycroftAI/mycroft-core#2737 now warns users if a different appstore is configured, right now we need to replace mycroft marketplace, my aim is to enable several marketplaces at once.

some more considerations, We don't really want skills duplicated across appstores, handling skill name collisions will be the tricky part, but matching is already not so good anyways and could use some love.

[ChanceNCounter]

Naming conventions could alleviate that. If Dismissal is always dismissal-skill.chancencounter, MSM could just check the skill folder. If the skill is already present, either refuse to install, or check version numbers and prompt.

Possible downsides:

• Skill changes hands, new dev renames it like dismissal-skill.gez or something
• Need to communicate to users when this happens
• Some devs will inevitably violate the convention

[JarbasAl]

i want to note that msm is a bit different from core, with msm being a command line util that already supports other marketplaces, my confusion here comes from this, msm supports random github repos already

is the issue about supporting multiple stores? (only want to support one app store at a time) or is the issue with pling not being a github based store? if pling used github like mycroft this PR wouldn't even be needed

[JarbasAl]

since i wanted to build up on this functionality i moved development to https://github.com/OpenVoiceOS/ovos_skill_manager