-
Notifications
You must be signed in to change notification settings - Fork 43
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature Idea: Deduped Steam Libraries #69
Comments
Not sure if this is something that we could add to Wolf or it just needs to be properly setup on the host..
The |
It could be something that Wolf doesn't do, but its a lot of state that specifically is useful to wolf, and seems to have advantages to be tied in to its lifecycle. Integrating some sort of plan here to avoid wasted diskspace and redownloading games for each streaming device will also make users less likely to just throw a single drive in mounts and put steam library on there. But maybe that would just work well enough, it feels like it's asking for race conditions messing everything up in a multi user environment. So for the goal of 1) reducing disk usage 2) reducing need to redownload:
This part of the feature is fairly easy to do outside, just have a cron job that dedupes the wolf folder once a day. However it makes a lot of sense to me if if Wolf schedules it when a container shuts down, to dedupe files added during that session.
| The steam image runs the full client, so everything should be available as is Right, but the steamcmd comment wasn't about a image spun up for a client. Those do already transfer from local network, and if you have Steam running on the host they will transfer from there, though not as fast as could be done between containers on a shared internal bridge. Building on that, if wolf can have with a container that's always on, and can be scripted to install games, it could be seamless that people don't have to download games multiple times if multiple moonlight devices play the same game on the wolf server. Another thing that could potentially be scripted externally. But here there is a risk partly installed games being transferred depending on implementation Wolf doing it when a client shuts down would make it somewhat atomic. SteamCMD is easier to script which is why I wonder if it can do the local network transfers. Another image like is spin up for clients would for sure work to serve games to the others, but I'm not sure sosohow to script it installing games. And wolf probably would need a way to connect to it's display to administer it (set it to allow all users on the local network to transfer games at least) Been thinking about a potentially easier path for 2) even if it does have some disadvantages: Another approach could be adding a LanCache container to wolfs network. I stopped running one at home because I had flakiness with steam deciding to bypass the container, but I hope it would be more reliable in the virtual network.Another downside is that update churn can cause extra disk space spent on no longer relevant files. It also wouldn't dedupe as cleanly as a "main" steam container approach because it stores compressed chunks and not the specific files. On the plus side it wouldn't need any sort of system to recognize what clients installed and install them on a master container. It would be a lot less moving parts, just some docker network design. There would still be advantages to having LanCache run by wolf and not set up externally. Mostly so it can be part of an internal bridge, because networking between containers is much faster that way than on the default bridge, DNS is handled better, and wolf can ensure DNS is set on the steam containers to use LanCache without a user having to mess with their local networks settings. |
The more I read about this the more I think this is getting outside the scope of Wolf but it's a very interesting discussion and you brought up a lot of good points, there's lots to digest so here are a few random counter points:
I'd be very interested in digging further into LanCache + Deduplication even if it's just going to end up being documentation and not actual code. I always wanted to checkout LanCache ony my homelab this might be the right time to actually do it! |
Thanks for reading and engaging!
I do like the idea of hooks and they will be useful for other stuff. I'm still tempted to say it should be something on by default for people but I can live with it not being :)
mm yeah. I suppose mostly assuming it was different client devices but same steam account, but this would be a hard part. Hmm.
I think with Lancache even if it's just publishing a docker compose example of it would massively help people out. Running it on a different machine will be much slower without multi gig networking.
I think something may still need to be done internally for dns, to make it easy to keep the traffic to the internal docker network if running one there. I might still explore scripting the main steam container some, because I don't trust lancache to work reliably :) If it ends up feeling easy I might have you take a look and see if you want to integrate. Or just figure out how to interface it with the hooks you add. |
I'm realizing the thing I'd actually more want to hack on is probably something to make reusing containers easier. I don't particularly need or want each device I connect from to have its own set of containers. Perhaps a user based collection. Or manual selection. Any thoughts for me on forking, vs preparing a PR or pointers on where I might want to start in the code? |
This is a valid point and I've been thinking about it for quite some time, I believe this is part of a more general feature: share sessions (aka: co-op for multiple users or as a specific case for a single user share the container between devices). I think there are different possible approaches to this:
Whilst the second approach might be a bit more complex it feels that it's the best way of achieving this:
Thoughts? |
That would be really cool 🤩 If I may add my 2 cents, this is how I see it:
is that similar to what you're thinking or am I dreaming too much? |
That's exactly what I have in mind, it's not a simple task, but it's definitely doable! Anyone wants to help out? 😅 |
I would love to help, but I'm gonna need some guidance on all the tech involved. |
The good news is that this will likely be a separate project from Wolf, so it can be done in any language (probably not C++). I have to define the requirements a bit better.. |
yeah, keep it as a separated project is the way to go. |
my toml block for steam: [[apps]]
title = "Steam"
start_virtual_compositor = true
[apps.runner]
type = "docker"
name = "WolfSteam"
image = "ghcr.io/games-on-whales/steam:edge"
mounts = [
"/mnt/games:/mnt/games",
"""
/home/apriestley/.steam/steam/userdata/19414271/config/shortcuts.vdf:/home/retro/.steam/steam/userdata/1941427\
1/config/shortcuts.vdf\
""",
"/home/apriestley/.steam/steam/compatibilitytools.d:/home/retro/.steam/steam/compatibilitytools.d",
"""
/home/apriestley/.steam/steam/config/libraryfolders.vdf:/home/retro/.steam/steam/steamapps/libraryfolders.vdf\
""",
]
env = [
"PROTON_LOG=1",
"RUN_GAMESCOPE=true",
"GOW_REQUIRED_DEVICES=/dev/input/* /dev/dri/* /dev/nvidia*",
]
devices = []
ports = []
base_create_json = """
{
"HostConfig": {
"IpcMode": "host",
"CapAdd": ["SYS_ADMIN", "SYS_NICE", "SYS_PTRACE", "NET_RAW", "MKNOD", "NET_ADMIN"],
"SecurityOpt": ["seccomp=unconfined", "apparmor=unconfined"],
"Ulimits": [{"Name":"nofile", "Hard":10240, "Soft":10240}],
"Privileged": false,
"DeviceCgroupRules": ["c 13:* rmw"]
}
}
\
""" The relevant bits are in the This isn't deduplication, but it works well. EDIT: I also have written this guide for setting up proper block-level deduplication using bees on top of btrfs, exposing storage shares over iSCSI. For this setup, the host and client are on the same machine. Potentially, each separate wolf user can own a discrete backstore pointed at the same disk image. |
It seems likely to cause problems to use the same steam library folder for multiple clients, at least at the same time.
So something that could be useful is running dedupe remove against the container folders for various clients. If mounted on BTRFS this could save a few gigabytes in the base image and many more in game downloads.
It would also be cool if there could be a main Steam container that installs all the games and is running when any steam containers for clients are open so a new client can install by steams feature to transfer from it over the "local-network". Which in this case can be a really quick bridge between containers.
Hard part on the main Steam container part might be identifying what a client steam container has installed to then installing on the "master" container. Maybe we can just look at "steamapps" grab the ids from the list of appmanifest_.acf files and then send those to steamcmd? But I'm not sure if a headless steam manage steamcmd implements the local network transfer stuff. Perhaps with a full fledged steam client we can just copy the files in? but I'm not sure if we'd need to trigger it to recognize them and offer them to other clients somehow.
The text was updated successfully, but these errors were encountered: