Add a multiplayer interface and visual nodes for SceneTree replication

[Faless]

Describe the project you are working on

A multiplayer game with Godot.

Describe the problem or limitation you are having in your project

Game state replication over network is hard to achieve in Godot, even for simple games.

Making multiplayer games has historically been a complex task, requiring ad-hoc optimizations and game-specific solutions. Still, two main concepts are almost ubiquitous in multiplayer games, some form of messaging, and some form of state replication (synchronization and reconciliation).

While Godot does provide a system for messaging (i.e. RPC), it does not provide a common system for replication.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

The goal of this document is to propose a replication API that can be used for the common cases, and is extensible via plugins, outlining what emerged during the #networking meetings while gathering and integrating feedback from a broader public.

In this sense, an initial "node interface" is also proposed incorporating the suggestions in #3359, as well as changes to the lower level API to make room for the more Godot-y approach to the subject.

Design Goals

Whatever system gets implemented (even if limited for the scope of Godot 4.0) should aim to:

• Provide an out-of-the-box solution for scene state replication across the network.
• Be easy to use for game developers (of course 😎).
• Allow for (almost) no-code prototyping.
• Be extensible with game-specific behaviours (custom reconciliation, interpolation, interest management, etc).
• Allow ex-post (incremental) optimizations of network code.

Glossary

• Spawn: Creating, or requesting remotely to create a new Object.
• Sync: Updating, or requesting remotely to update the state of an Object.

Brief

The idea is to add a fully customizable object replication interface to the MultiplayerAPI to deal with replication at the lower level, along with a set of higher level nodes to replicate the state in SceneTree as suggested in #3359.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

Higher level replication API (default SceneTree implementation)

Two extra nodes and a new resource type will be added to deal with scene replication:

• SceneReplicationConfig: Will allow to define which properties of a given node (or subnode) should be synchronized automatically.
  Somehow similar to Animation, the initial options will be limited but might grow over time to allow more and more features and transparent optimizations engine-side.
• MultiplayerSpawner: Will allow to spawn instances of predefined scenes via MultiplayerSpawner.spawn(node) according to their replication config.
• MultiplayerSynchronizer: Will allow synchronizing the remote and local state.

Nodes will behave according to the respective multiplayer authority (Node.get_multiplayer_authority()).

class SceneReplicationConfig extends Resource:

    # Not exposed directly
    # 4.0+: Potentially allow more options for properties in the future (e.g. encodings/compresison level/sync priority).
    class ReplicationProperty:
        # The property name
        var name : StringName # Or NodePath?
        # If the property is to be sent during spawn
        var spawn : bool
        # If the property is to be sent during sync
        var sync : bool

    var properties : Array[ReplicationProperty]

    ### Utility functions for encoding/decoding states
    func get_spawn_properties(obj: Object) -> Array[NodePath]
    func get_sync_properties(obj: Object) -> Array[NodePath]

    ### More functions for configuring it (add/remove/set), but also done via UI.


class MultiplayerSpawner extends Node:

    # The NodePath where the scenes will be spawned.
    var spawn_path : NodePath

    # If the nodes added to the spawn_path should be automatically spawned remotely.
    var auto_spawn : false

    # Array of packed scenes that can be spawned by this spawner node.
    var spawnable_scenes : Array[PackedScene]

    # Spawn a new object remotely using a custom spawn argument, and add it to "spawn_path".
    func spawn(arg)

    # Get the spawn argument that was used for a given object.
    func get_spawn_argument(ObjectID)

   # Virtual function that should be implement for custom spawning to work.
   func _spawn_custom(arg) -> Node


class MultiplayerSynchronizer extends Node:

    # Relative path to the root of the synchronization.
    # (must be direct the scene root for spawn state to work correctly... is this okay?)
    var sync_root : NodePath

    # Sync configuration for this sync node.
    var sync_config : MultiplayerReplicationConfig

    # Sync interval for this object (in seconds as float).
    var sync_interval : float = 0.0

Some implementation notes

MultiplayerSpawner:

• The spawner must know which scenes it can spawn, and for those scenes, how to encode and decode the spawn state.
• This is not known before the given scene is instantiated or added to tree (so the state cannot be properly validated before the scene is added to tree).
• Nodes will be automatically despawned when they leave the tree of the multiplayer authority or can be manually despawned.

MultiplayerSynchronizer:

• Sending each synchronized node separately would result in abysmal network performance.
• So by default, synchronizations will be sent in bulk, according to the sync_interval.

While we strive to support as many use cases as possible, the default behaviours might not fit all needs given the wide range of multiplayer games (from 3v3, to battle royale, to MMORPG), so while we should definitely provide more features out of the box (interpolation, interest management, etc) those are left out of this proposal, which will instead at least provide a way for game developers to deeply customize the spawn and sync process if needed.

Some optimization notes

Optimizations, and bandwidth optimizations in particular, are crucial to an effective networking.

• Selecting multiple properties and properties of child nodes is very useful in the prototyping stage, but bad in terms of potential optimizations.
• We should allow specifying compression levels and/or encoding types for the replicated properties as a mitigation.
• A very quick way to optimize the network code later on is to have a single property replicated, that return a tightly packed representation of the object state based on your game unique characteristics.
  When done properly, this is also going to be the most optimized state possible, that no tool can produce for you.
• We should still squeeze the state size as much as possible with the information we have.

Lower level replication API

A lower level replication API, (part of MultiplayerAPI) will be used by the higher level nodes to implement those functionalities and an extensible MultiplayerReplicatorInterface will allow overriding the behaviour globally, or on a per-node basis.

The lower level API will accept Object parameters (and not just nodes), but the default implementation will only work using the higher level nodes.

The MultiplayerReplicatorInterface will receive the MultiplayerSpawner or MultiplayerSynchronizer as configuration for the nodes being spawned/synced.

class MultiplayerReplicatorInterface extends RefCounted:

    ## Virtual functions, overridable (GDVIRTUAL)

    # Spawn, when a node is spawned via spawn, auto-spawned, or freed after spawn. "obj" is the node, "config" is the MultiplayerSpawner
    func _on_spawn(obj: Object, config: Variant) -> Error
    func _on_despawn(obj: Object, config: Variant) -> Error

    # Sync, when a MultiplayerSynchronizer enters/exit tree. obj is the node at root_path, config is the MultiplayerSynchronizer itself.
    func _on_replication_start(obj: Object, config: Variant) -> Error
    func _on_replication_stop(obj: Object, config: Variant) -> Error


# New members of the MultiplayerAPI
class MultiplayerAPI extends RefCounted:

    # Allow overriding the low level replication system.
    func set_replication_interface(interface: MultiplayerReplicatorInterface)

    # Notify a new node has been spawned.
    func spawn(obj: Object, , config: Variant)

    # Notify a previously spawned node has been freed.
    func despawn(obj: Object, , config: Variant)

    # Notify a new node has been marked/unmarked for replication (i.e. a MultiplayerSynchronizer with a valid root_path has entered or exited the tree)
    func replication_start(obj: Object, , config: Variant)
    func replication_stop(obj: Object, , config: Variant)

This changes the current API quite a bit, but should incorporate the same concepts discussed during meetings:

• The low level implementation no longer depends on resource IDs.
• The custom callables are replaced by the MultiplayerReplicatorInterface, which will also have GDExtension support for performant send/receive (using pointers instead of PackedByteArray).
• You should be able to customize the behaviour with only a 1 byte overhead (the MultiplayerAPI command byte) instead of 9 given the resource ID is gone.
• The sync_all method has been removed. Each node is synced independently (although bulking happens) replication start/stop will be called on each object instead (and you can decide to batch them as you wish with the custom interface).
• The track/untrack methods are also removed. Since spawn is now explicit, you will be able to track objects in the _on_spawn_send/_on_spawn_receive callback yourself.

Further Work

The long discussions over this topic spawned (pun intended!) a lot of ideas for further improvements but this proposal is already dense enough to further expand on them:

• It would be really nice to have the same interface-style override for RPCs (which also has been requested during meetings).
• Interest management should be explored, either via groups, visibility, ...
• The default server tick rate (rate at which the server sends update) will be configurable, but we should explore either specifying a per-replication tick rate, or (probably better) a priority based system for automatic throttling.

If this enhancement will not be used often, can it be worked around with a few lines of script?

It's a highly requested core feature of the engine.

Is there a reason why this should be core and not an add-on in the asset library?

It needs to be integrated to the multiplayer API.

[Faless]

CC @reduz @mhilbrunner @AndreaCatania @SaracenOne @Shatur @jordo @fire @Calinou @jonbonazza

[jordo]

I really appreciate the thought of building the higher level API abstraction on top of a Lower level replication API. This is a great design decision. I can imagine developers familiar with networked games wanting more flexibility at a lower level than a SceneTree implementation, so this is a nice touch.

For clarification, we should not confuse server tick-rate with the rate at which the server sends updates. These two are different things. But configuring the rate at which the server sends updates is definitely needed.

Also appreciate thought going in here: The custom callables are replaced by the MultiplayerReplicatorInterface, which will also have GDExtension support for malloc-free send/receive (using pointers instead of PackedByteArray). We have already made performance improvements to the engine in order to improve performance regarding Godot's memory operation overhead, so the above consideration is especially nice to see.

[reduz]

I think large part of the proposal, and large part of the initial implementation are derived from this single assumption:

The spawner must know which scenes it can spawn, and for those scenes, how to encode and decode the spawn state.

I think this is where the crux of our difference in perspective lies. In my view, spawn state is unnecessary and redundant, so I will try to make my point why:

How do you instantiate a scene normally in a regular game? (non-multiplayer)
Let's go for something simple like a game level.

var a = load("res://gamelevel.tscn")
add_child(a)

How do we do this in a multiplayer game? You instantiate the same scene.
Does this scene need spawn state synchronization for this? No, because the scene is the same level and same file on both ends, hence they by default initialize to the same state.

Sure, enemies may move around later, but this happens later, the initial state is the same and needs no synchronization.

Let's go for a bit more complex use case, like something that we will spawn multiple times, like a bullet in a single player game:

var b = load("res://bullet.tscn")
b.linear_velocity = $player.facing_axis
b.position = $player.gun.position
add_child(a)

What is the difference to the previous example? That spawning something multiple times requires code to set it up.

How does this work with multiplayer? This proposal assumes that we will probably set linear_velocity and position as spawn, so when added to the scene it will be synchronized. But is it really needed or the best way to do this?

Let's look at it from a different perspective. What we know for certain up to here is that code needs to run on initialization. In other words, the user will have to write this code, always.

So, instead of thinking in terms of spawn state, why don't we think in terms of spawn code? We now have the NetworkSpawn node, this node could have a virtual function that the user can hook into: Node _spawn(varargs). We write this code:

extends NetworkSpawn
class_name BulletSpawn

func _spawn():
  var b = load("res://bullet.tscn")
  b.linear_velocity = $player.facing_axis
  b.position = $player.gun.position
  return b

Then, all the client needs to do is call:

$bullet_spawner.spawn()

This will call the spawn function, spawn the node, replicate and create this node over the network.

Again, remember that we stated this code needs to exist anyway. All we did is put it in a place where it makes more sense.

Of course, I can see that this does not exactly the same, facing_axis and gun_position may not have been properly synchronized, so for some reason you may want to send this information over the network at spawn time, so instead it could be like:

extends NetworkSpawn
class_name BulletSpawn

func _spawn(position,velocity):
  #validate what comes from the server
  position = clamp(position,min_range,max_range)
  velocity = clamp(position,min_range,max_range)

  var b = load("res://bullet.tscn")
  b.linear_velocity = velocity
  b.position = position
  return b

then

$bullet_spawner.spawn($player.facing_axis,$player.gun.position)

For reconnection (in case a peer appears out of nowhere with the game already running) the spawn node can remember that this instance requires those parameters and send them again.

So to sum up, my point on this is that:

• The initial states of scenes is always the same, so this does not need synchronization.
• As such, you always need spawning setup code to specialize the scenes.
• This code always does what it's meant to do.
• This code will be present anyway on both ends of the peers, and code always does what it's meant to.
• As such in my view, spawn state is for the most part, and in the majority of cases, is redundant to spawn code, and hence unnecessary.

This is why when I read that, when for spawning a spawn state is required, It feels odd. Maybe there are times where this is the case, but in the vast majority of them all that matters is that the same code is used to create the instance, not the same state.

I think the proposal and idea of implementation was imagined based on the concept that scenes simply appear out of nowhere and in any state and was designed to solve that synchronization, but on further analysis it should be possible to prove that scenes will always come from the same state + an initializer code snippet. Only ensuring the initializer code snippet runs on both ends is enough to have proper state initialization.

Spawn states impose a much higher toll of complexity into the system, when they are really not as useful.

[jordo]

Again, remember that we stated this code needs to exist anyway. All we did is put it in a place where it makes more sense.

I would argue this is a worse place to put this logic.

The example you started with is simple game logic:

var b = load("res://bullet.tscn")
b.linear_velocity = $player.facing_axis
b.position = $player.gun.position
add_child(a)

The above example creates a new entity, and sets up various state data that this entity is intialized with. It's part of the logical state of the game's simulation, and is probably what a lot of developers are already doing. You want this logic to be part of the game simulation code (wherever that may be), NOT the netcode.

Putting this game logic into the proposed virtual function of a network node, is not good. You want isolation here:

extends NetworkSpawn
class_name BulletSpawn

func _spawn(position,velocity):
  #validate what comes from the server
  position = clamp(position,min_range,max_range)
  velocity = clamp(position,min_range,max_range)

  var b = load("res://bullet.tscn")
  b.linear_velocity = velocity
  b.position = position
  return b

Likewise overriding Node _spawn(varargs) with a varargs is not going to be maintainable, and is going to hinder fast iteration. When this vararg list gets to be 20 or 30 pieces of data long it's going to end up a mess. And isn't there a limit to the varargs binding number as well anyways? iirc last time I checked it was 7 or something, i could be wrong however.

[jordo]

But part of the idea is you don't want to have to be explicit in your game simulation logic about what should happen at the network sync level. This can be configured independently of your game logic. If you can mark certain scenes as replicated/synced, then you don't actually have to put this line:

$bullet_spawner.spawn($player.facing_axis,$player.gun.position)

anywhere into your game code, which is a good practice. You can operate as normal, as if you were building a single player game. And this is a good thing. Because if you want to end up adding advancements to this networking module in Godot (say client-side prediction, etc), it is going to be CRUCIAL to have the correct separations of concerns here. You do not want your netcode interleaved with your game logic code, because in advanced scenarios, (say client-side prediction, rollback and resimulation) these will run completely separate and independently.

[reduz]

@jordo Again, as I described, marking scenes replicated and synced is a no go because it's a large security risk. You will need to go via a spawner node that controls the location, types and lifetime of instantiated scenes. I think that part is not on discussion.

[reduz]

@Fales In any case, I think the proposal overall is really good, I am just nitpicking on details of spawning, because I would really like to have the ability to do as I described, even if alternatively, but for the most part I think it's great.

If I understood the limitation well, I don't think you really need to know the scene beforehand to synchronize the parameters when spawning it. As I described, this can be a simple hash of scene + properties contained in node that can group them internally.

[jordo]

@reduz That's a small part of the points I brought up, and you are misinterpretting... I was referring to this screenshot that @Faless posted, which are checkboxes (marking) on the spawner node.

There is zero security issues with the points I brought up. This above is a perfect design (according to your concerns) regarding security. Scene's are checked (allowed) at the spawn node, and a target node path is set for spawn. Is there anything else you are concerned with?

[reduz]

@jordo oh but it's the same, in @Faless proposal you do something like this instead:

var b = load("res://bullet.tscn")
b.linear_velocity = velocity
b.position = position
$bullet_spawner.spawn( b )

It's not too different, but the main change is that you don't really have much control on the spawning logic. What I argue is that I prefer to write the spawn code once and running in both peers, rather than having two separate systems doing this.

To me using spawn properties for this is a waste of time and making something more complex for no purpose. If you folks really want to use it and believe it's useful or better represents your way of thinking, I am not against having them. I would myself prefer any day to use spawn parameters having the choice, so I am advocating for this also being supported, even if you don't use them.

[jordo]

I will advise, with 100% certainty, that you do not want the developer to have to write your proposed 4th line of code here:

var b = load("res://bullet.tscn")
b.linear_velocity = velocity
b.position = position
$bullet_spawner.spawn( b )

Regardless of whether this fits into your design or @Faless design. Frankly, this is a must to avoid.

[jordo]

This should be

var b = load("res://bullet.tscn")
b.linear_velocity = velocity
b.position = position
$Players.add_child( b )

Which will work in both single player and multiplayer. And this is the pattern already used by developers using Godot right now

[reduz]

@jordo As I mentioned, thanks for your advise, but this is done explicitly for security reasons.

If you want to make a game where the server is 100% trusted, and you took measures again potential MITM, and your code does lots of checks to avoid problems with this, then that's up to you, but it won't be the case for all the other users. I prefer this is secure by default, even if it's a bit more hassle to have to force users to explicitly state what, where and when they will replicate scenes.

Explicit spawning to me is best. Nothing also prevents the spawn node from working single player if networking is not used.

[jordo]

Explicit spawning 'code' is going to end up sending the same data on the wire... Your argument for security falls flat, because you can MITM either way.

Can you explain to me (from the wire's perspective), and a potential MITM injector, what the difference would be on the wire for code that explicitly spawns an entity $spawnser.spawn(), and a code than spawns and entity with $Players.addChild(). There is no difference. The wire data, and security concerns, will be the exact same. The only difference is you've made implementing multiplayer much more difficult for the developer, and adding limitations to what can be done in the future with network prediction and rollback logic. (Which requires separation of network logic and game logic)

[jordo]

If you are making a MITM security argument, please explain what would be different in the wire format in each scenario. Because I can't see how they are different.

[reduz]

@jordo There is plenty of difference:

• With the spawner node you can control where the scene is instantiated. Otherwise it will be possible to instantiate it anywhere.
• With the spawner node you can control when the scene is instantiated, otherwise it may be possible to instantiate it at a time where it's not safe for it to be there.
• With the spawner node you can also control which type the scene is instantiated at the right time and place.

All these 3 are huge security improvements and the users dont have to worry about inadvertently making something they can mess up.

[reduz]

If you are making a MITM security argument, please explain what would be different in the wire format in each scenario. Because I can't see how they are different.

For the spawner node scenario, there is implicit validation, for the other scenario there is not.

[jordo]

All three of those security concerns are addressed in the screenshot @Faless posted, I will post again for your review:

1): Spawner node is the scene tree. (spawner node, left side) Check

2): Spawner node has a node path target. (spawn path, right side) Check

3): Spawner node has a limit of what scenes can be spawned. (replicated scenes, bottom) Check

[reduz]

@jordo Oh ok, either I misunderstood you or you missed a vital piece of information in your reasoning here (or both) :)

So having to guess, what you imply is that the spawner node is needed, but you still want it to magically detect when a node is added to the scene and synchronize it?

If this is the case, I am also against this, I really prefer explicitness, I am not a fan of automagically doing things for users. User is not dumb.

[jordo]

If you are making a MITM security argument, please explain what would be different in the wire format in each scenario. Because I can't see how they are different.

For the spawner node scenario, there is implicit validation, for the other scenario there is not.

This was honestly a legitimate question. If you want to make that argument, let me know what the difference is on the wire. there is implicit validation is incredibly hand wavy, and if you believe there is a difference you will have to explain from the wire format's perspective.

[reduz]

@jordo read the post above yours.

[jordo]

So having to guess, what you imply is that the spawner node is needed, but you still want it to magically detect when a node is added to the scene and synchronize it?

If you read any of my comments, I never once said or suggested anything against the spawner node... So I'm not sure where you got that idea. I prefer the node design as I referenced @Faless proposal screenshot.

If this is the case, I am also against this, I really prefer explicitness, I am not a fan of automagically doing things for users. User is not dumb.

If this ends up being case, this will end up being an unfortunate limitation for Godot. The explicitness is defined within the parameters of the SpawnerNode, and there is no need to complicate the game logic. As I mentioned, you really want separation of concerns here.

[reduz]

If this ends up being case, this will end up being an unfortunate limitation for Godot. The explicitness is defined within the parameters of the SpawnerNode, and there is no need to complicate the game logic. As I mentioned, you really want separation of concerns here.

It does not complicate the game logic, It's the same amount lines of code and the meaning of the code is more explicit. This is always good to me. I am sorry you may not share some of my views or values in software design, but I strongly abide by them.

[reduz]

To be clearer, anything that helps better understand the intention of the code when reading it is good for me.

[jordo]

I am sorry you may not share some of my views or values in software design, but I strongly abide by them. ? probably uncalled for, but OK.

[reduz]

probably uncalled for, but OK.

@jordo sorry, most likely cultural difference, did not meant to offend or anything of the sort. I just meant that in my experience, when given the choice, it is always better in the long run to make intention explicit and clear in APIs, programming languages, user interfaces, etc. It helps others (teammates or users) get familiar with what's going on faster.

[reduz]

@Faless How about what I suggested as a spawn_custom(args) / Node _spawn_custom(args) function instead? This way you can optionally use manually spawned/configured nodes if you wish.

[jordo]

OK, so as far as I can tell. @reduz, your first reason for changing the proposal was security related, but as discussed above, it seems your comments in your points (1,2, and 3) are resolved. So currently right now, as far as I can understand, your current issues are issues with:

magically detect when a node is added to the scene and synchronize it

and

I am sorry you may not share some of my views or values in software design, but I strongly abide by them.

In your first statement, you are just using the adjective 'magically' in a negative context to make your point. There's nothing wrong with the network sync system taking care of this for you, whether you want to call it 'magic' or not. It could be considered 'magic' when someone places a node in the scene tree and it appears on screen. Which is why you don't want to be explicit everywhere. This is why developers don't have to explicitly write say vulkan commands. This stuff is supposed to be 'magically' handled by the engine as it's just an abstraction.

The second statement is basically an argument that your software design principles are better than others, and it bothers me, because it's actually opposite of good software engineering principles, and even generally accepted OO principles. What you want to strive to build are systems that work together with HIGH cohesion, and LOOSE coupling. By requiring developers to explicitly add in network code to their game code, example: $bullet_spawner.spawn( b ), you are now TIGHTLY coupling these systems together. Why is this bad? Well it's bad for a number of reasons:

• If I want to replay game code, I can't replay game code without it touching network code. Bye bye client prediction and rollback, because you can't run your game logic code separately without it touching netcode (they are now interleaved).

• It makes maintainability harder now as now it's harder now to remove network code. With the current addChild proposal, if I want to take synchronization out, I just delete the SpawnerNode. With your suggestion of always being explicit in code, I now have to delete the SpawnerNode and delete the$bullet_spawner.spawn( b ) line, and also refactor the implementation of $bullet_spawner.spawn( b ) to be located elsewhere.

• Likewise, it makes it more difficult to add network capability to a scene. In the addChild proposal, all you would need to do is add a NetworkSpawner to a scene, and connect up the correct properties. This doesn't require the developer to change any additional code to add network capability, and Godot developers can write applications like they are used to like below:

var b = load("res://bullet.tscn")
b.linear_velocity = velocity
b.position = position
$Players.add_child( b )

• You won't be able to make any temporal adjustments if and when you need to. Maybe you want to lag render something, or you want to fast forward delta time a bit in game logic to account for a visual artifact due to latency. Well now modifying that process also modifies the network synchronization. If you de-couple the network synchronization code from the game logic code it allows you to do these things easier.

• And lastly and most importantly, you want to have the ability to clearly reason about your client game logic and your server game logic. The easiest way to do that, and make sure client and server run the exact same simulation given the same state, is to actually run THE EXACT SAME CODE on the client as on the server. You don't want to scatter a bunch of network .spawn( ) functions everywhere. What if you miss one, or add a child to a spawn path without calling .spawn() function, etc. Synchronizing what's added to the spawn_path by it's children takes care of all of this for you, there is no potential to screw up your state by missing anything.

At any rate, take it for what it is. I like the current proposal, with the exception of I would like to see func spawn and func despawn removed from the MultiplayerSpawner as I think it should just sync whatever the children of it's spawn target path are. (With the ability to provide a hook / virtual function for per peer filtering (that can be implemented in netcode) which can be organized correctly outside of game logic code). This has all the benefit of the above points, but especially the benefit of avoiding putting spawn() function calls everywhere in your game logic code.

[WolfgangSenff]

Sorry to butt in, @jordo, just wanted to say I think what @reduz meant re: the second point was not that your principles were inferior at all, but that there are legitimately good programmers that disagree with his principles on the point - i.e. there are a lot of people who would prefer the tersest but still sensible declaration of it (he's mentioned this before on Twitter, for context).

Further to the second point, unfortunately, as usual, the OO blade cuts both ways: by adding an extra responsibility to add_child, you're breaking the Single Responsibility Principle for the class itself. In reference to the first point, I can really see what reduz meant: I would hate it if add_child did something like this without my explicit desire/declaration, personally. I shouldn't need to know all the side-effects of the underlying calls I'm making, but it feels like in this case I would absolutely have to, and it has led to a very leaky abstraction.

That said, I do understand your point about putting extra work on the developers using the engine and that makes it a pain-point. I think having a pain-point that requires some refactoring and code updates is not the end of the world, but is sometimes the only thing that makes sense - you wouldn't want a single function having a bunch of extra code added into it for the sake of convenience (not that that's what you were suggesting - just an example).

But I digress. If I had my druthers, it would simply be a node that, when the game is run, it would connect to a singleton signal that is raised whenever a child is added anywhere in the tree. It feels like something of a middle-ground, albeit not necessarily super-OO either - by adding a raise call in add_child, you wouldn't really be breaking the SRP because that signal is still defined as a thing for adding children, and you'd still have the better maintainability aspect that comes with your/Faless suggestion. (For the record, I have no idea if, in context, this suggestion makes any sense at all; just thinking out loud as I'm working on my own stuff.)

[MartinHaeusler]

I attempted a multiplayer game once, in another engine, and it was a smash bros clone (platform fighter with physics). One thing that was impossible to achieve (without rewriting the entire engine) was deterministic physics across machines, i.e. you track the input of all players and timestamp them, and by replaying those inputs you get exactly (frame-by-frame) the same result. It never worked out; physics interactions worked ever so slightly differently every time, causing a butterfly effect. I guess it's more of a physics topic than networking, but I wanted to post this here anyway because it would be a huge win for Godot if it could do this out of the box, as any physics-based multiplayer game suffers from this issue.

[Calinou]

I attempted a multiplayer game once, in another engine, and it was a smash bros clone (platform fighter with physics). One thing that was impossible to achieve (without rewriting the entire engine) was deterministic physics across machines, i.e. you track the input of all players and timestamp them, and by replaying those inputs you get exactly (frame-by-frame) the same result. It never worked out; physics interactions worked ever so slightly differently every time, causing a butterfly effect. I guess it's more of a physics topic than networking, but I wanted to post this here anyway because it would be a huge win for Godot if it could do this out of the box, as any physics-based multiplayer game suffers from this issue.

This should be discussed in a separate proposal – see #2821.

Note that while there are plans to improve physics determinism in future 4.x releases, it's unlikely that GodotPhysics will ever be fully deterministic. If fully deterministic physics are required, it's best to write your own physics implementation or use a third-party physics engine. Also, the use of floating-point coordinates should be avoided whenever possible to improve determinism across platforms and CPU architectures.

[AndreaCatania]

Perfect determinism is impossible to achieve without sacrificing a lot of performance, but in any case Godot physics / bullet physics are already deterministic enough to create networked games. In past months I been creating samples with networked balls, characters, bullets: even with high latency the game was responding correctly. Tbh, it’s not even needed perfect determinisms at the physics level, because is expected to lose determinism by losing some sent packets on the road. What make a lot of difference is the recovery mechanism, that should be able to smoothly correct the player. If that algorithm works well, your game will be perfectly playable even with poor connections. So it’s much productive focus on the networking side, rather than the physics 😊🤘

[gedw99]

Great discussion .

What is the networking technology being used currently ?

i have been using NATS for games and other real time systems.
NATS makes it easy to standup a global cluster with 3 servers in the continents you want to serve.
Clients automatically connect to the closest cluster and automatically failover should a continent go offline .

Nats has c++, rust. Golang, js clients

100% open

single MacBook Air can do 8 million transactions a second . It’s highly performant.

Clients can run on anything . Iot. Web, mobiles. It’s very lean and has zero allocation design .

used by some big players .

Can someone let me know the current situation with the real one networking stack so I can get involved please ??

[dsnopek]

@MartinHaeusler: Most network synchronization techniques don't require true determinism, and like everyone above says, you can make really great online multiplayer games without it! That said, it just so happens that I created a 2D deterministic physics engine for Godot, and published a tutorial about getting started with it today:

https://www.snopekgames.com/tutorial/2021/getting-started-sg-physics-2d-and-deterministic-physics-godot

I specifically created it for implementing rollback & prediction netcode where you are synchronizing only the inputs (as opposed to some of the state as well), which is what it sounds like you were attempting to implement in your Smash Bro's clone.

[SaracenOne]

Sorry I haven't looked into networking stuff for a while, but wanted to chime in on this proposed revision. I can't really speak at the moment to any particularly strong opinions on how the high level nodes should be structured, but they seem decent enough. My main concern with the first API implementation was actually at the low level, that by having sync operate at per-scene rather than per-object, it essentially blocks the ability to do any kind of dynamic interest management. This draft seems a lot more flexible in that regard.

[jonbonazza]

by having sync operate at per-scene rather than per-object, it essentially blocks the ability to do any kind of dynamic interest management. This draft seems a lot more flexible in that regard.

thats actually a really good point. I havent had much time myself to read the proposal in detail. I believe I read an older iteration of it but i dont recall.

either way, being able to support dynamic interest management is certainly a critical requirement of whatever design we go with.

Hopefully Ill have some time soon to read through the current proposal and formulate some new opinions.

[jonbonazza]

Okay. Ive read through the existing comment thread and not only does it seem that not much more discussion has happened since I last read through it (in fact, most of the new discussion has been off topic or tangential at best), but it seems that there still hasn’t been a decision made in regards to @reduz ’s concerns and @jordo ’s rebuttal. I believe that resolving this dispute should be of a high priority since so many other network feature proposals—both existing and future—depend on this one.

@akien-mga what do we do about this?

[Faless]

@jonbonazza if you are referring to the automatic vs explicit spawn debate:
Both automatic spawning of scenes (defined via the editor), and explicit spawning via a spawn method (which requires a _spawn_custom virtual function) will be supported by the MultiplayerSpawner (see godotengine/godot#55950 ).

[nonunknown]

I'm just wondering if the replication sync will be able to be "disabled" by the client, because in my use case I have two situations where I would want to do this:

• Suppose we have 4 players connected: A,B,C and D

• player A is me, then when B and C get out of screen, I dont want to receive more information about their position anymore
• I have a game with N worlds, since the worlds are on the same network scene, I dont want to overload player A networks by receiving data from players on another worlds

[Calinou]

I'm just wondering if the replication sync will be able to be "disabled" by the client, because in my use case I have two situations where I would want to do this:

I believe this is known as interest management, which will be worked on in the future (but not in the initial implementation).

[agnosthesia]

While we're on that subject - are there standard techniques for that? It's not something I'm very familiar with..

For example, imagine you have things that are specific to your game or some context in your game. Like... there is a "cloaked player", and you want to make sure the network doesn't even tell other players where the cloaked player is (so the client side can't cheat and expose the person as if they were not cloaked). I would consider that a form of interest management, just not the typical ones listed above ("on different screens" etc).

Without thinking very hard, it seems like the type of thing where masks/layers (ala rendering and collisions) could be applicable..

[mhilbrunner]

FYI, just a small update about where things are: currently there is more work in my progress at godotengine/godot#55950

We're currently waiting for reduz to review those additional changes before proceeding in any direction. This may take a bit still, as reduz is currently understandably busy due to holidays and various IRL stuff. :)

We will update this issue once there is more information. Thanks to everyone providing feedback!

[jonbonazza]

@agnosthesia what you describe is indeed called “interest management” and while high level facilities for this won’t be included initially, it should certainly be doable if you wish to craft it yourself.

There are a couple common ways to do this:

1. a logical grid places on the works and only Showing netwirk objects that are in the same or adjacent cell to the player
2. A simple distance calculation to determine which entities can be seen.

Both have their own pros and cons.

there are others as well, but i believe these are the most common.

all of that said, this conversation is beginning to fall out of the scope of this particular proposal. Feel free to join the discord and/or developer chat to discuss it further with other interested parties (heh).

[agnosthesia]

Well, it came up via @nonunknown 's post because there's an obvious question of how this scene replication concept relates to interest management (if at all). I don't see anything wrong with discussing that - the work being done by @Faless and others is great too, and of course people can still discuss that.

After all, this is a public proposal board that asks for feedback from godot users about what's being discussed...

I'm definitely on the discord, but it's pretty hard to get a discussion like this going there in my experience.

[Faless]

Closing as this is now implemented via godotengine/godot#55950 .
Further discussion on interest management will continue in #3904 (will comment with my proposal in the coming days).

[kalysti]

I hope thats helpfull for the further development.

An real "authoritive server (fast-paced)" ->
https://github.com/sboron/godot4-fast-paced-network-fps-tps

[michaldev]

This is the final version? In latest alpha (and all other) this is very hard for debug. Errors like this without any details (source of code, or node id/name).

ERROR: ID 2 not found in cache of peer 518659612.
   at: get_cached_object (scene/multiplayer/scene_cache_interface.cpp:251)
ERROR: Condition "!sync || sync->get_multiplayer_authority() != p_from" is true. Returning: ERR_UNAUTHORIZED
   at: on_sync_receive (scene/multiplayer/scene_replication_interface.cpp:390)
ERROR: Unable to send packet on channel 2, max channels: 0
   at: send (modules/enet/enet_packet_peer.cpp:62)

[Faless]

This is the final version? In latest alpha (and all other) this is very hard for debug. Errors like this without any details (source of code, or node id/name).

Error reporting can be improved, reporting source of code or node name is not always possible unless you send that information along the wire, which is not viable if we want it to work within a reasonable bandwidth amount.
Possibly though, we can add some extra round-trip messages in debug mode when an error is detected in the node cache like in the first error.

What's really strange is this error:

Unable to send packet on channel 2, max channels: 0

Not sure if this happens after a disconnect, but the fact that the channels allocated by ENet is 0 is likely a bug, so if you can reliably reproduce that it would be great if you could open an issue. I'll poke around the code in any case.

[michaldev]

"Unable to send packet on channel 2, max channels: 0" - this is randomly. I don't see dependent between events (yet).

It also happens when player left the game. On player side error looks like this:

ERROR: Condition "!peers_info.has(p_peer)" is true.
   at: peer_sync_recv (scene/multiplayer/scene_replication_state.cpp:256)
ERROR: No cache found for peer 250216166.
   at: get_cached_object (scene/multiplayer/scene_cache_interface.cpp:248)
ERROR: Condition "!sync || sync->get_multiplayer_authority() != p_from" is true. Returning: ERR_UNAUTHORIZED
   at: on_sync_receive (scene/multiplayer/scene_replication_interface.cpp:390)

On the server side probably in the same time:
ERROR: Condition "!peers.has(p_to)" is true.

When player left the game, I remove node (by queue_free). This node is synchronized by "multiplayer synchronizer" node, and spawned with controller "multiplayer spawner".

In authority problem I understand the reason. Only source of problem - no.

I don't know the reason of the other errors. Maybe they are consequence of "authority problem".

I can send you my code (2 main scenes, max 10-15 nodes), but only on PM, because it's private. I don't except resolution/support. Just if you feel that it could help you with growing engine, I can send you.

[michaldev]

Can I use MultiplayerSpawner in Node spawned from other MultiplayerSpawner?

[japp-ctapuk]

Am I interested in having a room on the server? For example, the global world or the local world is temporarily generated by a scene that players can enter inside the scene.
Simply put, by default, the players enter the main scene, and if the player goes to another scene, it replaces the contents of the objects that he sees.

Tree structure example
-Root (Global Replicator Room)
-- Player 1
-- Player 2
-- Room 1 (Local Replicator Room)
--- Player 3
-- Room 2 (Global Replicator Room)
--- Player 4

As a result of the rest of the settings, this is already replication synchronization from the object to the parent, which is looking for descendants, if there is a room, then they will replicate if it turns off.

As you can see in the structure of the tree, player 1 and player 2 both see in the game but do not see other players and do not transmit information from players 3 and 4 since they are in another room and other players 2 and 3 also do not see other players, this will reduce huge packs when players accumulate in the main scene. You can also specify a replicator spawner not a list of objects.
So that it has a setting such as transmitting data from the player’s camera, if they are not included in the visibility range of the cameras, then data from the server will not be transmitted except for rpc

PS Sorry for the translation. I myself am Russian and I love to program but I can't write in English