From 50a212203724a7bba32fe30293aecd2e364ce98e Mon Sep 17 00:00:00 2001 From: Dekkonot Date: Wed, 26 Jul 2023 20:03:22 -0700 Subject: [PATCH 1/7] Add initial ARCHITECTURE document --- ARCHITECTURE.md | 69 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 69 insertions(+) create mode 100644 ARCHITECTURE.md diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md new file mode 100644 index 000000000..df0a60576 --- /dev/null +++ b/ARCHITECTURE.md @@ -0,0 +1,69 @@ +# Architecture + +Rojo is a rather large project with a bunch of moving parts. While it's not too complicated in practice, it tends to be overwhelming because it's a fair bit of Rust and not very clear where to begin. + +This document is a "what the heck is going on" level view of Rojo and the codebase that's written to make it more reasonable to jump into something. It won't go too into depth on *how* something is done, but it will go into depth on *what* is being done. + +## Overarching + +Rojo is divided into two main pieces: the server and the plugin. The server is what's ran on your computer (whether it be via the terminal or the visual studio code extension), with the plugin serving as its main client. + +When serving a project, the server gathers data on all of the files in that project, puts it into a nice format, and then sends it to the plugin. Then, when something changes on the file system, it does the same thing for only the changed files and sends them to the plugin. + +When it recieves a patch (whether it be the initial patch or any subsequent ones), the plugin reads through it and attempts to to apply it. Any sugar (the patch visualizer, as an example) happens on top of the patches received from the server. + +## Server + +Rojo's server component is divided into a few distinct pieces: + +- The web server +- The CLI +- The snapshotting system + +### The CLI + +The Comand Line Interface (CLI) of Rojo is the only interface for the program. It's initialized in `main.rs` but is hosted in `src/cli`. + +Each command for the CLI is hosted in its own file, with the `mod.rs` file for the `cli` module handling parsing and running each command. The commands are mostly self-contained, though may also interface with Rojo's other code when necessary. + +Specifically, they may interface with the web server and snapshotting system. + +### The Snapshotting System + +To do what it does, Rojo has to do two main things: it must decide how the file system should map to Roblox and then send changes from the file system to the plugin. To accomplish this, Rojo uses what's referred to as snapshots. + +Snapshots are essentially a capture of what a given Instance tree looks like at a given time. Once an initial snapshot is computed and sent to the plugin, any changes to the file system can be turned into a snapshot and compared directly against the previous snapshot, which Rojo can then use to make a set of patches that have to be applied by the plugin. + +These patches represent changes, additions, and removals to the Roblox tree that Rojo creates and manages. + +When generating snapshots, files are 'transformed' into Roblox objects through what's referred to as the `snapshot middleware`. As an example, this middleware takes files named `init.lua` and transforms them into a `ModuleScript` bearing the name of the parent folder. It's also responsible for things like JSON models and `.rbxm`/`.rbxmx` models being turned into snapshottable trees. + +Inquiring minds should look at `snapshot/mod.rs` and `snapshot_middleware` for a more detailed explanation. + +Because snapshots are designed to be translated into Instances anyway, this system is also used by the `build` command to turn a Rojo project into a complete file. The backend for serializing a snapshot into a file is provided by `rbx-dom`, which is a different project. + +### The Web Server + +Rojo uses a small web server to forward changes to the plugin. Once a patch is computed by the snapshot system, it's made available via the server's API. Then, the plugin requests regularly, and if a new patch exists, recieves it and applies it in Studio. + +The web server itself is very basic, consisting of around half a dozen endpoints. The bulk of the work is performed by either the snapshot system or the plugin, with the web server acting as a middleman. + +## The Plugin + +This section of the document is left incomplete. + +## Data Structures + +Rojo has many data structures and their purpose might not be immediately clear at a glance. To alleviate this, they are documented below. + +### Vfs + +### ServeSession + +### ChangeProcessor + +### RojoTree + +### LifeServer + +### InstanceSnapshot From b2aed34a02ccd2362fc40546c209b53b46bb1eaa Mon Sep 17 00:00:00 2001 From: utrain <63367489+u-train@users.noreply.github.com> Date: Thu, 27 Jul 2023 02:29:58 -0400 Subject: [PATCH 2/7] Add `memofs` architecture documentation --- ARCHITECTURE.md | 6 +++-- crates/memofs/ARCHITECTURE.md | 48 +++++++++++++++++++++++++++++++++++ 2 files changed, 52 insertions(+), 2 deletions(-) create mode 100644 crates/memofs/ARCHITECTURE.md diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index df0a60576..16b2e2269 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -10,7 +10,7 @@ Rojo is divided into two main pieces: the server and the plugin. The server is w When serving a project, the server gathers data on all of the files in that project, puts it into a nice format, and then sends it to the plugin. Then, when something changes on the file system, it does the same thing for only the changed files and sends them to the plugin. -When it recieves a patch (whether it be the initial patch or any subsequent ones), the plugin reads through it and attempts to to apply it. Any sugar (the patch visualizer, as an example) happens on top of the patches received from the server. +When it receives a patch (whether it be the initial patch or any subsequent ones), the plugin reads through it and attempts to to apply it. Any sugar (the patch visualizer, as an example) happens on top of the patches received from the server. ## Server @@ -22,7 +22,7 @@ Rojo's server component is divided into a few distinct pieces: ### The CLI -The Comand Line Interface (CLI) of Rojo is the only interface for the program. It's initialized in `main.rs` but is hosted in `src/cli`. +The Command Line Interface (CLI) of Rojo is the only interface for the program. It's initialized in `main.rs` but is hosted in `src/cli`. Each command for the CLI is hosted in its own file, with the `mod.rs` file for the `cli` module handling parsing and running each command. The commands are mostly self-contained, though may also interface with Rojo's other code when necessary. @@ -58,6 +58,8 @@ Rojo has many data structures and their purpose might not be immediately clear a ### Vfs +To learn more, read about [`memofs` architecture](crates/memofs/ARCHITECTURE.md). + ### ServeSession ### ChangeProcessor diff --git a/crates/memofs/ARCHITECTURE.md b/crates/memofs/ARCHITECTURE.md new file mode 100644 index 000000000..475377c7c --- /dev/null +++ b/crates/memofs/ARCHITECTURE.md @@ -0,0 +1,48 @@ +# Architecture + +As the `README.md` says, this is an incomplete library that's dedicated to serving Rojo's purposes for the time being. +Meaning, there's still plenty of work to be done and the API will need to change. + +For the time being, this documents the current state of affairs. + +## VFS Interface + +The predominant object of `memofs`, it provides an interface for an abstract filesystem (known as a backend in `memofs`). + +### Backends + +Instead of `Vfs` providing a backend for you, it's flipped that it consumes a backend to use. +Essentially, [late-binding](https://ericlippert.com/2012/02/06/what-is-late-binding/) the backend so it's decided at runtime instead of compile-time. + +This is useful because you can implement any arbitrary backend and pass it along to `Vfs` without any changes to your project! + +For example, if you want to use a network drive instead of a local file system, +all that's required is to implement a `VfsBackend` to interface with the network drive. +Now, you can swap your local file system for the new one! + +There are common use cases for this feature, hence `memofs` provides several backends. + +#### In-memory + +As the name implies, it keeps all files and directories in memory. +This is particularly useful for testing, as it's easy to build, snapshot, and teardown. + +To help, `memofs` provides a `VfsSnapshot` object to snapshot the filesystem. The in-memory backend has methods to load from and save to a `VfsSnapshot`. + +#### Noop + +As the name implies, it does nothing. +As the name doesn't imply, every operation will error. +This is useful if you want to verify that your software doesn't perform any read/write operations. + +#### Std + +As the name implies, it provides an interface to the `std`'s filesystem API. Particularly, `fs_err` for nicer error messages! + +### Filesystem events + +`Vfs` additionally provides an event bus via `Vfs::event_receiver()`. +For any changes detected by the backend, it will be sent down that channel. +Only `std` actually provides any events. + +Additionally, there is a `Vfs::commit_event()` method, which will unwatch a path if a remove event is passed. From 90c74132de6e85fb25009131d52669d2dca49251 Mon Sep 17 00:00:00 2001 From: Micah Date: Thu, 27 Jul 2023 12:40:11 -0700 Subject: [PATCH 3/7] Make description of client and server more clear Co-authored-by: boatbomber --- ARCHITECTURE.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 16b2e2269..cb4f9b5f4 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -6,7 +6,7 @@ This document is a "what the heck is going on" level view of Rojo and the codeba ## Overarching -Rojo is divided into two main pieces: the server and the plugin. The server is what's ran on your computer (whether it be via the terminal or the visual studio code extension), with the plugin serving as its main client. +Rojo is divided into two main pieces: the server and the Studio plugin. The server runs with access to your filesystem (whether it be via the terminal, the visual studio code extension, or a remote machine), with the Studio plugin acting as its main client. When serving a project, the server gathers data on all of the files in that project, puts it into a nice format, and then sends it to the plugin. Then, when something changes on the file system, it does the same thing for only the changed files and sends them to the plugin. From 94f7c70ce883261a59eff3eebc08436a8536f8ba Mon Sep 17 00:00:00 2001 From: Micah Date: Thu, 27 Jul 2023 12:45:28 -0700 Subject: [PATCH 4/7] Make it less ambiguous Co-authored-by: boatbomber --- ARCHITECTURE.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index cb4f9b5f4..6d31cedb4 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -10,7 +10,7 @@ Rojo is divided into two main pieces: the server and the Studio plugin. The serv When serving a project, the server gathers data on all of the files in that project, puts it into a nice format, and then sends it to the plugin. Then, when something changes on the file system, it does the same thing for only the changed files and sends them to the plugin. -When it receives a patch (whether it be the initial patch or any subsequent ones), the plugin reads through it and attempts to to apply it. Any sugar (the patch visualizer, as an example) happens on top of the patches received from the server. +When the plugin receives a patch from the server (whether it be the initial patch or any subsequent ones), the plugin reads through the patch and attempts to to apply the changes described by it. Any sugar (the patch visualizer, as an example) happens on top of the patches received from the server. ## Server From 3da203ddec0889d6cbf1305b5f2a67a30638fb12 Mon Sep 17 00:00:00 2001 From: Micah Date: Thu, 27 Jul 2023 12:46:06 -0700 Subject: [PATCH 5/7] Correctly state the plugin long polls instead of saying it polls Co-authored-by: boatbomber --- ARCHITECTURE.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 6d31cedb4..e0a4ad681 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -44,7 +44,7 @@ Because snapshots are designed to be translated into Instances anyway, this syst ### The Web Server -Rojo uses a small web server to forward changes to the plugin. Once a patch is computed by the snapshot system, it's made available via the server's API. Then, the plugin requests regularly, and if a new patch exists, recieves it and applies it in Studio. +Rojo uses a small web server to forward changes to the plugin. Once a patch is computed by the snapshot system, it's made available via the server's API. The plugin is requesting patches regularly using a technique called [long polling](https://en.wikipedia.org/wiki/Push_technology#Long_polling) to receive the patches from the server and apply them to the datamodel in Studio. The web server itself is very basic, consisting of around half a dozen endpoints. The bulk of the work is performed by either the snapshot system or the plugin, with the web server acting as a middleman. From cee9992eb058645386380a875d236dcee73e0b1c Mon Sep 17 00:00:00 2001 From: boatbomber Date: Sun, 30 Jul 2023 12:50:01 -0700 Subject: [PATCH 6/7] Add first draft of plugin architecture (#2) --- ARCHITECTURE.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index e0a4ad681..a4b9775f7 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -50,7 +50,11 @@ The web server itself is very basic, consisting of around half a dozen endpoints ## The Plugin -This section of the document is left incomplete. +As mentioned above, the Studio plugin uses the web server to receive patches that contain the changes, additions, and removals to the Studio datamodel. The patch does not actually point at instances directly but rather uses string IDs. The plugin has an InstanceMap that is a bidirectional map between instance IDs and Roblox instances. It lets us keep track of every instance that Rojo knows about. + +To actually apply the patch to the Studio datamodel, the plugin has a Reconciler. The Reconciler uses the InstanceMap to find the instances mentioned by the patch, and then it attempts to modify/create/destroy instances as specified in the patch. It will also create and return a new patch that contains all of the operations that failed to apply (effectively creating a patch that if applied, would fix the failures), which is used to display warnings to the end user. + +The first time the plugin connects to the server is a special case, since there are no changes yet. The server sends the plugin the initial state, which is first used to hydrate the InstanceMap with all the currently known instances. Then, the Reconciler computes a patch by diffing the datamodel with the server state. This is known as the "catch up" patch, and that is what is shown in the Confirming page in the plugin. When that catch up patch is applied, the Studio datamodel should match the Rojo server and subsequent patches will be able to apply cleanly. ## Data Structures From 549bb73048a51a4c976267b1584a4221c74ee114 Mon Sep 17 00:00:00 2001 From: utrain <63367489+u-train@users.noreply.github.com> Date: Wed, 2 Aug 2023 20:29:54 -0400 Subject: [PATCH 7/7] Push ServeSession to the start (#3) Co-authored-by: boatbomber Co-authored-by: Micah --- ARCHITECTURE.md | 77 +++++++++++++++++++++++++++++++++++-------------- 1 file changed, 56 insertions(+), 21 deletions(-) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index a4b9775f7..724604c22 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -6,27 +6,25 @@ This document is a "what the heck is going on" level view of Rojo and the codeba ## Overarching -Rojo is divided into two main pieces: the server and the Studio plugin. The server runs with access to your filesystem (whether it be via the terminal, the visual studio code extension, or a remote machine), with the Studio plugin acting as its main client. +Rojo is divided into several components, each layering on top of each other to provide Rojo's functionality. -When serving a project, the server gathers data on all of the files in that project, puts it into a nice format, and then sends it to the plugin. Then, when something changes on the file system, it does the same thing for only the changed files and sends them to the plugin. +At the core of Rojo lies [`ServeSession`](#servesession). As the name implies, it contains all of the components to keep a persistent DOM, react to events to update the DOM, and serve the DOM to consumers. -When the plugin receives a patch from the server (whether it be the initial patch or any subsequent ones), the plugin reads through the patch and attempts to to apply the changes described by it. Any sugar (the patch visualizer, as an example) happens on top of the patches received from the server. +Most of Rojo's uses are built upon `ServeSession`! For example, the [`sourcemap` command](#sourcemapcommand) uses `ServeSession` to generate the DOM and read it to build the `sourcemap.json` file. -## Server +### The Serve Command -Rojo's server component is divided into a few distinct pieces: +There are two main pieces in play when serving: the server and the Studio plugin. -- The web server -- The CLI -- The snapshotting system +The server runs a local [`LiveServer`](#liveserver) with access to your filesystem (whether it be via the terminal, the visual studio code extension, or a remote machine). It consumes a `ServeSession` and attaches a web server on top. The web server itself is very basic, consisting of around half a dozen endpoints. Generally, [`LiveServer`](#liveserver) acts as a middleman with the bulk of the work is performed by either the underlying `ServeSession` or the plugin. -### The CLI +To serve a project to a connecting plugin, the server gathers data on all of the files in that project, puts it into a nice format, and then sends it to the plugin. After that, when something changes on the file system, the underlying `ServeSession` emits new patches. The web server has an endpoint the plugin [long polls](https://en.wikipedia.org/wiki/Push_technology#Long_polling) to receive the patches from the server and apply them to the datamodel in Studio. -The Command Line Interface (CLI) of Rojo is the only interface for the program. It's initialized in `main.rs` but is hosted in `src/cli`. +When the plugin receives a patch it reads through the patch contents and attempts to to apply the changes described by it. Any sugar (the patch visualizer, as an example) happens on top of the patches received from the server. -Each command for the CLI is hosted in its own file, with the `mod.rs` file for the `cli` module handling parsing and running each command. The commands are mostly self-contained, though may also interface with Rojo's other code when necessary. - -Specifically, they may interface with the web server and snapshotting system. +### The Sourcemap Command +### The Build Command +### The Upload Command ### The Snapshotting System @@ -42,12 +40,6 @@ Inquiring minds should look at `snapshot/mod.rs` and `snapshot_middleware` for a Because snapshots are designed to be translated into Instances anyway, this system is also used by the `build` command to turn a Rojo project into a complete file. The backend for serializing a snapshot into a file is provided by `rbx-dom`, which is a different project. -### The Web Server - -Rojo uses a small web server to forward changes to the plugin. Once a patch is computed by the snapshot system, it's made available via the server's API. The plugin is requesting patches regularly using a technique called [long polling](https://en.wikipedia.org/wiki/Push_technology#Long_polling) to receive the patches from the server and apply them to the datamodel in Studio. - -The web server itself is very basic, consisting of around half a dozen endpoints. The bulk of the work is performed by either the snapshot system or the plugin, with the web server acting as a middleman. - ## The Plugin As mentioned above, the Studio plugin uses the web server to receive patches that contain the changes, additions, and removals to the Studio datamodel. The patch does not actually point at instances directly but rather uses string IDs. The plugin has an InstanceMap that is a bidirectional map between instance IDs and Roblox instances. It lets us keep track of every instance that Rojo knows about. @@ -64,12 +56,55 @@ Rojo has many data structures and their purpose might not be immediately clear a To learn more, read about [`memofs` architecture](crates/memofs/ARCHITECTURE.md). +### LiveServer + +LiveServer underlies the [`serve` command](#the-serve-command) and provides the web server which clients (such as the Studio plugin) can use to interface with a [`ServeSession`](#servesession). + +The web server has two components: a UI and the API used by clients. + +The UI provides information about the current project's [tree](#rojotree), including metadata. It also shows the project name, up-time, and version its Rojo is on. + +The API provides a simple JSON protocol to interact with and receive changes from the underlying [`ServeSession`](#servesession). Checkout the [`api.rs` file under the web module](src/web/api.rs) to learn more. + ### ServeSession +The `ServeSession` is the core of Rojo. It contains all of the required components to serve a given project file. + +Generally, to serve means: + +- Rojo maintains a DOM and exposes it to consumers; +- Rojo is able to accept events to cause changes to the DOM; +- Rojo is able to emit changes to the DOM to consumer. + +It depends on: + +- [`RojoTree`](#rojotree) to represent the DOM; +- `Project` to represent your root project file (e.g. `default.project.json`); +- [`Vfs`](#vfs) to provide a filesystem and emit events on changes; +- [`ChangeProcessor`](#changeprocessor) to process filesystem events from `Vfs` and consequently update the DOM through the [snapshotting system](#the-snapshotting-system); + +It also provides an API for the higher level components so it can be used with the outside world: + +- There is a [`MessageQueue`](#messagequeue) consumers listen on for patches applied to the DOM; +- There is a channel to send patches and update the DOM; +- And a `SessionId` to uniquely identify the `ServeSession`. + +The primary interface for a `ServeSession` is the web server used by the [`serve` command](#servecommand). Additionally, several of Rojo's commands are implemented using `ServeSession`. + +### MessageQueue + +`MessageQueue` manages a persistent history of messages and a way to subscribe asynchronously to messages past or future. + +It does this primarily by having the subscribers keep a cursor of the message they're currently on. So, when a subscription does occurs: + +- If the cursor is behind the latest message, it will be immediately be fired with all messages past the cursor; +- If the cursor is on the latest message, it will fire on the next message given; +- If the cursor is ahead of the latest message, it will fire on the message after the cursor. + +### SessionId + ### ChangeProcessor ### RojoTree -### LifeServer - ### InstanceSnapshot