App serializer POC

[seanimam]

Description

Proof of concept for a client side app serializer. This falls under the umbrella of solving the problem of having client derived data that can be accessible at rest/server side.

Understanding the code

The App serializer consists of three main components:

1. The Dependency class

This class encapsulates a given object (DDS) or primitive and is responsible for:

1. Retrieving the actual value you wish to track from said object via the getValue() method.
2. Optionally determining if the delta between two versions of the value type being tracked are "qualified" via the 'qualifier(prev, next)method. Checking if a dependency's value change is "qualified" is used downstream in the next class,DependencyChangeEffect`.

**Note that the Dependency class does not actually track the previous and next values.

2. The DependencyChangeEffect class

This class takes an array of Dependency's and one callback (aka the effect) as its constructor arguments. When the trigger() method of the DependencyChangeEffect is fired, the class will check if at least one of its provided Dependency classes has changed and uses the optional , Dependency.qualifier(prev,next) of each to determine if the change matters/is qualified. If atleast one Dependency has a change that is qualified then the DependencyChangeEffect class will fire the provided callback and save the results of said callback to an internal member variable.

In practice for app serialization, we provide one or more DDS's whose values we depend on each other to produce a single part of the serialized state of your app as Dependency's to the DependencyChangeEffect class and provide a callback/effect that produces a part of the serialized state of our app using said DDS's. Finally, we then listen to the onChange() handlers of each DDS and call the DependencyChangeEffect.trigger() method as needed which will then reproduce the piece of serialized state. Great!, now we have independently generated, serialized pieces of our application state. that are only regenerated as needed.

**Note that unlike the Dependency class, the DependencyChangeEffect class in charge of tracking the previous values of each of its dependencies.

3. The AppSerializer class

This class takes an array of DependencyChangeEffect classes, an interval time and a SharedCell DDS handle. We established above that each DependencyChangeEffect in this case should be used for producing independently generated, serialized pieces of our application state. The AppSerializer will then be in charge of combining the pieces serialized state into one string and saving them to a SharedCell DDS on a given millisecond interval, if and only if one or more of the pieces has actually changed.

**Note The AppSerializer class can efficiently check if one of its DependencyChangeEffect's has changed on each interval with simple number comparisons thanks to the effectResultIteration of each DependencyChangeEffect which is a monotonically increasing number that goes up by 1 each time a DependencyChangeEffect produces a new result.

Future design considerations

• Adding more defaults to classes to reduce the current verbosity
• Creating a new object syntax that each DependencyChangeEffect` will output that can be used to produce multiple different types of data formats instead of just one string format.

Example of the app:

Setting up the app serializer:

You'll see that there are a few parts to creating the different segments in our app serializer.

First we have each serializationSegment which is effectively an individual piece of serialized state that depends on some number of dependencies.

A segment is represented by the DependencyChangeEffect class which takes an array of Dependency's and an effect function. When DependencyChangeEffect.trigger() is called, the class will determine if its Dependency's changed and use the optional qualifier of each Dependency to determine if the effect should be triggered. We hook into each DDS's .on(<event>) function to determine how/when we trigger() each DependencyChangeEffect.

In the example below, we trigger the DependencyChangeEffect on every single change to the DDS, but in practice we can use algorithms such as debounce from lodash to make the trigger execution more performant.

Finally, once we assemble our serializationSegment's, we can pass them to the AppSerializer class which will assemble the segments into a single serialized string on a defined interval and save it to a shared cell. In the future, we could consider saving the data as json that can be reassembled into one of many output formats

examples/client-logger/app-insights-logger/src/components/App.tsx

	const [appSerializer, setAppSerializer] = useState<AppSerializer>();

	React.useEffect(() => {

		const serializationSegments: DependencyChangeEffect<string>[] = [];

		// Author field
		if (field1Input !== undefined) {
			const field1Dependency: Dependency<string> = {
				getValue: () => field1Input.getText(),
				qualifier: (prev, next) => prev !== next,
			};

			// tracker for detecting dependency changes and running an effect that produces a serialization
			const serializationSegment = new DependencyChangeEffect([field1Dependency], () => {
				const val = `# Application Security Report\n\n- Author: ${field1Input?.getText()} \n\n`;
				return val;
			});

			// Setting up when to trigger the segment serialization
			new SharedStringHelper(field1Input).on('textChanged', () => {
				serializationSegment.trigger(); //TODO: Debounce this
			});

			serializationSegments.push(serializationSegment);
		}


		// The second and third input fields together make up the second segment
		if (field2Input !== undefined && field3Input !== undefined) {
			// dependencies
			const field2Dependency: Dependency<string> = {
				getValue: () => field2Input.getText(),
				qualifier: (prev, next) => prev !== next,
			};
			const field3Dependency: Dependency<string> = {
				getValue: () => field3Input.getText(),
				qualifier: (prev, next) => prev !== next,
			};

			// tracker for detecting dependency changes and running an effect that produces a serialization
			const serializationSegment = new DependencyChangeEffect([field2Dependency, field3Dependency], () => {
				const val = `## The description of the application\n\n "${field2Input?.getText()}" \n\n`;
				const val2 = `## The way the applications front end communicates with back end services\n\n ${field3Input?.getText()} \n\n`
				return val + val2;
			});

			// Setting up when to trigger the segment serialization
			new SharedStringHelper(field2Input).on('textChanged', () => {
				serializationSegment.trigger(); //TODO: Debounce this
			});

			new SharedStringHelper(field3Input).on('textChanged', () => {
				serializationSegment.trigger(); //TODO: Debounce this
			});

			serializationSegments.push(serializationSegment);
		}

		// The number of customer face api's counter.
		if (counter1 !== undefined) {
			// dependencies
			const counter1Dependency: Dependency<number> = {
				getValue: () => counter1.value,
				qualifier: (prev, next) => prev !== next,
			};

			// tracker for detecting dependency changes and running an effect that produces a serialization
			const serializationSegment = new DependencyChangeEffect([counter1Dependency], () => {
				const val = `## The number of customer facing API endpoints: ${counter1?.value}`;
				return val;
			});

			// Setting up when to trigger the segment serialization
			counter1.on('incremented', () => {
				serializationSegment.trigger();
			});

			serializationSegments.push(serializationSegment);
		}

		if (serializationSegments.length === 3 && appSerializer === undefined) {
			const initializeAppSerializer = async () => {
				const metadata = props.containerInfo.container.initialObjects.metadata as ISharedMap;
				const sharedCellHandle = metadata.get(sharedCellKey) as IFluidHandle<ISharedCell>;
				const sharedCell = await sharedCellHandle.get();
				setAppSerializer(new AppSerializer(serializationSegments, 5000, sharedCell));

			}
			initializeAppSerializer();
		}

		return () => {
			appSerializer?.stop()
		}

	}, [field1Input, field2Input, field3Input, counter1]);

Console logging the serialization:

Pulling the data from the api:

-See
server/routerlicious/packages/tinylicious/src/routes/summaries/getLatestSummaryApi.ts

[alexvy86]

Overall looks pretty good!

One thing I had envisioned is the sharedCell being more build-in, or at least have a pre-defined key name dictated by us, that the application would need to use in order for it to be considered the serialized app data. And should we be showcasing the markdown from the sharedCell being accessible in the summaries API response, or a different route? The point of having that sharedCell is that we would know where it is and how to process it, so we could access the contents server-side without further user code, right?

[alexvy86]

For a final version, I'd hesitate to expose this in this form. The fact we use git for the summaries should be a non-exposed implementation detail, I think.

[alexvy86]

Before putting these in a final version, I'd check with the team if we should be treating them as permanently immutable across the lifetime of a document. I don't think we have plans to change them, but not sure if we should be extending their use, particularly to something server-side.

[Josmithr]

Overall looks pretty good!

One thing I had envisioned is the sharedCell being more build-in, or at least have a pre-defined key name dictated by us, that the application would need to use in order for it to be considered the serialized app data. And should we be showcasing the markdown from the sharedCell being accessible in the summaries API response, or a different route? The point of having that sharedCell is that we would know where it is and how to process it, so we could access the contents server-side without further user code, right?

+1. But what you have here makes sense as an initial prototype.

One additional comment: I don't think we would want the cell to live under the root shared-map. A single root map isn't how all apps will model their data. It would probably be a bit better to have a recommended pattern for the cell being a separate item in the container schema (so next to the map, rather than under it, in this case).

[microsoft-github-policy-service]

This PR has been automatically marked as stale because it has had no activity for 60 days. It will be closed if no further activity occurs within 8 days of this comment. Thank you for your contributions to Fluid Framework!