Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add initial Protobuf support #3448

Merged
merged 39 commits into from
Feb 17, 2023
Merged

Add initial Protobuf support #3448

merged 39 commits into from
Feb 17, 2023

Conversation

DrusTheAxe
Copy link
Member

@DrusTheAxe DrusTheAxe commented Feb 17, 2023

Add support for Google's Protocol Buffers (aka protobuf)

Using protobuf requires 3 components:

  1. protoc.exe -- 'Compiles' *.proto files generating *.pb.cc and *.pb.h code
  2. headers -- Needed to compile generated *.pb.cc
  3. libs -- Needed to link compiled bits from 2

Google provides a nuget containing a compiled protoc.exe but doesn't make headers or libs available via nuget. TL;DR we create a nuget for our use (Microsoft.WindowsAppSDK.Protobuf.3.21.12.nupkg). Details of what, why and how are in tools\nuget\protobuf\README.md.

TL;DR Developers working in WinAppSDK only need to know there's a nuget providing protobuf support. The messy details how to create that are only relevant to the developer creating the nuget (moi) or future devs if/when a new version is needed.

Added KozaniProtocol containing Kozani's protobuf messages and related definitions. The purpose of KozaniManageProtocol project is to contain all of Kozani's protobuf definitions and compile them to produce the generated code for use by other projects.

Updated KozaniManager to consume the protobuf code from KozaniProtocol and added wrappers showing how to use it.

Updated KozaniRemoteManager to reference to consume the protobuf code from KozaniProtocol.

General structure of our protobuf usage:

  1. Define a message in KozaniProtocol.
  • Split up by functional roles across *.proto files e.g. Kozani.Activation.proto for activation, Kozani.Process.proto for process management (e.g. if TaskManager kills local KozaniHostRuntime.exe we need to send message to server to terminate the associated back end process), etc.
  • Any sort of 'synchronous communication' would involve a pair of request+response messages. KozaniManager sends 'request' to server and KozaniRemoteManager sends a related 'response'.
  • The 'cookie' field is an example of a correlating id to match a request with a response. A 'conversionid', 'channelid', etc are other examples how to xref 2+ messages together into a larger context.
  1. Define a namespace with functions that internally use protobuf messages
  • Keep all protobuf usage internal to code using them. Protobuf is an implementation detail. Provide appropriate strongly typed functions for callers to drive activity which internally happen to use the protobuf generated code.
  • If you need context spanning multiple messages you can create a class with methods which internally use protobuf messages, plus additional attributes for any additional data needed for the context.
  1. Serialize messages to std::string
  • Protobuf can serialize messages to std::string or std::ostream. NOTE: The serialized data's just bytes, string is just a convenient container to pass the data around.
  • std::string is recommend when serializing a message to bytes.
  • std::string or std::istream is recommended when deserializing a message from bytes. Large messages may be more efficient via std::istream; either works well enough for small messages so use whichever is more convenient.
  1. Always encode strings as UTF8 before serialization.
  • Protobuf expresses message string fields as std::string. It does not do wide<->narrow conversions for you (unlike, say, SQLite) - that's the developer's responsibility. If you have a wide string (std::wstring, PCWSTR, HSTRING, etc) convert it to a UTF-8 string before assigning it to a protobuf field. Use functions in \dev\common\Microsoft.Utf8.h to convert wide->utf8 e.g.
PCWSTR appUserModelId{ L"LolzCatzVidz" };
const std::string appUserModelIdUtf8{ ::Microsoft::Utf8::ToUtf8(appUserModelId) };
  • When deserializing wide strings from protobuf serialized bytes don't forget to convert the UTF-8 bytes to a wide string. Use functions in \dev\common\Microsoft.Utf8.h to do this e.g.
Some::Protobuf::Message::Kitteh kitteh;
kitteh.ParseFromString(stream_containing_serialized_bytes);
const std::wstring name{ kitten.get_name() };
  1. Avoid making classes inherit from protobuf's generated classes.
  • Protobuf docs counsel against against inheriting and extending the generated classes. Treat protobuf's generated classes as structs of data you can access but not extend (use composition esp private composition, not inheritance).
  1. We use protobuf as a static library.
  • Protobuf can provide support code via libprotobuf.dll but recommends against it as that must have a compatible version as the generated code. To minimize complications we use protobuf as a static lib. This may be revisited in the future.

Everything compiles and links. dev\Kozani\KozaniManager\main.cpp has an example serializing a protobuf message to bytes (as a std::string). Changing and extending that for all the rest of our functionality and likewise parsing bytes to protobuf messages in KozaniRemoteManager (or vice versa) is left as an exercise for the reader :-)

…directory if present, and online if necessary. Add new option -OnlineVSWhere to ignore the local VSInstaller copy and use the latest VSWhere from the net.
…erProtocol for ProtocolBuffers definitions for (Local)Manager<->RemoteManager communication.
@DrusTheAxe DrusTheAxe added area-Infrastructure Build, test, source layout, package construction (TODO: move to Deployment, DeveloperTools) area-Kozani labels Feb 17, 2023
@DrusTheAxe DrusTheAxe added this to the 1.3 milestone Feb 17, 2023
@DrusTheAxe DrusTheAxe self-assigned this Feb 17, 2023
Copy link
Contributor

@rhuang-msft rhuang-msft left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

@DrusTheAxe
Copy link
Member Author

/azp run

@azure-pipelines
Copy link

Azure Pipelines could not run because the pipeline triggers exclude this branch/path.

@DrusTheAxe DrusTheAxe merged commit bdafab1 into develop Feb 17, 2023
@DrusTheAxe DrusTheAxe deleted the user/drustheaxe/protobuf branch February 17, 2023 23:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area-Infrastructure Build, test, source layout, package construction (TODO: move to Deployment, DeveloperTools)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants