Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generate the serialisation logic in the messaging layer with tooling instead of the custom case-by-case logic #18

Open
whisperity opened this issue Jan 11, 2023 · 0 comments
Assignees
Labels
Kind: Enhancement 🔮 New feature or request Target: Communication protocol 💬⚠️ A part of the core library which deals with client-server messaging Target: Core 💥 Core architectural support library

Comments

@whisperity
Copy link
Owner

Currently, all of the serialisation logic for the control message primitives are all built on top of hand-crafted code:

ENCODE_BASE(ProcessSpawnOptions)
{
std::ostringstream Buf;
Buf << "<PROCESS>";
{
Buf << "<IMAGE>" << Object.Program << "</IMAGE>";
Buf << "<ARGUMENTS Count=\"" << Object.Arguments.size() << "\">";
{
for (const std::string& Arg : Object.Arguments)
Buf << "<ARGUMENT Size=\"" << Arg.size() << "\">" << Arg
<< "</ARGUMENT>";
}
Buf << "</ARGUMENTS>";
Buf << "<ENVIRONMENT>";
{
Buf << "<DEFINE Count=\"" << Object.SetEnvironment.size() << "\">";
{
for (const std::pair<std::string, std::string>& EnvKV :
Object.SetEnvironment)
{
Buf << "<VARVAL>";
{
Buf << "<VAR Size=\"" << EnvKV.first.size() << "\">" << EnvKV.first
<< "</VAR>";
Buf << "<VAL Size=\"" << EnvKV.second.size() << "\">"
<< EnvKV.second << "</VAL>";
}
Buf << "</VARVAL>";
}
}
Buf << "</DEFINE>";
Buf << "<UNSET Count=\"" << Object.UnsetEnvironment.size() << "\">";
{
for (const std::string& EnvK : Object.UnsetEnvironment)
Buf << "<VAR Size=\"" << EnvK.size() << "\">" << EnvK << "</VAR>";
}
Buf << "</UNSET>";
}
Buf << "</ENVIRONMENT>";
}
Buf << "</PROCESS>";
return Buf.str();
}
DECODE_BASE(ProcessSpawnOptions)
{
ProcessSpawnOptions Ret;
HEADER_OR_NONE("<PROCESS>");
CONSUME_OR_NONE("<IMAGE>");
EXTRACT_OR_NONE(Image, "</IMAGE>");
Ret.Program = Image;
{
CONSUME_OR_NONE("<ARGUMENTS Count=\"");
EXTRACT_OR_NONE(ArgumentCount, "\">");
std::size_t ArgC = std::stoull(std::string{ArgumentCount});
Ret.Arguments.resize(ArgC);
for (std::size_t I = 0; I < ArgC; ++I)
{
CONSUME_OR_NONE("<ARGUMENT Size=\"");
EXTRACT_OR_NONE(ArgumentSize, "\">");
if (std::size_t S = std::stoull(std::string{ArgumentSize}))
Ret.Arguments.at(I) = splice(View, S);
CONSUME_OR_NONE("</ARGUMENT>");
}
CONSUME_OR_NONE("</ARGUMENTS>");
}
{
CONSUME_OR_NONE("<ENVIRONMENT>");
{
CONSUME_OR_NONE("<DEFINE Count=\"");
EXTRACT_OR_NONE(EnvDefineCount, "\">");
std::size_t SetC = std::stoull(std::string{EnvDefineCount});
Ret.SetEnvironment.resize(SetC);
for (std::size_t I = 0; I < SetC; ++I)
{
CONSUME_OR_NONE("<VARVAL>");
CONSUME_OR_NONE("<VAR Size=\"");
EXTRACT_OR_NONE(VarSize, "\">");
if (std::size_t S = std::stoull(std::string{VarSize}))
Ret.SetEnvironment.at(I).first = splice(View, S);
CONSUME_OR_NONE("</VAR>");
CONSUME_OR_NONE("<VAL Size=\"");
EXTRACT_OR_NONE(ValSize, "\">");
if (std::size_t S = std::stoull(std::string{ValSize}))
Ret.SetEnvironment.at(I).second = splice(View, S);
CONSUME_OR_NONE("</VAL>");
CONSUME_OR_NONE("</VARVAL>");
}
CONSUME_OR_NONE("</DEFINE>");
}
{
CONSUME_OR_NONE("<UNSET Count=\"");
EXTRACT_OR_NONE(EnvUnsetCount, "\">");
std::size_t UnsetC = std::stoull(std::string{EnvUnsetCount});
Ret.UnsetEnvironment.resize(UnsetC);
for (std::size_t I = 0; I < UnsetC; ++I)
{
CONSUME_OR_NONE("<VAR Size=\"");
EXTRACT_OR_NONE(VarSize, "\">");
if (std::size_t S = std::stoull(std::string{VarSize}))
Ret.UnsetEnvironment.at(I) = splice(View, S);
CONSUME_OR_NONE("</VAR>");
}
CONSUME_OR_NONE("</UNSET>");
}
CONSUME_OR_NONE("</ENVIRONMENT>");
}
BASE_FOOTER_OR_NONE("</PROCESS>");
return Ret;
}

While arguably this is the only part of the project that's actually tested, the logic behind the communication system follows a clear pattern, and there are not many data types that are generally transmitted (things are either strings, numbers, tuples, or an array of these...).

This means that the communication layer, the full serialisation logic, and perhaps even the automated tests for it (!) could be generated automatically from a sufficiently concise DSL. We should steer clear of heavyweight communication protocol libraries (such as Apache Thrift) or DSL parsers that depend on compiler-grade tooling (such as ODB implemented as a custom GCC frontend action) as such tools might make our compilation process too complex.

While this could be implemented as a fully backwards-compatible change, it might be worthwhile to do this together with (or as a precondition of) #17 and several other planned communication layer changes (e.g., #14). We might realise during making this that a different format is more suitable for comfortable automatically generated serialisation, and it would be great only to break ABI once if we can afford it.

@whisperity whisperity added Kind: Enhancement 🔮 New feature or request Target: Core 💥 Core architectural support library Target: Communication protocol 💬⚠️ A part of the core library which deals with client-server messaging labels Jan 11, 2023
@whisperity whisperity self-assigned this Oct 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Kind: Enhancement 🔮 New feature or request Target: Communication protocol 💬⚠️ A part of the core library which deals with client-server messaging Target: Core 💥 Core architectural support library
Projects
None yet
Development

No branches or pull requests

1 participant