Use Rust Loader for running examples #993

ipetr0v · 2020-05-14T16:26:10Z

This change:

Updates examples to use Rust Loader
Updates macro and scripts for running Rust Loader
Makes Oak CI to use Rust Loader for running examples
Builds Rust Oak Loader with Cargo

Fixes #725
Fixes #874
Fixes #901
Ref #945

Checklist

Pull request affects core Oak functionality (e.g. runtime, SDK, ABI)
- I have checked that these tests are run by Cloudbuild
- I have updated documentation accordingly.
- I have raised an issue to
  cover any TODOs and/or unfinished work.

tiziano88

Thanks!

tiziano88 · 2020-05-15T20:57:40Z

examples/abitest/module_0/rust/src/lib.rs

            roughtime: oak::roughtime::Roughtime::new("roughtime-client"),
            misconfigured_roughtime: oak::roughtime::Roughtime::new("roughtime-misconfig"),
        }
    }
 }

 #[no_mangle]
-pub extern "C" fn frontend_oak_main(in_handle: u64) {
+pub extern "C" fn frontend_oak_main(_: u64) {


Please keep the name of the parameter, just add an underscore as prefix.

tiziano88 · 2020-05-15T20:57:54Z

Cargo.lock

 "tokio 0.2.20 (registry+https://github.com/rust-lang/crates.io-index)",
- "tonic 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)",
+ "tonic 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)",


Is this downgrade intentional?

There were different versions of tonic in Cargo.lock

tiziano88 · 2020-05-15T21:00:44Z

examples/abitest/client/abitest.cc

+  std::string private_key_path = absl::GetFlag(FLAGS_private_key);
+  std::string cert_chain_path = absl::GetFlag(FLAGS_cert_chain);
+  if (private_key_path.empty()) {
+    OAK_LOG(FATAL) << "No private key file specified.";
+  }
+  if (cert_chain_path.empty()) {
+    OAK_LOG(FATAL) << "No certificate chain file specified.";
+  }
+  std::string private_key = oak::utils::read_file(private_key_path);
+  std::string cert_chain = oak::utils::read_file(cert_chain_path);
+
+  grpc::SslServerCredentialsOptions::PemKeyCertPair key_cert_pair = {private_key, cert_chain};
+  grpc::SslServerCredentialsOptions options;
+  options.pem_root_certs = ca_cert;
+  options.pem_key_cert_pairs.push_back(key_cert_pair);
+  return grpc::SslServerCredentials(options);


I don't remember why this was not needed before and now is needed. Is it because the Rust gRPC server pseudo-node requires TLS? Is there no way to make it work with insecure credentials? I don't know if it's still blocking you, but if it is, I think it would make sense to have support for insecure to make the transition easier, and then add TLS later on. Apologies if we already talked about it, I don't remember the details.

Presumably this is more because the Rust gRPC client pseudo-Node now always uses TLS, so the test server that it connects to (which runs in parallel inside the abitest client program) now needs to act as TLS server side?

tiziano88 · 2020-05-15T21:03:52Z

examples/private_set_intersection/client/private_set_intersection.cc

  // Retrieve intersection.
  std::vector<std::string> intersection_0 = RetrieveIntersection(stub_0.get());
  LOG(INFO) << "client 0 intersection:";
  for (auto item : intersection_0) {
    LOG(INFO) << "- " << item;
  }
+  assert(std::set<std::string>(intersection_0.begin(), intersection_0.end()) == expected_set);


Nice addition!

Moved to #1006

examples/aggregator/backend/src/main.rs

tiziano88 · 2020-05-19T13:12:10Z

examples/aggregator/backend/src/main.rs

+    let private_key = tokio::fs::read(&opt.tls_private_key).await?;
+    let certificate = tokio::fs::read(&opt.tls_certificate).await?;


Wow, so much parallelism :) (well, actually they are serialized anyways)

tiziano88 · 2020-05-19T13:13:55Z

examples/aggregator/scripts/run_server


 readonly APPLICATION="${PWD}/bazel-client-bin/examples/aggregator/config/config.bin"
-exec ./bazel-bin/oak/server/loader/oak_runner \
+exec ./bazel-clang-bin/oak/server/rust/oak_loader/oak_loader \


Wait, why are we not using the cargo version yet?

That's probably my fault – I switched to a Bazel-build rust_binary target in 210911e (back when we were planning to have a Bazel-built Rust + C++ cross-linked binary). We could/should revert that commit.

oak/server/rust/oak_runtime/src/node/grpc/client.rs

sdk/rust/oak/src/lib.rs

examples/abitest/client/abitest.cc

daviddrysdale · 2020-05-19T15:03:19Z

examples/abitest/client/abitest.cc

+  grpc::SslServerCredentialsOptions::PemKeyCertPair key_cert_pair = {private_key, cert_chain};
+  grpc::SslServerCredentialsOptions options;
+  options.pem_root_certs = ca_cert;
+  options.pem_key_cert_pairs.push_back(key_cert_pair);


(Aside: if we weren't about to delete it, I'd suggest commonizing this code with the equivalent code in oak_runner_main.cc. But we are so I won't.)

daviddrysdale · 2020-05-19T15:04:28Z

examples/abitest/client/abitest.cc

+  std::string private_key_path = absl::GetFlag(FLAGS_private_key);
+  std::string cert_chain_path = absl::GetFlag(FLAGS_cert_chain);
+  if (private_key_path.empty()) {
+    OAK_LOG(FATAL) << "No private key file specified.";
+  }
+  if (cert_chain_path.empty()) {
+    OAK_LOG(FATAL) << "No certificate chain file specified.";
+  }
+  std::string private_key = oak::utils::read_file(private_key_path);
+  std::string cert_chain = oak::utils::read_file(cert_chain_path);
+
+  grpc::SslServerCredentialsOptions::PemKeyCertPair key_cert_pair = {private_key, cert_chain};
+  grpc::SslServerCredentialsOptions options;
+  options.pem_root_certs = ca_cert;
+  options.pem_key_cert_pairs.push_back(key_cert_pair);
+  return grpc::SslServerCredentials(options);


Presumably this is more because the Rust gRPC client pseudo-Node now always uses TLS, so the test server that it connects to (which runs in parallel inside the abitest client program) now needs to act as TLS server side?

scripts/run_server

scripts/run_example

oak/server/rust/oak_runtime/src/node/storage.rs

oak/server/rust/oak_runtime/src/node/roughtime.rs

oak/server/rust/oak_runtime/src/node/mod.rs

rbehjati · 2020-05-19T16:27:03Z

examples/aggregator/backend/src/main.rs

+
+    let identity = Identity::from_pem(certificate, private_key);
+
+    let address = "[::1]:8888".parse()?;


Is this the same as localhost:8888? If so, can we use localhost everywhere?

It's a IPv6 synonym for localhost.
Changed to [::]:8888 that is a synonym to 0.0.0.0.
Using IPv6 so we won't need to change it everywhere if we switch to IPv6 (since it's backwards compatible).

examples/aggregator/scripts/run_example

rbehjati

LGTM!

tiziano88 · 2020-05-20T15:45:35Z

examples/abitest/config/config.textproto

@@ -26,17 +26,27 @@ node_configs {
    address: "test.invalid:9999"
  }
 }
+node_configs {


This is going away soon anyways.

oak/server/rust/oak_runtime/src/node/grpc/client.rs

Add gRPC client Add gRPC client Update comments Move client to a new grpc directory Move codec to a separate file Format Update comments Format Refactor Change after review Update Node macro Run running_average Update scripts Update examples Run examples with the Rust Oak Loader Delete default from rustfmt Decode GrpcResponse Use Rust Loader in aggregator Remove io Remove additional rust configs Update scripts Reenable tests; Add Node stubs Fix gRPC client connection Work on abitest fix Temporary comment gRPC abitest Update Aggregator scripts Update Aggregator scripts Format Change after review Fix backend wasm Nodes Format Fix non-Ok gRPC responses Disable gRPC stop via notification Fix gRPC abitest Fix example tests Use cargo build Fix build_docker Change after review Fix server termination Fix running_average

github-actions · 2020-05-21T13:53:40Z

Reproducibility index:

ff845b6131a554ea766bb3b159add982bf8553c007957c9724c81bdf6af23a62  ./target/wasm32-unknown-unknown/release/abitest_0_frontend.wasm
aba0ac3f1e56af8e41986b1d6c319f5bb82bf0f3362a50b4dc450d873eeda05d  ./target/wasm32-unknown-unknown/release/abitest_1_backend.wasm
33a09345a64145b9e16125f6dffc2d0c1cb50ce0f6d6fe2a9c008594a4400260  ./target/wasm32-unknown-unknown/release/aggregator.wasm
ff041be9b2da857dd04638b13d166dd32dad9b7be3372f7188e2f75121e609cc  ./target/wasm32-unknown-unknown/release/chat.wasm
d7a6787e91eacdf5ea5eb0c02d23df23c27dc1227457d045efc8e62503a0d510  ./target/wasm32-unknown-unknown/release/hello_world.wasm
20e392c183714a9ce051094f57795d5b59dac68f00f4d6bba05af8c7c7611266  ./target/wasm32-unknown-unknown/release/machine_learning.wasm
3d6c00733a87db29d47e20e5fe6b165f13fc1ffee8a04d49f55fef475673e81b  ./target/wasm32-unknown-unknown/release/private_set_intersection.wasm
0ac156fa51234414442efcfd6f0f397a01e2f4a8a41384c2e08c9d303f284015  ./target/wasm32-unknown-unknown/release/running_average.wasm
29af930f5095d77a27942766925cedac30fd3d0bec5f2712f977e100cbfa9141  ./target/wasm32-unknown-unknown/release/translator.wasm

Changes required by commit 68fbbe3 ("Use Rust Loader for running examples (#993)"): - Add and use a config that doesn't include a translator node. - Add C++ code that is equivalent to oak::grpc::server::init(). Change required by commit 64d3f90 ("Use bytes to represent gRPC message body (#833)"): - Re-work hand-coded response message for current GrpcResponse proto message definition. For #1009

ipetr0v added the WIP Work in progress label May 14, 2020

googlebot added the cla: yes label May 14, 2020

ipetr0v force-pushed the use_rust_loader branch from 03486cb to 69e11fc Compare May 14, 2020 19:31

ipetr0v mentioned this pull request May 15, 2020

Rust Runtime features for running Oak applications #818

Closed

15 tasks

ipetr0v force-pushed the use_rust_loader branch from ee309b6 to 53d44ec Compare May 18, 2020 10:56

ipetr0v mentioned this pull request May 19, 2020

Fix gRPC client/server Node termination #1002

Closed

ipetr0v removed the WIP Work in progress label May 19, 2020

ipetr0v marked this pull request as ready for review May 19, 2020 12:57

ipetr0v requested review from tiziano88, daviddrysdale and rbehjati May 19, 2020 12:57

tiziano88 reviewed May 19, 2020

View reviewed changes

daviddrysdale reviewed May 19, 2020

View reviewed changes

rbehjati reviewed May 19, 2020

View reviewed changes

ipetr0v requested review from rbehjati and daviddrysdale May 20, 2020 15:15

rbehjati approved these changes May 20, 2020

View reviewed changes

tiziano88 approved these changes May 20, 2020

View reviewed changes

ipetr0v mentioned this pull request May 21, 2020

Run C++ examples with Rust Oak Loader #1009

Closed

ipetr0v force-pushed the use_rust_loader branch from 8ccefc6 to a1d12de Compare May 21, 2020 12:57

ipetr0v merged commit 68fbbe3 into project-oak:master May 21, 2020

ipetr0v deleted the use_rust_loader branch May 21, 2020 13:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use Rust Loader for running examples #993

Use Rust Loader for running examples #993

ipetr0v commented May 14, 2020 •

edited

Loading

tiziano88 left a comment

tiziano88 May 15, 2020

tiziano88 May 15, 2020

ipetr0v May 21, 2020

tiziano88 May 15, 2020

daviddrysdale May 19, 2020

tiziano88 May 15, 2020

ipetr0v May 19, 2020

tiziano88 May 19, 2020

tiziano88 May 19, 2020

daviddrysdale May 19, 2020

daviddrysdale May 19, 2020

daviddrysdale May 19, 2020

rbehjati May 19, 2020

ipetr0v May 19, 2020

rbehjati left a comment

tiziano88 May 20, 2020

github-actions bot commented May 21, 2020

		let private_key = tokio::fs::read(&opt.tls_private_key).await?;
		let certificate = tokio::fs::read(&opt.tls_certificate).await?;


		let identity = Identity::from_pem(certificate, private_key);

		let address = "[::1]:8888".parse()?;

Use Rust Loader for running examples #993

Use Rust Loader for running examples #993

Conversation

ipetr0v commented May 14, 2020 • edited Loading

Checklist

tiziano88 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rbehjati left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented May 21, 2020

ipetr0v commented May 14, 2020 •

edited

Loading