Add support for an http streaming tracer (#148)

* Adds circular buffers for streaming. (#125) * Add chunk delimited circular buffer. * Add additional tests for MessageCircularBuffer. * s/MessageCircularBuffer/ChunkCircularBuffer/ * Document classes. * Document. * Fix build issues. * Fix clang-tidy error. * Fix clang-tidy error #2. * Fix clang-tidy #3. * Fix clang-tidy #4. * Fix magic numbers. * Add stub for http streaming recorder (#126) * Restructure source code. * Reorganize directory structure. * Run clang-format. * Reorging. * Add bazel build for grpc transporter. * Add top-level tracer libraries. * Add build for tests. * Reorging tests. * Port over recorder tests. * Don't expose internal header files. * Port dynamic loading test. * Add EventBase. * Add TimerEvent. * Support StreamRecorder as a tracer options. * Set up polling in StreamRecorder. * Fills out the StreamRecorder class. * Run clang-format. * Fix cmake build. * Convert benchmark to bazel. * Add benchmark build. * Add minimal cmake test. * Remove config.h * Port CI targets. * Fix ci build. * Fix memory leak. * Fix tsan target. * Fix typo. * Add exception handling to StreamRecorder. * Fix typo. * Add clang-tidy to CI. * Fix clang-tidy build. * Fix test case. * Adds an asynchronous dns resolver (#127) * Add code for ip addresses. * Add interface for dns resolution. * Adding in ares resolver. * Add Event class. * Add async dns resolution. * Add support for dns resolution without c-ares. * Add ChildProcessHandle. * Add a mock dns server. * Set up basic dns testing. * Add Socket. * Add addtional dns resolution tests. * Add more tests for dns resolver. * Add test coverage for no build with no dns resolution. * Fix naming. * Naming. * Fix tests with TSAN. * Add c-ares as cmake option. * Fix cmake file. * Fix do_ci.sh. * Add c-ares to plugin. * Add missing step to plugin build. * Fix clang-tidy errors. * Add commenting. * Add commenting. * Drop flaky test. * Fix lifetime issue with AresLibraryHandle. * Adds an endpoint manager to assign round-robin satellite connections (#128) * Change resolver construction function interface. * Set up classes for managing endpoints. * Separate out random code. * Add code to generate random durations. * Fill out code for endpoint manager. * Fix typo. * Fix assertion. * Fix memory issue in test. * Add ready callback to ResolutionManager. * Add test for dns resolution refreshes. * Add test coverage for retry. * Add test coverage for SatelliteEndpointManager. * Add sources to cmake. * Fix clang-tidy warnings. * Add satellite endpoints option to json. * Fix schema typos. * Add SatelliteStreamer (#129) * Add default dns resolution timeout. * Ignore case when comparing hosts. * Set up satellite connection classes. * Fill out SatelliteConnectionManager. * Add Noncopyable class. * s/SatelliteConnectionManager/SatelliteStreamer/ * Fill in SatelliteConnection. * Fix noexcept specifier. * Fill out parts of mock satellite. * Work on SatelliteStreamer. * Fix CI errors. * Add documentation. * Make connections as non-writable when they're closed * Add a mock satellite (#130) * Set up protobuf for go. * Add a mock satellite request handler. * Add reportprocessor. * Fill out mock satellite. * Add HttpConnection. * Add test case for mock satellite. * Add a handler for streaming report requests. * Don't reuse ReportRequest. * Add streaming test. * Check for chunked requests. * Don't modify request after posting to connection. * Fix format. * Fix clang-tidy. * Avoid unnecessary allocations. * Add network code for vectored IO (#131) * Add types for working with fragments. * Add coverage for FragmentInputStream. * Add test coverage for FragmentSet. * Add test coverage for vector-io. * Readability change. * Add additional coverage for Write. * Update CMake build files. * Update CMake build files. * Make FragmentInputStream final. * Fix clang-tidy error. * Add FunctionRef. * Use FunctionRef with ChunkCircularBuffer. * s/Serializer2/Serializer/ * Avoid implicit bool. * Use FunctionRef in FragmentSet. * Add documentation. * Add documentation. * Fix typo. * Remove unused code. * Fix typo. * Add documentation. * Assert that writev limit isn't exceeded. * Fix clang-tidy error. * Starts satellite streaming sessions. (#133) * Add library function for serializing embedded messages. * Generate the common stream header segment. * Add class for serializing metrics in stream header. * s/segment/fragment/ * Add test coverage for WriteStreamHeaderCommonFragment. * Add metrics to stream recorder. * Use FunctionRef in dns resolution. * Add classes for connection streaming. * Fill out code for ConnectionStream. * Fix assignment in FragmentInputStream. * s/FragmentInputStream/FixedFragmentInputStream/ * s/FragmentSet/FragmentInputStream/ * Change FragmentInputStream interface. * Add test coverage for Consume function. * Add HostHeader class. * Return the host name when requesting an endpoint. * Add FragmentArrayInputStream. * Fill out request part of stream header. * Integrate streaming classes. * Add test coverage for ConnectionStream. * Remove unused code. * Add test coverage for ConnectionStream. * Minor refactoring. * Minor refactoring. * Refactor connection code. * Add code for gracefully closing the satellite stream. * Fix post request. * Add test coverage for metrics. * Add files to cmake build. * Add missing files. * Fix clang-tidy error. * Fix clang-tidy errors. * Use preincrement. * Add documentation. * Remove unused include. * Remove magic numbers. * Stream spans (#134) * Fill out parts of SpanStream. * Add FindChunk method. * Fill out SpanStream functionality. * Add tracking from span remnants. * Allot and Consume bytes from the span stream. * Fill in SpanStream::Consume. * Add test coverage for SpanStream. * Put in fixes to SpanStream. * Add test coverage for ConnectionStream. * Add RandomSequencer. * Send spans to satellite. * Fix coverage. * Add missing cmake files. * Remove unnecessary capture. * Avoid recomputing message size. * Factor out code for binary number simulations. * Working on ConnectionStream simulation. * Get number simulation working with a single connection. * Finish ConnectionStream simulation. * Change number simulation functions to be static. * Check that buffer is empty. * Remove unused header. * tee clang-tidy output. * Document and test new circular buffer functions. * Increase clang-tidy timeout. * Document RandomTraverser. * Document GetRandomNumberGenerator. * Document ready method. * Document new ConnectionStream methods. * Fix ConnectionStream simulation. * Document FragmentArrayInputStream. * Document FragmentSpanInputStream. * Document Contains. * Document methods in SpanStream. * Document new test functions. * Document ZeroCopyConnectionInputStream. * Implement FlushWithTimeout (#136) * Work on supporing flush function. * Get rid of forwarding functions. * Use 64-bit number for head and tail in circular buffer. * Add support for flushing. * Fix flushing. * Add additional test coverage. * Fix typo. * Fix test. * Add example that streams to public satellites (#138) * Guard against self-move. * Parse the status code from satellite responses. * Make requests compatible. * Validate options. * Add stream example. * Add stream example. * Add missing cmake file. * Fix clang-tidy errors. * Fix clang-tidy issues. * Use a single ares channel per dns resolution. (#140) * Use a single ares channel per dns resolution. * Remove extra include. * Add missing release job. (#144) * Valid the streaming recorder's config. (#145) * Use smaller flush timeout (#142) * Use smaller flush timeout. * Move flush timeout to constant. * Support MetricsObserver in StreamRecorder. (#146) * Track chunk counts. * Move StreamRecorderMetrics to class. * Make metrics available to SpanStream. * Track spans sent. * Record spans sent. * Record flushes. * Change MetricsObserver to be noexcept. * Document StreamRecorderMetrics. * Fix sent spans count. * Add missing span sent observation. * Fix clang-tidy error. (#147)
lightstep · Apr 30, 2019 · 0916ee8 · 0916ee8
1 parent 8ccbd4d
commit 0916ee8
Show file tree

Hide file tree

Showing 244 changed files with 11,778 additions and 590 deletions.
diff --git a/.bazelrc b/.bazelrc
@@ -0,0 +1,2 @@
+build --incompatible_package_name_is_a_function=false
+query --incompatible_package_name_is_a_function=false
diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -1,105 +1,108 @@
 version: 2
 jobs:
-  asan:
+  cmake_no_grpc:
     docker:
       - image: ubuntu:18.04
     steps:
       - checkout
       - run: ./ci/setup_build_environment.sh
-      - run: ./ci/install_grpc.sh
       - run: ./ci/install_opentracing.sh
-      - run: ./ci/do_ci.sh cmake.asan
+      - run: ./ci/install_protobuf.sh
+      - run: ./ci/do_ci.sh cmake.minimal
       - store_artifacts:
           path: /build/Testing/Temporary/LastTest.log
           destination: Test.log
-  tsan:
+
+  cmake_with_grpc:
     docker:
       - image: ubuntu:18.04
     steps:
       - checkout
       - run: ./ci/setup_build_environment.sh
       - run: ./ci/install_grpc.sh
+      - run: ./ci/install_libevent.sh
+      - run: ./ci/install_cares.sh
       - run: ./ci/install_opentracing.sh
-      - run: ./ci/do_ci.sh cmake.tsan
+      - run: ./ci/do_ci.sh cmake.full
       - store_artifacts:
           path: /build/Testing/Temporary/LastTest.log
           destination: Test.log
-  clang_tidy:
+  asan:
     docker:
       - image: ubuntu:18.04
     steps:
       - checkout
       - run: ./ci/setup_build_environment.sh
-      - run: ./ci/install_grpc.sh
-      - run: ./ci/install_opentracing.sh
-      - run: ./ci/install_clang.sh
-      - run: ./ci/do_ci.sh cmake.clang-tidy
-      - store_artifacts:
-          path: /build/Testing/Temporary/LastTest.log
-          destination: Test.log
-  build_with_no_grpc:
+      - run: ./ci/install_bazel.sh
+      - run: ./ci/do_ci.sh bazel.asan
+
+  tsan:
     docker:
       - image: ubuntu:18.04
     steps:
       - checkout
       - run: ./ci/setup_build_environment.sh
-      - run: ./ci/install_opentracing.sh
-      - run: ./ci/install_protobuf.sh
-      - run: ./ci/do_ci.sh cmake.no_grpc
-      - store_artifacts:
-          path: /build/Testing/Temporary/LastTest.log
-          destination: Test.log
-  build_plugin:
+      - run: ./ci/install_bazel.sh
+      - run: ./ci/do_ci.sh bazel.tsan
+
+  release:
     docker:
       - image: ubuntu:18.04
     steps:
+      - run: apt-get -qq update; apt-get -y install git ssh
       - checkout
       - run: ./ci/setup_build_environment.sh
-      - run: ./ci/do_ci.sh plugin
-      - store_artifacts:
-          path: /liblightstep_tracer_plugin.so
-          destination: liblightstep_tracer_plugin.so
-  build_bazel:
+      - run: ./ci/do_ci.sh release
+
+  benchmark:
     docker:
       - image: ubuntu:18.04
-    resource_class: large
     steps:
       - checkout
       - run: ./ci/setup_build_environment.sh
       - run: ./ci/install_bazel.sh
-      - run: ./ci/do_ci.sh bazel.build
-  release:
+      - run: ./ci/do_ci.sh bazel.benchmark
+      - store_artifacts:
+          path: /benchmark
+          desination: benchmark
+
+  clang_tidy:
     docker:
       - image: ubuntu:18.04
     steps:
-      - run: apt-get -qq update; apt-get -y install git ssh
       - checkout
       - run: ./ci/setup_build_environment.sh
-      - run: ./ci/do_ci.sh release
+      - run: ./ci/install_clang.sh
+      - run: ./ci/install_bazel.sh
+      - run: 
+          command: ./ci/do_ci.sh clang_tidy
+          no_output_timeout: 1800
+      - store_artifacts:
+          path: /clang-tidy-result.txt
+          destination: /clang-tidy-result.txt
+      - store_artifacts:
+          path: /clang-tidy-errors.txt
+          destination: /clang-tidy-errors.txt
 
-  benchmark:
+  build_plugin:
     docker:
       - image: ubuntu:18.04
     steps:
       - checkout
       - run: ./ci/setup_build_environment.sh
-      - run: ./ci/install_grpc.sh
-      - run: ./ci/install_benchmark.sh
-      - run: ./ci/install_opentracing.sh
-      - run: ./ci/do_ci.sh benchmark
+      - run: ./ci/do_ci.sh plugin
       - store_artifacts:
-          path: /benchmark-results
-          desination: benchmark-results
+          path: /liblightstep_tracer_plugin.so
+          destination: liblightstep_tracer_plugin.so
 
   coverage:
     docker:
       - image: ubuntu:18.04
     steps:
       - checkout
       - run: ./ci/setup_build_environment.sh
-      - run: ./ci/install_grpc.sh
-      - run: ./ci/install_opentracing.sh
-      - run: ./ci/do_ci.sh coverage
+      - run: ./ci/install_bazel.sh
+      - run: ./ci/do_ci.sh bazel.coverage
       - store_artifacts:
           path: /coverage
           desination: coverage
@@ -114,11 +117,11 @@ workflows:
               ignore: /.*/
             tags:
               only: /^v[0-9]+(\.[0-9]+)*$/
+      - cmake_no_grpc
+      - cmake_with_grpc
       - asan
       - tsan
-      - clang_tidy
-      - build_with_no_grpc
-      - build_plugin
-      - build_bazel
-      - coverage
       - benchmark
+      - coverage
+      - build_plugin
+      - clang_tidy
diff --git a/.clang-tidy b/.clang-tidy
@@ -0,0 +1,2 @@
+Checks: '*'
+WarningsAsErrors: '*'
diff --git a/.gitignore b/.gitignore
@@ -43,3 +43,6 @@ linux
 # Ignore all bazel-* symlinks. There is no full list since this can change
 # based on the name of the directory bazel is cloned into.bazel-*
 /bazel-*
+
+# compilation database
+compile_commands.json
diff --git a/3rd_party/base64/BUILD b/3rd_party/base64/BUILD
@@ -0,0 +1,18 @@
+load(
+    "//bazel:lightstep_build_system.bzl",
+    "lightstep_cc_library",
+    "lightstep_package",
+)
+
+lightstep_package()
+
+lightstep_cc_library(
+    name = "base64_lib",
+    private_hdrs = [
+        "include/lightstep/base64/base64.h",
+    ],
+    srcs = [
+        "src/base64.cpp",
+    ],
+    is_3rd_party = True,
+)
diff --git a/3rd_party/catch2/BUILD b/3rd_party/catch2/BUILD
@@ -0,0 +1,27 @@
+load(
+    "//bazel:lightstep_build_system.bzl",
+    "lightstep_cc_library",
+    "lightstep_package",
+)
+
+lightstep_package()
+
+# From commit a575536abe20b58c48863600f8a71e93b4052b81
+lightstep_cc_library(
+    name = "catch2",
+    hdrs = [
+        "catch.hpp",
+    ],
+    is_3rd_party = True,
+)
+
+lightstep_cc_library(
+    name = "main_lib",
+    srcs = [
+        "catch_main.cpp",
+    ],
+    deps = [
+        ":catch2",
+    ],
+    is_3rd_party = True,
+)
diff --git a/...catch2/include/lightstep/catch2/catch.hpp → 3rd_party/catch2/catch.hpp b/...catch2/include/lightstep/catch2/catch.hpp → 3rd_party/catch2/catch.hpp
diff --git a/3rd_party/catch2/catch_main.cpp b/3rd_party/catch2/catch_main.cpp
@@ -0,0 +1,31 @@
+#define CATCH_CONFIG_RUNNER
+#include "catch.hpp"
+#include <stdio.h>
+#include <execinfo.h>
+#include <signal.h>
+#include <stdlib.h>
+#include <unistd.h>
+
+
+// Taken from https://stackoverflow.com/a/77336/4447365
+void handler(int sig) {
+  void *array[100];
+  size_t size;
+
+  // get void*'s for all entries on the stack
+  size = backtrace(array, 100);
+
+  // print out all the frames to stderr
+  fprintf(stderr, "Error: signal %d:\n", sig);
+  backtrace_symbols_fd(array, size, STDERR_FILENO);
+  exit(1);
+}
+
+int main( int argc, char* argv[] ) {
+  signal(SIGSEGV, handler);
+  signal(SIGTERM, handler);
+
+  int result = Catch::Session().run( argc, argv );
+
+  return result;
+}
diff --git a/3rd_party/randutils/BUILD b/3rd_party/randutils/BUILD
@@ -0,0 +1,15 @@
+load(
+    "//bazel:lightstep_build_system.bzl",
+    "lightstep_cc_library",
+    "lightstep_package",
+)
+
+lightstep_package()
+
+lightstep_cc_library(
+    name = "randutils_lib",
+    private_hdrs = [
+        "include/lightstep/randutils.h",
+    ],
+    is_3rd_party = True,
+)
diff --git a/3rd_party/randutils/include/lightstep/randutils.h b/3rd_party/randutils/include/lightstep/randutils.h
@@ -455,7 +455,7 @@ class auto_seeded : public SeedSeq {
     {
         // This is a constant that changes every time we compile the code
         constexpr uint32_t compile_stamp =
-            fnv(2166136261U, __DATE__ __TIME__ __FILE__);
+            fnv(2166136261U, "LKJLDSF23414lkjcas;liu23");
 
         // Some people think you shouldn't use the random device much because
         // on some platforms it could be expensive to call or "use up" vital

diff --git a/BUILD.bazel b/BUILD.bazel
@@ -1,27 +1,11 @@
-cc_library(
-    name = "lightstep_tracer",
-    srcs = glob([
-        "src/*.cpp",
-        "src/*.h",
-    ]),
-    hdrs = glob(["include/lightstep/*.h"]) + [
-        ":include/lightstep/config.h",
-        ":include/lightstep/version.h",
-    ],
-    strip_include_prefix = "include",
-    visibility = ["//visibility:public"],
-    deps = [
-        "//lightstep-tracer-common:collector_proto_cc",
-        "//lightstep-tracer-configuration:tracer_configuration_proto_cc",
-        "//lightstep-tracer-common:lightstep_carrier_proto_cc",
-        ":3rd_party_base64",
-        ":3rd_party_randutils",
-        ":3rd_party_catch",
-        # https://github.com/opentracing/opentracing-cpp
-        "@io_opentracing_cpp//:opentracing",
-    ],
+load(
+    "//bazel:lightstep_build_system.bzl",
+    "lightstep_package",
+    "lightstep_cc_library",
 )
 
+lightstep_package()
+
 genrule(
     name = "generate_version_h",
     srcs = glob([
@@ -32,41 +16,53 @@ genrule(
         "3rd_party/**/*",
     ]),
     outs = [
-        "include/lightstep/config.h",
-        "include/lightstep/version.h",
+        "gen-config/lightstep/version.h",
     ],
     cmd = """
     TEMP_DIR=$$(mktemp -d)
-    CONFIG_H_OUT=$${PWD}/$(location :include/lightstep/config.h)
-    VERSION_H_OUT=$${PWD}/$(location :include/lightstep/version.h)
+    VERSION_H_OUT=$${PWD}/$(location :gen-config/lightstep/version.h)
     LIGHTSTEP_ROOT=$$(dirname $${PWD}/$(location :CMakeLists.txt))
     cd $$TEMP_DIR
     cmake \\
         -DWITH_GRPC=OFF \\
         -DHEADERS_ONLY=ON \\
         -L \\
         $$LIGHTSTEP_ROOT
-    mv include/lightstep/config.h $$CONFIG_H_OUT
     mv include/lightstep/version.h $$VERSION_H_OUT
     rm -rf $$TEMP_DIR
     """,
 )
 
-cc_library(
-    name = "3rd_party_base64",
-    srcs = glob(["3rd_party/base64/src/*.cpp"]),
-    hdrs = glob(["3rd_party/base64/include/**/*.h"]),
-    strip_include_prefix = "3rd_party/base64/include",
+lightstep_cc_library(
+  name = "config_lib",
+  hdrs = [
+    ":gen-config/lightstep/version.h",
+  ],
+  strip_include_prefix = "gen-config",
 )
 
-cc_library(
-    name = "3rd_party_catch",
-    hdrs = glob(["3rd_party/catch/include/**/*.hpp"]),
-    strip_include_prefix = "3rd_party/catch/include",
+lightstep_cc_library(
+  name = "manual_tracer_lib",
+  deps = [
+    "//src/tracer:tracer_lib",
+    "//src/tracer:binary_carrier_lib",
+    "//src/recorder:transporter_lib",
+    "//src/tracer:no_default_ssl_roots_pem_lib",
+    "//src/recorder:no_grpc_transporter_lib",
+    "//src/recorder:no_stream_recorder_lib",
+  ],
 )
 
-cc_library(
-    name = "3rd_party_randutils",
-    hdrs = glob(["3rd_party/randutils/include/**/*.h"]),
-    strip_include_prefix = "3rd_party/randutils/include",
+lightstep_cc_library(
+  name = "tracer_lib",  
+  deps = [
+    "//src/tracer:tracer_lib",
+    "//src/tracer:binary_carrier_lib",
+    "//src/tracer:dynamic_load_lib",
+    "//src/tracer:no_default_ssl_roots_pem_lib",
+    "//src/recorder:transporter_lib",
+    "//src/recorder/grpc_transporter:grpc_transporter_lib",
+    "//src/recorder/stream_recorder:stream_recorder_lib",
+    "//src/network/ares_dns_resolver:ares_dns_resolver_lib",
+  ],
 )