Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add stub for http streaming recorder #126

Merged
merged 32 commits into from
Jan 27, 2019
Merged

Add stub for http streaming recorder #126

merged 32 commits into from
Jan 27, 2019

Conversation

rnburn
Copy link
Contributor

@rnburn rnburn commented Jan 23, 2019 •
edited
Loading

  • Adds a new StreamRecorder that serializes spans into the lock-free circular buffer. It has basic code to set up timers, but is missing the network code to send spans (to be added in a later PR) so for now it only consumes and throws away spans from the circular buffer.
  • Adds libevent as an option dependency (but will be required for the streaming recorder).
  • Reorganizes the build so that the testing code is now run under bazel. (This will make it easier to write more complicated tests with mock satellites for when the streaming network code is added.)

@rnburn
Restructure source code.
9ef8411
@rnburn
Reorganize directory structure.
eb15f2a
@rnburn
Run clang-format.
3371921
@rnburn
Reorging.
407831d
@rnburn
Add bazel build for grpc transporter.
17058d5
@rnburn
Add top-level tracer libraries.
0b9165d
@rnburn
Add build for tests.
59b467a
@rnburn
Reorging tests.
5529658
@rnburn
Port over recorder tests.
fd1ed5b
@rnburn
Don't expose internal header files.
87e4326
@rnburn
Port dynamic loading test.
b2cff31
@rnburn
Add EventBase.
5a17ca7
@rnburn
Add TimerEvent.
6a833d9
@rnburn
Support StreamRecorder as a tracer options.
71d6e88
@rnburn
Set up polling in StreamRecorder.
9ea7f50
@rnburn
Fills out the StreamRecorder class.
ce3b01a
@rnburn
Run clang-format.
65d08a3
@rnburn
Fix cmake build.
b21803a
@rnburn
Convert benchmark to bazel.
6b25dfe
@rnburn
Add benchmark build.
a5fb5ab
@rnburn
Add minimal cmake test.
893ebac
@rnburn
Remove config.h
294ff3f
@rnburn
Port CI targets.
909b72e
@rnburn
Fix ci build.
dc5302b
@rnburn
Fix memory leak.
4fbb6c4
@rnburn
Fix tsan target.
89ff795
@rnburn
Fix typo.
2093480
@rnburn
Add exception handling to StreamRecorder.
f6e8525
@rnburn rnburn requested a review from jmacd January 24, 2019 18:10
@rnburn rnburn merged commit d08552f into lightstep:http-streaming Jan 27, 2019
rnburn added a commit that referenced this pull request Apr 30, 2019
@rnburn
Add support for an http streaming tracer (#148)
0916ee8 
* Adds circular buffers for streaming. (#125)

* Add chunk delimited circular buffer.

* Add additional tests for MessageCircularBuffer.

* s/MessageCircularBuffer/ChunkCircularBuffer/

* Document classes.

* Document.

* Fix build issues.

* Fix clang-tidy error.

* Fix clang-tidy error #2.

* Fix clang-tidy #3.

* Fix clang-tidy #4.

* Fix magic numbers.

* Add stub for http streaming recorder (#126)

* Restructure source code.

* Reorganize directory structure.

* Run clang-format.

* Reorging.

* Add bazel build for grpc transporter.

* Add top-level tracer libraries.

* Add build for tests.

* Reorging tests.

* Port over recorder tests.

* Don't expose internal header files.

* Port dynamic loading test.

* Add EventBase.

* Add TimerEvent.

* Support StreamRecorder as a tracer options.

* Set up polling in StreamRecorder.

* Fills out the StreamRecorder class.

* Run clang-format.

* Fix cmake build.

* Convert benchmark to bazel.

* Add benchmark build.

* Add minimal cmake test.

* Remove config.h

* Port CI  targets.

* Fix ci build.

* Fix memory leak.

* Fix tsan target.

* Fix typo.

* Add exception handling to StreamRecorder.

* Fix typo.

* Add clang-tidy to CI.

* Fix clang-tidy build.

* Fix test case.

* Adds an asynchronous dns resolver (#127)

* Add code for ip addresses.

* Add interface for dns resolution.

* Adding in ares resolver.

* Add Event class.

* Add async dns resolution.

* Add support for dns resolution without c-ares.

* Add ChildProcessHandle.

* Add a mock dns server.

* Set up basic dns testing.

* Add Socket.

* Add addtional dns resolution tests.

* Add more tests for dns resolver.

* Add test coverage for no build with no dns resolution.

* Fix naming.

* Naming.

* Fix tests with TSAN.

* Add c-ares as cmake option.

* Fix cmake file.

* Fix do_ci.sh.

* Add c-ares to plugin.

* Add missing step to plugin build.

* Fix clang-tidy errors.

* Add commenting.

* Add commenting.

* Drop flaky test.

* Fix lifetime issue with AresLibraryHandle.

* Adds an endpoint manager to assign round-robin satellite connections (#128)

* Change resolver construction function interface.

* Set up classes for managing endpoints.

* Separate out random code.

* Add code to generate random durations.

* Fill out code for endpoint manager.

* Fix typo.

* Fix assertion.

* Fix memory issue in test.

* Add ready callback to ResolutionManager.

* Add test for dns resolution refreshes.

* Add test coverage for retry.

* Add test coverage for SatelliteEndpointManager.

* Add sources to cmake.

* Fix clang-tidy warnings.

* Add satellite endpoints option to json.

* Fix schema typos.

* Add SatelliteStreamer (#129)

* Add default dns resolution timeout.

* Ignore case when comparing hosts.

* Set up satellite connection classes.

* Fill out SatelliteConnectionManager.

* Add Noncopyable class.

* s/SatelliteConnectionManager/SatelliteStreamer/

* Fill in SatelliteConnection.

* Fix noexcept specifier.

* Fill out parts of mock satellite.

* Work on SatelliteStreamer.

* Fix CI errors.

* Add documentation.

* Make connections as non-writable when they're closed

* Add a mock satellite (#130)

* Set up protobuf for go.

* Add a mock satellite request handler.

* Add reportprocessor.

* Fill out mock satellite.

* Add HttpConnection.

* Add test case for mock satellite.

* Add a handler for streaming report requests.

* Don't reuse ReportRequest.

* Add streaming test.

* Check for chunked requests.

* Don't modify request after posting to connection.

* Fix format.

* Fix clang-tidy.

* Avoid unnecessary allocations.

* Add network code for vectored IO (#131)

* Add types for working with fragments.

* Add coverage for FragmentInputStream.

* Add test coverage for FragmentSet.

* Add test coverage for vector-io.

* Readability change.

* Add additional coverage for Write.

* Update CMake build files.

* Update CMake build files.

* Make FragmentInputStream final.

* Fix clang-tidy error.

* Add FunctionRef.

* Use FunctionRef with ChunkCircularBuffer.

* s/Serializer2/Serializer/

* Avoid implicit bool.

* Use FunctionRef in FragmentSet.

* Add documentation.

* Add documentation.

* Fix typo.

* Remove unused code.

* Fix typo.

* Add documentation.

* Assert that writev limit isn't exceeded.

* Fix clang-tidy error.

* Starts satellite streaming sessions. (#133)

* Add library function for serializing embedded messages.

* Generate the common stream header segment.

* Add class for serializing metrics in stream header.

* s/segment/fragment/

* Add test coverage for WriteStreamHeaderCommonFragment.

* Add metrics to stream recorder.

* Use FunctionRef in dns resolution.

* Add classes for connection streaming.

* Fill out code for ConnectionStream.

* Fix assignment in FragmentInputStream.

* s/FragmentInputStream/FixedFragmentInputStream/

* s/FragmentSet/FragmentInputStream/

* Change FragmentInputStream interface.

* Add test coverage for Consume function.

* Add HostHeader class.

* Return the host name when requesting an endpoint.

* Add FragmentArrayInputStream.

* Fill out request part of stream header.

* Integrate streaming classes.

* Add test coverage for ConnectionStream.

* Remove unused code.

* Add test coverage for ConnectionStream.

* Minor refactoring.

* Minor refactoring.

* Refactor connection code.

* Add code for gracefully closing the satellite stream.

* Fix post request.

* Add test coverage for metrics.

* Add files to cmake build.

* Add missing files.

* Fix clang-tidy error.

* Fix clang-tidy errors.

* Use preincrement.

* Add documentation.

* Remove unused include.

* Remove magic numbers.

* Stream spans (#134)

* Fill out parts of SpanStream.

* Add FindChunk method.

* Fill out SpanStream functionality.

* Add tracking from span remnants.

* Allot and Consume bytes from the span stream.

* Fill in SpanStream::Consume.

* Add test coverage for SpanStream.

* Put in fixes to SpanStream.

* Add test coverage for ConnectionStream.

* Add RandomSequencer.

* Send spans to satellite.

* Fix coverage.

* Add missing cmake files.

* Remove unnecessary capture.

* Avoid recomputing message size.

* Factor out code for binary number simulations.

* Working on ConnectionStream simulation.

* Get number simulation working with a single connection.

* Finish ConnectionStream simulation.

* Change number simulation functions to be static.

* Check that buffer is empty.

* Remove unused header.

* tee clang-tidy output.

* Document and test new circular buffer functions.

* Increase clang-tidy timeout.

* Document RandomTraverser.

* Document GetRandomNumberGenerator.

* Document ready method.

* Document new ConnectionStream methods.

* Fix ConnectionStream simulation.

* Document FragmentArrayInputStream.

* Document FragmentSpanInputStream.

* Document Contains.

* Document methods in SpanStream.

* Document new test functions.

* Document ZeroCopyConnectionInputStream.

* Implement FlushWithTimeout (#136)

* Work on supporing flush function.

* Get rid of forwarding functions.

* Use 64-bit number for head and tail in circular buffer.

* Add support for flushing.

* Fix flushing.

* Add additional test coverage.

* Fix typo.

* Fix test.

* Add example that streams to public satellites (#138)

* Guard against self-move.

* Parse the status code from satellite responses.

* Make requests compatible.

* Validate options.

* Add stream example.

* Add stream example.

* Add missing cmake file.

* Fix clang-tidy errors.

* Fix clang-tidy issues.

* Use a single ares channel per dns resolution. (#140)

* Use a single ares channel per dns resolution.

* Remove extra include.

* Add missing release job. (#144)

* Valid the streaming recorder's config. (#145)

* Use smaller flush timeout (#142)

* Use smaller flush timeout.

* Move flush timeout to constant.

* Support MetricsObserver in StreamRecorder. (#146)

* Track chunk counts.

* Move StreamRecorderMetrics to class.

* Make metrics available to SpanStream.

* Track spans sent.

* Record spans sent.

* Record flushes.

* Change MetricsObserver to be noexcept.

* Document StreamRecorderMetrics.

* Fix sent spans count.

* Add missing span sent observation.

* Fix clang-tidy error. (#147)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jmacd jmacd jmacd approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@rnburn @jmacd