Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure ImportJobTest is not flaky by checking WriteToStore metric and requesting adequate resources for testing #332

Merged

Commits on Nov 27, 2019

  1. Print Redis INFO when key not found in ImportJobTest

    Also print random Redis element to debug that some FeatureRow has been ingested properly
    davidheryanto committed Nov 27, 2019
    Configuration menu
    Copy the full SHA
    b1bbbda View commit details
    Browse the repository at this point in the history
  2. Request CPU and memory for the pod running tests

    Some tests (like ingestion test) expect the operation to complete in certain amount of time. This can only be guaranteed if the process have adequate CPU and memory.
    Without it, when the test cluster is overloaded, the test process may get little CPU time allocated and the expected completion time is no longer valid
    davidheryanto committed Nov 27, 2019
    Configuration menu
    Copy the full SHA
    08ca71e View commit details
    Browse the repository at this point in the history

Commits on Nov 28, 2019

  1. Add metric WriteToStore:elements_written

    So we can obtain information about no of elements have been written in the pipeline without resorting to external metrics collector
    This method makes use built in metrics util in Apache Beam
    davidheryanto committed Nov 28, 2019
    Configuration menu
    Copy the full SHA
    36ac0f9 View commit details
    Browse the repository at this point in the history
  2. Add a check for all elements to be written to store before checking t…

    …he ingestion result in ImportJobTest
    davidheryanto committed Nov 28, 2019
    Configuration menu
    Copy the full SHA
    55fc69e View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    eb83163 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    d17a28d View commit details
    Browse the repository at this point in the history