Port Python tests of log input to Golang with filestream input

[kvch]

Why?

The goal of this issue is to move all Python tests related to the log input to Golang and use the new filestream input. These existing tests cover valuable knowledge about edge cases of the log input. To make sure filestream input is at least as stable as the log input, we need to run the same tests for the new input.

How?

All end-to-end tests are written in Python for the log input. However, some of these test does not require starting Filebeat, asserting that specific log lines are present, etc. It is much faster to run the input itself from Golang and assert that events have arrived and offsets are updated in the registry. Furthermore, these test no longer depend on log messages so changing logging will not break existing tests.

Note: It might turn out that some of the test below are better suited for Python E2E tests. In this case, we are going to run the same test for both input types with the help of parameterized.

What?

• filebeat/tests/system/test_harvester.py

  • test_bom_utf8 Finish porting all tests from Python to Golang from test_harvester.py #24397
    - [ ] test_boms_0_utf_8 duplicate of prev test
  • test_boms_1_utf_16be_bom Finish porting all tests from Python to Golang from test_harvester.py #24397
  • test_boms_2_utf_16le_bom Finish porting all tests from Python to Golang from test_harvester.py #24397
  • test_close_eof Port two Harvester tests of log input to filestream in Golang #24190
  • test_close_removed Port four Harvester tests of log input to filestream in Golang #24250
  • test_close_renamed Port two Harvester tests of log input to filestream in Golang #24190
  • test_close_timeout Finish porting all tests from Python to Golang from test_harvester.py #24397
    - [ ] test_debug_reader asserts debug log contents
    - [ ] test_decode_error assert error log
  • test_empty_line Port four Harvester tests of log input to filestream in Golang #24250
  • test_empty_lines_only Port four Harvester tests of log input to filestream in Golang #24250
  • test_exceed_buffer Port four Harvester tests of log input to filestream in Golang #24250
  • test_ignore_symlink
  • test_symlink_and_file Finish porting all tests from Python to Golang from test_harvester.py #24397
  • test_symlink_removed Finish porting all tests from Python to Golang from test_harvester.py #24397
  • test_symlink_rotated Finish porting all tests from Python to Golang from test_harvester.py #24397
  • test_symlinks_enabled Finish porting all tests from Python to Golang from test_harvester.py #24397
  • test_truncate Finish porting all tests from Python to Golang from test_harvester.py #24397
  • test_truncated_file_closed Finish porting all tests from Python to Golang from test_harvester.py #24397
  • test_truncated_file_open Finish porting all tests from Python to Golang from test_harvester.py #24397

• filebeat/tests/system/test_input.py

  • test_close_inactive Port close_inactive* tests from test_input.py #24706
  • test_close_inactive_file_removal Port close_inactive* tests from test_input.py #24706
  • test_close_inactive_file_rotation_and_removal Port close_inactive* tests from test_input.py #24706
  • test_close_inactive_file_rotation_and_removal2 Port close_inactive* tests from test_input.py #24706
  • test_disable_recursive_glob
  • test_exclude_files: fswatch_test.go:TestFileScanner/skip_excluded_files
  • test_files_added_late
  • test_harvester_limit Add harvester_limit to filestream #21631
  • test_ignore_older_files: prospector_test.go:TestProspectorNewAndUpdatedFiles/old_files_with_ignore_older_configured
  • test_inode_marker_based_identity_tracking
  • test_inode_marker_based_identity_tracking_to_path_based
  • test_no_paths_defined: config_test.go:TestConfigValidate/paths_cannot_be_empty
  • test_not_ignore_old_files
  • test_path_based_identity_tracking
  • test_restart_recursive_glob
  • test_rotating_close_inactive_larger_write_rate Port test_rotating_close_inactive_larger_write_rate to Golang #32039
  • test_rotating_close_inactive_low_write_rate Port test_rotating_close_inactive_low_write_rate to Golang #32045
    ~~ - [ ] test_shutdown_no_inputs~~ not relevant to filestream
  • test_skip_symlinks: fswatch_test_non_windows.go:TestFileScannerSymlinks/skip_symlinks

• filebeat/tests/system/test_registrar.py

  • test_clean_inactive
  • test_clean_removed
  • test_clean_removed_with_clean_inactive
  • test_ignore_older_state
  • test_ignore_older_state_clean_inactive

• filebeat/tests/system/test_json.py
  - [ ] test_config_no_msg_key_filtering specific to log input
  - [ ] test_config_no_msg_key_multiline specific to log input

  • test_docker_logs Add support for parsers in filestream input #24763
  • test_docker_logs_filtering Add support for parsers in filestream input #24763
  • test_id_in_message
    - [ ] test_integer_condition must be E2E test, it involves drop_event processor
    - [ ] test_json_add_tags must stay Python as tags is in pipeline
    - [ ] test_json_decoding_error_false must keep E2E, the test checks if there is no error logged in the log file of the Beat
    - [ ] test_json_decoding_error_true must keep E2E, the test checks if there is an error logged in the log file of the Beat
  • test_simple_json_overwrite Add support for parsers in filestream input #24763
  • test_timestamp_in_message Add support for parsers in filestream input #24763
  • test_type_in_message Port test_type_in_message to filestream Golang tests #32032

• filebeat/tests/system/test_multiline.py

  • test_c_style_log Add support for parsers in filestream input #24763
  • test_close_timeout_with_multiline Add support for parsers in filestream input #24763
  • test_consecutive_newline Add support for parsers in filestream input #24763
  • test_invalid_config Add tests to cover test_invalid_config from test_multiline.py #24644
  • test_java_elasticsearch_log Add support for parsers in filestream input #24763
  • test_max_bytes Add support for parsers in filestream input #24763
  • test_max_lines Add support for parsers in filestream input #24763
  • test_rabbitmq_multiline_log Add support for parsers in filestream input #24763
  • test_timeout Add support for parsers in filestream input #24763

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[kvch]

The tests that are already running in Golang using the filestream input can be deleted from the Python test suite. We no longer support the log input, so there is no need to run duplicate tests for it.

[kvch]

Significant tests that are missing from filestream:

• test_rotating_close_inactive_larger_write_rate Port test_rotating_close_inactive_larger_write_rate to Golang #32039
• test_rotating_close_inactive_low_write_rate Port test_rotating_close_inactive_low_write_rate to Golang #32045

The tests from test_registry.py should not be migrated because most of the are low-value and specific to the log input. Some of the tests like test_clean_inactive or test_clean_removed is tested from the package input-logfile. But as the registry behaves differently in filestream, it does not really make sense to compare the two tests.

Remaining tests from test_json.py are also valuable as an INTEGRATION test, not as E2E tests. JSON tests cover the common code paths used by both log and filestream inputs.

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!