Error log in successful ingest

[PSeitz]

On a short ingest, there's also an error log, but overall the ingest was successful, so this is maybe a false error?

2022-05-13T08:54:39.529Z ERROR {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=1}::{index=hdfs gen=0}:{actor=Publisher}:{msg_id=2}: quickwit_indexing::actors::publisher: fail-send-suggest-truncate

Complete Logs

quickwit index create --index-config ./hdfs_index_config.yaml --config=quickwit.yaml
2022-05-13T08:54:06.847Z  INFO quickwit: version="0.2.1" commit="2b8e4c9"
2022-05-13T08:54:06.848Z  WARN quickwit_config::config: Cluster ID is not set, falling back to default value. cluster_id="quickwit-default-cluster"
2022-05-13T08:54:06.848Z  WARN quickwit_config::config: Seed list is empty.
2022-05-13T08:54:06.848Z  INFO quickwit_cli: Loaded Quickwit config. config_uri=file:///home/pascal/LinuxData/Development/quickwit-indices/quickwit.yaml config=QuickwitConfig { version: 0, node_id: "node-old-tq1c", cluster_id: "quickwit-default-cluster", listen_address: "127.0.0.1", rest_listen_port: 7280, gossip_listen_port: 7280, grpc_listen_port: 7281, peer_seeds: [], data_dir_path: "./qwdata", metastore_uri: Uri { uri: "file:///home/pascal/LinuxData/Development/quickwit-indices/qwdata/indexes", protocol_idx: 4 }, default_index_root_uri: Uri { uri: "file:///home/pascal/LinuxData/Development/quickwit-indices/qwdata/indexes", protocol_idx: 4 }, indexer_config: IndexerConfig { split_store_max_num_bytes: Byte(100000000000), split_store_max_num_splits: 1000 }, searcher_config: SearcherConfig { fast_field_cache_capacity: Byte(1000000000), split_footer_cache_capacity: Byte(500000000), max_num_concurrent_split_streams: 100 }, storage_config: None }
2022-05-13T08:54:06.850Z  INFO quickwit_core::index: Index config does not specify `index_uri`, falling back to default value. index_id=hdfs index_uri=file:///home/pascal/LinuxData/Development/quickwit-indices/qwdata/indexes/hdfs
Index `hdfs` successfully created.
➜  quickwit-indices cat ./hdfs_small.json | quickwit index ingest --index hdfs --config=quickwit.yaml
2022-05-13T08:54:39.482Z  INFO quickwit: version="0.2.1" commit="2b8e4c9"
2022-05-13T08:54:39.483Z  WARN quickwit_config::config: Cluster ID is not set, falling back to default value. cluster_id="quickwit-default-cluster"
2022-05-13T08:54:39.483Z  WARN quickwit_config::config: Seed list is empty.
2022-05-13T08:54:39.483Z  INFO quickwit_cli: Loaded Quickwit config. config_uri=file:///home/pascal/LinuxData/Development/quickwit-indices/quickwit.yaml config=QuickwitConfig { version: 0, node_id: "node-winter-Co4k", cluster_id: "quickwit-default-cluster", listen_address: "127.0.0.1", rest_listen_port: 7280, gossip_listen_port: 7280, grpc_listen_port: 7281, peer_seeds: [], data_dir_path: "./qwdata", metastore_uri: Uri { uri: "file:///home/pascal/LinuxData/Development/quickwit-indices/qwdata/indexes", protocol_idx: 4 }, default_index_root_uri: Uri { uri: "file:///home/pascal/LinuxData/Development/quickwit-indices/qwdata/indexes", protocol_idx: 4 }, indexer_config: IndexerConfig { split_store_max_num_bytes: Byte(100000000000), split_store_max_num_splits: 1000 }, searcher_config: SearcherConfig { fast_field_cache_capacity: Byte(1000000000), split_footer_cache_capacity: Byte(500000000), max_num_concurrent_split_streams: 100 }, storage_config: None }

---------------------------------------------------
 Connectivity checklist
 ✔ metastore
 ✔ storage
 ✔ .cli-ingest-source

2022-05-13T08:54:39.492Z  INFO {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=1}::{index=hdfs gen=0}: quickwit_indexing::actors::indexing_pipeline: spawn-indexing-pipeline root_dir=./qwdata/indexing/hdfs/.cli-ingest-source merge_policy=StableMultitenantWithTimestampMergePolicy { demux_enabled: false, demux_factor: 8, demux_field_name: None, min_level_num_docs: 100000, merge_enabled: true, merge_factor: 10, max_merge_factor: 12, split_num_docs_target: 10000000 }
2022-05-13T08:54:39.492Z  INFO {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=1}::{index=hdfs gen=0}:{actor=GarbageCollector}: quickwit_indexing::actors::garbage_collector: garbage-collect-operation
2022-05-13T08:54:39.493Z  INFO {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=1}::{index=hdfs gen=0}:{actor=FileSource{source_id=.cli-ingest-source}}: quickwit_indexing::source::file_source: EOF
2022-05-13T08:54:39.493Z  INFO {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=1}::{index=hdfs gen=0}:{actor=FileSource{source_id=.cli-ingest-source}}: quickwit_actors::runner: actor-exit actor_id=SourceActor-floral-bee1 exit_status=Success
2022-05-13T08:54:39.504Z  INFO {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=1}::{index=hdfs gen=0}:{actor=Indexer}:{msg_id=2}: quickwit_indexing::actors::indexer: new-split split_id=01G2YA12M79SJK0A9FF85F1NNH
2022-05-13T08:54:39.504Z  INFO {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=1}::{index=hdfs gen=0}:{actor=Indexer}: quickwit_indexing::actors::indexer: send-to-packager commit_trigger=NoMoreDocs split="01G2YA12M79SJK0A9FF85F1NNH" num_docs=26
2022-05-13T08:54:39.504Z  INFO {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=1}::{index=hdfs gen=0}:{actor=Indexer}: quickwit_actors::runner: actor-exit actor_id=Indexer-broken-hLK5 exit_status=Success
2022-05-13T08:54:39.504Z  INFO {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=1}::{index=hdfs gen=0}:{actor=Packager}:{msg_id=1 num_splits=1}: quickwit_indexing::actors::packager: start-packaging-splits split_ids=["01G2YA12M79SJK0A9FF85F1NNH"]
2022-05-13T08:54:39.504Z  INFO {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=1}::{index=hdfs gen=0}:{actor=Packager}:{msg_id=1 num_splits=1}: quickwit_indexing::actors::packager: commit-split split_id=01G2YA12M79SJK0A9FF85F1NNH
2022-05-13T08:54:39.525Z  INFO {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=1}::{index=hdfs gen=0}:{actor=Packager}:{msg_id=1 num_splits=1}: quickwit_indexing::actors::packager: create-packaged-split split_id="01G2YA12M79SJK0A9FF85F1NNH"
2022-05-13T08:54:39.527Z  INFO {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=1}::{index=hdfs gen=0}:{actor=Uploader}:{msg_id=2 num_splits=1}: quickwit_indexing::actors::uploader: start-stage-and-store-splits split_ids=["01G2YA12M79SJK0A9FF85F1NNH"]
2022-05-13T08:54:39.527Z  INFO {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=1}::{index=hdfs gen=0}:{actor=Uploader}:{msg_id=2 num_splits=1}: quickwit_indexing::actors::uploader: staging-split split_id="01G2YA12M79SJK0A9FF85F1NNH"
2022-05-13T08:54:39.528Z  INFO {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=1}::{index=hdfs gen=0}:{actor=Uploader}:{msg_id=2 num_splits=1}: quickwit_indexing::actors::uploader: storing-split split_id="01G2YA12M79SJK0A9FF85F1NNH"
2022-05-13T08:54:39.528Z  INFO {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=1}::{index=hdfs gen=0}:{actor=Uploader}:{msg_id=2 num_splits=1}: quickwit_indexing::split_store::indexing_split_store: store-split-remote-start
2022-05-13T08:54:39.528Z  INFO {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=1}::{index=hdfs gen=0}:{actor=Uploader}:{msg_id=2 num_splits=1}: quickwit_indexing::split_store::indexing_split_store: store-split-remote-success split_size_in_megabytes=0 elapsed_secs=0.000774282 throughput_mb_s=0 is_mature=false
2022-05-13T08:54:39.528Z  INFO {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=1}::{index=hdfs gen=0}:{actor=Uploader}:{msg_id=2 num_splits=1}: quickwit_indexing::split_store::indexing_split_store: store-in-cache
2022-05-13T08:54:39.529Z  INFO {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=1}::{index=hdfs gen=0}:{actor=Uploader}:{msg_id=2 num_splits=1}: quickwit_indexing::actors::uploader: success-stage-and-store-splits
2022-05-13T08:54:39.529Z  INFO {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=1}::{index=hdfs gen=0}:{actor=Publisher}:{msg_id=2}: quickwit_indexing::actors::publisher: new-split-start
2022-05-13T08:54:39.529Z  INFO {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=1}::{index=hdfs gen=0}:{actor=Publisher}:{msg_id=2}: quickwit_metastore::checkpoint: delta=∆() checkpoint=Ckpt()
2022-05-13T08:54:39.529Z  INFO {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=1}::{index=hdfs gen=0}:{actor=Publisher}:{msg_id=2}: quickwit_indexing::actors::publisher: new-split-success
2022-05-13T08:54:39.529Z  INFO {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=1}::{index=hdfs gen=0}:{actor=Publisher}:{msg_id=2}: quickwit_indexing::actors::publisher: publish-new-splits new_split="01G2YA12M79SJK0A9FF85F1NNH" tts=0.025229309 checkpoint_delta=∆()
2022-05-13T08:54:39.529Z  INFO {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=1}::{index=hdfs gen=0}:{actor=Publisher}:{msg_id=2}: quickwit_metastore::checkpoint: delta=∆() checkpoint=Ckpt()
2022-05-13T08:54:39.529Z ERROR {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=1}::{index=hdfs gen=0}:{actor=Publisher}:{msg_id=2}: quickwit_indexing::actors::publisher: fail-send-suggest-truncate
2022-05-13T08:54:39.728Z  INFO {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=1}::{index=hdfs gen=0}:{actor=Packager}: quickwit_actors::runner: actor-exit actor_id=Packager-damp-muw6 exit_status=Success
2022-05-13T08:54:39.929Z  INFO {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=1}::{index=hdfs gen=0}:{actor=Uploader}: quickwit_actors::runner: actor-exit actor_id=Uploader-muddy-Cqc5 exit_status=Success
2022-05-13T08:54:39.931Z  INFO {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=1}::{index=hdfs gen=0}:{actor=Publisher}: quickwit_actors::runner: actor-exit actor_id=Publisher-frosty-tTFp exit_status=Success
2022-05-13T08:54:39.931Z  INFO {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=1}::{index=hdfs gen=0}:{actor=GarbageCollector}: quickwit_actors::runner: actor-exit actor_id=GarbageCollector-hidden-3LOy exit_status=Success
2022-05-13T08:54:39.931Z  INFO {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=1}::{index=hdfs gen=0}:{actor=MergePlanner}: quickwit_actors::runner: actor-exit actor_id=MergePlanner-weathered-pySf exit_status=Success
2022-05-13T08:54:40.097Z  INFO {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=1}::{index=hdfs gen=0}:{actor=MergeSplitDownloader}: quickwit_actors::runner: actor-exit actor_id=MergeSplitDownloader-snowy-utlI exit_status=Success
2022-05-13T08:54:40.297Z  INFO {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=1}::{index=hdfs gen=0}:{actor=MergeExecutor}: quickwit_actors::runner: actor-exit actor_id=MergeExecutor-billowing-VzpD exit_status=Success
2022-05-13T08:54:40.498Z  INFO {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=1}::{index=hdfs gen=0}:{actor=MergePackager}: quickwit_actors::runner: actor-exit actor_id=MergePackager-broken-i9l1 exit_status=Success
2022-05-13T08:54:40.499Z  INFO {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=1}::{index=hdfs gen=0}:{actor=MergeUploader}: quickwit_actors::runner: actor-exit actor_id=MergeUploader-nameless-MvE3 exit_status=Success
2022-05-13T08:54:40.699Z  INFO {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=1}::{index=hdfs gen=0}:{actor=MergePublisher}: quickwit_actors::runner: actor-exit actor_id=MergePublisher-crimson-UInX exit_status=Success
 Num docs      26 Parse errs     0 PublSplits   1 Input size     0MB Thrghput  0.00MB/s Time 00:00:02
2022-05-13T08:54:41.496Z  INFO {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=1}::{msg_id=15}: quickwit_indexing::actors::indexing_pipeline: indexing-pipeline-success index=hdfs gen=1
2022-05-13T08:54:41.496Z  INFO {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=1}:: quickwit_actors::runner: actor-exit actor_id=IndexingPipeline-fragrant-xMaX exit_status=Success
 Num docs      26 Parse errs     0 PublSplits   1 Input size     0MB Thrghput  0.00MB/s Time 00:00:03
 Num docs      26 Parse errs     0 PublSplits   1 Input size     0MB Thrghput  0.00MB/s Time 00:00:03

Indexed 26 documents in 3s
Now, you can query the index with the following command:
quickwit index search --index hdfs --config ./config/quickwit.yaml --query "my query"
Clearing local cache directory...
2022-05-13T08:54:42.493Z  INFO quickwit_core::index: Clearing cache directory. path=./qwdata/indexing/hdfs/.cli-ingest-source/cache
2022-05-13T08:54:42.497Z  INFO {actor=Scheduler}: quickwit_actors::runner: actor-exit actor_id=Scheduler-shy-Wx3z exit_status=Killed
2022-05-13T08:54:42.694Z  INFO {actor=quickwit_indexing::actors::indexing_service::IndexingService}: quickwit_actors::runner: actor-exit actor_id=quickwit_indexing::actors::indexing_service::IndexingService-nameless-RCh8 exit_status=Killed

[guilload]

Closing in favor of #2150.