Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Compact deduplication and retention is not working as expected #4592

Closed
yasharne opened this issue Aug 23, 2021 · 23 comments
Closed

Compact deduplication and retention is not working as expected #4592

yasharne opened this issue Aug 23, 2021 · 23 comments
Labels
component: compact needs-more-info

Comments

@yasharne
Copy link

Thanos, Prometheus and Golang version used:
Thanos 0.22.0
Golang go1.16.6

Object Storage Provider:
S3 (Minio)

What happened:
We have two Prometheus servers with the exact same configuration to scrape the targets, both with the same cluster and different replica external label and a compactor with the below flags:

          --compact.enable-vertical-compaction
          --deduplication.replica-label="replica"
          --deduplication.func=penalty
          --retention.resolution-raw=20d
          --retention.resolution-5m=90d

What you expected to happen:
See only one block of data for a time range and no raw chunk older than 20 days, but here is the status of the blocks:
screenshot_4
This block is obviously older than 20 days but it is still present in the object store:
screenshot_5
@yeya24
Copy link
Contributor

yeya24 commented Aug 24, 2021 •
edited
Loading

Are you sure your compactor is actually running? The compactor has to finish all the compaction and downsampling jobs first and then start applying retention. I think it is still busy with compacting / downsampling other blocks so it doesn't start the retention.

The issue itself is not related to deduplication I believe. You need to have more concurrency on your compactor or use some other ways to scale it.

If the compactor falls behind, you may want to try the bucket retention tool to delete the old blocks first. For example, see #4406.

@yasharne
Copy link
Author

Hi @yeya24 Thanks for your explanations, yes the compactor is busy with the compacting, but what about deduplication? is that happen at the end of the compaction as well?

@yeya24
Copy link
Contributor

yeya24 commented Aug 24, 2021 •
edited
Loading

Hi @yeya24 Thanks for your explanations, yes the compactor is busy with the compacting, but what about deduplication? is that happen at the end of the compaction as well?

No. It happens at the same time as compaction. The screenshot you showed only mentions the retention problem I think. How do you know the deduplication is not working?

Btw deduplication only works for new blocks (not fully compacted to level4), not for old blocks (level 4 blocks). For the block you showed, the end time of that block is June 10th. If it was compacted to level 4 before you deployed the new deduplication compactor feature, then it won't be recompacted again.
If this is what you want, we already have an old issue #2348.

@yasharne
Copy link
Author

@yeya24 The first screenshot shows the duplicated blocks from both sources, if compactor is not doing this for the previous blocks how can I handle this?

@yeya24
Copy link
Contributor

yeya24 commented Aug 24, 2021

@yeya24 The first screenshot shows the duplicated blocks from both sources, if compactor is not doing this for the previous blocks how can I handle this?

Not possible right now, until we fix #2348.

@yasharne
Copy link
Author

@yeya24 Even the new blocks are also duplicated:
screenshot_6
I'm using --deduplication.replica-label="replica" flag

@yeya24
Copy link
Contributor

yeya24 commented Aug 24, 2021

@yeya24 Even the new blocks are also duplicated:
screenshot_6
I'm using --deduplication.replica-label="replica" flag

Can you show the labels of the newly compacted blocks (for example, compacted today or yesterday)?

@yasharne
Copy link
Author

Sure @yeya24 The compaction happened yesterday and the cluster labels are the same
screenshot_7
screenshot_8

@yeya24
Copy link
Contributor

yeya24 commented Aug 24, 2021

Sure @yeya24 The compaction happened yesterday and the cluster labels are the same
screenshot_7
screenshot_8

This looks weird. Can you attach full logs of your compactor? Btw when did you deploy the compactor?

@yasharne
Copy link
Author

@yeya24 Compactor was working for 11 days but today I redeployed the compactor so I don't have the logs for yesterday but I will send the logs for today's blocks compaction

@yasharne
Copy link
Author

@yeya24 Here is the log output, the rest of the lines are the same:

level=info ts=2021-08-24T06:46:51.876394568Z caller=compact.go:459 msg="compact blocks" count=6 mint=1629504000000 maxt=1629676800000 ulid=01FDVD7D475VP0P15ZEQWECCH0 sources="[01FDM9R46ZY27311CK2WTJA55D 01FDN7YXKKZXFJVQK70NJENTB5 01FDNZYDP3Z195MGA6BE8ZYH1Z 01FDPTPXYPFBK101XBDK8H23ZT 01FDQXEQHVSE6B5YZN3HS9RY8W 01FDVAVDFYVQA3K1SYV3RG2TP8]" duration=23m50.262145375s
level=info ts=2021-08-24T06:46:52.331215914Z caller=compact.go:786 group="0@{cluster=\"bla-vm\", replica=\"0\"}" groupKey=0@16634987252461542783 msg="compacted blocks" new=01FDVD7D475VP0P15ZEQWECCH0 blocks="[/var/thanos/compact/compact/0@16634987252461542783/01FDM9R46ZY27311CK2WTJA55D /var/thanos/compact/compact/0@16634987252461542783/01FDN7YXKKZXFJVQK70NJENTB5 /var/thanos/compact/compact/0@16634987252461542783/01FDNZYDP3Z195MGA6BE8ZYH1Z /var/thanos/compact/compact/0@16634987252461542783/01FDPTPXYPFBK101XBDK8H23ZT /var/thanos/compact/compact/0@16634987252461542783/01FDQXEQHVSE6B5YZN3HS9RY8W /var/thanos/compact/compact/0@16634987252461542783/01FDVAVDFYVQA3K1SYV3RG2TP8]" duration=23m50.716974763s overlapping_blocks=false
level=info ts=2021-08-24T06:47:01.763666181Z caller=compact.go:459 msg="compact blocks" count=6 mint=1629504000000 maxt=1629676800000 ulid=01FDVD7SJAGNKXD2ZPEMV36G08 sources="[01FDM9DZ1ZNK9GV6RVJ24N46BD 01FDN7YQBCVXCFBPSQN50SFMVV 01FDNZYM9KX05K5PDR8VXXEAY3 01FDPTQ0Y3CEVE55NG3BND81M0 01FDQXESBJSX1XCG6E3RNER09H 01FDVAVDQ6Q64ZZEB280MRVB3Q]" duration=23m45.217234424s
level=info ts=2021-08-24T06:47:02.401511004Z caller=compact.go:786 group="0@{cluster=\"bla-vm\", replica=\"1\"}" groupKey=0@12213323787495744540 msg="compacted blocks" new=01FDVD7SJAGNKXD2ZPEMV36G08 blocks="[/var/thanos/compact/compact/0@12213323787495744540/01FDM9DZ1ZNK9GV6RVJ24N46BD /var/thanos/compact/compact/0@12213323787495744540/01FDN7YQBCVXCFBPSQN50SFMVV /var/thanos/compact/compact/0@12213323787495744540/01FDNZYM9KX05K5PDR8VXXEAY3 /var/thanos/compact/compact/0@12213323787495744540/01FDPTQ0Y3CEVE55NG3BND81M0 /var/thanos/compact/compact/0@12213323787495744540/01FDQXESBJSX1XCG6E3RNER09H /var/thanos/compact/compact/0@12213323787495744540/01FDVAVDQ6Q64ZZEB280MRVB3Q]" duration=23m45.855080972s overlapping_blocks=false
level=info ts=2021-08-24T06:47:34.862321645Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=741.746883ms cached=57 returned=57 partial=1
level=info ts=2021-08-24T06:48:34.898493813Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=780.376463ms cached=57 returned=57 partial=2
level=info ts=2021-08-24T06:49:34.794791125Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=676.120849ms cached=57 returned=57 partial=2
level=info ts=2021-08-24T06:50:34.153363165Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1.587276291s cached=57 returned=57 partial=2
level=info ts=2021-08-24T06:50:34.160234557Z caller=clean.go:33 msg="started cleaning of aborted partial uploads"
level=info ts=2021-08-24T06:50:34.160306983Z caller=clean.go:60 msg="cleaning of aborted partial uploads done"
level=info ts=2021-08-24T06:50:34.160326094Z caller=blocks_cleaner.go:43 msg="started cleaning of blocks marked for deletion"
level=info ts=2021-08-24T06:50:34.160348688Z caller=blocks_cleaner.go:57 msg="cleaning of blocks marked for deletion done"
level=info ts=2021-08-24T06:50:34.69923179Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=580.801933ms cached=57 returned=57 partial=2
level=info ts=2021-08-24T06:51:34.88654375Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=767.283845ms cached=57 returned=57 partial=2
level=info ts=2021-08-24T06:52:34.818723121Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=700.620778ms cached=57 returned=57 partial=2
level=info ts=2021-08-24T06:53:34.888453398Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=770.089883ms cached=57 returned=57 partial=2
level=info ts=2021-08-24T06:54:34.924116824Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=806.226184ms cached=57 returned=57 partial=2
level=info ts=2021-08-24T06:55:34.045234414Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1.482166849s cached=57 returned=57 partial=2
level=info ts=2021-08-24T06:55:34.053640039Z caller=clean.go:33 msg="started cleaning of aborted partial uploads"
level=info ts=2021-08-24T06:55:34.053796494Z caller=clean.go:60 msg="cleaning of aborted partial uploads done"
level=info ts=2021-08-24T06:55:34.053819041Z caller=blocks_cleaner.go:43 msg="started cleaning of blocks marked for deletion"
level=info ts=2021-08-24T06:55:34.053843439Z caller=blocks_cleaner.go:57 msg="cleaning of blocks marked for deletion done"
level=info ts=2021-08-24T06:55:34.697549777Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=578.304177ms cached=57 returned=57 partial=2
level=info ts=2021-08-24T06:56:34.88444679Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=765.985184ms cached=57 returned=57 partial=2
level=info ts=2021-08-24T06:56:38.932142934Z caller=compact.go:824 group="0@{cluster=\"bla-vm\", replica=\"0\"}" groupKey=0@16634987252461542783 msg="uploaded block" result_block=01FDVD7D475VP0P15ZEQWECCH0 duration=9m31.371113081s
level=info ts=2021-08-24T06:56:44.173189489Z caller=compact.go:846 group="0@{cluster=\"bla-vm\", replica=\"0\"}" groupKey=0@16634987252461542783 msg="marking compacted block for deletion" old_block=01FDM9R46ZY27311CK2WTJA55D
level=info ts=2021-08-24T06:56:44.508808481Z caller=block.go:209 group="0@{cluster=\"bla-vm\", replica=\"0\"}" groupKey=0@16634987252461542783 msg="block has been marked for deletion" block=01FDM9R46ZY27311CK2WTJA55D
level=info ts=2021-08-24T06:56:46.191263928Z caller=compact.go:846 group="0@{cluster=\"bla-vm\", replica=\"0\"}" groupKey=0@16634987252461542783 msg="marking compacted block for deletion" old_block=01FDN7YXKKZXFJVQK70NJENTB5
level=info ts=2021-08-24T06:56:46.626783003Z caller=block.go:209 group="0@{cluster=\"bla-vm\", replica=\"0\"}" groupKey=0@16634987252461542783 msg="block has been marked for deletion" block=01FDN7YXKKZXFJVQK70NJENTB5
level=info ts=2021-08-24T06:56:47.931522191Z caller=compact.go:846 group="0@{cluster=\"bla-vm\", replica=\"0\"}" groupKey=0@16634987252461542783 msg="marking compacted block for deletion" old_block=01FDNZYDP3Z195MGA6BE8ZYH1Z
level=info ts=2021-08-24T06:56:48.254312021Z caller=block.go:209 group="0@{cluster=\"bla-vm\", replica=\"0\"}" groupKey=0@16634987252461542783 msg="block has been marked for deletion" block=01FDNZYDP3Z195MGA6BE8ZYH1Z
level=info ts=2021-08-24T06:56:51.518039835Z caller=compact.go:846 group="0@{cluster=\"bla-vm\", replica=\"0\"}" groupKey=0@16634987252461542783 msg="marking compacted block for deletion" old_block=01FDPTPXYPFBK101XBDK8H23ZT
level=info ts=2021-08-24T06:56:51.851897461Z caller=block.go:209 group="0@{cluster=\"bla-vm\", replica=\"0\"}" groupKey=0@16634987252461542783 msg="block has been marked for deletion" block=01FDPTPXYPFBK101XBDK8H23ZT
level=info ts=2021-08-24T06:56:55.176985987Z caller=compact.go:846 group="0@{cluster=\"bla-vm\", replica=\"0\"}" groupKey=0@16634987252461542783 msg="marking compacted block for deletion" old_block=01FDQXEQHVSE6B5YZN3HS9RY8W
level=info ts=2021-08-24T06:56:55.500219494Z caller=block.go:209 group="0@{cluster=\"bla-vm\", replica=\"0\"}" groupKey=0@16634987252461542783 msg="block has been marked for deletion" block=01FDQXEQHVSE6B5YZN3HS9RY8W
level=info ts=2021-08-24T06:56:56.998619603Z caller=compact.go:846 group="0@{cluster=\"bla-vm\", replica=\"0\"}" groupKey=0@16634987252461542783 msg="marking compacted block for deletion" old_block=01FDVAVDFYVQA3K1SYV3RG2TP8
level=info ts=2021-08-24T06:56:57.44385401Z caller=block.go:209 group="0@{cluster=\"bla-vm\", replica=\"0\"}" groupKey=0@16634987252461542783 msg="block has been marked for deletion" block=01FDVAVDFYVQA3K1SYV3RG2TP8
level=info ts=2021-08-24T06:57:01.433330585Z caller=compact.go:824 group="0@{cluster=\"bla-vm\", replica=\"1\"}" groupKey=0@12213323787495744540 msg="uploaded block" result_block=01FDVD7SJAGNKXD2ZPEMV36G08 duration=9m44.131186463s
level=info ts=2021-08-24T06:57:01.623700114Z caller=compact.go:846 group="0@{cluster=\"bla-vm\", replica=\"1\"}" groupKey=0@12213323787495744540 msg="marking compacted block for deletion" old_block=01FDM9DZ1ZNK9GV6RVJ24N46BD
level=info ts=2021-08-24T06:57:02.037127878Z caller=block.go:209 group="0@{cluster=\"bla-vm\", replica=\"1\"}" groupKey=0@12213323787495744540 msg="block has been marked for deletion" block=01FDM9DZ1ZNK9GV6RVJ24N46BD
level=info ts=2021-08-24T06:57:02.230833137Z caller=compact.go:846 group="0@{cluster=\"bla-vm\", replica=\"1\"}" groupKey=0@12213323787495744540 msg="marking compacted block for deletion" old_block=01FDN7YQBCVXCFBPSQN50SFMVV
level=info ts=2021-08-24T06:57:02.559639606Z caller=block.go:209 group="0@{cluster=\"bla-vm\", replica=\"1\"}" groupKey=0@12213323787495744540 msg="block has been marked for deletion" block=01FDN7YQBCVXCFBPSQN50SFMVV
level=info ts=2021-08-24T06:57:02.780170156Z caller=compact.go:846 group="0@{cluster=\"bla-vm\", replica=\"1\"}" groupKey=0@12213323787495744540 msg="marking compacted block for deletion" old_block=01FDNZYM9KX05K5PDR8VXXEAY3
level=info ts=2021-08-24T06:57:03.122680897Z caller=block.go:209 group="0@{cluster=\"bla-vm\", replica=\"1\"}" groupKey=0@12213323787495744540 msg="block has been marked for deletion" block=01FDNZYM9KX05K5PDR8VXXEAY3
level=info ts=2021-08-24T06:57:03.384227267Z caller=compact.go:846 group="0@{cluster=\"bla-vm\", replica=\"1\"}" groupKey=0@12213323787495744540 msg="marking compacted block for deletion" old_block=01FDPTQ0Y3CEVE55NG3BND81M0
level=info ts=2021-08-24T06:57:03.870293533Z caller=block.go:209 group="0@{cluster=\"bla-vm\", replica=\"1\"}" groupKey=0@12213323787495744540 msg="block has been marked for deletion" block=01FDPTQ0Y3CEVE55NG3BND81M0
level=info ts=2021-08-24T06:57:04.212274685Z caller=compact.go:846 group="0@{cluster=\"bla-vm\", replica=\"1\"}" groupKey=0@12213323787495744540 msg="marking compacted block for deletion" old_block=01FDQXESBJSX1XCG6E3RNER09H
level=info ts=2021-08-24T06:57:04.682904507Z caller=block.go:209 group="0@{cluster=\"bla-vm\", replica=\"1\"}" groupKey=0@12213323787495744540 msg="block has been marked for deletion" block=01FDQXESBJSX1XCG6E3RNER09H
level=info ts=2021-08-24T06:57:05.016977953Z caller=compact.go:846 group="0@{cluster=\"bla-vm\", replica=\"1\"}" groupKey=0@12213323787495744540 msg="marking compacted block for deletion" old_block=01FDVAVDQ6Q64ZZEB280MRVB3Q
level=info ts=2021-08-24T06:57:05.263845688Z caller=block.go:209 group="0@{cluster=\"bla-vm\", replica=\"1\"}" groupKey=0@12213323787495744540 msg="block has been marked for deletion" block=01FDVAVDQ6Q64ZZEB280MRVB3Q
level=info ts=2021-08-24T06:57:07.268245346Z caller=compact.go:948 msg="start sync of metas"
level=info ts=2021-08-24T06:57:09.121535718Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1.853173871s cached=59 returned=47 partial=0
level=info ts=2021-08-24T06:57:09.132162152Z caller=compact.go:953 msg="start of GC"
level=info ts=2021-08-24T06:57:09.134566034Z caller=compact.go:976 msg="start of compactions"
level=info ts=2021-08-24T06:57:09.136669741Z caller=compact.go:1008 msg="compaction iterations done"
level=info ts=2021-08-24T06:57:09.136979538Z caller=compact.go:420 msg="start first pass of downsampling"
level=info ts=2021-08-24T06:57:10.948745844Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1.811665065s cached=59 returned=47 partial=0
level=info ts=2021-08-24T06:57:34.944379597Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=826.473111ms cached=59 returned=59 partial=0
level=info ts=2021-08-24T06:58:34.972554074Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=839.939063ms cached=59 returned=59 partial=0
level=info ts=2021-08-24T06:59:34.889864957Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=770.71425ms cached=59 returned=59 partial=0
level=info ts=2021-08-24T07:00:34.153349695Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1.582483828s cached=59 returned=47 partial=0
level=info ts=2021-08-24T07:00:34.175332464Z caller=clean.go:33 msg="started cleaning of aborted partial uploads"
level=info ts=2021-08-24T07:00:34.175470122Z caller=clean.go:60 msg="cleaning of aborted partial uploads done"
level=info ts=2021-08-24T07:00:34.175511203Z caller=blocks_cleaner.go:43 msg="started cleaning of blocks marked for deletion"
level=info ts=2021-08-24T07:00:34.175571321Z caller=blocks_cleaner.go:57 msg="cleaning of blocks marked for deletion done"
level=info ts=2021-08-24T07:00:34.688950415Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=570.668749ms cached=59 returned=59 partial=0
level=info ts=2021-08-24T07:01:34.890083978Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=771.109435ms cached=61 returned=61 partial=0
level=info ts=2021-08-24T07:02:34.895082663Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=777.038801ms cached=61 returned=61 partial=0
level=info ts=2021-08-24T07:03:34.868318454Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=749.874346ms cached=61 returned=61 partial=0
level=info ts=2021-08-24T07:04:34.888530931Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=770.338362ms cached=61 returned=61 partial=0
level=info ts=2021-08-24T07:05:34.111855905Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1.549185421s cached=61 returned=47 partial=0
level=info ts=2021-08-24T07:05:34.119027495Z caller=clean.go:33 msg="started cleaning of aborted partial uploads"
level=info ts=2021-08-24T07:05:34.119117871Z caller=clean.go:60 msg="cleaning of aborted partial uploads done"
level=info ts=2021-08-24T07:05:34.119141466Z caller=blocks_cleaner.go:43 msg="started cleaning of blocks marked for deletion"
level=info ts=2021-08-24T07:05:34.119165976Z caller=blocks_cleaner.go:57 msg="cleaning of blocks marked for deletion done"
level=info ts=2021-08-24T07:05:34.713589202Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=595.464149ms cached=61 returned=61 partial=0
level=info ts=2021-08-24T07:06:35.06044861Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=942.199203ms cached=61 returned=61 partial=0
level=info ts=2021-08-24T07:07:34.899210276Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=780.010572ms cached=61 returned=61 partial=0
level=info ts=2021-08-24T07:08:35.884124977Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1.766119s cached=61 returned=61 partial=0
level=info ts=2021-08-24T07:09:34.871944435Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=754.026925ms cached=61 returned=61 partial=0
level=info ts=2021-08-24T07:10:34.038657055Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1.476052979s cached=61 returned=47 partial=0
level=info ts=2021-08-24T07:10:34.051236094Z caller=clean.go:33 msg="started cleaning of aborted partial uploads"
level=info ts=2021-08-24T07:10:34.051742014Z caller=clean.go:60 msg="cleaning of aborted partial uploads done"
level=info ts=2021-08-24T07:10:34.051857222Z caller=blocks_cleaner.go:43 msg="started cleaning of blocks marked for deletion"
level=info ts=2021-08-24T07:10:34.051900063Z caller=blocks_cleaner.go:57 msg="cleaning of blocks marked for deletion done"
level=info ts=2021-08-24T07:10:34.699361739Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=581.350119ms cached=61 returned=61 partial=0
level=info ts=2021-08-24T07:11:34.924863077Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=804.88282ms cached=61 returned=61 partial=0
level=info ts=2021-08-24T07:12:34.890494754Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=771.128337ms cached=61 returned=61 partial=0
level=info ts=2021-08-24T07:13:34.996064077Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=877.883469ms cached=61 returned=61 partial=0
level=info ts=2021-08-24T07:14:34.916670009Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=798.386332ms cached=61 returned=61 partial=0
level=info ts=2021-08-24T07:15:34.249174027Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1.68652217s cached=61 returned=47 partial=0
level=info ts=2021-08-24T07:15:34.259163133Z caller=clean.go:33 msg="started cleaning of aborted partial uploads"
level=info ts=2021-08-24T07:15:34.25925183Z caller=clean.go:60 msg="cleaning of aborted partial uploads done"
level=info ts=2021-08-24T07:15:34.259268813Z caller=blocks_cleaner.go:43 msg="started cleaning of blocks marked for deletion"
level=info ts=2021-08-24T07:15:34.259316657Z caller=blocks_cleaner.go:57 msg="cleaning of blocks marked for deletion done"
level=info ts=2021-08-24T07:15:34.692806581Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=574.914168ms cached=61 returned=61 partial=0
level=info ts=2021-08-24T07:16:34.72876326Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=609.93562ms cached=61 returned=61 partial=0
level=info ts=2021-08-24T07:17:34.912461776Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=793.704979ms cached=61 returned=61 partial=0
level=info ts=2021-08-24T07:18:34.903932698Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=784.529419ms cached=61 returned=61 partial=0
level=info ts=2021-08-24T07:19:34.929712757Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=811.46678ms cached=61 returned=61 partial=0
level=info ts=2021-08-24T07:20:34.203834133Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1.641204461s cached=61 returned=47 partial=0
level=info ts=2021-08-24T07:20:34.213186647Z caller=clean.go:33 msg="started cleaning of aborted partial uploads"
level=info ts=2021-08-24T07:20:34.21327345Z caller=clean.go:60 msg="cleaning of aborted partial uploads done"
level=info ts=2021-08-24T07:20:34.213296007Z caller=blocks_cleaner.go:43 msg="started cleaning of blocks marked for deletion"
level=info ts=2021-08-24T07:20:34.213332584Z caller=blocks_cleaner.go:57 msg="cleaning of blocks marked for deletion done"
level=info ts=2021-08-24T07:20:34.690922583Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=572.977207ms cached=61 returned=61 partial=0
level=info ts=2021-08-24T07:21:34.898822405Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=780.969826ms cached=61 returned=61 partial=0
level=info ts=2021-08-24T07:22:34.807041869Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=687.657415ms cached=61 returned=61 partial=0
level=info ts=2021-08-24T07:23:34.898008092Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=779.211897ms cached=61 returned=61 partial=0
level=info ts=2021-08-24T07:24:34.862841062Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=744.603361ms cached=61 returned=61 partial=0
level=info ts=2021-08-24T07:25:34.018655174Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1.456035268s cached=61 returned=47 partial=0
level=info ts=2021-08-24T07:25:34.057396249Z caller=clean.go:33 msg="started cleaning of aborted partial uploads"
level=info ts=2021-08-24T07:25:34.057503579Z caller=clean.go:60 msg="cleaning of aborted partial uploads done"
level=info ts=2021-08-24T07:25:34.057519568Z caller=blocks_cleaner.go:43 msg="started cleaning of blocks marked for deletion"
level=info ts=2021-08-24T07:25:34.057561952Z caller=blocks_cleaner.go:57 msg="cleaning of blocks marked for deletion done"
level=info ts=2021-08-24T07:25:34.706578367Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=588.625554ms cached=61 returned=61 partial=0
level=info ts=2021-08-24T07:26:34.899417109Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=780.798361ms cached=61 returned=61 partial=0
level=info ts=2021-08-24T07:27:35.502480271Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1.384398033s cached=61 returned=61 partial=0
level=info ts=2021-08-24T07:28:34.836767345Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=717.056593ms cached=61 returned=61 partial=0
level=info ts=2021-08-24T07:29:34.911799726Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=793.858889ms cached=61 returned=61 partial=0
level=info ts=2021-08-24T07:30:34.309852623Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1.747126414s cached=61 returned=49 partial=0
level=info ts=2021-08-24T07:30:34.329245826Z caller=clean.go:33 msg="started cleaning of aborted partial uploads"
level=info ts=2021-08-24T07:30:34.329318925Z caller=clean.go:60 msg="cleaning of aborted partial uploads done"
level=info ts=2021-08-24T07:30:34.329338323Z caller=blocks_cleaner.go:43 msg="started cleaning of blocks marked for deletion"
level=info ts=2021-08-24T07:30:34.704137328Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=586.010456ms cached=61 returned=61 partial=0
level=info ts=2021-08-24T07:30:35.647209005Z caller=blocks_cleaner.go:53 msg="deleted block marked for deletion" block=01FDPTQ0Y3CEVE55NG3BND81M0
level=info ts=2021-08-24T07:30:37.071683655Z caller=blocks_cleaner.go:53 msg="deleted block marked for deletion" block=01FDNZYDP3Z195MGA6BE8ZYH1Z
level=info ts=2021-08-24T07:30:38.482015279Z caller=blocks_cleaner.go:53 msg="deleted block marked for deletion" block=01FDVAVDFYVQA3K1SYV3RG2TP8
level=info ts=2021-08-24T07:30:40.091380766Z caller=blocks_cleaner.go:53 msg="deleted block marked for deletion" block=01FDM9R46ZY27311CK2WTJA55D
level=info ts=2021-08-24T07:30:41.785387166Z caller=blocks_cleaner.go:53 msg="deleted block marked for deletion" block=01FDQXESBJSX1XCG6E3RNER09H
level=info ts=2021-08-24T07:30:43.288736073Z caller=blocks_cleaner.go:53 msg="deleted block marked for deletion" block=01FDVAVDQ6Q64ZZEB280MRVB3Q
level=info ts=2021-08-24T07:30:45.275277019Z caller=blocks_cleaner.go:53 msg="deleted block marked for deletion" block=01FDN7YXKKZXFJVQK70NJENTB5
level=info ts=2021-08-24T07:30:46.973124872Z caller=blocks_cleaner.go:53 msg="deleted block marked for deletion" block=01FDNZYM9KX05K5PDR8VXXEAY3
level=info ts=2021-08-24T07:30:48.642749549Z caller=blocks_cleaner.go:53 msg="deleted block marked for deletion" block=01FDPTPXYPFBK101XBDK8H23ZT
level=info ts=2021-08-24T07:30:50.414015633Z caller=blocks_cleaner.go:53 msg="deleted block marked for deletion" block=01FDM9DZ1ZNK9GV6RVJ24N46BD
level=info ts=2021-08-24T07:30:51.932827928Z caller=blocks_cleaner.go:53 msg="deleted block marked for deletion" block=01FDN7YQBCVXCFBPSQN50SFMVV
level=info ts=2021-08-24T07:30:53.721956576Z caller=blocks_cleaner.go:53 msg="deleted block marked for deletion" block=01FDQXEQHVSE6B5YZN3HS9RY8W
level=info ts=2021-08-24T07:30:53.722025024Z caller=blocks_cleaner.go:57 msg="cleaning of blocks marked for deletion done"
level=info ts=2021-08-24T07:31:35.779756633Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1.661593964s cached=49 returned=49 partial=0
level=info ts=2021-08-24T07:32:34.970804506Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=851.884513ms cached=49 returned=49 partial=0
level=info ts=2021-08-24T07:33:12.852340125Z caller=downsample.go:320 msg="downloaded block" id=01FDR905HRYX3KXSC572YV6NWB duration=36m1.896258437s
level=info ts=2021-08-24T07:33:34.883460875Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=764.95008ms cached=49 returned=49 partial=0
level=info ts=2021-08-24T07:34:34.720295344Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=602.128393ms cached=49 returned=49 partial=0
level=info ts=2021-08-24T07:35:33.959977995Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1.396322204s cached=49 returned=49 partial=0
level=info ts=2021-08-24T07:35:33.97408917Z caller=clean.go:33 msg="started cleaning of aborted partial uploads"
level=info ts=2021-08-24T07:35:33.974162063Z caller=clean.go:60 msg="cleaning of aborted partial uploads done"
level=info ts=2021-08-24T07:35:33.974181906Z caller=blocks_cleaner.go:43 msg="started cleaning of blocks marked for deletion"
level=info ts=2021-08-24T07:35:33.974197513Z caller=blocks_cleaner.go:57 msg="cleaning of blocks marked for deletion done"
``

@yasharne
Copy link
Author

Hi @yeya24 Is it safe to delete objects (compressed) from one of the sources? if yes how can I do this?

@yeya24
Copy link
Contributor

yeya24 commented Sep 1, 2021 •
edited
Loading

It looks really weird from your compactor logs.

If deduplication.replica-label flag is specified correctly, then the replica label should be removed by the label removal modifier and disappears. But from the logs, the label is still present so I am wondering is your configuration the latest?

level=info ts=2021-08-24T06:47:02.401511004Z caller=compact.go:786 group="0@{cluster=\"bla-vm\", replica=\"1\"}" groupKey=0@12213323787495744540

Another way to check is to enable debug level log on your compactor and see if there is any log line with replica label removed.

@yasharne
Copy link
Author

yasharne commented Sep 4, 2021

@yeya24 here is the list flags the compactor is running with:

          - compact
          - --wait 
          - --wait-interval=5m
          - --consistency-delay=30m
          - --compact.enable-vertical-compaction
          - --deduplication.replica-label="replica"
          - --deduplication.func=penalty
          - --retention.resolution-raw=20d
          - --retention.resolution-5m=28d
          - --delete-delay=30m
          - --compact.concurrency=2
          - --data-dir=/var/thanos/compact
          - |
            --objstore.config=type: S3

@yeya24
Copy link
Contributor

yeya24 commented Sep 4, 2021

@yeya24 here is the list flags the compactor is running with:


          - compact

          - --wait 

          - --wait-interval=5m

          - --consistency-delay=30m

          - --compact.enable-vertical-compaction

          - --deduplication.replica-label="replica"

          - --deduplication.func=penalty

          - --retention.resolution-raw=20d

          - --retention.resolution-5m=28d

          - --delete-delay=30m

          - --compact.concurrency=2

          - --data-dir=/var/thanos/compact

          - |

            --objstore.config=type: S3

Can you enable the debug log?

@yasharne
Copy link
Author

yasharne commented Sep 6, 2021

@yeya24 I let the compactor run for two days but I don't see anything like replica label removed:

oc logs -f thanos-compact-0 | grep -i "removed"

Shows nothing, also here are some sample of the debug output:

oc logs -f thanos-compact-0  | grep -i debug
level=debug ts=2021-09-06T20:14:22.177730824Z caller=downsample.go:230 msg="downsampling bucket" concurrency=1
level=debug ts=2021-09-06T20:14:22.178008751Z caller=fetcher.go:321 component=block.BaseFetcher msg="fetching meta data" concurrency=32
level=debug ts=2021-09-06T20:14:22.412834631Z caller=block.go:233 msg="deleted file" file=01FEXK3B45N616Z9Q6VC1EFZ88/meta.json bucket="tracing: thanos"
level=debug ts=2021-09-06T20:14:22.731444668Z caller=block.go:276 msg="deleted file" file=01FEXK3B45N616Z9Q6VC1EFZ88/index bucket="tracing: thanos"
level=debug ts=2021-09-06T20:14:23.036385257Z caller=block.go:276 msg="deleted file" file=01FEXK3B45N616Z9Q6VC1EFZ88/chunks/000001 bucket="tracing: thanos"
level=debug ts=2021-09-06T20:14:23.302650605Z caller=block.go:256 msg="deleted file" file=01FEXK3B45N616Z9Q6VC1EFZ88/deletion-mark.json bucket="tracing: thanos"
level=debug ts=2021-09-06T20:14:23.549013948Z caller=block.go:233 msg="deleted file" file=01FEXK3B489BPKVRQ3QWD43PZD/meta.json bucket="tracing: thanos"
level=debug ts=2021-09-06T20:14:23.799326537Z caller=block.go:276 msg="deleted file" file=01FEXK3B489BPKVRQ3QWD43PZD/index bucket="tracing: thanos"
level=debug ts=2021-09-06T20:14:24.059709812Z caller=block.go:276 msg="deleted file" file=01FEXK3B489BPKVRQ3QWD43PZD/chunks/000001 bucket="tracing: thanos"
level=debug ts=2021-09-06T20:14:24.303787771Z caller=block.go:256 msg="deleted file" file=01FEXK3B489BPKVRQ3QWD43PZD/deletion-mark.json bucket="tracing: thanos"
level=debug ts=2021-09-06T20:14:24.554820321Z caller=block.go:233 msg="deleted file" file=01FEXC7KW87NTHTW3M0BMESERC/meta.json bucket="tracing: thanos"
level=debug ts=2021-09-06T20:14:24.806709151Z caller=block.go:276 msg="deleted file" file=01FEXC7KW87NTHTW3M0BMESERC/index bucket="tracing: thanos"
level=debug ts=2021-09-06T20:14:25.062789582Z caller=block.go:276 msg="deleted file" file=01FEXC7KW87NTHTW3M0BMESERC/chunks/000001 bucket="tracing: thanos"
level=debug ts=2021-09-06T20:14:25.389379575Z caller=block.go:256 msg="deleted file" file=01FEXC7KW87NTHTW3M0BMESERC/deletion-mark.json bucket="tracing: thanos"
level=debug ts=2021-09-06T20:14:25.649826248Z caller=block.go:233 msg="deleted file" file=01FEXSZ2C8WW60TGMS7GDWB6M6/meta.json bucket="tracing: thanos"
level=debug ts=2021-09-06T20:14:25.998547549Z caller=block.go:276 msg="deleted file" file=01FEXSZ2C8WW60TGMS7GDWB6M6/index bucket="tracing: thanos"
level=debug ts=2021-09-06T20:14:26.325386928Z caller=block.go:276 msg="deleted file" file=01FEXSZ2C8WW60TGMS7GDWB6M6/chunks/000001 bucket="tracing: thanos"
level=debug ts=2021-09-06T20:14:26.584220808Z caller=block.go:256 msg="deleted file" file=01FEXSZ2C8WW60TGMS7GDWB6M6/deletion-mark.json bucket="tracing: thanos"
level=debug ts=2021-09-06T20:14:26.831705749Z caller=block.go:233 msg="deleted file" file=01FEXSZ2C583DD4KBBYKGGXYN1/meta.json bucket="tracing: thanos"
level=debug ts=2021-09-06T20:14:27.078772537Z caller=block.go:276 msg="deleted file" file=01FEXSZ2C583DD4KBBYKGGXYN1/index bucket="tracing: thanos"
level=debug ts=2021-09-06T20:14:27.53999946Z caller=block.go:276 msg="deleted file" file=01FEXSZ2C583DD4KBBYKGGXYN1/chunks/000001 bucket="tracing: thanos"
level=debug ts=2021-09-06T20:14:27.891457189Z caller=block.go:256 msg="deleted file" file=01FEXSZ2C583DD4KBBYKGGXYN1/deletion-mark.json bucket="tracing: thanos"
level=debug ts=2021-09-06T20:14:28.128874173Z caller=block.go:233 msg="deleted file" file=01FEXC7KW5DXRM7QRBREP9YWH2/meta.json bucket="tracing: thanos"
level=debug ts=2021-09-06T20:14:28.546011691Z caller=block.go:276 msg="deleted file" file=01FEXC7KW5DXRM7QRBREP9YWH2/index bucket="tracing: thanos"
level=debug ts=2021-09-06T20:14:28.797824494Z caller=block.go:276 msg="deleted file" file=01FEXC7KW5DXRM7QRBREP9YWH2/chunks/000001 bucket="tracing: thanos"
level=debug ts=2021-09-06T20:14:29.127959785Z caller=block.go:256 msg="deleted file" file=01FEXC7KW5DXRM7QRBREP9YWH2/deletion-mark.json bucket="tracing: thanos"
level=debug ts=2021-09-06T20:14:29.128101253Z caller=fetcher.go:321 component=block.BaseFetcher msg="fetching meta data" concurrency=32
level=debug ts=2021-09-06T20:15:18.7626315Z caller=fetcher.go:321 component=block.BaseFetcher msg="fetching meta data" concurrency=32
level=debug ts=2021-09-06T20:16:18.763021022Z caller=fetcher.go:321 component=block.BaseFetcher msg="fetching meta data" concurrency=32
level=debug ts=2021-09-06T20:17:18.762911952Z caller=fetcher.go:321 component=block.BaseFetcher msg="fetching meta data" concurrency=32
level=debug ts=2021-09-06T20:18:18.763002026Z caller=fetcher.go:321 component=block.BaseFetcher msg="fetching meta data" concurrency=32
level=debug ts=2021-09-06T20:19:17.56787997Z caller=fetcher.go:321 component=block.BaseFetcher msg="fetching meta data" concurrency=32
level=debug ts=2021-09-06T20:19:18.763591536Z caller=fetcher.go:321 component=block.BaseFetcher msg="fetching meta data" concurrency=32
level=debug ts=2021-09-06T20:19:19.936175284Z caller=fetcher.go:321 component=block.BaseFetcher msg="fetching meta data" concurrency=32

@yeya24
Copy link
Contributor

yeya24 commented Sep 6, 2021

@yeya24 I let the compactor run for two days but I don't see anything like replica label removed:


oc logs -f thanos-compact-0 | grep -i "removed"

Shows nothing, also here are some sample of the debug output:


oc logs -f thanos-compact-0  | grep -i debug

level=debug ts=2021-09-06T20:14:22.177730824Z caller=downsample.go:230 msg="downsampling bucket" concurrency=1

level=debug ts=2021-09-06T20:14:22.178008751Z caller=fetcher.go:321 component=block.BaseFetcher msg="fetching meta data" concurrency=32

level=debug ts=2021-09-06T20:14:22.412834631Z caller=block.go:233 msg="deleted file" file=01FEXK3B45N616Z9Q6VC1EFZ88/meta.json bucket="tracing: thanos"

level=debug ts=2021-09-06T20:14:22.731444668Z caller=block.go:276 msg="deleted file" file=01FEXK3B45N616Z9Q6VC1EFZ88/index bucket="tracing: thanos"

level=debug ts=2021-09-06T20:14:23.036385257Z caller=block.go:276 msg="deleted file" file=01FEXK3B45N616Z9Q6VC1EFZ88/chunks/000001 bucket="tracing: thanos"

level=debug ts=2021-09-06T20:14:23.302650605Z caller=block.go:256 msg="deleted file" file=01FEXK3B45N616Z9Q6VC1EFZ88/deletion-mark.json bucket="tracing: thanos"

level=debug ts=2021-09-06T20:14:23.549013948Z caller=block.go:233 msg="deleted file" file=01FEXK3B489BPKVRQ3QWD43PZD/meta.json bucket="tracing: thanos"

level=debug ts=2021-09-06T20:14:23.799326537Z caller=block.go:276 msg="deleted file" file=01FEXK3B489BPKVRQ3QWD43PZD/index bucket="tracing: thanos"

level=debug ts=2021-09-06T20:14:24.059709812Z caller=block.go:276 msg="deleted file" file=01FEXK3B489BPKVRQ3QWD43PZD/chunks/000001 bucket="tracing: thanos"

level=debug ts=2021-09-06T20:14:24.303787771Z caller=block.go:256 msg="deleted file" file=01FEXK3B489BPKVRQ3QWD43PZD/deletion-mark.json bucket="tracing: thanos"

level=debug ts=2021-09-06T20:14:24.554820321Z caller=block.go:233 msg="deleted file" file=01FEXC7KW87NTHTW3M0BMESERC/meta.json bucket="tracing: thanos"

level=debug ts=2021-09-06T20:14:24.806709151Z caller=block.go:276 msg="deleted file" file=01FEXC7KW87NTHTW3M0BMESERC/index bucket="tracing: thanos"

level=debug ts=2021-09-06T20:14:25.062789582Z caller=block.go:276 msg="deleted file" file=01FEXC7KW87NTHTW3M0BMESERC/chunks/000001 bucket="tracing: thanos"

level=debug ts=2021-09-06T20:14:25.389379575Z caller=block.go:256 msg="deleted file" file=01FEXC7KW87NTHTW3M0BMESERC/deletion-mark.json bucket="tracing: thanos"

level=debug ts=2021-09-06T20:14:25.649826248Z caller=block.go:233 msg="deleted file" file=01FEXSZ2C8WW60TGMS7GDWB6M6/meta.json bucket="tracing: thanos"

level=debug ts=2021-09-06T20:14:25.998547549Z caller=block.go:276 msg="deleted file" file=01FEXSZ2C8WW60TGMS7GDWB6M6/index bucket="tracing: thanos"

level=debug ts=2021-09-06T20:14:26.325386928Z caller=block.go:276 msg="deleted file" file=01FEXSZ2C8WW60TGMS7GDWB6M6/chunks/000001 bucket="tracing: thanos"

level=debug ts=2021-09-06T20:14:26.584220808Z caller=block.go:256 msg="deleted file" file=01FEXSZ2C8WW60TGMS7GDWB6M6/deletion-mark.json bucket="tracing: thanos"

level=debug ts=2021-09-06T20:14:26.831705749Z caller=block.go:233 msg="deleted file" file=01FEXSZ2C583DD4KBBYKGGXYN1/meta.json bucket="tracing: thanos"

level=debug ts=2021-09-06T20:14:27.078772537Z caller=block.go:276 msg="deleted file" file=01FEXSZ2C583DD4KBBYKGGXYN1/index bucket="tracing: thanos"

level=debug ts=2021-09-06T20:14:27.53999946Z caller=block.go:276 msg="deleted file" file=01FEXSZ2C583DD4KBBYKGGXYN1/chunks/000001 bucket="tracing: thanos"

level=debug ts=2021-09-06T20:14:27.891457189Z caller=block.go:256 msg="deleted file" file=01FEXSZ2C583DD4KBBYKGGXYN1/deletion-mark.json bucket="tracing: thanos"

level=debug ts=2021-09-06T20:14:28.128874173Z caller=block.go:233 msg="deleted file" file=01FEXC7KW5DXRM7QRBREP9YWH2/meta.json bucket="tracing: thanos"

level=debug ts=2021-09-06T20:14:28.546011691Z caller=block.go:276 msg="deleted file" file=01FEXC7KW5DXRM7QRBREP9YWH2/index bucket="tracing: thanos"

level=debug ts=2021-09-06T20:14:28.797824494Z caller=block.go:276 msg="deleted file" file=01FEXC7KW5DXRM7QRBREP9YWH2/chunks/000001 bucket="tracing: thanos"

level=debug ts=2021-09-06T20:14:29.127959785Z caller=block.go:256 msg="deleted file" file=01FEXC7KW5DXRM7QRBREP9YWH2/deletion-mark.json bucket="tracing: thanos"

level=debug ts=2021-09-06T20:14:29.128101253Z caller=fetcher.go:321 component=block.BaseFetcher msg="fetching meta data" concurrency=32

level=debug ts=2021-09-06T20:15:18.7626315Z caller=fetcher.go:321 component=block.BaseFetcher msg="fetching meta data" concurrency=32

level=debug ts=2021-09-06T20:16:18.763021022Z caller=fetcher.go:321 component=block.BaseFetcher msg="fetching meta data" concurrency=32

level=debug ts=2021-09-06T20:17:18.762911952Z caller=fetcher.go:321 component=block.BaseFetcher msg="fetching meta data" concurrency=32

level=debug ts=2021-09-06T20:18:18.763002026Z caller=fetcher.go:321 component=block.BaseFetcher msg="fetching meta data" concurrency=32

level=debug ts=2021-09-06T20:19:17.56787997Z caller=fetcher.go:321 component=block.BaseFetcher msg="fetching meta data" concurrency=32

level=debug ts=2021-09-06T20:19:18.763591536Z caller=fetcher.go:321 component=block.BaseFetcher msg="fetching meta data" concurrency=32

level=debug ts=2021-09-06T20:19:19.936175284Z caller=fetcher.go:321 component=block.BaseFetcher msg="fetching meta data" concurrency=32

Then I don't think your compactor is running with the flags you gave. Looks like deduplication is disabled.

@yasharne
Copy link
Author

@yeya24 compactor is running with the given deduplication flags:
screenshot_17

@jnauska
Copy link

jnauska commented Sep 22, 2021

Replica label should be without ""

evel=info ts=2021-09-22T05:13:01.945232765Z caller=compact.go:241 msg="deduplication.replica-label specified, enabling vertical compaction" dedupReplicaLabels=replica
level=info ts=2021-09-22T05:13:01.945344035Z caller=compact.go:246 msg="vertical compaction is enabled" compact.enable-vertical-compaction=true

....
level=debug ts=2021-09-22T05:13:03.689823716Z caller=fetcher.go:717 msg="replica label removed" label=replica
level=debug ts=2021-09-22T05:13:03.689871928Z caller=fetcher.go:717 msg="replica label removed" label=replica
level=debug ts=2021-09-22T05:13:03.689897056Z caller=fetcher.go:717 msg="replica label removed" label=replica
level=debug ts=2021-09-22T05:13:03.689905195Z caller=fetcher.go:717 msg="replica label removed" label=replica
....

@rsommer rsommer mentioned this issue Sep 22, 2021
Closed
@rsommer
Copy link

rsommer commented Sep 22, 2021

While running only compact, retention and deletion does not work for us, too. If I apply retention and cleanup as standalone commands, everything works as expected. The same goes with downsampling (see #4531).

@yeya24
Copy link
Contributor

yeya24 commented Sep 22, 2021

While running only compact, retention and deletion does not work for us, too. If I apply retention and cleanup as standalone commands, everything works as expected. The same goes with downsampling (see #4531).

As I mentioned, it is not a bug. This means that the compactor has too much compaction work to do so that it never starts retention and downsampling because these steps are executed after all compaction work is done.

Try scaling your compactor more and increase resources to keep up with the compaction work.

@yeya24
Copy link
Contributor

yeya24 commented Oct 2, 2021

Replica label should be without ""

evel=info ts=2021-09-22T05:13:01.945232765Z caller=compact.go:241 msg="deduplication.replica-label specified, enabling vertical compaction" dedupReplicaLabels=replica
level=info ts=2021-09-22T05:13:01.945344035Z caller=compact.go:246 msg="vertical compaction is enabled" compact.enable-vertical-compaction=true

....
level=debug ts=2021-09-22T05:13:03.689823716Z caller=fetcher.go:717 msg="replica label removed" label=replica
level=debug ts=2021-09-22T05:13:03.689871928Z caller=fetcher.go:717 msg="replica label removed" label=replica
level=debug ts=2021-09-22T05:13:03.689897056Z caller=fetcher.go:717 msg="replica label removed" label=replica
level=debug ts=2021-09-22T05:13:03.689905195Z caller=fetcher.go:717 msg="replica label removed" label=replica
....

Seems that's the root cause. Thanks @jnauska. @yasharne Does removing double quotes work for you?

@yeya24
Copy link
Contributor

yeya24 commented Oct 3, 2021

I will close this issue as this is a configuration issue mentioned by @jnauska

@yeya24 yeya24 closed this as completed Oct 3, 2021
@JayChanggithub JayChanggithub mentioned this issue Feb 16, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
component: compact needs-more-info
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@rsommer @yasharne @yeya24 @jnauska