Skip to content

Commit

Permalink
[gitea] fix: Elasticsearch: Request Entity Too Large go-gitea#28117 (g…
Browse files Browse the repository at this point in the history
…o-gitea#29062)

Fix for gitea putting everything into one request without batching and
sending it to Elasticsearch for indexing as issued in go-gitea#28117

This issue occured in large repositories while Gitea tries to
index the code using ElasticSearch.

I've applied necessary changes that takes batch length from below config
(app.ini)
```
[queue.code_indexer]
BATCH_LENGTH=<length_int>
```
and batches all requests to Elasticsearch in chunks as configured in the
above config

(cherry picked from commit 5c0fc90)
  • Loading branch information
inferno-umar authored and earl-warren committed Feb 10, 2024
1 parent 75937b5 commit 00370f1
Showing 1 changed file with 11 additions and 5 deletions.
16 changes: 11 additions & 5 deletions modules/indexer/code/elasticsearch/elasticsearch.go
Original file line number Diff line number Diff line change
Expand Up @@ -180,11 +180,17 @@ func (b *Indexer) Index(ctx context.Context, repo *repo_model.Repository, sha st
}

if len(reqs) > 0 {
_, err := b.inner.Client.Bulk().
Index(b.inner.VersionedIndexName()).
Add(reqs...).
Do(ctx)
return err
esBatchSize := 50

for i := 0; i < len(reqs); i += esBatchSize {
_, err := b.inner.Client.Bulk().
Index(b.inner.VersionedIndexName()).
Add(reqs[i:min(i+esBatchSize, len(reqs))]...).
Do(ctx)
if err != nil {
return err
}
}
}
return nil
}
Expand Down

0 comments on commit 00370f1

Please sign in to comment.