Ripgrep interactive mode is extremely slow

[hahuang65]

• Category
  • Question
  • Bug
  • Suggestion
• OS
  • Linux
  • macOS
  • Windows
  • Etc.
• Vim
  • Vim
  • Neovim

Using

map <leader>/ :Rg<Cr>

command! -bang -nargs=* Rg
  \ call fzf#vim#grep(
  \   'rg --column --line-number --no-heading --hidden --fixed-strings --follow --ignore-case --no-ignore --glob "!.git/*" --color=always '.shellescape(<q-args>), 1,
  \   <bang>0 ? fzf#vim#with_preview({'options': '--delimiter : --nth 4..'}, 'up:60%')
  \           : fzf#vim#with_preview({'options': '--delimiter : --nth 4..'}, 'right:50%:hidden', '?'),
  \   <bang>0)

vs.

map <leader>/ :Ag<Cr>

has some interesting issues:

1. Using :Rg <term> <CR> is LIGHTNING fast.
2. Using <leader>/ to use :Rg in interactive mode is 10 times slower than :Ag in interactive mode.

Is there any reason why interactive mode for ripgrep is much slower than the_silver_searcher? Is it the way I configured :Rg?

[hahuang65]

Attaching a small gif that shows what I mean. First I do a straight :Rg search which is fast enough. Then I show the comparison between the interactive modes of :Rg and :Ag
The :Ag runs shows results and scans them MUCH faster than :Rg.

[junegunn]

No idea what's going on there. Make sure to install the latest version of ripgrep (0.7.1).

[hahuang65]

Yeah, I've got the latest installed. Do you have any tips for me to debug this, or is this something I just need to live with?

Is this something I need to file with ripgrep?

[junegunn]

Since it's not an issue of fzf, I can't really help. If ag works great, why don't you just use it? It's sufficiently fast for most use cases. By the way, I'm not sure starting ag or rg without any keyword is a good strategy since they are much more efficient for searching inside files and fzf will have to need a lot of memory to keep every line in memory.

These are the binding I use with ag.

https://github.com/junegunn/dotfiles/blob/255248256762bd8b1f997da8c786db57975979cb/vimrc#L1646

[hahuang65]

Yeah, I'll do that. Thanks for taking a look at this. Appreciate all your work!

[alok]

@hahuang65 The color=always may have something to do with it, because I get instant results with color=never and slow ones with color always.

[hahuang65]

Oh good tip. I'll try that. Thanks

[unphased]

So I wrote up this whole thing but i got ahead of myself so i'm rewriting my comment.

It looks like ag and rg have quite different methods of outputting content, and rg is (for now) much slower at output (I/O). rg is quite more speedy at doing actual searching however. But the way you're using this is such that hundreds of thousands of lines (no doubt something like nearly every nonempty line getting matched and sent into FZF for fuzzy matching) are produced, so rg happens to be sluggish in comparison to ag and grep. I've noticed this.

It's pretty clear that this is what happens (and it may not be obvious): FZF is adept at accepting megabytes upon megabytes of buffer to let you rapidly fuzzy search over. Indeed, it's so fast that the bottleneck of rg's output I/O is what's become conspicuous here. That's it. The discrepancy observed here is it takes rg a few seconds to get through assembling what is likely to be hundreds of MB of content whereas ag can do it 5 times faster or so. If you already have a starting point of some token to search for, you would be very well served to provide it as an arg to :Rg or such (which in almost all cases will narrow you down to a few K of content to flip through). If not, ag or grep is currently a faster tool for use with non-searching brute force workflow.

As nice as it is that FZF can consume these buffers pretty much as fast as they can be generated, it's probably overkill in most cases, but very powerful! It's simply another testament to the greatness of this software.

[hahuang65]

Thanks for doing the investigation and explaining it to me. It really helps!

[BurntSushi]

@unphased Can you provide a reproducible test case outside of fzf and contribute it to ripgrep so that we can fix it? A bug was already filed but it lacks details: BurntSushi/ripgrep#696

[unphased]

Further investigation indicates that rg output is never slow, but nevertheless, in vim-fzf, and only when outputting color, the performance is reduced. The reason is unclear at this time.

The quick workaround for OP and others troubled by the behavior is to configure an rg command for fzf (when used in vim) without color.