Specializing high-memory classes (or any other random things)

[devsnek]

I ran into an issue with the unicode \w using a lot of memory because it contains many char ranges. This becomes worse with repetition expansion as the entire class is repeated. After a couple hours of messing around I've come up with a pretty simple solution, simply adding a special instruction for \w (master...devsnek:regex:specialization). This code is very mess as I was just experimenting, but I am curious how folks feel about this approach and whether it should be used in this crate. I also know that @BurntSushi is working on rewrites of some internals, so I would want to wait for that before implementing this I think.

[BurntSushi]

Yeah like I was saying in chat, this approach doesn't work in the current or future architecture. It doesn't work in the current one because this only applies to one of the three NFAs compiled. And in the future architecture, it won't apply to any of the NFAs it compiles because the ability to match on char is completely removed. And this approach fundamentally requires matching on char.

[devsnek]

@BurntSushi i think this approach still works for bytes, you would just need multiple instructions for the specialization. The complexity of that approach is higher though. for the dfa and reverse cases, I think it could make sense for this optimization to just not apply (not that they are impossible but the effort doesn't seem worth it, at least to me...).

[BurntSushi]

I don't see how it works for bytes for the same reason it doesn't work for DFAs.

In the current architecture, it works for 1/3 NFAs. So it saves memory in just one of those NFAs. But the other two NFAs are still bloated by the word class, so the aavings doesn't really drop by that much, and just isn't worth doing in its current form.

In the future architecture, this optimization will apply to 0/2 NFAs.

The idea is good. But it needs a very different implementation and needs to be worked into the byte based NFA representation somehow for it to have a future.