Skip to content

Biased samping

Compare
Choose a tag to compare
@gsuuon gsuuon released this 10 Aug 22:08
· 116 commits to main since this release

This release adds the ability to modify logits before the sampling phase for each expression. You can now enforce that a field is only numbers, or that it's far more likely to be 'nun-chucks'.

Screenshot 2023-08-10 102252

The naive way of implementing this didn't add much accuracy to my desired inference outcomes - it turns out that we can't just encode our desired strings and modify the logits of those tokens, we have to encode them as they would appear in the existing (and upcoming) completion. I figured there were two main uses for modifying logits - one is to modify the odds of specific sequences, the other is to target specific character classes (eg. numbers). These require slightly different handling in terms of figuring out the desired tokens -- I'm not sure I got it right in this first pass, especially with character classes, but I'll keep improving it as I bump into bugs.

There's also an auto-generated docs site now: https://gsuuon.github.io/ad-llama

Hope it makes the API easier to understand!