Allow users to generate texts longer than 1024 tokens

[minimaxir]

It likely isn't possible to do it at the generation level (like other frameworks), but we can hack it by:

1. Generate full text.
2. Feed latter half of previous text as a prefix.
3. Repeat until done.

[aletote]

Why half of the previous text and not all?

[yeldarby]

Would this be something you'd be willing to accept a PR on? I'd be willing to give it a go next week.

[rafaeelaudibert]

Is there some work on this? I would like to have this feature implemented

[woctezuma]

Why half of the previous text and not all?

I guess that the length is computed on the generated text, including the prefix.

If you feed the whole previous text as a prefix, then you would not be able to generate anything more if the length of the input is already at the max.

By feeding half of the previous text, you are guaranteed to have space(*) left for the rest of the generation process. This allows to circumvent the length constraint by iterating.

(*) at least half the length

[minimaxir]

I'm currently not working on this; if there's a PR, I'll merge it.

The hard part is that the 1024 limit is done at the tensor level; not sure what's necessary to shift it to handle it efficiently. (especially in the batch case)

[rafaeelaudibert]

Yeah, the batch case is where I fail to get it working, as I already am using a code with it working for only one batch, but can't figure it out how to properly and efficiently make it work for multiple batches, so I can't create a PR right now.

Yet, I'd be glad if someone would create a PR that solves this problem

[cedspam]

Why half of the previous text and not all?
you need some space left on the model sequence to generate your text, so it can be up to max len minus one token. it's a compromize between generation speed and generated text contextuality