Add with_sequence for decode stream

[ArthurZucker]

No description provided.

[njhill]

Thank you for this @ArthurZucker!

What do you think about having a version of step that can take a sequence tokens? That could be used for prefilling and also for incrementing the stream with chunks of tokens when needed?

I'm also thinking through how this would be used in practice. For very long prompts, we ideally don't want to decode the whole thing since we would typically already have just tokenized the text prompt. But we need the last couple of prompt tokens to ensure we can continue the prompt text cleanly such that the concatenation of the first streamed string with the original prompt is exactly equal to all of the tokens being decoded together.

Perhaps that's up to the user of the API to sort out, but it might be nice for the prefilled tokens to be excluded from the subsequent step output (or at least have the option for that).

[ArthurZucker]

For sure! I am actually a lot less familiar than you about the actual use-cases! Super thankful for the feedback!
Indeed makes senses that you don't want it all. Was wondering if this is also compatible with batches in general or not, as each sample needs a stream with the current implementation