Add image-text-to-text task guide #31777
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
Co-authored-by: Steven Liu <[email protected]>
Co-authored-by: Steven Liu <[email protected]>
Co-authored-by: Steven Liu <[email protected]>
Co-authored-by: Steven Liu <[email protected]>
Co-authored-by: Steven Liu <[email protected]>
Co-authored-by: Steven Liu <[email protected]>
Co-authored-by: Steven Liu <[email protected]>
Co-authored-by: Steven Liu <[email protected]>
Co-authored-by: Steven Liu <[email protected]>
Co-authored-by: Steven Liu <[email protected]>
Co-authored-by: Steven Liu <[email protected]>
| processor = AutoProcessor.from_pretrained("HuggingFaceM4/idefics2-8b") | ||
| ``` | ||
|
|
||
| This model has a [chat template](./chat_templating) format that's required for the input. Moreover, the model can also accept multiple images as input in a single conversation or message. We will now prepare the inputs. |
There was a problem hiding this comment.
This isn't quite right, we don't need to use a chat template for the model inputs. It's just useful to correctly format the prompt in the case of message-style inputs
There was a problem hiding this comment.
You likely know better, I thought when fine-tuning these chat templates are included in fine-tuning data, thus it is required to use chat templates no? e.g. Mistral one has <INST> </INST>
| img_urls =["https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/cats.png", | ||
| "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"] | ||
| images = [Image.open(requests.get(img_urls[0], stream=True).raw), | ||
| Image.open(requests.get(img_urls[1], stream=True).raw)] |
|
|
||
| acc_text = "" | ||
| for text_token in streamer: | ||
| time.sleep(0.04) |
There was a problem hiding this comment.
Why do we need to add this?
There was a problem hiding this comment.
otherwise the text flows super fast which is essentially against streaming (and also from my experience it was crashing too)
| target=model.generate, | ||
| kwargs=generation_args, | ||
| ) | ||
| thread.start() |
There was a problem hiding this comment.
Will this be safely closed?
| quantized_model = Idefics2ForConditionalGeneration.from_pretrained(model_id, device_map="cuda", quantization_config=quantization_config) | ||
| ``` | ||
|
|
||
| And that's it, we can use the model the same way with no changes. |
There was a problem hiding this comment.
It would be good here to note what kind of change this makes e.g. x% reduction in memory footprint
Co-authored-by: amyeroberts <[email protected]>
Co-authored-by: amyeroberts <[email protected]>
Co-authored-by: amyeroberts <[email protected]>
Co-authored-by: amyeroberts <[email protected]>
Co-authored-by: amyeroberts <[email protected]>
Co-authored-by: amyeroberts <[email protected]>
Added shortly
image-text-to-texttask guide that includes streaming and more