Why there is no LlavaForSequenceClassification ?

[Zuhashaik]

Feature request

Need of LlavaForSequenceClassification class in 'src/transformers/models/llava/modeling_llava.py'..

Motivation

If there is a LlavaForSequenceClassification class we can use decoder only models for classification tasks.

Your contribution

Allow me to contribute to this as I am already working on this model for mental health meme classification for the upcoming AAAI conference.

[amyeroberts]

Hi @Zuhashaik, thanks for opening this issue!

We only add specific task heads for models if there are official checkpoints available; the task is described in the original paper or there's a large community demand.

[NielsRogge]

Note that you could technically also use LlavaForConditionalGeneration by training the model to only spit out one classification token per image.

[Zuhashaik]

Note that you could technically also use LlavaForConditionalGeneration by training the model to only spit out one classification token per image.

Yeah I've attached an score layer at the top of the model by passing those last layer hidden states. While doing this, some cuda related issues ("data in different devices cuda:0 and cuda:1") came up.

So I've changed my LlavaForConditionalGeneration class itself in modelling_llava.py file (added score layer) and used that for classification.

Thankyou.

[Zuhashaik]

I am facing lots of data type and quantization errors while defining classifier (score) separately please initiate LlavaForSequenceClassification and SequenceClassification for vision models..

Hi @Zuhashaik, thanks for opening this issue!

We only add specific task heads for models if there are official checkpoints available; the task is described in the original paper or there's a large community demand.

Thank you!