Extend OpenAI finish_reason handling

[steven-solomon]

As part of #1979, I added:

• explicit handling for the finish_reason values: length, content_filter, and stop
• handlingfinish_reason for any unspecified cases
• log repr_doc_ids in all error cases to improve debugging

Fixes #1979

[MaartenGr]

Apologies for the delay, I left some comments here and there.

[steven-solomon]

@MaartenGr, I made the following updates based on your feedback:

• Renamed choice to output
• Introduce a logging prefix OpenAI Topic Representation
• Improve copy of logs based on your feedback
  • Modified references to doc_ids to use document IDs
  • Fixed typos

Open questions:

• Would you like me to grab the logger off of the topic_model?
• Should I defer any changes to the prompt?

[MaartenGr]

@steven-solomon Apologies for the delay, last few weeks were busy. It seems that there are some conflicts with the main branch. Other than that, I just ran the pipeline but expect this to run without any issues. Moreover, it seems there might be an issue with selecting the correct output variable.

[steven-solomon]

Moreover, it seems there might be an issue with selecting the correct output variable.

I think I see what you mean. The definition of output on L221 should be up another level so it is accessible in L243. Is that correct or is there something else amiss?

I am planning on adding the change to use the model's logger in the next day or two.

[MaartenGr]

@steven-solomon I think I see what you mean. The definition of output on L221 should be up another level so it is accessible in L243. Is that correct or is there something else amiss?

No, I meant that the output variable response on L242 and L240 are not doing anything:

                if self.exponential_backoff:
                    response = completions_with_backoff(self.client, model=self.model, prompt=prompt, **self.generator_kwargs)
                else:
                    response = self.client.completions.create(model=self.model, prompt=prompt, **self.generator_kwargs)

since they are not accessed on L243:

                label = output.text.strip()

[steven-solomon]

@MaartenGr I have made the switch the pull the logger from the topic_model parameter and added a fix for L243.

[MaartenGr]

@steven-solomon There is no topic_model.logger attribute so that wouldn't work, unfortunately. What was wrong with the previous approach?