Skip to content

Issues: huggingface/lighteval

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

47 Open 40 Closed
47 Open 40 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Dataset loading issue for german_rag_evals on Windows
#211 opened Jul 4, 2024 by Pommel4711
20
Add evaluations for function calling
#210 opened Jul 3, 2024 by clefourrier
Shape mismatches in MMLU
#204 opened Jun 17, 2024 by marcobellagente93
1
Handling last token in tokenized_continuation
#203 opened Jun 15, 2024 by simon900314
2
Different results for gsm8k via lighteval compared to internal pipeline
#202 opened Jun 12, 2024 by Mugariya
quantized model not loading
#200 opened Jun 7, 2024 by rankofootball
1
6
The helm|piqa task is generative but has generation_size=-1.
#198 opened May 30, 2024 by yonatano
1
@clefourrier
3
Expose stop_sequence at command line
#196 opened May 27, 2024 by lewtun
An apparent bug in drop's dealing with multi-span answer
#195 opened May 21, 2024 by sadra-barikbin
3
LightevalTask.process_results() is not aligned with LightevalTask.get_request_type()
#193 opened May 20, 2024 by sadra-barikbin
1
@sadra-barikbin
4
apply_target_perplexity_metric pops only the first response
#192 opened May 20, 2024 by sadra-barikbin
Issue when saving results with fsspec==2023.12.1
#191 opened May 15, 2024 by Hamza-Alobeidli
1
Evaluate EncoderDecoderModels
#183 opened May 3, 2024 by Bachstelze
2
Performance compared to lm-evaluation-harness
#179 opened Apr 29, 2024 by geoalgo
6
Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0!
#176 opened Apr 26, 2024 by gody7334
4
Error: ModuleNotFoundError: No module named 'openai'.
#175 opened Apr 26, 2024 by PhilipMay
5
Add Sympy equivalence for MATH / GSM8K?
#170 opened Apr 23, 2024 by lewtun
1
Could not initialize the JudgeOpenAI model and openi import error bug Something isn't working
#166 opened Apr 22, 2024 by lewtun
@NathanHB
1
DROP Evaluation with Llama3 (vs. lm-evaluation-harness)
#165 opened Apr 21, 2024 by vipulraheja
1
Expose a few model predictions / gold answers in the logs
#164 opened Apr 21, 2024 by lewtun
1
MMLU evaluation fails with Mistral
#159 opened Apr 18, 2024 by sanchit-gandhi
6
add transformers model to be used as judge feature request New feature/request
#153 opened Apr 10, 2024 by NathanHB
2
LatexTableWriter created but never used.
#151 opened Apr 9, 2024 by PhilipMay
1
Add Code-Centric Interface to LightEval for Enhanced Usability
#148 opened Apr 7, 2024 by adithya-s-k
4
Add a nanotron model to the test suite
#144 opened Apr 4, 2024 by clefourrier
ProTip! Updated in the last three days: updated:>2024-07-13.