Skip to content

Issues: open-compass/opencompass

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

132 Open 283 Closed
132 Open 283 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

评测结果的稳定性如何保证？
#1323 opened Jul 16, 2024 by ccp123456789
1 task
@liushz
1
[Feature]configs里支持gpt4o的eval配置
#1321 opened Jul 14, 2024 by ChrisXYF
1 task
@tonysy
[Feature] MBPP数据集的CN版本 mbpp_cn 能够release一下吗 ORZ，感谢大佬
#1320 opened Jul 14, 2024 by SrWYG
1 task
@tonysy
[Bug] AttributeError: 'str' object has no attribute 'shape'
#1319 opened Jul 12, 2024 by wenhuiwwh
2 tasks done
@tonysy
2
[Feature] Document Refine
#1317 opened Jul 12, 2024 by liushz
1 task
@liushz
[Bug] model attribute - max_out_len do not take effect
#1309 opened Jul 11, 2024 by jojo195
2 tasks done
1
运行评测结果为空[Bug]
#1304 opened Jul 11, 2024 by badmic
2 tasks done
1
[Bug] Qwen2-7b 模型 GPU 显存加载不均匀
#1301 opened Jul 10, 2024 by baymax591
2 tasks done
@MaiziXiao
3
[Bug] Alignbench无法使用VLLM模型评测，eval阶段卡住并报错
#1298 opened Jul 8, 2024 by IcyFeather233
2 tasks done
9
internlm2-7B-base CMMLU评测结果异常
#1281 opened Jul 1, 2024 by poisonwine
2 tasks done
@Leymore
[Bug] 多卡时，GPU7显存占用比其他卡多30G+
#1277 opened Jun 26, 2024 by dhcode-cpp
2 tasks done
@liushz
2
[Bug] flames的flames-scorer无法正确加载
#1275 opened Jun 25, 2024 by shiroko98
2 tasks done
@bittersweet1999
1
[Bug] qwen1.5-7B base 版本 在math测试集下得分仅有2.6分左右 远低于 官方评测给出的结果
#1274 opened Jun 25, 2024 by 1moye
2 tasks done
@bittersweet1999
1
[Bug] 按文档使用gpt3.5 测试数据集报错
#1270 opened Jun 24, 2024 by luckfu
2 tasks done
@Leymore
3
[Feature] Improve the Documentation for Subjective Evaluation
#1269 opened Jun 24, 2024 by tonysy
1 of 3 tasks
@Leymore
有人配置过mmlu_pro数据集么？求分享代码~
#1262 opened Jun 20, 2024 by wll-design
1 task
@Leymore @kennymckormick
[Bug] When I attempted to perform the agent evaluation, the console returned an error: "AttributeError: 'OpenAI' object has no attribute 'chat'".
#1259 opened Jun 20, 2024 by CaptainJi
2 tasks done
@kennymckormick
1
[Feature] 为啥我开始评测一直卡在这里
#1258 opened Jun 20, 2024 by GEK1
1 task
@kennymckormick
1
[Bug] Find scikit-learn version conflict in requirements/runtime.txt and requirements/extra.txt
#1256 opened Jun 19, 2024 by BIGWangYuDong
@liushz
1
[Feature] Cached Dataset load
#1254 opened Jun 18, 2024 by Chensem
1 task
@bittersweet1999
[Bug] llama3 8b 基座模型在ARC-C PPL数据集上的评估，accuracy只有41，不正常
#1253 opened Jun 18, 2024 by linboyang
2 tasks done
@bittersweet1999
[Bug] 大佬们，这个函数好像写的有问题，只能解析出来[BEGIN]到[DONE]中间的代码，然而基座模型最先输出的代码不是以[BEGIN]开头的。
#1251 opened Jun 17, 2024 by linboyang
2 tasks done
@Leymore
1
[Bug] 增加数据集时失败
#1245 opened Jun 15, 2024 by YanxingLiu
2 tasks done
1
meta-llama/Meta-Llama-3-8B-Instruct evaluated results is not consistent with hugging face's official results
#1243 opened Jun 13, 2024 by hzgdeerHo
2 tasks done
@kennymckormick
2
[Feature] Difficulty in Evaluating Custom Models with OpenCompass
#1239 opened Jun 13, 2024 by jiangjiadi
1 task
@kennymckormick
1
ProTip! no:milestone will show everything without a milestone.