Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Model] Update Molmo Eval to Match Official Implementation #648

Merged
merged 2 commits into from
Dec 30, 2024
Merged

[Model] Update Molmo Eval to Match Official Implementation #648

merged 2 commits into from
Dec 30, 2024

Conversation

jamespark3922
Copy link
Contributor

Changes:

  • Process up to max 36 crops per image instead of 12 crops in default setting.
  • Integrate prompts from the official Molmo implementation to match the reported numbers in the paper. (https://molmo.allenai.org/paper.pdf).

@jamespark3922
Copy link
Contributor Author

@kennymckormick Could you take a look at this PR?

@kennymckormick
Copy link
Member

Hi, @jamespark3922 ,
Will merge this PR after validating the implementation with 1 model.

@kennymckormick
Copy link
Member

  • TODO: Re-evaluate Molmo Series

@kennymckormick kennymckormick merged commit c08ab64 into open-compass:main Dec 30, 2024
1 check passed
@kennymckormick
Copy link
Member

Hi, @jamespark3922 ,
I find that when using this piece of codes for evaluation, the scores of Molmo-7B-O and Molmo-7B-D on MMVet dropped by around 10% (absolute). Would you please evaluate these two models on MMVet to double check?

kennymckormick pushed a commit to TobiasLee/VLMEvalKit that referenced this pull request Jan 1, 2025
@jamespark3922 @kennymckormick
[Model] Update Molmo Eval to Match Official Implementation (open-comp…
40bbc75 
…ass#648)

* add molmo prompts

* fix lint format
kennymckormick added a commit that referenced this pull request Jan 1, 2025
@TobiasLee @kennymckormick
@jamespark3922 @CMeteor @XuGW-Kevin
[Benchmark] Add VL-RewardBench (#703)
276d90a 
* update vlrewardbench

* pre-commit fix

* formatter

* [Improvement] Better `AUTO_SPLIT` and model split for InternVL2

* [Minor] Improve CC-OCR Import

* [Model] Support QVQ

* [Model] Update Molmo Eval to Match Official Implementation (#648)

* add molmo prompts

* fix lint format

* [Fix] Refine Qwen-VL2 device assignment

* [Fix] Fix RealWorldQA md5

* update MMMU_DEV_VAL tsv

* [Fix] Fix confusing image width&height (#704)

Co-authored-by: Yuan Ye <yuany2@chinatelecom.cn>

* Update llama_vision.py (#705)

* [Fix] Fix Lint

* Fix Lint

* Fix Lint

---------

Co-authored-by: kennymckormick <dhd.efz@gmail.com>
Co-authored-by: jamespark3922 <jspark96@cs.washington.edu>
Co-authored-by: CMeteor <CMeteor@users.noreply.github.com>
Co-authored-by: Yuan Ye <yuany2@chinatelecom.cn>
Co-authored-by: Guowei Xu <113534787+XuGW-Kevin@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kennymckormick kennymckormick kennymckormick approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@jamespark3922 @kennymckormick