Code

Add Granite Vision Support#11794

Code

Merged

ngxson merged 40 commits intoggml-org:masterggml-org/llama.cpp:masterfrom alex-jw-brooks:granite_visionalex-jw-brooks/llama.cpp:granite_visionCopy head branch name to clipboardFeb 24, 2025

+235-42

Commits

Commits on Feb 24, 2025

• Add super wip scripts for multimodal granite gguf

  Show description for 7db1c51
  alex-jw-brooks

  committed
  7db1c51

  Copy full SHA for 7db1c51

• Add example for converting mmgranite to gguf

  Show description for f6fbfc2
  alex-jw-brooks

  committed
  f6fbfc2

  Copy full SHA for f6fbfc2

• remove hardcoded path

  Show description for 8b60107
  alex-jw-brooks

  committed
  8b60107

  Copy full SHA for 8b60107

• Add vision feature layer to gguf params

  Show description for 72c53dd
  alex-jw-brooks

  committed
  72c53dd

  Copy full SHA for 72c53dd

• Clean up llava surgery and remove name substitution hacks

  Show description for 4212f37
  alex-jw-brooks

  committed
  4212f37

  Copy full SHA for 4212f37

• Add transformers llava next tensor name mapping

  Show description for 77ce6f2
  alex-jw-brooks

  committed
  77ce6f2

  Copy full SHA for 77ce6f2

• Make siglip / openclip mutuall exclusive

  Show description for 3fc67a9
  alex-jw-brooks

  committed
  3fc67a9

  Copy full SHA for 3fc67a9

• Fix projector linear substitution

  Show description for e3c791c
  alex-jw-brooks

  committed
  e3c791c

  Copy full SHA for e3c791c

• Fix linear 2 substitution index

  Show description for 2d45e0d
  alex-jw-brooks

  committed
  2d45e0d

  Copy full SHA for 2d45e0d

• Increase max flattened gridpoints to 64

  Show description for 3e3eebc
  alex-jw-brooks

  committed
  3e3eebc

  Copy full SHA for 3e3eebc

• Fix hardcoded concat for multiple feature layers

  Show description for 1ceef1a
  alex-jw-brooks

  committed
  1ceef1a

  Copy full SHA for 1ceef1a

• Pull vision feature layers out of gguf keys

  Show description for c788a45
  alex-jw-brooks

  committed
  c788a45

  Copy full SHA for c788a45

• fix num gridpoints and use all layers

  Show description for 6725d6c
  alex-jw-brooks

  committed
  6725d6c

  Copy full SHA for 6725d6c

• Avoid dropping last image encoder layer in llava models

  Show description for 403575c
  alex-jw-brooks

  committed
  403575c

  Copy full SHA for 403575c

• Use 10 for max number of patches

  Show description for 095b836
  alex-jw-brooks

  committed
  095b836

  Copy full SHA for 095b836

• Standardize vision feature layers

  Show description for ff00515
  alex-jw-brooks

  committed
  ff00515

  Copy full SHA for ff00515

• Cleanup logs

  Show description for eceee7f
  alex-jw-brooks

  committed
  eceee7f

  Copy full SHA for eceee7f

• Update comment for vision feature layer init

  Show description for be204f0
  alex-jw-brooks

  committed
  be204f0

  Copy full SHA for be204f0

• Update notes for alternative to legacy llm conversion script

  Show description for 4588b90
  alex-jw-brooks

  committed
  4588b90

  Copy full SHA for 4588b90

• Fix notes rendering

  Show description for 264c2ca
  alex-jw-brooks

  committed
  264c2ca

  Copy full SHA for 264c2ca

• Add v prefix to vision feature layer log

  Show description for 86b43da
  alex-jw-brooks

  committed
  86b43da

  Copy full SHA for 86b43da

• Use current defaults for feature layer

  Show description for eb54540
  alex-jw-brooks

  committed
  eb54540

  Copy full SHA for eb54540

• Use constant for max gridpoints / feat layers, style fixes

  Show description for 243a899
  alex-jw-brooks

  committed
  243a899

  Copy full SHA for 243a899

• clarify non-negative feature layers

  Show description for ee2cf62
  alex-jw-brooks

  committed
  ee2cf62

  Copy full SHA for ee2cf62

• Remove CLIP_API from func signature

  Show description for b5735ba
  alex-jw-brooks

  committed
  b5735ba

  Copy full SHA for b5735ba

• USE MAX_IMAGE_FEATURE_LAYERS const in layer calc

  Show description for 3670d0e
  alex-jw-brooks

  committed
  3670d0e

  Copy full SHA for 3670d0e

• Clarify feature layers are non negative ints and not uint

  Show description for b973f37
  alex-jw-brooks

  committed
  b973f37

  Copy full SHA for b973f37

• Fix condition for reading feature layers

  Show description for 82e8852
  alex-jw-brooks

  committed
  82e8852

  Copy full SHA for 82e8852

• pop last llava layer when feature layers are unset

  Show description for ee6fb4d
  alex-jw-brooks

  committed
  ee6fb4d

  Copy full SHA for ee6fb4d

• Fix unset vision layer 0

  Show description for 07e7716
  alex-jw-brooks

  committed
  07e7716

  Copy full SHA for 07e7716

• Update examples/llava/clip.cpp

  Show description for 5f69fdb
  

  alex-jw-brooks

  and

  ngxson

  committed
  5f69fdb

  Copy full SHA for 5f69fdb

• Reenable assertion for out of bounds get_rows

  Show description for ab522d7
  alex-jw-brooks

  committed
  ab522d7

  Copy full SHA for ab522d7

• Use std vector for gridpoints and feature layers

  Show description for bb3e03a
  alex-jw-brooks

  committed
  bb3e03a

  Copy full SHA for bb3e03a

• Caculate max feature layer at load time

  Show description for 7bab305
  alex-jw-brooks

  committed
  7bab305

  Copy full SHA for 7bab305

• Include base patch for granite vision allocation

  Show description for 6557523
  alex-jw-brooks

  committed
  6557523

  Copy full SHA for 6557523

• Fix trailing whitespace

  Show description for cce01b8
  alex-jw-brooks

  committed
  cce01b8

  Copy full SHA for cce01b8

• Add max num patches = 10 back for minicpmv

  Show description for 188bfb0
  alex-jw-brooks

  committed
  188bfb0

  Copy full SHA for 188bfb0

• Use unordered set to store feature layers

  Show description for 8676316
  

  alex-jw-brooks

  and

  ngxson

  committed
  8676316

  Copy full SHA for 8676316

• Use max feature layer for postnorm

  Show description for 49c0863
  alex-jw-brooks

  committed
  49c0863

  Copy full SHA for 49c0863

• Apply suggestions from code review

  ngxson

  authored
  bec9ef1

  Copy full SHA for bec9ef1