llama : add functions to get the model's metadata

[slaren]

These functions are meant to provide an easy way to read the model's metadata without needing to use one of the GGUF libraries.

• Arrays are excluded
• Also prints metadata values on load and adds bounds checks to some gguf functions

[KerfuffleV2]

I kind of have mixed feelings about this. I definitely agree that being able to access those values is sorely needed, but most of the time I'd really rather access it through a function like llama_n_vocab instead of having to know details about the GGUF file format.

[slaren]

In part, the motivation to add this is to be able to add the jinja prompt template metadata in the future, which will also require having a reasonable way to access it. I don't really see an issue with being able to enumerate metadata directly, and I am not sure that we should add specific functions to obtain metadata that is not directly relevant to the functions of llama.cpp.

[KerfuffleV2]

I don't really see an issue with being able to enumerate metadata directly

Sorry I wasn't clear. I wasn't saying there was an issue with adding it, just that I'd like to access metadata (that is relavant to the functions of llama.cpp) via functions like llama_n_vocab rather than having to know details about the GGUF format and what specific metadata keys are called.

Just for example, there's no way (as far as I know) to determine how many layers are in the currently loaded model.

[KerfuffleV2]

Hmm, ironically I actually need this now but it doesn't seem to add enough metadata access methods.

I can use llama_model_meta_count and llama_model_meta_key_by_index to get the keys but I don't have a way to find the value's type. Being able to fetch the value as a string is also pretty awkward and apparently I'll get "???" in my string result if the value happened to be an array. It would also be ambiguous whether the value is the string "1" or an integer value 1.

I think this kind of needs to work like the GGUF metadata functions in ggml except that it takes a const struct llama_model *. An alternative would be to nuke most of this and just add a function to access the struct gguf_context in the model. It's kind of unfortunate the functions in ggml take struct gguf_context* instead of struct gguf_kv * otherwise you wouldn't have to export the whole gguf_context.

I hate to say it but accessing non-string values as a string with the weird stuff like having to do escaping seems like it's not going to be useful most of the time.

[slaren]

Arrays are explicitly excluded from the metadata returned by these functions. You won't get a "???" if you try to read an array, you will get an error. I think you are misunderstanding the code, the string "???" is only returned with arrays of arrays, which is not supported by GGUF (or at least not by the C library). And even then, that is only relevant during in the info prints during loading.

The reason for returning only strings is for simplicity from the user perspective. If you need something more advanced, you are free to use ggml or gguf.py directly to read all the metadata. There is no point replicating the GGUF library in llama.cpp. The goal here is to provide a very simply API that will allow applications to access metadata such as the jinja templates.

[KerfuffleV2]

I think you are misunderstanding the code

You're correct, sorry, I did misread it.

There is no point replicating the GGUF library in llama.cpp.

I agree, but can't we at least make it so if someone already has a llama_context * or a llama_model * they can use the existing metadata functions from GGML? That would be as simple as adding some way to get the gguf_context *... Or from looking at the code, it looks like that doesn't get saved after the loading process? So I guess it would need to also be saved in the model or context structure.

[slaren]

I don't really think there is a good enough reason to do that. As you say, currently the gguf_context is released after loading, and this would force us to keep it in memory. Additionally, this would mean exposing internal details that may restrict our ability to make changes in the future. In contrast, this exposes an abstraction instead of the internals, and the implementation can be easily changed without changing the interface. The applications can easily do the same with just an additional call to gguf_init_from_file.

[KerfuffleV2]

The applications can easily do the same with just an additional call to gguf_init_from_file.

It seems incredibly strange to me that after we've loaded the file and all the metadata and returned a handle to the model that the only way for the user to access the metadata is to load the file a second time.

I don't really like the return the whole gguf_context idea either. The problem is the gguf_get_val_ functions expect a gguf_context instead of what they really need, just the KV stuff. If we are allowed to change that API, then I think there should be a struct containing the number of KVs and struct gguf_kv array and then accessor functions for it. Then llama.cpp could just wrap those with it's own get_val type functions, or save and allow accessing the KV structure so that the functions in GGML could be used on it.

But literally having to reload the file that was already loaded just makes absolutely no sense to me.

[slaren]

The cost of loading the metadata again from disk is not really significant, and it doesn't happen in a performance sensitive path, it happens during load time. Nobody really cares if loading takes 10ms more because the metadata needs to be loaded twice. So I don't think that this is something that needs to be optimized at this point. Never forget the first rule of optimization!

[KerfuffleV2]

What's this doing here then? Why aren't you taking your own suggestion to just gguf_init_from_file again?

All I'm saying is save this information in a format that's not only useful for Jinja templates or whatever. Where there's a way to reliably access it and something like 1 can be differentiated from "1".

---

Never forget the first rule of optimization!

And yes, yes, I know. "Premature optimization is the root of all evil." To be honest, I never forget it but pretty much always disregard it. My objection here isn't really about optimization, but more for a logical relatively intuitive API where you don't have to do weird stuff reload the same file from disk multiple times.

When you do that, there's also no way to even know for sure you reloaded the same file as the model that's in memory.

[slaren]

I am not trying to be snarky, but it would take a lot more code to re-load the gguf when using the metadata reading functions, and the only purpose would be to save a few bytes of memory (the reason arrays are excluded is precisely because it would waste too much memory for something of questionable utility). So doing that would be another case of premature optimization.

To be clear, I am not opposed to extending the API to support more types of metadata, that's why the function are called llama_model_meta_val_str and not just llama_model_meta_val, I expected that somebody may want to extend the API in the future. I am just not convinced that it is really worth the effort and complexity to do that because the expected use of these functions is to get very specific metadata like the jinja templates, which is trivial to convert to a number if needed, or just to dump a bunch of useless info to the user, in which case it doesn't matter what the type is. Making everything accesible as a string is simpler than having to check the correct function to call depending on the type. If this was a C++17 library I would be happy to make it return a std::variant, but that's not what it is.