Support for GGML Files of Falcon 40b?

[karrtikiyer]

As i understand for Falcon 40b, there is some change w.r.t ggml because of which this new fork has been created: https://github.com/cmp-nct/ggllm.cpp

I would like to know from the community if someone has tried using python bindings llama-cpp-python with falcon 40b GGML files from hugging face? https://huggingface.co/TheBloke/falcon-40b-instruct-GGML

If so, can you please share if any changes were needed to make this work?

Thanks in advance,

[matthoffner]

ctransformers just added support for this and it seems to be working. It's been my goto for any model that doesn't work in llama https://github.com/marella/ctransformers/tree/main

I've got a space setup using it based on the Falcon demo here: https://huggingface.co/spaces/matthoffner/falcon-mini

40b version: https://huggingface.co/spaces/matthoffner/falcon-fastapi

[karrtikiyer]

Thanks a lot Matt. This is very helpful.

On Sat, 24 Jun 2023 at 05:52, Matt Hoffner ***@***.***> wrote: ctransformers just added support for this and it seems to be working. It's been my goto for any model that doesn't work in llama https://github.com/marella/ctransformers/tree/main I've got some spaces setup based on their demo setup here: https://huggingface.co/spaces/matthoffner/falcon-mini — Reply to this email directly, view it on GitHub <#418 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABBMHMDGQNNZD3VIWMCJ2QTXMYXK3ANCNFSM6AAAAAAZRIZUFU> . You are receiving this because you authored the thread.Message ID: ***@***.*** com>

-- Thanks and regards, Karrtik

[linuxmagic-mp]

Well, I know it doesn't want to work out of the box with a qauntized version..

llama.cpp: loading model from ~/models/falcon-40b-ggml/ggml-model-qt_k_m.bin
error loading model: unknown (magic, version) combination: 67676363, 0000000a; is this really a GGML file?
llama_load_model_from_file: failed to load model

or a regular one..

llama.cpp: loading model from ~/models/falcon-40b-ggml/ggml-model--f32.bin
error loading model: llama.cpp: tensor '' should not be 1966080-dimensional
llama_load_model_from_file: failed to load model

[linuxmagic-mp]

Been a while, but thought I would poke back at this.. of course, I know I am running an unsupported model (Falcon) but just wanted to see where the status of the server was..

Update to the latest (0.1.77), and server (1.1.1)

Simple test..

 python3 -m llama_cpp.server --model /models/WizardLM-Uncensored-Falcon-40b/ggml-model-falcon-40b-wizardlm-qt_k5.bin

/usr/local/lib/python3.10/dist-packages/pydantic/_internal/_fields.py:127: UserWarning: Field "model_alias" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ('settings_',)`.
  warnings.warn(

llama.cpp: loading model from /models/WizardLM-Uncensored-Falcon-40b/ggml-model-falcon-40b-wizardlm-qt_k5.bin
error loading model: unknown (magic, version) combination: 67676363, 0000000a; is this really a GGML file?
llama_load_model_from_file: failed to load model
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.10/dist-packages/llama_cpp/server/__main__.py", line 46, in <module>
    app = create_app(settings=settings)
  File "/usr/local/lib/python3.10/dist-packages/llama_cpp/server/app.py", line 313, in create_app
    llama = llama_cpp.Llama(
  File "/usr/local/lib/python3.10/dist-packages/llama_cpp/llama.py", line 313, in __init__
    assert self.model is not None
AssertionError
Exception ignored in: <function Llama.__del__ at 0x7f57ce42d1b0>
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/llama_cpp/llama.py", line 1510, in __del__
    if self.ctx is not None:
AttributeError: 'Llama' object has no attribute 'ctx'

Just thought I would report this...

But as Matt said..
https://github.com/marella/ctransformers/tree/main
That is working.. only problem no server..

Managed to get https://github.com/ParisNeo/lollms-webui working with ctransformers, but yeah.. not out of the box, had to hack at a few things, and while he has done incredible work and it has a lot of bells and whistles.. TOO much automation ;0 A lot on his plate I am sure, so the smaller things are not the most important.. @cmp-nct has some plans in the future to look at the server again as well.

[RiEnClVi]

I wonder here too, the correct way seems to be to integrate:

https://github.com/cmp-nct/ggllm.cpp

I wonder if this is a target in the future?