How to load Llama 2 based 70B models with the llama_cpp.server?

[vmajor]

How do I load Llama 2 based 70B models with the llama_cpp.server? we need to declare n_gqa=8 but as far as I can tell llama_cpp.server takes no arguments.

[vmajor]

Never mind, I got it to give me 'usage' after trying to set it using a non-server flag. First time I tried something similar it just raised an error, did not give a list of acceptable llama_cpp.server runtime arguments, hence my original question. For reference here they are:

usage: __main__.py [-h] [--model MODEL] [--model_alias MODEL_ALIAS] [--n_ctx N_CTX]
                   [--n_gpu_layers N_GPU_LAYERS] [--tensor_split TENSOR_SPLIT]
                   [--rope_freq_base ROPE_FREQ_BASE] [--rope_freq_scale ROPE_FREQ_SCALE]
                   [--seed SEED] [--n_batch N_BATCH] [--n_threads N_THREADS]
                   [--f16_kv F16_KV] [--use_mlock USE_MLOCK] [--use_mmap USE_MMAP]
                   [--embedding EMBEDDING] [--low_vram LOW_VRAM]
                   [--last_n_tokens_size LAST_N_TOKENS_SIZE] [--logits_all LOGITS_ALL]
                   [--cache CACHE] [--cache_type CACHE_TYPE] [--cache_size CACHE_SIZE]
                   [--vocab_only VOCAB_ONLY] [--verbose VERBOSE] [--host HOST]
                   [--port PORT] [--interrupt_requests INTERRUPT_REQUESTS] [--n_gqa N_GQA]
                   [--rms_norm_eps RMS_NORM_EPS]

[vmajor]

Actually, no that does not work:

python -m llama_cpp.server --n_gqa 8
__main__.py: error: argument --n_gqa: invalid Optional value: '8'

MODEL is in path.

[pradhyumna85]

@vmajor, there is an open issue regarding the same - #528
For now you can pass n_gqa via environement variable

export N_GQA=8 && python -m llama_cpp.server