[FEAT]: Evaluate llama.cpp as alternative to Ollama for improved performance

📝 Description

Investigate and evaluate whether implementing llama.cpp provides better performance compared to the current Ollama-based language model integration in FireForm.

💡 Rationale

The current implementation uses Ollama for language model inference. llama.cpp is a lighter-weight alternative that may offer improved latency, throughput, or resource efficiency. Evaluating this could enhance FireForm's performance for emergency department documentation workflows.

🛠️ Proposed Solution

A prototype or proof-of-concept implementation comparing both approaches:

Implement equivalent functionality using llama.cpp
Create reproducible benchmarks for both Ollama and llama.cpp
Compare resource usage (CPU, memory, inference time)
Document findings and performance metrics

✅ Acceptance Criteria

How will we know this is finished?

Benchmark tests exist for both Ollama and llama.cpp under identical conditions.
Performance results documented with test environment, model configuration, and workload details.
Recommendation provided on whether to adopt llama.cpp based on performance gains vs. integration effort.
Findings documented in docs/ or as a technical summary.

📌 Additional Context

Add any performance requirements, specific models being tested, or references to performance optimization research here.

@marcvergees

Hi @marcvergees — I was looking at this as a potential first contribution and wanted to flag something in case it's useful. The good first issue label seems to sit a little at odds with the scope: the acceptance criteria describe a full comparative benchmark (reproducible tests for both runtimes, CPU/memory/inference profiling, and an adoption recommendation), and it's also filed under the 16-week GSoC track — which reads more like a substantial project than a newcomer task.

Would it help to pin down the benchmark parameters up front — which models, what hardware/workload, and what performance gain would justify adopting llama.cpp? Right now, two contributors could produce quite different numbers and neither would clearly be 'done.'

Happy to help however is useful. Thanks!

@marcvergees

Hi @marcvergees, I’d be interested in working on #702 as my first contribution to FireForm.

Is the issue still available for someone to work on?

@Abhishekpandey123-git

Hey @Abhishekpandey123-git and @ElDShell, this issue is not trivial and it might take a long time for you guys if you're new to this community.

There was a real concern that Ollama was wasting a lot of time that maybe llama.cpp could solve it faster in terms of the response time. That's why we thought that re-implementing everything with a provider different than Ollama would be useful so that we could check if there was a real time performance gain.

As you might have known the way we've been testing our pipeline is been through benchmarking, and these benchmark allowed us to check first of all how fast it was, and how accurate our pipeline was.

At the end of the day, our picture that we have on our mind regarding this issue is that we are gonna create a provider, which is gonna be completely agnostic of the service we're asking the information to, that means the LLM basically. So that we will be able to change this part of information without changing the whole architecture. This is gonna be probably done in a separate repository.

In my opinion, and that's why I removed the good first issue tag, that's not an initial good first issue because it not just requires coding, but also you need guys to understand the whole story of FireForm, the business goals and the utility of the benchmark.

Next days we will be posting new good first issues, so that maybe is more appropiate for you guys.

Thanks,
Marc

Thanks for the clear explanation, that makes sense. A provider-agnostic layer in a separate repo sounds like the right direction. I'll keep an eye out for the new good first issues you mentioned and would be glad to pick one up.

Thanks for the explanation, Marc.

I’ll keep an eye out for the upcoming issues and look forward to contributing.

@Abhishekpandey123-git

Hi @Abhishekpandey123-git and @ElDShell, since it looks like you're both guys new here, maybe you could hop in our discord channel to keep updated of what's going on. We have online weekly meetings too where we discuss these kind of strategical things. It might be interesting for you if you prefer business-goals issues.

Marc

@marcvergees

Hi @Abhishekpandey123-git and @ElDShell, since it looks like you're both guys new here, maybe you could hop in our discord channel to keep updated of what's going on. We have online weekly meetings too where we discuss these kind of strategical things. It might be interesting for you if you prefer business-goals issues.

Marc

Hi @marcvergees, I tried to join the Discord using the invite link in the main FireForm page, but it says that the invite is invalid. I just wanted to let you know in case the link needs to be updated.

@ElDShell

Hi @ElDShell, thanks for letting us know. Already changed it.

@marcvergees

Hi @marcvergees — I just tried joining the Discord to catch up on the weekly meetings, and it looks like the invite link is showing as invalid/expired again (Discord defaults to 7-day temporary invites unless "Expire After" is set to Never when generating the link). Could you drop a non-expiring link when you get a chance? Thanks!

@Abhishekpandey123-git

@Abhishekpandey123-git thanks, done!

Progress update — llama.cpp evaluation

I've implemented and tested the provider/benchmark proof of concept on branch feat/llama-cpp-provider in my fork: https://github.com/Abhishekpandey123-git/FireForm/tree/feat/llama-cpp-provider

Implementation and checks

Added the llama.cpp provider path and benchmark diagnostics for comparing it with the existing Ollama path.
Local repository checks passed: Ruff clean and 187 pytest tests passed.
Built and ran llama-server on a Colab Tesla T4 using Qwen2.5-1.5B-Instruct-Q4_K_M.gguf.
A small generation smoke test reached about 56 tokens/s (not a full-workload throughput result).

Initial extraction benchmark

Fixture	Raw accuracy	Populated-only accuracy	Missing	Unsupported	Request time
`ics_206_1`	87.55%	88.69%	7	6	~7.6 s
`ics_203_1`	13.89%	16.36%	53	0	~8.8 s

These are preliminary results from one run per fixture, so they should not be treated as a definitive Ollama-vs-llama.cpp comparison. The LLM judge was unavailable in this environment, so the figures above are raw benchmark accuracy, not judge-scored results.

Next steps

Restore/rebuild the llama.cpp server after the Colab runtime reset and rerun both fixtures.
Run Ollama and llama.cpp against the same model, prompts, fixtures, and runtime conditions.
Capture comparable latency, throughput, CPU and memory metrics, then document the recommendation.

The current evidence confirms the llama.cpp path runs, but a controlled side-by-side benchmark is still needed before recommending a production switch.

Note

The implementation and benchmark results are currently under investigation by AI-assisted analysis and me. I’m continuing to validate the findings, investigate performance bottlenecks, and run further tests to ensure the results are accurate and reproducible. Please review the current progress and share any feedback or suggestions. I’ll update this issue as the investigation progresses.

@Abhishekpandey123-git

Hi @Abhishekpandey123-git, would you think you could integrate it with the current LLMProvider class we have?

Furthermore, it'd be interesting to have a frontend option to select llama.cpp from the providers list, you can check it out in the tab settings of the current frontend. The idea would be implementing one more item in the selectable list that is "llama.cpp" and then select the model.

This is a big implementation.

What do you think about it @vharkins1, @chetanr25 and @jansans04?

@marcvergees

Hi @marcvergees,

I reviewed the current provider architecture and noticed that PR #725 added support for OpenAI-compatible endpoints, including llama.cpp.

I'd like to continue working on #702 by focusing on a reproducible Ollama vs. llama.cpp benchmark using equivalent model configurations and FireForm workloads, measuring latency, throughput, CPU/memory usage, and form-filling accuracy.

Before I proceed, could you confirm whether you would prefer the benchmark tooling and results in this repository or in a separate repository? Are there specific models or hardware configurations you would like us to prioritize?

Thanks!

@Abhishekpandey123-git

Hi @Abhishekpandey123-git, it's nice to have you here as a new contributor of FireForm, I would invite you to hop on our discord channel so that you can keep engaged with our community discussions in our weekly meetings.

Your issue has been discussed in today's meeting. Currently, there's an implementation that it's gonna be merged of both frontend and backend regarding support for open AI compatible end points, including llama.cpp.

Definitely as you were pointing out, this new feature is gonna open us to other providers like llama.cpp. That's why we want you to invite to work on that, the implementation should have a front and back features. The new merged stuff has a specifically tab for setting providers in which we should find llama.cpp in the left column. Let me know if you have any questions regarding your implmentation of both front and back (you can create issue for frontend too and we'll assign it to you).

Furthermore, since your work should be adapted to what @chetanr25 developed, maybe you can ping him too if you have any doubt.

Marc

@chetanr25

Thanks Marc — I appreciate the opportunity. I’ll coordinate with @chetanr25 and work from the current provider registry rather than the closed proof of concept. I’m treating the backend provider/config/tests and frontend selector as a coordinated change, and I’ll keep the benchmark report separate so it only contains measured results. Chetan, I’d appreciate your guidance on the preferred provider ID/config contract, especially how you want the explicit llama_cpp provider to interact with the generic custom provider and the in-progress configuration API.

Comparative benchmark status — blocked by hardware constraints

The llama.cpp provider changes and benchmark tooling have been submitted for review in PR #751. Local backend validation passed (420 tests; Ruff passed), but my current hardware does not have sufficient resources to run a fair Ollama-versus-llama.cpp comparison.

I have not produced latency, throughput, CPU, or memory measurements, and I will not infer or fabricate performance results. The comparative performance experiment remains blocked / not measured. Could a maintainer or contributor with suitable hardware run the documented workloads, or point me to an available CI/benchmark runner? I can then help capture and interpret the output and update the report.

Navigation Menu

Repository navigation

[FEAT]: Evaluate llama.cpp as alternative to Ollama for improved performance #702

📝 Description

💡 Rationale

🛠️ Proposed Solution

✅ Acceptance Criteria

📌 Additional Context

Progress update — llama.cpp evaluation

Implementation and checks

Initial extraction benchmark

Next steps

Note

Comparative benchmark status — blocked by hardware constraints

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[FEAT]: Evaluate llama.cpp as alternative to Ollama for improved performance #702

Description

📝 Description

💡 Rationale

🛠️ Proposed Solution

✅ Acceptance Criteria

📌 Additional Context

Activity

Abhishekpandey123-git commented on Sep 30, 2026

ElDShell commented on Oct 1, 2026

marcvergees commented on Oct 2, 2026

Abhishekpandey123-git commented on Oct 2, 2026

ElDShell commented on Oct 2, 2026

marcvergees commented on Oct 4, 2026

ElDShell commented on Oct 7, 2026

marcvergees commented on Oct 12, 2026

Abhishekpandey123-git commented on Oct 14, 2026

marcvergees commented on Oct 14, 2026

Abhishekpandey123-git commented on Oct 18, 2026

Progress update — llama.cpp evaluation

Implementation and checks

Initial extraction benchmark

Next steps

Note

marcvergees commented on Oct 24, 2026

Abhishekpandey123-git commented on Oct 26, 2026

marcvergees commented on Oct 26, 2026

Abhishekpandey123-git commented on Oct 27, 2026

Abhishekpandey123-git commented on Oct 27, 2026

Comparative benchmark status — blocked by hardware constraints

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions