Kernel BUG in `cas_cache` 26.03.2 on Linux 6.18.33 during cached read-hit completion: `ocf_alock_unlock_one_rd`

[bitactive]

Summary

We are seeing repeatable kernel crashes with Open CAS Linux 26.03.2 on Linux 6.18.33 under a high-throughput nginx fileserver workload.

The crash is always inside the CAS kernel module, in:

• cas_cache/src/ocf/utils/utils_alock.c:319
• cas_cache/src/ocf/utils/utils_alock.c:368

Both crashes happen from the cached read-hit completion path:

_ocf_read_generic_hit_complete
ocf_req_unlock
ocf_alock_unlock_one_rd

The machine reboots via kdump. This does not look like mdraid, XFS, nginx, NIC, or userspace failure. The direct failure appears to be an OCF alock state invariant violation: CAS is trying to unlock/transfer a read lock while the cache-line access state is already WR (0xff).

Environment

• OS/userland: RHEL 9 based
• Kernel: Linux 6.18.33
• Open CAS Linux: 26.03.2.0000.release
• CAS kernel module: 26.03.2.0000.release
• OCF submodule in release: d54f2fc9938cc9aa57f69c06cb26084a5322248c
• Kernel config:
  • CONFIG_PREEMPT_NONE=y
  • CONFIG_PREEMPT_NONE_BUILD=y
  • CONFIG_HZ_1000=y
  • CONFIG_HZ=1000
  • CONFIG_NO_HZ_IDLE=y
• Workload:
  • nginx fileserver
  • heavy read traffic
  • direct/AIO reads
  • high network throughput
• Storage:
  • HDD mdraid6 core device
  • NVMe CAS cache device
  • XFS on exported CAS core
• CAS mode:
  • write-around
  • cache line size: 64 KiB
  • promotion policy: nhit
  • sequential cutoff policy: always
  • sequential cutoff threshold: 16384 KiB
• CAS module parameters:
  • max_writeback_queue_size=65536
  • writeback_queue_unblock_size=60000
  • use_io_scheduler=1
  • unaligned_io=1
  • seq_cut_off_mb=1

Actual Result

The server crashed multiple times with the same CAS BUG signature.

Example 1:

kernel BUG at cas_cache/src/ocf/utils/utils_alock.c:368!
Oops: invalid opcode: 0000 [#1] SMP NOPTI
CPU: 58 PID: 0 Comm: swapper/58 Kdump: loaded Tainted: G OE 6.18.33 #2
RIP: 0010:ocf_alock_unlock_one_rd+0x2af/0x2c0 [cas_cache]

RAX: 00000000000000ff
...
Call Trace:
 <IRQ>
 ocf_req_unlock+0x62/0x100 [cas_cache]
 _ocf_read_generic_hit_complete+0x3c/0x60 [cas_cache]
 cas_bd_forward_end_callback+0x11/0x20 [cas_cache]
 md_end_clone_io+0x45/0x100
 blk_mq_end_request_batch+0xf1/0x4e0
 nvme_irq+0x86/0xa0 [nvme]

Example 2:

kernel BUG at cas_cache/src/ocf/utils/utils_alock.c:319!
Oops: invalid opcode: 0000 [#1] SMP NOPTI
CPU: 48 UID: 65534 PID: 67341 Comm: nginx Kdump: loaded Tainted: G OE 6.18.33 #2
RIP: 0010:ocf_alock_unlock_one_rd+0x2b7/0x2c0 [cas_cache]

RAX: 00000000000000ff
...
Call Trace:
 <IRQ>
 ocf_req_unlock+0x62/0x100 [cas_cache]
 _ocf_read_generic_hit_complete+0x3c/0x60 [cas_cache]
 cas_bd_forward_end_callback+0x11/0x20 [cas_cache]
 md_end_clone_io+0x45/0x100
 blk_mq_end_request_batch+0xf1/0x4e0
 nvme_irq+0x86/0xa0 [nvme]

 <TASK>
 xfs_file_dio_read+0xa1/0xf0 [xfs]
 xfs_file_read_iter+0xb3/0xd0 [xfs]
 aio_read+0xeb/0x1b0
 io_submit_one+0xce/0x360
 __x64_sys_io_submit+0x82/0x170

Why This Looks Like CAS/OCF Lock State Corruption

In OCF utils_alock.c from the 26.03.2 OCF submodule:

#define OCF_CACHE_LINE_ACCESS_WR    ((unsigned char)(-1))
#define OCF_CACHE_LINE_ACCESS_IDLE  0
#define OCF_CACHE_LINE_ACCESS_ONE_RD 1

Crash registers show:

RAX: 00000000000000ff

That matches OCF_CACHE_LINE_ACCESS_WR.

The failing assertions are:

ENV_BUG_ON(v == OCF_CACHE_LINE_ACCESS_WR);

in the read unlock/read-to-read transfer path.

So CAS appears to be completing a read-hit request and trying to release a read lock, but the alock entry is already in write-lock state.

Reproduction Pattern

The issue appeared after running Open CAS 26.03.2 on Linux 6.18.33 under production read load.

Observed crashes:

• First crash after long uptime under load
• Subsequent crashes within around 20 minutes after reboot under resumed load
• Same BUG signature each time

No manual reboot/shutdown command triggered these events; kdump captured vmcores.

Relevant Upstream Observation

Open CAS Linux v26.03.2 pins OCF:

d54f2fc9938cc9aa57f69c06cb26084a5322248c

OCF master currently contains later metadata/concurrency-related commits.

This one looks especially relevant:

7a524fe53f49dc2f1444834552cf9b170b73e0f8
metadata: Make collision updates atomic

Commit message:

Allow to accessing collision metadata for read without acquiring hash bucket lock.

It changes collision metadata next/prev updates from separate bitfield accesses to an atomic 64-bit entry update/read.

utils_alock.c itself appears unchanged between OCF d54f2fc and current OCF master, so the alock BUG may be a symptom of earlier metadata/collision state corruption rather than a bug directly inside the alock function.

v26.03.2 already includes:

50b7ee24fa4702bb70435563fd7002907cadc975
metadata: Prevent read hoisting in hash bucket lock

but does not include the later atomic collision metadata update commit.

Expected Result

CAS should not hit ENV_BUG_ON() in ocf_alock_unlock_one_rd() during normal cached read-hit completion under read-heavy production load.

Questions

1. Is this a known issue in Open CAS 26.03.2 / OCF d54f2fc?
2. Is 7a524fe5 metadata: Make collision updates atomic expected to fix this class of alock state corruption?
3. Are there other OCF/Open CAS commits after 26.03.2 that should be backported for Linux 6.18.x and high-concurrency read workloads?
4. Would you recommend testing current Open CAS master, or backporting only the metadata/collision fixes onto 26.03.2?

Additional Notes

• The kernel is built with CONFIG_PREEMPT_NONE=y, so this is not a PREEMPT_DYNAMIC/full-preempt runtime issue.
• The crash happens in IRQ completion from NVMe/mdraid completion path.
• The workload uses nginx AIO/direct reads, which may increase concurrency on cached read-hit completion.
• CAS reported no user-visible CAS errors before the BUG.

[robertbaldyga]

@bitactive Hi! Thank you for the detailed report.

1. Is this a known issue in Open CAS 26.03.2 / OCF d54f2fc?

No, it is not.

2. Is 7a524fe5 metadata: Make collision updates atomic expected to fix this class of alock state corruption?

Not really. This patch is intended to make 9c8dea15 metadata: Add ocf_metadata_is_hit_no_lock() possible. All the other paths in OCF code are protected with appropriate locks, so atomicity of collision metadata is not required.

3. Are there other OCF/Open CAS commits after 26.03.2 that should be backported for Linux 6.18.x and high-concurrency read workloads?

None I'm aware of.

4. Would you recommend testing current Open CAS master, or backporting only the metadata/collision fixes onto 26.03.2?

I'd consider current master branch unstable. So if you want to test it, I'd not recommend doing it on the actual production cluster. It contains substantial amount of changes, which were regression tested, but we did not run our full test suite on them yet.

I'm looking at this issue, but for now I have neither a plausible explanation nor a reproduction on my side.

A few questions that may help me pinpoint the issue:

• Is the moment this bug starts to reproduce correlated with the cache reaching near 100% occupancy? That would suggest potential issue with eviction, which is very likely, as lines on the freelist are not really subjects of that many potential race conditions. That would also explain why the issue starts to reproduce more frequently after the cache fills up.
• Can you show OpenCAS statistics just after loading the cache and just before the crash happens?
• Can you test it on a smaller cache (1GiB or even 100MiB) to see if it reproduces more frequently?
• Can you test it on 4k cache line size?

[bitactive]

@robertbaldyga Thank you for your reply.

• Is the moment this bug starts to reproduce correlated with the cache reaching near 100% occupancy? That would suggest potential issue with eviction, which is very likely, as lines on the freelist are not really subjects of that many potential race conditions. That would also explain why the issue starts to reproduce more frequently after the cache fills up.

I will verify this, but I am fairly confident that the answer is yes. The issue did not occur during the first day, when the cache was still mostly free.

• Can you show OpenCAS statistics just after loading the cache and just before the crash happens?

I will set up a cron job to collect OpenCAS statistics every minute, so we have the data from shortly before the crash.

• Can you test it on a smaller cache (1GiB or even 100MiB) to see if it reproduces more frequently?

I will also try to set up a smaller test environment and will share the results once available.

• Can you test it on 4k cache line size?

I will test the 4K cache line size as well and follow up with more details shortly.

[robertbaldyga]

@bitactive Any updates?

[robertbaldyga]

@bitactive Do you still observe this bug? Do you have any new data about it?
I'm currently preparing v26.03.3 version with kernel 7.0 support and few minor fixes. However, as I was not able to reproduce this specific issue in my environment, and I have no other reports that would allow me to pinpoint the root cause. I'm going to leave this issue open.

[bitactive]

Hi,

the bug was difficult to reproduce under a normal workload, but we collected memory dumps and analyzed them with GPT and Claude. Both identified the same root cause:

Kernel panic when 8-bit cache-line read-lock counter reaches the write-lock sentinel

Summary

Open CAS/OCF can panic when 255 concurrent readers acquire the same cache-line lock.

The cache-line lock uses one env_atomic8 value for two purposes:

• 0x00: unlocked
• 0x01–0xFE: reader count
• 0xFF: write-lock sentinel

The read-lock operation prevents incrementing only when the value is already 0xFF:

#define OCF_CACHE_LINE_ACCESS_WR ((unsigned char)(-1))

return !!env_atomic8_add_unless(access, 1,
        OCF_CACHE_LINE_ACCESS_WR);

Consequently, when 254 readers hold a line, the next reader is allowed to increment the counter:

0xFE → 0xFF

The state is now indistinguishable from a write lock. When a reader subsequently unlocks the line, OCF reaches:

ENV_BUG_ON(v == OCF_CACHE_LINE_ACCESS_WR);

and panics the kernel.

Affected versions observed

The same failure was observed with:

• Open CAS 26.03.2.0000.release, kernel 6.18.33
• Open CAS 26.03.3.0000.release, kernel 6.18.38

The defective change appears to have been introduced by OCF commit:

8e28976f5633d0a8ca2c9054e040636bb9c4b0d8
alock: Change atomic to atomic8

The commit message says that the change reduces the supported concurrent read locks to 254, but the implementation does not enforce that limit.

The older Open CAS 25.12.0.0997 implementation used a 32-bit env_atomic counter with INT_MAX as the write-lock sentinel and did not have this practical saturation problem.

Production evidence

Seven kernel panics occurred between 2026-06-01 and 2026-08-01.

Six occurred on Open CAS 26.03.2, and the latest occurred after upgrading to Open CAS 26.03.3.

All seven have:

• RIP in ocf_alock_unlock_one_rd
• RAX = 0x00000000000000ff in the actual panic register block
• The read-hit completion path from an NVMe interrupt
• The same OCF invariant failure
• Different CPUs and tasks
• Two independently built Open CAS modules and two kernel versions

Six dumps report:

kernel BUG at cas_cache/src/ocf/utils/utils_alock.c:368
RIP: ocf_alock_unlock_one_rd+0x2af/0x2c0 [cas_cache]
RAX: 00000000000000ff

One dump reports the adjacent assertion:

kernel BUG at cas_cache/src/ocf/utils/utils_alock.c:319
RIP: ocf_alock_unlock_one_rd+0x2b7/0x2c0 [cas_cache]
RAX: 00000000000000ff

The common call chain is:

nvme_irq
  → blk_mq_end_request_batch
  → md_end_clone_io
  → cas_bd_forward_end_callback
  → _ocf_read_generic_hit_complete
  → ocf_req_unlock
  → ocf_alock_unlock_one_rd
  → BUG

The two BUG locations are consistent with the same invalid state:

• Plain reader unlock
• Reader-to-reader waiter handoff

Disassembly of the deployed module confirms that the trapped value is the loaded 8-bit cache-line access counter and that the ud2 instruction corresponds to the v == 0xFF assertion.

Expected behavior

A cache line must never reach the write-lock sentinel through reader-count increments.

When 254 readers already hold the line, additional readers should enter the existing waiter path until an active reader releases its reference.

Actual behavior

Reader 255 successfully changes the counter from 0xFE to 0xFF. A subsequent read unlock treats that value as an impossible write-locked state and triggers ENV_BUG_ON.

Suggested fix

ocf_alock_trylock_entry_rd() should refuse both:

• OCF_CACHE_LINE_ACCESS_WR
• The maximum reader count immediately below it

A cmpxchg loop is one possible implementation:

#define OCF_CACHE_LINE_ACCESS_MAX_RD \
        ((unsigned char)(OCF_CACHE_LINE_ACCESS_WR - 1))

static inline bool ocf_alock_trylock_entry_rd(
        struct ocf_alock *alock, ocf_cache_line_t entry)
{
        env_atomic8 *access = &alock->access[entry];
        uint8_t v = env_atomic8_read(access);

        while (v != OCF_CACHE_LINE_ACCESS_WR &&
                        v != OCF_CACHE_LINE_ACCESS_MAX_RD) {
                uint8_t old = env_atomic8_cmpxchg(access, v, v + 1);

                if (old == v)
                        return true;

                v = old;
        }

        return false;
}

Using only the following is unsafe:

env_atomic8_add_unless(access, 1,
        OCF_CACHE_LINE_ACCESS_MAX_RD);

If the line is genuinely write-locked at 0xFF, that operation does not match the 0xFE exclusion value and could increment 0xFF → 0x00, destroying the active write lock.

Proposed regression test

A deterministic unit test should:

1. Acquire 254 read locks on one cache line.
2. Attempt to acquire reader 255.
3. Verify that reader 255 is queued rather than incrementing the counter.
4. Release an active reader.
5. Verify that the queued reader receives the lock correctly.
6. Verify that a real 0xFF write lock cannot be modified by a read-lock attempt.

On the unpatched implementation, acquiring reader 255 changes the counter to 0xFF, and the next reader unlock reproduces the kernel assertion.

Environment

• High-throughput video CDN workload, approximately 47 Gbit/s
• Write-around cache
• 64 KiB cache lines
• Cache device: RAID0 across two NVMe devices
• Cache size: approximately 14.3 TiB
• 72 logical CPUs

This workload can create hundreds of overlapping reads against a hot cache line.

[robertbaldyga]

Hi @bitactive ! Thank you, that's a really great analysis.

What made it so hard to spot by reviewing the code is the fact that ocf_alock_trylock_entry_rd already used env_atomic8_add_unless. It seemingly protected from the overflow, which turned out to be not the case at all, because it first compares and only then performs the addition.

I already prepared pyocf test reproducing this problem on OCF and a patch with fix: Open-CAS/ocf#968

[bitactive]

While validating the OCF PR #968 fix, we found a separate stop-status bug in OpenCAS 26.03.3; current master also appears affected.

casadm -T / casctl stop --flush returns KCAS_ERR_STOPPED_DIRTY even though:

• dirty blocks are zero;
• the flush and metadata save complete;
• the kernel logs Cache cache1 successfully stopped;
• the cache disappears from casadm -L and reloads correctly.

cache_priv->stop_context is allocated with non-zeroing env_malloc(). The real local flush_status is set in cache_mngt_exit_instance(), but is never copied to context->flush_status. Later, exit_instance_finish() reads that uninitialized field:

if (!ctx->error && ctx->flush_status)
    result = -KCAS_ERR_STOPPED_DIRTY;

The apparent fix is to assign the real result before stopping:

context->flush_status = flush_status;
status = _cache_mngt_cache_stop_sync(cache);

We reproduced the false error repeatedly while the cache stopped and reloaded cleanly. Could you check this too?

[robertbaldyga]

You are right. I just merged the fix to the master #1789

[robertbaldyga]

Both issues fixed, tested and available in a new minor release: https://github.com/Open-CAS/open-cas-linux/releases/tag/v26.03.4