Repository navigation
Conversation
|
Don't worry about 7.4 - we'll generally forward port commits to any active kernels. Claude (note the minors don't need fixing for this PR, but may be of interest): Review: staging: vchiq_core: Store a msg_queue entry before publishing itLooks correct to me. The ring has one producer (slot handler thread, via the vc-sm-cma callback) and one consumer (the vc-sm-cma io thread), so a release/acquire pair per index is enough, and each side reading its own index without a barrier is fine. Minor points:
|
|
Thanks for the review, and for the note on 7.4.
|
pelwell
left a comment
There was a problem hiding this comment.
I'm relieved to note that the bug didn't exist in the early VCHIQ code, it being added as part of a cleanup.
Approved, assuming the respin addresses those minor issues.
vchiq_msg_queue_push() advances msg_queue_write before it stores the header in the entry that the new index makes visible, and vchiq_msg_hold() advances msg_queue_read before it loads the header from the entry that the new index frees. Each side publishes its index before it has touched the entry. The thread on the other side can run in between: on one CPU when the first thread is preempted between the two statements, and on SMP at any time. The consumer can see the new write index and load the entry before the producer has stored it. It then gets the header left in that entry one lap (VCHIQ_MAX_SLOTS messages) earlier. That message was released long ago and its slot recycled, so the consumer parses whatever the VPU has written at that address since, and then releases it. The message that was actually pushed is never seen and never released, so the slot that holds it is never recycled. In the first lap the stale entry is NULL. With a full queue the producer can likewise see the advanced read index and overwrite an entry before the consumer has loaded it: one message is lost and the next one is delivered twice. vc-sm-cma, the queue's only user, logs the stale read as "vc_sm_cma_vchi_rx_ack: received response 1818324333, throw away..." (0x6c616d6d, the MMAL header magic of another service's message now at the stale address) or as a response of 0. On a Raspberry Pi 4 decoding short H.264 clips in a loop through bcm2835-codec, 14 messages were lost in 23.5 hours, about 35,500 decoder teardowns. Each lost message pins one of the VPU's transmit slots for the rest of the boot, and a pinned slot stays charged to the slot quota of every VPU service that has a message in it. At 14 pinned slots the VPU stopped sending on its MMAL and SMEM services at its next slot boundary. That matches vchiq's default per-service quota for 31 slots, 15: 14 pinned plus the one being written. From then on every attempt to create ril.video_decode failed with -ETIME until the next reboot. Every lost message that was a VC_SM_MSG_TYPE_RELEASED also left an imported dma-buf referenced for the rest of the boot. Store the entry before publishing the index, and load it before freeing the entry, with smp_store_release() for each index and smp_load_acquire() where the other side reads it. The queue has one producer (the slot handler thread, through the service callback) and one consumer (the service's own thread), so a release/acquire pair per index is all the ordering it needs, as in Documentation/core-api/circular-buffers.rst. No lock is required. The new order of the statements is what closes the race on one CPU. The barriers keep that order visible across CPUs. The comparison kernels carried a test-only delay between the two statements of vchiq_msg_queue_push(). Even at 0 that hook adds a load and a branch between them, which widens the race. With this patch, a kernel ran 3,001 decoder teardowns, half with the delay at 0 and half at 500 us, without losing a message. Built the same way without the patch, it lost hardware decode after about 600 teardowns, twice, with the delay at 0. The cause was found, and this patch written, with an LLM coding assistant (Claude Code), working from the log line above and the vchiq and vc-sm-cma sources. Fixes: 77b3f6c ("staging: vchiq: Move message queue into struct vchiq_service") Link: https://lore.kernel.org/r/20261116072339.2821648-1-jussi@firstview.fi Signed-off-by: Jussi Alanärä <jussi@firstview.fi>
60c9f6e to
4554d03
Compare
|
Respun with the two minor points addressed:
No code change. The reference leak and the dead wait loop will come separately. |
vc-sm-cma loses messages from the VPU because
vchiq_msg_queue_push()publishes the queue's write index before it stores the entry, andvchiq_msg_hold()frees an entry before it reads it. The symptom isvc_sm_cma_vchi_rx_ack: received response 1818324333, throw away...(orresponse 0).What each loss costs:
VC_SM_MSG_TYPE_RELEASED, it also leaves an imported dma-buf referenced.ril.video_decodecreate then fails with -62 until reboot.On a Pi 4B decoding short H.264 clips in a loop through bcm2835-codec, 14 messages were lost in 23.5 hours (about 35,500 decoder teardowns) on 6.18.50, with stock vchiq and vc-sm-cma.
The fix stores before it publishes and reads before it frees, with
smp_store_release()/smp_load_acquire()on each index. It is the same change as the mainline patch, with the file at its staging path:https://lore.kernel.org/r/20261116072339.2821648-1-jussi@firstview.fi
Why a PR here as well: the mainline patch has no stable tag, because no released mainline kernel has an in-tree caller of this queue (vc-sm-cma is only now in the maintainer's
drivers/next). So 6.18.y will not get it through stable, while rpi-6.18.y calls the queue today. If review on the list changes the patch, I will update this PR to match.Tested on a Pi 4B with 6.18.50 kernels built the same way, each carrying a test-only delay between the two statements of
vchiq_msg_queue_push():On the current rpi-6.18.y head it is build-tested only: arm64,
bcm2711_defconfig,CONFIG_WERROR=y,drivers/staging/vc04_services/. The test details and the tool-use statement are below the fold of the mainline posting.The same code is on rpi-7.4.y at
drivers/platform/raspberrypi/vchiq-interface/, where the mainline patch applies with a line offset. I can open a second PR for that branch if you want one.