The Linux Audit Subsystem

Auditing on Linux is a kernel feature, not a logging library. The kernel notices things — a syscall touching a watched path, a BPF program being loaded, a PAM session opening, a systemd service starting — and emits records describing them. Everything after that is a delivery problem, and the delivery problem is where the interesting failures live.

Events are not records

The first thing worth internalising, because every capacity calculation depends on it: one logical event is usually several records. A single watched-file access produces a SYSCALL record, a PROCTITLE record, often PATH and CWD too — all sharing one event serial number. A BPF program load produces SYSCALL + BPF + PROCTITLE.

Rule of thumb on a host like chiba: roughly 3 records per event. So a limit expressed in records is hit at about a third of the event count you'd naively expect, and anything that reasons about "how many events until we hit the ceiling" without that factor is off by 3×.

kauditd, and its two completely separate delivery paths

kauditd is a kernel thread. Its job is to take records off the queue and deliver them. It has two delivery mechanisms, and they are not alternatives to each other:

• Unicast, over netlink, to a single registered userspace daemon — auditd. The daemon registers itself with the kernel, and the kernel remembers its PID. This is the path the kernel considers the consumer.
• Multicast, to any number of passive subscribers. On a systemd host this is where systemd-journald-audit.socket attaches — that's how audit records end up in the journal tagged _TRANSPORT=audit.

Here is the sentence that matters more than anything else on this page:

Multicast does not replace the unicast consumer.

If no process has registered as auditd, the kernel treats delivery as failed — even while journald is happily receiving every single record over multicast. Journald being full of audit records is not evidence that the audit subsystem is healthy. It is evidence of exactly one thing: that the multicast hook is attached.

The three queues

There are three, and only one of them is visible in the obvious place:

• audit_queue — the main queue. This is what auditctl -s reports as backlog.
• audit_retry_queue
• audit_hold_queue — not shown by auditctl -s at all.

All three are checked against the same audit_backlog_limit. So backlog: 0 is a statement about the main queue only. It is entirely consistent with a hold queue sitting pinned at the limit, dropping every new record.

What the code actually does

From kernel/audit.c, kauditd_send_queue() — read against v6.18, the kernel this was diagnosed on, rather than recalled:

while ((skb != skb_tail) && (skb = skb_dequeue(queue))) {
    if (skb_hook)
        (*skb_hook)(skb);          /* multicast -> journald. ALWAYS, and first. */
    if (!sk) {                      /* no auditd registered */
        if (err_hook)
            (*err_hook)(skb, -ECONNREFUSED);
        continue;
    }
    ...

Two things fall out of that ordering, and both are load-bearing.

The multicast hook runs first, unconditionally. Before the if (!sk) check, before any error handling. Journald gets the record whether or not a unicast consumer exists. So a broken unicast path never blinds the journal-based pipeline.

With no auditd, sk is NULL and every record goes to err_hook — which is kauditd_hold_skb():

static void kauditd_hold_skb(struct sk_buff *skb, int error)
{
    kauditd_printk_skb(skb);        /* the "callbacks suppressed" spam */
    if (!audit_default)
        goto drop;
    if (error == -EAGAIN) { ... retry queue ... }
    if (!audit_backlog_limit ||
        skb_queue_len(&audit_hold_queue) < audit_backlog_limit) {
        skb_queue_tail(&audit_hold_queue, skb);
        return;
    }
    audit_log_lost("kauditd hold queue overflow");
drop:
    kfree_skb(skb);
}

The bucket, not the burst

Read that function as a state machine and the failure mode names itself.

Without auditd, the hold queue grows monotonically. Every record is held. Nothing ever drains it — draining is what the absent daemon was for. It grows until it reaches audit_backlog_limit, and from that instant every further record is lost, permanently, 1:1 with ordinary audit traffic.

That is a bucket filling up, not a burst overflowing. The distinction is the whole diagnostic value:

• A burst problem spikes and recovers. Spike, quiet, spike, quiet.
• A bucket problem is flat at exactly zero for hours — and then starts climbing and never stops.

If a loss counter has that second shape, no amount of raising the limit will fix it. A bigger bucket takes longer to fill. That is all it does.

The held records are pinned kernel memory. They're visible in /proc/slabinfo as skbuff_head_cache, and comparing its active count against audit_backlog_limit is a direct read of how full the invisible queue is.

The audit_default branch

audit_default is 1 when auditing was enabled via the kernel command line (audit=1) — which is what the NixOS security.audit module sets. That's the branch that reaches skb_queue_tail(&audit_hold_queue, skb).

Were it 0, records would be dropped rather than held — still printk-spammed into dmesg, but not accumulating pinned memory. Worth knowing which side of that branch a given host is on, because it decides whether "no consumer" costs you memory or just noise.

Where this actually stands

Budding. The delivery path is nailed down — read from the source, confirmed against a live host, and it explains a real incident end to end (that's the neighbouring note: The kauditd Hold Queue Incident).

What this note deliberately does not cover: audit rule syntax beyond the -w path -p perms -k key watches that prompted it, auditctl tuning past backlog_limit, ausearch/aureport, or how any of this differs on non-systemd distributions. Those are all real and none of them were needed to explain the failure, so they're not asserted here rather than half-remembered.