[{"data":1,"prerenderedAt":1205},["ShallowReactive",2],{"page-de-\u002Fgarden\u002Flinux-audit-subsystem":3,"backlinks-de-\u002Fgarden\u002Flinux-audit-subsystem":555},{"doc":4,"isFallback":544},{"id":5,"title":6,"body":7,"description":540,"draft":541,"extension":542,"meta":543,"navigation":544,"notice":545,"path":546,"seo":547,"stage":548,"stem":549,"tags":550,"topic":553,"__hash__":554},"garden_en\u002Flinux-audit-subsystem.md","The Linux Audit Subsystem",{"type":8,"value":9,"toc":530},"minimark",[10,14,23,28,65,78,82,92,123,126,134,144,148,151,183,194,198,209,274,277,287,306,391,395,398,413,416,431,434,455,463,479,486,490,500,526],[11,12,6],"h1",{"id":13},"the-linux-audit-subsystem",[15,16,17,18,22],"p",{},"Auditing on Linux is a kernel feature, not a logging library. The kernel notices things — a syscall touching a watched path, a BPF program being loaded, a PAM session opening, a systemd service starting — and emits ",[19,20,21],"em",{},"records"," describing them. Everything after that is a delivery problem, and the delivery problem is where the interesting failures live.",[24,25,27],"h2",{"id":26},"events-are-not-records","Events are not records",[15,29,30,31,35,36,38,39,43,44,47,48,51,52,55,56,58,59,58,62,64],{},"The first thing worth internalising, because every capacity calculation depends on it: one logical ",[32,33,34],"strong",{},"event"," is usually several ",[32,37,21],{},". A single watched-file access produces a ",[40,41,42],"code",{},"SYSCALL"," record, a ",[40,45,46],{},"PROCTITLE"," record, often ",[40,49,50],{},"PATH"," and ",[40,53,54],{},"CWD"," too — all sharing one event serial number. A BPF program load produces ",[40,57,42],{}," + ",[40,60,61],{},"BPF",[40,63,46],{},".",[15,66,67,68,73,74,77],{},"Rule of thumb on a host like ",[69,70,72],"a",{"href":71},"\u002Fgarden\u002Fkauditd-hold-queue-overflow","chiba",": roughly ",[32,75,76],{},"3 records per event",". So a limit expressed in records is hit at about a third of the event count you'd naively expect, and anything that reasons about \"how many events until we hit the ceiling\" without that factor is off by 3×.",[24,79,81],{"id":80},"kauditd-and-its-two-completely-separate-delivery-paths","kauditd, and its two completely separate delivery paths",[15,83,84,87,88,91],{},[40,85,86],{},"kauditd"," is a kernel thread. Its job is to take records off the queue and deliver them. It has ",[32,89,90],{},"two"," delivery mechanisms, and they are not alternatives to each other:",[93,94,95,110],"ul",{},[96,97,98,101,102,105,106,109],"li",{},[32,99,100],{},"Unicast, over netlink",", to a single registered userspace daemon — ",[40,103,104],{},"auditd",". The daemon registers itself with the kernel, and the kernel remembers its PID. This is the path the kernel considers ",[19,107,108],{},"the"," consumer.",[96,111,112,115,116,119,120,64],{},[32,113,114],{},"Multicast",", to any number of passive subscribers. On a systemd host this is where ",[40,117,118],{},"systemd-journald-audit.socket"," attaches — that's how audit records end up in the journal tagged ",[40,121,122],{},"_TRANSPORT=audit",[15,124,125],{},"Here is the sentence that matters more than anything else on this page:",[127,128,129],"blockquote",{},[15,130,131],{},[32,132,133],{},"Multicast does not replace the unicast consumer.",[15,135,136,137,139,140,143],{},"If no process has registered as ",[40,138,104],{},", the kernel treats delivery as ",[19,141,142],{},"failed"," — even while journald is happily receiving every single record over multicast. Journald being full of audit records is not evidence that the audit subsystem is healthy. It is evidence of exactly one thing: that the multicast hook is attached.",[24,145,147],{"id":146},"the-three-queues","The three queues",[15,149,150],{},"There are three, and only one of them is visible in the obvious place:",[93,152,153,166,171],{},[96,154,155,158,159,162,163,64],{},[40,156,157],{},"audit_queue"," — the main queue. This is what ",[40,160,161],{},"auditctl -s"," reports as ",[40,164,165],{},"backlog",[96,167,168],{},[40,169,170],{},"audit_retry_queue",[96,172,173,176,177],{},[40,174,175],{},"audit_hold_queue"," — ",[32,178,179,180,182],{},"not shown by ",[40,181,161],{}," at all.",[15,184,185,186,189,190,193],{},"All three are checked against the same ",[40,187,188],{},"audit_backlog_limit",". So ",[40,191,192],{},"backlog: 0"," is a statement about the main queue only. It is entirely consistent with a hold queue sitting pinned at the limit, dropping every new record.",[24,195,197],{"id":196},"what-the-code-actually-does","What the code actually does",[15,199,200,201,204,205,208],{},"From ",[40,202,203],{},"kernel\u002Faudit.c",", ",[40,206,207],{},"kauditd_send_queue()"," — read against v6.18, the kernel this was diagnosed on, rather than recalled:",[210,211,216],"pre",{"className":212,"code":213,"language":214,"meta":215,"style":215},"language-c shiki shiki-themes github-light github-dark","while ((skb != skb_tail) && (skb = skb_dequeue(queue))) {\n    if (skb_hook)\n        (*skb_hook)(skb);          \u002F* multicast -> journald. ALWAYS, and first. *\u002F\n    if (!sk) {                      \u002F* no auditd registered *\u002F\n        if (err_hook)\n            (*err_hook)(skb, -ECONNREFUSED);\n        continue;\n    }\n    ...\n","c","",[40,217,218,226,232,238,244,250,256,262,268],{"__ignoreMap":215},[219,220,223],"span",{"class":221,"line":222},"line",1,[219,224,225],{},"while ((skb != skb_tail) && (skb = skb_dequeue(queue))) {\n",[219,227,229],{"class":221,"line":228},2,[219,230,231],{},"    if (skb_hook)\n",[219,233,235],{"class":221,"line":234},3,[219,236,237],{},"        (*skb_hook)(skb);          \u002F* multicast -> journald. ALWAYS, and first. *\u002F\n",[219,239,241],{"class":221,"line":240},4,[219,242,243],{},"    if (!sk) {                      \u002F* no auditd registered *\u002F\n",[219,245,247],{"class":221,"line":246},5,[219,248,249],{},"        if (err_hook)\n",[219,251,253],{"class":221,"line":252},6,[219,254,255],{},"            (*err_hook)(skb, -ECONNREFUSED);\n",[219,257,259],{"class":221,"line":258},7,[219,260,261],{},"        continue;\n",[219,263,265],{"class":221,"line":264},8,[219,266,267],{},"    }\n",[219,269,271],{"class":221,"line":270},9,[219,272,273],{},"    ...\n",[15,275,276],{},"Two things fall out of that ordering, and both are load-bearing.",[15,278,279,282,283,286],{},[32,280,281],{},"The multicast hook runs first, unconditionally."," Before the ",[40,284,285],{},"if (!sk)"," check, before any error handling. Journald gets the record whether or not a unicast consumer exists. So a broken unicast path never blinds the journal-based pipeline.",[15,288,289,301,302,305],{},[32,290,291,292,204,294,297,298],{},"With no ",[40,293,104],{},[40,295,296],{},"sk"," is NULL and every record goes to ",[40,299,300],{},"err_hook"," — which is ",[40,303,304],{},"kauditd_hold_skb()",":",[210,307,309],{"className":212,"code":308,"language":214,"meta":215,"style":215},"static void kauditd_hold_skb(struct sk_buff *skb, int error)\n{\n    kauditd_printk_skb(skb);        \u002F* the \"callbacks suppressed\" spam *\u002F\n    if (!audit_default)\n        goto drop;\n    if (error == -EAGAIN) { ... retry queue ... }\n    if (!audit_backlog_limit ||\n        skb_queue_len(&audit_hold_queue) \u003C audit_backlog_limit) {\n        skb_queue_tail(&audit_hold_queue, skb);\n        return;\n    }\n    audit_log_lost(\"kauditd hold queue overflow\");\ndrop:\n    kfree_skb(skb);\n}\n",[40,310,311,316,321,326,331,336,341,346,351,356,362,367,373,379,385],{"__ignoreMap":215},[219,312,313],{"class":221,"line":222},[219,314,315],{},"static void kauditd_hold_skb(struct sk_buff *skb, int error)\n",[219,317,318],{"class":221,"line":228},[219,319,320],{},"{\n",[219,322,323],{"class":221,"line":234},[219,324,325],{},"    kauditd_printk_skb(skb);        \u002F* the \"callbacks suppressed\" spam *\u002F\n",[219,327,328],{"class":221,"line":240},[219,329,330],{},"    if (!audit_default)\n",[219,332,333],{"class":221,"line":246},[219,334,335],{},"        goto drop;\n",[219,337,338],{"class":221,"line":252},[219,339,340],{},"    if (error == -EAGAIN) { ... retry queue ... }\n",[219,342,343],{"class":221,"line":258},[219,344,345],{},"    if (!audit_backlog_limit ||\n",[219,347,348],{"class":221,"line":264},[219,349,350],{},"        skb_queue_len(&audit_hold_queue) \u003C audit_backlog_limit) {\n",[219,352,353],{"class":221,"line":270},[219,354,355],{},"        skb_queue_tail(&audit_hold_queue, skb);\n",[219,357,359],{"class":221,"line":358},10,[219,360,361],{},"        return;\n",[219,363,365],{"class":221,"line":364},11,[219,366,267],{},[219,368,370],{"class":221,"line":369},12,[219,371,372],{},"    audit_log_lost(\"kauditd hold queue overflow\");\n",[219,374,376],{"class":221,"line":375},13,[219,377,378],{},"drop:\n",[219,380,382],{"class":221,"line":381},14,[219,383,384],{},"    kfree_skb(skb);\n",[219,386,388],{"class":221,"line":387},15,[219,389,390],{},"}\n",[24,392,394],{"id":393},"the-bucket-not-the-burst","The bucket, not the burst",[15,396,397],{},"Read that function as a state machine and the failure mode names itself.",[15,399,400,406,407,409,410],{},[32,401,402,403,405],{},"Without ",[40,404,104],{},", the hold queue grows monotonically."," Every record is held. Nothing ever drains it — draining is what the absent daemon was for. It grows until it reaches ",[40,408,188],{},", and from that instant ",[32,411,412],{},"every further record is lost, permanently, 1:1 with ordinary audit traffic.",[15,414,415],{},"That is a bucket filling up, not a burst overflowing. The distinction is the whole diagnostic value:",[93,417,418,425],{},[96,419,420,421,424],{},"A ",[32,422,423],{},"burst"," problem spikes and recovers. Spike, quiet, spike, quiet.",[96,426,420,427,430],{},[32,428,429],{},"bucket"," problem is flat at exactly zero for hours — and then starts climbing and never stops.",[15,432,433],{},"If a loss counter has that second shape, no amount of raising the limit will fix it. A bigger bucket takes longer to fill. That is all it does.",[15,435,436,439,440,443,444,447,448,451,452,454],{},[32,437,438],{},"The held records are pinned kernel memory."," They're visible in ",[40,441,442],{},"\u002Fproc\u002Fslabinfo"," as ",[40,445,446],{},"skbuff_head_cache",", and comparing its ",[40,449,450],{},"active"," count against ",[40,453,188],{}," is a direct read of how full the invisible queue is.",[24,456,458,459,462],{"id":457},"the-audit_default-branch","The ",[40,460,461],{},"audit_default"," branch",[15,464,465,467,468,471,472,475,476,64],{},[40,466,461],{}," is 1 when auditing was enabled via the kernel command line (",[40,469,470],{},"audit=1",") — which is what the NixOS ",[40,473,474],{},"security.audit"," module sets. That's the branch that reaches ",[40,477,478],{},"skb_queue_tail(&audit_hold_queue, skb)",[15,480,481,482,485],{},"Were it 0, records would be ",[19,483,484],{},"dropped"," rather than held — still printk-spammed into dmesg, but not accumulating pinned memory. Worth knowing which side of that branch a given host is on, because it decides whether \"no consumer\" costs you memory or just noise.",[24,487,489],{"id":488},"where-this-actually-stands","Where this actually stands",[15,491,492,495,496,499],{},[32,493,494],{},"Budding."," The delivery path is nailed down — read from the source, confirmed against a live host, and it explains a real incident end to end (that's the neighbouring note: ",[69,497,498],{"href":71},"The kauditd Hold Queue Incident",").",[15,501,502,503,506,507,510,511,514,515,204,518,521,522,525],{},"What this note deliberately does ",[19,504,505],{},"not"," cover: audit rule syntax beyond the ",[40,508,509],{},"-w path -p perms -k key"," watches that prompted it, ",[40,512,513],{},"auditctl"," tuning past ",[40,516,517],{},"backlog_limit",[40,519,520],{},"ausearch","\u002F",[40,523,524],{},"aureport",", or how any of this differs on non-systemd distributions. Those are all real and none of them were needed to explain the failure, so they're not asserted here rather than half-remembered.",[527,528,529],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":215,"searchDepth":228,"depth":228,"links":531},[532,533,534,535,536,537,539],{"id":26,"depth":228,"text":27},{"id":80,"depth":228,"text":81},{"id":146,"depth":228,"text":147},{"id":196,"depth":228,"text":197},{"id":393,"depth":228,"text":394},{"id":457,"depth":228,"text":538},"The audit_default branch",{"id":488,"depth":228,"text":489},"What has to be true for an audit record to actually arrive — kauditd, its two delivery paths, and the queue nobody shows you.",false,"md",{},true,null,"\u002Fgarden\u002Flinux-audit-subsystem",{"title":6,"description":540},"budding","linux-audit-subsystem",[551,552],"infra","security","linux-audit","gG-WNw6YDJpV72Rx-6KBNxKf1eJClGO27WDnSeJY_7s",[556],{"id":557,"title":498,"body":558,"description":1198,"draft":541,"extension":542,"meta":1199,"navigation":544,"notice":545,"path":71,"seo":1200,"stage":548,"stem":1201,"tags":1202,"topic":553,"__hash__":1204},"garden_en\u002Fkauditd-hold-queue-overflow.md",{"type":8,"value":559,"toc":1178},[560,563,566,579,583,586,589,597,601,627,630,640,655,659,666,677,688,691,695,698,709,712,738,753,756,761,776,783,787,793,797,800,874,878,883,886,890,896,917,924,930,934,961,980,985,998,1006,1024,1039,1043,1046,1090,1109,1113,1119,1146,1149,1152,1154,1159,1176],[11,561,498],{"id":562},"the-kauditd-hold-queue-incident",[15,564,565],{},"This one took three rounds. Two of them produced fixes that looked like they worked, and neither of them was a fix. Writing it down mostly for the shape of the mistake, which is more portable than the specifics.",[15,567,568,569,571,572,575,576,578],{},"The mechanism underneath is in the neighbouring note — ",[69,570,6],{"href":546}," — and this story doesn't really parse without it. The one-line version: the kernel delivers audit records over ",[32,573,574],{},"two independent paths",", unicast to a registered ",[40,577,104],{}," and multicast to passive subscribers like journald, and the second does not substitute for the first.",[24,580,582],{"id":581},"the-host","The host",[15,584,585],{},"chiba: a headless Hetzner box running NixOS on kernel 6.18.45, btrfs root, impermanence (the root filesystem is wiped every boot). It runs the git forge, Matrix, mail and webmail.",[15,587,588],{},"Its intrusion detection uses five kernel audit watches — three tamper watches and two canary baits:",[210,590,595],{"className":591,"code":593,"language":594},[592],"language-text","-w \u002Froot\u002F.ssh                            -p wa -k chiba-tamper-ssh\n-w \u002Fhome\u002Fsynapse\u002F.ssh                    -p wa -k chiba-tamper-ssh\n-w \u002Fpersist\u002Fvar\u002Flib\u002Fsops-nix\u002Fkey.txt     -p wa -k chiba-tamper-agekey\n-w \u002Froot\u002F.aws\u002Fcredentials                -p r  -k chiba-canary-bait\n-w \u002Fhome\u002Fsynapse\u002F.aws\u002Fcredentials        -p r  -k chiba-canary-bait\n","text",[40,596,593],{"__ignoreMap":215},[24,598,600],{"id":599},"the-actual-bug-stated-up-front","The actual bug, stated up front",[15,602,603,604,607,608,611,612,615,616,619,620,622,623,626],{},"In NixOS, ",[40,605,606],{},"security.audit.enable"," turns on ",[32,609,610],{},"only the kernel side"," and loads the rules. The daemon is a ",[19,613,614],{},"separate"," option: ",[40,617,618],{},"security.auditd.enable",". It was never set. ",[40,621,161],{}," showed ",[40,624,625],{},"pid 0"," — no registered consumer, the entire time.",[15,628,629],{},"The reason nobody looked there is a single sentence in this project's own design documentation, which read roughly:",[127,631,632],{},[15,633,634,636,637,64],{},[40,635,118],{}," is active, so audit records reach journald ",[19,638,639],{},"without the userspace auditd daemon",[15,641,642,643,646,647,650,651,654],{},"That sentence is ",[32,644,645],{},"true",". It was read as ",[19,648,649],{},"\"therefore we don't need auditd\"",", which is ",[32,652,653],{},"false",". Journald subscribes over multicast; the kernel still wants a unicast consumer and treats its absence as a delivery failure. That one misreading cost three rounds of diagnosis.",[24,656,658],{"id":657},"round-1-the-cold-boot-burst-2026-08-21","Round 1 — the cold-boot burst (2026-08-21)",[15,660,661,662,665],{},"After a genuine cold start — the first in a while, since most deploys had been live ",[40,663,664],{},"switch","es rather than power cycles — every hardened systemd service loaded its seccomp-BPF filter at more or less the same moment. Each BPF program load is its own audit event. The burst blew straight through the then-current limit of 1024.",[15,667,668,669,672,673,676],{},"Symptoms: ",[40,670,671],{},"kauditd hold queue overflow",", thousands of suppressed callbacks, kauditd pegged, and the network stack unresponsive for minutes. From outside it looked like a hard crash — no ping, no SSH. But ",[40,674,675],{},"journalctl -b -1"," showed clean, orderly shutdowns. No panic.",[15,678,679,680,683,684,687],{},"Diagnosis: a simultaneous-start burst. Fix: raise ",[40,681,682],{},"backlogLimit",", ending at 16384 (8192 having been patched live into ",[40,685,686],{},"grub.cfg"," from Hetzner rescue mode as the emergency measure).",[15,689,690],{},"Plausible. Also wrong.",[24,692,694],{"id":693},"round-2-its-not-boot-specific-2026-08-22","Round 2 — it's not boot-specific (2026-08-22)",[15,696,697],{},"It came back: 216 overflow lines inside a single three-hour window, with no reboot in between.",[15,699,700,701,704,705,708],{},"Diagnosis this time: the BPF cascade isn't unique to boot. It fires on ",[32,702,703],{},"every PAM login session"," that spins up a fresh ",[40,706,707],{},"user@\u003Cuid>.service"," — systemd builds and tears down the whole per-user instance, cgroup-BPF programs included, per login, when the user has no lingering session to stay warm between connections.",[15,710,711],{},"Two accounts drove it:",[93,713,714,726],{},[96,715,716,721,722,725],{},[32,717,718],{},[40,719,720],{},"synapse"," — every individual ",[40,723,724],{},"ssh chiba '\u003Ccmd>'"," is its own login\u002Flogout cycle. Automation that opens a fresh connection per command rather than multiplexing generates a lot of these.",[96,727,728,733,734,737],{},[32,729,730],{},[40,731,732],{},"forgejo"," — and this is the one that scales badly. ",[19,735,736],{},"Every"," git-over-SSH operation rides system sshd as that one shared system user. So every contributor pushing or pulling against the forge multiplies the same PAM churn, not just the operator.",[15,739,740,741,744,745,204,747,51,750,752],{},"Fix: ",[40,742,743],{},"users.users.\u003Cname>.linger = true"," for ",[40,746,720],{},[40,748,749],{},"root",[40,751,732],{},", keeping the per-user instances resident instead of cycling per connection.",[15,754,755],{},"Also plausible. Also wrong — though linger stayed, because it removed real churn and is correct on its own terms.",[757,758,760],"h3",{"id":759},"the-instrumentation-that-was-the-bug-it-was-measuring","The instrumentation that was the bug it was measuring",[15,762,763,764,775],{},"Worth preserving for the sheer shape of it. The metric built in round 2, specifically to make this visible, was ",[32,765,766,767,770,771,774],{},"first implemented as a ",[40,768,769],{},"systemd.timer"," firing a ",[40,772,773],{},"Type=oneshot"," service every 30 seconds"," — which is to say, precisely the pattern that had just been diagnosed as the cause, running unconditionally and more often than the SSH traffic it was meant to observe.",[15,777,778,779,782],{},"It surfaced because paging continued after the linger deploy, and the overflow timestamps landed exactly on that service's start times. Rebuilt as a long-running process with a ",[40,780,781],{},"sleep"," loop.",[24,784,786],{"id":785},"round-3-the-real-one-2026-08-24","Round 3 — the real one (2026-08-24)",[15,788,789,790,792],{},"It came back again. Linger active. ",[40,791,682],{}," at 16384.",[757,794,796],{"id":795},"the-evidence","The evidence",[15,798,799],{},"The counter shape is what cracked it:",[93,801,802,816,831,839,853,868],{},[96,803,804,807,808,811,812,815],{},[40,805,806],{},"node_audit_lost_total"," sat at ",[32,809,810],{},"exactly 0 for 50 hours"," after the 08-22 boot, then began climbing at ",[32,813,814],{},"20:13:42"," and never stopped. No reboot in between.",[96,817,818,819,822,823,826,827,830],{},"The audit event serial at onset was ",[32,820,821],{},"~5580",". At ~3 records per event that's ",[32,824,825],{},"~16.7k records"," — crossing ",[40,828,829],{},"backlog_limit = 16384"," right about there.",[96,832,833,622,835,838],{},[40,834,442],{},[40,836,837],{},"skbuff_head_cache active = 17328"," against that same limit of 16384. The held records were directly visible as pinned kernel memory.",[96,840,841,842,845,846,849,850,64],{},"After onset, ",[40,843,844],{},"lost"," grew by ",[32,847,848],{},"18–19 per 10-minute check cycle",", and one cycle generates ~17 audit records. Loss running ",[32,851,852],{},"1:1 with every record produced",[96,854,855,857,858,861,862,865,866,182],{},[40,856,165],{}," read ",[32,859,860],{},"0"," the entire time — because that's the ",[19,863,864],{},"main"," queue, which drains fine. The hold queue isn't in ",[40,867,161],{},[96,869,870,871,873],{},"The host's only error-level logs in 24 hours: 137 lines, all 137 ",[40,872,671],{},". Otherwise completely healthy — zero failed units, disk 5%, RAM 2.7G of 15G, backups green, certificates valid.",[757,875,877],{"id":876},"the-signature-worth-memorising","The signature worth memorising",[15,879,880],{},[32,881,882],{},"A counter that is exactly zero for hours, then climbs monotonically and never stops, describes a bucket filling up — not a burst.",[15,884,885],{},"A burst problem is jagged: spike, recover, spike, recover. This was flat, then a knee, then a straight line. That shape rules out \"too much traffic at once\" and points at \"something that should be draining isn't.\"",[757,887,889],{"id":888},"why-both-earlier-fixes-could-never-have-held","Why both earlier fixes could never have held",[15,891,892,893],{},"They were ",[32,894,895],{},"time dials, not fixes.",[93,897,898,911],{},[96,899,900,901,903,904,907,908,64],{},"Raising ",[40,902,682],{}," makes the bucket bigger (and pins more kernel memory). It moved onset from ",[19,905,906],{},"minutes after boot"," to ",[19,909,910],{},"~50 hours after boot",[96,912,913,916],{},[40,914,915],{},"linger"," lowered the record rate. It moved onset the same way, for the same reason.",[15,918,919,920,923],{},"And ",[32,921,922],{},"every reboot reset the counter",", which made both look like they had worked.",[15,925,926,927],{},"That's the transferable lesson: ",[19,928,929],{},"if a fix only moves the moment the problem returns, it isn't a fix.",[24,931,933],{"id":932},"the-fix","The fix",[210,935,939],{"className":936,"code":937,"language":938,"meta":215,"style":215},"language-nix shiki shiki-themes github-light github-dark","security.auditd = {\n  enable = true;\n  settings.write_logs = false;\n};\n","nix",[40,940,941,946,951,956],{"__ignoreMap":215},[219,942,943],{"class":221,"line":222},[219,944,945],{},"security.auditd = {\n",[219,947,948],{"class":221,"line":228},[219,949,950],{},"  enable = true;\n",[219,952,953],{"class":221,"line":234},[219,954,955],{},"  settings.write_logs = false;\n",[219,957,958],{"class":221,"line":240},[219,959,960],{},"};\n",[15,962,963,965,966,969,970,972,973,976,977,979],{},[40,964,104],{}," registers as ",[40,967,968],{},"audit_pid"," on the unicast netlink socket. kauditd's send now succeeds, so ",[40,971,175],{}," ",[32,974,975],{},"drains"," instead of filling, the printk spam stops, and ",[40,978,844],{}," stands still.",[15,981,982,984],{},[40,983,682],{}," stays at 16384. With a consumer present it guards only the main queue against genuine simultaneous-start bursts — which is what it was always meant to do.",[15,986,987,989,990,993,994,997],{},[40,988,104],{}," was also added to the central unit registry so its failure pages ",[32,991,992],{},"immediately",". Otherwise its absence only announces itself ~2 days later as ",[40,995,996],{},"AuditRecordsLost",", long after the cause.",[757,999,1001,1002,1005],{"id":1000},"on-write_logs-no-a-trade-off-not-a-trick","On ",[40,1003,1004],{},"write_logs = no"," — a trade-off, not a trick",[15,1007,1008,1009,1012,1013,1016,1017,1020,1021,1023],{},"Deliberate ",[32,1010,1011],{},"for this host",", and not a general recommendation. journald plus Vector is already chiba's evidence path, and ",[40,1014,1015],{},"\u002Fvar\u002Flog"," lives in ",[40,1018,1019],{},"\u002Fpersist",", so a second on-disk copy of the same records would grow every backup while covering nothing that isn't already covered. Here, ",[40,1022,104],{}," is wanted purely as the queue consumer, not as a second store.",[15,1025,1026,1027,1030,1031,1034,1035,1038],{},"If you don't already have another durable log path, you want the real ",[40,1028,1029],{},"audit.log"," with rotation. (",[40,1032,1033],{},"write_logs"," is a genuine ",[40,1036,1037],{},"auditd.conf(5)"," keyword, checked against the shipped man page for audit 4.2.1.)",[24,1040,1042],{"id":1041},"verification-after-deploy","Verification after deploy",[15,1044,1045],{},"Measured live, before → after:",[93,1047,1048,1060,1068,1075,1081],{},[96,1049,1050,1052,1053,1056,1057],{},[40,1051,161],{}," → ",[40,1054,1055],{},"pid",": ",[32,1058,1059],{},"0 → 216431",[96,1061,1062,1056,1065],{},[40,1063,1064],{},"skbuff_head_cache active",[32,1066,1067],{},"17328 → 1147",[96,1069,1070,1056,1072],{},[40,1071,844],{},[32,1073,1074],{},"+18–19 per 10-minute cycle → frozen at 4802",[96,1076,1077,1078],{},"overflow lines: ",[32,1079,1080],{},"continuous → 0",[96,1082,1083,1056,1086,1089],{},[40,1084,1085],{},"\u002Fvar\u002Flog\u002Faudit",[32,1087,1088],{},"empty",", as intended",[15,1091,1092,1093,1095,1096,1098,1099,1101,1102,1105,1106,1108],{},"Then a 14-minute window at 60-second sampling: ",[40,1094,844],{}," flat across all 14 samples, ",[40,1097,165],{}," 0 throughout, ",[40,1100,446],{}," between 1147 and 1236 with no trend, zero overflow lines in every bucket — and ",[32,1103,1104],{},"319 audit records delivered to journald in that same window, of which 0 were lost."," Before the fix, ",[40,1107,844],{}," had been growing 1:1 with every record. That's the closing proof. The Alertmanager alert cleared itself.",[24,1110,1112],{"id":1111},"the-correction-that-matters-most","The correction that matters most",[15,1114,1115,1116,64],{},"This project's own alert text claimed that during an overflow, tamper watches and canary reads had ",[32,1117,1118],{},"\"NOT recorded\"",[15,1120,1121,1124,1125,1128,1129,1132,1133,1135,1136,1138,1139,1142,1143,64],{},[32,1122,1123],{},"That was wrong."," The multicast hook runs ",[19,1126,1127],{},"before"," the failing unicast send — that's the ",[40,1130,1131],{},"if (skb_hook)"," line above the ",[40,1134,285],{}," branch in ",[40,1137,207],{},". journald, and therefore the whole intrusion-detection pipeline, received ",[32,1140,1141],{},"every"," record throughout. Intrusion detection was ",[32,1144,1145],{},"never blind at any point",[15,1147,1148],{},"What was lost was exclusively the copy destined for a daemon that wasn't there.",[15,1150,1151],{},"So: this was a monitoring and memory problem, not a security hole. Worth stating plainly, because the alert text made it sound like the latter for two days, and an alert that overstates its own blast radius is its own kind of bug — it was retracted at every site.",[24,1153,489],{"id":488},[15,1155,1156,1158],{},[32,1157,494],{}," Closed, deployed, and verified against live measurement rather than reasoning. The three things worth carrying forward:",[1160,1161,1162,1165,1173],"ol",{},[96,1163,1164],{},"Multicast subscribers don't satisfy a kernel that wants a unicast consumer — and \"the records are showing up in the journal\" is not evidence that they aren't being lost.",[96,1166,1167,1169,1170,1172],{},[40,1168,161],{}," doesn't show the hold queue, so ",[40,1171,192],{}," proves less than it looks like it proves.",[96,1174,1175],{},"A fix that only moves the recurrence date isn't a fix — and a reboot that resets the counter will happily let you believe it was.",[527,1177,529],{},{"title":215,"searchDepth":228,"depth":228,"links":1179},[1180,1181,1182,1183,1186,1191,1195,1196,1197],{"id":581,"depth":228,"text":582},{"id":599,"depth":228,"text":600},{"id":657,"depth":228,"text":658},{"id":693,"depth":228,"text":694,"children":1184},[1185],{"id":759,"depth":234,"text":760},{"id":785,"depth":228,"text":786,"children":1187},[1188,1189,1190],{"id":795,"depth":234,"text":796},{"id":876,"depth":234,"text":877},{"id":888,"depth":234,"text":889},{"id":932,"depth":228,"text":933,"children":1192},[1193],{"id":1000,"depth":234,"text":1194},"On write_logs = no — a trade-off, not a trick",{"id":1041,"depth":228,"text":1042},{"id":1111,"depth":228,"text":1112},{"id":488,"depth":228,"text":489},"Three rounds of diagnosis, two plausible root causes that were both wrong, and the one counter shape that settled it.",{},{"title":498,"description":1198},"kauditd-hold-queue-overflow",[551,552,1203],"self-hosting","9qhAky0zs70XnEYq3vhPTe-NbfXeffYhuMN4MxEnN_M",1789414696475]