{"id":720,"date":"2020-09-11T07:35:44","date_gmt":"2020-09-11T05:35:44","guid":{"rendered":"http:\/\/blog.pi3.com.pl\/?p=720"},"modified":"2020-09-11T07:35:44","modified_gmt":"2020-09-11T05:35:44","slug":"cve-2020-14356-2020-25220","status":"publish","type":"post","link":"https:\/\/blog.pi3.com.pl\/?p=720","title":{"rendered":"CVE: 2020-14356 & 2020-25220"},"content":{"rendered":"\n

The short story of 1 Linux Kernel Use-After-Free bug and 2 CVEs (CVE-2020-14356 and CVE-2020-25220)<\/span><\/strong><\/p>\n\n\n\n

Name:     Linux kernel Cgroup BPF Use-After-Free
Author:   Adam Zabrocki (pi3@pi3.com.pl<\/a>)
Date:       May 27, 2020<\/p>\n\n\n\n\n\n\n\n

First things first – short history:<\/span><\/strong><\/span><\/p>\n\n\n\n

In 2019 Tejun Heo discovered a racing problem with lifetime of the cgroup_bpf which could result in double-free and other memory corruptions. This bug was fixed in kernel 5.3. More information about the problem and the patch can be found here:<\/p>\n\n\n\n

https:\/\/lore.kernel.org\/patchwork\/patch\/1094080\/<\/a><\/p>\n\n\n\n

Roman Gushchin discovered another problem with the newly fixed code which could lead to use-after-free vulnerability. His report and fix can be found here:<\/p>\n\n\n\n

https:\/\/lore.kernel.org\/bpf\/20191227215034.3169624-1-guro@fb.com\/<\/a><\/p>\n\n\n\n

During the discussion on the fix, Alexei Starovoitov pointed out that walking through the cgroup hierarchy without holding cgroup_mutex might be dangerous:<\/p>\n\n\n\n

https:\/\/lore.kernel.org\/bpf\/20200104003523.rfte5rw6hbnncjes@ast-mbp\/<\/a><\/p>\n\n\n\n

However, Roman and Alexei concluded that it shouldn’t be a problem:<\/p>\n\n\n\n

https:\/\/lore.kernel.org\/bpf\/20200106220746.fm3hp3zynaiaqgly@ast-mbp\/<\/a><\/p>\n\n\n\n

Unfortunately, there is another Use-After-Free bug related to the Cgroup BPF release logic.<\/p>\n\n\n\n

The “new” bug – details (a lot of details ;-)):<\/span><\/strong><\/p>\n\n\n\n

During LKRG development and tests, one of my VMs was generating a kernel crash during shutdown procedure. This specific machine had the newest kernel at that time (5.7.x) and I compiled it with all debug information as well as SLAB DEBUG feature. When I analyzed the crash, it had nothing to do with LKRG. Later I confirmed that kernels without LKRG are always hitting that issue:<\/p>\n\n\n\n

      KERNEL: linux-5.7\/vmlinux\n    DUMPFILE: \/var\/crash\/202006161848\/dump.202006161848  [PARTIAL DUMP]\n        CPUS: 1\n        DATE: Tue Jun 16 18:47:40 2020\n      UPTIME: 14:09:24\nLOAD AVERAGE: 0.21, 0.37, 0.50\n       TASKS: 234\n    NODENAME: oi3\n     RELEASE: 5.7.0-g4\n     VERSION: #28 SMP PREEMPT Fri Jun 12 18:09:14 UTC 2020\n     MACHINE: x86_64  (3694 Mhz)\n      MEMORY: 8 GB\n       PANIC: \"Oops: 0000 [#1] PREEMPT SMP PTI\" (check log for details)\n         PID: 1060499\n     COMMAND: \"sshd\"\n        TASK: ffff9d8c36b33040  [THREAD_INFO: ffff9d8c36b33040]\n         CPU: 0\n       STATE:  (PANIC)\n\ncrash> bt\nPID: 1060499  TASK: ffff9d8c36b33040  CPU: 0   COMMAND: \"sshd\"\n #0 [ffffb0fc41b1f990] machine_kexec at ffffffff9404d22f\n #1 [ffffb0fc41b1f9d8] __crash_kexec at ffffffff941c19b8\n #2 [ffffb0fc41b1faa0] crash_kexec at ffffffff941c2b60\n #3 [ffffb0fc41b1fab0] oops_end at ffffffff94019d3e\n #4 [ffffb0fc41b1fad0] page_fault at ffffffff95c0104f\n    [exception RIP: __cgroup_bpf_run_filter_skb+401]\n    RIP: ffffffff9423e801  RSP: ffffb0fc41b1fb88  RFLAGS: 00010246\n    RAX: 0000000000000000  RBX: ffff9d8d56ae1ee0  RCX: 0000000000000028\n    RDX: 0000000000000000  RSI: ffff9d8e25c40b00  RDI: ffffffff9423e7f3\n    RBP: 0000000000000000   R8: 0000000000000000   R9: 0000000000000000\n    R10: 0000000000000003  R11: 0000000000000000  R12: 0000000000000000\n    R13: 0000000000000000  R14: 0000000000000000  R15: 0000000000000001\n    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018\n #5 [ffffb0fc41b1fbd0] ip_finish_output at ffffffff957d71b3\n #6 [ffffb0fc41b1fbf8] __ip_queue_xmit at ffffffff957d84e1\n #7 [ffffb0fc41b1fc50] __tcp_transmit_skb at ffffffff957f4b27\n #8 [ffffb0fc41b1fd58] tcp_write_xmit at ffffffff957f6579\n #9 [ffffb0fc41b1fdb8] __tcp_push_pending_frames at ffffffff957f737d\n#10 [ffffb0fc41b1fdd0] tcp_close at ffffffff957e6ec1\n#11 [ffffb0fc41b1fdf8] inet_release at ffffffff9581809f\n#12 [ffffb0fc41b1fe10] __sock_release at ffffffff95616848\n#13 [ffffb0fc41b1fe30] sock_close at ffffffff956168bc\n#14 [ffffb0fc41b1fe38] __fput at ffffffff942fd3cd\n#15 [ffffb0fc41b1fe78] task_work_run at ffffffff94148a4a\n#16 [ffffb0fc41b1fe98] do_exit at ffffffff9412b144\n#17 [ffffb0fc41b1ff08] do_group_exit at ffffffff9412b8ae\n#18 [ffffb0fc41b1ff30] __x64_sys_exit_group at ffffffff9412b92f\n#19 [ffffb0fc41b1ff38] do_syscall_64 at ffffffff940028d7\n#20 [ffffb0fc41b1ff50] entry_SYSCALL_64_after_hwframe at ffffffff95c0007c\n    RIP: 00007fe54ea30136  RSP: 00007fff33413468  RFLAGS: 00000202\n    RAX: ffffffffffffffda  RBX: 00007fff334134e0  RCX: 00007fe54ea30136\n    RDX: 00000000000000ff  RSI: 000000000000003c  RDI: 00000000000000ff\n    RBP: 00000000000000ff   R8: 00000000000000e7   R9: fffffffffffffdf0\n    R10: 000055a091a22d09  R11: 0000000000000202  R12: 000055a091d67f20\n    R13: 00007fe54ea5afa0  R14: 000055a091d7ef70  R15: 000055a091d70a20\n    ORIG_RAX: 00000000000000e7  CS: 0033  SS: 002b<\/code><\/pre>\n\n\n\n
1060499 <\/strong>is a sshd’s child:<\/p>\n\n\n\n
...\r\nroot        5462  0.0  0.0  12168  7276 ?        Ss   04:38   0:00 sshd: \/usr\/sbin\/sshd -D [listener] 0 of 10-100 startups\r\n...\r\nroot     1060499  0.0  0.1  13936  9056 ?        Ss   17:51   0:00  \\_ sshd: pi3 [priv]\r\npi3      1062463  0.0  0.0  13936  5852 ?        S    17:51   0:00      \\_ sshd: pi3@pts\/3\r\n...<\/span><\/span><\/span><\/code><\/pre>\n\n\n\n
Crash happens in function “__cgroup_bpf_run_filter_skb”, exactly in this piece of code:<\/p>\n\n\n\n
0xffffffff9423e7ee <__cgroup_bpf_run_filter_skb+382>: callq  0xffffffff94153cb0 <preempt_count_add>\n0xffffffff9423e7f3 <__cgroup_bpf_run_filter_skb+387>: callq  0xffffffff941925a0 <__rcu_read_lock>\n0xffffffff9423e7f8 <__cgroup_bpf_run_filter_skb+392>: mov 0x3e8(%rbp),%rax\n0xffffffff9423e7ff <__cgroup_bpf_run_filter_skb+399>: xor %ebp,%ebp\n0xffffffff9423e801 <__cgroup_bpf_run_filter_skb+401>: mov 0x10(%rax),%rdi\n                                                          ^^^^^^^^^^^^^^^\n0xffffffff9423e805 <__cgroup_bpf_run_filter_skb+405>: lea 0x10(%rax),%r14\n0xffffffff9423e809 <__cgroup_bpf_run_filter_skb+409>: test %rdi,%rdi<\/span><\/span><\/code><\/span><\/pre>\n\n\n\n
where RAX: 0000000000000000. However, when I was playing with repro under SLAB_DEBUG, I often got RAX: 6b6b6b6b6b6b6b6b:<\/p>\n\n\n\n
    [exception RIP: __cgroup_bpf_run_filter_skb+401]\n    RIP: ffffffff9123e801  RSP: ffffb136c16ffb88  RFLAGS: 00010246\n    RAX: 6b6b6b6b6b6b6b6b  RBX: ffff9ce3e5a0e0e0  RCX: 0000000000000028\n    RDX: 0000000000000000  RSI: ffff9ce3de26b280  RDI: ffffffff9123e7f3\n    RBP: 0000000000000000   R8: 0000000000000000   R9: 0000000000000000\n    R10: 0000000000000003  R11: 0000000000000000  R12: 0000000000000000\n    R13: 0000000000000000  R14: 0000000000000000  R15: 0000000000000001<\/code><\/pre>\n\n\n\n
So we have kind of a Use-After-Free bug. This bug is triggerable from user-mode. I’ve looked under IDA for the binary:<\/p>\n\n\n\n
.text:FFFFFFFF8123E7EE skb = rbx      ; sk_buff * ; PIC mode\n.text:FFFFFFFF8123E7EE type = r15     ; bpf_attach_type\n.text:FFFFFFFF8123E7EE save_sk = rsi  ; sock *\n.text:FFFFFFFF8123E7EE        call    near ptr preempt_count_add-0EAB43h\n.text:FFFFFFFF8123E7F3        call    near ptr __rcu_read_lock-0AC258h ; PIC mode\n.text:FFFFFFFF8123E7F8        mov     ret, [rbp+3E8h]\n.text:FFFFFFFF8123E7FF        xor     ebp, ebp\n.text:FFFFFFFF8123E801 _cn = rbp      ; u32\n.text:FFFFFFFF8123E801        mov     rdi, [ret+10h]  ; prog\n.text:FFFFFFFF8123E805        lea     r14, [ret+10h]<\/code><\/pre>\n\n\n\n
and this code is referencing cgroups from the socket. Source code:<\/p>\n\n\n\n
int __cgroup_bpf_run_filter_skb(struct sock *sk,\n\t\t\t\tstruct sk_buff *skb,\n\t\t\t\tenum bpf_attach_type type)\n{\n    ...\n\tstruct cgroup *cgrp;\n    ...
\n    ...\n\tcgrp = sock_cgroup_ptr(&sk->sk_cgrp_data);\n    ...\n\tif (type == BPF_CGROUP_INET_EGRESS) {\n\t\tret = BPF_PROG_CGROUP_INET_EGRESS_RUN_ARRAY(\n\t\t\tcgrp->bpf.effective[type], skb, __bpf_prog_run_save_cb);\n    ...\n    ...\n}<\/span><\/span><\/span><\/code><\/pre>\n\n\n\n
Debugger:<\/p>\n\n\n\n
crash> x\/4i 0xffffffff9423e7f8\n   0xffffffff9423e7f8:  mov    0x3e8(%rbp),%rax\n   0xffffffff9423e7ff:  xor    %ebp,%ebp\n   0xffffffff9423e801:  mov    0x10(%rax),%rdi\n   0xffffffff9423e805:  lea    0x10(%rax),%r14\ncrash> p\/x (int)&((struct cgroup*)0)->bpf\n$2 = 0x3e0\ncrash> ptype struct cgroup_bpf\ntype = struct cgroup_bpf {\n    struct bpf_prog_array *effective[28];\n    struct list_head progs[28];\n    u32 flags[28];\n    struct bpf_prog_array *inactive;\n    struct percpu_ref refcnt;\n    struct work_struct release_work;\n}\ncrash> print\/a sizeof(struct bpf_prog_array)\n$3 = 0x10\ncrash> print\/a ((struct sk_buff *)0xffff9ce3e5a0e0e0)->sk\n$4 = 0xffff9ce3de26b280\ncrash> print\/a ((struct sock *)0xffff9ce3de26b280)->sk_cgrp_data\n$5 = {\n  {\n    {\n      is_data = 0x0,\n      padding = 0x68,\n      prioidx = 0xe241,\n      classid = 0xffff9ce3\n    },\n    val = 0xffff9ce3e2416800\n  }\n}<\/code><\/pre>\n\n\n\n
We also know that R15: 0000000000000001 == type == BPF_CGROUP_INET_EGRESS<\/p>\n\n\n\n
crash> p\/a ((struct cgroup *)0xffff9ce3e2416800)->bpf.effective[1]\n$6 = 0x6b6b6b6b6b6b6b6b\ncrash> x\/20a 0xffff9ce3e2416800\n0xffff9ce3e2416800:     0x6b6b6b6b6b6b016b      0x6b6b6b6b6b6b6b6b\n0xffff9ce3e2416810:     0x6b6b6b6b6b6b6b6b      0x6b6b6b6b6b6b6b6b\n0xffff9ce3e2416820:     0x6b6b6b6b6b6b6b6b      0x6b6b6b6b6b6b6b6b\n0xffff9ce3e2416830:     0x6b6b6b6b6b6b6b6b      0x6b6b6b6b6b6b6b6b\n0xffff9ce3e2416840:     0x6b6b6b6b6b6b6b6b      0x6b6b6b6b6b6b6b6b\n0xffff9ce3e2416850:     0x6b6b6b6b6b6b6b6b      0x6b6b6b6b6b6b6b6b\n0xffff9ce3e2416860:     0x6b6b6b6b6b6b6b6b      0x6b6b6b6b6b6b6b6b\n0xffff9ce3e2416870:     0x6b6b6b6b6b6b6b6b      0x6b6b6b6b6b6b6b6b\n0xffff9ce3e2416880:     0x6b6b6b6b6b6b6b6b      0x6b6b6b6b6b6b6b6b\n0xffff9ce3e2416890:     0x6b6b6b6b6b6b6b6b      0x6b6b6b6b6b6b6b6b\ncrash><\/code><\/pre>\n\n\n\n
This pointer (struct cgroup *)<\/p>\n\n\n\n
\tcgrp = sock_cgroup_ptr(&sk->sk_cgrp_data);<\/code><\/pre>\n\n\n\n
Points to the freed object. However, kernel still keeps eBPF rules attached to the socket under cgroups. When process (sshd) dies (do_exit() call) and cleanup is executed, all sockets are being closed. If such socket has “pending” packets, the following code path is executed:<\/p>\n\n\n\n
do_exit -> ... -> sock_close -> __sock_release -> inet_release -> tcp_close -> __tcp_push_pending_frames -> tcp_write_xmit -> __tcp_transmit_skb -> __ip_queue_xmit -> ip_finish_output -> __cgroup_bpf_run_filter_skb<\/code><\/pre>\n\n\n\n
However, there is nothing wrong with such logic and path. The real problem is that cgroups disappeared while still holding active clients. How is that even possible? Just before the crash I can see the following entry in kernel logs:<\/p>\n\n\n\n
[190820.457422] ------------[ cut here ]------------\n[190820.457465] percpu ref (cgroup_bpf_release_fn) <= 0 (-70581) after switching to atomic\n                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n[190820.457511] WARNING: CPU: 0 PID: 9 at lib\/percpu-refcount.c:161 percpu_ref_switch_to_atomic_rcu+0x112\/0x120\n[190820.457511] Modules linked in: [last unloaded: p_lkrg]\n[190820.457513] CPU: 0 PID: 9 Comm: ksoftirqd\/0 Kdump: loaded Tainted: G           OE     5.7.0-g4 #28\n[190820.457513] Hardware name: VMware, Inc. VMware Virtual Platform\/440BX Desktop Reference Platform, BIOS 6.00 04\/13\/2018\n[190820.457515] RIP: 0010:percpu_ref_switch_to_atomic_rcu+0x112\/0x120\n[190820.457516] Code: eb b6 80 3d 11 95 5a 02 00 0f 85 65 ff ff ff 48 8b 55 d8 48 8b 75 e8 48 c7 c7 d0 9f 78 93 c6 05 f5 94 5a 02 01 e8 00 57 88 ff <0f> 0b e9 43 ff ff ff 0f 0b eb 9d cc cc cc 8d 8c 16 ef be ad de 89\n[190820.457516] RSP: 0018:ffffb136c0087e00 EFLAGS: 00010286\n[190820.457517] RAX: 0000000000000000 RBX: 7ffffffffffeec4a RCX: 0000000000000000\n[190820.457517] RDX: 0000000000000101 RSI: ffffffff949235c0 RDI: 00000000ffffffff\n[190820.457517] RBP: ffff9ce3e204af20 R08: 6d6f7461206f7420 R09: 63696d6f7461206f\n[190820.457517] R10: 7320726574666120 R11: 676e696863746977 R12: 00003452c5002ce8\n[190820.457518] R13: ffff9ce3f6e2b450 R14: ffff9ce2c7fc3100 R15: 0000000000000000\n[190820.457526] FS:  0000000000000000(0000) GS:ffff9ce3f6e00000(0000) knlGS:0000000000000000\n[190820.457527] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033\n[190820.457527] CR2: 00007f516c2b9000 CR3: 0000000222c64006 CR4: 00000000003606f0\n[190820.457550] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000\n[190820.457551] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400\n[190820.457551] Call Trace:\n[190820.457577]  rcu_core+0x1df\/0x530\n[190820.457598]  ? smpboot_register_percpu_thread+0xd0\/0xd0\n[190820.457609]  __do_softirq+0xfc\/0x331\n[190820.457629]  ? smpboot_register_percpu_thread+0xd0\/0xd0\n[190820.457630]  run_ksoftirqd+0x21\/0x30\n[190820.457649]  smpboot_thread_fn+0x195\/0x230\n[190820.457660]  kthread+0x139\/0x160\n[190820.457670]  ? __kthread_bind_mask+0x60\/0x60\n[190820.457671]  ret_from_fork+0x35\/0x40\n[190820.457682] ---[ end trace 63d2aef89e998452 ]---<\/span><\/code><\/pre>\n\n\n\n
I was testing the same scenario a few times and I had the following results:<\/p>\n\n\n\n
 percpu ref (cgroup_bpf_release_fn) <= 0 (-70581) after switching to atomic\n percpu ref (cgroup_bpf_release_fn) <= 0 (-18829) after switching to atomic\n percpu ref (cgroup_bpf_release_fn) <= 0 (-29849) after switching to atomic\n<\/span><\/code><\/pre>\n\n\n\n
Let’s look at this function:<\/p>\n\n\n\n
\/**\n * cgroup_bpf_release_fn() - callback used to schedule releasing\n *                           of bpf cgroup data\n * @ref: percpu ref counter structure\n *\/\nstatic void cgroup_bpf_release_fn(struct percpu_ref *ref)\n{\n\tstruct cgroup *cgrp = container_of(ref, struct cgroup, bpf.refcnt);\n\n\tINIT_WORK(&cgrp->bpf.release_work, cgroup_bpf_release);\n\tqueue_work(system_wq, &cgrp->bpf.release_work);\n}<\/span><\/code><\/pre>\n\n\n\n
So that’s the callback used to release bpf cgroup data. Sounds like it is being called while there could be still active socket attached to such cgroup:<\/p>\n\n\n\n
\/**\n * cgroup_bpf_release() - put references of all bpf programs and\n *                        release all cgroup bpf data\n * @work: work structure embedded into the cgroup to modify\n *\/\nstatic void cgroup_bpf_release(struct work_struct *work)\n{\n\tstruct cgroup *p, *cgrp = container_of(work, struct cgroup,\n\t\t\t\t\t       bpf.release_work);\n\tstruct bpf_prog_array *old_array;\n\tunsigned int type;\n\n\tmutex_lock(&cgroup_mutex);\n\n\tfor (type = 0; type < ARRAY_SIZE(cgrp->bpf.progs); type++) {\n\t\tstruct list_head *progs = &cgrp->bpf.progs[type];\n\t\tstruct bpf_prog_list *pl, *tmp;\n\n\t\tlist_for_each_entry_safe(pl, tmp, progs, node) {\n\t\t\tlist_del(&pl->node);\n\t\t\tif (pl->prog)\n\t\t\t\tbpf_prog_put(pl->prog);\n\t\t\tif (pl->link)\n\t\t\t\tbpf_cgroup_link_auto_detach(pl->link);\n\t\t\tbpf_cgroup_storages_unlink(pl->storage);\n\t\t\tbpf_cgroup_storages_free(pl->storage);\n\t\t\tkfree(pl);\n\t\t\tstatic_branch_dec(&cgroup_bpf_enabled_key);\n\t\t}\n\t\told_array = rcu_dereference_protected(\n\t\t\t\tcgrp->bpf.effective[type],\n\t\t\t\tlockdep_is_held(&cgroup_mutex));\n\t\tbpf_prog_array_free(old_array);\n\t}\n\n\tmutex_unlock(&cgroup_mutex);\n\n\tfor (p = cgroup_parent(cgrp); p; p = cgroup_parent(p))\n\t\tcgroup_bpf_put(p);\n\n\tpercpu_ref_exit(&cgrp->bpf.refcnt);\n\tcgroup_put(cgrp);\n}<\/span><\/code><\/pre>\n\n\n\n
while:<\/p>\n\n\n\n
static void bpf_cgroup_link_auto_detach(struct bpf_cgroup_link *link)\n{\n\tcgroup_put(link->cgroup);\n\tlink->cgroup = NULL;\n}<\/code><\/pre>\n\n\n\n
So if cgroup dies, all the potential clients are being auto_detached. However, they might not be aware about such situation. When is cgroup_bpf_release_fn() executed?<\/p>\n\n\n\n
\/**\n * cgroup_bpf_inherit() - inherit effective programs from parent\n * @cgrp: the cgroup to modify\n *\/\nint cgroup_bpf_inherit(struct cgroup *cgrp)\n{\n    ...\n  \tret = percpu_ref_init(&cgrp->bpf.refcnt, cgroup_bpf_release_fn, 0,\n\t\t\t      GFP_KERNEL);\n    ...\n}<\/span><\/code><\/pre>\n\n\n\n
It is automatically executed when cgrp->bpf.refcnt drops to 1. However, in the warning logs before kernel had crashed, we saw that such reference counter is below 0. Cgroup was already freed.<\/p>\n\n\n\n
Originally, I thought that the problem might be related to the code walking through the cgroup hierarchy without holding cgroup_mutex, which was pointed out by Alexei. I’ve prepared the patch and recompiled the kernel:<\/p>\n\n\n\n
$ diff -u cgroup.c linux-5.7\/kernel\/bpf\/cgroup.c\n--- cgroup.c    2020-05-31 23:49:15.000000000 +0000\n+++ linux-5.7\/kernel\/bpf\/cgroup.c       2020-07-17 16:31:10.712969480 +0000\n@@ -126,11 +126,11 @@\n                bpf_prog_array_free(old_array);\n        }\n\n-       mutex_unlock(&cgroup_mutex);\n-\n        for (p = cgroup_parent(cgrp); p; p = cgroup_parent(p))\n                cgroup_bpf_put(p);\n\n+       mutex_unlock(&cgroup_mutex);\n+\n        percpu_ref_exit(&cgrp->bpf.refcnt);\n        cgroup_put(cgrp);\n }<\/code><\/pre>\n\n\n\n
Interestingly, without this patch I was able to generate this kernel crash every time when I was rebooting the machine (100% repro). After this patch crashing ratio dropped to around 30%. However, I was still able to hit the same code-path and generate kernel dump. The patch indeed helps but it looks like it’s not the real problem since I can still hit the crash (just much less often).<\/p>\n\n\n\n
I stepped back and looked again where the bug is. Corrupted pointer (struct cgroup *) is comming from that line:<\/p>\n\n\n\n
\tcgrp = sock_cgroup_ptr(&sk->sk_cgrp_data);<\/code><\/pre>\n\n\n\n
this code is related to the CONFIG_SOCK_CGROUP_DATA. Linux source has an interesting comment about it in “cgroup-defs.h” file:<\/p>\n\n\n\n
\/*\n * sock_cgroup_data is embedded at sock->sk_cgrp_data and contains\n * per-socket cgroup information except for memcg association.\n *\n * On legacy hierarchies, net_prio and net_cls controllers directly set\n * attributes on each sock which can then be tested by the network layer.\n * On the default hierarchy, each sock is associated with the cgroup it was\n * created in and the networking layer can match the cgroup directly.\n *\n * To avoid carrying all three cgroup related fields separately in sock,\n * sock_cgroup_data overloads (prioidx, classid) and the cgroup pointer.\n * On boot, sock_cgroup_data records the cgroup that the sock was created\n * in so that cgroup2 matches can be made; however, once either net_prio or\n * net_cls starts being used, the area is overriden to carry prioidx and\/or\n * classid.  The two modes are distinguished by whether the lowest bit is\n * set.  Clear bit indicates cgroup pointer while set bit prioidx and\n * classid.\n *\n * While userland may start using net_prio or net_cls at any time, once\n * either is used, cgroup2 matching no longer works.  There is no reason to\n * mix the two and this is in line with how legacy and v2 compatibility is\n * handled.  On mode switch, cgroup references which are already being\n * pointed to by socks may be leaked.  While this can be remedied by adding\n * synchronization around sock_cgroup_data, given that the number of leaked\n * cgroups is bound and highly unlikely to be high, this seems to be the\n * better trade-off.\n *\/<\/span><\/code><\/pre>\n\n\n\n
and later:<\/p>\n\n\n\n
\/*\n * There's a theoretical window where the following accessors race with\n * updaters and return part of the previous pointer as the prioidx or\n * classid.  Such races are short-lived and the result isn't critical.\n *\/<\/code><\/pre>\n\n\n\n
This means that sock_cgroup_data “carries” the information whether net_prio or net_cls starts being used and in such case sock_cgroup_data overloads (prioidx, classid) and the cgroup pointer. In our crash we can extract this information:<\/p>\n\n\n\n
crash> print\/a ((struct sock *)0xffff9ce3de26b280)->sk_cgrp_data\n$5 = {\n  {\n    {\n      is_data = 0x0,\n      padding = 0x68,\n      prioidx = 0xe241,\n      classid = 0xffff9ce3\n    },\n    val = 0xffff9ce3e2416800\n  }\n}<\/code><\/pre>\n\n\n\n
Described socket keeps the “sk_cgrp_data” pointer with the information of being “attached” to the cgroup2. However, cgroup2 has been destroyed.
Now we have all the information to solve the mystery of this bug:<\/p>\n\n\n\n
    \n
  1. Process creates a socket and both of them are inside some cgroup v2 (non-root)\n