My dumping ground for what I've been upto

Hello! I’m Joel and I run this site! I work at Google. My interests are scheduler, tracing, synchronization and kernel internals. I also love contributing to the upstream Linux kernel and other open source projects.

Connect with me on Twitter, and LinkedIn. Or, drop me an email at: joel at joelfernandes dot org

Here’s a list of recent kernel patches I submitted. I got featured on hackaday and have written for LWN as well. Check out my resume and also see a list of past talks/presentations. is a resource I created as a collection of articles and resources exploring Linux kernel and internals topics.

Full list of all posts on this site:
  • 22 Dec 2018   Dumping User and Kernel stacks on Kernel events [linuxinternals]
  • 15 Jun 2018   RCU and dynticks-idle mode [linuxinternals]
  • 10 Jun 2018   Single-stepping the kernel's C code [linux]
  • 10 May 2018   RCU-preempt: What happens on a context switch [linuxinternals]
  • 11 Feb 2018   USDT for reliable Userspace event tracing [linuxinternals]
  • 08 Jan 2018   BPFd- Running BCC tools remotely across systems [linuxinternals]
  • 01 Jan 2017   ARMv8: flamegraph and NMI support [linuxinternals]
  • 19 Jun 2016   Ftrace events mechanism [linuxinternals]
  • 20 Mar 2016   TIF_NEED_RESCHED: why is it needed [linuxinternals]
  • 25 Dec 2015   Tying 2 voltage sources/signals together [electronics,linuxinternals]
  • 04 Jun 2014   MicroSD card remote switch [linuxinternals]
  • 07 May 2014   Linux Spinlock Internals [linuxinternals]
  • 24 Apr 2014   Studying cache-line sharing effects on SMP systems [linuxinternals]
  • 23 Apr 2014   Design of fork followed by exec in Linux [linuxinternals]
  • Most Recept Post:

    Dumping User and Kernel stacks on Kernel events

    | Comments

    Dumping the native kernel and userspace stack when a certain path in the kernel or userspace occurs, can be useful to understand which code paths triggered a certain behavior that you’re trying to debug, such as an error you found in the log. One such case is when you notice Selinux denial messages in logs but want to know which path triggered it.

    In this article we will show you how to use kernel instrumentation and BCC to dump the both the user and kernel stack. The article applies both to Android and regular Linux kernels.

    Example: Understanding which path triggered an SELinux denial

    Step 1: Add a tracepoint to the kernel

    Apply the following diff to your kernel. It adds a tracepoint at precisely the point where an SELinux denial is logged. If not cleanly applying, patch it in manually.

    Diff to add a tracepoint for selinux denials
    diff --git a/include/trace/events/selinux.h b/include/trace/events/selinux.h
    new file mode 100644
    index 000000000000..dac185062634
    --- /dev/null
    +++ b/include/trace/events/selinux.h
    @@ -0,0 +1,34 @@
    +#undef TRACE_SYSTEM
    +#define TRACE_SYSTEM selinux
    +#if !defined(_TRACE_SELINUX_H) || defined(TRACE_HEADER_MULTI_READ)
    +#define _TRACE_SELINUX_H
    +#include <linux/ktime.h>
    +#include <linux/tracepoint.h>
    +	TP_PROTO(int cls, int av),
    +	TP_ARGS(cls, av),
    +	TP_STRUCT__entry(
    +		__field(	int,		cls	)
    +		__field(	int,		av	)
    +	),
    +	TP_fast_assign(
    +		__entry->cls = cls;
    +		__entry->av = av;
    +	),
    +	TP_printk("denied %d %d",
    +		__entry->cls,
    +		__entry->av)
    +#endif /* _TRACE_SELINUX_H */
    +/* This part ust be outside protection */
    +#include <trace/define_trace.h>
    diff --git a/security/selinux/avc.c b/security/selinux/avc.c
    index 84d9a2e2bbaf..ab04b7c2dd01 100644
    --- a/security/selinux/avc.c
    +++ b/security/selinux/avc.c
    @@ -34,6 +34,9 @@
     #include "avc_ss.h"
     #include "classmap.h"
    +#include <trace/events/selinux.h>
     #define AVC_CACHE_SLOTS			512
     #define AVC_DEF_CACHE_THRESHOLD		512
     #define AVC_CACHE_RECLAIM		16
    @@ -713,6 +716,12 @@ static void avc_audit_pre_callback(struct audit_buffer *ab, void *a)
     	struct common_audit_data *ad = a;
     	audit_log_format(ab, "avc:  %s ",
     			 ad->selinux_audit_data->denied ? "denied" : "granted");
    +	if (ad->selinux_audit_data->denied) {
    +		trace_selinux_denied(ad->selinux_audit_data->tclass,
    +				     ad->selinux_audit_data->audited);
    +	}
     	avc_dump_av(ab, ad->selinux_audit_data->tclass,
     	audit_log_format(ab, " for ");

    Step 2: Install adeb

    Run the command:

    adeb prepare --full

    This also installs BCC on the Android device which contains the ‘trace’ utility we need for the next step. For regular Linux kernels, you may have to manually install BCC or find a package for it.

    Step 3: Start tracing the user and kernel stacks

    Running the following command:

    adeb shell
    trace -K -U 't:selinux:selinux_denial'

    You should see something like this when denials are triggered:

    2286    2434    Binder:2286_4   selinux_denied   
            avc_audit_pre_callback+0xd8 [kernel]
            avc_audit_pre_callback+0xd8 [kernel]
            common_lsm_audit+0x64 [kernel]
            slow_avc_audit+0x74 [kernel]
            avc_has_perm+0xb8 [kernel]
            selinux_binder_transfer_file+0x158 [kernel]
            security_binder_transfer_file+0x50 [kernel]
            binder_translate_fd+0xcc [kernel]
            binder_transaction+0x1b64 [kernel]
            binder_ioctl+0xadc [kernel]
            do_vfs_ioctl+0x5c8 [kernel]
            sys_ioctl+0x88 [kernel]
            __sys_trace_return+0x0 [kernel]
            __ioctl+0x8 []
            android::IPCThreadState::talkWithDriver(bool)+0x104 []
            android::IPCThreadState::waitForResponse(android::Parcel*, int*)+0x40
            android::IPCThreadState::executeCommand(int)+0x460 []
            android::IPCThreadState::getAndExecuteCommand()+0xa0 []
            android::IPCThreadState::joinThreadPool(bool)+0x40 []
            [unknown] []
            android::Thread::_threadLoop(void*)+0x12c []
            android::AndroidRuntime::javaThreadShell(void*)+0x90 []
            __pthread_start(void*)+0x28 []
            __start_thread+0x48 []

    The same trick can be used for dumping the stack on syscalls, random kernel functions using kprobes and more! Just change the arguments passed to the ‘trace’ command.