|
|
eBPF is considered Technology Preview and is not intended for production use. For information on Red Hat scope of support for Technology Preview features, see: Technology Preview Features Support Scope |
In this unit, we will get familiar with eBPF. eBPF stands for Extended Berkeley Packet Filter and is effectively an in-kernel virtual machine upon which several applications have been built. To date, these applications center around working with the network stack (eXpress Data Path, Traffic Control) and also offering efficient kernel tracing (BPF Compiler Collection). In today’s lab, we will be focusing on the kernel tracing aspect of eBPF.
|
💡
|
Now is a great time to use the multi session capability of tmux. Using Ctrl-b c create another session. You can now cycle back and forth between them with CTRL-b n (next) and CTRL-b p (previous). Just make sure you are the user root on workstation.example.com in order to sucessfully run the rootkit script.
|
There is a dedicated VM we will use for the container exercises. From workstation.example.com, you should be able to ssh to node2.example.com as 'root' without any prompts for credentials.
ssh node2.example.com
Verify that you are on the right host for these exercises.
cheat-ebpf-checkhost.sh
Now you are ready to begin your exercises with eBPF Tracing.
Start by installing the BPF Compiler Collection bcc-tools package:
yum install bcc-tools -y
Now, we have a lot of interesting tools installed in /usr/share/bcc/tools. These all have man pages for them.
The tcplife tool monitors tcp connections on the system and measures the amount of time that they’ve been open, the local and remote ports involved in the connection and the amount of data transferred and received in kilobytes. To run this tool, do:
/usr/share/bcc/tools/tcplife
Wait for this to get started and you’ll see the following header print out:
PID COMM LADDR LPORT RADDR RPORT TX_KB RX_KB MS|
💡
|
Remember that tmux is your friend here. If you have any questions, ask the instructor on how to use it |
Now from another workstation terminal session, run:
ssh node2.example.com cheat-ebpf-rootkit.sh
This command should run for about 10 seconds and then exit.
On the node2 terminal sessions you should now see the output of tcplife similar to:
PID COMM LADDR LPORT RADDR RPORT TX_KB RX_KB MS
28478 sshd 10.0.0.12 22 10.0.0.10 33514 14 2 9091.95Doing the math with my numbers above, my session was open for about 9 seconds and was initiated by IP 10.0.0.10, the workstation.
On the terminal running tcp-life, go ahead and Ctrl-C the process. You will see a python traceback that looks like:
^CTraceback (most recent call last):
File "./tcplife", line 506, in <module>
b.perf_buffer_poll()
File "/usr/lib/python3.6/site-packages/bcc/__init__.py", line 1216, in perf_buffer_poll
lib.perf_reader_poll(len(readers), readers, timeout)
KeyboardInterruptThis is safe to ignore and the expected behavior of a Ctrl-C on this process.
The execsnoop script monitors all calls to execve() and catches all processes that follow the fork→exec sequence, as well as processes that re-exec() themselves. Processes that fork() but do not exec() won’t be caught by this script.
To run this script, do:
/usr/share/bcc/tools/execsnoop
You will see a header print that looks like:
PCOMM PID PPID RET ARGSNow from another workstation terminal, run:
ssh node2.example.com cheat-ebpf-rootkit.sh
You can immediately exit your session and return to the workstation. In the execsnoop terminal, you should see output similar to:
PCOMM PID PPID RET ARGS sshd 28512 749 0 /usr/sbin/sshd -D [email protected],[email protected],aes256-ctr,aes256-cbc,[email protected],aes128-ctr,aes128-cb [email protected],[email protected],[email protected],[email protected],hmac-sha2- -oGSSAPIKexAlgorithms=gss-gex-sha1-,gss-group14-sha1- [email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-excha -oHostKeyAlgorithms=rsa-sha2-256,ecdsa-sha2-nistp256,[email protected],ecdsa-sha2-nistp384,ecdsa-sha2-nis -oPubkeyAcceptedKeyTypes=rsa-sha2-256,ecdsa-sha2-nistp256,[email protected],ecdsa-sha2-nistp384,ecdsa-sha -R unix_chkpwd 28514 28512 0 /usr/sbin/unix_chkpwd root chkexpiry bash 28516 28515 0 /bin/bash -c cheat-ebpf-rootkit.sh grepconf.sh 28517 28516 0 /usr/libexec/grepconf.sh -c grep 28518 28517 0 /usr/bin/grep -qsi ^COLOR.*none /etc/GREP_COLORS grepconf.sh 28519 28516 0 /usr/libexec/grepconf.sh -c grep 28520 28519 0 /usr/bin/grep -qsi ^COLOR.*none /etc/GREP_COLORS grepconf.sh 28521 28516 0 /usr/libexec/grepconf.sh -c grep 28522 28521 0 /usr/bin/grep -qsi ^COLOR.*none /etc/GREP_COLORS sed 28524 28523 0 /usr/bin/sed -r -e s/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d /etc/locale.conf uname 28525 28516 0 /usr/bin/uname -a sleep 28526 28516 0 /usr/bin/sleep 1 who 28527 28516 0 /usr/bin/who sleep 28528 28516 0 /usr/bin/sleep 1 grep 28530 28516 0 /usr/bin/grep root /etc/passwd sleep 28531 28516 0 /usr/bin/sleep 1 grep 28532 28516 0 /usr/bin/grep root /etc/shadow sleep 28533 28516 0 /usr/bin/sleep 1 cat 28534 28516 0 /usr/bin/cat /etc/fstab sleep 28535 28516 0 /usr/bin/sleep 1 ps 28536 28516 0 /usr/bin/ps -ef sleep 28537 28516 0 /usr/bin/sleep 1 netstat 28538 28516 0 /usr/bin/netstat -tulpn sleep 28539 28516 0 /usr/bin/sleep 1 getenforce 28540 28516 0 /usr/sbin/getenforce sleep 28541 28516 0 /usr/bin/sleep 1 firewall-cmd 28542 28516 0 /usr/bin/firewall-cmd --state
This shows you all the processes that ran exec during that ssh login, their PID, their parent PID, their return code, and the arguments that were sent to the process. You could keep monitoring this for quite some time to catch potential bad actors on the system.
Go to the terminal with execsnoop running and issue a Ctrl-C. This will end the process with a python traceback. That is the expected behavior.
Similar in nature to execsnoop, opensnoop traces the open() syscall, which shows which processes are attempting to open which files.
To run this script, do:
/usr/share/bcc/tools/opensnoop
You will see a header that prints out like:
PID COMM FD ERR PATHIn a workstation terminal, perform the following steps to connect to node2 as a student:
ssh student@node2
You probably saw a lot of data go by in the window running opensnoop as there are a lot of files opened during the establishing of an ssh session and the corresponding login activity.
Now on node2, let’s do:
cat /etc/fstab
You will see a lot of locale data being loaded in addition to /etc/fstab in the opensnoop window like so:
31474 cat 3 0 /etc/ld.so.cache
31474 cat 3 0 /lib64/libc.so.6
31474 cat -1 2 /usr/lib/locale/locale-archive
31474 cat 3 0 /usr/share/locale/locale.alias
31474 cat -1 2 /usr/lib/locale/en_US.UTF-8/LC_IDENTIFICATION
31474 cat 3 0 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION
31474 cat 3 0 /usr/lib64/gconv/gconv-modules.cache
31474 cat -1 2 /usr/lib/locale/en_US.UTF-8/LC_MEASUREMENT
31474 cat 3 0 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT
31474 cat -1 2 /usr/lib/locale/en_US.UTF-8/LC_TELEPHONE
31474 cat 3 0 /usr/lib/locale/en_US.utf8/LC_TELEPHONE
31474 cat -1 2 /usr/lib/locale/en_US.UTF-8/LC_ADDRESS
31474 cat 3 0 /usr/lib/locale/en_US.utf8/LC_ADDRESS
31474 cat -1 2 /usr/lib/locale/en_US.UTF-8/LC_NAME
31474 cat 3 0 /usr/lib/locale/en_US.utf8/LC_NAME
31474 cat -1 2 /usr/lib/locale/en_US.UTF-8/LC_PAPER
31474 cat 3 0 /usr/lib/locale/en_US.utf8/LC_PAPER
31474 cat -1 2 /usr/lib/locale/en_US.UTF-8/LC_MESSAGES
31474 cat 3 0 /usr/lib/locale/en_US.utf8/LC_MESSAGES
31474 cat 3 0 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES
31474 cat -1 2 /usr/lib/locale/en_US.UTF-8/LC_MONETARY
31474 cat 3 0 /usr/lib/locale/en_US.utf8/LC_MONETARY
31474 cat -1 2 /usr/lib/locale/en_US.UTF-8/LC_COLLATE
31474 cat 3 0 /usr/lib/locale/en_US.utf8/LC_COLLATE
31474 cat -1 2 /usr/lib/locale/en_US.UTF-8/LC_TIME
31474 cat 3 0 /usr/lib/locale/en_US.utf8/LC_TIME
31474 cat -1 2 /usr/lib/locale/en_US.UTF-8/LC_NUMERIC
31474 cat 3 0 /usr/lib/locale/en_US.utf8/LC_NUMERIC
31474 cat -1 2 /usr/lib/locale/en_US.UTF-8/LC_CTYPE
31474 cat 3 0 /usr/lib/locale/en_US.utf8/LC_CTYPE
31474 cat 3 0 /etc/fstabThe fourth column is the exit code and you can see that some of these files exit with non-zero codes meaning they don’t exist or the user doesn’t have appropriate permissions. Let’s take a look at /usr/lib/locale/locale-archive:
ls /usr/lib/locale/locale-archive
and we see the following returned:
ls: cannot access '/usr/lib/locale/locale-archive': No such file or directoryNow let’s try to cat /etc/shadow as the student user:
cat /etc/shadow
We see the following returned in opensnoop:
31480 cat 3 0 /etc/ld.so.cache
31480 cat 3 0 /lib64/libc.so.6
31480 cat -1 2 /usr/lib/locale/locale-archive
31480 cat 3 0 /usr/share/locale/locale.alias
31480 cat -1 2 /usr/lib/locale/en_US.UTF-8/LC_IDENTIFICATION
31480 cat 3 0 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION
31480 cat 3 0 /usr/lib64/gconv/gconv-modules.cache
31480 cat -1 2 /usr/lib/locale/en_US.UTF-8/LC_MEASUREMENT
31480 cat 3 0 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT
31480 cat -1 2 /usr/lib/locale/en_US.UTF-8/LC_TELEPHONE
31480 cat 3 0 /usr/lib/locale/en_US.utf8/LC_TELEPHONE
31480 cat -1 2 /usr/lib/locale/en_US.UTF-8/LC_ADDRESS
31480 cat 3 0 /usr/lib/locale/en_US.utf8/LC_ADDRESS
31480 cat -1 2 /usr/lib/locale/en_US.UTF-8/LC_NAME
31480 cat 3 0 /usr/lib/locale/en_US.utf8/LC_NAME
31480 cat -1 2 /usr/lib/locale/en_US.UTF-8/LC_PAPER
31480 cat 3 0 /usr/lib/locale/en_US.utf8/LC_PAPER
31480 cat -1 2 /usr/lib/locale/en_US.UTF-8/LC_MESSAGES
31480 cat 3 0 /usr/lib/locale/en_US.utf8/LC_MESSAGES
31480 cat 3 0 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES
31480 cat -1 2 /usr/lib/locale/en_US.UTF-8/LC_MONETARY
31480 cat 3 0 /usr/lib/locale/en_US.utf8/LC_MONETARY
31480 cat -1 2 /usr/lib/locale/en_US.UTF-8/LC_COLLATE
31480 cat 3 0 /usr/lib/locale/en_US.utf8/LC_COLLATE
31480 cat -1 2 /usr/lib/locale/en_US.UTF-8/LC_TIME
31480 cat 3 0 /usr/lib/locale/en_US.utf8/LC_TIME
31480 cat -1 2 /usr/lib/locale/en_US.UTF-8/LC_NUMERIC
31480 cat 3 0 /usr/lib/locale/en_US.utf8/LC_NUMERIC
31480 cat -1 2 /usr/lib/locale/en_US.UTF-8/LC_CTYPE
31480 cat 3 0 /usr/lib/locale/en_US.utf8/LC_CTYPE
31480 cat -1 13 /etc/shadow
31480 cat -1 2 /usr/share/locale/en_US.UTF-8/LC_MESSAGES/libc.mo
31480 cat -1 2 /usr/share/locale/en_US.utf8/LC_MESSAGES/libc.mo
31480 cat -1 2 /usr/share/locale/en_US/LC_MESSAGES/libc.mo
31480 cat -1 2 /usr/share/locale/en.UTF-8/LC_MESSAGES/libc.mo
31480 cat -1 2 /usr/share/locale/en.utf8/LC_MESSAGES/libc.mo
31480 cat -1 2 /usr/share/locale/en/LC_MESSAGES/libc.mowith the key line being:
31480 cat -1 13 /etc/shadowThe -13 return code indicates that we did not have permission to open the requested resource.
As you can see, this tool is very useful for trying to determine where an application has its configurations or for seeing why an application may be failing.
Go to the terminal with opensnoop running and issue a Ctrl-C. This will end the process with a python traceback. That is the expected behavior.
|
|
Please check which filesystem your host is using with the command df -T /. If you host is configured with ext4, then substitute the command ext4slower in place of xfsslower.
|
The purpose of the xfsslower tool (also ext4slower and nfsslower) is to show you filesystem operations slower than 10ms. It traces reads, writes, opens, and syncs and then prints out the timestamp of the operation, the process name, the ID, the type of operation, the file offset in kilobytes, the latency of the I/O measured from when it was issued by VFS to the filesystem to when it was completed, and finally, the filename being operated on.
To run this script, do:
/usr/share/bcc/tools/xfsslower
Tracing XFS operations slower than 10 ms
TIME COMM PID T BYTES OFF_KB LAT(ms) FILENAMENow in another node2 terminal window, let’s run:
dd if=/dev/urandom of=bigfile bs=1024 count=20000
This writes out a 20M file called bigfile and should not register on your xfsslower window.
Now, let’s execute the above command in a for loop so that we get more I/O going in parallel:
for i in $(seq 1 10); do dd if=/dev/urandom of=bigfile$i bs=1024 count=20000 & done
Now you should see similar output in your xfsslower window:
TIME COMM PID T BYTES OFF_KB LAT(ms) FILENAME
20:44:43 b'dd' 32446 W 1024 778 44.11 b'bigfile1'
20:44:43 b'dd' 32455 W 1024 818 55.11 b'bigfile10'
20:44:43 b'dd' 32452 W 1024 1712 44.11 b'bigfile7'
20:44:43 b'dd' 32455 W 1024 1778 55.02 b'bigfile10'
20:44:43 b'dd' 32451 W 1024 2850 44.11 b'bigfile6'
20:44:43 b'dd' 32447 W 1024 3598 44.10 b'bigfile2'
20:44:43 b'dd' 32451 W 1024 3805 55.11 b'bigfile6'
20:44:43 b'dd' 32446 W 1024 4612 44.28 b'bigfile1'
20:44:43 b'dd' 32446 W 1024 5529 33.01 b'bigfile1'
20:44:43 b'dd' 32454 W 1024 4504 55.11 b'bigfile9'
20:44:43 b'dd' 32447 W 1024 7335 44.10 b'bigfile2'
20:44:43 b'dd' 32455 W 1024 7545 44.02 b'bigfile10'
20:44:43 b'dd' 32446 W 1024 8344 49.16 b'bigfile1'
20:44:43 b'dd' 32448 W 1024 8183 44.18 b'bigfile3'
20:44:43 b'dd' 32447 W 1024 9168 55.10 b'bigfile2'
20:44:43 b'dd' 32449 W 1024 9728 54.10 b'bigfile4'
20:44:43 b'dd' 32454 W 1024 10244 33.11 b'bigfile9'
20:44:43 b'dd' 32447 W 1024 10989 55.02 b'bigfile2'
20:44:43 b'dd' 32453 W 1024 11276 54.10 b'bigfile8'
20:44:43 b'dd' 32453 W 1024 12169 33.10 b'bigfile8'
20:44:43 b'dd' 32451 W 1024 13292 91.11 b'bigfile6'
20:44:43 b'dd' 32453 W 1024 13108 47.24 b'bigfile8'
20:44:43 b'dd' 32448 W 1024 13788 44.01 b'bigfile3'
20:44:43 b'dd' 32454 W 1024 14137 44.23 b'bigfile9'
20:44:43 b'dd' 32446 W 1024 16076 44.02 b'bigfile1'
20:44:43 b'dd' 32447 W 1024 15796 44.26 b'bigfile2'
20:44:44 b'dd' 32446 W 1024 17004 44.10 b'bigfile1'
20:44:44 b'dd' 32455 W 1024 16697 44.16 b'bigfile10'
20:44:44 b'dd' 32450 W 1024 18505 44.01 b'bigfile5'
20:44:44 b'dd' 32451 W 1024 19056 44.17 b'bigfile6'
20:44:44 b'dd' 32446 W 1024 19868 44.38 b'bigfile1'
20:44:44 b'dd' 32452 W 1024 19272 44.14 b'bigfile7'
20:44:44 b'dd' 32455 W 1024 19168 30.75 b'bigfile10'
20:44:44 b'dd' 32453 W 1024 19612 31.16 b'bigfile8'
20:44:44 b'dd' 32454 W 1024 19460 24.59 b'bigfile9'
20:44:44 b'dd' 32447 W 1024 19508 36.20 b'bigfile2'So we can see that when writing these files in parallel, we have xfs operations taking longer than 10ms to complete.
Go to the terminal with xfsslower running and issue a Ctrl-C. This will end the process with a python traceback. That is the expected behavior.
The cachestat tool traces kernel page cache functions and prints per-second summaries to aid you in workload characterization.
To run this script, do:
cd /usr/share/bcc/tools ./cachestat
You should see the following header print:
TOTAL MISSES HITS DIRTIES BUFFERS_MB CACHED_MBIn another root terminal on node 2, run our dd for loop from the xfsslower section:
for i in $(seq 1 10); do dd if=/dev/urandom of=bigfile$i bs=1024 count=20000 & done
In the cachestat window, you should output similar to:
TOTAL MISSES HITS DIRTIES BUFFERS_MB CACHED_MB
1135 0 1135 0 5 1464
9852 31 9821 0 5 1464This shows that we had 31 page cache misses while running the above loop, but during that same second, there were 9,821 hits, indicating great performance from the page cache.
Go to the terminal with cachestat running and issue a Ctrl-C. This will end the process with the message "Detaching…". That is the expected behavior.
This tool is a swiss army knife allowing you to specify functions to trace and messages to be printed when certain conditions are met. You can read more about this by running:
man 8 trace
Let’s do a simple trace in which we will dynamically trace the do_sys_open() kernel function and print the names of the files opened. Run this with:
cd /usr/share/bcc/tools ./trace 'p::do_sys_open "%s", arg2'
Now in another node2 terminal, run:
cat /etc/fstab
In the window with trace running, you will see something similar to:
32559 32559 cat do_sys_open b'/etc/ld.so.cache'
32559 32559 cat do_sys_open b'/lib64/libc.so.6'
32559 32559 cat do_sys_open b'/usr/lib/locale/locale-archive'
32559 32559 cat do_sys_open b'/usr/share/locale/locale.alias'
32559 32559 cat do_sys_open b'/usr/lib/locale/en_US.UTF-8/LC_IDENTIFICATION'
32559 32559 cat do_sys_open b'/usr/lib/locale/en_US.utf8/LC_IDENTIFICATION'
32559 32559 cat do_sys_open b'/usr/lib64/gconv/gconv-modules.cache'
32559 32559 cat do_sys_open b'/usr/lib/locale/en_US.UTF-8/LC_MEASUREMENT'
32559 32559 cat do_sys_open b'/usr/lib/locale/en_US.utf8/LC_MEASUREMENT'
32559 32559 cat do_sys_open b'/usr/lib/locale/en_US.UTF-8/LC_TELEPHONE'
32559 32559 cat do_sys_open b'/usr/lib/locale/en_US.utf8/LC_TELEPHONE'
32559 32559 cat do_sys_open b'/usr/lib/locale/en_US.UTF-8/LC_ADDRESS'
32559 32559 cat do_sys_open b'/usr/lib/locale/en_US.utf8/LC_ADDRESS'
32559 32559 cat do_sys_open b'/usr/lib/locale/en_US.UTF-8/LC_NAME'
32559 32559 cat do_sys_open b'/usr/lib/locale/en_US.utf8/LC_NAME'
32559 32559 cat do_sys_open b'/usr/lib/locale/en_US.UTF-8/LC_PAPER'
32559 32559 cat do_sys_open b'/usr/lib/locale/en_US.utf8/LC_PAPER'
32559 32559 cat do_sys_open b'/usr/lib/locale/en_US.UTF-8/LC_MESSAGES'
32559 32559 cat do_sys_open b'/usr/lib/locale/en_US.utf8/LC_MESSAGES'
32559 32559 cat do_sys_open b'/usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES'
32559 32559 cat do_sys_open b'/usr/lib/locale/en_US.UTF-8/LC_MONETARY'
32559 32559 cat do_sys_open b'/usr/lib/locale/en_US.utf8/LC_MONETARY'
32559 32559 cat do_sys_open b'/usr/lib/locale/en_US.UTF-8/LC_COLLATE'
32559 32559 cat do_sys_open b'/usr/lib/locale/en_US.utf8/LC_COLLATE'
32559 32559 cat do_sys_open b'/usr/lib/locale/en_US.UTF-8/LC_TIME'
32559 32559 cat do_sys_open b'/usr/lib/locale/en_US.utf8/LC_TIME'
32559 32559 cat do_sys_open b'/usr/lib/locale/en_US.UTF-8/LC_NUMERIC'
32559 32559 cat do_sys_open b'/usr/lib/locale/en_US.utf8/LC_NUMERIC'
32559 32559 cat do_sys_open b'/usr/lib/locale/en_US.UTF-8/LC_CTYPE'
32559 32559 cat do_sys_open b'/usr/lib/locale/en_US.utf8/LC_CTYPE'
32559 32559 cat do_sys_open b'/etc/fstab'Go ahead and Ctrl-C trace and then let’s do one more trace, this time, tracing the return values out of trace:
cd /usr/share/bcc/tools ./trace 'r::do_sys_open "ret: %d", retval'
Now in another node2 terminal, run:
cat /etc/fstab
and in your trace window, you will see output similar to:
PID TID COMM FUNC -
32576 32576 cat do_sys_open ret: 3
32576 32576 cat do_sys_open ret: 3
32576 32576 cat do_sys_open ret: -2
32576 32576 cat do_sys_open ret: 3
32576 32576 cat do_sys_open ret: -2
32576 32576 cat do_sys_open ret: 3
32576 32576 cat do_sys_open ret: 3
32576 32576 cat do_sys_open ret: -2
32576 32576 cat do_sys_open ret: 3
32576 32576 cat do_sys_open ret: -2
32576 32576 cat do_sys_open ret: 3
32576 32576 cat do_sys_open ret: -2
32576 32576 cat do_sys_open ret: 3
32576 32576 cat do_sys_open ret: -2
32576 32576 cat do_sys_open ret: 3
32576 32576 cat do_sys_open ret: -2
32576 32576 cat do_sys_open ret: 3
32576 32576 cat do_sys_open ret: -2
32576 32576 cat do_sys_open ret: 3
32576 32576 cat do_sys_open ret: 3
32576 32576 cat do_sys_open ret: -2
32576 32576 cat do_sys_open ret: 3
32576 32576 cat do_sys_open ret: -2
32576 32576 cat do_sys_open ret: 3
32576 32576 cat do_sys_open ret: -2
32576 32576 cat do_sys_open ret: 3
32576 32576 cat do_sys_open ret: -2
32576 32576 cat do_sys_open ret: 3
32576 32576 cat do_sys_open ret: -2
32576 32576 cat do_sys_open ret: 3
32576 32576 cat do_sys_open ret: 3Go to the terminal with trace running and issue a Ctrl-C. This will end the process and return you to the command line.
There is a lot more that you can do with this tool when you actually need to start tracing what is getting passed into kernel functions and what is being returned by those kernel functions.
|
ℹ️
|
You are not required to reference any additional resources for these exercises. This is informational only. |