fix: [bug description] Not able to generate any vuln report in STDOUT nor SBOMs for mounted qcow2

[omkhard]

Description

Not able to generate any vuln report in STDOUT nor SBOMs for mounted qcow2

To reproduce

okhard@keep-platform-utility:~/$ cve-bin-tool /mnt/
[11:20:22] INFO cve_bin_tool - CVE Binary Tool v3.4 cli.py:624
INFO cve_bin_tool - This product uses the NVD API but is not endorsed or cli.py:625
certified by the NVD.
INFO cve_bin_tool - For potentially faster NVD downloads, mirrors are available cli.py:628
using -n json-mirror
[11:20:23] INFO cve_bin_tool - Getting NVD CVE data... nvd_source.py:389
INFO cve_bin_tool - Getting GitLab Advisory Database CVEs... gad_source.py:86
INFO cve_bin_tool - Getting RedHat CVEs... redhat_source.py:69
INFO cve_bin_tool - Getting PURL2CPE data... purl2cpe_source.py:36
ERROR CVEDB - Unable to fetch EPSS, skipping EPSS. epss_source.py:158
Downloading CVEs... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:41
[11:24:55] INFO cve_bin_tool - Adding 35598 RedHat CVE entries redhat_source.py:136
[11:29:12] INFO cve_bin_tool - Getting Open Source Vulnerability Database CVEs... osv_source.py:161
Killed

Expected behaviour: Should show the vulnerable packages with CVE in stdout as well as in SBOM if sbom tags are given.
Actual behaviour: Not happening as Expected.

Version/platform info

Version of CVE-bin-tool( e.g. output of cve-bin-tool --version):
Installed from pypi or github?
Operating system: Linux/Windows (other platforms are unsupported but feel free to report issues anyhow)

• okhard@keep-platform-utility:~/$ uname -a
  Linux keep-platform-utility 5.15.0-122-generic 132-Ubuntu SMP Thu Aug 29 13:45:52 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
• okhard@keep-platform-utility:~/$ python3 --version
  Python 3.10.12
  Running in any particular CI environment we should know about? (e.g. Github Actions)

[terriko]

That last line is just Killed which tells me that your setup killed the job for some reason, probably a timeout but it could be disk space.

Potential things to try:

• make sure there's no timeouts that are terminating the job early. How long you need will depend on your network connection/disk speed for data updates but probably you need at least a half hour the first time you run things and after that you'll have a cache so it should be faster?
• Make sure you have enough disk space. The full CVE data takes over 2.5G of space.
• Try skipping some of the data sources and seeing if that helps
• Try separating out the data cache update and the scan into separate jobs so a random network timeout won't block your scan.

[omkhard]

There is over ~ 9G of space present in root fs ,

okhard@keep-platform-utility:~/$ df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 1.6G 1.2M 1.6G 1% /run
/dev/vda1 78G 69G 9.2G 89% /
tmpfs 7.9G 168K 7.9G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 7.9G 216K 7.9G 1% /run/qemu
/dev/vda15 105M 6.1M 99M 6% /boot/efi
tmpfs 1.6G 12K 1.6G 1% /run/user/1006
/dev/loop7 2.3G 2.3G 0 100% /tmp/mntsomething
/dev/mapper/debian-root 25G 6.0G 18G 26% /mnt

still the cve-bin-tool keeps on getting killed , may I know what are the other minimum requirements for running cve-bin-tool?

I tried -l debug which even didn't give any error regarding space , disk or io operation related.

[terriko]

Maybe it was killed due to memory issues? I can't really debug this from the cve-bin-tool side, so you'll need to use your linux utilities to figure it out. I'd start with dmesg | less personally. Those logs rotate so you might have to try running it again just so you can watch and see what happens.

In case you've never debugged something similar, here's a random blog post that explains a bit more about figuring out when something was killed by the system: https://www.baeldung.com/linux/what-killed-a-process -- it might not be the answer for you but I don't want to assume that you've ever read a dmesg trace before so I figured I'd include something with more detail.

[terriko]

Also, I don't think you're using github actions but I will say that I've been seeing jobs killed after about 30-45 minutes even when their timeouts are set considerably longer, so I won't rule out being killed by a timeout if you're running in a CI or cloud system that may have additional restrictions enabled.

[omkhard]

Yeah I am not using github actions or not with any CI/CD libraries.
I had installed cve-bin-tool (version 3.4) using pip install

When I am seeing kern.log , seems oom-killer occurs before the cve-bin-tool gets Killed.

ubuntu@moubuntu:~/$ grep cve-bin-tool /var/log/kern.log
Jan  5 14:55:22 moubuntu kernel: [ 7013.572383] cve-bin-tool invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Jan  5 14:55:22 moubuntu kernel: [ 7013.572399] CPU: 0 PID: 7749 Comm: cve-bin-tool Not tainted 5.15.0-48-generic #54-Ubuntu
Jan  5 14:55:22 moubuntu kernel: [ 7013.572773] [   7749]  1000  7749  4212500  3992204 32612352        0             0 cve-bin-tool
Jan  5 14:55:22 moubuntu kernel: [ 7013.572778] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-2.scope,task=cve-bin-tool,pid=7749,uid=1000
Jan  5 14:55:22 moubuntu kernel: [ 7013.572845] Out of memory: Killed process 7749 (cve-bin-tool) total-vm:16850000kB, anon-rss:15968816kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:31848kB oom_score_adj:0
Jan  5 15:39:41 moubuntu kernel: [ 9672.370703] cve-bin-tool invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Jan  5 15:39:41 moubuntu kernel: [ 9672.370722] CPU: 7 PID: 61333 Comm: cve-bin-tool Not tainted 5.15.0-48-generic #54-......

In dmesg.

ubuntu@moubuntu:~/$ sudo dmesg -T | egrep -i 'killed process'
[Sun Jan  5 14:55:21 2025] Out of memory: Killed process 7749 (cve-bin-tool) total-vm:16850000kB, anon-rss:15968816kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:31848kB oom_score_adj:0
[Sun Jan  5 15:39:40 2025] Out of memory: Killed process 61333 (cve-bin-tool) total-vm:17263688kB, anon-rss:15954716kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:31856kB oom_score_adj:0
[Sun Jan  5 17:44:42 2025] Out of memory: Killed process 169786 (cve-bin-tool) total-vm:16834804kB, anon-rss:15946240kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:31808kB oom_score_adj:0
[Mon Jan  6 10:16:54 2025] Out of memory: Killed process 289895 (cve-bin-tool) total-vm:16760068kB, anon-rss:15946128kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:31812kB oom_score_adj:0
[Wed Jan  8 05:53:45 2025] Out of memory: Killed process 1041734 (cve-bin-tool) total-vm:17270384kB, anon-rss:15945564kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:31884kB oom_score_adj:0
[Fri Jan 10 06:52:18 2025] Out of memory: Killed process 1243772 (cve-bin-tool) total-vm:15847988kB, anon-rss:14533604kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:29096kB oom_score_adj:0
[Fri Jan 10 07:35:32 2025] Out of memory: Killed process 1301593 (cve-bin-tool) total-vm:15312808kB, anon-rss:14522492kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:28972kB oom_score_adj:0

What I could guess is the cve-bin-tool is taking a loot of memory as expected, maybe due to multiprocessing/multithreading.

While top :

1355782 ubuntu    20   0 1533572 928848  15572 R 100.0   5.7   1:42.78 cve-bin-tool            1355757 root      20   0       0      0      0 I   5.0   0.0   0:04.38 kworker/u16:3-events_u+
1355760 root      20   0       0      0      0 I   1.7   0.0   0:01.12 kworker/u16:2-events_u+     340 root      20   0       0      0      0 S   0.3   0.0   0:17.40 jbd2/vda1-8              886587 ubuntu    20   0   13908   7988   2988 S   0.3   0.0   0:06.08 tmux: server
1240488 root      20   0 4631556   1.3g   2652 S   0.3   8.2   9:30.44 qemu-system-x86         1355702 root      20   0       0      0      0 I   0.3   0.0

It seems cve-bin-tool is taking 5% of Memory which seems to pretty much , should it be using this much memory ?

before vs during running cve-bin-tool , this is the memory status
before

ubuntu@moubuntu:~/$ free -h
               total        used        free      shared  buff/cache   available
Mem:            15Gi       1.6Gi        13Gi       1.0Mi       396Mi        13Gi
Swap:             0B          0B          0B

during

ubuntu@moubuntu:~/$ free -h
               total        used        free      shared  buff/cache   available
Mem:            15Gi       3.6Gi       4.0Gi       1.0Mi       8.1Gi        11Gi
Swap:             0B          0B          0B

Question : Just wanted to confirm if this is the expected behaviour of cve-bin-tool ?

[terriko]

I haven't done a memory profile on it since I've never seen an OOM myself, but that doesn't seem utterly unreasonable given that it's processing gigs of vuln data in those steps. Maybe someone who's profiled the tool more recently can compare numbers with you? I'll ask in our gitter chat and see if anyone's got numbers or wants to do some profiling.

[frangocm]

Hello! Thank you for the suggestion. I would like to work on this issue. Please let me know if there are any specific steps or requirements I should follow. Thank you!

[AryanBakliwal]

I tested memory usage of cve-bin-tool using the top command on my computer running Ubuntu 24.04.1 and having Intel Core i5-7200U processor and 8 GB of memory.

When I ran cve-bin-tool to scan a SBOM and a package lockfile, memory usage was very low and around 1%.
Then I deleted the database so cve-bin-tool would rebuild it and in this case, the memory usage was not consistent, with gradually increasing from 0% to around 22.5%, then it didn't show in the processes list for couple of minutes and by the time it finished, it utilized a maximum of around 40% memory.

[omkhard]

Any suggestions to avoid OOM-killer , and to reduce memory/space consumption ?

[AryanBakliwal]

Not sure but disabling some data sources might help

[Snehallaldas]

why don't you use Swap file. It will prevent the OOM Killer for terminating process.

[omkhard]

Yeah disabling the sources and excluding few expressions indeed helped, Thanks.