Clamav - clamd continuous out-of-memory conditions

OMV needs to retest their clamd scan configuration, which unfortunately due to Clamav, can become an out-of-control memory hog.

The tiny VM (configuration below) happily serves a bunch of users, but on average, two simultaneously without any glaring issues and certainly

none related to memory. This even as users are manually migrating files/folders from old storage onto this updated OMV node.

Until the dreaded ClamAV is turned on.

ClamAV (ClamAV 1.0.8/27541) has successfully taken-over my laptop (12 core, 16GB Memory) scanning approximately 80GB of data in the past as an FYI.

Any sort of mis-configuration will wildly takeover any system and in my case, crash our tiny OMV VM.

I've checked the forum for any appropriate solution and unfortunately, I see nothing practical, admittedly, may be due to my search results.

Please re-test the clamd.conf configuration as it will not just affect memory but will also bog down all assigned/installed CPUs.

Version

7.6.0-1 (Sandworm)

Processor
Intel(R) Celeron(R)

Kernel
Linux 6.1.0-31-amd64

Other

VM, 2 CPU cores, 5GB Memory

Absolutely!

Note: anything below that starts with a '$' is just a variable that is set as part of a script, so substitute the file and/or directory as needed.

This scans approx. 8GB (reported in error as 80GB above) of data excluding a whole bunch of filetypes and as per my last run ends-up scanning approx. 1 GB of data (see below).

Here is the clamscan command used on my Laptop:

clamscan -i -o -r --official-db-only=yes -l $LOGFILE --cross-fs=no --follow-dir-symlinks=0 --follow-file-symlinks=0 --copy=$COPYDIR --detect-pua=yes --phishing-sigs=yes --quiet --phishing-scan-urls=yes --heuristic-alerts=yes --remove=no --exclude-dir=/.*/* --exclude-dir=/Filerxx/ --exclude-dir=/Fileryy/ --exclude-dir=/Downloads/ClamAV/* --exclude=*-vc --exclude='\\.(xba.*|crypt*|obb|sdtid|hc|dcv|sdoc|acsm|xlsl|kdb|kdbx|tc|hds|db|log|fve|efs|idb|iso|mar|mdf|accdt|accde|accft|odb|mdb|sdf|dbf|nsf|ndf|db2|tmd|jet|ns3|musicdb|file|vmdk|vdi)$' $DIRTOSCAN;

...and here are the results:

----------- SCAN SUMMARY -----------

Known viruses: 8720422

Engine version: 1.0.8

Scanned directories: 0 <-- this is actually incorrect, since I get a warning (below) that shows a sub-directory item being scanned, potential bug!

Scanned files: 45

Infected files: 0

Total errors: 1

Data scanned: 1014.11 MB

Data read: 8034.51 MB (ratio 0.13:1)

Time: 275.852 sec (4 m 35 s)

Start Date: 2025:03:11 17:16:28

End Date: 2025:03:11 17:21:04

WARNING: /home/xxxxxxxx/Pictures/variety-copied-wallpaper-a7c2097f1ec23a684dce4ba3b0ca244d.jpg: Can't access file

Oops! Noticed a bug and an opportunity in my command. Here are the corrections:

clamscan -i -o -r --official-db-only=yes -l $LOGFILE --cross-fs=no --follow-dir-symlinks=0 --follow-file-symlinks=0 --copy=$COPYDIR --detect-pua=yes --phishing-sigs=yes --quiet --phishing-scan-urls=yes --heuristic-alerts=yes --remove=no --exclude-dir='$DIRTOSCAN/(\.*/*|Filerxx/*|Fileryy/*|Downloads/ClamAV/*)' --exclude=*-vc --exclude='\\.(xba|xba.*|crypt*|obb|sdtid|hc|dcv|sdoc|acsm|xlsl|kdb|kdbx|tc|hds|db|log|fve|efs|idb|iso|mar|mdf|accdt|accde|accft|odb|mdb|sdf|dbf|nsf|ndf|db2|tmd|jet|ns3|musicdb|file|vmdk|vdi)$' $DIRTOSCAN;

The odd thing is that you have to play with the extensions to exclude (--exclude=). I also find that anything that has an '*' is buggy. In some cases they will still get scanned.

Also, in actuality, the scan did involve ~80G of data and consumed ~1.4GB (one point four) out of 16G of memory while other things were being done on the laptop alongside the scan. It didn't takeover the machine at anytime. Also, ~90% of 10% CPU utilization on a twelve-thread processor, on average.

However, the scan is still going and if I have the time, I'll post the stats.

Hope this helps!!

And, here is the scan summary results of the above command:

----------- SCAN SUMMARY -----------

Known viruses: 8720422

Engine version: 1.0.8

Scanned directories: 13237

Scanned files: 238865

Infected files: 3

Data scanned: 20842.83 MB

Data read: 95560.34 MB (ratio 0.22:1)

Time: 2489.782 sec (41 m 29 s)

Start Date: 2025:03:11 18:55:06

End Date: 2025:03:11 19:36:36

It actually scanned ~21GB out of ~95GB of data and took 41 minutes.

Here also is my clamd.conf file in case that's needed:

cat /etc/clamav/clamd.conf

#Automatically Generated by clamav-daemon postinst

#To reconfigure clamd run #dpkg-reconfigure clamav-daemon

#Please read /usr/share/doc/clamav-daemon/README.Debian.gz for details

LocalSocket /var/run/clamav/clamd.ctl

FixStaleSocket true

LocalSocketGroup clamav

LocalSocketMode 666

# TemporaryDirectory is not set to its default /tmp here to make overriding

# the default with environment variables TMPDIR/TMP/TEMP possible

User clamav

ScanMail true

ScanArchive true

ArchiveBlockEncrypted false

MaxDirectoryRecursion 15

FollowDirectorySymlinks false

FollowFileSymlinks false

ReadTimeout 180

MaxThreads 12

MaxConnectionQueueLength 15

LogSyslog false

LogRotate true

LogFacility LOG_LOCAL6

LogClean false

LogVerbose false

PreludeEnable no

PreludeAnalyzerName ClamAV

DatabaseDirectory /var/lib/clamav

OfficialDatabaseOnly false

SelfCheck 3600

Foreground false

Debug false

ScanPE true

MaxEmbeddedPE 10M

ScanOLE2 true

ScanPDF true

ScanHTML true

MaxHTMLNormalize 10M

MaxHTMLNoTags 2M

MaxScriptNormalize 5M

MaxZipTypeRcg 1M

ScanSWF true

ExitOnOOM false

LeaveTemporaryFiles false

AlgorithmicDetection true

ScanELF true

IdleTimeout 30

CrossFilesystems true

PhishingSignatures true

PhishingScanURLs true

PhishingAlwaysBlockSSLMismatch false

PhishingAlwaysBlockCloak false

PartitionIntersection false

DetectPUA false

ScanPartialMessages false

HeuristicScanPrecedence false

StructuredDataDetection false

CommandReadTimeout 30

SendBufTimeout 200

MaxQueue 100

ExtendedDetectionInfo true

OLE2BlockMacros false

AllowAllMatchScan true

ForceToDisk false

DisableCertCheck false

DisableCache false

MaxScanTime 120000

MaxScanSize 100M

MaxFileSize 25M

MaxRecursion 16

MaxFiles 10000

MaxPartitions 50

MaxIconsPE 100

PCREMatchLimit 10000

PCRERecMatchLimit 5000

PCREMaxFileSize 25M

ScanXMLDOCS true

ScanHWP3 true

MaxRecHWP3 16

StreamMaxLength 25M

LogFile /var/log/clamav/clamav.log

LogTime true

LogFileUnlock false

LogFileMaxSize 0

Bytecode true

BytecodeSecurity TrustSigned

BytecodeTimeout 60000

OnAccessMaxFileSize 5M

Btw. I also paused the scan on the server until this gets fixed. I believe there may be something up with the multiscan option in the scheduled scans task - not saying I'm right. I'll test this if I get the time.

Sorry for any mistakes, but I hope this helps for a next release!

Update:

Zee bugs in clamscan continue. The command option from above,

--exclude-dir='$DIRTOSCAN/(\.*/*|Filerxx/*|Fileryy/*|Downloads/ClamAV/*)'

seems to be ignored or not working as it should. Although the "Downloads/ClamAV/*" is excluded, it continues to scan all of the folders anyway.

The only workaround it seems is to code --exclude-dir on separate lines.

--exclude-dir=$DIRTOSCAN/.config/ \

--exclude-dir=$DIRTOSCAN/.local/ \

--exclude-dir=$DIRTOSCAN/.var/ \

--exclude-dir=$DIRTOSCAN/Filerxx/ \

--exclude-dir=$DIRTOSCAN/Fileryy/ \

--exclude-dir=$DIRTOSCAN/Downloads/ClamAV/ \

Try to also avoid using '*' (asterisk).

The other situation is that infected files are copied to a quarantine sub-folder in /Downloads/ClamAV/ which gets re-scanned because the --exclude-dir doesn't work. Which means the one file that is quarantined (in my case) gets copied over-and-over again and somehow adds another 15GB of data to the quarantine sub-folder until somehow the scan process gives up. And, this one file (in my case) is a file-type that is set to be excluded from the scan. Clamscan is definitely buggy or my code is buggy - I'm not sure. Below is the result of the latest scan with the updated code changes:

----------- SCAN SUMMARY -----------

Known viruses: 8721510

Engine version: 1.0.8

Scanned directories: 595

Scanned files: 6596

Infected files: 0

Total errors: 7

Data scanned: 4613.92 MB

Data read: 75152.67 MB (ratio 0.06:1)

Time: 746.281 sec (12 m 26 s)

Start Date: 2025:03:28 15:05:38

End Date: 2025:03:28 15:18:04

The amount of data (75152.67 MB) aligns with what's currently on my disk. And with the new code changes, only a small portion (4613.92 MB) of this data is scanned due to all the exclusions, I guess.

From above we see that a total of 7 errors occurred. More weirdness. As I understand it, the --quiet option is supposed to print any errors, but didn't receive any such thing in the output log. From clamscan --help:

--quiet Only output error messages

I also checked the quarantine sub-directory and did not find the one infected file there, so this, I hope, confirms the new --exclude-dir approach is working.

-----------------------

New Code - for those who want to try:

clamscan -i -o -r --official-db-only=yes -l $LOGFILE --cross-fs=no --follow-dir-symlinks=0 --follow-file-symlinks=0 --copy=$COPYDIR \

--detect-pua=yes --phishing-sigs=yes --quiet --phishing-scan-urls=yes --heuristic-alerts=yes --remove=no \

--exclude-dir=$DIRTOSCAN/.config/ \

--exclude-dir=$DIRTOSCAN/.local/ \

--exclude-dir=$DIRTOSCAN/.var/ \

--exclude-dir=$DIRTOSCAN/Filerxx/ \

--exclude-dir=$DIRTOSCAN/Fileryy/ \

--exclude-dir=$DIRTOSCAN/Downloads/ClamAV/ \

--exclude='\\.(-vc|xba|crypt|obb|sdtid|hc|dcv|sdoc|acsm|xlsl|kdb|kdbx|tc|hds|db|log|fve|efs|idb|iso|mar|mdf|accdt|accde|accft|odb|mdb|sdf|dbf|nsf|ndf|db2|tmd|jet|ns3|musicdb|file|vmdk|vdi)$' $DIRTOSCAN;

Hope this helps!!

Final Update

Un-checking "Multiscan" for both of my scheduled scan jobs with my minimal 5GB memory allocation seems to have done the trick.

Based on both scans performed on Saturday and Monday past, the CPU-User averaged about 80%, and the memory averaged around 1.6G.

The only issue with removing Multiscan is that the scan process could take longer due to the amount of storage (in my case approx. 300 & 120GB). But, with the 'excludes' in place the actual scanned GB's for both drives was ~70GB. The 'excludes' are important and will help reduce the amount of time. The little VM didn't hang or crash.

Also, using clamscan (with the aforementioned 'excludes') on my laptop, the scan barely impacted the performance and finished with success:

----------- SCAN SUMMARY -----------

Known viruses: 8721685

Engine version: 1.0.8

Scanned directories: 596

Scanned files: 6552

Infected files: 0

Data scanned: 4649.95 MB

Data read: 75752.24 MB (ratio 0.06:1)

Time: 741.102 sec (12 m 21 s)

Start Date: 2025:04:11 14:00:01

End Date: 2025:04:11 14:12:22

ClamAV is great, but needs some fine tuning and better documentation.

Hope this helps!!