View Issue Details
|ID||Project||Category||View Status||Date Submitted||Last Update|
|0005832||SUMo||Refactoring||public||2020-01-27 08:01||2020-02-23 15:51|
|Summary||0005832: Suggestions for improving disk scan and "loading" step performance.|
|Description||When scanning for software, the utilization of 4-core CPU is:|
SSD scan 32%
HDD scan 19%
SSD scan 6%
HDD scan 34%
"Loading" step has similar problem utilizing only 1 thread (30% percent CPU) and updating file counter each time instead of once in a second).
1) When scanning use more that 1 thread (configurable, because HDD scan might not become faster, but it will definitely speed up scanning on SSD at least)
2) Do not update the text label containing currently scanned path on every directory/file because that possibly causes unneeded CPU consumption. Should be updated once in a second.
|Steps To Reproduce||1. Fresh install SUMo|
2. Configure to scan a whole volume on SSD
3. Configure to scan another whole volume on HDD
4. Monitor Disk and CPU performance (utilization)
5. Start initial scan.
|Tags||optimization, Performance, scan|
||Good idea. Wil lbe tested soon.|
How do you get this Windows task manager configuration?
I had similar configurations in times of Windows XP or before but didn't get such a configuration with Windows 7 or later. Your screenshots report that you're using a dual-core CPU with multithreading. So you've two active threads per core makeing a total of 4 active threads on a dual-core CPU.
I have different priorities leading to different constraints. So I don't want such an implementation unreflected. It needs to be configurable in order to match different users priorities, much more than proposed by you. When running on my quad-core CPU with multithreading, I've to wait very long for SUMo startup and SUMo scan, and long for SUMo check operation. And if done in a certain manner, I would be blocked for more than an hour on such a device as I've configured SUMo to scan intensively although not to scan any whole volume.
On your first suggestion, it is necessary to have it configurable to which extent CPU resources are used. Maximum usage will create inacceptable side-effects. So if increasing CPU usage, a maximum should be configurable in order not to starve any other service or to get killed by some monitoring and device health service. On your second suggestion, the current behaviour should remain possible and allowing your proposed behaviour by configuration. And the update interval of the state display should be configurable similar to Windows task manager or Linux/Unix top. High update interval shall remain to display every directory resp. file while various slower display updates may only report the then current directory with possible intervals of 1, 5 and 8 seconds. Or do you prefer any interval configurable in steps of seconds?
I'm not sure if I understood your description further. The text tables don't seem to be accurate. I guess that you reported some observation at some moment, not a real monitoring of exact values and averages nor its relation of processing steps of SUMo. (The latter is probably out of your scope and more for the task of the tool publisher.)
Please provide more info about your configuration with what you mean for the steps to reproduce. Did you configure standard scan or deep scan? Did you mean to scan the whole volume or did you also configure some exceptions like temp folders? Which of those volumes you selected is the system volume or did you experiment only on other volumes excluding the system volume?
This is default Task Manager from Windows 10 x64 1903
My screenshot reports 4-core CPU, single logical processor per core, so no HT. This is Intel i3 8350k @ 4GHz.
32% CPU utilization might mean that 25% (1/4 cores) is taken by most demanding thread, and the rest is Task Manager, other apps or GUI updating.
I will add further comments later.
0. Restart PC so no hdd caching is in memory, so HDD scan will run not from cache.
1. Configured in-depth scan in settings
2. Configured additional folders as C D (which are on the same ssd), E which is on HDD
3. No exclusions were configured
4. Started scan F6
Scan step SSD volumes (small window with stop button and not fully visible path):
CPU utilization for SUMo process was 25-30%, for me this is indication of single-threaded scanning process.
SSD active time was under 10%, so SSD has potential to perform faster and is just waiting most of the time.
CPU utilization for SUMo process dropped to 12-25 (18% average)
HDD active time was about 50% this time.
Loading step (progressbar at the bottom):
CPU utilization for SUMo process was 25-30% for apps on SSD, and dropped to 12-25% when listing apps on HDD
1. Configurable scanning thread count - from 1 to 4 (or maximum thread count detected)
2. Multithread scanning only for SSD. I believe 2+ thread scanning for HDD will make everything worse because of low HDD IOPS when doing random reads. At all times when accessing HDD, the CPU is not the bottleneck.
3. SSD scan should be run with profiler attached to see which calls are most demanding: scanning, or updating the GUI on each file. (should be done by dev)
4. I don't have any idea on whether updating should occur on dir change, or just limited to 1 per second, and full path left. Depends on current actual GUI refresh call count and profiling results (from 3rd suggestion). Some other apps (antivirus scanners) are just doing 1 update per second.
||No profiling/or monitoring capture was done. Only observations in task manager in middle of the scanning process.|
||Thanks ! I'll do a debugger based profiling and i'll post potential benefits here before validating the change.|
The following tool, run with admin privileges in commandline without arguments, can be used to clean HDD cache, to make tests show correct results.
|2020-01-27 08:01||wadim-al||New Issue|
|2020-01-27 08:01||wadim-al||Tag Attached: optimization|
|2020-01-27 08:01||wadim-al||Tag Attached: Performance|
|2020-01-27 08:01||wadim-al||Tag Attached: scan|
|2020-01-27 08:01||wadim-al||File Added: Taskmgr_2020-01-27_09-42-06.png|
|2020-01-27 08:01||wadim-al||File Added: Taskmgr_2020-01-27_09-44-31.png|
|2020-01-27 20:35||Kyle_Katarn||Assigned To||=> Kyle_Katarn|
|2020-01-27 20:35||Kyle_Katarn||Status||new => confirmed|
|2020-01-27 20:35||Kyle_Katarn||Note Added: 0003841|
|2020-01-27 20:36||Kyle_Katarn||Target Version||=> 5.10.9|
|2020-01-27 20:36||Kyle_Katarn||Description Updated||View Revisions|
|2020-01-27 20:36||Kyle_Katarn||Steps to Reproduce Updated||View Revisions|
|2020-01-27 20:36||Kyle_Katarn||Relationship added||related to 0005667|
|2020-02-04 23:01||wolf||Note Added: 0003849|
|2020-02-05 07:20||wadim-al||Note Added: 0003850|
|2020-02-05 08:00||wadim-al||Note Added: 0003851|
|2020-02-05 08:03||wadim-al||Note Added: 0003852|
|2020-02-05 20:00||Kyle_Katarn||Note Added: 0003853|
|2020-02-14 08:55||wadim-al||Note Added: 0003854|
|2020-02-23 15:51||Kyle_Katarn||Target Version||5.10.9 => 5.11.x|