View Issue Details

IDProjectCategoryView StatusLast Update
0005832SUMoRefactoringpublic2020-02-23 15:51
Reporterwadim-al Assigned ToKyle_Katarn  
PrioritylowSeverityminorReproducibilityalways
Status confirmedResolutionopen 
Product Version5.10.8 
Target Version5.10.x 
Summary0005832: Suggestions for improving disk scan and "loading" step performance.
DescriptionWhen scanning for software, the utilization of 4-core CPU is:
SSD scan 32%
HDD scan 19%

Disk utilization:
SSD scan 6%
HDD scan 34%

"Loading" step has similar problem utilizing only 1 thread (30% percent CPU) and updating file counter each time instead of once in a second).

SUGGESTIONS:
1) When scanning use more that 1 thread (configurable, because HDD scan might not become faster, but it will definitely speed up scanning on SSD at least)
2) Do not update the text label containing currently scanned path on every directory/file because that possibly causes unneeded CPU consumption. Should be updated once in a second.



Steps To Reproduce1. Fresh install SUMo
2. Configure to scan a whole volume on SSD
3. Configure to scan another whole volume on HDD
4. Monitor Disk and CPU performance (utilization)
5. Start initial scan.
Tagsoptimization, Performance, scan

Relationships

related to 0005667 assignedKyle_Katarn Accessibility improvement - blinking & frequency 

Activities

wadim-al

2020-01-27 08:01

reporter  

Taskmgr_2020-01-27_09-42-06.png (115,893 bytes)   
Taskmgr_2020-01-27_09-42-06.png (115,893 bytes)   
Taskmgr_2020-01-27_09-44-31.png (119,761 bytes)   
Taskmgr_2020-01-27_09-44-31.png (119,761 bytes)   

Kyle_Katarn

2020-01-27 20:35

administrator   ~0003841

Good idea. Wil lbe tested soon.

wolf

2020-02-04 23:01

reporter   ~0003849

How do you get this Windows task manager configuration?

I had similar configurations in times of Windows XP or before but didn't get such a configuration with Windows 7 or later. Your screenshots report that you're using a dual-core CPU with multithreading. So you've two active threads per core makeing a total of 4 active threads on a dual-core CPU.

I have different priorities leading to different constraints. So I don't want such an implementation unreflected. It needs to be configurable in order to match different users priorities, much more than proposed by you. When running on my quad-core CPU with multithreading, I've to wait very long for SUMo startup and SUMo scan, and long for SUMo check operation. And if done in a certain manner, I would be blocked for more than an hour on such a device as I've configured SUMo to scan intensively although not to scan any whole volume.

On your first suggestion, it is necessary to have it configurable to which extent CPU resources are used. Maximum usage will create inacceptable side-effects. So if increasing CPU usage, a maximum should be configurable in order not to starve any other service or to get killed by some monitoring and device health service. On your second suggestion, the current behaviour should remain possible and allowing your proposed behaviour by configuration. And the update interval of the state display should be configurable similar to Windows task manager or Linux/Unix top. High update interval shall remain to display every directory resp. file while various slower display updates may only report the then current directory with possible intervals of 1, 5 and 8 seconds. Or do you prefer any interval configurable in steps of seconds?

I'm not sure if I understood your description further. The text tables don't seem to be accurate. I guess that you reported some observation at some moment, not a real monitoring of exact values and averages nor its relation of processing steps of SUMo. (The latter is probably out of your scope and more for the task of the tool publisher.)

Please provide more info about your configuration with what you mean for the steps to reproduce. Did you configure standard scan or deep scan? Did you mean to scan the whole volume or did you also configure some exceptions like temp folders? Which of those volumes you selected is the system volume or did you experiment only on other volumes excluding the system volume?

wadim-al

2020-02-05 07:20

reporter   ~0003850

This is default Task Manager from Windows 10 x64 1903
My screenshot reports 4-core CPU, single logical processor per core, so no HT. This is Intel i3 8350k @ 4GHz.
32% CPU utilization might mean that 25% (1/4 cores) is taken by most demanding thread, and the rest is Task Manager, other apps or GUI updating.
I will add further comments later.

wadim-al

2020-02-05 08:00

reporter   ~0003851

Retested again
Steps:
0. Restart PC so no hdd caching is in memory, so HDD scan will run not from cache.
1. Configured in-depth scan in settings
2. Configured additional folders as C D (which are on the same ssd), E which is on HDD
3. No exclusions were configured
4. Started scan F6

Scan step SSD volumes (small window with stop button and not fully visible path):
CPU utilization for SUMo process was 25-30%, for me this is indication of single-threaded scanning process.
SSD active time was under 10%, so SSD has potential to perform faster and is just waiting most of the time.

HDD volume:
CPU utilization for SUMo process dropped to 12-25 (18% average)
HDD active time was about 50% this time.

Loading step (progressbar at the bottom):
CPU utilization for SUMo process was 25-30% for apps on SSD, and dropped to 12-25% when listing apps on HDD

Suggestions:
1. Configurable scanning thread count - from 1 to 4 (or maximum thread count detected)
2. Multithread scanning only for SSD. I believe 2+ thread scanning for HDD will make everything worse because of low HDD IOPS when doing random reads. At all times when accessing HDD, the CPU is not the bottleneck.
3. SSD scan should be run with profiler attached to see which calls are most demanding: scanning, or updating the GUI on each file. (should be done by dev)
4. I don't have any idea on whether updating should occur on dir change, or just limited to 1 per second, and full path left. Depends on current actual GUI refresh call count and profiling results (from 3rd suggestion). Some other apps (antivirus scanners) are just doing 1 update per second.

wadim-al

2020-02-05 08:03

reporter   ~0003852

No profiling/or monitoring capture was done. Only observations in task manager in middle of the scanning process.

Kyle_Katarn

2020-02-05 20:00

administrator   ~0003853

Thanks ! I'll do a debugger based profiling and i'll post potential benefits here before validating the change.

wadim-al

2020-02-14 08:55

reporter   ~0003854

The following tool, run with admin privileges in commandline without arguments, can be used to clean HDD cache, to make tests show correct results.
https://wj32.org/wp/software/empty-standby-list/

Issue History

Date Modified Username Field Change
2020-01-27 08:01 wadim-al New Issue
2020-01-27 08:01 wadim-al Tag Attached: optimization
2020-01-27 08:01 wadim-al Tag Attached: Performance
2020-01-27 08:01 wadim-al Tag Attached: scan
2020-01-27 08:01 wadim-al File Added: Taskmgr_2020-01-27_09-42-06.png
2020-01-27 08:01 wadim-al File Added: Taskmgr_2020-01-27_09-44-31.png
2020-01-27 20:35 Kyle_Katarn Assigned To => Kyle_Katarn
2020-01-27 20:35 Kyle_Katarn Status new => confirmed
2020-01-27 20:35 Kyle_Katarn Note Added: 0003841
2020-01-27 20:36 Kyle_Katarn Target Version => 5.10.9
2020-01-27 20:36 Kyle_Katarn Description Updated View Revisions
2020-01-27 20:36 Kyle_Katarn Steps to Reproduce Updated View Revisions
2020-01-27 20:36 Kyle_Katarn Relationship added related to 0005667
2020-02-04 23:01 wolf Note Added: 0003849
2020-02-05 07:20 wadim-al Note Added: 0003850
2020-02-05 08:00 wadim-al Note Added: 0003851
2020-02-05 08:03 wadim-al Note Added: 0003852
2020-02-05 20:00 Kyle_Katarn Note Added: 0003853
2020-02-14 08:55 wadim-al Note Added: 0003854
2020-02-23 15:51 Kyle_Katarn Target Version 5.10.9 => 5.10.x