Is there a limit on the number of items you can have in the exclude list?

Discussion related to "Everything" 1.5 Alpha.
Post Reply
DerekZiemba
Posts: 44
Joined: Thu Sep 27, 2018 4:46 pm

Is there a limit on the number of items you can have in the exclude list?

Post by DerekZiemba »

Since well before the holidays (for sure with at least versions 1361, 1363, & 1366), Everything rebuilds the index everytime it's launched. I didn't look into it too much earlier bcus it's only a problem after restarting the PC, and it probably avgs 10days between restarts. Figured it just needed to do it's thing after a restart for some reason.

Well today I changed the UI Scale level in options, which restarts everything.... Indexing Properties is an hour+ ordeal that lags the PC & gives the CPU a 220watt exercise so I've definitely now picked up on it being an issue. There's over 8 million items that the exclude list cuts down to under 1.8 million. Indexing properties for that many items isn't feasible & I've found makes it prone to crashing.
So I run 2 instances. Everything "Actually" that doesn't exclude anything or index properties (is blazing fast) & then the default Everything that excludes most junk and lets me search indexed properties quickly. I noticed the "Actually" instance doesn't ever need to rebuild so I began looking into the differences between them & it came down to the exclude list.

It took a whole lot of time (8million item with 14 custom properties....), but I've verified it only rebuilds on each launch when "Enable exclude list" is checked. I combed through the list and nothing really stands out to me besides the number of entries, 257 rules in total.... And I do recall adding 3 or so rules roughly around the time the issue started. Maybe a `uint8_t` kind of issue??
To test this theory, I converted & combined many of the wildcard entries into a few RegularExpressions. After doing so, I can now launch with exclude list enabled without it doing a full index rebuild.

Is there some kind of overflow thing going on? Or was one of my Wildcard rules somehow breaking things?

Here are my original exclude rules that cause the issue:

Code: Select all

(178 entries)
exclude_folders=$Recycle.Bin,**.indexeddb.blob\**,**\.git\objects\*,**\AppData\Local\Spotify\Data,**\AppData\Local\Spotify\Storage,**\cache\assets,**\Daedalus Mainnet\chain\immutable,**\image\packages\**\node_modules\*,**\node_modules\*\**\node_modules\*,**\node_modules\@storybook\*,**\resources\app\dist\*,**\SettingsV2\*\*,**\share\man\**,**\Steam\appcache\**,**\Steam\logs\**,**\steamapps\common\**\Assets,**\steamapps\common\**\package,**\steamapps\common\**\resource,**\steamapps\common\**\resources,**\steamapps\common\**cache**,**\steamapps\downloading\*,**\tmp\cache,.dropbox.cache,.IdentityService,__pycache__,_cacache,app.asar.unpacked,arm,arm64,C:\$WinREAgent,C:\**\BaseImages\**,C:\**\D3DSCache\*,C:\**\Microsoft\Vault\*,C:\**\User Data\**\Network,C:\**\User Data\**\WebStorage,C:\Config.Msi,C:\Games\AppData.Roaming\Daedalus Mainnet\chain,C:\inetpub,C:\Program Files*\**\Images,C:\Program Files*\**\Resources,C:\Program Files*\Adobe\Acrobat*\*\WebResources,C:\Program Files*\Microsoft Office\*\*\sdxs,C:\Program Files\WindowsApps\**\Assets,C:\Program Files\WindowsApps\**\Images,C:\Program Files\WindowsApps\**\res,C:\Program Files\WindowsApps\**\resources,C:\Program Files\WindowsApps\*\AppxMetadata,C:\Program*\Microsoft*\**\IDE\**\SecurityIssueAnalysis,C:\ProgramData\Docker\windowsfilter,C:\ProgramData\Dropbox,C:\ProgramData\Microsoft\Crypto\RSA,C:\ProgramData\Microsoft\MapData,C:\ProgramData\Microsoft\NetFramework\BreadcrumbStore,C:\ProgramData\Microsoft\Search\Data,C:\ProgramData\Microsoft\Windows Defender,C:\ProgramData\Microsoft\Windows\AppRepository,C:\ProgramData\Microsoft\Windows\Caches,C:\ProgramData\Microsoft\Windows\ClipSVC,C:\ProgramData\Microsoft\Windows\Hyper-V,C:\ProgramData\NVIDIA Corporation\Drs,C:\ProgramData\regid.*,C:\ProgramData\USOPrivate,C:\ProgramData\USOShared,C:\Users\*\.local\**\assets,C:\Users\*\.local\**\images,C:\Users\*\.local\share\gem\ruby\**\.git,C:\Users\*\.local\share\gem\ruby\**\.github,C:\Users\*\.local\share\gem\ruby\**\doc,C:\Users\*\.local\share\gem\ruby\**\fonts,C:\Users\*\.local\share\gem\ruby\**\rdoc,C:\Users\*\.vscode\**\node_modules,C:\Users\*\.vscode\**\pythonFiles,C:\Users\*\.vscode\extensions\**\node_modules,C:\Users\*\.vscode\extensions\**\resources,C:\Users\*\.vscode\extensions\*\dist,C:\Users\*\.vscode\extensions\*\media,C:\Users\*\.vscode\extensions\*\out,C:\Users\*\AppData\**\blobs,C:\Users\*\AppData\**\ep_weather_host,C:\Users\*\AppData\**\IndexedDB,C:\Users\*\AppData\**\Local Storage,C:\Users\*\AppData\**\LocalState,C:\Users\*\AppData\**\Network,C:\Users\*\AppData\**\safebrowsing,C:\Users\*\AppData\**\Session Storage,C:\Users\*\AppData\**\Spotify\*,C:\Users\*\AppData\**\thumbnails,C:\Users\*\AppData\**\User Data\**\_manifest,C:\Users\*\AppData\**\User Data\**\_metadata,C:\Users\*\AppData\**\User Data\**\Extensions\*\**\icons,C:\Users\*\AppData\**\User Data\**\Extensions\*\**\image,C:\Users\*\AppData\**\User Data\**\Extensions\*\**\images,C:\Users\*\AppData\**\User Data\**\Extensions\*\**\img,C:\Users\*\AppData\**\User Data\**\popup\browsers,C:\Users\*\AppData\**\User Data\AutofillStates,C:\Users\*\AppData\**\User Data\Safe Browsing,C:\Users\*\AppData\**\User Data\Snapshots,C:\Users\*\AppData\*\Microsoft\Credentials,C:\Users\*\AppData\*\Microsoft\Office\SolutionPackages,C:\Users\*\AppData\*\Microsoft\Windows\Caches,C:\Users\*\AppData\*\Mozilla\Firefox\Profiles\*\settings\main\quicksuggest,C:\Users\*\AppData\Local\**\node_modules,C:\Users\*\AppData\Local\**\TokenBroker\Cache,C:\Users\*\AppData\Local\Clover\User Data\Default\JumpListIcons*,C:\Users\*\AppData\Local\Dropbox,C:\Users\*\AppData\Local\Microsoft\Media Player,C:\Users\*\AppData\Local\Microsoft\Office\**\Wef,C:\Users\*\AppData\Local\Microsoft\vscode-cpptools,C:\Users\*\AppData\Roaming\*\Network,C:\Users\*\AppData\Roaming\Code\CachedData,C:\Users\*\AppData\Roaming\MAXON,C:\Users\Derek\AppData\Local\Microsoft\Windows\History,C:\Windows\appcompat,C:\Windows\Installer,C:\Windows\Logs\NetSetup,C:\Windows\Microsoft.NET,C:\Windows\Prefetch,C:\Windows\rescache,C:\Windows\ServiceProfiles,C:\Windows\ServiceState,C:\Windows\servicing,C:\Windows\SoftwareDistribution,C:\Windows\System32\CatRoot,C:\Windows\System32\catroot2,C:\Windows\System32\config,C:\Windows\System32\config\systemprofile\AppData,C:\Windows\System32\LogFiles\WMI\RtBackup,C:\Windows\System32\Microsoft\Protect,C:\Windows\System32\Speech_OneCore,C:\Windows\System32\sru,C:\Windows\System32\wbem,C:\Windows\SystemApps\**\Assets,C:\Windows\SystemApps\**\Images,C:\Windows\SystemApps\**\res,C:\Windows\SystemApps\**\resources,C:\Windows\SystemApps\*\AppxMetadata,C:\Windows\WinSxS,cacache,cache2,Cache_Data,CacheStorage,ccache,Code Cache,CryptnetUrlCache,DawnCache,DiagOutputDir,GPUCache,GraphiteDawnCache,GrShaderCache,IndexedDB,INetCache,JumpListIcons,JumpListIconsMostVisited,JumpListIconsRecentClosed,JumpListIconsRecentWorkspacesV2,leveldb,man,man1,man3,man5,man7,NetworkService,OfflineCache,Profile_null,regex:^(?:_(?:locales?)|a(?:f(?:[-_][AZS][EA])?|r(?:[-_][A-Z]{2})?|m[-_]ET|s[-_]IN|z-Latn-AZ)|b(?:g|[gns][-_](?:Latn-)?[BI][ADGN])|c(?:y|a|(?:[ays][-_][CEG][BHSZ](?:-VALENCIA)?))|d(?:a(?:[-_]D[EK])?|e(?:[-_](?:D[EK]|AT|CH|L[IU]))?)|e(?:u(?:la|[-_]ES)?|[ns](?:[-_][A-Z]{2})?|t(?:[-_][EC][EN])?|l(?:[-_]GR)?)|f[ira](?:[-_][A-Z]{2}|l[-_]PH)?|g[adlu](?:[-_][IE][ES])?|h(?:[eru](?:[-_][BIH][ALRU])?|i[-_]IN|y-AM|a-Latn-NG)|i(?:18n|[ts]|nt(?:l|ernationalizations?)|[tsd][-_][IC][HDTS]|g[-_]NG)|ja(?:[-_]JP)?|k(?:[akno](?:[-_][GK][ERZ])?|ok?|m[-_]KH|u-Arab)|l(?:ang(?:uages?)?|egal|ocal(?:es?|i(?:z[ea](tions?)?)|nguistics?)|[tv](?:[-_]L[TV])?|b-LU)|m(?:l-in|i[-_]NZ|s[-_][BM][NY]|t[-_]MT|k[-_]MK)|n(?:l|[lbne][-_][BN][EOLP]|so[-_]ZA)|or-IN|p(?:a(?:-Arab-PK)?|[lt](?:[-_][BP][RLT])?)|r[ou](?:[-_][MR][DOU])?|s(?:[iklv]|[ikl][-_]S[KIL]|[lqv][-_][SA][IE]|r-(?:Cyrl|Latn)-[BCRM][AES])|t(?:ranslations?|[hr]|n[-_]ZA|[rh][-_]T[RH])|u(?:k(?:[-_]UA)?|z[-_]Latn-UZ)|vi[-_]VN|xh-ZA|yo-NG|z(?:oneinfo|h[-_](?:TW|CN|CH[NST]|Han[st])))$,regex:C:\\((?!Users)[\w\s\)\(]+|Users\\\w+?\\AppData)\\.+?\\-?\d+$,ScriptCache,Session Storage,ShaderCache,System Volume Information,terminfo,TFSTemp,VSLogs,WebCache,WebStorage,win-arm64,xamarintvos10,xamarinwatchos10

(79 entries)
exclude_files=_._;LOG;LICENSE;Local State;Network Persistent State;TransportSecurity;LICENSE.txt;MANIFEST-000001;cache.bin;CertificateInformation.dat;*.aux;*.cache;*.cachedmsg;*.DAT.LOG2;*.db-wal;*.etag;*.feedsdb-ms;*.fingerprint;*.forge;*.hyb;*.idx;*.jsonlz4;*.mui;*.mum;*.nlp;*.obj;*.old;*.p;*.pb;*.p7x;*.pack;*.pma;*.pri;*.pyc;*.regtrans-ms;*.rels;*.resources.???;*.ri;*.toc;*.usage;*.vlpset;*.wasm;*.wbcat;*.wbverify;*.winmd;*.WMF;*db-journal;??????.ldb;**\lib\**.a;**\DXCache\*.bin;**\startupCache\*;C:\**\Thunderbird*\.profile\**.msf;C:\Users\*\.local\share\gem\ruby\**\*.html;C:\Users\*\AppData\*\**\*Cache*\**;C:\Users\*\AppData\**\resources\**;C:\Users\*\AppData\**\storage\**;C:\Users\*\AppData\**\User Data\**\LOCK;C:\Users\*\AppData\**\User Data\**\CURRENT;C:\Users\*\AppData\**\User Data\**-journal;C:\Users\*\AppData\**\User Data\**.png;C:\Users\*\AppData\**\User Data\**.svg;C:\Users\*\AppData\**\User Data\**.gif;C:\Users\*\AppData\**\User Data\**.jpg;C:\Users\*\AppData\**\User Data\**.jpeg;C:\Users\*\AppData\**\User Data\**.woff2;C:\Users\*\AppData\**\User Data\**.woff;C:\Program Files*\Reference Assemblies\**.xml;C:\Program Files\WindowsApps\*\**.xml;C:\Program Files\WindowsApps\*\**.xbf;C:\Windows\System32\winevt\Logs\*.evtx;"regex:^\W?[0-9a-fA-F]{8,}\W(?:[0-9a-fA-F]{4}\W){3}[0-9a-fA-F]{12,}\W*?(?:\.(?:tmp|node|lock|val|[xy]ml|vsidx|json))*$";*.tc;*.extra;*.eza;*.scx;*.feature;D:\appliedsystems\products\ezlynx\**.xlsx;D:\appliedsystems\products\ezlynx\**.xls;**\node_modules\**.map

I only changed exclude_folders. Merging many rules into few RegularExpression decreased total rule count to 183.

Code: Select all

(104 entries)
exclude_folders=$Recycle.Bin,**.indexeddb.blob\**,**\.git\objects\*,**\AppData\Local\Spotify\Data,**\AppData\Local\Spotify\Storage,**\cache\assets,**\Daedalus Mainnet\chain\immutable,**\image\packages\**\node_modules\*,**\node_modules\*\**\node_modules\*,**\node_modules\@storybook\*,**\resources\app\dist\*,**\SettingsV2\*\*,**\share\man\**,**\Steam\appcache\**,**\Steam\logs\**,**\steamapps\downloading\*,regex:\\steamapps\\common\\.+\\(?:Assets|package|Images|res(ources?)?|AppxMetadata|appcache)$,**\tmp\cache,.dropbox.cache,.IdentityService,__pycache__,app.asar.unpacked,C:\$WinREAgent,C:\**\D3DSCache\*,C:\**\Microsoft\Vault\*,C:\Config.Msi,C:\Games\AppData.Roaming\Daedalus Mainnet\chain,C:\inetpub,C:\Program Files*\Microsoft Office\*\*\sdxs,regex:\\Program\sFiles.*?\\.+\\(?:Assets|Images|sdxs|(web)?res(ources?)?|AppxMetadata)$,regex:\\WindowsApps\\.+\\(?:Assets|Images|res(ources?)?|AppxMetadata)$,C:\Program*\Microsoft*\**\IDE\**\SecurityIssueAnalysis,C:\ProgramData\Docker\windowsfilter,C:\ProgramData\Dropbox,C:\ProgramData\Microsoft\Crypto\RSA,C:\ProgramData\Microsoft\MapData,C:\ProgramData\Microsoft\NetFramework\BreadcrumbStore,C:\ProgramData\Microsoft\Search\Data,C:\ProgramData\Microsoft\Windows Defender,C:\ProgramData\Microsoft\Windows\AppRepository,C:\ProgramData\Microsoft\Windows\Caches,C:\ProgramData\Microsoft\Windows\ClipSVC,C:\ProgramData\Microsoft\Windows\Containers,C:\ProgramData\Microsoft\Windows\Hyper-V,C:\ProgramData\NVIDIA Corporation\Drs,C:\ProgramData\regid.*,C:\ProgramData\USOPrivate,C:\ProgramData\USOShared,regex:^C:\\Users\\\w+\\\.local\\.+\\(assets|images|fonts|\.git(?:hub)?|r?doc)$,regex:^C:\\Users\\\w+\\\.vscode\\.+\\(?:node_modules|pythonFiles)$,regex:^C:\\Users\\\w+\\\.vscode\\extensions\\.+\\(?:resources|dist|media|out)$,C:\Users\*\AppData\*\Microsoft\Credentials,C:\Users\*\AppData\*\Microsoft\Office\SolutionPackages,C:\Users\*\AppData\*\Microsoft\Windows\Caches,C:\Users\*\AppData\*\Mozilla\Firefox\Profiles\*\settings\main\quicksuggest,C:\Users\*\AppData\Local\**\node_modules,C:\Users\*\AppData\Local\**\TokenBroker\Cache,C:\Users\*\AppData\Local\Clover\User Data\Default\JumpListIcons*,C:\Users\*\AppData\Local\Dropbox,C:\Users\*\AppData\Local\Microsoft\Media Player,C:\Users\*\AppData\Local\Microsoft\Office\**\Wef,C:\Users\*\AppData\Local\Microsoft\vscode-cpptools,C:\Users\*\AppData\Local\Microsoft\Windows\History,C:\Users\*\AppData\Roaming\*\Network,C:\Users\*\AppData\Roaming\Code\CachedData,C:\Users\*\AppData\Roaming\MAXON,C:\Users\*\AppData\Roaming\ExplorerPatcher\ep_weather_host,C:\Windows\appcompat,C:\Windows\Installer,C:\Windows\Logs\NetSetup,C:\Windows\Microsoft.NET,C:\Windows\Prefetch,C:\Windows\rescache,C:\Windows\ServiceProfiles,C:\Windows\ServiceState,C:\Windows\servicing,C:\Windows\SoftwareDistribution,C:\Windows\System32\CatRoot,C:\Windows\System32\catroot2,C:\Windows\System32\config,C:\Windows\System32\config\systemprofile\AppData,C:\Windows\System32\LogFiles\WMI\RtBackup,C:\Windows\System32\Microsoft\Protect,C:\Windows\System32\Speech_OneCore,C:\Windows\System32\sru,C:\Windows\System32\wbem,C:\Windows\WinSxS,CryptnetUrlCache,DiagOutputDir,Profile_null,NetworkService,System Volume Information,terminfo,TFSTemp,VSLogs,regex:(?-i)^[Jj]umpList(?:Cache|Icons(?:Old|Recent(?:Closed|WorkspacesV2)?|MostVisited|TopSites)?)?$,regex:(?-i)^C:\\((?!Users)[\w\s\)\(]+|Users\\\w+?\\AppData)\\.+?\\-?\d+$,regex:(?-i)^man[0-9]?$,regex:(?-i:(?:\\(?:AppData(?:\.\w+)?|User\sData|steamapps|(?:My\s)?[Gg]ames))\\)(?:.+\\)?(?i:_m(?:anifest|etadata)|icons|im(?:g|ages?)|popup\\browsers|AutofillStates|Safe\sBrowsing|Snapshots|thumbnails|safebrowsing|blobs|LocalState|Network)$,regex:(?-i:^C:|(?:\\(?:\.[a-z]+|AppData(?:\.\w+)?|User\sData|Apps|steamapps|(?:My\s)?[Gg]ames))\\)(?:.+\\)?(?i:[a-f0-9]{2}|(?:_?ca?|Code\s|CryptnetUrl|Dawn|G(PU|r(?:Shader|aphiteDawn))|INet|Offline|Script|web|shader-?)Cache|(?:Indexed|Level)\s?DB|(?:Web|(?:Local|Session)\s|blob_|Cache)Storage|Cache_Data)$,regex:(?i)\bARM(?:64)?e?(?:c|l|abi)?(?:v[46]i?)?(?:\b|$),regex:(?i)^(?:_(?:locales?)|a(?:f(?:[-_][AZS][EA])?|r(?:[-_][A-Z]{2})?|m[-_]ET|s[-_]IN|z-Latn-AZ)|b(?:g|[gns][-_](?:Latn-)?[BI][ADGN])|c(?:y|a|(?:[ays][-_][CEG][BHSZ](?:-VALENCIA)?))|d(?:a(?:[-_]D[EK])?|e(?:[-_](?:D[EK]|AT|CH|L[IU]))?)|e(?:u(?:la|[-_]ES)?|[ns](?:[-_][A-Z]{2})?|t(?:[-_][EC][EN])?|l(?:[-_]GR)?)|f[ira](?:[-_][A-Z]{2}|l[-_]PH)?|g[adlu](?:[-_][IE][ES])?|h(?:[eru](?:[-_][BIH][ALRU])?|i[-_]IN|y-AM|a-Latn-NG)|i(?:18n|[ts]|nt(?:l|ernationalizations?)|[tsd][-_][IC][HDTS]|g[-_]NG)|ja(?:[-_]JP)?|k(?:[akno](?:[-_][GK][ERZ])?|ok?|m[-_]KH|u-Arab)|l(?:ang(?:uages?)?|egal|ocal(?:es?|i(?:z[ea](tions?)?)|nguistics?)|[tv](?:[-_]L[TV])?|b-LU)|m(?:l-in|i[-_]NZ|s[-_][BM][NY]|t[-_]MT|k[-_]MK)|n(?:l|[lbne][-_][BN][EOLP]|so[-_]ZA)|or-IN|p(?:a(?:-Arab-PK)?|[lt](?:[-_][BP][RLT])?)|r[ou](?:[-_][MR][DOU])?|s(?:[iklv]|[ikl][-_]S[KIL]|[lqv][-_][SA][IE]|r-(?:Cyrl|Latn)-[BCRM][AES])|t(?:ranslations?|[hr]|n[-_]ZA|[rh][-_]T[RH])|u(?:k(?:[-_]UA)?|z[-_]Latn-UZ)|vi[-_]VN|xh-ZA|yo-NG|z(?:oneinfo|h[-_](?:TW|CN|CH[NST]|Han[st])))$,regex:(?i)^Xamarin(?:\.?(?:mac|(?:i|tv|watch)os|Android)[0-9]*)?(?:(?:app|bindings|library|wear|crossplat|form|ui|test)+(?:\.zip)?)?$,regex:^C:\\Windows\\SystemApps\\.+\\(?:Assets|Images|res(ources)?|Appx?Metadata)$

Unfortunately, I didn't realize until much later that the rule count was only 2 over 255. If I had, I would have been careful to remove only 3 & see if it would rebuild. But now that it's finally rebuilt again & working, I loathe the thought of potentially causing another.
Post Reply