Slow to query after sitting for a few hours

[DerekZiemba]

If I haven't performed a query in Everything for an hour or so (ex: window open but haven't changed the query for a while) or if it's been sitting in the background and I then change the query or open a new window it will get stuck `Querying...` for up to a minute. This never happened on `v1.4`. I've been using 1.5 for months and it's been this way for months.
Is this a known issue?

The database size is 554MB & the screenshot shows the # of items it's dealing with

Capture.PNG (9.54 KiB) Viewed 8976 times

Attached are `Everything-1.5a.ini`, `Temporary Excludes-1.5a.csv`, & `Filters-1.5a.csv`

[void]

Thanks for the bug report DerekZiemba,

This can occur if Everything is paged to disk.
Searching for content (eg: Text Plain Line Count / MD5) may exacerbate the issue.

It might be the Text Plain Line Count column and the MD5 column.

Please try removing these column from all your filters and also removing it from the current result list.
Does the issue persist?



Note: The Text Plain Line Count property will read all the file content to calculate the number of lines.

[void]

To prevent Everything from being paged to disk (not recommended):

• In Everything, type in the following search and press ENTER:
  /min_working_set_size=0xfffffffffffffffe
• Type in the following search and press ENTER:
  /max_working_set_size=0xfffffffffffffffe
• Type in the following search and press ENTER:
  /virtual_lock=1
• Type in the following search and press ENTER:
  /restart

The default settings are:
/min_working_set_size=0
/max_working_set_size=0
/virtual_lock=0


min_working_set_size
max_working_set_size
virtual_lock

[DerekZiemba]

Searching for content (eg: Text Plain Line Count / MD5) may exacerbate the issue.
It might be the Text Plain Line Count column and the MD5 column.

The MD5 column is only present in the "Executable" category and is not an indexed property. Searches with that column present do not seem to be impacted, it just slowly populates the MD5 column in the background for the items currently scrolled in to view, which is nice.
The "Plain Text Line Count" is under most categories and is indexed according to the rules below. Would only removing the column be enough? Also, I really like this feature...

Code: Select all

Maximum size: 1800KB
Include only files:
*.txt;*.ini;*.md;*.bat;*.cmd;*.sh;*.csv;*.json;*.config;*.xml;*.yml;*.js;*.jsx;*.jsm;*.ts;*.tsx;
*.css;*.htm;*.html;*.ps1;*.psm1;*.psd1;*.cs;*.vb;*.rb;*.c;*.cc;*.cpp;*.cxx;*.h;*.hpp;*.hx;*.java;
Exclude folders:
C:\Windows\;C:\ProgramData\;C:\Program Files\;C:\Program Files (x86)\;C:\msys64\;\AppData\;\node_modules\;
\packages\;\resources\;\*cache*\;\tmp\;\temp\;\.git\;\.svn\;\obj\;\.nuget\;\Nuget\;\NuGetFallbackFolder\;
\.dotnet\;\Dictionary\;\Help\;\EULA\;\Legal\;\licenses\;\Translation\;\intl\;\Localiazation\;
\locale\;\_locale\;\locales\;\_locales\;\locales-src\;\lang\;\langpack\;\language*\;
\de\;\de-DE\;\en_CA\;\en_GB\;\es\;\es-ES\;\fr\;\fr-FR\;\fr_CA\;\it\;\it-IT\;\ja\;\ja-JP\;\ko\;
\ko-KR\;\pl\;\pt-BR\;\ru\;\ru-RU\;\tr\;\zh-CHS\;\zh-CHT\;\zh-CN\;\zh-TW\;\zh-Hans\;\zh-Hant\

Regarding paging to disk:
I have 32GB of ram so haven't considered that. After checking process statistics, paging may be a very likely culprit. I can already see the "Working set" is significantly less than "Private bytes", suggesting 780mb of the 941mb has been paged out.
I like this process more than the others, so I'm going to do the "(not recommended)" thing and hope the system can just find some other process to page out.
I wouldn't call this typical utilization, it actually surprised me to see how little resources were remaining. Apparently having tabs open to charts on pro.coinbase.com causes msedge to eat ream. Usually I see something like `Commit charge: ~28GB | Physical memory: ~18GB`.


BTW, shouldn't the process paging back in be relatively quick? The time it takes to "Query" when this happens is substantial. A fresh installation using default settings (no indexed properties/content) can build its index and perform its first query in possibly less time.

[void]

Thanks for your reply DerekZiemba,

The MD5 column should be fine.

The indexed Plain Text Line Count property should be fine.

It does appear Everything is paged to disk (high private bytes, low working set size)
Is Everything still indexing properties? (status is shown on the right side of the statusbar)



Debug logs might help identify the issue:

• In Everything 1.5, from the Tools menu, under the Debug submenu, check Console.
• Leave Everything running in the background for an hour or so.
• Open a new Everything window.
• After you get a long Querying.... status, what is shown in the debug console? (right click the console window caption and under the Edit menu, click Select All, right click the selected text to copy it to the clipboard)
• Please send the debug output to support@voidtools.com

[DerekZiemba]

In the screenshotted scenario Everything was actually working normally without issue.

The issue is actually very hard to reproduce or deliberately cause. The long "Querying..." episodes happen randomly once then afterwards the frequency of it reoccurring skyrockets or occur nearly every time it has been left unused for a few dozens of minutes. Once it reaches peak annoyance and I consider posting about the issue, it just kinda stops doing it for a while again. I made a post this time only because of the number of prior episodes and was curious if anyone else has experienced this.

The issue has not re-occurred in the past few days. When it does happen the UI remains responsive so I believe I'll be able to open the debug console. When it's stuck in the "Querying..." state no results are displayed. Any newly opened windows will also display "Querying..." and if the search query on any previously opened windows is changed, they too will get stuck "Querying...". After some time, all windows stuck "Querying..." resolve simultaneously.
If I'm remember correctly, Everything's CPU usage while "Querying..." isn't above normal. Perhaps an issue with a synchronization primitive?
-------------
Note that since the last post I performed the steps mentioned to prevent it being paged to disk. I also deleted "Everything-1.5a.db" after realizing I haven't tried rebuilding the database.
Upon next launch it rebuilt the db and spent a while re-indexing properties. The new database size is down to 472mb from 554mb, which I found odd because shouldn't it contain the same data as before?

The last time the database was rebuilt likely would have been due to the release of 1.5.0.1285, when it looks like I went from 1282 to 1289. I do remember thinking that since the DB was being rebuilt, "hmm maybe that'll fix that querying issue". So rebuilding it this time shouldn't prevent the issue from occurring again.


The dates and times I updated are likely to coincide to when the issue has occurred. I likely wouldn't have remembered to check for an update otherwise. It's also an incomplete picture because sometimes instead of clicking "Save to" I've clicked "Open" or there was no update available.

[void]

Do you have the preview pane shown?

I have noticed some incorrectly installed preview handlers that are missing dlls will cause Everything to freeze on Querying for several minutes.

If I'm remember correctly, Everything's CPU usage while "Querying..." isn't above normal. Perhaps an issue with a synchronization primitive?

There's two different Querying... statuses you should be aware of:

Querying...
The search is not complete.
Old results are still visible.

Querying... x objects
The search is partially complete.
Some results will be shown.
Everything is gathering properties to complete the search request.


Which Querying... status are you seeing?
If you are seeing Querying... x objects, it most likely means Everything is stuck trying to read a file property.

Verbose debug output will show more information.

Upon next launch it rebuilt the db and spent a while re-indexing properties. The new database size is down to 472mb from 554mb, which I found odd because shouldn't it contain the same data as before?

This is a little odd.
Indexed properties will be allocated as they are read.
There might be indexed properties that have not been read yet.

There might have been indexed properties included from offline volumes in the old database.

Tools -> Debug -> Statistics shows database allocation information.

[DerekZiemba]

Was able to reproduce it today. Believe its memory related as suspected. OmniSharp ate all the memory so Everything64.exe exhibited the issue.
Not sure this is an actual issue now. I don't think it's reasonable to expect Everything to work properly when no memory is available.
Note that while this screenshot was taken, Everything was stuck "Querying...". So 2.73% more accurately defines "CPU usage... isn't above normal"



At the time this gif was taken, it was already "Querying" for well over a minute. The gif is over 2min long.
It was triggered by simply deleting the prior query "test". Towards the end there you can also see a lot of "VirtualLock" entries.
It may be easier to consume the gif as mp4 https://i.imgur.com/sCHLGWX.mp4


Notice towards the end I exit vscode. This freed up ram and then the results list was almost immediately updated.

Attached is the copied console output as well as Everything Debug Log-1.5a.txt

Which Querying... status are you seeing?
If you are seeing Querying... x objects, it most likely means Everything is stuck trying to read a file property.

Just regular "Querying..." as seen at the start of the gif before I interacted with the menu and caused it to get stuck on "Contains help commands." Can also be seen at 1:41 after cancelling the query.

[void]

Thank you for your reply, screenshots and debug logs DerekZiemba,


Everything allocates room for results as needed.

It's interesting this allocation takes such a long time in a low memory situation.
My guess is memory is being paged to disk to make RAM available.

Normally if Everything is unable to allocate memory it will just throw a fatal error and exit.


I will experiment with an option to preallocate room for the results and get back to you.
preallocating room for results will use additional memory (~8MB per 1 million files)


Please feel free to disable min_working_set_size, max_working_set_size and virtual_lock as it appears Everything is not being paged to disk:
virtual_lock also fails due to low system memory.

In Everything, type in the following searches and press ENTER:

/min_working_set_size=0
/max_working_set_size=0
/virtual_lock=0

[void]

Everything 1.5.0.1299a adds a mem_trim ini setting.

When disabled, Everything will keep resources available for new result arrays.
Disabling mem_trim will make Everything consume slightly more ram (~8MB per 1million files)

This should prevent Everything from allocating new memory when performing a query.

Please try disabling mem_trim and see if the issue persists:

• In Everything, type in the following search and press ENTER:
  /mem_trim=0
• If successful, mem_trim=0 is shown in the status bar for a few seconds.

It is possible Everything will be paged to disk again, in which case, please try re-enabling min_working_set_size, max_working_set_size and virtual_lock:

• In Everything, type in the following search and press ENTER:
  /min_working_set_size=0xfffffffffffffffe
• Type in the following search and press ENTER:
  /max_working_set_size=0xfffffffffffffffe
• Type in the following search and press ENTER:
  /virtual_lock=1
• Type in the following search and press ENTER:
  /restart

[cwm9]

OMG, this post saved me.

For years now I've been dealing with slow performance when opening Everything. I finally managed to trace the problem to Everything being paged out when I minimize the window. (I have 64GB of RAM and 48TB HDD space that is indexed). Upon reopening, it was taking (literally!) 60+ seconds before Everything could respond again because Windows was restoring Everything and its entire in-memory index out of the page file on every restore from Tray... despite the fact that in memory it was "only" taking up 500 megabytes of my 64 GB and I have over 32 GB free. (The restore was happening a 6MB per second! 500 MiB/6 MiB/sec = 83 seconds! )

Disabling paging has made Everything fly again, the way I remember it being.

Please consider adding a "dummy" switch to the in-app settings that turns off paging. "Speed up Everything on systems with lots of RAM by preventing Everything from being paged to disk". Perhaps include a warning you suggest that it only be enabled on systems with 32GB or more RAM.

You could also consider replacing "querying..." with "restoring from pagefile..." when appropriate so users are more aware of what is happening.

[void]

Thank you for your feedback cwm9,

I will consider an option or an advanced option to prevent Everything from being paged to disk.

I would like to avoid enabling any settings by default to prevent Everything from being paged to disk.
This will likely make a lot of issues.



Memory being paged to and from disk is hidden from Everything.
However, I will consider a "restoring from pagefile" status if technically possible.

Thank you for the suggestions.

[V@no]

Would windows merge into page file if there is more then 15gb of free ram left? Cause I'm experiencing the same issue, but I'd think there is still pretty of ram to spare...In my case Everthing is running as service, but when I open its window, it takes a minute for it to "query"

[void]

Yes, Windows can page Everything to disk, even when there's plenty of available RAM.


To help prevent the OS from paging Everything to disk:

• In Everything, from the Tools menu, click Options.
• Click the Advanced tab on the left.
• To the right of show settings containing, search for:
  working
• Select max_working_set_size
• Set the value to: 0x7fffffff
• Select min_working_set_size
• Set the value to: 0x7fffffff
• To the right of show settings containing, search for:
  lock
• Select virtual_lock
• Set the value to: true
• Click OK.

Please note: Enabling min_working_set_size, max_working_set_size and virtual_lock can prevent other programs from allocating memory in low memory conditions.