Everything1.5.0.1331a - Default Instance Crashes when Indexing Props on C:\ Drive

Discussion related to "Everything" 1.5 Alpha.
Post Reply
DerekZiemba
Posts: 44
Joined: Thu Sep 27, 2018 4:46 pm

Everything1.5.0.1331a - Default Instance Crashes when Indexing Props on C:\ Drive

Post by DerekZiemba »

I've recently had two 2TB Samsung 870 Evos fail (drive's manufactured end of last year are failing prematurely en masse). This caused a need to rebuild my Index, but now the default instance of Everything stops responding as soon as it comes time to index properties.
The work around I've arrived at is creating a new instance & launching with -instance "Main" argument, which for some reason works. Initially I did this just to see if it would work with stock settings, then slowly added my settings to it to see which one was causing the problem.
I've deleted the AppData\Roaming\Everything files the default instance uses to no avail. In my 2nd instance, all the settings are the same. As far as I can tell, the problem seems to be because the default instance detects a "PortableBaseLayer" drive that doesn't get detected by the non-default instances. And the problem only happens if I index drive C:\ on this default instance. I can index C:\ no problem using the -instance "Main" instance.
I think I've had this problem before as well, but in the past, I manually deleted PortableBaseLayer from Everything-1.5a.ini & it stayed gone. Now for some reason, Everything is redetecting it & it keeps coming back.

Everything-1.5a.ini default instance without drive C:\ included (so it doesn't lockup & crash):

Code: Select all

ntfs_volume_paths=A:,C:,C:\ProgramData\Microsoft\Windows\Containers\BaseImages\61004803-ee21-4570-abe2-26ab99876538\BaseLayer,D:,E:,H:,P:,V:,Y:,Z:
ntfs_volume_guids=\\?\Volume{},\\?\Volume{},\\?\Volume{},\\?\Volume{},\\?\Volume{},\\?\Volume{},\\?\Volume{},\\?\Volume{},\\?\Volume{},\\?\Volume{}
ntfs_volume_roots=,,,,,,,,,
ntfs_volume_includes=1,0,0,0,0,1,0,1,0,0
ntfs_volume_include_onlys=,,,,,,,,,
ntfs_volume_label=Docs,Windows,PortableBaseLayer,CrucialM4,Evo840,HomeMedia,HDDStorage,HyperV,Evo870Bad,Evo870Poor
ntfs_volume_monitors=1,1,1,1,1,1,1,1,1,1
ntfs_volume_multithreaded=0,0,0,0,0,0,0,0,0,0
ntfs_volume_drive_type=3,3,3,3,3,3,3,3,3,3
ntfs_volume_load_recent_changes=0,0,0,0,0,0,0,0,0,0
ntfs_volume_user_added=0,0,0,0,0,0,0,0,0,0
ntfs_volume_real=1,1,1,1,1,1,1,1,1,1
Everything-Main.ini custom instance (doesn't have that PortableBaseLayer)

Code: Select all

ntfs_volume_paths=A:,C:,D:,E:,H:,P:,V:,Y:,Z:
ntfs_volume_guids=\\?\Volume{},\\?\Volume{},\\?\Volume{},\\?\Volume{},\\?\Volume{},\\?\Volume{},\\?\Volume{},\\?\Volume{},\\?\Volume{}
ntfs_volume_roots=,,,,,,,,
ntfs_volume_includes=1,1,0,0,1,0,1,0,0
ntfs_volume_include_onlys=,,,,,,,,
ntfs_volume_label=Docs,Windows,CrucialM4,Evo840,HomeMedia,HDDStorage,HyperV,Evo870Bad,Evo870Poor
ntfs_volume_monitors=1,1,1,1,1,1,1,1,1
ntfs_volume_multithreaded=0,0,0,0,0,0,0,0,0
ntfs_volume_drive_type=3,3,3,3,3,3,3,3,3
ntfs_volume_load_recent_changes=1,1,0,0,1,0,1,1,1
ntfs_volume_user_added=0,0,0,0,0,0,0,0,0
ntfs_volume_real=1,1,1,1,1,1,1,1,1
The only other difference is Everything-Main.ini starts with:

Code: Select all

app_data=0
run_as_admin=0
index_as_admin=0
exit_indexing_process_on_exit=1
stop_service_on_exit=0
lvm=1
ipc=1
ipc_pipe=1
allow_window_message_filter_ipc=1
allow_window_message_filter_dragdrop=0
localization_strings=
allow_open=1
allow_context_menu=1
allow_delete=1
allow_rename=1
allow_cut=1
allow_copy=1
allow_paste=1
allow_drag_drop=1
allow_path_ldrop=0
allow_path_rdrop=1
allow_path_ldrop_modifier_key=1
allow_options=1
allow_run_on_system_startup=1
allow_everything_service=1
allow_folder_context_menu=1
allow_start_menu_shortcuts=1
allow_desktop_shortcut=1
allow_quick_launch_shortcut=1
allow_url_protocol=1
allow_efu_association=1
allow_export=1
allow_check_for_updates=1
allow_clear_search_history=1
allow_clear_run_history=1
allow_force_rebuild=1
allow_rescan_now=1
allow_admin_options=1
alpha_instance=1
Back when all the drives were good and everything worked (used VolumeShadowCopy to pull this from 2022.12.09), there wasn't a PortableBaseLayer either.

Code: Select all

ntfs_volume_paths=A:,B:,C:,H:,M:,P:,V:,W:
ntfs_volume_guids=\\?\Volume{},\\?\Volume{},\\?\Volume{},\\?\Volume{},\\?\Volume{},\\?\Volume{},\\?\Volume{},\\?\Volume{}
ntfs_volume_roots=,,,,,,,
ntfs_volume_includes=1,1,1,1,1,1,1,1
ntfs_volume_include_onlys=,,,,,,,
ntfs_volume_label=Docs,RAID0,Windows,HomeMedia,Movies,HDDStorage,HyperV,WindowsBackup
ntfs_volume_monitors=1,1,1,1,1,1,1,0
ntfs_volume_multithreaded=1,0,1,1,1,0,0,0
ntfs_volume_drive_type=3,3,3,3,3,3,3,3
ntfs_volume_load_recent_changes=1,1,1,1,1,1,1,1
ntfs_volume_user_added=0,0,0,0,0,0,0,0
ntfs_volume_real=1,1,1,1,1,1,1,1
Evo870Bad & Evo870Poor were part of a drivepool servicing Docs, RAID0, HomeMedia, Movies, & HyperV - they were moved to motherboard SATA controller so Samsung Magician could work with them. With the concurrent failure of those two drives, RAID0 & Movies volumes were sacrificed so there was enough redundancy to recover Docs, HomeMedia, & HyperV. CrucialM4 & Evo840 are temporarily to store files until I get 870Evo replacements. WindowsBackup temporarily doesn't exist because I ran out of SATA ports.
Last edited by void on Mon Dec 19, 2022 12:03 pm, edited 1 time in total.
Reason: removed volume names
therube
Posts: 4985
Joined: Thu Sep 03, 2009 6:48 pm

Re: Everything1.5.0.1331a - Default Instance Crashes when Indexing Props on C:\ Drive

Post by therube »

So I take it that currently none of your drives are LABEL'd "PortableBaseLayer".
(Disk Management or Windows Explorer should show the labels attached to particular partitions.)


(And that being the case, I can only guess that... in attempting to maintain/distinguish between "drives" that something to do with the guid is causing "PortableBaseLayer" to turn up?)


(CrystaslDiskInfo: Hmm. "Raw Values". I suppose that DEC(imal) is more readable then HEX?)
void
Developer
Posts: 16773
Joined: Fri Oct 16, 2009 11:31 pm

Re: Everything1.5.0.1331a - Default Instance Crashes when Indexing Props on C:\ Drive

Post by void »

Thank you for the crash report DerekZiemba,

Could you please re-add your C: drive to your index and send a mini crash dump:
  • Re-include your C: drive in your index from Tools -> Options -> NTFS -> C: -> Include in database.
  • Keep the crash dialog open.
  • Download Process Explorer from Microsoft:
    https://docs.microsoft.com/en-us/sysinternals/downloads/process-explorer
  • In Process Explorer, select Everything.exe (If there is two, select the one using the most memory)
  • Right click Everything.exe and under Create Dump, select Create Mini Dump.
  • Choose a filename and click Save.
  • Please send this file to support@voidtools.com
Privacy
DerekZiemba
Posts: 44
Joined: Thu Sep 27, 2018 4:46 pm

Re: Everything1.5.0.1331a - Default Instance Crashes when Indexing Props on C:\ Drive

Post by DerekZiemba »

Hmm, now that I try to replicate it, "(offline)" was next "PortableBaseLayer". Previously "remove" was always greyed out. So I clicked it. And well, now I can't replicate the issue. It's successfully indexing.
I haven't restarted the PC so nothing should have changed, IDK why I can now do that.
I've tried adding it back through the config but now that it's offline, I can't get it to do it. I'm not sure what triggers that volume to go online. The internet says it's something to do with Windows Sandbox, but I launched it and the volume still didn't come online.

Upon looking at the EventLog, the chain of events seems to be:
  • System Restore Successfully created restore point
  • MsiInstaller Windows Installer installs, updates, reconfigured, or uninstalls something
  • VSTTExecution (devenv.exe, PID 15564, Thread 1) QualityToolsPackage Package Initialization Failure: Failed to get service EnvDTE._DTE This one doesn't always happen, VS2022 just happened to trigger the latest update
  • Windows Search Service The Windows Search service stopped the Protocol Host process because it was consuming too many resources.
  • Windows Error Reporting Event Name: RADAR_PRE_LEAK_64 Response: Not available Cab Id: 0 Problem signature: P1: Everything64.exe P2: 1.5.0.1331 P3: 10.0.19044.2.0.0
  • Microsoft-Windows-Winsrv The following application was terminated because it was hung: Everything64.exe.
  • EventLog The system uptime is 30 seconds. (after PC auto restarts)
  • Windows Error Reporting Event Name: AppHangB1 Response: Not available Cab Id: 0 Problem signature: P1: Everything64.exe P2: 1.5.0.1331 P3: 63916d05 P4: 109d P5: 134217728
  • Application Hang The program Everything64.exe version 1.5.0.1331 stopped interacting with Windows and was closed.
The date that PortableBaseImage modified does coincide with WindowsUpdates. So my best guess is that PortableBaseImage gets mounted and updated whenever Windows runs updates.
I'll keep you posted if it does it again. I currently already have all the latest Windows updates.
Keep the crash dialog open.
So it never actually crashes (while I'm watching it trying to use it). It would just go unresponsive while sucking down ~12.5% CPU (1 core). The Window does not respond to any input, can't be moved, etc. It would freeze roughly 2-3seconds after launching the window, but seemed to have been properly indexing stuff in the background up to that point. Windows presents a "Everything is not responding. Close the program or Wait for the program to respond" dialog.
I think this results in different log entries like:
  • Windows Error Reporting Event Name: APPCRASH Response: Not available Cab Id: 0 Problem signature: P1: Everything64.exe P2: 1.5.0.1331 P3: 63916d05 P4: Everything64.exe P5: 1.5.0.1331
  • Windows Error Reporting Event Name: AppHangTransient Response: Not available Cab Id: 0 Problem signature: P1: Everything64.exe P2: 1.5.0.1331 P3: 63916d05
void
Developer
Posts: 16773
Joined: Fri Oct 16, 2009 11:31 pm

Re: Everything1.5.0.1331a - Default Instance Crashes when Indexing Props on C:\ Drive

Post by void »

Thanks for the information DerekZiemba,

If Everything hangs again, could you please send a mini crash dump: Privacy
DerekZiemba
Posts: 44
Joined: Thu Sep 27, 2018 4:46 pm

Re: Everything1.5.0.1331a - Default Instance Crashes when Indexing Props on C:\ Drive

Post by DerekZiemba »

It's doing something weird again. Kind of the same thing, but there's no PortableBaseLayer this time and it freezes @~3% sorting if I don't touch it, otherwise it freezes instantly Querying ...

I can currently replicate the issue on demand & can screen record a launch if needed.
I'm pretty sure I can fix it by deleting the database and the fat_volume_*, ntfs_volume_*, & refs_volume_* section/properties from the .ini before launching again as that's what's always worked in past. Though I dread to do it during the day bcus it takes several hours, like 8-12GB RAM, & max CPU to index all the content properties I've opted-in to (7.5TB & probably 8 million items without exclusions (lots of code, code artifacts, & dependencies))- also I feel I may be able to better help you if I keep it in this broken state.
Everything initially starts but immediately locks up if I try to switch the filter category or do a search. If I let it sit with an open Window, it'll lock up after a minute or two (during sorting). Without interaction, it appears to run in the background fine. For this screenshot, I immediately started the debug logger then tried to go into the settings to see if a PortableBaseLayer or other odd volume was present. It made it about 2 minutes before locking up.
Also pictured it a working instance of Everything 1.5.0.1331a, this is my alternative instance I use when the default installed instance breaks. I only just upgraded the default instance to 1332a bcus it was already broken & thought maybe the new version would change something. I didn't bother upgrading the "A:\Everything\Everything64.exe" -instance "Main", I just needed to know where the log files for the broken main install were placed. (more details in bottom quarter of screenshot as red & light green text).
ScreenshotAtMiddleWithExplainers.png
ScreenshotAtMiddleWithExplainers.png (535.13 KiB) Viewed 3600 times
After finishing this post, Everything no longer shows any process activity & just appears to be dead.
ScreenshotNowAppearsDeadWithNoThreadActivity.PNG
ScreenshotNowAppearsDeadWithNoThreadActivity.PNG (443.73 KiB) Viewed 3600 times
------------------------------------------------------------------------
Note: I've had a suspicion for a while now that my issue may somehow be related to the usage of RegularExpressions.
Thinking back, I don't think I've ever had issues with Everything until I started using them. Usage of RegularExpressions is one of the few differences I have between the default instance and that alternative instance. On the default instance, I use the expressions below at the Indexes->Exclude stage. In the alternative instance, I do it via "Organize Filters". The trade off is the default instance searches instantly, but rebuilding the database & indexing content takes hours. The alternative instance rebuilds & indexes in about an hour, but searches can sometimes take a second or ten. The differences between the default instance & the alternative instance(that has yet to have a problem) are:
  • The alternative instance does NOT index Archival/Backup/VM drives (HyperV, CrucialM4, Evo840, or HDDStorage)
  • The default instance runs the following RegExps at Indexing time, whereas the alternative instance at Search time depending on the Filter used
    • Hexadecimal: regex:^[\[{]*[0-9a-fA-F=~+_\)\(\-]{12,}[\]{}=+_-]*$
    • Hashes/Hexadecimal/caches for Folders: regex:\\[0-9a-fA-F\\%_-]{8,}$
    • Hashes/Hexadecimal/caches/git/svn Files: regex:/[0-9a-f]{2}/[0-9a-fA-F_]{32,}
    • GUIDs: regex:{[0-9a-fA-F]{8,}-(?:[0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12,}}
    • Base64: regex:^(?:[a-zA-Z0-9+_-]{2})+(?:[a-zA-Z0-9+_-]{2}=|[a-zA-Z0-9+_-]{3})+=(?:\.[a-z]{2}[a-z0-9]+)?$
    • Localizations: regex:/(_?local((es?|iz(?:e|ations?)))|a(f-ZA|r-SA)|b(n-BD|g-BG)|c(a-ch)|s-(CZ|ES)|y-GB|d(a-DK|e(-DE)?)|e(ULA|[lstu][-_](EE|ES|GR|MX))|f(a-IR|i-FI)|gl-ES|h(e(lp|-IL)|i-IN|r-HR|u-HU)|i(d-ID|t-IT|s-IS|18n|nt(l|ernationalization))|ja(-JP)?|k(a-GE|k-KZ|o(-KR)?)|l(t-LT|v-LV|ang(uages?)?|egal|inguistics?)|ms-MY|n[bln]-N[OL]|p[lt]-(BR|P[LT])|r(o-RO|u(-RU)?)|s(k-SK|l-SI|q-AL|v-SE)|t(r(-TR|anslations?)|h-TH)|uk-UA|vi-VN|fr([_-](CA|FR))?|zh([_-](TW|CN|CH[NS]|Han[ts]))?)(/|$)
      I spent a lot of time optimizing this one bcus it's SLOW, but I hate all the localization files laying around. Grouping words together that had letters in common brought it from 7-10sec to 2-<3sec. Was surprised it even made a difference/thought the RegEx engine would do that for you.
  • Indexing File Contents (default instance only): regex:(\.[a-z]{2,12}rc$)|CMAkeLists|MAKEFILE|CHANGELOG|README|CONTRIBUTING; *.csv;*.log;*.md;*.txt;*.nfo;*.chm;*.rtf;*.xml;*.yml;*.json;*.ini;*.settings;*.manifest;*.vsixmanifest;*.profile;*.bash_profile;*.js;*.js;*.jsx;*.jsm;*.cjs;*.mjs;*.coffee;*.ts;*.tsx;*.tsm;*.rb;*.tsconfig;*.css;*.scss;*.less;*.htm;*.html;*.cs;*.cshtml;*.vb;*.vbhtml;*.resx;*.asp;*.aspx;*.sql;*.c;*.cc;*.cpp;*.cxx;*.h;*.hpp;*.hxx;*.idl;*.rb;*.erb;*.ru;*.gemfile;*.bat;*.cmd;*.sh;*.bash;*.ps1;*.vbs;*.psm1;*.java;*.lua;*.php;*.rs;*.go;*.xml;*.yml;*.json;*.ini;*.csproj;*.vbproj;*.vcxproj;*.sqlproj;*.sln;*.manifest;*.vsixmanifest;*.props;*.targets;*.pubxml;*.config;*.jsconfig;*.tsconfig;*.editorconfig;*.gitconfig;*.gitmodules;*.gitignore;*.gitattributes;
I use "Organize Filters" & Indexes->Exclude instead of the Enable Result Omissions feature because I learned very quickly it often would result in "Not Responding" & "Cancel Current Query?". Whereas that's not so much a problem using the two other methods (or maybe it is the problem???). Adding those expressions to Result Omissions actually just pretty easily got me to lock up the alternative instance, but it came back after 10 seconds.
UsingThoseRegexsInEnableResultOmissions.png
UsingThoseRegexsInEnableResultOmissions.png (26.31 KiB) Viewed 3575 times
------------------------------------------------------------------------
EDIT: The default instance just came back after 3hrs of not responding. I had shut down the original instance earlier and tried relaunching it at 9:50AM. I was doing something else when suddenly two "Cancel Current Query?" windows popped up out of nowhere. I confirmed the cancellations then three Everything windows that I must have tried opening sometime earlier appeared. I didn't notice but looks like it was busy actually doing something this time with >11hrs CPU time, possibly RegExpressions on what looks like 227.65GB of files?
CameBackAfter3hrs.png
CameBackAfter3hrs.png (83.36 KiB) Viewed 3575 times
------------------------------------------------------------------------
EDIT2: Actually, I don't know what's going on. My filters.csv got janked somehow. The Filters-1.5a.csv I attached appears to be like a mishmash from two different versions from some time ago & doesn't actually filter anything properly. Additionally for some reason the VolumeShadowCopies for that file specifically are all corrupt and full of null values. Yesterday I uninstalled NVidia Nsight for VisualStudio, and I know it created a restore point, but there's no available ShadowCopy entry for it. There might be a decent chance there's nothing wrong with Everything & something wrong with my computer. Hardware seems to be dieing rather frequently...
ShadowCopiesCorrupt.PNG
ShadowCopiesCorrupt.PNG (131.69 KiB) Viewed 3574 times
------------------------------------------------------------------------
Everything-Logs.7z contains:
  • "Process Hacker (Everything64.exe) Memory.txt"
  • "Process Hacker (Everything64.exe) Modules.txt"
  • "Process Hacker (Everything64.exe) Threads.txt"
  • "Everything Debug Log-1.5a.txt"
  • "Everything-1.5a.ini"
  • "Filters-1.5a.csv"
  • "ScreenshotAtMiddleWithExplainers.png"
  • "ScreenshotAtStart.png"
  • "ScreenshotNowAppearsDeadWithNoThreadActivity.PNG"
Apparently ProcessHackers MiniDumps are actually full dumps, so I have full dumps if needed at various stages of it crashing. I only realized after Everything64.exe ceased all activity, then installed ProcessExplorer to get the attached minidump.
  • Everything64.exe-2023-01-04_07-57-12.7z (Initial - Dump made right after lockup) --void: removed dumps--
  • Everything64.exe-2023-01-04_08-07-44.7z (Middle - The dump made in ScreenshotAtMiddleWithExplainers) --void: removed dumps--
  • Everything64.exe-2023-01-04_08-38-07.7z (AppearsDead - Dump made after final screenshot) --void: removed dumps--
Last edited by void on Thu Jan 05, 2023 12:26 am, edited 1 time in total.
Reason: removed logs
void
Developer
Posts: 16773
Joined: Fri Oct 16, 2009 11:31 pm

Re: Everything1.5.0.1331a - Default Instance Crashes when Indexing Props on C:\ Drive

Post by void »

Thank you for the information, debug logs and dumps.

I am looking into the issue.
void
Developer
Posts: 16773
Joined: Fri Oct 16, 2009 11:31 pm

Re: Everything1.5.0.1331a - Default Instance Crashes when Indexing Props on C:\ Drive

Post by void »

The debug logs show that the console is in mark mode and almost all threads are waiting to exit mark mode.

You have an active mark in the console window
USN CLOSE DATA OVERWRITE.... is selected (Red on White background)

Everything will pause until you exit mark mode in the debug console.



This is most likely unrelated to the issue you are seeing.
Could you please try sending another crash dump when Everything hangs / crashes?
-Please make sure the console is not in mark mode.


I can currently replicate the issue on demand & can screen record a launch if needed.
Yes please.
Please send to support@voidtools.com
If I can reproduce the issue my end, it will make things a lot easier ;)



The debug logs show you are indexing a lot of properties.
It looks like Everything is currently scanning a lot of files under a Dropbox folder.
The issue might be related to property indexing the Dropbox folder.

There's a log file under C:\inetpub\logs that keeps getting its properties reindexed.
Please try excluding this folder.

Please check your Index menu -> Index Journal for any other files that are constantly being modified.
-Please make sure they are excluded from your property indexing.


Code: Select all

regex:^[\[{]*[0-9a-fA-F=~+_\)\(\-]{12,}[\]{}=+_-]*$ 
There's a space at the end of your exclude filters that make these searches match nothing.


"Not Responding" & "Cancel Current Query?"
A mini crash dump when you see "Not Responding" or the "Cancel Current Query" will help me fix the issue.


My filters.csv got janked somehow.
I would love to know what hardware you are using for the drive storing your filters.csv.
This is a little concerning as Everything will only write to filters.csv when you make a change to your filters from the UI.
DerekZiemba
Posts: 44
Joined: Thu Sep 27, 2018 4:46 pm

Re: Everything1.5.0.1331a - Default Instance Crashes when Indexing Props on C:\ Drive

Post by DerekZiemba »

> The debug logs show that the console is in mark mode and almost all threads are waiting to exit mark mode. Everything will pause until you exit mark mode in the debug console.
Woops.

> The debug logs show you are indexing a lot of properties.
I only recently realized that indexed properties & content are not automatically searched in general queries without explicit functions like subject:"x" description:"x" programName:"x" title:"x" content:"x".
So for the past ~2 weeks/after that realization I've stopped indexing content & reduced the list of indexed props to ones used in Filters: AlternateDateStreamNames, DateSaved, Dimensions, Height, Length, & Owner.
I have yet to have a crash since, but that's not the only thing I changed.

> There's a log file under C:\inetpub\logs that keeps getting its properties reindexed. Please check your Index menu -> Index Journal for any other files that are constantly being modified.
I didn't know about this tool. I now check it every so often then add anything egregious to the Index Exclusions. With the reduced props and lack of content indexing, it now indexes pretty fast again so rebuilding isn't such a big deal. Also freed about 1GB ram.

I haven't had any recent problems.
Post Reply