Understanding Indexing and USN Journal - 3 Questions

[ChrisGreaves]

Since Everything-1.2.1.371.exe I have used Everything as a simple file search utility, but have never been able to set up the index simply. Today I

Code: Select all

Used RevoUninstaller and scan to remove Everything-1.4.1.969.x64
Rebooted Win10
Extract all to T:\Appl\Everything\Everything-1.5.0.1309a.x64
Ran Everything64.exe

Everything built an index from its default settings:-

Every001.png (26.63 KiB) Viewed 5938 times

Question 1: Why is drive A: included in the index but not drive B:?
At first I thought that, just perhaps, Everything thought that drive A: was a floppy drive, but then B: was not indexed; I have files in both A: and B:, neither of which is a floppy drive.
My boot sequence uses the DOS SUBST command to map folders to drive letters for my convenience when I am working on a project.
Thus: "subst U: T:\Greaves\Products\DEVEL\Turing\"
and: "subst A: C:\Users\%UserName%\AppData\Roaming\Greaves"
and: "subst B: T:\Blotter\%DAILY%"
Drive T: is the decrypted version of a Veracrypt partition of the sole physical hard drive on the laptop.
Drive A always takes me to my Roaming Appdata folders (generally INI files for various projects)
Drive B always takes me to a desktop blotter folder, unique for the day. Today the drive B represents "T:\Blotter\20220421"
Drives L, U, V, and W point to active projects and do not appear in the index, which is fine by me, since the host drive T: does appear in the index and I can always find the file that way.

Every002.png (66.02 KiB) Viewed 5938 times

Question 2: What should I change in the area outlined above? I have over the years tried checking ON USN Journal without knowing what I am doing.

Every003.png (34.58 KiB) Viewed 5938 times

Question 3[/b]:I experimented with my drive L: but (perhaps fortunately) Everything threw up this error message. I can't remember seeing this message over the past nine years of foolish fiddling (grin), but I assume that somewhere between Everything and Windows10 an alarm bell goes off because I have SUBSTituted a drive letter on an encrypted volume. Is this close to the truth?

The only drives I really need indexed are my boot partition (C:) and my data partition (T:).
Thanks well in advance for any insights.
ChrisGreaves

[therube]

Unless you have specific reason to do so, you should not be running Everything "As Administrator" - use the Everything Service instead.

> Why is drive A: included in the index but not drive B:?

B: looks to be there, but is empty?

> Question 2: What should I change in the area outlined above?

Don't know, particularly. I just leave those settings at defaults.

[ChrisGreaves]

Hello rube, and thanks for the comments.

Unless you have specific reason to do so, you should not be running Everything "As Administrator" - use the Everything Service instead.

I can change this, possibly. I have found in the past that NOT running as Administrator causes problems with some BATch files or with assignment of shortcut keys, pinning to taskbar, etc. In short, non-Administrator has been a problem since Win7, for me. I suspect that this "Administrator privilges" is aimed at the run-of-the-mill user who browses the internet and downloads dangerous programs, rather than someone who never grew up from the Days of DOS(grin)

> Why is drive A: included in the index but not drive B:?
B: looks to be there, but is empty?

Drive B: is never empty.
The letter for Drive B is assigned by my boot sequence, which SUBSTitutes B for the daily blotter folder AND copies three dictionaries into B: as a short-term backup; also Audacity config files and any other text files that I might mess up during a day's running.

Untitled.png (32.09 KiB) Viewed 5893 times

I cleared out a set of blotter files two days ago, so today only the last two daily folders remain.
Seven instances of "offline disk B:" remain in that scrollable window, but only three of the seven are visible in my screen shot.
"Enable USN journal is checked ON"

In the first screenshot in my first post, note that "B:" is described as "0KB", but once the boot sequence is completed, drive B: will never be empty (well, OK, if I then go in and manually delete my local backup copies ...)

> Question 2: What should I change in the area outlined above?
Don't know, particularly. I just leave those settings at defaults.

And herein lies the mystery.
When I reinstalled Everything some three or four days ago, I refrained from changing anything at all in the options, reasoning that it was, prehaps, my playing around with Options that has caused problems inmy previous installations.
(later)

Untitled2.png (10.05 KiB) Viewed 5893 times

I found a file "links.xml" which appears in a regular (non SUBST path) in both the boot partition C: and in the decrypted data partition (T:), but NOT in my A: or B: drive. I copied the links.xml from T: into both the A: and the B: drives. So now two SUBST drives - one that points to the fixed Users\Roaming and one that points to the variable daily folder that is drive B: - and then used Everything to search for Links.xml.
As show above, four such files are now listed:-
(1) This is the Proper path for a file copied to my A: drive
(2) This is the proper path to the original file in my C: drive.
(3) This is the Proper path for a file copied to my daily B: drive
(4) This is the proper path to the original file in my decrypted T: drive

I will have to think about this a bit more. Today I think that I have four distinct copies of a file in four distinct folders. Also that Everything-1.5.0.1309a.x64 is indexing both my A and my B drives, but reports them not as A or B, but translates these faked paths into a Proper path on the only two drives I really want to be indexed.

That is, maybe Everything-1.5.0.1309a.x64 with default settings is working as it should.
Which, of course, is to be expected.

I am still unsure about USN journal; I suspect that for my simple needs I do not need it.

Thanks, Chris

[NotNull]

I can change this, possibly. I have found in the past that NOT running as Administrator causes problems with some BATch files or with assignment of shortcut keys, pinning to taskbar, etc. In short, non-Administrator has been a problem since Win7, for me. I suspect that this "Administrator privilges" is aimed at the run-of-the-mill user who browses the internet and downloads dangerous programs, rather than someone who never grew up from the Days of DOS(grin)

Best practice is to run Everything using the same credentials as your Windows login-account. That means enabling 'Everything Service' and unchecking 'Run as administrator'
When running Everything 'As Administrator', all programs, like Notepad, started through Everything will be started as administrator too. Furthermore, a SUBST is per-user. So as administrator you will not see SUBST'ed drives of the Windows login user (if those are different accounts; if you already log in as non-elevated admin, it should work).
Same goes for mapped network disks. They are per-user too.
There are more reasons why you shouldn't run Everything elevated, but his should do for now.


When 'Automatically include new fixed volumes' is NOT enabled (see your screenshots), a SUBST'ed drive needs to be added manually:
In the 'Local NTFS volumes:' box, select your B:-drive and enable 'Include in database'.


Batch-files can be run elevated by right-clicking them and choose Run as administrator from the context menu.
Be aware that elevated CMD windows will have a default working directory of C:\Windows\Sysytem32, so you might have to add a command pushd "%~dp0" to your script to start in the script-folder.


Assigning shortcut keys and pinning to the taskbar should not be an issue when not running as an elevated administrator.


Are you running Everything 1.4 or 1.5 or both at the moment? (you posted screenshots from both)


Let's find out what issues remain after implementing the suggestions above. Those can very likely be fixed too.

[void]

Everything will see your B: drive as a new drive each day.

Please try enabling Automatically include new fixed volumes and enabling automatically remove offline volumes:

• In Everything, from the Tools menu, click Options.
• Click the NTFS tab on the left.
• Check Automatically include new fixed volumes.
• Check Automatically remove offline volumes.
• Click OK.

If Everything is still not including your B: drive, could you please send some debug output:

• Please make sure the Everything Options window is closed.
• From the Tools menu, under the Debug menu, click Start Debug Logging.
• From the Tools menu, click Options.
• Click the NTFS tab on the left.
• Close the Options window.
• From the Tools menu, under the Debug menu, click Stop Debug Logging.
  ---This will open your Everything Debug Log.txt
• Could you please send this file to support@voidtools.com

[ChrisGreaves]

Everything will see your B: drive as a new drive each day.

Thank you, Void; this statement is correct. I had thought of my Blotter drive always being present, but I suppose that Everything in its infinte wisdom (smile) sees it as T:\Blotter\20220501\ today which is different from yesterday, when it was T:\Blotter\20220430

[*]Check Automatically include new fixed volumes.
[*]Check Automatically remove offline volumes.

This I have done, and now my Everything screen looks as i expect/want it to be, showing results only from my boot partition C: and my decrypted data partition T: (clapping)

If Everything is still not including your B: drive, could you please send some debug output:

My poorly-worded leading post is at fault here.
When I asked "Why is drive A: included in the index but not drive B:?" I was concerned that drive A was showing up, rather than drive B not showing up. It was the presence of my Substituted drive A that caused me concern. I work in splendid isolation and as long as I can see the "pure" full file name my mind easily translates this to a substituted drive.

Thank you again for your help
Chris

[ChrisGreaves]

When 'Automatically include new fixed volumes' is NOT enabled (see your screenshots), a SUBST'ed drive needs to be added manually:
In the 'Local NTFS volumes:' box, select your B:-drive and enable 'Include in database'.

Thank you NotNull. As acknowledged in my reply to Void, indeed this was the setting that caused the A-drive to appear in the listing when I really didn't want it to be shown; seeing the original file name on my boot partition satisfies me.
When one file is listed twice (on A as well as on C) I get confused!

Are you running Everything 1.4 or 1.5 or both at the moment? (you posted screenshots from both)

I am running "Version 1.4.1.969 (x64)"
The web site shows "Download Everything 1.4.1.1015" as the latest public offering.
Please and thank you, in your opinion what is the latest stable version, and where do I find it?
Thanks again, Chris

[void]

The latest stable release can always be found on the downloads page.

1.4.1.1015 is the current stable release.

[ChrisGreaves]

If Everything is still not including your B: drive, could you please send some debug output:

I apologize for my delay in replying to this post.
I have emailed a package as you suggested.

I found a documentation bug. My version of Everything does not show a menu item Tools, Debug.
i found this page http://www.voidtools.com/support/everything/troubleshooting/ which suggested "See viewtopic.php?f=6&t=1713 for more information."

The page http://www.voidtools.com/forum/viewtopic.php?f=6&t=1713 suggested several options for start/stopping debug.
In the end i edited the INI file as:-

Code: Select all

debug=1
debug_log=1
verbose=1

and that created a text file of about two dozen lines.

Thanks, Chris

[horst.epp]

I guess there is no documentation bug.
Tools / Debug
is found in the Alpha version 1.5

[ChrisGreaves]

I guess there is no documentation bug.
Tools / Debug
is found in the Alpha version 1.5

Thank you Horst; I am using the current stable release, so that clears that up.
Cheers, Chris

[void]

Thank you for helping look into the issues Chris,


We found:


Everything is installed correctly.



Debug Logs confirmed Everything indexed the B: drive.
Everything was likely indexing the B: drive, but was not shown in the result list because there was no folders.



Renaming a SUBST volume target will cause Everything to lose its index of the SUBST volume.



Launching Everything after the SUBST volumes are created appears to have helped detect the B: drive.



Everything 1.4 is not indexing SUBST volume targets correctly (U: drive and L: drive).
The target was to an encrypted volume. (Not a real device, a virtually mounted volume)
Everything would index the whole target volume, not the target folder (for example: T: instead of T:\folder\childfolder)
I was unable to produce the issue on my end.
Adding the SUBSTed volumes as a folder index may help.
For now, we removed the SUBST volumes from Tools -> Options -> NTFS.


We found that all programs are running elevated.
Unsure of the reason for now, there's no scheduled tasks starting Explorer.
This might be causing Everything to see different SUBST volumes targets.

[ChrisGreaves]

Thank you for helping look into the issues Chris, ...

Thank you David.
We have spent about a month on resolving these issues on a daily basis.
From my point of view my system (encrypted data partition, use of SUBST commands in an AutoExec.bat file) is now well-supported.

More than anything else, as a long-time software developer I am impressed by David's determined and dedicated approach to resolving user problems.

Chris Greaves

P.S I shall now strike fear into David's heart by installing Everything 1.5 Alpha and populating that forum with posts.
C.

[ChrisGreaves]

... Question 1[/b]: Why is drive A: included in the index but not drive B:? ...

Two months later with a new laptop and Win11 instead of Win 7 & Win10 ... I finally understand it (I think!)
My problem as I saw it is that my SUBSTituted drives were popping up like Polar Bears in Melrose.

Indexes_05.png (24.39 KiB) Viewed 2815 times

My A: drive lists with C: drive because:-
The A drive is

Code: Select all

subst A: C:\Users\%UserName%\AppData\Roaming\Greaves

The APPData folder is part of the C: drive so, by definition, it must be NTFS (because the C: drive is NTFS)

Indexes_06.png (30.08 KiB) Viewed 2815 times

The B:, V:, and W: drives show up as FAT because they are mapped to folders of the T: drive which is by definition a FAT drive.

Code: Select all

		if not exist B:\NUL subst B: T:\Blotter\%DAILY%
		if not exist V:\NUL subst V: T:\Pers\Places\LivingInBonavista
		if not exist W:\NUL subst W: T:\Greaves\Admin\Domains

(E: is DataRam disk and F: is an 8GB memory key)

Back when I was young I couldn't understand why, for example, the A: drive was shacked up with the C: drive.
I should have twigged to this sooner because the start of my nightly backup copies my USERS folder (hence oin drive C) to drive T (my data) which means I capture my Appdata as part of the backup.

Cheers, Chris

[ChrisGreaves]

Back when I was young I couldn't understand why, for example, the A: drive was shacked up with the C: drive.

I have been documenting what I (think I ) know about Indexes.
I think that Void has mentioned the possibility of rewriting the dialog boxes for Tools, Options, Indexes.
I now see that - as they are - they work, but perhaps are most easily understood by developers and experts.
I am going to propose a new layout/format that better serves the beginner user, but supports all features evident in the current dialogue boxes, and perhaps this will reduce the number and vagueness of some problems that we encounter with indexes.
Cheers, Chris