What exactly does "rebuild index" mean?

If you are experiencing problems with "Everything", post here for assistance.
Post Reply
cabbage
Posts: 9
Joined: Tue Feb 15, 2022 2:51 pm

Re: Keep offline volumes in the index

Post by cabbage »

void wrote: Mon Jul 07, 2014 6:43 am
[*]Offline indexes will be lost when the index is rebuilt. All volumes must be online when Everything builds its index.[/list]
What exactly does "rebuild index" mean? I'm very confused. When does it happen? Every time Everything is started? In which case then how can there be such a thing as "keep offline volumes in index"? Is "rebuild index" the same thing as monitoring internal hard drive for changes and updating? Do I ever have to rebuild the index? How do I know if it's happening? Is there a way to stop it?

What I do know is that offline drives seem to randomly disappear and I have no idea why. Sometimes they stay for months and sometimes for only for a few minutes after detaching and I can't see any pattern.
void
Developer
Posts: 16773
Joined: Fri Oct 16, 2009 11:31 pm

Re: What exactly does "rebuild index" mean?

Post by void »

Moved from Keep offline volumes in the index.



Everything should only index once.

Everything will perform a quick reindex if you add or remove a volume from your index.


When does it happen?
When you start Everything for the first time.
-or-
When you add or remove a volume from your index.


In which case then how can there be such a thing as "keep offline volumes in index"?
When reindexing, Everything will keep an old copy of your index.
The filenames from your offline volume are gathered from your old index.
Once the new index is created the old index is destroyed.


Is "rebuild index" the same thing as monitoring internal hard drive for changes and updating?
No.
Rebuild index is creating a full and fresh scan of your volumes. (expensive)
Monitor changes watches your file system for changes and updates your existing indexes. (inexpensive)


Do I ever have to rebuild the index?
Generally, no.

You will need to rebuild if you:
delete hardlinks
add hardlinks in Everything 1.4 or earlier.
After running a check disk that made a change to your volume.



Everything will automatically reindex if:
a lot of changes were made to your file system while Everything was not running.
An NTFS USN Journal is deleted or re-created.
Your Everything.db was missing or corrupt.


How do I know if it's happening?
Everything will show reindexing or scanning in the status bar at the bottom left.


Is there a way to stop it?
Once started, a reindex cannot be stopped.



Is Everything missing files or still showing deleted files?

-Please make sure Everything is installed correctly:
  • In Everything, from the Tools menu, click Options.
  • Click the General tab on the left.
  • Check Store settings and data in %APPDATA%\Everything.
  • Uncheck Run as administrator.
  • Check Everything Service. (Please make sure this is tick-checked and not square-checked)
  • Click OK.
  • Exit Everything (right click the Everything tray icon and click Exit).
  • Restart Everything.


Is Everything reindexing often?

-Please make sure Everything is installed correctly:
  • In Everything, from the Tools menu, click Options.
  • Click the General tab on the left.
  • Check Store settings and data in %APPDATA%\Everything.
  • Click OK.
-Please try disabling automatically include new volumes and automatically remove offline volumes:
  • In Everything, from the Tools menu, click Options.
  • Click the NTFS tab on the left.
  • Uncheck Automatically include new fixed volumes.
  • Uncheck Automatically include new removable volumes.
  • Uncheck Automatically remove offline volumes.
  • Click OK.
-Please try increasing your USN Journal size to at least 131072 KB:
  • In Everything, from the Tools menu, click Options.
  • Click the NTFS tab on the left.
  • For each NTFS volume:
    • Set Maximum size to: 131072 KB
  • Click OK.
-Please make sure Windows is shutting down gracefully. (bluescreens will cause Everything to lose recent changes to your file systems)
-Please make sure Everything is shutting down gracefully on system shutdown. (don't terminate Everything on shutdown)
-Please make sure there's no pending Windows updates that are stuck. (stuck updates can make millions of file changes triggering a reindex)
-Please try storing your Everything database on another drive (Tools -> Options -> Indexes -> Database location)
Thy Grand Voidinesss
Posts: 691
Joined: Wed Jun 01, 2022 5:01 pm

Re: What exactly does "rebuild index" mean?

Post by Thy Grand Voidinesss »

void wrote: Sat Mar 02, 2024 4:52 am [...]
Do I ever have to rebuild the index?
Generally, no.

You will need to rebuild if you:
[...]
After running a check disk that made a change to your volume.
[...]
What kind of changes? And how is a user suppose to know that such changes have been made?

I use chkdsk rarely and with the /F parameter in hopes of fixing things when something is apparently not working
Thy Grand Voidinesss
Posts: 691
Joined: Wed Jun 01, 2022 5:01 pm

Re: What exactly does "rebuild index" mean?

Post by Thy Grand Voidinesss »

Thy Grand Voidinesss wrote: Sat Apr 20, 2024 6:42 am What kind of changes?
[...]
Well?
void
Developer
Posts: 16773
Joined: Fri Oct 16, 2009 11:31 pm

Re: What exactly does "rebuild index" mean?

Post by void »

Any errors fixed by chkdsk.

chkdsk will report detected errors and fixes.
Thy Grand Voidinesss
Posts: 691
Joined: Wed Jun 01, 2022 5:01 pm

Re: What exactly does "rebuild index" mean?

Post by Thy Grand Voidinesss »

So even a single chkdsk fix that affects [i.e. corrects] accessibility of just 1 item of minuscule size that happens to be located on a 22 TB HDD will require from user a manual follow-up in Everything in form of re-inxdexing that whole drive? Because there is no way for Everything to notice just that 1 fixed item?
void
Developer
Posts: 16773
Joined: Fri Oct 16, 2009 11:31 pm

Re: What exactly does "rebuild index" mean?

Post by void »

Correct.

chkdsk fixes are not added the USN Journal.

Everything will be unaware of any chkdsk fixes to your volume.

Reindex the volume to pickup the change.
Thy Grand Voidinesss
Posts: 691
Joined: Wed Jun 01, 2022 5:01 pm

Re: What exactly does "rebuild index" mean?

Post by Thy Grand Voidinesss »

Could at least there be a way for Everything [through an update] to take some sort of checksum / ID of a freshly indexed / properly working drive - and then to compare this snapshot info with a re-check up performed every time in background by Everything every single time it would be opened? And if mismatch is detected to then display a pop-up window to user with information of arose issue - and thus a need for re-indexation?

Because with how things works now it takes just a single slip of mind of user who will perform chkdsk but then forget to check its results and [if needed] to do also an re-indexing in order to start getting false results of not finding an item that a user remembers having. And that is assuming a user even knew in the first place about that nasty caveat; as my empirical example shows, I personally got to know about this only because the topic of this thread seemed like it might be of value of me in context of my own issues [with keeping volume C being indexed] and then indulged in posting until I got an answer understandable to my slightly above mediocre level of computing skills
void
Developer
Posts: 16773
Joined: Fri Oct 16, 2009 11:31 pm

Re: What exactly does "rebuild index" mean?

Post by void »

Everything does keep track of the Journal ID.
Everything will automatically reindex your volume if the Journal ID changes.

If chkdsk is fixing errors you have bigger issues to worry about than Everything missing an item.
Thy Grand Voidinesss
Posts: 691
Joined: Wed Jun 01, 2022 5:01 pm

Re: What exactly does "rebuild index" mean?

Post by Thy Grand Voidinesss »

void wrote: Sun May 12, 2024 12:02 pm [...]
If chkdsk is fixing errors you have bigger issues to worry about than Everything missing an item.
Yes- I know

And no- not if it do fixes them correctly; right?


I just wanted to emphasize how many users might unbeknownst brake the Everything monitoring everything concept
Last edited by Thy Grand Voidinesss on Sun May 12, 2024 1:22 pm, edited 3 times in total.
Thy Grand Voidinesss
Posts: 691
Joined: Wed Jun 01, 2022 5:01 pm

Re: What exactly does "rebuild index" mean?

Post by Thy Grand Voidinesss »

void wrote: Sun May 12, 2024 12:02 pm [...]
Everything will automatically reindex your volume if the Journal ID changes.
[...]
And what can cause that [positive fali-safe]?
void
Developer
Posts: 16773
Joined: Fri Oct 16, 2009 11:31 pm

Re: What exactly does "rebuild index" mean?

Post by void »

And no- not if it do fixes them correctly; right?
It could be nothing, it could be a program not loading settings or missing data, it could be Windows unable to load your registry.
Please make sure Windows is shutting down gracefully and consider replacing this drive if chkdsk is consistently fixing errors.


I just wanted to emphasize how many users might unbeknownst brake the Everything monitoring everything concept
Everything should continue to monitor changes to files.
Only the chkdsk-repaired-files might be out of sync in Everything.
Everything will resync if these repaired files are modified outside of chkdsk.
If you know the repaired filenames, you could resync them by adding the read-only attribute and then removing the read-only attribute.
I will consider automatically reindexing NTFS volumes once a month.
For now, this is an action left up to the user under Tools -> Options -> Index -> Force Rebuild.


Everything will automatically reindex your volume if the Journal ID changes.
And what can cause that [positive fali-safe]?
Deleting and recreating the USN Journal will generate a new USN Journal ID.
It's possible for chkdsk to recreate the USN Journal.
Thy Grand Voidinesss
Posts: 691
Joined: Wed Jun 01, 2022 5:01 pm

Re: What exactly does "rebuild index" mean?

Post by Thy Grand Voidinesss »

void wrote: Mon May 13, 2024 1:26 am [...]
I will consider automatically reindexing NTFS volumes once a month.
Maybe there could / should be a friendly pop-up remainder about this, with a brief explanation of just why the user could be in need of re-indexation?

A first one could be obligatory [i.e. turned on by default] but that popup window could have an option in it to easily either turn off this remainder off or to adjust its frequency

Thy Grand Voidinesss wrote: Sun May 12, 2024 1:19 pm
void wrote: Sun May 12, 2024 12:02 pm [...]
Everything will automatically reindex your volume if the Journal ID changes.
[...]
And what can cause that [positive fali-safe]?
void wrote: Mon May 13, 2024 1:26 am Deleting and recreating the USN Journal will generate a new USN Journal ID.
It's possible for chkdsk to recreate the USN Journal.
But possible as in chkdsk has such option [but I do not see it in its help] - or possible as in chkdsk might perform recreation when it decides on its own that it is needed [and thus a user of Everything who does not know about all of this might be wondering why Everything is performing a rescan]?
Post Reply