Content Indexing >1GB of text - adding extension .msg (Outlook) - program hangs partially

[tuska]

Hi,

Content Indexing
I am aware of the following:

Everything will keep content in memory.
Content indexing is intended for indexing user documents only.
I do not recommend indexing over 1GB of text.
For the best performance, set an include only folder.

On the occasion of a forum post, I have added the file extension .MSG under Indexes - Content for test purposes.
Previously, I had this file type indexed ONLY by Windows Search.

My other settings - exclusive *.msg files - are listed here.

1. 16.5.2023: Everything.db ... 1 239 283 809 Bytes
2. 16.5.2023: *.msg added in "Options - 1.5.0.1346a (x64)" under "Content" - Include only files:
   *.csv;*.doc;*.docx;*.eml;*.ini;*.ion;*.md;*.msg;*.odt;*.pdf;*.pps;*.ppt;*.pptx;*.txt;*.xls;*.xlsm;*.xlsx;*.xlt;*.xltm;*.xltx
3. 16.5.2023: Indexing properties... 41% -> Content indexing stopped while indexing a .msg file
4. 16.5.2023: PC cold boot performed
5. 16.5.2023: Content indexing continued and stopped at 99% when indexing a .msg file
6. 16.5.2023: PC shut down
7. 17.5.2023: PC booted up
8. 17.5.2023: Indexing properties... 92% -> Content indexing stopped while indexing a .msg file
9. 17.5.2023: Everything.db ... 1 297 419 641 Bytes
10. 17.5.2023: File - Exit, Terminate ... startet Everything!? ...
11. 17.5.2023: This time content indexing was finished!
12. 17.5.2023: Everything.db ... 1 389 321 195 Bytes
13. 18.05.2023 C:\Everything\Everything64.exe -update ... Everything.db saved without closing Everything 1.5.0.1346a (x64) first.
    Everything - ESC pressed > icon in the systray. --> OK, i.e. no problem (not even when PC booted up)
    Everything.db - 1.29 GB.
14. 19.05.2023 Indexes > Force Rebuild: OK (18 minutes) - ~600 000 objects (~485 000 files, ~110 000 folders)

The search queries in Everything could and can always be carried out without any problems!
At the moment I can work (including searching for file contents in *.msg files) as usual and without any problems in Everything.

I have more than enough RAM (only 20% to 25% is usually used) and the processor load is minimal
(I have listed the exact details below under "Device specifications").

For me the following questions arise:

1. Should the recommendation of "limit of 1GB of text" be considered as a guideline or should not be exceeded at all,
   not even with enough RAM?
2. Could a "Force Rebuild" be performed after a new file extension has been captured instead of an immediate indexing?
   There has been no problem with a "Force Rebuild".
3. Should separate Everything instances be created for certain file types, e.g. *.pdf?
   I currently see this as a possibility file content indexing for even more topics (file types, properties)
   to be carried out in Everything - taking into account the available main memory (RAM).
   1. RAM currently: 20% - 25% (depending on the number of browser windows, etc.)
      1. Everything 1.5.0.1346a (x64) - RAM: 1 552,8 MB
      2. Everything Service - RAM: 1,3 MB

Device specifications

• OS Name Microsoft Windows 11 Pro
• Version 10.0.22621 Build 22621
• Processor Intel(R) Core(TM) i5-12600K 12th Gen, 3.70 GHz, 10 Cores, 16 Logical Processors
• Installed RAM 32.0 GB (31.8 GB usable)
• System type 64-bit operating system, x64-based processor
• ---
• Edition Windows 11 Pro
• Version 22H2
• Installed on 05/10/2022
• OS build Build 22621.1702
• Experience Windows Feature Experience Pack 1000.22641.1000.0

[void]

Thank you for the issue report Tuska,

Could you please force a rebuild and send a mini crash dump when Everything gets stuck loading content from an msg file.

I suspect the iFilter for msg files is getting stuck reading content.



Could you please also send some debug output:
When Everything gets stuck reading msg content, could you please:

• In Everything, from the Tools menu, under the Debug submenu, check Verbose.
• From the Tools menu, under the Debug submenu, click Start Debug Logging....
• Let Everything sit for a minute.
• In Everything, from the Tools menu, under the Debug submenu, click Stop Debug Logging....
  ---This will open your Everything Debug Log.txt in Notepad---
• Could you please send this file to support@voidtools.com
  This log will show the iFilter calls, which might be stuck in a loop.

Should the recommendation of "limit of 1GB of text" be considered as a guideline or should not be exceeded at all,
not even with enough RAM?

Avoid going over 50% of total ram.
Otherwise, Everything will swap to disk too much during reindexing.

Could a "Force Rebuild" be performed after a new file extension has been captured instead of an immediate indexing?

Everything should automatically perform a quick reindex when adding a file extension to your content index.
I'm not quite sure what you are asking here.

Should separate Everything instances be created for certain file types, e.g. *.pdf?

It shouldn't be necessary.

[tuska]

2void
Thank you for your prompt response.

Could you please force a rebuild and send a mini crash dump
when Everything gets stuck loading content from an msg file.

The error did not occur with a "Force Rebuild" and still does not occur.
On 21.5. I did a "Force Rebuild" again (duration: 19 minutes).

Could you please also send some debug output:
When Everything gets stuck reading msg content,...

The error regarding indexing only occurred after I added *.msg
in "Options - 1.5.0.1346a (x64)" under "Content" - Include only files: ...,
namely after indexing ("quick reindex") immediately followed (WITHOUT "Force Rebuild").

I will therefore remove *.msg again, let do the "quick reindex", do a "Force Rebuild" and then
add *.msg again to check if this behavior occurs again.
I will report on the result and, if necessary, provide the desired "debug outputs".

Avoid going over 50% of total ram.
Otherwise, Everything will be swapped to disk too much.

Thank you for this information
I will not exceed this value from now on.

Could a "Force Rebuild" be performed after a new file extension has been captured instead of an immediate indexing?

Everything should automatically perform a quick reindex when adding a file extension to your content index.
I'm not quite sure what you are asking here.

After a "Force Rebuild" worked for me WITHOUT any problems, I was wondering if I could prevent the "quick reindex"
and instead in such a case do only a "force rebuild".

Thanks for the information and support!

[tuska]

2void
Test again on 22.5.2023: Surprise!!

- Options - 1.5.0.1346a (x64) > Content: *.msg removed ... quick reindex: ~1 minute > OK!
- Options - 1.5.0.1346a (x64) > Indexes: "Force Rebuild": 18 minutes > OK!
- Options - 1.5.0.1346a (x64) > Content: *.msg added ... quick reindex: ~1 minute > OK!
- Test with an email filed about 2 hours ago: The search query (ext:msg content:xxx) was successful!

It is not clear to me how the test results at that time came about.
I will follow up on the issue and occasionally include a different file extension in the file content indexing.

Thanks again for the valuable information and support!