[Solved] Huge RAM leakage leading to a system crash + high CPU consumption.

[Turok]

(Definitive solution in my case here: viewtopic.php?p=73393#p73393)

Hello,
First I'd like to thank the creator of such an awesome program, I really love it and couldn't live without it. A million thanks for keeping updating it and adding key features like sessions, which helped me a lot

Now the problem I'm facing is that over time Everything decides to eat all my ram and fill up the pagefile as well until my system crashes.
I don't understand its behaviour because yes, I have dozens of tabs opened at the same time, but without opening more or even interacting at all with the program for a while, the program starts to consume more and more ram, from 15-30gb (it increases over time, but slowly for a while) to all of it and even a lot from a pagefile of more than 100GB (specially huge for Everything to last longer before making the system crash).

I don't know if it eats all of the available pagefile or just an enough amount to make the system unable to handle it and lead to a crash. Honestly, I don't know if this is possible. What I've seen is that when the free ram is close or lower than 1GB and I check the pagefile usage, the amount in use is about a 40-50% of the total available. But then I quit from Everything, so I don't know how much it ends consuming until the crash happens. After closing Everything up to 40GB of ram are recovered. Then restoring the previous session leaves the system with around 20-25GB of free ram. It keeps using a similar amount for a few hours, until suddenly the above mentioned happens, and if I'm not paying attention it ends up in the system crashing due to the system getting out of ram.

It has also a CPU usage of more than a 15%, is this normal?
I'm sitting on a 64GB system and without Everything running the normal consumption can reach around 30GB due to other programs running.
DB size is only 1.5GB.

To add just something I've noticed is that the only property I have set (lenght) takes a lot to get updated, and it does it in bulks, not in real time as new files are stored but in a long process at a certain moment. I think in that process it consumes more ram. Is this the expected behaviour? any way to make it update that attribute in real time?

Screenshot taken short after starting Everything. I'll add others later as it keeps using more resources.

image.png (8.71 KiB) Viewed 2566 times

[therube]

None of what you say is "normal".

Your screenshot (since it doesn't show up), https://ibb.co/6v4pHRq.
(That's a lot of RAM .)

What version of Everything? [1.5.x ?]
How many files are we dealing with?
Aside from indexing just these files, are you also indexing any file Properties (if using 1.5)? [Length]
Post, Help | Troubleshooting Information.

[Turok]

Version 1.5.0.1383a (x64)
The total amount is around 25M, mostly are empty folders (I need to keep this folders tree). Time ago the total amount reached more than 100M (99% empty folders) and I had not this issue. It indeed used to use a lot of ram, but I don't remember it eating all the free RAM + Pagefile until making the system crash. I had then a 15-30GB .db, but it used to take a lot of time at each start for the program to be usable.
About the properties, only Length is being indexed.
Thank you for fixing the image

Now my index settings looks like this:


At some point I'd like to recover the +100M tree folder structure but I'm scared it won't be possible not even doubling the RAM if the Everything's voracity persists

[void]

Are you indexing content?

Please try disabling content indexing under Tools -> Options -> Content.

[Turok]

Nope, that option is already disabled.

[Turok]

Version 1.5.0.1383a (x64)
The total amount is around 25M, mostly are empty folders (I need to keep this folders tree). Time ago the total amount reached more than 100M (99% empty folders) and I had not this issue. It indeed used to use a lot of ram, but I don't remember it eating all the free RAM + Pagefile until making the system crash. I had then a 15-30GB .db, but it used to take a lot of time at each start for the program to be usable.
About the properties, only Length is being indexed.
Thank you for fixing the image

Now my index settings looks like this:


At some point I'd like to recover the +100M tree folder structure but I'm scared it won't be possible not even doubling the RAM if the Everything's voracity persists

I've been thinking and indeed when I had +100M folders sometimes the program was using a lot of ram + pagefile, just to free it up both after a while. Sometimes closing some other apps first, I don't remember if just to speed up the process or as a requirement for it to happen. I used to do this because how slow was to get the program ready after closing it. Now it's almost instant so I can restart it when I notice the leakage. The problem is when I don't notice it. Many times I leave the PC unattended for 10-20 minutes, I come back just to find the system has crashed. Checking the events viewer it says the system got out of virtual/memory.

[void]

Please send your Help -> Troubleshooting Information in a bugreport

For 100 million folders, I would expect Everything to use around 10GB of RAM.

[Turok]

Please send your Help -> Troubleshooting Information in a bugreport

For 100 million folders, I would expect Everything to use around 10GB of RAM.

Impressive, there is hope then.

I'll do it right away.

[void]

Thanks for the Help -> Troubleshooting Information.

Your setup looks OK.

request_extra_fileinfo_end=2000

This setting could be causing trouble.
This will prefetch 2000 thumbnails.

To change this setting:

• In Everything 1.5, from the Tools menu, click Options.
• Click the Advanced tab on the left.
• To the right of Show settings containing, search for:
  extra
• Select: request_extra_fileinfo_end
• Set the value to: 0
  (0 is the default)
• Click OK.

See if the issue persists when this is set to: 0

request_extra_fileinfo_end



Please try lowering your thumbnail size:

• In Everything 1.5, from the Tools menu, click Options.
• Click the Advanced tab on the left.
• To the right of Show settings containing, search for:
  thumbnail
• Select: thumbnail_load_size
• Set the value to: 64
• Click OK.

Could you please send your Tools -> Debug -> Statistics in a bugreport

[Turok]

Thanks for the Help -> Troubleshooting Information.

Your setup looks OK.

request_extra_fileinfo_end=2000

This setting could be causing trouble.
This will prefetch 2000 thumbnails.

To change this setting:

• In Everything 1.5, from the Tools menu, click Options.
• Click the Advanced tab on the left.
• To the right of Show settings containing, search for:
  extra
• Select: request_extra_fileinfo_end
• Set the value to: 0
  (0 is the default)
• Click OK.

See if the issue persists when this is set to: 0

request_extra_fileinfo_end



Please try lowering your thumbnail size:

• In Everything 1.5, from the Tools menu, click Options.
• Click the Advanced tab on the left.
• To the right of Show settings containing, search for:
  thumbnail
• Select: thumbnail_load_size
• Set the value to: 64
• Click OK.

Could you please send your Tools -> Debug -> Statistics in a bugreport

Hello, I've been testing this workaround, and it seemed very promising for a few hours. It was using only 5-6gb of ram most of the time, with some spikes to 10 and a few times to 25, but going down short after. Then, after a while, I noticed this:

image.png (6 KiB) Viewed 2308 times

Never seen such an high amount of ram being used by a process nor by adding them all up. Actually I thought that this value in Process Hacker was only ram, but my system has 64gb, so I must assume it also takes into account the virtual memory? weird.
Since then the program keeps using similar values to those wrote in my first paragraph most of the time, so it's an improvement, but time to time it still reaches an high usage (26, 38, 45) that leaves me with 1-2GB free and the program needs a restart. This, again, not even interacting with Everything or having done any new search (beyond those already in place after restoring the session) since the last restart. So if there is any other tweak I could try I'd be glad to hear.
Actually I wonder, as I mentioned I have dozens of tabs with their respective searches, and I restore this session on every restart. My logic is that the RAM consumption should stay stable with this method after the session is restored. But is this really the expected behavior? or may this be causing those increases in RAM usage as time goes by?

Your help is very appreciated
I'll send the statistics log in a while.

[therube]

Again, as your image is not displaying, https://imgur.com/EBI8zVz.
(I didn't see where imgur allows hotlinking?)

[void]

Are you showing thumbnails?
-Does the issue only persist when viewing thumbnails?
You might have a bad third party thumbnail handler..

[Turok]

Again, as your image is not displaying, https://imgur.com/EBI8zVz.
(I didn't see where imgur allows hotlinking?)

I don't know about hotlinking, sorry. I thought it wasn't being showed due to the host I used in my first post, so I swapped to IMGUR for my second and it worked there. Got it, I must use the raw link. Thank you for fixing it.

[Turok]

Are you showing thumbnails?
-Does the issue only persist when viewing thumbnails?
You might have a bad third party thumbnail handler..

You're right! I use Icaros. I have been using it for quite a few years now, since I suffered a bug with the thumbnails not being displayed or something similar, and never had a problem again.
"-Does the issue only persist when viewing thumbnails?"
Well, from the many tabs I have opened yes, many of them are showed as thumbnails.
So do you think this is causing the RAM leakage? would you be so kind to explain why, if it's a conflict or? No need for a detailed explanation, just a summarize, I'd just like to understand.
And what should I do then, uninstall Icaros? or is there some setting in Everything I can tweak to make it work better?

PS: Just noticed that Icaros it must be outdated, never worried about as it was fulfilling its purpose. I'll update it and check if it fixes this problem.

[void]

So do you think this is causing the RAM leakage?

It's possible thumbnails are causing the issue.

A thumbnail handler is most likely the cause if you can't reproduce the issue without showing thumbnails.

And what should I do then, uninstall Icaros? or is there some setting in Everything I can tweak to make it work better?

Some ideas:

Use a smaller thumbnail load size:

• In Everything 1.5, from the Tools menu, click Options.
• Click the Advanced tab on the left.
• To the right of Show settings containing, search for:
  thumbnail
• Select: thumbnail_load_size
• Set the value to: 64
  (where 64 is the desired thumbnail size in logical pixels)
• Click OK.

Limit thumbnails to the desired file extensions only:

• In Everything 1.5, from the Tools menu, click Options.
• Click the Advanced tab on the left.
• To the right of Show settings containing, search for:
  thumbnail
• Select: thumbnail_include_only_files
• Set the value to: *.png;*.jpg
  (where *.png;*.jpg is the semicolon (;) list of desired extensions)
• Click OK.

[Turok]

So do you think this is causing the RAM leakage?

It's possible thumbnails are causing the issue.

A thumbnail handler is most likely the cause if you can't reproduce the issue without showing thumbnails.

And what should I do then, uninstall Icaros? or is there some setting in Everything I can tweak to make it work better?

Some ideas:

Use a smaller thumbnail load size:

• In Everything 1.5, from the Tools menu, click Options.
• Click the Advanced tab on the left.
• To the right of Show settings containing, search for:
  thumbnail
• Select: thumbnail_load_size
• Set the value to: 64
  (where 64 is the desired thumbnail size in logical pixels)
• Click OK.

Limit thumbnails to the desired file extensions only:

• In Everything 1.5, from the Tools menu, click Options.
• Click the Advanced tab on the left.
• To the right of Show settings containing, search for:
  thumbnail
• Select: thumbnail_include_only_files
• Set the value to: *.png;*.jpg
  (where *.png;*.jpg is the semicolon (;) list of desired extensions)
• Click OK.

Ok, thank you for your advice, it's amazing to see such a nice technical support for an app that you bring us for free
If after updating Icaros the problem isn't solved I'll try those workarounds and I will come back to present the results.

[void]

Thank you for the statistics.

You are indexing 27 million files.
Everything is using 2.4 GB of RAM for the database.
Any other RAM usage is from the UI and most likely thumbnails.

[anmac1789]

Thank you for the statistics.

You are indexing 27 million files.
Everything is using 2.4 GB of RAM for the database.
Any other RAM usage is from the UI and most likely thumbnails.

Why are thumbnails causing so much CPU usage ? arent thumbnails just compressed lesser quality images?

[void]

The Everything thumbnail cache can grow rather large.
Everything only frees thumbnail cache when you close the window/tab or press F5.

The default thumbnail size is 256x256. (can be set with thumbnail_load_size -see above)
The bit depth is 24bits.
Each thumbnail is 200KB.

5000 thumbnails = 1GB of RAM.

[anmac1789]

The Everything thumbnail cache can grow rather large.
Everything only frees thumbnail cache when you close the window/tab.

The default thumbnail size is 256x256. (can be set with thumbnail_load_size -see above)
The bit depth is 24bits.
Each thumbnail is 200KB.

5000 thumbnails = 1GB of RAM.

oh my lord, okay ....definitly overlooked that one. Is there a way to flush out the thumbnails after it reaches a certain threshold so that this leak or overload of CPU resources does not happen ?

[void]

F5 will flush the cache.
Closing the tab/window will flush the cache.

I have on my TODO list to set a thumbnail cache limit.

[Turok]

It's Solved! The problem was my diogenes syndrome.
Sorry for the delay in my answer. In this lapse I've tried the solutions proposed and these were my (now useless) conclussions:

Updating Icaros didn't work.
In Everything the thumbnail_load_size was set at 0. After changing it to 64 yes, it seemed to solve the issue, but I'm not 100% sure since just left it for a short time because I can't handle such little and low resolution thumbnails
So I tried the other tip and limited the extensions that are shown as thumbnails (just videos and pictures, without these it would be useless to me). This didn't work and RAM leakage happened again, it just took longer for the leaking process to start.
On my end pressing F5 to flush the cache frees up to 3-4 gb only, and short after this amount is used again by the program itself.
After noticing a non-Everything related issue but also of what apparently was a case of RAM leakage (lots of gigs used but not by processes), I saw this option in RamMap: "Empty working sets" and tried it. It actually helped, freeing up more than a 50% of that ghost RAM.
Even if after a while it was in use again, it helped in anticipating the Everything leakage to avoid the final PC crash, and also seemed to help in extending the time it took for the Everything leak to happen.

After that and feeling there was not a satisfying way to solve this I just decided to live with it and to live "paranoid", trying to be extra sensible to any strange lag due to a low ram usage and looking at this value in a bit obsessive way to prevent the massive leakage from happening, as well as emptying working sets when noticing a low free memory amount. And it worked, I've had almost no PC crashes in the past few weeks.
But today I've finally done what I should have done from the start. Closing a few dozens of accumulated opened tabs in Everything totally solved the issues, getting a massive reduction in both RAM and CPU usage. These are my values now:

image.png (21.12 KiB) Viewed 1029 times

image.png (144.45 KiB) Viewed 1029 times

So my solution and advice is: Stay minimalist
Thank you for your help and advice.