Any advice? I have included screenshots of the Everything config for the data it harvests in case you're wondering. Thank you for continued help guys.\\
Huge DB files that keep growing
Huge DB files that keep growing
I have the 1.5 Everything Search utility set up to work with Directory Opus, and love it. Good stuff. However, I have a RAMdisk and its about 14gb in size, and slowly but surely the space is dwindling away, and I found today that the folder I have on the RAMdisk for Everything is nearly 6gb in size. it has the db file, which is 2.1gb, the backup db file, same size. and the tmp file which is 1.18gb in size. I have been struggling with longer and longer folder loading times, and wondering if this is playing a part in it. I just now tried rebuilding the db, and it didn't do anything besides make the files larger lol.
Any advice? I have included screenshots of the Everything config for the data it harvests in case you're wondering. Thank you for continued help guys.\\
Any advice? I have included screenshots of the Everything config for the data it harvests in case you're wondering. Thank you for continued help guys.\\
Re: Huge DB files that keep growing
Have you enabled content indexing ?
Re: Huge DB files that keep growing
There is no need to store the Everything database(s) on a RAM-drive,
Everything keeps its entire database in RAM and will only write it to disk when exiting
(and after 24 hours of activity).
When Everything starts, the database will be read from disk.
And that is about all disk activity regarding the database.
Typically 1 million items (folders/files) will take around 100MB, so if you have (ballpark) 20 million items indexed - the actual amount is shown in the statusbar - this 2GB size of the database is to be expected.
However ...
Do you have also indexed other file properties (beside the ones shown in your screenshot)? (under Options => Indexes => Properties)
Do you have indexed file content? (under Options => Indexes => Content)
File properties and -content are added to the datbase too and can make up a large portion of the size of it.Especially file content...
Everything keeps its entire database in RAM and will only write it to disk when exiting
(and after 24 hours of activity).
When Everything starts, the database will be read from disk.
And that is about all disk activity regarding the database.
Typically 1 million items (folders/files) will take around 100MB, so if you have (ballpark) 20 million items indexed - the actual amount is shown in the statusbar - this 2GB size of the database is to be expected.
However ...
Do you have also indexed other file properties (beside the ones shown in your screenshot)? (under Options => Indexes => Properties)
Do you have indexed file content? (under Options => Indexes => Content)
File properties and -content are added to the datbase too and can make up a large portion of the size of it.Especially file content...
Re: Huge DB files that keep growing
Yessir I do, that first picture shows the things that I am telling Everything to index.
For other readers interest: I have moved the database location away from the RAM Disk and back on a local physical drive. Just like the above guy shared, it being on the RAM disk provided absolutely zero increase in performance (judged by the software's performance now that its back on a local drive. I notice zero discernable change to search times or the like since moving, I know - not exactly using measurable data points to use as evidence but its enough for me). Returned a large chuck on my RAM drive's space. Ill provide updates if anything relevant happens after this change. Thank you guys for the quick response and all the help
Re: Huge DB files that keep growing
No, it does not !
It looks like you don't know anything about content indexing.
To show what you have enabled in Content indexing you have to show us a picture like the following
- Screenshot - 25.08.2024 , 10_25_39.png (31.86 KiB) Viewed 1558 times
Re: Huge DB files that keep growing
You are right, I am very sorry for my ignorance. I really do enjoy seeing your enthusiasm... I'm glad youre using it to help and not do the opposite lol.
Here is the short of it - nothing is selected on that page . here is a SS:
In fact, every other panel in the options panel is blank except for exclude due to a few specific files and folders I found were driving the journal insane. Again, I'm sorry for my stupidity and frustrating you. Thank you for taking the time to help me.
Here is the short of it - nothing is selected on that page . here is a SS:
In fact, every other panel in the options panel is blank except for exclude due to a few specific files and folders I found were driving the journal insane. Again, I'm sorry for my stupidity and frustrating you. Thank you for taking the time to help me.
Re: Huge DB files that keep growing
How many files are being indexed?
Is it expected to be seeing (normally) the .db.tmp file?
As in, presumably ? it should "write out" to disk, then be renamed to .db, & at that point all there should be would be .db & .backup.db.
(Now, maybe with 2GB it's taking a bit of time [to "write out"], but still the .db.tmp should not be hanging around, forever.)
Is it expected to be seeing (normally) the .db.tmp file?
As in, presumably ? it should "write out" to disk, then be renamed to .db, & at that point all there should be would be .db & .backup.db.
(Now, maybe with 2GB it's taking a bit of time [to "write out"], but still the .db.tmp should not be hanging around, forever.)
Re: Huge DB files that keep growing
Here is my stats, which by trying to answer this question, I found this nifty little bit of info. I know its alot, but if any extra knowledge can come from looking through this, then its worth posting! Thanks for taking interest in my post, therube.therube wrote: ↑Mon Aug 26, 2024 4:02 pm How many files are being indexed?
Is it expected to be seeing (normally) the .db.tmp file?
As in, presumably ? it should "write out" to disk, then be renamed to .db, & at that point all there should be would be .db & .backup.db.
(Now, maybe with 2GB it's taking a bit of time [to "write out"], but still the .db.tmp should not be hanging around, forever.)
Code: Select all
Database
Location: E:\Indexes\Everything-1.5a.db
Indexed file properties: Name, Path, Size, Date Created, Date Modified, Attributes, Reparse Tag, Reparse Target
Indexed folder properties: Name, Path, Size, Date Created, Date Modified, Attributes, Reparse Tag, Reparse Target
Fast sorts: Name, Path, Size, Date Modified
Folder count: 1,477,135
File count: 11,732,245
Total item count: 13,209,380
FAT index count: 1
NTFS index count: 15
ReFS index count: 0
Network drive index count: 0
Folder index count: 0
File list index count: 0
Network index count: 0
Total index count: 16
Folder data size: 111,298,069 bytes
File data size: 842,904,979 bytes
Total data size: 954,203,048 bytes
Average folder data size: 75 bytes
Average file data size: 71 bytes
Folder index size: 47,268,320 bytes
File index size: 375,431,840 bytes
Total index size: 422,700,160 bytes
Total size: 1,376,903,208 bytes
Folders created: 1,225
Folders modified: 341,134
Folders deleted: 555
Folders moved: 193
Files created: 71,978
Files modified: 245,494
Files deleted: 17,090
Files moved: 2,634
Journal
Enabled: Yes
ID: x
Size: 278,970 bytes
Max size: 1,048,576 bytes
First item ID: 1
Next item ID: 1984
Item count: 1,983
Build
Count: 10
Total duration: 09:57
Minimum duration: 27.954561 seconds
Maximum duration: 116.478890 seconds
Average duration: 59.737028 seconds
Last duration: 27.954561 seconds
Last build date: 8/25/2024 7:42 PM
Last rebuild reason: Excludes changed.
Update
Count: 484,657
Total duration: 00:09
Minimum duration: 0.000000 seconds
Maximum duration: 0.040543 seconds
Average duration: 0.000020 seconds
Last duration: 0.000003 seconds
Last update date: 8/26/2024 12:08 PM
Load
Count: 9
Total duration: 00:42
Minimum duration: 2.787504 seconds
Maximum duration: 6.300581 seconds
Average duration: 4.719368 seconds
Last duration: 3.273314 seconds
Last load date: 8/26/2024 12:06 PM
Save
Count: 14
Total duration: 01:16
Minimum duration: 3.203993 seconds
Maximum duration: 7.022858 seconds
Average duration: 5.429008 seconds
Last duration: 3.780808 seconds
Last save date: 8/26/2024 5:48 AM
Next scheduled save date: 8/27/2024 4:00 AM
Total bytes written: 21,698,707,512
Query
Count: 4,235
Total duration: 06:17
Minimum duration: 0.000002 seconds
Maximum duration: 1.822352 seconds
Average duration: 0.089026 seconds
Last duration: 0.000024 seconds
Last query date: 8/26/2024 12:06 PM
Total result count: 95,215,195
Maximum result count: 8,012,085
Average result count: 22,482
Last result count: 0
Sort
Count: 3
Total duration: 00:00
Minimum duration: 0.000339 seconds
Maximum duration: 0.006275 seconds
Average duration: 0.003792 seconds
Last duration: 0.000339 seconds
Last sort date: 8/26/2024 3:30 AM
FAT Index
Volume name: \\?\Volume{a6}
Path: A:
Root:
Include only:
Drive type: Fixed
Label: RAMDisk
Index number: 0
Date indexed: 8/23/2024 11:13 AM
Out of date: Yes
Online: No
Disk device index:
Multithreaded: Separate device thread
Folder count: 1
File count: 0
Last rescan date: 8/26/2024 3:00 AM
Last rescan successful: No
Last successful rescan date:
Last successful rescan duration:
Next scheduled rescan date: 8/27/2024 3:00 AM
NTFS Index
Volume name: \\.\A:
Path: A:
Root:
Include only:
Drive type: Fixed
Label: RAMDISK
Index number: 1
Date indexed: 8/23/2024 11:13 AM
Out of date: No
Online: Yes
Disk device index:
Multithreaded: Separate device thread
Folder count: 16
File count: 37
USN Journal ID: 01daf2901fdbc95d
Next USN: 0000000003745e10
NTFS Index
Volume name: \\?\Volume{a6}
Path: C:
Root:
Include only:
Drive type: Fixed
Label: 990 Pro 2GB
Index number: 2
Date indexed: 8/23/2024 11:13 AM
Out of date: No
Online:
Disk device index: 3
Multithreaded: Yes
Folder count: 129,470
File count: 1,089,284
USN Journal ID: 01dad8f9fac122b8
Next USN: 0000000107807c70
NTFS Index
Volume name: \\?\Volume{fa}
Path: D:
Root:
Include only:
Drive type: Fixed
Label: 860 QVO
Index number: 3
Date indexed: 8/23/2024 11:13 AM
Out of date: No
Online: Yes
Disk device index: 5
Multithreaded: Yes
Folder count: 313,145
File count: 418,433
USN Journal ID: 01d7f2a04902f223
Next USN: 00000002c8ef6a98
NTFS Index
Volume name: \\?\Volume{70}
Path: E:
Root:
Include only:
Drive type: Fixed
Label: 970 Evo+
Index number: 4
Date indexed: 8/23/2024 11:13 AM
Out of date: No
Online: Yes
Disk device index: 11
Multithreaded: Yes
Folder count: 29
File count: 1,677
USN Journal ID: 01d8f42eca71b928
Next USN: 00000000990b3960
NTFS Index
Volume name: \\?\Volume{70}
Path: H:
Root:
Include only:
Drive type: Fixed
Label: WD Blue
Index number: 5
Date indexed: 8/23/2024 11:13 AM
Out of date: No
Online: Yes
Disk device index: 4
Multithreaded: Yes
Folder count: 337
File count: 4,531
USN Journal ID: 01d7f2a0490b90fb
Next USN: 00000002fe792ca8
NTFS Index
Volume name: \\?\Volume{45}
Path: I:
Root:
Include only:
Drive type: Fixed
Label: 980 Pro
Index number: 6
Date indexed: 8/23/2024 11:13 AM
Out of date: No
Online: Yes
Disk device index: 2
Multithreaded: Yes
Folder count: 2,786
File count: 108,432
USN Journal ID: 01da37dc84aaf23d
Next USN: 0000000004f37660
NTFS Index
Volume name: \\?\Volume{35}
Path: J:
Root:
Include only:
Drive type: Fixed
Label: 4TB Juliett
Index number: 7
Date indexed: 8/23/2024 11:13 AM
Out of date: No
Online: Yes
Disk device index: 10
Multithreaded: Separate device thread
Folder count: 930
File count: 18,855
USN Journal ID: 01d933d0cadfcca8
Next USN: 000000000349a7b8
NTFS Index
Volume name: \\?\Volume{6b}
Path: K:
Root:
Include only:
Drive type: Fixed
Label: 990 Pro 1TB
Index number: 8
Date indexed: 8/23/2024 11:13 AM
Out of date: No
Online: Yes
Disk device index: 12
Multithreaded: Yes
Folder count: 51,037
File count: 1,023,537
USN Journal ID: 01daa0ac55dacfed
Next USN: 00000000aa243da0
NTFS Index
Volume name: \\?\Volume{03}
Path: T:
Root:
Include only:
Drive type: Fixed
Label: 4TB Juliet
Index number: 9
Date indexed: 8/23/2024 11:13 AM
Out of date: No
Online: Yes
Disk device index: 9
Multithreaded: Separate device thread
Folder count: 999
File count: 25,103
USN Journal ID: 01da1e27276caa47
Next USN: 000000000128c9f8
NTFS Index
Volume name: \\?\Volume{a6}
Path: U:
Root:
Include only:
Drive type: Fixed
Label: 5TB India
Index number: 10
Date indexed: 8/23/2024 11:13 AM
Out of date: No
Online: Yes
Disk device index: 8
Multithreaded: Separate device thread
Folder count: 1,479
File count: 13,733
USN Journal ID: 01d8b3dafd3a39d4
Next USN: 0000000000ef8108
NTFS Index
Volume name: \\?\Volume{70}
Path: V:
Root:
Include only:
Drive type: Fixed
Label: 8TB Hotel
Index number: 11
Date indexed: 8/23/2024 11:13 AM
Out of date: No
Online: Yes
Disk device index: 6
Multithreaded: Separate device thread
Folder count: 30,312
File count: 405,958
USN Journal ID: 01d7f2a0490c7aba
Next USN: 000000001dbbe880
NTFS Index
Volume name: \\?\Volume{3f}
Path: W:
Root:
Include only:
Drive type: Fixed
Label: 8TB Gulf
Index number: 12
Date indexed: 8/23/2024 11:13 AM
Out of date: No
Online: Yes
Disk device index: 13
Multithreaded: Separate device thread
Folder count: 933,534
File count: 8,248,908
USN Journal ID: 01d7f2a04913cb88
Next USN: 0000000078420210
NTFS Index
Volume name: \\?\Volume{da}
Path: X:
Root:
Include only:
Drive type: Fixed
Label: 4TB Delta
Index number: 13
Date indexed: 8/23/2024 11:13 AM
Out of date: No
Online: Yes
Disk device index: 1
Multithreaded: Separate device thread
Folder count: 7,754
File count: 358,186
USN Journal ID: 01d7f2a04917375b
Next USN: 0000000033bd35e8
NTFS Index
Volume name: \\?\Volume{81}
Path: Y:
Root:
Include only:
Drive type: Fixed
Label: 8TB Foxtrot
Index number: 14
Date indexed: 8/23/2024 11:13 AM
Out of date: No
Online: Yes
Disk device index: 7
Multithreaded: Separate device thread
Folder count: 2,869
File count: 7,857
USN Journal ID: 01d7f2a0494e48ac
Next USN: 00000000033fbe20
NTFS Index
Volume name: \\?\Volume{a6}
Path: Z:
Root:
Include only:
Drive type: Fixed
Label: 5TB Tango
Index number: 15
Date indexed: 8/23/2024 11:13 AM
Out of date: No
Online: Yes
Disk device index: 0
Multithreaded: Separate device thread
Folder count: 2,437
File count: 7,714
USN Journal ID: 01d7ffca5c445088
Next USN: 000000000be25f80
Last edited by void on Tue Aug 27, 2024 12:14 am, edited 2 times in total.
Reason: trimmed log
Reason: trimmed log
Re: Huge DB files that keep growing
By default, Everything will use about 100MB per 1million files.
13 million files will require about 1300MB of RAM.
Enabling Date Created indexing will use about 8 MB per 1 million files (100MB total in this case)
Enabling Attributes indexing will use about 4 MB per 1 million files (50MB total in this case)
Please try disabling 'include files' for the reparse tag and reparse target properties:
Optimal settings for Everything
13 million files will require about 1300MB of RAM.
Enabling Date Created indexing will use about 8 MB per 1 million files (100MB total in this case)
Enabling Attributes indexing will use about 4 MB per 1 million files (50MB total in this case)
Please try disabling 'include files' for the reparse tag and reparse target properties:
- In Everything 1.5, from the Tools menu, click Options.
- Click the Properties tab on the left.
- To the right of Show settings containing, search for:
- Select Reparse Tag
- Uncheck Include files. (this will save about 100MB)
- Select Reparse Target
- Uncheck Include files. (this will save about 100MB)
- Click OK.
Optimal settings for Everything
Re: Huge DB files that keep growing
Thank you for posting the optimal setting info you did. very useful information for me to have in the future. Yes, the reparse information that i have designated was to help me find other junctions, soft links, and hard links and see how they were being set up and used throughout windows. I really appreciate your helping with this mess I call a configuration of mine. 
