Index all datastreams of a file

[NotNull]

Possibly not a mainstream suggestion, but it might be useful to also index the alternate/ alternative/ named datastreams (short:ADS).

Couple of use cases:

• Some applications store metadata and tags of documents in this ADS. That would make this metadata searchable.
• (My main usecase:)Search for Zone.Identifier datastreams so that they can be deleted (frm within Everything).
  
  This Zone.Identifier datastream gets added to files you download from the internet. You have to unblock those (file properties > Unblock) before further processing is possible. Deleting the datastream does the same.
  Currently I use a script for that; it deletes all Zone.Identifier streams from all the files in my download folders.
• This gives a more accurate representation of actual disk space usage (currently only the unnamed datastream is 'counted')

[void]

It's on my TODO list, currently Everything ignores named data streams.

Thanks for the suggestion.

[collinchaffin]

I wasn't sure if I should post here, or resurrect the decade-old thread here: viewtopic.php?f=4&t=4481

Either way after searching to see if ADS indexing had been added yet to Everything, I came across that above thread asking for a mainstream example of app(s) that actively use xattr alternate data streams and was blown away that the one that is WILDLY popular in the last decade was never mentioned since it's entire backup/restore operation (primary) depends on xattr ADS.

That tool which I'm certain void has not only heard of, but may have even used is FILEBOT. If you go grab v4.79 (the last fully freeware) although even the > 4.80beta/final (now shareware) still maintain this functionality. Filebot stores the original filename and some other metadata within ADS of the files it manipulates which it then uses as the primary to "undo". It obviously is not the only app utilizing ADS that without specialized one-off utils like Nirsoft or scripts today makes them virtually invisible to most users. ES could change all that.

With that said, back to the real-world example - anyone who specifically uses Filebot against large media libraries (tens-hundreds of TB+) would absolutely see Everything as a godsend to be able to have it index, store, then use it's instant-db query to report on the origin of the files etc.

Today, there are different scripted solutions (I myself am a Powershell guy and have now been doing PoSH ADS querying) but we rely on (if we even store non-volatile at all) implementing our own back-end storage methods from flat data files, to sqlite etc. I myself already use Powershell to query Everything though, too and boy since I already store all my media library standard file info from all those source drives into Everything how awesome would that be even if it was a folder-like manual refresh process to just have that data in ES not only for my scripts, but heck just for quick searches!

[verviers]

[*](My main usecase:)Search for Zone.Identifier datastreams so that they can be deleted (frm within Everything).

It's on my TODO list

I have exactly the same purpose. Five years have passed. Is it still not possible?

[void]

Everything 1.5 will have support for a Zone ID, Referrer URL and Host URL properties.

These properties can be indexed.

Zone ID, Referrer URL and Host URL properties

[verviers]

These properties can be indexed.

OK. But it is still not possible to find ADS named "Zone.Identifier" and to delete them in Everything GUI? Did I understand correctly?

[void]

It can be done with the Alternate Data Stream Names property in Everything 1.5.

Example search:

adsname:"Zone.Identifier"

adsname:



Everything doesn't have any functionality to add, remove or modify alternate data streams.

[verviers]

Everything doesn't have any functionality to add, remove or modify alternate data streams.

It is clear to me now. Thanks. It will be hard enough for me to delete thousands of "Zone.Identifier" streams manually.

[NotNull]

I wouldn't dare to spoil your fun removing thousands of "Zone.Identifier" streams manually, but if you get RSI or cramp halfway, one method to remove them all at once (the safest method I know):

• Search for all files with a Zone.Identifier dfatastream:

  Code: Select all

  C:|X:   alternate-data-stream-names:zone.identifier

  ( C:|X: = limit the results to your NTFS formatted volumes only)
• Menu:File => Export
• Save as type: TXT Text Files (*.txt)
• Start PowerShell
• Older versions of PowerShell did not support Unblock-File, so test first if version is OK by typing the following command (+ENTER)

  Code: Select all

  unblock-file "C:\windows\notepad.exe"

• If this command runs without error, continue.
  If not: report back (there are more ways ..)
• Type or paste the following command:

  Code: Select all

  gc T:\unblock.txt | %  {unblock-file $_ } 

  (where T:\unblock.txt is the name of the file with the exported filenames)
• Done!

[void]

To add a right click context menu item to unblock files:

• From the Start menu, search for:
  regedit
• Right click the Registry Editor and click Run as administrator.
  ---
• In the Registry Editor, navigate to:
  HKEY_CLASSES_ROOT\*\shell
• On the right side, right click and click New -> Key
• Set the Key name to: Unblock
  ---
• Navigate to:
  HKEY_CLASSES_ROOT\*\shell\Unblock
• On the right side, right click and click New -> Key
• Set the Key name to: command
  ---
• Navigate to:
  HKEY_CLASSES_ROOT\*\shell\Unblock\command
• Set the (default) value data to:
  powershell.exe -noprofile -Command "& {unblock-file -path '%1'}"

Search for your files with a zone.identifier in Everything with:
C:|X: alternate-data-stream-names:zone.identifier
Select all results
Right click and click Unblock (I recommend performing this action for less than 15 files at a time*)

*edit: 15 or less files otherwise the menu item is not shown (see below)



To remove the right click context menu item to unblock files:

• From the Start menu, search for:
  regedit
• Right click the Registry Editor and click Run as administrator.
• Navigate to:
  HKEY_CLASSES_ROOT\*\shell\Unblock
• Delete this key.

[hamid56]

i recommend stream remover v1(made in 2018) from nice sordum website for removing ADS.
https://www.sordum.org/11263/streams-remover-v1-0/
or
https://www.sordum.org/files/download/s ... emover.zip

[NotNull]

Thanks for sharing @hamid56!

[verviers]

Have tested three methods (NotNull, void, hamid56). void and hamid56 "right click solution" works only if I select up to 15 files.

screen_2023.11.07.001.png (2.13 KiB) Viewed 11323 times

If the number of selected files is greater than 15, these menu items are not shown at all. Windows Server 2016x64.

So the only way for me now - to follow NotNull`s guide. Thanks everyone guys.

[NotNull]

if I select up to 15 files.

Windows limitation. You can increase this number in the registry.

More info

[verviers]

Windows limitation. You can increase this number in the registry.

Yep. Didn't get this information, because had faced to some PowerShell errors, connected with UTF-8 encoding, regarding to your and void`s sentences, and spent a lot of time trying to solve the issues. As a result, I had to modify commands as

Code: Select all

gc -encoding utf8 D:\del_zone.txt | %  {unblock-file $_ }

and

Code: Select all

powershell.exe -noprofile -Command "& {unblock-file -path '%1'}" -encoding utf8

-----------------------------------------------------------------------
I have to note, that Sordum Streams Remover looks not very suitable, because it creates notification window for every processed file from the list. So if there are about 40 000 files, it will be disaster. Not sure if we can turn off this feature.

[NotNull]

Well done!


Another alternative could be:
- Search for the Zone.Identifier files
- 'CTRL + A' to select them all (or select a few for testing)
- 'CTRL + SHIFT + C' to copy filenames including path to the clipboard
- In PowerShell, run the following command:
Get-Clipboard | % {unblock-file $_ }

(This command doesn't like quoted paths (like "c:\file.txt" ), so make sure Double quote copy as path is unchecked under Options > Results )



I tested Sordum Streams Remover(briefly)

The dialog boxes can be removed:

• Start Register Editor (regedit.exe) as administrator
• Browse to
  HKEY_CLASSES_ROOT\*\shell\ZXY-Streams-Remover\command
• Double-click (Default) in the right pane
• Remove the /I from the command, so the new command looks like:

  Code: Select all

  "C:\Program Files (x86)\Streams Remover\sRemover.exe"  "%1"

• Exit Register Editor

The process was still quite slow though: removing Zone.Identifier stream from all 14 matches took quite a while. Felt like 10 seconds.

[DeRobertis]

i recommend stream remover v1(made in 2018) from nice sordum site for removing ADS.

Another alternative: streams64, which is part of Microsoft's excellent (and free) Sysinternals Suite https://learn.microsoft.com/en-us/sysin ... nals-suite.

The shell command I use is

Code: Select all

C:\PATH_TO\SysinternalsSuite\streams64.exe -nobanner -d "%1"

[NotNull]

Thanks for sharing, DeRobertis!

I used this in the past too. It did remove *all* ADS streams however.
Some applications actually use ADS (on my system a couple of file managers)

Don't know the current situation of streams; my version is 7 years old ...

[horst.epp]

The actual Streams version is 1.6 from April 2020.
It deletes all streams.

You can use AlternateStreamView from Nirsoft.
It displays every stream of files and allows to just delete one named stream from them.
https://www.nirsoft.net/utils/alternate ... reams.html

[NotNull]

Thanks for testing, Horst!

[verviers]

Finally, I`d like to prefer these methods, mentioned above by NotNull for a large number of files with zone.identifiers:

Code: Select all

Menu:File => Export
Save as type: TXT Text Files (*.txt)
Start PowerShell
gc T:\unblock.txt | %  {unblock-file $_ } 

OR

Code: Select all

- Search for the Zone.Identifier files
- 'CTRL + A' to select them all (or select a few for testing)
- 'CTRL + SHIFT + C' to copy filenames including path to the clipboard
- In PowerShell, run the following command:
Get-Clipboard | % {unblock-file $_ }

Advantages:
- no need to modify registry, right click context menu etc
- no need to download external tools
- it`s easier to detect issues in Powershell console, when command fails due to specific errors (too long path; unicode path; wildcard character pattern issue).

[DeRobertis]

I used this in the past too. It did remove *all* ADS streams however.

To clarify: do you wish to remove all alternate data streams from any file that has a :Zone.Identifier, or do you wish to remove only :Zone.Identifier from files that contain it and other ADS?

a. if file contains(:Zone.Identifier) remove all ADS streams

b. if file contains(:Zone.Identifier) and other streams, remove only :Zone.Identifier

[NotNull]

For me it is b. : Remove just Zone.Identifier streams.

Can't speak for @verviers, but by the look of it (s)he is looking for b. too.

[DeRobertis]

Did you check how many files you have that require such special handling?

E.g., I have hundreds of files with Filebot ADS metadata, but when I search

Code: Select all

ads-count:>1 ads-name:Zone.Identifier

I see nothing. Perhaps it depends on one's workflow. I tend to clear Zone.Identifier first, before further processing.

[NotNull]

ads-count:>1

Ah, I see the miscommunication now ...

a. if file contains(:Zone.Identifier) remove all ADS streams
b. if file contains(:Zone.Identifier) and other streams, remove only :Zone.Identifier

The answer for me (and @verviers) is:

c. if file contains (:Zone.Identifier), remove only :Zone.Identifier

[DeRobertis]

As described above, Powershell Unblock-File is the most straightforward way to remove :Zone.Identifier via CLI.

I avoid Powershell where alternatives exist, as it is slow, especially when working with hundreds/thousands of files at a time. IME cmd, Java, and Groovy scripts can handle file queries and transformations in a fraction of the time.

My workflow generally involves filtering via Everything, then dragging the results to a scrap window of my file manager, Zabkat xplorer². From there I can call shell commands, custom scripts, etc.

Find what works best for your common use cases and technical abilities.