Compare larger folders for migration

[Just Visiting]

I am in the process of slowly migrating one my large folders with thousands of files from my legacy W7 PC to a new laptop. At the moment, I wish to keep both folders on both drives and then merge the folders back and forth as necessary.

Is there a simple Everything script or command I can use to scan both folders (the one my laptop and the one on my desktop) when I copy them over to a temporary directly on each PC and show me only the most recent ones? Or perhaps there's an even simpler approach you can recommend? For example. The main folder is called MYdocuments on Laptop and the other is called the same on my PC. I use a USB drive to copy the folder onto and then would run Everything to compare the differences.

[horst.epp]

Thats a easy job for any file manager you may try (not Windows Explorer of course).
Synchronize Directories in Total Commander for example.

[therube]

If it's a straight 1 directory to 1 directory comparison, I'd agree with the file manager argument that should easily point out differences.

For a mixed method, also including Everything, see [freefilesync] Problem using the Mirror function.

[Just Visiting]

Ok. so I have downloaded Total Commander and Free. Now what do I do?

How do I use Freefilesync?

If it's a straight 1 directory to 1 directory comparison, I'd agree with the file manager argument that should easily point out differences.

For a mixed method, also including Everything, see [freefilesync] Problem using the Mirror function.

Thats a easy job for any file manager you may try (not Windows Explorer of course).
Synchronize Directories in Total Commander for example.

[horst.epp]

Ok. so I have downloaded Total Commander and Free. Now what do I do?

How do I use Freefilesync?

If it's a straight 1 directory to 1 directory comparison, I'd agree with the file manager argument that should easily point out differences.

For a mixed method, also including Everything, see [freefilesync] Problem using the Mirror function.

Thats a easy job for any file manager you may try (not Windows Explorer of course).
Synchronize Directories in Total Commander for example.

In TC open the root of your file sets in the 2 panels.
Then execute the command cm_FileSync in the command line on the bottom.
There are ways to start it from the Button bar but I don't know your layout.
See the 4 buttons in the Show section and select what you want to see and hit Compare.

Screenshot - 17.06.2021 , 08_16_31.png (58.97 KiB) Viewed 17617 times

[NotNull]

Be aware that Total Commander's Synchronize Directories is not reliable with 'modern' files. Simple (DOS-like) files should be OK though.

For example: TC doesn't handle files with named datastreams (/ADS) well. Maybe other 'modern' files too, but I stopped testing after that.
(TC 9.50B7 x64 btw)


I don't have an alternative to offer.

[horst.epp]

Be aware that Total Commander's Synchronize Directories is not reliable with 'modern' files. Simple (DOS-like) files should be OK though.

For example: TC doesn't handle files with named datastreams (/ADS) well. Maybe other 'modern' files too, but I stopped testing after that.
(TC 9.50B7 x64 btw)


I don't have an alternative to offer.

What do you mean with "modern" files ?
TC copies files with the official Windows API.
There are no known problems with any sort of files as the content is not relevant.

[NotNull]

It is not about the copying, it's about the comparing.

Try this:
Create a new folder, start PowerShell and go to that folder.
Run the following commands (just select all and paste it in PowerShell).

That will create a abc.txt in folder1 as well as folder2.
Difference being that they have different named datastreams and those datastreams not only have a different name, they even have different content.
TC sync comparison sees both folders as being equal ....

Code: Select all

md .\Folder1, .\Folder2
Set-Content .\folder1\abc.txt -Stream 'Folder1' "This is Folder1 stream"
Set-Content .\folder2\abc.txt -Stream 'Folder2' "Something else"
(gi .\Folder1\abc.txt).LastWriteTime = (gi .\Folder2\abc.txt).LastWriteTime
cat .\Folder1\abc.txt:Folder1
cat .\Folder2\abc.txt:Folder2

[horst.epp]

It is not about the copying, it's about the comparing.

Try this:
Create a new folder, start PowerShell and go to that folder.
Run the following commands (just select all and paste it in PowerShell).

That will create a abc.txt in folder1 as well as folder2.
Difference being that they have different named datastreams and those datastreams not only have a different name, they even have different content.
TC sync comparison sees both folders as being equal ....

Code: Select all

md .\Folder1, .\Folder2
Set-Content .\folder1\abc.txt -Stream 'Folder1' "This is Folder1 stream"
Set-Content .\folder2\abc.txt -Stream 'Folder2' "Something else"
(gi .\Folder1\abc.txt).LastWriteTime = (gi .\Folder2\abc.txt).LastWriteTime
cat .\Folder1\abc.txt:Folder1
cat .\Folder2\abc.txt:Folder2

Zero sized files with the same name and time stamp are of no practical relevance.
Where do you have such files ?
Updating datastreams normaly also updates the time stamps
if not done by an virus.

[NotNull]

This is just a simple, minimal demo to show the effect. It would not be very hard to imagine a more realistic scenario.

The main point here is that TC does not detect this, where it should. It's a shortcoming in it's comparison routine. Enabling ignore date for comparisons would make it even worse.

[Just Visiting]

Thanks everyone.

Unfortunately, you all LOST me completely. This is way beyond my level of computer expertise.

I will add that TC is the easiest to look at even though it uses a very primitive 1994 DOS like interface. Command lines are for Linux and power users, not modern day Windows users who look and need "a turn key solution"...in my humble opinion. I became totally confused with the RED X marks, also shown in the above screen shot.

I also played around with Freesync and each time Freesync kept the duplicate files scattered from everywhere. The interface started to become confusing the more I tried to find a way to delete the duplicates during the process.

With TC, on ONE occasion [ONLY] it deleted the duplicates, but other times it kept them BOTH as well. Fortunately, I had my backup copy and I lost a few hours messing with these applications. TC deleted NON duplicate files.

Clearly, these two applications are not simple and effective enough for someone who wants an automated process to compare, add, and delete files that are the same and different while prioritizing a particular drive or directory.

In other words, if my large directory called Dell has been modified on another PC, but needs to be merged into the same directory onto another PC called ASUS, that's really all I want. Any files which have been deleted, say from, Dell would have more priority over the files that still remain on ASUS. They would also be deleted [automatically] on ASUS as part of the process. At least that's what I need.

From this point on, I think it's best save any new files or documents on the cloud instead of repeating this time process hoping to obtain the results I need.

I will make one last attempt on Reddit to find a suitable solution for me. Perhaps I'll have better luck there. Thanks again for your efforts.

[tuska]

Hi,

In other words, if my large directory called Dell has been modified on another PC,
but needs to be merged into the same directory onto another PC called ASUS, that's really all I want.

Any files which have been deleted, say from, Dell would have more priority over the files that still remain on ASUS.
They would also be deleted [automatically] on ASUS as part of the process. At least that's what I need.

 

2021-06-18_Compare larger folders for migration_10141.png (22.24 KiB) Viewed 17476 times

 

Asymmetric
If this option is checked, it is assumed that a copy of the left side (folder: "Dell") should be created on the right side (folder: Asus).

Simply press the F1 key in the "Synchronize directories" window to call up the help.
Pay close attention to the buttons I have pressed(!)

I became totally confused with the RED X marks, also shown in the above screen shot.

Blue: Copy from right to left, or delete on the right side (asymmetric mode, in this case, the file has a cross symbol x in the middle).

If the folder "Dell" is to be prioritised, then this means that after an asymmetrical synchronisation
all files/subfolders in the "Asus" folder should also be present ("1:1") - just as in the "Dell" folder.
With thousands of files, I think it is impossible that there would be no files to delete in the "Asus" folder...

It is best to test beforehand with 2 test folders and a few files that differ in these folders,
to get clarity about the process.

Regards,
Karl

PS:
If you want to synchronise the two folders "Dell" and "Asus" against each other, then it is necessary to uncheck "Asymmetric".
The following also applies in this case:
It is best to test beforehand with 2 test folders and a few files that differ in these folders.

PPS:
If you have questions or problems regarding synchronisations in Total Commander,
then I would recommend that you post a request in the Total Commander forums,
because this is not the right place...
________________________________________________
Duplicate files

• Search for duplicate files in Total Commander
• Alt+Shift+F7 Search in separate process...
• Click on tab "Advanced"
• Tick "Find duplicate files",...
• Click on button "Start search"
• Click on button "Feed to listbox"

More information is available on request in the TC Forum.

[ChrisGreaves]

I am in the process of slowly migrating one my large folders with thousands of files from my legacy W7 PC to a new laptop. At the moment, I wish to keep both folders on both drives and then merge the folders back and forth as necessary.

Hi Just Visiting. I am a bit late to the party, but thought to add my tuppenorth for latecomers.
I suspect that you did not find a ready solution in Everything, but that after two years your "merge" project is probably completed by some other method.

Two years ago i would have suggested the Windows utility RoboCopy as a solution, and could have suggested a set of parameters.
I use RoboCopy for my nightly and weekly backups, and often enough to determine "what is going on?" in various project folders.
Cheers, Chris

[therube]

As far as FreeFileSync, it has various Synchronization methods.

You do need to understand just what you want, & just what each method entails (so you get the results you want - not unwanted results).


IMO, The Default Sync Method Should Be - Update - because it is safer. (Others disagree .)

[ChrisGreaves]

You do need to understand just what you want, & just what each method entails (so you get the results you want - not unwanted results).

Quite so.
I was raised on each project having a written Goal, broken down into a series (not a collection) of Objectives, each objective with a Measurable Deliverable, so if I were drafting, say, a RoboCopy command line, I would do it according to an agreed and well-defined Goal statement.
Cheers, Chris