Robocopy Not Terminating

Post by David E. Ross
Windows 7 Ultimate SP1 (x64)
When I do my backups, I send the resulting files to an internal drive of
my PC. I then encrypt the files and move them to a removable external
drive that I store remotely.
To move the encrypted files, I use a script that uses Robocopy.exe. A
search tool shows 6 files in my multi-drive system named Robocopy.exe.
The one I am using is located at [C:\Windows\System32\Robocopy.exe],
which has version #5.1.10.1027.
For encrypted files whose sizes are less than 1.5 GB, I have no problem
running the script. For encrypted files whose sizes are in the range 10
GB - 21 GB, however, Robocopy.exe fails to terminate. This means my
script does not complete unless I manually terminate Robocopy.exe via
Windows Task Manager, which then allows my script to continue.
Is there a more recent x64 version of Robocopy than 5.1.10.1027? Is
there something else I could do in my script to facilitate the
termination of Robocopy.exe, possibly in one of its options?

What sort of messages is it leaving behind (I'm assuming you have
a log file)?
It might be just taking forever to get the "paperwork" done, do
you have a /W or /r option set? (Wait & repeat- default for Wait is
thirty second before retrying, default number for Repeat is one
million attempts.)

--
pyotr filipivich
Next month's Panel: Graft - Boon or blessing?

David E. Ross

2017-01-11 06:42:56 UTC

Post by pyotr filipivich

I use the script only weekly, and only one of three encrypted backup
files meets the 10 GB - 21 GB criterion for the problem (a different
backup each week). I will add /W, /r, and logging to the script before
I next use it; but I need advice for what logging options to use.
--
David E. Ross
<http://www.rossde.com/>

When the President of the United States makes a statement of
national importance, I want to see his face as he is talking.
At the least, I want to hear his voice. Presidents should
not be making public statements that are of no importance.

Donald: Stop tweeting. Otherwise, how do we know the message
really comes from you?

David E. Ross

2017-01-11 07:03:15 UTC

Post by pyotr filipivich

Oops! I just looked at the script. It already had /R:10 /W:30. But I
need advice regarding logging.
--
David E. Ross
<http://www.rossde.com/>

When the President of the United States makes a statement of
national importance, I want to see his face as he is talking.
At the least, I want to hear his voice. Presidents should
not be making public statements that are of no importance.

Donald: Stop tweeting. Otherwise, how do we know the message
really comes from you?

pyotr filipivich

2017-01-11 17:34:16 UTC

Post by pyotr filipivich

Oops! I just looked at the script. It already had /R:10 /W:30. But I
need advice regarding logging.

/Tee /NP /Log+:filename.rtf

That will append to Filename.rtf what goes to the screen (/Tee). The
/NP turns off the "Notify of Progress" (the % copied) - which I shut
off because I have line after line of
1%
1.5%
2%
...
99.9%
100%
Which does not help me.
I use rtf files as they import into Wordpad, or you favorite
Wordproceser with a minim of fuss.

--
pyotr filipivich
Next month's Panel: Graft - Boon or blessing?

VanguardLH

2017-01-11 04:49:12 UTC

That's the one you should be using.

Post by David E. Ross
which has version #5.1.10.1027.

https://technet.microsoft.com/en-us/library/ee851678.aspx
"Robocopy (the name is short for Robust File Copy) was introduced with
the Windows Server 2003 Resource Kit and is included in all editions of
Windows 7."

robocopy.exe comes in Windows 7. When I run robocopy.exe, it says
"Microsoft Windows [Version 6.1.7601]" (which is just the NT version of
Windows 7). When I right-click on the robocopy.exe file in the system32
folder, the Details pane shows its version as 6.1.7601.23403. I did not
see a /version or similar argument to have robocopy divulge its own
version rather than report Windows' version.

So from where did you get the version 5 of robocopy.exe?

Post by David E. Ross
For encrypted files whose sizes are less than 1.5 GB, I have no problem
running the script. For encrypted files whose sizes are in the range 10
GB - 21 GB, however, Robocopy.exe fails to terminate. This means my
script does not complete unless I manually terminate Robocopy.exe via
Windows Task Manager, which then allows my script to continue.
Is there a more recent x64 version of Robocopy than 5.1.10.1027? Is
there something else I could do in my script to facilitate the
termination of Robocopy.exe, possibly in one of its options?

Is the destination path in a FAT or NTFS partition? Maximum file size
in a FAT32 partition is 2GB. Unlike compression, encryption will
enlarge the size of a file. So you could start with a file that is
under 2GB but encryption wants to make it larger than 2GB.

Are you sure no other file has a handle on the source file you feed to
robocopy? That is, could another process have a lock on the file so
robocopy keeps waiting for the file to get released? Robocopy does not
support VSC (Volume Shadow Copying). You need to use another tool to
create shadow copies of the inuse files, assign a drive to the shadow
copy folder, and then use robocopy to copy the shadow copy (source) to
your destination. I remember reading about vshadow.exe but haven't
looked at it for a long time. An old article on how to create the
shadow copy that robocopy can then access is describe at
http://rakhesh.com/windows/how-to-backup-open-pst-via-robocopy/.

I remember looking at VSScopy to copy files that may be inuse. As I
recall, I was trying to save a copy of a .pst file (message store) for
MS Outlook which I leave running 24x7. From its description, VSScopy
looks like it can replace robocopy as VSScopy is also a command-line
(console mode) program. See:

http://vsscopy.com/

I'll omit any discussing of encryption possibly enlarged the output file
since the problem is with robocopy and before your script does any
encrypting. I don't recall any args or switchs in robocopy that lets
you have it do the encryption. There is /efsraw but that merely has
robocopy do a raw copy of already EFS-encrypted files; i.e., don't have
robocopy end up descrypting and saving files that are accessed in an EFS
volume/folder. I suspect /efsraw accesses and copies the binary stream
without requiring using the certificate because the stream is not being
decrypted, just copied.

What command-line arguments/parameters are you using with robocopy.exe?
Could be it has to walk through a huge folder hierarchy whether it is
filled with files or contains empty folders. How long did you wait to
determine robocopy was hung? Could be one of those args is causing the
long delay or hang.

What type of external drive are you using? Is it USB attached? If so,
is it one of those "green" drives where its firmware spins it down after
a period of no activity? The backup program (robocopy but I've seen
Acronis TrueImage do this, too) might think the media is unavailable
because it doesn't respond immediately. The time for the drive to
acknowledge a new read or write and the time to spin up might be too
long for the backup program to wait. A better type of USB drive would
eliminate that problem. The external USB HDD that I built using a WDC
black 7200 RPM drive did not have the errors by the backup program
reported when using the WDC-built USB drive (forget its product name).
I think WDC has a tool to alter the default firmware configuration to
override the idle timeout and power down so the green drive keeps
spinning as long as it has power. Between the time the backup program
has to buffer up a huge chuck of the source data (and possible to
encrypt it, too) means the USB drive might power off to save power. I
don't get green drives because of this power cycling but then I'm using
them on desktops or laptops where I'm not concerned about battery drain.
On a desktop, battery drain is not an issue. On a laptop, I don't have
the USB HDD attached unless I'm actually using it. But I'll stay away
from those damn green drives or USB HDDs that incorporate firmware to
power cycle the drive when it is idle "too long".

Char Jackson

2017-01-11 06:05:30 UTC

Is the destination path in a FAT or NTFS partition? Maximum file size
in a FAT32 partition is 2GB.

I've seen that so many times over the years that I'm almost starting to
believe it myself, but the fact is that the file-size limit on FAT32 is
(2^32)-1 bytes, or one byte less than a full 4GB.

I think the confusion creeps in when you consider that the max partition
size for FAT32 is 2TB, so maybe people confuse the partition size with
the file size while keeping the GB unit of measure.

Post by VanguardLH
Unlike compression, encryption will
enlarge the size of a file. So you could start with a file that is
under 2GB but encryption wants to make it larger than 2GB.

Actually, compression can also increase the size of a file. ;-)
Granted, that's not usually the case, and it's far from the desired
outcome, but it's possible if the source file is already heavily
compressed.

--
Char Jackson

VanguardLH

2017-01-11 06:56:29 UTC

Post by Char Jackson

Is the destination path in a FAT or NTFS partition? Maximum file
size in a FAT32 partition is 2GB.

I've seen that so many times over the years that I'm almost starting
to believe it myself, but the fact is that the file-size limit on
FAT32 is (2^32)-1 bytes, or one byte less than a full 4GB.

Oops, you're correct.

The OP says the problem crops up with files in the 10-21 GB range. He
doesn't mention trying anything between 1.5GB to 10GB.

David E. Ross

2017-01-11 07:18:10 UTC

That's the one you should be using.

Post by David E. Ross
which has version #5.1.10.1027.

https://technet.microsoft.com/en-us/library/ee851678.aspx
"Robocopy (the name is short for Robust File Copy) was introduced with
the Windows Server 2003 Resource Kit and is included in all editions of
Windows 7."
robocopy.exe comes in Windows 7. When I run robocopy.exe, it says
"Microsoft Windows [Version 6.1.7601]" (which is just the NT version of
Windows 7). When I right-click on the robocopy.exe file in the system32
folder, the Details pane shows its version as 6.1.7601.23403. I did not
see a /version or similar argument to have robocopy divulge its own
version rather than report Windows' version.
So from where did you get the version 5 of robocopy.exe?

It is what I found when I did a local-drive search for "robocopy.exe".
Actually, I found several instances of "robocopy.exe"; but this had the
latest date. Strangely, ALL instances have the verion #5.1.10.1027
although they have various dates and various sizes.

I will have to answer this tomorrow since I already moved the external
drive to its remote storage.

Post by VanguardLH
Are you sure no other file has a handle on the source file you feed to
robocopy? That is, could another process have a lock on the file so
robocopy keeps waiting for the file to get released? Robocopy does not
support VSC (Volume Shadow Copying). You need to use another tool to
create shadow copies of the inuse files, assign a drive to the shadow
copy folder, and then use robocopy to copy the shadow copy (source) to
your destination. I remember reading about vshadow.exe but haven't
looked at it for a long time. An old article on how to create the
shadow copy that robocopy can then access is describe at
http://rakhesh.com/windows/how-to-backup-open-pst-via-robocopy/.

Both the successful terminations of Robocopy.exe and the unsuccessful
terminations are for source files that have been generated by PGP
10.1.2. No other application touches those source files. No
application other than Robocopy.exe touches the destination files.

Post by VanguardLH
I remember looking at VSScopy to copy files that may be inuse. As I
recall, I was trying to save a copy of a .pst file (message store) for
MS Outlook which I leave running 24x7. From its description, VSScopy
looks like it can replace robocopy as VSScopy is also a command-line
http://vsscopy.com/
I'll omit any discussing of encryption possibly enlarged the output file
since the problem is with robocopy and before your script does any
encrypting. I don't recall any args or switchs in robocopy that lets
you have it do the encryption. There is /efsraw but that merely has
robocopy do a raw copy of already EFS-encrypted files; i.e., don't have
robocopy end up descrypting and saving files that are accessed in an EFS
volume/folder. I suspect /efsraw accesses and copies the binary stream
without requiring using the certificate because the stream is not being
decrypted, just copied.

Actually, I run the script AFTER encrypting. I encrypt all the backup
files before first running the script. The encrypted backup files are
slightly smaller than their unencrypted source backup files.
Apparently, Acronis True Image (my backup application) does significant
compression so that the added compression by PGP is minimal.

Post by VanguardLH
What command-line arguments/parameters are you using with robocopy.exe?
Could be it has to walk through a huge folder hierarchy whether it is
filled with files or contains empty folders. How long did you wait to
determine robocopy was hung? Could be one of those args is causing the
long delay or hang.

The arguments are:
/COPY:DT /V /NP /XF *.tib /R:10 /W:30
Thhe /XF *.tib is to prevent the copying of .tib files since they are
not encrypted. Only a file with the extension .tib.pgp is copied. The
failure case has a source folder on an NTFS drive with five large files;
four have the extension .tib, and one has .tib.pgp. The destination
folder has three large files; all with the extension .tib.pgp.

I waited about two minutes. Windows Task Manager indicated only minimal
activity (0%-2% of CPU usage) for Robocopy.exe.

Post by VanguardLH
What type of external drive are you using? Is it USB attached? If so,
is it one of those "green" drives where its firmware spins it down after
a period of no activity? The backup program (robocopy but I've seen
Acronis TrueImage do this, too) might think the media is unavailable
because it doesn't respond immediately. The time for the drive to
acknowledge a new read or write and the time to spin up might be too
long for the backup program to wait. A better type of USB drive would
eliminate that problem. The external USB HDD that I built using a WDC
black 7200 RPM drive did not have the errors by the backup program
reported when using the WDC-built USB drive (forget its product name).
I think WDC has a tool to alter the default firmware configuration to
override the idle timeout and power down so the green drive keeps
spinning as long as it has power. Between the time the backup program
has to buffer up a huge chuck of the source data (and possible to
encrypt it, too) means the USB drive might power off to save power. I
don't get green drives because of this power cycling but then I'm using
them on desktops or laptops where I'm not concerned about battery drain.
On a desktop, battery drain is not an issue. On a laptop, I don't have
the USB HDD attached unless I'm actually using it. But I'll stay away
from those damn green drives or USB HDDs that incorporate firmware to
power cycle the drive when it is idle "too long".

The external drive is USB2 Western Digital Passport. As I indicated,
further information about this drive will have to wait until I retrieve
it late tomorrow. I do know that the file is indeed copied; the copying
appears successful. Next week, when I again do backups, I will get a
SHA512 checksum on the source and destination to verify success.
--
David E. Ross
<http://www.rossde.com/>

When the President of the United States makes a statement of
national importance, I want to see his face as he is talking.
At the least, I want to hear his voice. Presidents should
not be making public statements that are of no importance.

Donald: Stop tweeting. Otherwise, how do we know the message
really comes from you?

pyotr filipivich

2017-01-11 17:34:16 UTC

That's the one you should be using.

Post by David E. Ross
which has version #5.1.10.1027.

https://technet.microsoft.com/en-us/library/ee851678.aspx
"Robocopy (the name is short for Robust File Copy) was introduced with
the Windows Server 2003 Resource Kit and is included in all editions of
Windows 7."
robocopy.exe comes in Windows 7. When I run robocopy.exe, it says
"Microsoft Windows [Version 6.1.7601]" (which is just the NT version of
Windows 7). When I right-click on the robocopy.exe file in the system32
folder, the Details pane shows its version as 6.1.7601.23403. I did not
see a /version or similar argument to have robocopy divulge its own
version rather than report Windows' version.
So from where did you get the version 5 of robocopy.exe?

Is Windows - a good first approximation of a working OS. (Snark
off, I don't like what has been developed at the interface, I no
longer really care what's happening "under the hood")

--
pyotr filipivich
Next month's Panel: Graft - Boon or blessing?

VanguardLH

2017-01-11 19:34:34 UTC

Since Acronis TrueImage (ATI) can optionally encrypt its outputted
backup file, why add the extra step of using PGP after using ATI?

http://www.acronis.com/en-us/support/documentation/ATIH2012/#7937.html
#7 refers to "Disk backup options" described at:
http://www.acronis.com/en-us/support/documentation/ATIH2012/#7940.html

I don't see there an option for encryption. However, there is a link to
"Backup protection" at:

http://www.acronis.com/en-us/support/documentation/ATIH2012/index.html#3407.html

The above are help web pages for ATI 2012. I did find:

http://www.acronis.com/en-us/support/documentation/ATIH2011/index.html#3407.html

for ATI 2011 that does have a "Backup protection" setting in the
Advanced tab for "Disk backup options". From what I read in Acronis
forums, "Backup protection is only available when you set up a new
backup task. As soon as you have done a backup, the backup encryption
option is not showing up." That is because once you set a password
string (used to encrypt), even a blank one, the password cannot be
changed later in the backup job. You have to enable encryption when you
first define a backup job.

In addition, if you set the backup location to Acronis Secure Zone (ASZ)
when defining a backup job, encryption will not be available. However,
for you to copy it elsewhere means that you would need to mount the
image through ATI. The ASZ is not only hidden (no drive letter assigned
to the partition) but also uses a non-standard partition type in the
partition record in the partition table back in the MBR or UEFI. Hiding
it prevents malware and malicious/ignorant users from destroying your
backups and using a non-standard partition type prevents most disk tools
from monkeying with that partition. I wish they had added disabling the
device (I can do it using devcon.exe - a command-line equivalent to
Device Manager) and also setting a Windows policy to prevent writing to
that volume. Paragon's Backup & Restore has its similar Backup Capsule
which works the same way (programmers migrated between Acronis and
Paragon and why the similar protection scheme).

So when first defining a backup job and as long at the backup location
is not ASZ, the encryption option is available. Then you can encrypt
while you backup instead of having to perform encryption as a separate
and follow up task.

Back to the hang issue, I'm wondering if there is still an open handle
on the backup file. A write protected (locked) file would prevent
robocopy from opening the file. robocopy should immediately fail with
an access error on the file. However, if the file is not locked but
still open (another process has a handle still open on the file), maybe
robocopy is still waiting for the file to close before it can finialize
its read of the same file. You could use something like Unlocker or
SysInternals' handle.exe to see if there is still a handle on the file
that you are trying to process in the subsequent PGP run.

Are you using a batch (script) file to run the ATI job and then follow
with PGP and then follow with robocopy? If so, are you checking the
return status of the prior command to ensure it completed okay? If you
list commands one line at a time in a .bat file, they get executed in
sequence regardless of what happened with the prior program. You could
use IF ERRORLEVEL in the batch file to check status is zero from the
prior command. You actually have to test if ERRORLEVEL is 1, or
greater, and if so then use goto to error code that bypasses subsequent
commands. IF ERRORLEVEL returns true on any value equal to *or greater*
so testing for zero is tough. Instead use IF %ERRORLEVEL% LSS 1 to be
sure zero got returned. Alternatively, you can use conditional chaining
within a command line. & is used to chain commands within one command
line. && is used to conditionally chain commands within the same
command line: the next command doesn't run unless the return status from
the prior command is zero (okay). cmd1 & cmd2 will run cmd2 whether
cmd1 ran okay or not. cmd1 && cmd2 won't run cmd2 unless cmd1 completed
okay.

In fact, a program may load and still be running but the .bat file
continues running subsequent commands that are probably inapplicable at
that time. That is, program 1 gets loaded but immediately returns
status despite it is still running so program 2 runs; however, program 1
has not yet completed its task. program 1 loaded it GUI or shell and
returns status but it is still working and will exit sometime later.
You need to put the shell on hold in which you run the batch file so
that program 2 has to wait until program 1 has actually *exited*, not
just that it loaded okay. You need to ensure the ATI backup is not
still running, has actually ended and the ATI program exited, before you
run the following PGP command. To make the shell wait until program 1
exits, use the start command, like:

start "Acronis job" ...

I always add the title arg (within double quotes). If you don't specify
the title arg, any other arg using double quotes, like to delimit a path
(/D path) or command/program file (including a path to it) because it
has spaces will result in start parsing the arg as the title. So I add
"title" so in case any other arg has double quotes then that arg won't
get confused as the title. I add "title" and it is the first arg so it
is sure to get parsed as the title arg.

I haven't bothered to run ATI from a command line but I suspect it does
support a CLI (command-line interface) to use it that way. I don't know
if its CLI will pend the shell execution (of the batch file) until the
command to ATI has completed (backup done *and* ATI unloads). See if
using start makes sure ATI has loaded AND ran AND exited before the next
command in the batch file gets ran.

Or are you using the post-command feature in ATI to run a program or
batch file after ATI has supposedly completed its backup job? That
means ATI has not exited when the post-command gets run. I remember
having to do something goofy in my batch files when called as
post-command by an ATI backup job but been too long since I last used
ATI. The batch file where I used diskpart to remove the drive letter
and devcon to disable the device wouldn't run correctly when called by
ATI as a post-command after a backup job. Don't remember what I had to
modify to get it to work okay as a post-command in ATI.

David E. Ross

2017-01-12 03:01:35 UTC

Post by VanguardLH
Since Acronis TrueImage (ATI) can optionally encrypt its outputted
backup file, why add the extra step of using PGP after using ATI?

I trust the OpenPGP concept and thus the PGP application. As far as I
know, it has never been broken.
--
David E. Ross
<http://www.rossde.com/>

When the President of the United States makes a statement of
national importance, I want to see his face as he is talking.
At the least, I want to hear his voice. Presidents should
not be making public statements that are of no importance.

Donald: Stop tweeting. Otherwise, how do we know the message
really comes from you?

Zaidy036

2017-01-13 01:47:05 UTC

Post by David E. Ross
/COPY:DT /V /NP /XF *.tib /R:10 /W:30
Thhe /XF *.tib is to prevent the copying of .tib files since they are
not encrypted. Only a file with the extension .tib.pgp is copied. The
failure case has a source folder on an NTFS drive with five large files;
four have the extension .tib, and one has .tib.pgp. The destination
folder has three large files; all with the extension .tib.pgp.

Why use a positive command with source as "[folder]\*.tib.pgp" ?

If everything is local why not use /R:0 /W:0 ?

Specify a log file and add /V for test to run RoboCopy but not do the copy.

Open a window to the destination and watch if the copied file is
increasing in size which indicates it is slow writing to HDD caused by
slow transmission line, cache, or HDD speed.

David E. Ross

2017-01-12 02:59:46 UTC

Post by VanguardLH
Is the destination path in a FAT or NTFS partition? Maximum file size
in a FAT32 partition is 2GB. Unlike compression, encryption will
enlarge the size of a file. So you could start with a file that is
under 2GB but encryption wants to make it larger than 2GB.

The removable external hard drive is NTFS. It has an overall capacity
of 465 GB of which 241 are unused.
--
David E. Ross
<http://www.rossde.com/>

When the President of the United States makes a statement of
national importance, I want to see his face as he is talking.
At the least, I want to hear his voice. Presidents should
not be making public statements that are of no importance.

Donald: Stop tweeting. Otherwise, how do we know the message
really comes from you?

Paul

2017-01-11 05:41:02 UTC

This caught my eye.

https://en.wikipedia.org/wiki/Robocopy

"Robocopy does not copy open files.

Any process may open files for exclusive read access
by withholding the FILE_SHARE_READ flag during opening.

Even Robocopy's Backup mode will not touch those files.
"

You could try experimenting with "Shadowspawn". which is
in the reference section of that article.

https://github.com/candera/shadowspawn

HTH,
Paul

David E. Ross

2017-01-11 07:20:03 UTC

This caught my eye.
https://en.wikipedia.org/wiki/Robocopy
"Robocopy does not copy open files.
Any process may open files for exclusive read access
by withholding the FILE_SHARE_READ flag during opening.
Even Robocopy's Backup mode will not touch those files.
"
You could try experimenting with "Shadowspawn". which is
in the reference section of that article.
https://github.com/candera/shadowspawn
HTH,
Paul

But the 20 GB file is copied. After copying it, however, Robocopy.exe
does not terminate.
--
David E. Ross
<http://www.rossde.com/>

When the President of the United States makes a statement of
national importance, I want to see his face as he is talking.
At the least, I want to hear his voice. Presidents should
not be making public statements that are of no importance.

Donald: Stop tweeting. Otherwise, how do we know the message
really comes from you?

Paul

2017-01-11 08:29:44 UTC

Post by David E. Ross
But the 20 GB file is copied. After copying it, however, Robocopy.exe
does not terminate.

Watch the transaction with Process Monitor ?

https://technet.microsoft.com/en-us/sysinternals/processmonitor

I've actually recorded an entire Macrium backup with ProcMon (20 minutes),
and the log was 9GB in size. That's the 64-bit version of ProcMon
(it launches the right one for you). If I did that on a 32-bit
system, I would have run out of resources.

The purpose of doing that, is to see if Robocopy is actually
checking a resource in a repetitive fashion or not. So you can
figure out what it's stuck on. It's also possible for a program
to "deadlock" if it's threaded, which might be harder to diagnose
and figure out.

You can try bringing over a more recent Robocopy, but I don't
know if they would check versioning for a thing like that or not.
The Solitaire game is sensitive to OS (unless you patch it
with a hex editor).

*******

Another thing you could try, as a debugging step, would
be to complete your encryption operation, then reboot.
Then start the Robocopy phase. The purpose of this, is to
make sure any "handles" from the previous steps of your
conversion, are laid to rest.

Paul

Stan Brown

2017-01-12 23:19:54 UTC

Post by Paul
This caught my eye.
https://en.wikipedia.org/wiki/Robocopy
"Robocopy does not copy open files.
Any process may open files for exclusive read access
by withholding the FILE_SHARE_READ flag during opening.
Even Robocopy's Backup mode will not touch those files.
"

But the 20 GB file is copied. After copying it, however, Robocopy.exe
does not terminate.

Caching?

I copy 2 GB files over a USB2 cable to my external hard drive, via
click-and-drag, and the progress window stays at "5 seconds
remaining" for about 15 seconds. That's because the main copy
operation thinks it's finished, but Windows background caching is
still doing the write operations.

Unless Robocopy avoids Windows caching (and I don't see how it
could), maybe that's the explanation.

--
Stan Brown, Oak Road Systems, Tompkins County, New York, USA
http://BrownMath.com/
http://OakRoadSystems.com/
Shikata ga nai...

David E. Ross

2017-01-12 23:59:04 UTC

Post by Stan Brown

But the 20 GB file is copied. After copying it, however, Robocopy.exe
does not terminate.

Caching?
I copy 2 GB files over a USB2 cable to my external hard drive, via
click-and-drag, and the progress window stays at "5 seconds
remaining" for about 15 seconds. That's because the main copy
operation thinks it's finished, but Windows background caching is
still doing the write operations.
Unless Robocopy avoids Windows caching (and I don't see how it
could), maybe that's the explanation.

I will check the properties of the external drive. If caching is
enabled, I will disable it.

I will not be doing the backups and copying until sometime next week.
--
David E. Ross
<http://www.rossde.com/>

When the President of the United States makes a statement of
national importance, I want to see his face as he is talking.
At the least, I want to hear his voice. Presidents should
not be making public statements that are of no importance.

Donald: Stop tweeting. Otherwise, how do we know the message
really comes from you?

Big Al

2017-01-11 21:29:27 UTC

THis may be true but I've done backups of open files and I get notices
on the screen that the file cannot be copy (or is in use) and robocopy
is waiting 30 seconds. And this is with no switches to enable this.
So yes, it won't copy but it also won't skip without user intervention.

Post by Paul
You could try experimenting with "Shadowspawn". which is
in the reference section of that article.
https://github.com/candera/shadowspawn
HTH,
Paul

Nil

2017-01-11 22:58:27 UTC

Post by Big Al
THis may be true but I've done backups of open files and I get
notices on the screen that the file cannot be copy (or is in use)
and robocopy is waiting 30 seconds. And this is with no switches
to enable this. So yes, it won't copy but it also won't skip
without user intervention.

Yes, it will... after the default 1,000,000 attempts. You can (and
should, IMO) reduce the retry number to something more reasonable with
a command line switch or setting a new default (also a command like
argument.)

Mr. Man-wai Chang

2017-01-11 12:39:52 UTC

To terminate a running process in Command Prompt, the process must
output something to the console! When you terminate the screen output in
a Command Prompt, it then terminates the process that outputs the text
to the screen.

--
@~@ Remain silent! Drink, Blink, Stretch! Live long and prosper!!
/ v \ Simplicity is Beauty!
/( _ )\ May the Force and farces be with you!
^ ^ (x86_64 Ubuntu 9.10) Linux 2.6.39.3
不借貸! 不詐騙! 不援交! 不打交! 不打劫! 不自殺! 請考慮綜援 (CSSA):
http://www.swd.gov.hk/tc/index/site_pubsvc/page_socsecu/sub_addressesa

pyotr filipivich

2017-01-11 17:34:16 UTC

I've not had problems with that, but then, I'm not copying 10 gig
files.

--
pyotr filipivich
Next month's Panel: Graft - Boon or blessing?

David E. Ross

2017-01-12 06:37:07 UTC

For files of size 1.5 GB or less, Robocopy.exe terminates, allowing the
script to continue with additional commands. For files of size 10 GB or
more, Robocopy.exe does NOT terminate, preventing the script to
continue. The same script either has the problem (file larger than 10
GB) or does not have the problem (file less than 1.5 GB).

I am not sure where between 1.5 GB and 10 GB the problem arises.
--
David E. Ross
<http://www.rossde.com/>

When the President of the United States makes a statement of
national importance, I want to see his face as he is talking.
At the least, I want to hear his voice. Presidents should
not be making public statements that are of no importance.

Donald: Stop tweeting. Otherwise, how do we know the message
really comes from you?

Paul

2017-01-12 10:11:46 UTC

I just tested your version of Robocopy, on a folder with
two 18.6GB files, transferred to a second disk drive. And
both copied no problem at all.

*******

I tested gpg4win 2.3.3 and Kleopatra. Kleopatra is grabby.
It's cobbled together, spawns gpg2 to do encryption. If you
click "cancel", stuff keeps running in the background.
Stuff you can see with Task Manager.

I would suggest, if that *is* what you're feeding this process
with, when the encryption stage is finished, you should manually
use Task Manager to kill all the sub-threads. So *nothing* can have
an attachment to the file you're trying to copy.

Kleopatra is too slow for me to wait until it has
finished encrypting one of my 18.6GB files. And attempts
to override asymmetric RSA2048 with symmetric AES128 aren't
working (and why should they, when GPG2 is launched with
a perfectly good set of command line parameters). So I cannot
let this test case continue, unless it's going to finish
sometime soon.

Paul

Mr. Man-wai Chang

2017-01-20 15:05:45 UTC

Post by Paul
I just tested your version of Robocopy, on a folder with
two 18.6GB files, transferred to a second disk drive. And
both copied no problem at all.

Is the Robocopy.exe in 64-bit Windows also a 64-bit program?

Paul

2017-01-21 05:38:35 UTC

Post by Paul
I just tested your version of Robocopy, on a folder with
two 18.6GB files, transferred to a second disk drive. And
both copied no problem at all.

Is the Robocopy.exe in 64-bit Windows also a 64-bit program?

When the NTFS file system was invented, it *already* needed
to support a 64 bit file length field, even in 32 bit
executables. It's been doing that for at least 17 years.
We've been able to copy >4GB files in NTFS, since NTFS
came out! It doesn't matter whether the program is
32 bit or 64 bit, since we were able to write 32 bit
programs in the year 2000 that could copy large files.
The 32-bit nature of the program would be a red herring.

Since I already tested the exact version of Robocopy
as the OP is using, and transferred a 25GB file with it,
that proves that the program can handle something which
is >4GB.

In the old days (FAT32 era), you might have written
programs with stat(). From the year NTFS was invented,
it would have been wise to start using stat64(), which
exists even in 32-bit programs. Then, you can properly
handle files with sizes >4GB.

Paul

Mr. Man-wai Chang

2017-01-20 15:04:41 UTC

Post by David E. Ross
For files of size 1.5 GB or less, Robocopy.exe terminates, allowing the
script to continue with additional commands. For files of size 10 GB or
more, Robocopy.exe does NOT terminate, preventing the script to
continue. The same script either has the problem (file larger than 10
GB) or does not have the problem (file less than 1.5 GB).
I am not sure where between 1.5 GB and 10 GB the problem arises.

Robocopy is a 32-bit program.

I think the maximum size of file should be limited to 2 GB?

Need to port and compile the program using 64-bit C/C++ compilers?

For the meantime, use a 64-bit 7-zip to compress and split the 10GB
files into multiple 1.5GB files?

Zaidy036

2017-01-20 17:09:16 UTC

Robocopy is a 32-bit program.
I think the maximum size of file should be limited to 2 GB?
Need to port and compile the program using 64-bit C/C++ compilers?
For the meantime, use a 64-bit 7-zip to compress and split the 10GB
files into multiple 1.5GB files?

Win 7 Home 64 bit

Robocopy 6.1.7601.23403

I have a batch using Robocopy from NAS over CAT 5 to PC to USB3 1 TB HDD
to backup an Acronis Images files (tib) 500GB (full) and 30 to 50 GB
(incremental) without problems except the time it takes to finish
writing the larger file.

Here are my central batch cmds:
SET "ipNAS=\\xxx.xxx.x.xx"
SET "_Set0=%ipNAS%\Image\i7\Set0"
SET _LOG=/LOG:F:\BackupSet0.txt
SET _ACT=/PURGE /R:0 /W:0 /NDL /NFL %_LOG%
START "" /WAIT Robocopy %_Set0% K:\Images_Set0 %_ACT%

I suggest monitoring the file size being written to view what is
happening. This can be done using a batch loop with SET _Size=%%~zI

David E. Ross

2017-01-21 05:55:37 UTC

Robocopy is a 32-bit program.
I think the maximum size of file should be limited to 2 GB?
Need to port and compile the program using 64-bit C/C++ compilers?
For the meantime, use a 64-bit 7-zip to compress and split the 10GB
files into multiple 1.5GB files?

Win 7 Home 64 bit
Robocopy 6.1.7601.23403
I have a batch using Robocopy from NAS over CAT 5 to PC to USB3 1 TB HDD
to backup an Acronis Images files (tib) 500GB (full) and 30 to 50 GB
(incremental) without problems except the time it takes to finish
writing the larger file.
SET "ipNAS=\\xxx.xxx.x.xx"
SET "_Set0=%ipNAS%\Image\i7\Set0"
SET _LOG=/LOG:F:\BackupSet0.txt
SET _ACT=/PURGE /R:0 /W:0 /NDL /NFL %_LOG%
START "" /WAIT Robocopy %_Set0% K:\Images_Set0 %_ACT%
I suggest monitoring the file size being written to view what is
happening. This can be done using a batch loop with SET _Size=%%~zI

Where did you get Robocopy 6.1.7601.23403?
--
David E. Ross
<http://www.rossde.com/>

When the President of the United States makes a statement of
national importance, I want to see his face as he is talking.
At the least, I want to hear his voice.

Donald: Stop tweeting. Otherwise, how do we know the message
really comes from you?

Zaidy036

2017-01-21 15:46:09 UTC

Robocopy is a 32-bit program.
I think the maximum size of file should be limited to 2 GB?
Need to port and compile the program using 64-bit C/C++ compilers?
For the meantime, use a 64-bit 7-zip to compress and split the 10GB
files into multiple 1.5GB files?

Win 7 Home 64 bit
Robocopy 6.1.7601.23403
I have a batch using Robocopy from NAS over CAT 5 to PC to USB3 1 TB HDD
to backup an Acronis Images files (tib) 500GB (full) and 30 to 50 GB
(incremental) without problems except the time it takes to finish
writing the larger file.
SET "ipNAS=\\xxx.xxx.x.xx"
SET "_Set0=%ipNAS%\Image\i7\Set0"
SET _LOG=/LOG:F:\BackupSet0.txt
SET _ACT=/PURGE /R:0 /W:0 /NDL /NFL %_LOG%
START "" /WAIT Robocopy %_Set0% K:\Images_Set0 %_ACT%
I suggest monitoring the file size being written to view what is
happening. This can be done using a batch loop with SET _Size=%%~zI

Where did you get Robocopy 6.1.7601.23403?

"C:\Windows\System32\Robocopy.exe" Properties

David E. Ross

2017-01-21 16:01:23 UTC

Post by Zaidy036
Win 7 Home 64 bit
Robocopy 6.1.7601.23403
I have a batch using Robocopy from NAS over CAT 5 to PC to USB3 1 TB HDD
to backup an Acronis Images files (tib) 500GB (full) and 30 to 50 GB
(incremental) without problems except the time it takes to finish
writing the larger file.
SET "ipNAS=\\xxx.xxx.x.xx"
SET "_Set0=%ipNAS%\Image\i7\Set0"
SET _LOG=/LOG:F:\BackupSet0.txt
SET _ACT=/PURGE /R:0 /W:0 /NDL /NFL %_LOG%
START "" /WAIT Robocopy %_Set0% K:\Images_Set0 %_ACT%
I suggest monitoring the file size being written to view what is
happening. This can be done using a batch loop with SET _Size=%%~zI

Where did you get Robocopy 6.1.7601.23403?

"C:\Windows\System32\Robocopy.exe" Properties

At that location, I have Robocopy.exe 5.1.10.1027, last modified 20
November 2010.
--
David E. Ross
<http://www.rossde.com/>

When the President of the United States makes a statement of
national importance, I want to see his face as he is talking.
At the least, I want to hear his voice.

Donald: Stop tweeting. Otherwise, how do we know the message
really comes from you?

Zaidy036

2017-01-21 16:30:11 UTC

Robocopy is a 32-bit program.
I think the maximum size of file should be limited to 2 GB?
Need to port and compile the program using 64-bit C/C++ compilers?
For the meantime, use a 64-bit 7-zip to compress and split the 10GB
files into multiple 1.5GB files?

Win 7 Home 64 bit
Robocopy 6.1.7601.23403
I have a batch using Robocopy from NAS over CAT 5 to PC to USB3 1 TB HDD
to backup an Acronis Images files (tib) 500GB (full) and 30 to 50 GB
(incremental) without problems except the time it takes to finish
writing the larger file.
SET "ipNAS=\\xxx.xxx.x.xx"
SET "_Set0=%ipNAS%\Image\i7\Set0"
SET _LOG=/LOG:F:\BackupSet0.txt
SET _ACT=/PURGE /R:0 /W:0 /NDL /NFL %_LOG%
START "" /WAIT Robocopy %_Set0% K:\Images_Set0 %_ACT%
I suggest monitoring the file size being written to view what is
happening. This can be done using a batch loop with SET _Size=%%~zI

Where did you get Robocopy 6.1.7601.23403?

"C:\Windows\System32\Robocopy.exe" Properties

Just did an Everything search for RoboCopy.exe "Executable:"

5.1.10.1027
"C:\Windows\winsxs\amd64_microsoft-windows-robocopy_31bf3856ad364e35_6.1.7601.17514_none_252d34f00303c6fa\Robocopy.exe"

6.1.7601.23403
"C:\Windows\winsxs\amd64_microsoft-windows-robocopy_31bf3856ad364e35_6.1.7601.23403_none_25c070011c1a6a70\Robocopy.exe"

5.1.10.1027
"C:\Windows\winsxs\x86_microsoft-windows-robocopy_31bf3856ad364e35_6.1.7601.17514_none_c90e996c4aa655c4\Robocopy.exe"

6.1.7601.23403
"C:\Windows\winsxs\x86_microsoft-windows-robocopy_31bf3856ad364e35_6.1.7601.23403_none_c9a1d47d63bcf93a\Robocopy.exe"

Two are 5.1.10.1027 and two 6.1.7601.23403 **BUT** why does the list
**NOT** include "C:\Windows\System32\Robocopy.exe" ???

Nothing in the RoboCopy Log indicates which version was being used but
since CMD comes up using C:\Windows\System32 I expect
"C:\Windows\System32\Robocopy.exe".

Paul

2017-01-21 17:57:45 UTC

Robocopy is a 32-bit program.
I think the maximum size of file should be limited to 2 GB?
Need to port and compile the program using 64-bit C/C++ compilers?
For the meantime, use a 64-bit 7-zip to compress and split the 10GB
files into multiple 1.5GB files?

Win 7 Home 64 bit
Robocopy 6.1.7601.23403
I have a batch using Robocopy from NAS over CAT 5 to PC to USB3 1 TB HDD
to backup an Acronis Images files (tib) 500GB (full) and 30 to 50 GB
(incremental) without problems except the time it takes to finish
writing the larger file.
SET "ipNAS=\\xxx.xxx.x.xx"
SET "_Set0=%ipNAS%\Image\i7\Set0"
SET _LOG=/LOG:F:\BackupSet0.txt
SET _ACT=/PURGE /R:0 /W:0 /NDL /NFL %_LOG%
START "" /WAIT Robocopy %_Set0% K:\Images_Set0 %_ACT%
I suggest monitoring the file size being written to view what is
happening. This can be done using a batch loop with SET _Size=%%~zI

Where did you get Robocopy 6.1.7601.23403?

"C:\Windows\System32\Robocopy.exe" Properties

Just did an Everything search for RoboCopy.exe "Executable:"
5.1.10.1027
"C:\Windows\winsxs\amd64_microsoft-windows-robocopy_31bf3856ad364e35_6.1.7601.17514_none_252d34f00303c6fa\Robocopy.exe"
6.1.7601.23403
"C:\Windows\winsxs\amd64_microsoft-windows-robocopy_31bf3856ad364e35_6.1.7601.23403_none_25c070011c1a6a70\Robocopy.exe"
5.1.10.1027
"C:\Windows\winsxs\x86_microsoft-windows-robocopy_31bf3856ad364e35_6.1.7601.17514_none_c90e996c4aa655c4\Robocopy.exe"
6.1.7601.23403
"C:\Windows\winsxs\x86_microsoft-windows-robocopy_31bf3856ad364e35_6.1.7601.23403_none_c9a1d47d63bcf93a\Robocopy.exe"
Two are 5.1.10.1027 and two 6.1.7601.23403 **BUT** why does the list
**NOT** include "C:\Windows\System32\Robocopy.exe" ???
Nothing in the RoboCopy Log indicates which version was being used but
since CMD comes up using C:\Windows\System32 I expect
"C:\Windows\System32\Robocopy.exe".

C:\Windows\System32\Robocopy.exe is a hardlinked object.

A filenum (entry in the $MFT) can have multiple file pointers.
Each one appears in the file system as if it was a separate file,
but they're all handled under the sane filenum entry.

The "nfi.exe" utility can show when a file has two or
more $FILE_NAME entries. But the utility won't print out the
contents of the hard links.

File 25918
\Windows\System32\Robocopy.exe
$STANDARD_INFORMATION (resident)
$FILE_NAME (resident) <--- the WinSXS one ???
$FILE_NAME (resident) <--- the System32 one ???
$DATA (nonresident)
logical sectors 20151712-20151967 (0x1337da0-0x1337e9f)

It turns out that Everything.exe just reads the first one, and
not any of the others.

https://www.voidtools.com/forum/viewtopic.php?t=459

"Only the first hardlink of a file will be indexed and monitored.
Files that are not the first hardlink will not be indexed or monitored.

This is a limitation of the USN Change Journal.

I have plans to index all hard links in the future.
However, you will have to update the indexes manually
as the USN Change Journal does not support hardlinks."

Now, if you run the nfi.exe command (a utility from Win2K days),
it lists all the winsxs entries first. Because, naturally, first
a file is stored in winsxs (as part of Component Based Servicing
or CBS), and afterwards the file is hardlinked into C:\Windows\System32.

So that gives a kind of answer, as to why you can see all
the WinSXS entries, but the actual entry that counts, the
one we use, because it comes second in the $MFT table used
to store the pointer, Everything.exe cannot list it.

*******

Something else I just learned, is more disturbing. I saw a
thread very recently, where someone was claiming the release
control system was busted on Robocopy. Now, I didn't pay this
any mind, because I was concentrating on something else.

However, now I realize the gravity of what that person was saying.

Try this:

In an Administrator Command Prompt window...

fsutil hardlink list C:\Windows\System32\robocopy.exe

Mine comes back with this.

\Windows\System32\Robocopy.exe

\Windows\winsxs\amd64_microsoft-windows-robocopy_31bf3856ad364e35_6.1.7601.17514_none_252d34f00303c6fa\Robocopy.exe

The fricken *folder* in WinSXS has a release number
with the correct format for Win7SP1. A value of 7600
is the original Windows. And 7601 is SP1. "6.1.7601.17514"

However, if you do Properties on the file,
*all* the files use the (dishonest) 5.1.10.1027 identifier.

So for this file, the *only* way to get the release
information, is using the fsutil command!!! All the
files being hardlinked into the system32 folder,
will have the same 5.1.10.1027 number each time.

So if you were to complain to Microsoft

"My 5.1.10.1027 robocopy.exe is broken..."

the staff at Microsoft would laugh at you. Because
the actual number (6.1.7601.17514) is only available
by an indirect fsutil method.

So even if David had followed my instructions to
install the Convenience Rollup, visually, doing Properties
on the file would show 5.1.10.1027, and he'd tell
me I was full of shit.

*******

So *now* I understand, why what looked like a crusty copy
of Robocopy on my system, wasn't actually that crusty.
And we won't know until David runs the fsutil command,
what he is actually using.

How does this stuff happen ?

Simple.

Robocopy started life as an article on a webpage called
"Utility Spotlight". That means, it was the private
creation of a Microsoft employee and not part of the OS.

Later, Microsoft decided to bundle the file with the OS.

However, this release number snafu, indicates the stubborn
people at Microsoft, *will not* support this program
as if it is part of the OS. By *not* stamping robocopy,exe
properties with "6.1.7601.17514", they're saying it is
not part of the OS. And that's why the release number
involved here is "broken". That's a subtle hint - if you
ask for help, they will give you the middle finger.

Purely a guess,
Paul

Zaidy036

2017-01-21 20:06:05 UTC

Robocopy is a 32-bit program.
I think the maximum size of file should be limited to 2 GB?
Need to port and compile the program using 64-bit C/C++ compilers?
For the meantime, use a 64-bit 7-zip to compress and split the 10GB
files into multiple 1.5GB files?

Win 7 Home 64 bit
Robocopy 6.1.7601.23403
I have a batch using Robocopy from NAS over CAT 5 to PC to USB3 1 TB HDD
to backup an Acronis Images files (tib) 500GB (full) and 30 to 50 GB
(incremental) without problems except the time it takes to finish
writing the larger file.
SET "ipNAS=\\xxx.xxx.x.xx"
SET "_Set0=%ipNAS%\Image\i7\Set0"
SET _LOG=/LOG:F:\BackupSet0.txt
SET _ACT=/PURGE /R:0 /W:0 /NDL /NFL %_LOG%
START "" /WAIT Robocopy %_Set0% K:\Images_Set0 %_ACT%
I suggest monitoring the file size being written to view what is
happening. This can be done using a batch loop with SET _Size=%%~zI

Where did you get Robocopy 6.1.7601.23403?

"C:\Windows\System32\Robocopy.exe" Properties

Just did an Everything search for RoboCopy.exe "Executable:"
5.1.10.1027
"C:\Windows\winsxs\amd64_microsoft-windows-robocopy_31bf3856ad364e35_6.1.7601.17514_none_252d34f00303c6fa\Robocopy.exe"
6.1.7601.23403
"C:\Windows\winsxs\amd64_microsoft-windows-robocopy_31bf3856ad364e35_6.1.7601.23403_none_25c070011c1a6a70\Robocopy.exe"
5.1.10.1027
"C:\Windows\winsxs\x86_microsoft-windows-robocopy_31bf3856ad364e35_6.1.7601.17514_none_c90e996c4aa655c4\Robocopy.exe"
6.1.7601.23403
"C:\Windows\winsxs\x86_microsoft-windows-robocopy_31bf3856ad364e35_6.1.7601.23403_none_c9a1d47d63bcf93a\Robocopy.exe"
Two are 5.1.10.1027 and two 6.1.7601.23403 **BUT** why does the list
**NOT** include "C:\Windows\System32\Robocopy.exe" ???
Nothing in the RoboCopy Log indicates which version was being used but
since CMD comes up using C:\Windows\System32 I expect
"C:\Windows\System32\Robocopy.exe".

C:\Windows\System32\Robocopy.exe is a hardlinked object.
A filenum (entry in the $MFT) can have multiple file pointers.
Each one appears in the file system as if it was a separate file,
but they're all handled under the sane filenum entry.
The "nfi.exe" utility can show when a file has two or
more $FILE_NAME entries. But the utility won't print out the
contents of the hard links.
File 25918
\Windows\System32\Robocopy.exe
$STANDARD_INFORMATION (resident)
$FILE_NAME (resident) <--- the WinSXS one ???
$FILE_NAME (resident) <--- the System32 one ???
$DATA (nonresident)
logical sectors 20151712-20151967 (0x1337da0-0x1337e9f)
It turns out that Everything.exe just reads the first one, and
not any of the others.
https://www.voidtools.com/forum/viewtopic.php?t=459
"Only the first hardlink of a file will be indexed and monitored.
Files that are not the first hardlink will not be indexed or monitored.
This is a limitation of the USN Change Journal.
I have plans to index all hard links in the future.
However, you will have to update the indexes manually
as the USN Change Journal does not support hardlinks."
Now, if you run the nfi.exe command (a utility from Win2K days),
it lists all the winsxs entries first. Because, naturally, first
a file is stored in winsxs (as part of Component Based Servicing
or CBS), and afterwards the file is hardlinked into C:\Windows\System32.
So that gives a kind of answer, as to why you can see all
the WinSXS entries, but the actual entry that counts, the
one we use, because it comes second in the $MFT table used
to store the pointer, Everything.exe cannot list it.
*******
Something else I just learned, is more disturbing. I saw a
thread very recently, where someone was claiming the release
control system was busted on Robocopy. Now, I didn't pay this
any mind, because I was concentrating on something else.
However, now I realize the gravity of what that person was saying.
In an Administrator Command Prompt window...
fsutil hardlink list C:\Windows\System32\robocopy.exe
Mine comes back with this.
\Windows\System32\Robocopy.exe
\Windows\winsxs\amd64_microsoft-windows-robocopy_31bf3856ad364e35_6.1.7601.17514_none_252d34f00303c6fa\Robocopy.exe
The fricken *folder* in WinSXS has a release number
with the correct format for Win7SP1. A value of 7600
is the original Windows. And 7601 is SP1. "6.1.7601.17514"
However, if you do Properties on the file,
*all* the files use the (dishonest) 5.1.10.1027 identifier.
So for this file, the *only* way to get the release
information, is using the fsutil command!!! All the
files being hardlinked into the system32 folder,
will have the same 5.1.10.1027 number each time.
So if you were to complain to Microsoft
"My 5.1.10.1027 robocopy.exe is broken..."
the staff at Microsoft would laugh at you. Because
the actual number (6.1.7601.17514) is only available
by an indirect fsutil method.
So even if David had followed my instructions to
install the Convenience Rollup, visually, doing Properties
on the file would show 5.1.10.1027, and he'd tell
me I was full of shit.
*******
So *now* I understand, why what looked like a crusty copy
of Robocopy on my system, wasn't actually that crusty.
And we won't know until David runs the fsutil command,
what he is actually using.
How does this stuff happen ?
Simple.
Robocopy started life as an article on a webpage called
"Utility Spotlight". That means, it was the private
creation of a Microsoft employee and not part of the OS.
Later, Microsoft decided to bundle the file with the OS.
However, this release number snafu, indicates the stubborn
people at Microsoft, *will not* support this program
as if it is part of the OS. By *not* stamping robocopy,exe
properties with "6.1.7601.17514", they're saying it is
not part of the OS. And that's why the release number
involved here is "broken". That's a subtle hint - if you
ask for help, they will give you the middle finger.
Purely a guess,
Paul

Interesting

fsutil hardlink list C:\Windows\System32\robocopy.exe returned the
following:

\Windows\winsxs\amd64_microsoft-windows-robocopy_31bf3856ad364e35_6.1.7601.23403_none_25c070011c1a6a70\Robocopy.exe
\Windows\System32\Robocopy.exe

Note: 6.1.7601.23403

Paul

2017-01-21 16:36:48 UTC

Robocopy is a 32-bit program.
I think the maximum size of file should be limited to 2 GB?
Need to port and compile the program using 64-bit C/C++ compilers?
For the meantime, use a 64-bit 7-zip to compress and split the 10GB
files into multiple 1.5GB files?

Win 7 Home 64 bit
Robocopy 6.1.7601.23403
I have a batch using Robocopy from NAS over CAT 5 to PC to USB3 1 TB HDD
to backup an Acronis Images files (tib) 500GB (full) and 30 to 50 GB
(incremental) without problems except the time it takes to finish
writing the larger file.
SET "ipNAS=\\xxx.xxx.x.xx"
SET "_Set0=%ipNAS%\Image\i7\Set0"
SET _LOG=/LOG:F:\BackupSet0.txt
SET _ACT=/PURGE /R:0 /W:0 /NDL /NFL %_LOG%
START "" /WAIT Robocopy %_Set0% K:\Images_Set0 %_ACT%
I suggest monitoring the file size being written to view what is
happening. This can be done using a batch loop with SET _Size=%%~zI

Where did you get Robocopy 6.1.7601.23403?

"C:\Windows\System32\Robocopy.exe" Properties

https://blogs.technet.microsoft.com/askpfeplat/2016/05/20/windows-7-sp1-and-server-2008-r2-sp1-convenience-roll-up-now-available-at-a-download-location-near-you-kb3125574/

"Windows 7 SP1 and Server 2008 R2 SP1 convenience roll-up
now available at a download location near you! (KB3125574)"

I took a look in my copy of that download. Because I use this when
testing Windows Update for Win7SP1 (in a test VM, not on my real copy).

AMD64-all-windows6.1-kb3125574-v4-x64_2dafb1d203c8964239af3048b5dd4b1264cd93b9.msu

500,046,015 bytes

You can open "_manifest_.cix.xml" in the archive. For example,
this entry looks interesting.

<File id="45586" name="x86_microsoft-windows-robocopy_31bf3856ad364e35_6.1.7601.23403_none_c9a1d47d63bcf93a\robocopy.exe" length="101888" time="131034005737750000" attr="32">
<Hash alg="SHA1" value="903fff4b49ee8f38c18af4573ecdad636890a2ed" />
<Delta>
<Source type="PA30" name="11086">
<Hash alg="SHA1" value="856973479dd460752613c113d2e6439c1b8719de" />
</Source>
<Basis file="32850" />
</Delta>
</File>

Now, with that information, I'm not able to say anything.
I couldn't find any of the referenced material.

So what a person could do, is

1) visit catalog.update.microsoft.com and get a copy of kb3125574
2) Backup the OS.
3) Install it.
4) Grab the essential materials (EXE files and MUI maybe ?)
5) Restore from backup.

That's if there was actually something good about that version.

The convenience rollup is supposed to be made from
existing patches. So I don't know why the release number
of the robocopy.exe executable would be changed, just for
the rollup. I'm not able to find any other reference to that
particular one.

Paul

Zaidy036

2017-01-21 17:16:26 UTC

Robocopy is a 32-bit program.
I think the maximum size of file should be limited to 2 GB?
Need to port and compile the program using 64-bit C/C++ compilers?
For the meantime, use a 64-bit 7-zip to compress and split the 10GB
files into multiple 1.5GB files?

Win 7 Home 64 bit
Robocopy 6.1.7601.23403
I have a batch using Robocopy from NAS over CAT 5 to PC to USB3 1 TB HDD
to backup an Acronis Images files (tib) 500GB (full) and 30 to 50 GB
(incremental) without problems except the time it takes to finish
writing the larger file.
SET "ipNAS=\\xxx.xxx.x.xx"
SET "_Set0=%ipNAS%\Image\i7\Set0"
SET _LOG=/LOG:F:\BackupSet0.txt
SET _ACT=/PURGE /R:0 /W:0 /NDL /NFL %_LOG%
START "" /WAIT Robocopy %_Set0% K:\Images_Set0 %_ACT%
I suggest monitoring the file size being written to view what is
happening. This can be done using a batch loop with SET _Size=%%~zI

Where did you get Robocopy 6.1.7601.23403?

"C:\Windows\System32\Robocopy.exe" Properties

https://blogs.technet.microsoft.com/askpfeplat/2016/05/20/windows-7-sp1-and-server-2008-r2-sp1-convenience-roll-up-now-available-at-a-download-location-near-you-kb3125574/
"Windows 7 SP1 and Server 2008 R2 SP1 convenience roll-up
now available at a download location near you! (KB3125574)"
I took a look in my copy of that download. Because I use this when
testing Windows Update for Win7SP1 (in a test VM, not on my real copy).
AMD64-all-windows6.1-kb3125574-v4-x64_2dafb1d203c8964239af3048b5dd4b1264cd93b9.msu
500,046,015 bytes
You can open "_manifest_.cix.xml" in the archive. For example,
this entry looks interesting.
<File id="45586"
name="x86_microsoft-windows-robocopy_31bf3856ad364e35_6.1.7601.23403_none_c9a1d47d63bcf93a\robocopy.exe"
length="101888" time="131034005737750000" attr="32">
<Hash alg="SHA1"
value="903fff4b49ee8f38c18af4573ecdad636890a2ed" />
<Delta>
<Source type="PA30" name="11086">
<Hash alg="SHA1"
value="856973479dd460752613c113d2e6439c1b8719de" />
</Source>
<Basis file="32850" />
</Delta>
</File>
Now, with that information, I'm not able to say anything.
I couldn't find any of the referenced material.
So what a person could do, is
1) visit catalog.update.microsoft.com and get a copy of kb3125574
2) Backup the OS.
3) Install it.
4) Grab the essential materials (EXE files and MUI maybe ?)
5) Restore from backup.
That's if there was actually something good about that version.
The convenience rollup is supposed to be made from
existing patches. So I don't know why the release number
of the robocopy.exe executable would be changed, just for
the rollup. I'm not able to find any other reference to that
particular one.
Paul

kb3125574 is installed on my PC

Char Jackson

2017-01-21 01:46:28 UTC

On Fri, 20 Jan 2017 23:04:41 +0800, "Mr. Man-wai Chang"

Robocopy is a 32-bit program.
I think the maximum size of file should be limited to 2 GB?

No. Even if you meant 4GB-1, still no.

Post by Mr. Man-wai Chang
Need to port and compile the program using 64-bit C/C++ compilers?

No.

Post by Mr. Man-wai Chang
For the meantime, use a 64-bit 7-zip to compress and split the 10GB
files into multiple 1.5GB files?

No.

--
Char Jackson

David E. Ross

2017-01-18 06:35:33 UTC

Here is what I discovered with my latest attempt to use Robocopy.

I copied two files using my script, which relies on Robocopy.exe. One
was 9.6 MB; the other was 670.8 MB. The files were encrypted and
digitally signed with PGP. I was able to decrypt the files
successfully, and their digital signatures indicated they were not
corrupted.

I copied one file using the script. The file was 20,534.1 MB (20.5 GB).
The file was encrypted and digitally signed with PGP. I was unable to
decrypt the file, and its digital signature indicated it was corrupted.
I generated SHA512 hashes of the source and destination files, and they
did not compare.

I captured a log of the Robocopy.exe execution. The Robocopy.exe
process remained active at lest 3 minutes after the last write to the

Post by David E. Ross
-------------------------------------------------------------------------------
ROBOCOPY :: Robust File Copy for Windows
-------------------------------------------------------------------------------
Started : Tue Jan 17 22:23:05 2017
Source : D:\Backups\Backup C and J\C-disc\
Dest : K:\David\C and J\C-disc\
Files : *.pgp
Options : /V /TEE /COPY:DT /NP /R:10 /W:30
------------------------------------------------------------------------------
1 D:\Backups\Backup C and J\C-disc\
*EXTRA File 951.8 m C delta 10Jan17_inc_b2_s3_v1.tib.pgp
*EXTRA File 1.1 g C delta 3Jan17_inc_b2_s2_v1.tib.pgp
*EXTRA File 19.7 g C full 27Dec16_full_b2_s1_v1.tib.pgp
New File 19.5 g C full 17Jan17_full_b2_s1_v1.tib.pgp

I got equivalent bad results using the RobocopyGW.exe 3.1.2.0 user
interface for Robocopy.exe. While RobocopyGW.exe terminated normally,
Robocopy.exe itself remained active.

Conclusion: Robocopy.exe has a maximum file size, beyond which it is
not reliable. It is not truly "robust". Unfortunately, it is for very
large files that Robocopy.exe is most often used.

Does anyone know how I might report this problem to Microsoft?
--
David E. Ross
<http://www.rossde.com/>

When the President of the United States makes a statement of
national importance, I want to see his face as he is talking.
At the least, I want to hear his voice.

Donald: Stop tweeting. Otherwise, how do we know the message
really comes from you?

Paul

2017-01-18 07:49:48 UTC

Here is what I discovered with my latest attempt to use Robocopy.
I copied two files using my script, which relies on Robocopy.exe. One
was 9.6 MB; the other was 670.8 MB. The files were encrypted and
digitally signed with PGP. I was able to decrypt the files
successfully, and their digital signatures indicated they were not
corrupted.
I copied one file using the script. The file was 20,534.1 MB (20.5 GB).
The file was encrypted and digitally signed with PGP. I was unable to
decrypt the file, and its digital signature indicated it was corrupted.
I generated SHA512 hashes of the source and destination files, and they
did not compare.
I captured a log of the Robocopy.exe execution. The Robocopy.exe
process remained active at lest 3 minutes after the last write to the

You could try a Hotfix version for Win7.

https://support.microsoft.com/en-us/kb/2639043

*******

I would add an additional test case to what you're doing.

1) Do your PGP run. Prepare all .tib.pgp files.
2) *Reboot* the computer. This ensures there are no open handles,
no cached data, and so on. This is to put Robocopy on a level
playing field, where "files are just files".
3) Verify the signature of the files is still good.
This is intended to catch files open on write when
you rebooted.

*******

Another test case you can try.

1) Download dd.exe . The site main page has usage info
if you need it.

http://www.chrysocome.net/downloads/dd-0.6beta3.zip

2) cd \path\to\where\you\normally\work (Make sure dd.exe is in there)

dd.exe if=/dev/random of=some.tib.php bs=1048576 count=20000

That will make a file full of random data, just to
test your hard drive.

Now, robocopy one of those to the destination, as
if it was one of your tib.php. Run SHA512 on both
source and dest file. Does this work OK ?

*******

As for reporting your bug, I would try to reproduce in
Win10 Insider Edition, then I would use the Feedback Hub
(which is registered to my MSA Microsoft Account) and file
my bug that way.

Otherwise, it's pretty difficult to deliver a completely
anonymous report to Microsoft. They're "data gluttons" and
they need to track you. While there is Microsoft Connect,
it's limited in scope, and rather useless.

HTH,
Paul

VanguardLH

2017-01-18 16:07:55 UTC

Here is what I discovered with my latest attempt to use Robocopy.
I copied two files using my script, which relies on Robocopy.exe. One
was 9.6 MB; the other was 670.8 MB. The files were encrypted and
digitally signed with PGP. I was able to decrypt the files
successfully, and their digital signatures indicated they were not
corrupted.
I copied one file using the script. The file was 20,534.1 MB (20.5 GB).
The file was encrypted and digitally signed with PGP. I was unable to
decrypt the file, and its digital signature indicated it was corrupted.
I generated SHA512 hashes of the source and destination files, and they
did not compare.
I captured a log of the Robocopy.exe execution. The Robocopy.exe
process remained active at lest 3 minutes after the last write to the

With a 3-minute lag after what appears to be when robocopy completes the
copy process, it looks like robocopy returns status before it has
actually completed the copy. Could be it issue a request via file I/O
API in Windows so robocopy is done but not the OS. With robocopy
returning a status (before the file has completed copying), the next
command in the script executes (but the file is not yet complete).

Have you tried using the /z option (restartable). I've read it is
recommended when copying large files, especially over unstable
connections or to recover from corruption during the copy. Sometimes
copy speed is dramatically slowed when using the /z option.

/zb will try to use /z restartable mode unless access it denied in which
case /b backup mode gets used which is needed if you get access denied
errors and includes the files' attributes and permissions. robocopy
needs to run under an admin-level Windows account to use the /b option.

Since the Windows Vista version of robocopy, it will default to
multi-threading to concurrently copy multiple files. You could try to
force robocopy to copy only 1 file at a time by using /mt:1.

I did not see an option that forces robocopy to not return status until
it knows the file system has completed the copy operation.

Can you ensure the source files are not getting touched during the long
copy operation? robocopy does not use VSS. Because of the omission,
I've seen several articles on how to employ VSS with robocopy (e.g.,
http://www.robgolding.com/blog/2009/01/14/leveraging-vss-and-robocopy-for-robust-backups/).
That's why I suggested using VSScopy instead of robocopy so a VSS
snapshot of the source file got copied without problems caused by the
source files possibly changing during the copy. Before Microsoft came
out with VSS (in Windows XP), users had to do backups by rebooting and
booting into the backup program so all the OS and any apps were
quiescent to prevent files changing during the copy.

Paul

2017-01-19 08:19:01 UTC

Here is what I discovered with my latest attempt to use Robocopy.
I copied two files using my script, which relies on Robocopy.exe. One
was 9.6 MB; the other was 670.8 MB. The files were encrypted and
digitally signed with PGP. I was able to decrypt the files
successfully, and their digital signatures indicated they were not
corrupted.
I copied one file using the script. The file was 20,534.1 MB (20.5 GB).
The file was encrypted and digitally signed with PGP. I was unable to
decrypt the file, and its digital signature indicated it was corrupted.
I generated SHA512 hashes of the source and destination files, and they
did not compare.
I captured a log of the Robocopy.exe execution. The Robocopy.exe
process remained active at lest 3 minutes after the last write to the

Could you share the key lines in your script with us ? How do you
detect that pgp2.exe or whatever, is finished ?

I ran another test here, with a 25GB file, fed it to Kleopatra
(which calls pgp2.exe) and after Kleopatra said the operation
was complete, I ran Robocopy, it terminated, and both the SRC
and DEST folders had the same SHA1SUM.

I feel it's somehow related to what the other program running
on the system at the time is doing. It's either that, or there
is some kind of hardware problem present.

If you *do not* encrypt files, and just work with a 25GB test
file, what happens then ? Does the file copy ? Do the SRC and
DEST checksums match ? This is one way to make a test file.

dd.exe if=/dev/random of=some.tib bs=1048576 count=25000

I finally figured out in my tests here, what Kleopatra was
doing. The program appears to apply gzip to the source file,
before running RSA2048 on it. Once I switched to using a
random file, the unencrypted and encrypted files had closer
to the same size (as a random data file doesn't compress well).

HTH,
Paul

David E. Ross

2017-01-25 22:53:01 UTC

Today, I discovered I merely had to wait a while for Robocopy to
terminate. When I indicate "wait", I mean "wait, wait, and wait".

If copying 1 GB takes 11 seconds (which is indeed the time reported in
the Robocopy log), I would expect copying 10 GB to take ten times
longer: 110 seconds (1.8 minutes). According to the log, however, 10 GB
took over 6 minutes, more than 30 times longer. Nevertheless, the
source file and its copy now have the same SHA512 hash.

The question mark after "SOLVED" is because I am not sure there is a bug
that causes Robocopy to take disproportionally longer to copy
exceptionally large files.

--
David E. Ross
<http://www.rossde.com/>

When the President of the United States makes a statement of
national importance, I want to see his face as he is talking.
At the least, I want to hear his voice.

Donald: Stop tweeting. Otherwise, how do we know the message
really comes from you?

Paul

2017-01-26 12:03:22 UTC