Mem86+ and floppy drives

[John B. Smith]

Read on usenet that MEM86+ test is better than MEM86 for showing up
problems. Looked up with Google, its a free download. Wrote an .iso to
a flashdrive and it worked. One snag – my 2008 machine does not have a
fast-boot button to choose the flashdrive. I have to enter BIOS and
change the boot order. I'm used to having a 3.5" MEM86 floppy residing
in my floppy drive, all I have to do it push it in and boot. So I
wanted to write MEM86+ on a floppy. I found a site that promised that.
https://www.vogons.org/viewtopic.php?t=90258

I downloaded version 1.11 (tried earlier versions first). I ran the
install.bat and it writes a bootable mem86+ to the floppy, seemingly
with no errors. But then it doesn't want to boot.

I eventually shoved my usable MEM86 floppy in by mistake (wanted to
save that) and it produced a MEM86+ that does boot successfully. I
started trying a bunch of floppys, ages circa 2008, and they were ALL
failing in the same way. I tried doing a full format on one (1.4 megs
takes forever!) and it worked. But then I couldn't find another floppy
that this trick would work on.

I had been working in xp (that program that writes the floppy doesn't
work in a 64bit OS), but booted into Win10 and formatted a floppy
there. (I was thinking I could try FAT32 there, but no, it would still
only format FAT.
And when I wrote the MEM86+ on it it still wouldn't boot.

Are my 15 year old floppies all bad or is my probably older that 15
year old floppy drive going bad? I see Amazon wants $15 for a pack of
diskettes. If I can't solve this I'll probably waste the price of a
decent dinner to find out.

[Paul]

Post by John B. Smith
Read on usenet that MEM86+ test is better than MEM86 for showing up
problems. Looked up with Google, its a free download. Wrote an .iso to
a flashdrive and it worked. One snag – my 2008 machine does not have a
fast-boot button to choose the flashdrive. I have to enter BIOS and
change the boot order. I'm used to having a 3.5" MEM86 floppy residing
in my floppy drive, all I have to do it push it in and boot. So I
wanted to write MEM86+ on a floppy. I found a site that promised that.
https://www.vogons.org/viewtopic.php?t=90258
I downloaded version 1.11 (tried earlier versions first). I ran the
install.bat and it writes a bootable mem86+ to the floppy, seemingly
with no errors. But then it doesn't want to boot.
I eventually shoved my usable MEM86 floppy in by mistake (wanted to
save that) and it produced a MEM86+ that does boot successfully. I
started trying a bunch of floppys, ages circa 2008, and they were ALL
failing in the same way. I tried doing a full format on one (1.4 megs
takes forever!) and it worked. But then I couldn't find another floppy
that this trick would work on.
I had been working in xp (that program that writes the floppy doesn't
work in a 64bit OS), but booted into Win10 and formatted a floppy
there. (I was thinking I could try FAT32 there, but no, it would still
only format FAT.
And when I wrote the MEM86+ on it it still wouldn't boot.
Are my 15 year old floppies all bad or is my probably older that 15
year old floppy drive going bad? I see Amazon wants $15 for a pack of
diskettes. If I can't solve this I'll probably waste the price of a
decent dinner to find out.

You can "dd" the image off the floppy (as a 1440K file) and
examine it in a hex editor.

This is the command the install.bat uses in one instance.
It's a raw copy of an executable file, to a floppy diskette,
starting at sector 0. The file system information is just overwritten.
This file is not "stored in FAT32" or anything. It does not work that way.
It's the most primal way of running a computer. Sequential load off the
floppy as if it was a tiny tape drive.

dd.exe if=memtestp.bin of=\\.\a: bs=18k

"The HD (High Density) format of ​a 1.44MB 3.5-inch floppy disks
has 80 tracks with 18 sectors per track and 512 bytes per sector
and two sides for each disk. So 80 * 18 * 512 *2 equals 1,474,560 bytes.
If you divide that number by 1024, you get the 1440K size nomenclature."

If you were to do "disktype 1440K.bin" it would tell you there
was no file system on the thing. But we don't even need to do that.

You need to dd the content off the success floppy and the fail floppy,
and see what is different or what is missing. You would compare the
memtestp.bin file to the first part of the 1440K.bin file you make.

The "dd" they're using is traceable to

http://www.chrysocome.net/download

because the person who wrote dd.exe originally, called it
rawwrite instead. The name changed to dd.exe later.

http://www.chrysocome.net/downloads/dd-0.6beta3.zip

I think if you do your forensics, the problem won't be
magical or anything.

If you wish, you can also use dd to zero the floppy diskette
from end to end. It should be possible to "erase" a floppy,
and not in the form of a FAT file system either. Just zero
the damn thing.

dd.exe if=/dev/zero of=\\.\a: bs=18k # It will stop when it runs out of space.

You can dd it back and verify it is OK.

dd.exe if=\\.\a: of=1440K.bin bs=18k

Paul

[John B. Smith]

Post by Paul

Post by John B. Smith
Read on usenet that MEM86+ test is better than MEM86 for showing up
problems. Looked up with Google, its a free download. Wrote an .iso to
a flashdrive and it worked. One snag – my 2008 machine does not have a
fast-boot button to choose the flashdrive. I have to enter BIOS and
change the boot order. I'm used to having a 3.5" MEM86 floppy residing
in my floppy drive, all I have to do it push it in and boot. So I
wanted to write MEM86+ on a floppy. I found a site that promised that.
https://www.vogons.org/viewtopic.php?t=90258
I downloaded version 1.11 (tried earlier versions first). I ran the
install.bat and it writes a bootable mem86+ to the floppy, seemingly
with no errors. But then it doesn't want to boot.
I eventually shoved my usable MEM86 floppy in by mistake (wanted to
save that) and it produced a MEM86+ that does boot successfully. I
started trying a bunch of floppys, ages circa 2008, and they were ALL
failing in the same way. I tried doing a full format on one (1.4 megs
takes forever!) and it worked. But then I couldn't find another floppy
that this trick would work on.
I had been working in xp (that program that writes the floppy doesn't
work in a 64bit OS), but booted into Win10 and formatted a floppy
there. (I was thinking I could try FAT32 there, but no, it would still
only format FAT.
And when I wrote the MEM86+ on it it still wouldn't boot.
Are my 15 year old floppies all bad or is my probably older that 15
year old floppy drive going bad? I see Amazon wants $15 for a pack of
diskettes. If I can't solve this I'll probably waste the price of a
decent dinner to find out.

You can "dd" the image off the floppy (as a 1440K file) and
examine it in a hex editor.
This is the command the install.bat uses in one instance.
It's a raw copy of an executable file, to a floppy diskette,
starting at sector 0. The file system information is just overwritten.
This file is not "stored in FAT32" or anything. It does not work that way.
It's the most primal way of running a computer. Sequential load off the
floppy as if it was a tiny tape drive.
dd.exe if=memtestp.bin of=\\.\a: bs=18k
"The HD (High Density) format of ?a 1.44MB 3.5-inch floppy disks
has 80 tracks with 18 sectors per track and 512 bytes per sector
and two sides for each disk. So 80 * 18 * 512 *2 equals 1,474,560 bytes.
If you divide that number by 1024, you get the 1440K size nomenclature."
If you were to do "disktype 1440K.bin" it would tell you there
was no file system on the thing. But we don't even need to do that.
You need to dd the content off the success floppy and the fail floppy,
and see what is different or what is missing. You would compare the
memtestp.bin file to the first part of the 1440K.bin file you make.
The "dd" they're using is traceable to
http://www.chrysocome.net/download
because the person who wrote dd.exe originally, called it
rawwrite instead. The name changed to dd.exe later.
http://www.chrysocome.net/downloads/dd-0.6beta3.zip
I think if you do your forensics, the problem won't be
magical or anything.
If you wish, you can also use dd to zero the floppy diskette
from end to end. It should be possible to "erase" a floppy,
and not in the form of a FAT file system either. Just zero
the damn thing.
dd.exe if=/dev/zero of=\\.\a: bs=18k # It will stop when it runs out of space.

So far this command is all I've done. It zeros the 3.5 floppy in an
instant compared to trying to format it. His bat file tells you it
wants a 'formatted' floppy to start with, but if it works like you say
I don't think it really cares. And there's no check whether the copy
WAS made successfully after the write is done. If I was gungho
programmer maybe I'd try. Although on one attempt the bat file DID
say the write was unsuccessful for whatever reason. I ordered some new
floppies on Amazon. I'm sure you're all breathlessly awaiting the
result of this important experiment.

I greatly appreciate the explanation of dd and rawrite. My only
previous brush with it was checking its box when trying to get a
Kaspersky flash drive to boot. Didn't have the foggiest what I was
doing but that made it work. I've kind of gotten disillusioned with K
since then though.

Post by Paul
You can dd it back and verify it is OK.
dd.exe if=\\.\a: of=1440K.bin bs=18k
Paul

[John B. Smith]

Post by John B. Smith

Post by Paul

Post by John B. Smith
Read on usenet that MEM86+ test is better than MEM86 for showing up
problems. Looked up with Google, its a free download. Wrote an .iso to
a flashdrive and it worked. One snag – my 2008 machine does not have a
fast-boot button to choose the flashdrive. I have to enter BIOS and
change the boot order. I'm used to having a 3.5" MEM86 floppy residing
in my floppy drive, all I have to do it push it in and boot. So I
wanted to write MEM86+ on a floppy. I found a site that promised that.
https://www.vogons.org/viewtopic.php?t=90258
I downloaded version 1.11 (tried earlier versions first). I ran the
install.bat and it writes a bootable mem86+ to the floppy, seemingly
with no errors. But then it doesn't want to boot.
I eventually shoved my usable MEM86 floppy in by mistake (wanted to
save that) and it produced a MEM86+ that does boot successfully. I
started trying a bunch of floppys, ages circa 2008, and they were ALL
failing in the same way. I tried doing a full format on one (1.4 megs
takes forever!) and it worked. But then I couldn't find another floppy
that this trick would work on.
I had been working in xp (that program that writes the floppy doesn't
work in a 64bit OS), but booted into Win10 and formatted a floppy
there. (I was thinking I could try FAT32 there, but no, it would still
only format FAT.
And when I wrote the MEM86+ on it it still wouldn't boot.
Are my 15 year old floppies all bad or is my probably older that 15
year old floppy drive going bad? I see Amazon wants $15 for a pack of
diskettes. If I can't solve this I'll probably waste the price of a
decent dinner to find out.

You can "dd" the image off the floppy (as a 1440K file) and
examine it in a hex editor.
This is the command the install.bat uses in one instance.
It's a raw copy of an executable file, to a floppy diskette,
starting at sector 0. The file system information is just overwritten.
This file is not "stored in FAT32" or anything. It does not work that way.
It's the most primal way of running a computer. Sequential load off the
floppy as if it was a tiny tape drive.
dd.exe if=memtestp.bin of=\\.\a: bs=18k
"The HD (High Density) format of ?a 1.44MB 3.5-inch floppy disks
has 80 tracks with 18 sectors per track and 512 bytes per sector
and two sides for each disk. So 80 * 18 * 512 *2 equals 1,474,560 bytes.
If you divide that number by 1024, you get the 1440K size nomenclature."
If you were to do "disktype 1440K.bin" it would tell you there
was no file system on the thing. But we don't even need to do that.
You need to dd the content off the success floppy and the fail floppy,
and see what is different or what is missing. You would compare the
memtestp.bin file to the first part of the 1440K.bin file you make.
The "dd" they're using is traceable to
http://www.chrysocome.net/download
because the person who wrote dd.exe originally, called it
rawwrite instead. The name changed to dd.exe later.
http://www.chrysocome.net/downloads/dd-0.6beta3.zip
I think if you do your forensics, the problem won't be
magical or anything.
If you wish, you can also use dd to zero the floppy diskette
from end to end. It should be possible to "erase" a floppy,
and not in the form of a FAT file system either. Just zero
the damn thing.
dd.exe if=/dev/zero of=\\.\a: bs=18k # It will stop when it runs out of space.

So far this command is all I've done. It zeros the 3.5 floppy in an
instant compared to trying to format it. His bat file tells you it
wants a 'formatted' floppy to start with, but if it works like you say
I don't think it really cares. And there's no check whether the copy
WAS made successfully after the write is done. If I was gungho
programmer maybe I'd try. Although on one attempt the bat file DID
say the write was unsuccessful for whatever reason. I ordered some new
floppies on Amazon. I'm sure you're all breathlessly awaiting the
result of this important experiment.

Received the new diskettes - same problem. I now think my ancient
floppy drive is the problem. I know I threw away a couple of these
drives with their pcs before I built my home-built. Wish I could have
forseen a time when they were no longer available. Anyhoo the problem
is not worth chasing further (or even this far). I have two working
Mem86+ diskettes and a flashdrive with the latest Mem86+, they'll
suffice. Thanks for all the help.

Post by John B. Smith
I greatly appreciate the explanation of dd and rawrite. My only
previous brush with it was checking its box when trying to get a
Kaspersky flash drive to boot. Didn't have the foggiest what I was
doing but that made it work. I've kind of gotten disillusioned with K
since then though.

Post by Paul
You can dd it back and verify it is OK.
dd.exe if=\\.\a: of=1440K.bin bs=18k
Paul

[Paul]

Post by John B. Smith
Received the new diskettes - same problem. I now think my ancient
floppy drive is the problem. I know I threw away a couple of these
drives with their pcs before I built my home-built. Wish I could have
forseen a time when they were no longer available. Anyhoo the problem
is not worth chasing further (or even this far). I have two working
Mem86+ diskettes and a flashdrive with the latest Mem86+, they'll
suffice. Thanks for all the help.

There used to be a variety of head cleaning kits for
various sizes of floppy.

Cleaning kits generally have a "fiber" disc inside,
which does the cleaning. For some cleaning kits (depending
on what kind of storage device they're for), they come
with a small supply of an alcohol substance, and you
can put a few drops on the cleaning thing before
inserting it.

Loading Image...

I never really found a problem with 3.5" floppies
and dirty heads. I would be surprised if the head
was fouled on one of those. There are other head types,
such as the helical scan head on a digital tape drive,
those will only run 30-40 hours, before you clean them.
And we did have cases of data loss, due to helical scan
heads being filthy. But for me at least, the 3.5"
floppies seemed to be pretty good.

*******

As for the general topic of cleaning, things like optical
drives, the cleaning kits for those can be a scam. You're
more likely to damage an optical drive with a cleaning
kit, than actually clean it.

Not every storage type, fouls. But the ones that do,
they can be pretty bad. While others can go for years,
with no attention at all.

I did a fair number of cleanings for 8" floppy drives,
and the media on those used ferric oxide particles.

If a fool manages to keep the 3.5" heads loaded for hours
on end, then sure, the heads can get damaged. On a
floppy drive, reversing the ribbon cable causes
the light to come on, and the heads just stay loaded.
You're not supposed to leave the machine that way,
when you do that by accident :-) Power off, and
correct the mistake.

Paul

[c***@nycap.rr.com]

Post by Paul

Post by John B. Smith
Received the new diskettes - same problem. I now think my ancient
floppy drive is the problem. I know I threw away a couple of these
drives with their pcs before I built my home-built. Wish I could have
forseen a time when they were no longer available. Anyhoo the problem
is not worth chasing further (or even this far). I have two working
Mem86+ diskettes and a flashdrive with the latest Mem86+, they'll
suffice. Thanks for all the help.

There used to be a variety of head cleaning kits for
various sizes of floppy.
Cleaning kits generally have a "fiber" disc inside,
which does the cleaning. For some cleaning kits (depending
on what kind of storage device they're for), they come
with a small supply of an alcohol substance, and you
can put a few drops on the cleaning thing before
inserting it.
https://en.m.wikipedia.org/wiki/File:Disk-cleaning-kit-front-and-rear.jpg
I never really found a problem with 3.5" floppies
and dirty heads. I would be surprised if the head
was fouled on one of those. There are other head types,
such as the helical scan head on a digital tape drive,
those will only run 30-40 hours, before you clean them.
And we did have cases of data loss, due to helical scan
heads being filthy. But for me at least, the 3.5"
floppies seemed to be pretty good.
*******

Well, MAYBE I found the answer to my Mem86+ and floppy disk problem. I
bought a cleaning kit. I think it was this one:
https://www.etsy.com/listing/1368860147/head-cleaning-pulisci-testine-disk-drive?ga_order=most_relevant&ga_search_type=all&ga_view_type=gallery&ga_search_query=diskette+cleaner&ref=sr_gallery-1-15&sts=1&content_source=9517f8b8d966d9543262b9dbe629c7357a15969c%253A1368860147&search_preloaded_img=1&organic_search_click=1

I thought I got really screwed on the shipping, they wouldn't tell you
till the last minute when they already had my credit card. But when
the package finally arrived it was from ITALY! So I guess the shipping
($13 postage on the package) wasn't too far off. Though I probably
would have passed on it if I'd known up front. Anyway, I ran the
cleaning disk with alcohol thru several times, then tried writing the
Mem86+ floppy version1.111 and when I booted the floppy it ran Mem86+.
I stopped testing at that point.

Post by Paul
As for the general topic of cleaning, things like optical
drives, the cleaning kits for those can be a scam. You're
more likely to damage an optical drive with a cleaning
kit, than actually clean it.
Not every storage type, fouls. But the ones that do,
they can be pretty bad. While others can go for years,
with no attention at all.
I did a fair number of cleanings for 8" floppy drives,
and the media on those used ferric oxide particles.
If a fool manages to keep the 3.5" heads loaded for hours
on end, then sure, the heads can get damaged. On a
floppy drive, reversing the ribbon cable causes
the light to come on, and the heads just stay loaded.
You're not supposed to leave the machine that way,
when you do that by accident :-) Power off, and
correct the mistake.
Paul

[David E. Ross]

Post by John B. Smith
Read on usenet that MEM86+ test is better than MEM86 for showing up
problems. Looked up with Google, its a free download. Wrote an .iso to
a flashdrive and it worked. One snag – my 2008 machine does not have a
fast-boot button to choose the flashdrive. I have to enter BIOS and
change the boot order. I'm used to having a 3.5" MEM86 floppy residing
in my floppy drive, all I have to do it push it in and boot. So I
wanted to write MEM86+ on a floppy. I found a site that promised that.
https://www.vogons.org/viewtopic.php?t=90258
I downloaded version 1.11 (tried earlier versions first). I ran the
install.bat and it writes a bootable mem86+ to the floppy, seemingly
with no errors. But then it doesn't want to boot.
I eventually shoved my usable MEM86 floppy in by mistake (wanted to
save that) and it produced a MEM86+ that does boot successfully. I
started trying a bunch of floppys, ages circa 2008, and they were ALL
failing in the same way. I tried doing a full format on one (1.4 megs
takes forever!) and it worked. But then I couldn't find another floppy
that this trick would work on.
I had been working in xp (that program that writes the floppy doesn't
work in a 64bit OS), but booted into Win10 and formatted a floppy
there. (I was thinking I could try FAT32 there, but no, it would still
only format FAT.
And when I wrote the MEM86+ on it it still wouldn't boot.
Are my 15 year old floppies all bad or is my probably older that 15
year old floppy drive going bad? I see Amazon wants $15 for a pack of
diskettes. If I can't solve this I'll probably waste the price of a
decent dinner to find out.

Your floppies might all be good, but they might not be big enough. A
floppy can only hold 1.4 Mb. The latest Memtest86+ (10.7) requires a
medium with 1 Gb. Version 4.3.7 (compatible with Windows XP) likely
requires more than 4 Mb, not including the user guide.

These sizes represent what is required to create a bootable medium.
Once that is created, however, the medium can be copied to a smaller
medium for actual booting. This difference is caused by the fact that
the medium must hold both the installer package and the resulting
executable.

When the executable is completely created, the installer package is
apparently deleted. I generate a bootable 10.7 on a 34 Gb flash drive
and then move the executable to a 260 Mb flash drive for use.

[David E. Ross]

Post by David E. Ross

Post by John B. Smith
Read on usenet that MEM86+ test is better than MEM86 for showing up
problems. Looked up with Google, its a free download. Wrote an .iso to
a flashdrive and it worked. One snag – my 2008 machine does not have a
fast-boot button to choose the flashdrive. I have to enter BIOS and
change the boot order. I'm used to having a 3.5" MEM86 floppy residing
in my floppy drive, all I have to do it push it in and boot. So I
wanted to write MEM86+ on a floppy. I found a site that promised that.
https://www.vogons.org/viewtopic.php?t=90258
I downloaded version 1.11 (tried earlier versions first). I ran the
install.bat and it writes a bootable mem86+ to the floppy, seemingly
with no errors. But then it doesn't want to boot.
I eventually shoved my usable MEM86 floppy in by mistake (wanted to
save that) and it produced a MEM86+ that does boot successfully. I
started trying a bunch of floppys, ages circa 2008, and they were ALL
failing in the same way. I tried doing a full format on one (1.4 megs
takes forever!) and it worked. But then I couldn't find another floppy
that this trick would work on.
I had been working in xp (that program that writes the floppy doesn't
work in a 64bit OS), but booted into Win10 and formatted a floppy
there. (I was thinking I could try FAT32 there, but no, it would still
only format FAT.
And when I wrote the MEM86+ on it it still wouldn't boot.
Are my 15 year old floppies all bad or is my probably older that 15
year old floppy drive going bad? I see Amazon wants $15 for a pack of
diskettes. If I can't solve this I'll probably waste the price of a
decent dinner to find out.

Your floppies might all be good, but they might not be big enough. A
floppy can only hold 1.4 Mb. The latest Memtest86+ (10.7) requires a
medium with 1 Gb. Version 4.3.7 (compatible with Windows XP) likely
requires more than 4 Mb, not including the user guide.
These sizes represent what is required to create a bootable medium.
Once that is created, however, the medium can be copied to a smaller
medium for actual booting. This difference is caused by the fact that
the medium must hold both the installer package and the resulting
executable.
When the executable is completely created, the installer package is
apparently deleted. I generate a bootable 10.7 on a 34 Gb flash drive
and then move the executable to a 260 Mb flash drive for use.

By the way, I know that the installer package is actually loaded onto a
drive of your PC, not on the executable's media. It seems that the
process of creating the executable, however, uses some of the media for
some temporary files. That is what I was told when I could not generate
the executable on a 250 Mb medium for version 10.6. PassMark (the
developer of Memtest+) informed me that I needed a medium with at least
500 Mb. When I finished generating the executable, I discovered that it
occupied less than 250 Mb. Thus, I moved the contents of the medium to
a smaller flash drive.

[croy]

Post by John B. Smith
Are my 15 year old floppies all bad or is my probably older that 15
year old floppy drive going bad? I see Amazon wants $15 for a pack of
diskettes. If I can't solve this I'll probably waste the price of a
decent dinner to find out.

If it's worth it to you, you could try SpinRite version 5--it saved a few
of my floppies back in the day.

[David E. Ross]

Post by John B. Smith
Read on usenet that MEM86+ test is better than MEM86 for showing up
problems. Looked up with Google, its a free download. Wrote an .iso to
a flashdrive and it worked. One snag – my 2008 machine does not have a
fast-boot button to choose the flashdrive. I have to enter BIOS and
change the boot order. I'm used to having a 3.5" MEM86 floppy residing
in my floppy drive, all I have to do it push it in and boot. So I
wanted to write MEM86+ on a floppy. I found a site that promised that.
https://www.vogons.org/viewtopic.php?t=90258
I downloaded version 1.11 (tried earlier versions first). I ran the
install.bat and it writes a bootable mem86+ to the floppy, seemingly
with no errors. But then it doesn't want to boot.
I eventually shoved my usable MEM86 floppy in by mistake (wanted to
save that) and it produced a MEM86+ that does boot successfully. I
started trying a bunch of floppys, ages circa 2008, and they were ALL
failing in the same way. I tried doing a full format on one (1.4 megs
takes forever!) and it worked. But then I couldn't find another floppy
that this trick would work on.
I had been working in xp (that program that writes the floppy doesn't
work in a 64bit OS), but booted into Win10 and formatted a floppy
there. (I was thinking I could try FAT32 there, but no, it would still
only format FAT.
And when I wrote the MEM86+ on it it still wouldn't boot.
Are my 15 year old floppies all bad or is my probably older that 15
year old floppy drive going bad? I see Amazon wants $15 for a pack of
diskettes. If I can't solve this I'll probably waste the price of a
decent dinner to find out.

I was just now reviewing some old E-mails I exchanged with the Memtest
developers.

I always installed Memtest86+ on a flash drive, but a recent (not the
latest) installation repeatedly failed. In one exchange, the developers
informed me that I needed a flash drive with at least 512MB capacity to
install Memtest86+. "There needs to be some empty space on the drive as
we are now writing logs and reports to the USB drive."

I installed Memtest86+ on a flash drive with a very large capacity. (It
is labeled 32 GB, but I doubt my Windows 7 PC can address that much.)
However, the result required only a little over 8 MB; so I moved it to a
smaller 248 MB flash drive. You might try installing on a 700 MB CD if
you do not have large-capacity flash drives. I do not think you can
install any version of Memtest86+ on a floppy.

In another exchange of E-mail, the developers informed me that the
latest version of Memtest86+ can be used on any PC that can support UEFI
booting. That includes some Windows XP systems. The issue is NOT the
version of Windows because Memtest86+ does not run under Windows but
instead of Windows. Instead, the issue is what kind of boot is
available. If UEFI is not available, then Memtest+ version 4 should be
used.

[J. P. Gilliver]

In message <v2h24d$de8g$***@dont-email.me> at Mon, 20 May 2024 19:49:47,
David E. Ross <***@nowhere.invalid> writes
[]

Post by David E. Ross
I installed Memtest86+ on a flash drive with a very large capacity. (It
is labeled 32 GB, but I doubt my Windows 7 PC can address that much.)

[]
I have a "32G" flash drive plugged into this W7 (32 bit) laptop, and it
doesn't seem to be having any problem with it. (Well, scanner - from
http://steffengerlach.de/freeware/ - shows it as 28.8 GBytes total, 9.71
GBytes used.)

[Paul]

[]

I installed Memtest86+ on a flash drive with a very large capacity.  (It
is labeled 32 GB, but I doubt my Windows 7 PC can address that much.)

[]
I have a "32G" flash drive plugged into this W7 (32 bit) laptop, and it doesn't seem to be having any problem with it. (Well, scanner - from http://steffengerlach.de/freeware/ - shows it as 28.8 GBytes total, 9.71 GBytes used.)

There is no limit to how much you can have on a flash drive.

That's because a SCSI CDB is used in the stack. Windows uses
SCSI stack as the "stack for foreign storage entities". Any time
a drive isn't an IDE drive, you have to do something to fit the
device to the system. And SCSI is one level of the stack. The
first SATA drives (on a Silicon Image SIL3112), there used to be
a stub SCSI stack in the driver arrangement.

https://superuser.com/questions/308492/is-there-a-size-limit-on-external-usb-hard-drives

"READ(16) and READ(32) use 64-bit LBAs imposing an 8 ZiB limit (that's 8 giga-terabytes!)."

Windows 7 supports GPT partitioning, which removes the 32-bit MBR limitation
when defining storage. If you were using MSDOS partitioning on Windows 7, that
has a 2.2TB limit. When you buy a big drive, you use GPT so that all of
the disk can be used without a problem.

*******

And yes, of course versions of memory tester have fit on floppy diskettes.
That's what I have sitting in front of me, within reach. My main PCs no longer
have floppy drives (no controller on the motherboard), but I still have a
USB to floppy box which does what is needed. Only an older PC that did not
support USB booting, might not be able to find a USB floppy drive. And those
older machines, have a floppy connector on the motherboard, for the legacy
kind of floppy drive.

With the floppy version, the program absolutely did not write on the floppy,
because... the floppy had no file system on it. You cannot "dir" an old
memtest floppy. It has no drive letter.

The new design is entirely different. How much of that is due to the Intel
promise to go "UEFI only" on new hardware, I do not know. Without CSM
compatibility boot, some day the older stuff might no longer boot
on PCs. This is one reason I won't be buying motherboards when they
are UEFI only. Not interested. As a Scot would say "now you're getting
in my road".

Paul

[J. P. Gilliver]

In message <v2hr8v$hmrj$***@dont-email.me> at Tue, 21 May 2024 05:58:54,
Paul <***@needed.invalid> writes
[]

Post by Paul
With the floppy version, the program absolutely did not write on the floppy,
because... the floppy had no file system on it. You cannot "dir" an old
memtest floppy. It has no drive letter.

Interesting! I don't think I ever tried; assuming I'd booted into DOS or
Windows, I'd guess the _drive_ would have a letter; I don't think I ever
tried to dir a memtest floppy, though.

Post by Paul
The new design is entirely different. How much of that is due to the Intel
promise to go "UEFI only" on new hardware, I do not know. Without CSM
compatibility boot, some day the older stuff might no longer boot
on PCs. This is one reason I won't be buying motherboards when they
are UEFI only. Not interested. As a Scot would say "now you're getting
in my road".

I'd assumed a fair number of them already are, if not all; you give me
hope! (Though I can't see me ever assembling a PC again.)

Post by Paul
Paul

John

[Paul]

Post by J. P. Gilliver
I'd assumed a fair number of them already are, if not all;
you give me hope! (Though I can't see me ever assembling a PC again.)

That's where Sparkle Ponies come from.

The ability to shop until you find what you want.

A Dell or an HP is just gonna "go with the flow".

Paul

[c***@nycap.rr.com]

Post by Paul

[]

I installed Memtest86+ on a flash drive with a very large capacity.  (It
is labeled 32 GB, but I doubt my Windows 7 PC can address that much.)

[]
I have a "32G" flash drive plugged into this W7 (32 bit) laptop, and it doesn't seem to be having any problem with it. (Well, scanner - from http://steffengerlach.de/freeware/ - shows it as 28.8 GBytes total, 9.71 GBytes used.)

There is no limit to how much you can have on a flash drive.
That's because a SCSI CDB is used in the stack. Windows uses
SCSI stack as the "stack for foreign storage entities". Any time
a drive isn't an IDE drive, you have to do something to fit the
device to the system. And SCSI is one level of the stack. The
first SATA drives (on a Silicon Image SIL3112), there used to be
a stub SCSI stack in the driver arrangement.
https://superuser.com/questions/308492/is-there-a-size-limit-on-external-usb-hard-drives
"READ(16) and READ(32) use 64-bit LBAs imposing an 8 ZiB limit (that's 8 giga-terabytes!)."
Windows 7 supports GPT partitioning, which removes the 32-bit MBR limitation
when defining storage. If you were using MSDOS partitioning on Windows 7, that
has a 2.2TB limit. When you buy a big drive, you use GPT so that all of
the disk can be used without a problem.

Is it possible to have both types of hard drive
partitioning on your system, GPT drives and MBR drives and your modern
BIOS would accomodate both and Windows Explorer would just 'see' both
drives without complaint?

Post by Paul
*******
And yes, of course versions of memory tester have fit on floppy diskettes.
That's what I have sitting in front of me, within reach. My main PCs no longer
have floppy drives (no controller on the motherboard), but I still have a
USB to floppy box which does what is needed. Only an older PC that did not
support USB booting, might not be able to find a USB floppy drive. And those
older machines, have a floppy connector on the motherboard, for the legacy
kind of floppy drive.
With the floppy version, the program absolutely did not write on the floppy,
because... the floppy had no file system on it. You cannot "dir" an old
memtest floppy. It has no drive letter.
The new design is entirely different. How much of that is due to the Intel
promise to go "UEFI only" on new hardware, I do not know. Without CSM
compatibility boot, some day the older stuff might no longer boot
on PCs. This is one reason I won't be buying motherboards when they
are UEFI only. Not interested. As a Scot would say "now you're getting
in my road".
Paul

[Char Jackson]

Post by c***@nycap.rr.com

Post by Paul
Windows 7 supports GPT partitioning, which removes the 32-bit MBR limitation
when defining storage. If you were using MSDOS partitioning on Windows 7, that
has a 2.2TB limit. When you buy a big drive, you use GPT so that all of
the disk can be used without a problem.

Is it possible to have both types of hard drive
partitioning on your system, GPT drives and MBR drives and your modern
BIOS would accomodate both and Windows Explorer would just 'see' both
drives without complaint?

Yes, absolutely.

[Paul]

Post by Char Jackson

Post by c***@nycap.rr.com

Post by Paul
Windows 7 supports GPT partitioning, which removes the 32-bit MBR limitation
when defining storage. If you were using MSDOS partitioning on Windows 7, that
has a 2.2TB limit. When you buy a big drive, you use GPT so that all of
the disk can be used without a problem.

Is it possible to have both types of hard drive
partitioning on your system, GPT drives and MBR drives and your modern
BIOS would accomodate both and Windows Explorer would just 'see' both
drives without complaint?

Yes, absolutely.

It's nice when stuff works :-) That's for sure.

*******

The next area of interest, coming up, is NTFS only has four billion clusters.
According to Wiki:

2^64 clusters − 1 cluster (format); [No idea what this means, 2^64 is a CDB limit maybe]

256 TB size − 64 KB clusters ( early Windows 10 or less )
8 PB − 2 MB clusters ( late Windows 10 had more cluster sizes added, NotBackwardCompatible )
( if you show this partition to Win7, it "offers to format it", out of spite )

From this, we might conclude

16 TB size - C: drive Windows 10 install wants only 4 KB clusters (enables compression)
- It's possible Win7 might still accept 64 KB clusters for a Windows Install.

As a test of VirtualBox, I tried to create a large disk using a .vdi container.
This is the best I could do. In the Properties pie chart area for F: ,
File Explorer was unable to work out how much free space there was,
whereas good old Command Prompt worked it out.

F:\>dir
Volume in drive F is BIG
Volume Serial Number is EE25-2764

Directory of F:\

05/22/2024 07:21 AM 82,110 test.txt
1 File(s) 82,110 bytes
0 Dir(s) 562,949,287,706,624 bytes free

F:\>

We may be approaching a point, where some things "act up again",
just after the "trauma of 2.2TB" had passed :-)

Anyway, this little investigation makes me wonder what the OS
did with my backup drive. I had a few problems with partitioning it,
and may have stepped into this issue, without recognizing what
was going on. The OS is pretty crafty, and if you offer a large
volume (in modern Win10), it just restricts your cluster size
choices and does not offer any info you might benefit from,
for planning purposes. The above 512TB volume is using 128KB clusters,
as a guess as to how that worked out.

Size: 82.110 bytes
Size on disk: 131,072 bytes <== 128KB cluster, can't work with Win7

Paul

[Kerr-Mudd, John]

Note: xposted to afc

On Wed, 22 May 2024 09:23:02 -0400

Post by Paul

Post by Char Jackson

Post by c***@nycap.rr.com

Post by Paul
Windows 7 supports GPT partitioning, which removes the 32-bit MBR limitation
when defining storage. If you were using MSDOS partitioning on Windows 7, that
has a 2.2TB limit. When you buy a big drive, you use GPT so that all of
the disk can be used without a problem.

Is it possible to have both types of hard drive
partitioning on your system, GPT drives and MBR drives and your modern
BIOS would accomodate both and Windows Explorer would just 'see' both
drives without complaint?

Yes, absolutely.

It's nice when stuff works :-) That's for sure.
*******
The next area of interest, coming up, is NTFS only has four billion clusters.
2^64 clusters − 1 cluster (format); [No idea what this means, 2^64 is a CDB limit maybe]
256 TB size − 64 KB clusters ( early Windows 10 or less )
8 PB − 2 MB clusters ( late Windows 10 had more cluster sizes added, NotBackwardCompatible )
( if you show this partition to Win7, it "offers to format it", out of spite )
From this, we might conclude
16 TB size - C: drive Windows 10 install wants only 4 KB clusters (enables compression)
- It's possible Win7 might still accept 64 KB clusters for a Windows Install.
As a test of VirtualBox, I tried to create a large disk using a .vdi container.
This is the best I could do. In the Properties pie chart area for F: ,
File Explorer was unable to work out how much free space there was,
whereas good old Command Prompt worked it out.
F:\>dir
Volume in drive F is BIG
Volume Serial Number is EE25-2764
Directory of F:\
05/22/2024 07:21 AM 82,110 test.txt
1 File(s) 82,110 bytes
0 Dir(s) 562,949,287,706,624 bytes free
F:\>
We may be approaching a point, where some things "act up again",
just after the "trauma of 2.2TB" had passed :-)
Anyway, this little investigation makes me wonder what the OS
did with my backup drive. I had a few problems with partitioning it,
and may have stepped into this issue, without recognizing what
was going on. The OS is pretty crafty, and if you offer a large
volume (in modern Win10), it just restricts your cluster size
choices and does not offer any info you might benefit from,
for planning purposes. The above 512TB volume is using 128KB clusters,
as a guess as to how that worked out.
Size: 82.110 bytes
Size on disk: 131,072 bytes <== 128KB cluster, can't work with Win7
Paul

WIWAL, as a "HelpDesk Consultant"! some dept wanted to store huge number
of I dunno, some things, trouble was that they ran out of space much faster
than they imagined as every file on the Big Disk was say c. 1kB of
text, but the cluster size was (again making it up) c. 16kB.

[Paul]

Post by Kerr-Mudd, John
Note: xposted to afc
On Wed, 22 May 2024 09:23:02 -0400

Post by Paul

Post by Char Jackson

Post by c***@nycap.rr.com

Post by Paul
Windows 7 supports GPT partitioning, which removes the 32-bit MBR limitation
when defining storage. If you were using MSDOS partitioning on Windows 7, that
has a 2.2TB limit. When you buy a big drive, you use GPT so that all of
the disk can be used without a problem.

Is it possible to have both types of hard drive
partitioning on your system, GPT drives and MBR drives and your modern
BIOS would accomodate both and Windows Explorer would just 'see' both
drives without complaint?

Yes, absolutely.

It's nice when stuff works :-) That's for sure.
*******
The next area of interest, coming up, is NTFS only has four billion clusters.
2^64 clusters − 1 cluster (format); [No idea what this means, 2^64 is a CDB limit maybe]
256 TB size − 64 KB clusters ( early Windows 10 or less )
8 PB − 2 MB clusters ( late Windows 10 had more cluster sizes added, NotBackwardCompatible )
( if you show this partition to Win7, it "offers to format it", out of spite )
From this, we might conclude
16 TB size - C: drive Windows 10 install wants only 4 KB clusters (enables compression)
- It's possible Win7 might still accept 64 KB clusters for a Windows Install.
As a test of VirtualBox, I tried to create a large disk using a .vdi container.
This is the best I could do. In the Properties pie chart area for F: ,
File Explorer was unable to work out how much free space there was,
whereas good old Command Prompt worked it out.
F:\>dir
Volume in drive F is BIG
Volume Serial Number is EE25-2764
Directory of F:\
05/22/2024 07:21 AM 82,110 test.txt
1 File(s) 82,110 bytes
0 Dir(s) 562,949,287,706,624 bytes free
F:\>
We may be approaching a point, where some things "act up again",
just after the "trauma of 2.2TB" had passed :-)
Anyway, this little investigation makes me wonder what the OS
did with my backup drive. I had a few problems with partitioning it,
and may have stepped into this issue, without recognizing what
was going on. The OS is pretty crafty, and if you offer a large
volume (in modern Win10), it just restricts your cluster size
choices and does not offer any info you might benefit from,
for planning purposes. The above 512TB volume is using 128KB clusters,
as a guess as to how that worked out.
Size: 82.110 bytes
Size on disk: 131,072 bytes <== 128KB cluster, can't work with Win7
Paul

WIWAL, as a "HelpDesk Consultant"! some dept wanted to store huge number
of I dunno, some things, trouble was that they ran out of space much faster
than they imagined as every file on the Big Disk was say c. 1kB of
text, but the cluster size was (again making it up) c. 16kB.

Sounds like "someone didn't budget for enough disk drives" :-)

Disk drives are cheaper than consultants.

For me, this would be a research project. XFS-on-Dokan ? NTFS new compression ?
Storing <700 byte files in the $MFT 1KB slot ? I've had four million files
in a partition, where they were all stored in the $MFT. No clusters needed.

This sounds like a "somebody elses problem" type problem :-)

Stick a bunch of them in a .cab file, and use the .cab integration ?
(Haven't tested that. Don't particularly like .cab) Maybe the Federated
Search can look inside the cab, and then the Windows.edb file blows out.

I think Windows also has .tar integration now, if you want to test that.
Stick your small files in a tar file. Have the Federated Search index it.

So much untested stuff.

But I agree, that small files are not a lot of fun. They're a curse,
and no matter how fast your computer is, the results will not be pretty.

Paul

[Ahem A Rivet's Shot]

On Wed, 22 May 2024 13:27:14 -0400

Post by Paul
But I agree, that small files are not a lot of fun. They're a curse,
and no matter how fast your computer is, the results will not be pretty.

A battery backed RAM based FS could easily be optimised for small
files.

[Paul]

Post by Ahem A Rivet's Shot
On Wed, 22 May 2024 13:27:14 -0400

Post by Paul
But I agree, that small files are not a lot of fun. They're a curse,
and no matter how fast your computer is, the results will not be pretty.

A battery backed RAM based FS could easily be optimised for small
files.

It's the file system stack that's the issue.

This is what I use for storage testing. HDTune really isn't
meant for testing a thing like this, because the program
came from the HDD era. This is drive D: . Naturally, there
are PCIe Rev5 NVMe faster than this.

[Picture]

Loading Image...

If I pick a bench from the collection -

4:53:30 4:58:09 Windows Defender running, writeidx3.exe 1048576 1048576 / 279 = 3758 files/sec

D:\TEMP

4.00 GB (4,294,967,296 bytes) 1048576 files, 4096 zeros each
1,048,576 Files, 69,906 Folders

delete rate via shift-delete, 6700 items per second

5:04:30 5:06:05 WD off, writeidx3.exe 1048576 1048576 / 95 = 11038 files/sec

Last files written: D:\TEMP\0\0\0\F\F\F\F\000FFFF0.txt .. 000FFFFF.txt

File creation would go slower, if they were all in the same directory (flat test).

On TMPFS on Linux, I've hit 186,000 created files a second,
for comparison. The only problem with TMPFS, is the performance
does not stay that fast forever. It's only good after a fresh boot.
But if you need to bench to make impressive figures, that is the
better way to do it. And that's not a recent TMPFS test, so it's
not really fair to Linux. The other result, is fresh off W11.

Paul

[Kerr-Mudd, John]

On Wed, 22 May 2024 13:27:14 -0400

Post by Paul

Post by Kerr-Mudd, John
Note: xposted to afc
On Wed, 22 May 2024 09:23:02 -0400

Post by Paul
Windows 7 supports GPT partitioning, which removes the 32-bit MBR limitation
when defining storage. If you were using MSDOS partitioning on Windows 7, that
has a 2.2TB limit. When you buy a big drive, you use GPT so that all of

[]

Post by Paul

Post by Kerr-Mudd, John
WIWAL, as a "HelpDesk Consultant"! some dept wanted to store huge number
of I dunno, some things, trouble was that they ran out of space much faster
than they imagined as every file on the Big Disk was say c. 1kB of
text, but the cluster size was (again making it up) c. 16kB.

Sounds like "someone didn't budget for enough disk drives" :-)
Disk drives are cheaper than consultants.

This was way back - When I Was A Lad - the Big Disk was a big spend.
around 1986?

Post by Paul
For me, this would be a research project. XFS-on-Dokan ? NTFS new compression ?
Storing <700 byte files in the $MFT 1KB slot ? I've had four million files
in a partition, where they were all stored in the $MFT. No clusters needed.
This sounds like a "somebody elses problem" type problem :-)
Stick a bunch of them in a .cab file, and use the .cab integration ?
(Haven't tested that. Don't particularly like .cab) Maybe the Federated
Search can look inside the cab, and then the Windows.edb file blows out.

pre-cab

Post by Paul
I think Windows also has .tar integration now, if you want to test that.

pre-Windows also I think.

Post by Paul
Stick your small files in a tar file. Have the Federated Search index it.

They had bought a Program that saved everything into these small individual
files - horrible, I know.

Post by Paul
So much untested stuff.
But I agree, that small files are not a lot of fun. They're a curse,
and no matter how fast your computer is, the results will not be pretty.
Paul

[Lawrence D'Oliveiro]

Post by Paul
But I agree, that small files are not a lot of fun. They're a curse,
and no matter how fast your computer is, the results will not be pretty.

Linux copes with them quite nicely, as long as you don’t try to use some
GUI tool to view the resulting directory. In the early days of online
video, I did a system for a client which would split the raw camera
footage into individual JPEG frames, from which bespoke movies could be
automatically generated for customers according to a template. You could
easily end up with 100,000 video-frame files in a single directory.

Another aspect in which NTFS is showing its age?

[Paul]

Post by Lawrence D'Oliveiro

Post by Paul
But I agree, that small files are not a lot of fun. They're a curse,
and no matter how fast your computer is, the results will not be pretty.

Linux copes with them quite nicely, as long as you don’t try to use some
GUI tool to view the resulting directory. In the early days of online
video, I did a system for a client which would split the raw camera
footage into individual JPEG frames, from which bespoke movies could be
automatically generated for customers according to a template. You could
easily end up with 100,000 video-frame files in a single directory.
Another aspect in which NTFS is showing its age?

They did try to spin another file system, and they
weren't happy with the results. So what are you
going to do ?

It has a lot of "features". An IT person once tried to
explain how permissions work, and there must have been
fifty pages of information on the web site. And near the
end, the guy said "there are a couple more features,
but nobody has likely ever heard of them, so there is
no point in writing them up". It's like a bottomless
pit full of handcuffs.

Why should a thing like that be fast ? If you change anything
on it, you'd hear "you broke my whatsit, how dare you". As
a result, you cannot exactly simplify it. There could be
a paying customer handcuffed to it.

And a particular test case to break it, only requires the
transfer of 50-60GB of data, after which the message
"insufficient system resources" appears on the screen.
That is just one reason for inventing a second compression
feature for the file system ("New Compression"). I assume it
passes the test case, or they wouldn't have created it. There
are two types of compression, plus there are reparse points
to add "custom features" on top of the "standard" they've created.

Paul

[Lawrence D'Oliveiro]

Post by Lawrence D'Oliveiro
Another aspect in which NTFS is showing its age?

They did try to spin another file system, and they weren't happy with
the results. So what are you going to do ?

You mean ReFS?

At least Windows NT could have had a virtual filesystem layer, like Unix
had as far back as the 1980s, before NT was even thought of. Linux also
has this, and this allows it to support such a great variety of
filesystems.

And a particular test case to break it, only requires the transfer of
50-60GB of data, after which the message "insufficient system resources"
appears on the screen.

Interesting that the problems with copying large numbers of files seem to
be more due to limitations in Windows itself, rather than NTFS: Linux
itself can do those things on NTFS just fine
<https://www.theregister.com/2010/09/24/sysadmin_file_tools/>.

[Andy Burns]

Post by Lawrence D'Oliveiro

They did try to spin another file system, and they weren't happy with
the results. So what are you going to do ?

You mean ReFS?

ResFS, it dropped a few features (8.3 filenames, compression) and adds a
few (disk scrubbing to proactively detect bad sectors, fixing volumes
while online, rather than offline chkdsk)

I've used it on a few hyper-V servers, where the SAN has gone for cost
reasons and storage is all local on the server with h/w RAID)

[Paul]

Post by Lawrence D'Oliveiro

Post by Lawrence D'Oliveiro
Another aspect in which NTFS is showing its age?

They did try to spin another file system, and they weren't happy with
the results. So what are you going to do ?

You mean ReFS?
At least Windows NT could have had a virtual filesystem layer, like Unix
had as far back as the 1980s, before NT was even thought of. Linux also
has this, and this allows it to support such a great variety of
filesystems.

And a particular test case to break it, only requires the transfer of
50-60GB of data, after which the message "insufficient system resources"
appears on the screen.

Interesting that the problems with copying large numbers of files seem to
be more due to limitations in Windows itself, rather than NTFS: Linux
itself can do those things on NTFS just fine
<https://www.theregister.com/2010/09/24/sysadmin_file_tools/>.

One way to move files from one place to another like that is: Macrium.

Or for that matter, any of 20-30 backup/clone/restore utilities.

They move files at the cluster level.

When I attempt pathological cases here, for test, I store them using
Macrium so that it is easier (and faster) to reconstitute later.

Robocopy has retry capability, so if the problem was hardware related,
you might get a complete set that way. It also keeps a log, and you can
see which files it had trouble with.

It's even possible the source drive needed a CHKDSK. Or, there was
a bad block in there. With that many files in there, you'd want to be
careful about *which* version of CHKDSK you ran. I would want to know
your "backup status", before suggesting such a thing. CHKDSK, like
an fsck, can destroy a partition, and if "someone elses data" is involved,
that's the first thing I want to know, is how prepared you are for a disaster.

Macrium can also be used for defragmentation, if you restore a "larger"
partition to a "smaller" space, on purpose. It, in effect, switches
to file-by-file restoration, and the fragmentation is significantly
reduced. You would backup to a solid file, carry the solid file to
the second machine, then do the restore there.

60 million files is a lot of files. I can well imagine the challenge
of doing that. Not easy. A scaling problem. And the Wikipedia idea that NTFS
can handle 4 billion files (via some sort of "theoretical calculation") is
absurd. The thing would slow down so much, it would be heat death
of the universe before it finished. But as a technical person,
I would want to record the reason for each of these failures during
the 60 million copy, to see if we can really blame some aspect of
the implementation, versus some other reason. If you switch to a
cluster-level copy, it's an entirely different experience.

I hope this wasn't their backup strategy :-) The file-copying idea.
One reason I don't do things that way, is that's hard on the disk drive
if using conventional HDD. That's the impetus to do cluster-level transfer.
A cluster-level transfer, moves sequentially over the surface of the
disk, and the only "thumping" from the drive, is due to the use
of NTFS TXF (transactional NTFS). We don't control that, it's the
developers of the backup program who use that. But it's only two long
seeks per second, versus the "quiet" data transfer process.

Conventional HDD wear out in about a year, if you thrash them constantly.
I've not seen a post-mortem analysis as to whether the ribbon to the
head snapped, or what happened to the device.

Robocopy started out as a project by one employee on the side, and
today, it is an officially supported program shipped with the OS.
If it is having trouble with such a copy operation, you can use
the Feedback Hub and report a bug against it.

Paul

[Lawrence D'Oliveiro]

60 million files is a lot of files. I can well imagine the challenge of
doing that. Not easy. A scaling problem.

And yet rsync handled it all without skipping a beat.

[Ahem A Rivet's Shot]

On Thu, 23 May 2024 07:49:48 -0000 (UTC)

Post by Lawrence D'Oliveiro

60 million files is a lot of files. I can well imagine the challenge of
doing that. Not easy. A scaling problem.

And yet rsync handled it all without skipping a beat.

Yes rsync is a very well written piece of code.

[Scott Lurndal]

Post by Ahem A Rivet's Shot
On Thu, 23 May 2024 07:49:48 -0000 (UTC)

Post by Lawrence D'Oliveiro

60 million files is a lot of files. I can well imagine the challenge of
doing that. Not easy. A scaling problem.

And yet rsync handled it all without skipping a beat.

Yes rsync is a very well written piece of code.

It is worth pointing out that this thread is cross-posted to
the windows7 newsgroup. Windows filesystems basically suck.

[Lawrence D'Oliveiro]

Post by Scott Lurndal
Windows filesystems basically suck.

They do, but even in spite of that, Linux seems to do a better job of
managing them than Windows can manage.

[Paul]

Post by Lawrence D'Oliveiro

60 million files is a lot of files. I can well imagine the challenge of
doing that. Not easy. A scaling problem.

And yet rsync handled it all without skipping a beat.

This is my test log.

D:\TEMP>writeidx4 67108864 # These files are very small. My storage device is the limitation. All files in $MFT.
Writing 67108864 files
Start time (epoch seconds) 1716478386
Stop time (epoch seconds) 1716483434
Total time 5048
Done 67108864 files

Used space 86,366,027,776 7.45GB left

writeidx4-64million-Ddrive-736391-00-00.mrimg 2,381,377,130 bytes

It took Macrium 7 minutes to scan the file system and "make a list"
of the 60+ million files (it's future-proofing, for when it is
asked during a restore, to resize the partition). And then it took
only 3 minutes to write the clusters. It can't go faster than around
300MB/sec because it computes hashes as it goes. The file is protected
by hashes, to detect corruption.

I don't think I have anything else even remotely close as a recipe.
That's basically handling 100,000 files a second.

Yes, NTFS works. A little bit :-) It's like a premium piece of
swampland in Florida, above water at least twice a year.

And we can thank the availability of larger RAM sticks,
for making this project worth trying.

Just about any other interaction with the file collection,
takes too long.

Paul

[j***@astraweb.com]

Post by Lawrence D'Oliveiro

Post by Paul
But I agree, that small files are not a lot of fun. They're a curse,
and no matter how fast your computer is, the results will not be pretty.

Linux copes with them quite nicely, as long as you don’t try to use some
GUI tool to view the resulting directory. In the early days of online
video, I did a system for a client which would split the raw camera
footage into individual JPEG frames, from which bespoke movies could be
automatically generated for customers according to a template. You could
easily end up with 100,000 video-frame files in a single directory.
Another aspect in which NTFS is showing its age?

And I would bet a dollar to a donut that someone, somewhere, has either created or modified a file
manager so that it will collapse files in a directory that meets a literal file portion in a user
defined template -- maybe in the style: <template>. * (9610)
Or, alternatively, there are at least two that will display only files meeting a user defined template
in either xtree or ztree -- the "Filespec" function (F).

--

jim

[Paul]

Post by j***@astraweb.com

Post by Lawrence D'Oliveiro

Post by Paul
But I agree, that small files are not a lot of fun. They're a curse,
and no matter how fast your computer is, the results will not be pretty.

Linux copes with them quite nicely, as long as you don’t try to use some
GUI tool to view the resulting directory. In the early days of online
video, I did a system for a client which would split the raw camera
footage into individual JPEG frames, from which bespoke movies could be
automatically generated for customers according to a template. You could
easily end up with 100,000 video-frame files in a single directory.
Another aspect in which NTFS is showing its age?

And I would bet a dollar to a donut that someone, somewhere, has either created or modified a file
manager so that it will collapse files in a directory that meets a literal file portion in a user
defined template -- maybe in the style: <template>. * (9610)
Or, alternatively, there are at least two that will display only files meeting a user defined template
in either xtree or ztree -- the "Filespec" function (F).
--
jim

D:\TEMP>dir /s 0 # 3,885,286,833 bytes worth of file listing...

Directory of D:\TEMP\0\0\3\F\F\F\F

05/23/2024 12:57 PM <DIR> .
05/23/2024 12:57 PM <DIR> ..
05/23/2024 12:57 PM 7 000003FFFF00.txt
...
05/23/2024 12:57 PM 7 000003FFFFFF.txt <=== the 67108864th or so file (256 files per bottom level directory)
256 File(s) 1,792 bytes

Total Files Listed:
67108864 File(s) 469,762,048 bytes
838865 Dir(s) 7,819,739,136 bytes free

Some things which are using GetNextFile, they
smoke File Explorer and make File Explorer look stoopid.

D:\TEMP>dir /s 0 | D:\pigz.exe -c > d:\dir-out.txt.gz

The faster case, is when you have a large number of
files in a single folder. dir can do things that
just kill File Explorer.

File Explorer, at least on one machine with plenty of RAM,
stops malloc'ing at around 15GB, while trying to "explore"
a large flat directory, and the "wheel of death" just
spins forever. I think it's neat that somebody slapped
a quota on that puppy. I wanted it to consume all the memory
on the machine, because that's what the memory is for.

Paul

[Lawrence D'Oliveiro]

Post by j***@astraweb.com
And I would bet a dollar to a donut that someone, somewhere, has either
created or modified a file manager so that it will collapse files in a
directory that meets a literal file portion in a user defined template

Some filesystems have support for “tail packing”. That is, that last
sector that is not completely filled by the last part of one file is
shared among multiple files. Being done in the filesystem, this is
completely transparent to applications.

[Carlos E.R.]

Post by Lawrence D'Oliveiro

Post by j***@astraweb.com
And I would bet a dollar to a donut that someone, somewhere, has either
created or modified a file manager so that it will collapse files in a
directory that meets a literal file portion in a user defined template

Some filesystems have support for “tail packing”. That is, that last
sector that is not completely filled by the last part of one file is
shared among multiple files. Being done in the filesystem, this is
completely transparent to applications.

Reiserfs.

Which behaves very well with small files, millions of them in a single
direrctory.