I’ve been on Windows 7 since the RC was released — that coupled with the fact that I’m a blogger makes me an expert, you should trust me and take my word on everything unconditionally — and while I’ve been overall pretty happy (20 tips and tricks, 8GB performance) there have been a few niggles here and there (Windows rot).
One of the more prominent ones I’ve noticed is the poor performance when dealing with large amounts of small files. Before Windows 7 was officially released, generally slower file operations (Copy, Move, etc.) were reported and to some extent fixed in the Beta 2 and RC releases, and a bit more with a follow-up hotfix to the public release. I don’t know if SP1 had additional fixes for file ops or not.
At the time I had an inkling that the performance issues weren’t so much that Microsoft’s device drivers had gotten worse, but that too much work was being done to copy, move or delete files. The most likely culprit was updating the system-wide search index with the second culprit being Shadow-Copied files needing to have their revision records updated.
Given that Shadow Copy is optional and must be turned on, I looked at indexing first.
ASIDE: As an Eclipse Java PM in my past life, I had workspaces that contained 100s of thousands of .java source files, so trying to erase or move these took hours — I decided it was long enough to investigate the issue and try and trim down the time.
I first ran into this problem when I tried to delete a set of workspaces I had on my hard drive. Executing the Delete operation hung at the “Calculating…” discovery step of the delete for more than 10 mins before I tried to figure out what was going on:
No files were getting deleted, the damn thing was just counting the files it would delete in a day or two once it was done counting.
I popped open the properties on the parent directory to see what the hold-up was and found out it was pretty huge:
So I did some digging online and found folks suggesting that you disable ‘Indexing’ on directories with tons of files that you want to perform a mass-operation on. You can do that from the Advanced… menu:
This operation itself took about an hour to apply, but given that I was looking at a 6hr delete for the previous operation, it seemed to be worth it:
My guess is that Windows 7 went through and updated the NTFS metadata on every single file contained in that parent directory to flip the switch on indexing. Not sure what Mac or Linux/EXT4 or Sun/ZFS would do in this case or if properties can be hierarchical, but that seemed like a very 1990s way of adjusting file properties.
Once the operation had finished I re-executed the Delete operation on the parent folder and saw the “Calculating…” finish much faster this time, maybe about 3 or 4 mins. The delete operation itself was scheduled to finish in under 30 mins, quite a bit different from the previous estimate of “LOL, you will die waiting” that Windows 7 had given me before:
As far as I know this tip effects directories with tons of files, not necessarily directories with very large files.
If you guys have any of your own favorite Windows 7 tips, let us know!