why does it take longer to transfer hundreds of files totalling 4 gigabytes than transferring one big 4 gigabyte file?

16 views
0

why does it take longer to transfer hundreds of files totalling 4 gigabytes than transferring one big 4 gigabyte file?

In: Technology

Imagine opening hundreds of containers, pouring them into other containers you just made, and closing them – versus doing that with one big container. The making, opening, and closing takes time and thought.

Think of each file like a book in a library. The computer needs to make sure the book is there, find the book, take the book to a table, sit down, open the book, flip through the book, close the book, stand up, walk back to where the book was found and then start again for each book. Now think about if the computer is strong enough and if a single book contained all of those smaller books, he could simply do the same activity just once.

Overhead

Each individual file is not only the data it contains but also it’s corresponding entries in the NTFS file table.

If you think of a hard drive like a library, then the files are books, and the FAT (File Table) is the index that tells you where to find these books.

Every time you write a file you have to update the index as well, and that adds overhead. So imagine being in a library and having to put a couple hundred books back on the appropriate shelves. You have run back and forth between the shelves and the index, putting the books away and then updating the index. It’s not very efficient. Vs putting a single set of encyclopedias back on the shelf in order.

This is worse on spinning hard drives because the hammer has to move around a lot to access the index, then write the file. Each move of the hammer is time spent not writing data to disk, that’s latency and it adds up.

On top of that the built in Windows File transfer utility doesn’t handle large file transfers well, so the amount of time it takes to move data increases on a curve instead of a straight line.

When I’m migrating a lot of data in the server world I use a tool call Teracopy (which is free for home users) that replaces the built-in Windows File utility for this very reason.

Each file has some additional metadata. It’s location, file size, address of contents, last modified time, etc. This scales with the number of files.