Confusion About Disk Drive Space

The Setup & The Problem

You’re running out of storage on your computer for your photos, so you decide to buy an external disk drive to hold your new images. You start doing research on what’s size drives are available and how much they cost. Maybe you go back through your images to see how much space you are currently using and how many images you’ve been shooting each year.

You do the math to figure out how much space you should need for five years of images, plus some extra space, just in case you start shooting & keeping more. You figure out the average size of the raw image files your camera(s) produces. And since you will probably upgrade your camera sometime in the next 5 years and probably get a camera with a larger sensor that will increase the size of your raw and/or jpg files from the camera, you bump it up a bit more. The math says you will need a 4TB drive. The web says $200 dollars. You order one.

It arrives two days later. You unbox it and connect it to your Windows or Apple computer. You boot your computer and go into the file manager and look at the drives on your system and are immediately confused. The system shows the capacity of your drive as 3.63TB, with 3.63TB of free space.

Huh?

It’s a 4TB drive. The web site said it would hold 4TB of stuff. Why does the system say it’s only 3.63TB? Have I been ripped off? Who do I complain to? Should I demand my money back? Who stole 0.37TBs of my storage? That’s almost 10% of the space I thought I was buying!

The Answers

There are three answers to that last question: 1) simple math, 2) the file system, and 3) the glass is half empty.

Part 1

The math is a bit confusing when you first go through it, but when all is said and done, it’s really a matter of mathematical perspective. Let’s start with the operating system.

You’ve probably all heard of kilobytes (KB), megabytes (MB), gigabytes (GB) and terabytes (TB). Those equate to thousands, millions, billions and trillions. Kind of.

Internally, computers work in binary. A single character is stored in a byte of memory. That byte of memory is composed of eight bits. Each bit can be either on or off, which means a zero or a one. That means that one byte can hold 256 different values, zero (all bits off) to 255 (all bits on). 256 is 2x2x2x2x2x2x2x2, or more mathematically precise, 2⁸, or in words, 2 to the eight power.

Since the beginning of the universe (maybe just the beginning of computers), computers, specifically the operating systems that run them, have consider a kilobyte of storage to be 2¹⁰. That comes to 2x2x2x2x2x2x2x2x2x2, or 1024. Since they wanted 1KB to be as close to one thousand, but also be a power of 2, they went with 1024.

Using that logic, one megabyte is 2²⁰, or 1024×1024. That comes to 1,048,575. One gigabyte is 2³⁰, or 1024x1024x1024. That comes to 1,073,741,824. And a terabyte is 2⁴⁰, or 1024x1024x1024x1024. That’s a whopping 1,099,511,627,776.

But the operating system perspective is only part the story. The second portion is the disk drive manufacturers. They are humans. Most humans aren’t computer geeks. They may be familiar with kilometers and kilograms. A kilogram is 1,000 grams and a kilometer is 1,000 meters. So rather than using 1,024 when computing disk space, they use 1,000 for one kilobyte. Their math makes one terabyte 1000x1000x1000x1000, or 1,000,000,000,000.

To complete the math lesson, from a computers’ perspective, 4TB is 4,398,046,511,104 bytes and from the drive makers perspective, 4TB is 4,000,000,000,000 bytes.

Guess what 4,000,000,000,000/1,099,511,627,776 turns out to be?

3.637978807TB

Look familiar? If not, go re-read the third paragraph of this article.

Click to learn more

This is paragraph 1. This is paragraph 1. This is paragraph 1. This is paragraph 1. This is paragraph 1. This is paragraph 1. This is paragraph 1. This is paragraph 1. This is paragraph 1. This is paragraph 1. This is aragraph 1. This is paragraph 1. This is paragraph 1. This is paragraph 1.

Paragraph 2

qwd

That’s the math half of the answer. It’s a little strange and confusing, but now you understand why it starts with (significantly) less space than you thought it would.

Part 2

As you start to load your images to the drive, it seems to be filing up faster than you think it should. You’ve been shooting a lot lately and have loaded about 200GB of image files. But the free space on the drive is more than 200GB less than it was before you loaded the images.

And once again, you think, Huh?

The second of three parts was the file system.

Each disk drive if formatted by the operating system when it is installed. For Windows, there are several different formats that can be chosen. The two major ones are FAT32 and NTFS. FAT32 has been around wince Windows 98. NTFS is much newer, more resilient, more capable and more space efficient and it’s the one we will assume you are using for your new disk space.

In addition to storing the actual contents of files, a file system must keep track of the information about each of the files that are contained within that file system. Storing this information also takes up disk space. How much you ask? That’s a really tough one to answer. The easiest way to answer it is to say: it depends.

There is some basic space used to create some system tables, log files and a map of the entire drive showing which clusters have been used to hold files and which are available for future use. On a large disk volume in the 1-10TB range, that will normally consume between 200MB and 1GB. So basically, not enough to really even notice or care about. That’s good!

Next is the space for the file location and metadata information. For NTFS, this is the Master File Table, or MFT. Basically, the stuff that shows up in the file manager when you look at a directory. Including all of the stuff it doesn’t show by default, but you can get by adding it to the list of data items to show. Obviously, the more files and directories you have on a disk drive, the more space they will take up on the disk. Each MFT entry (file or directory) is about 1KB, so the MFT space for 1 million files & directories is roughly 1GB. Again, not a lot of space. Unless you start creating a lot of small files.

Will the space used for the MFT ever be as much as the space used for the data in the files? Almost never. The only time they should ever even come close to using the same amount of space is if you have a disk drive filled with millions of very small files. But even that isn’t always true. One of the features of the NTFS file system is that, to save space and speed up the system, if a file is just a couple of hundred bytes or less, the contents of the file are actually kept in the directory itself, rather than in a separate file. If you are using an NTFS file system, then all of those very small files won’t actually take up a ton of disk space. If you are using FAT32, you could be wasting a very significant amount of disk space. More importantly, you really shouldn’t be using FAT32 for any file system created in the last 10 years.

So, does the portion of the file system used to hold the file metadata and location information consume a lot of space? Not normally. But there is a catch. When the file system is initially created, NTFS will designate roughly 12.5% of the total space on the drive for the MFT. WHAT? That’s crazy. You would have to create tens of millions, or even hundreds of millions of files to need that much space for the MFT. The information stored in the MFT is critical to the functioning of your system and finding and reading the contents of your files, so NTFS wants to make sure it has room to grow into. However, once you have filled up the “normal” free space on the drive with files, NTFS will start using space initially designated for the MFT to hold regular files and directories. While that 12.5% is ‘designated’ as future MFT space, it is smart enough to start using it for files so it doesn’t just sit there with nothing in it.

Since we are trying to estimate how much space gets used for the MFT, this is good start:

If you will have mostly large files, such as videos, camera raw images & PSD files, Lightroom catalogues, backups: the NTFS overhead will be 0.1-0.3% of the total usable space on the disk.
If you are going to have millions of tiny files: the overhead can reach 2-5% of the total usable space on the disk
And if you are going to have a reasonable mixture of small, medium and large files: the overhead will be in the 0.5-1% range of the total usable space on the volume

Part 3

What did I mean when I said the third part is “the glass is half empty”?

On Windows, if you open the File Manager and right click on a directory, and select the Properties option, the system will read through the contents of the directory and produce a summary of the space used. It will show two numbers:

Size: The actual size of the contents of a file
Size on disk: How much space that file is actually using on the disk

To get a concrete example so explain this, I used two examples from my computer:

the c:\Users directory
A full drive of my images

	Example 1	Example 2
Size:	72.5GB / 77,872,567,003 bytes	3.22TB / 3,549,719,629,490 bytes
Size On Disk:	72.7GB / 78,100,729,856 bytes	3.22TB / 3,549,870,821,376 bytes)
Files / Directories:	124,422 files / 15,656 folders	73,879 files / 448 folders
Total diff in size:	0.2GB / 228,161,853 bytes	0.0TB / 151,191,886 bytes
Avg diff per file:	1,834 bytes	2,046 bytes

If you subtract the “size” from the “size on disk”, you get the total difference in size.

If you divide the total difference in size by the number of files, you get the average ‘wasted’ space per file.

Example 1 is a directory that contains lots of small files, some medium sized and a small number of large files. Example 2 is a directory that contains nothing but large files. The smallest files there are roughly 20-25MB with the largest being over 1GB.

What’s it using the extra space for? The simple answer is: nothing.

It all has to do with how the file system allocates space for each file. When a program starts writing data to a file, such as saving a new PSD file in Photoshop or exporting a JPG file in Lightroom or copying in a file from an SD card, the file system has no idea of how big this file will be once the application finishes writing the file and then closing it.

To make the allocation of space easier, the file system doesn’t just grab space in individual bytes. The overhead for that would be immense. Instead, it grabs space one cluster at a time as a file is created (or expanded). So how big is a cluster? The default cluster size for NTFS or 4K, or 4096 bytes. That means that the 4TB drive you purchased, has roughly 1 billion 4K clusters.

Since a cluster can only be used for one file, it means that the last cluster allocated to any file will, statistically speaking, only be half full. Or half empty. It doesn’t matter whether the file consists of a single cluster or a thousand clusters. The amount of ‘wasted’ space in a file only depends on how empty the final cluster is. If you have a reasonably large number of files, say more than a couple of tens of thousand, you should have a pretty even distribution of space in use from 1 to 4096 bytes in that last cluster. What’s half of 4,096 bytes? 2048 bytes. Almost exactly what the files in the two sample directories came out to be.

What does this mean for our 4TB disk drive? Well, if you have a lot of very large files, you won’t really be “wasting” a lot of space. However, the smaller the files stored on the drive, the higher the percentage of wasted space. If your average file size is 400,000 bytes, the wasted space is 0.5%. At 200,000 bytes, the wasted space is about 1%. But at 2000 bytes, the wasted space is close to 50%.

The good news is that even jpg files from current cameras typically run 15-35MB, so the wasted percentage is quite small.

Summary

When you look at the second and third parts of the answer above, they really don’t amount to a lot of storage used, except under some special conditions, primarily a large drive filled with very small files.

For example, let’s keep using that 4TB drive you bought above. If you fill it with 54MB files (roughly the size of the raw files produced by my current camera), you will be able to load approximately 77,283 of them to it before you run out of space. However, if you use 2048 byte files (the size of a large xmp sidecar file from Lightroom), you can load roughly 1,068,373,114 (yes that is slightly over 1 BILLION) of them before it is full. Actually, you really can’t fit that many, because a chunk of space will be consumed by the MFT with all of those file entries. But I’m trying to keep this simple.

Of course, generating 1 billion xmp files would require 1 billion files in a Lightroom catalog with xmp generation turned on. That would be a VERY large Lightroom catalog. I can’t imagine how long it would take to shoot & load that many images. As photographers, we all take a lot of pictures. However, if we included all of the images that everyone in the club has ever taken, we probably couldn’t come up with 1% of that number.

The Conclusion

Hopefully this article has cleared up some of the confusion when looking at the space on a disk drive. If it has made it more confusing, then I truly apologize.

P.S.

Do you want to be really depressed? A 10TB disk drive only gives you 9.09TB or usable space. Almost a whole terabyte of storage missing. Of course it can’t be missing, since it was really never there. Or was it?

P.P.S.

In part 1 of the answer section above, we spoke of kilobytes being either 1000 (decimal context) or 1024 (binary context). That’s not entirely accurate. A completely separate set of names and abbreviations was created for the binary set of numbers/names. The table below details both naming conventions. However, the use of the binary names has never really caught on, and even all of us computer geeks still use the decimal names rather than the binary names. Probably because if we did use the binary names, it would drive the rest of the world crazy trying to understand what we are talking about.

I included this table because you may run across these names while reading other web sites.

Decimal			Normal Math	Binary
Name	Abbrev.	Math Notation		Name	Abbrev	Math Notation
Kilobyte	KB	10³	Thousands	Kibibyte	KiB	2¹⁰
Megabyte	MB	10⁶	Millions	Mebibyte	MiB	2²⁰
Gigabyte	GB	10⁹	Billions	Gibibyte	GiB	2³⁰
Terabyte	TB	10¹²	Trillions	Tebibyte	TiB	2⁴⁰
Petabyte	PB	10¹⁵	Quadrillions	Pebibyte	PiB	2⁵⁰
Exabyte	EB	10¹⁸	Quintillion	Exbibyte	EiB	2⁶⁰
Zettabyte	ZB	10²¹	Sextillion	Zibibyte	ZiB	2⁷⁰
Yottabyte	YB	10²⁴	Septillion	Yobibyte	YiB	2⁸⁰
Ronnabyte	RB	10_²⁷	Octillion	Robibyte	RiB	2⁹⁰
Quettabyte	QB	10³⁰	Nonillion	No name yet		2¹⁰⁰

Racine Camera Club