Disk Provisioning and storage demystification

CocCoc Techblog
May 23 · 5 min read
Image for post
Image for post

Foreword

Many of us have heard of Thin/Thick provisioning — the very well-known features advertised widely on the market. In this blogpost we’re not only going to explain what are they, but we will dig deeper into what’s going on in the system for the features, with a few simple lines of C code, by our own hands! There’re even some interesting facts about the storage that we’re using day by day. Let’s go on.

For system admins, devops, developers… And all of those who are studious, curious and want to learn more about small things working under the hood.

First, What is Disk Provisioning?

Imagine, when you want to create a Virtual Machine (VM) with a 50GB disk, the VM service (KVM / VMware…) will create you a box, with requested specifications: CPU, memory, disk… Providing CPUs to the VM is fast, so is memory. What about disk? Depending the mechanism that the service is using, it will give you either choice:

  • A disk image that takes whole 50GB preallocated in the host OS
  • A disk image that takes a very small initial amount of space (say, 50MB), and claimed to be grown upon demand until the maximum allocated capacity (50GB)

So basically the above situation is the example of the differences between Thick — the former and Thin — the latter. The ideas are simple, aint they? Each mechanism comes with pros and cons, simply:

  • Thin provisioning is quick for create, and it can save a lot of space, for what is called “Storage overcommitment” — which is not very common, but if you’re already familiar with Memory overcommitment, then the idea is literally the same. Supposingly the host machine has only 100G of disk, but because of thin provisioning, it can allocate 10 VMs with 50G disk each, with the hope of not many VMs use all the disk space they’re allocated. This feature has been working perfectly since it’s introduced, and it is set default for many services that have storage provisioning: Ceph, KVM…
  • Thick provisioning cannot do like that, and continue with above example, the host is able to barely allocate 2 VMs, which will be overly underutilized in term of storage usage for most situations. It is, however, better in term of performace and security in comparision with the counterpart.

We’re not going to discuss this technique in a very detailed manner, but there’s infomation necessary to understand before we can go further.

  • For thick provisioning, all the preallocated space will be zero’ed out at a time of creation so new data is ready to be written right after that — This accounts for why disk space initialization is so slow but having better performance — it doesnt need zero’ing out disk blocks any more.
  • For thin provisioning, the demanded capacity is only registered to the OS, no data is actually consumed. Whenever new data is written, the OS still needs to zero’ed out the space beforehand.

Enough theory, now we’re gonna have our hands a bit dirty to demonstrate the ideas of Storage Provisioning.

Experiments

Thick Provisioning

Let’s try with thick provisioning first, the most primitive technique

#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>
#include <string.h>
int main() {
// Let's say we want to create a 500M disk
int capacity = 1024*1024*500;
int fd = open("thick.disk", O_RDWR | O_CREAT | O_TRUNC, 0664);
// Create a big zero block for faster zeroing out
int buf_size = 4096;
char zero[1024];
memset(zero, 0, sizeof(zero));
// Zeroing out the entire space
for (int i=0; i<capacity/buf_size; i++) {
// Write "0"
write(fd, zero, buf_size);
}
return 0;
}
# time ./thick
real 0m2.882s
# ls -lh
-rw-r--r-- 1 root root 500M May 23 15:53 thick.disk
# du -h thick.disk
500M thick.disk

As can be seen in this experiment, creating 500MB of diskspace took ~3s to complete. The outcome is very similar to what created by using dd command, as below. dd is even slower than ours because it needs read-then-write, while read is unnecessary in our example.

time dd if=/dev/zero of=thick.disk bs=4K count=128000
real 0m3.995s

Moreover, ls and du report the same size, so it can be sure that registered size == actually used size — what thick provisioning actually does.

Thin Provisioning

Making example of this technique is also pretty simple. As from definition, it doesnt zero out all the needed space, instead, it only registers the capacity it’ll be using in the near future. With that in mind, we can assume that du will not show the image takes any space. Let’s try

int main() {
// Let's say we want to create a 500M disk
int capacity = 1024*1024*500;
int fd = open("thin.disk", O_RDWR | O_CREAT | O_TRUNC, 0664);
// Get the pointer to the end of file, without zeroing out
lseek(fd, capacity, SEEK_CUR);
// Write only one character to the end
// for OS to know what's the end of the image
write(fd, "\0", 1);
return 0;
}
# time ./thin
real 0m0.004s
# ls -lh thin.disk
-rw-r--r-- 1 root root 501M May 23 16:26 thin.disk
# du -h thin.disk
4.0K thin.disk

See the differences? Thin provisioning does not actually take any time and space to create an image in comparision with thick one. While ls reports the image is ~500M, du says it takes only 4K on diskspace usage (1 block — 4K). The difference is because ls reports file info, while du shows real diskusage by default. The 2 tools agree with eachother if du uses the option “ — apparent-size

# du -h --apparent-size thin.disk
501M thin.disk

Basically we created a very big hole inside that image (with NULL terminated at the end) — which is called sparse file, and is used commonly in many situations, for example Kafka’s also ultilizing this technique for index files:

# ls -lh
total 4.0K
-rw-r--r-- 1 cp-kafka confluent 10M May 21 17:33 00000000000000240001.index
-rw-r--r-- 1 cp-kafka confluent 0 May 21 17:33 00000000000000240001.log
-rw-r--r-- 1 cp-kafka confluent 10M May 21 17:33 00000000000000240001.timeindex
-rw-r--r-- 1 cp-kafka confluent 13 May 21 17:29 leader-epoch-checkpoint
# du -sh
8.0K .

Storage Overcommitment

Let’s go beyond the provisioning a bit. We mentioned Storage Overcommitment earlier, but we havent experimented that. Let’s try creating a sparse file with very huge hole inside, even bigger than total space that the OS has. Is it possible?

/* long long is necessary, as int will be eventually overflowed
long long capacity = 1024LL*1024*1024*500; // 500G
# ls -lh
-rw-r--r-- 1 root root 501G May 23 16:57 thin.disk
# df -h /
/dev/vda2 99G 44G 51G 47% /

A 500GB image in a 100G system, what a surprise, isn’t it?

Conclusion

Storage is always one of the biggest as well as slowest part of any enterprise system that needs extensive care from system admins. Having more knowledge about it may open paths for performance tuning, which is critical for storage usage.

Understanding 2 types of Storage Provisioning and their pros and cons is quite fundamental nowadays, because many services are making use of these cool features. We now know when and why Thin provisioning should be used, so as Thick. Even more than that, we experimented it by our own hands in low level system, and understood some unexpectations with the tools we think we know them best, which are also kinda great to some extent.

The Startup

Medium's largest active publication, followed by +731K people. Follow to join our community.

CocCoc Techblog

Written by

From engineers who’re devoting their time to the Vietnamese browser and search engine.

The Startup

Medium's largest active publication, followed by +731K people. Follow to join our community.

CocCoc Techblog

Written by

From engineers who’re devoting their time to the Vietnamese browser and search engine.

The Startup

Medium's largest active publication, followed by +731K people. Follow to join our community.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store