Google Cloud Storage: Everything You Need to Know (Object, File, Block)
Data has become increasingly critical for any type of business. A company’s ability to compete largely determined by how well it leverages data to gain a competitive advantage. It is imperative that every business has a data storage and protection strategy.
Where do you store your data? And how do you secure the data? The answer to this problem lies in Google Cloud storage offerings.
The most common forms of data storage are object storage, archive storage, file storage, and block storage, and each type is fit for different purposes. Google’s storage service offerings have undergone significant diversification over time to meet requirements for a wide range of applications.
Virtually all use cases, including those with high-performance demands, data residency restrictions, ambiguous or shifting access patterns, or archival storage are covered by some or all of the storage products in google cloud kitty.
GCP Storage Suite
The same storage system that powers Google Cloud also underpins Google’s most popular products, supporting globally available services with more than billion users across the globe, like YouTube, Drive, Gmail, Photos, workspace suite of products, etc.
- Persistent Disk: Flexible, Durable Block Storage
- Google Cloud Storage: Scalable, Secure Object storage bucket
- Filestore: Simple, predictable, elastic file storage
Apart from the storage options mentioned above, GCP also offers various tools that can help facilitate transfer across different cloud providers and meeting the RPO/RTO requirements of enterprises for their HA/DR setup.
- Storage Transfer Service: Transfer data quickly and securely
- Google Cloud Backup and DR: Protect workloads running in GCP & on-prem
Persistent Disk
Google Persistent Disk is a block storage service provided by Google Cloud Platform. It allows you to create and attach disks to virtual machine instances running in Google Cloud. These disks persist independently from the lifetime of the instance, allowing you to store data on the disks that can be used by multiple instances over time.
You can use Persistent Disk to store data such as application code, logs, and databases. Persistent Disk is designed for high performance, with the ability to read and write data concurrently. It is also designed for durability, with multiple copies of your data stored across multiple locations.
You can create Persistent Disks in a variety of sizes and configurations, including standard and SSD disks. You can also create disks with different performance tiers to match your workload requirements.
To use Persistent Disk, you need to create a disk and then attach it to a virtual machine instance. You can do this using the Google Cloud Console, the gcloud command-line tool, or the Compute Engine API. Once the disk is attached, you can use it like any other disk on the instance. You can either create a blank persistent disk, or create a disk from a data source. You can use a persistent disk as a boot disk for a virtual machine (VM) instance, or as a data disk that you attach to a VM.
Key Features
- High-performance block storage for any workload Persistent Disk performance scales with the size of the disk and with the number of vCPUs on your VM instance.
- Durability and availability that keep your business running Persistent Disks are designed for durability. It automatically stores your data redundantly to ensure the highest level of data integrity.
- Automatic security and encryption Automatically encrypt your data before it travels outside of your instance to Persistent Disk storage. Each Persistent Disk remains encrypted with system-defined keys or with customer-supplied keys. Google distributes Persistent Disk data across multiple physical disks, ensuring the ultimate level of security. When a disk is deleted, the keys are discarded, rendering the data irretrievable.
Use Cases
- Databases and SAP workloads — Connect to VM’s as block storage
- Boot images
- Google Compute (GCE) and Kubernetes Engine (GKE)
- High-Performance Computing
Pricing
Standard, SSD, and balanced persistent disks are priced by the amount of provisioned space per disk. For these disk types, I/O operations are included in the price for provisioned space. Since disk performance grows linearly with the size of your disk, consider your I/O needs when choosing the size of your disk.
Extreme persistent disks are priced by the amount of provisioned space per disk and the number of provisioned IOPS per disk.
After you successfully delete a persistent disk, you are no longer charged for that disk.
Google Cloud Storage
Cloud Storage is a managed service for storing unstructured data. Store any amount of data and retrieve it as often as you like. Cloud Storage provides 11 9’s durability by redundantly storing objects across multiple devices located in multiple availability zones and encrypts data in-flight and at rest to ensure its security, integrity, and privacy.
Cloud Storage offers high availability and performance, while being strongly consistent. To use Cloud Storage, you’ll first create a bucket, a basic container that holds your data in Cloud Storage. You’ll then upload objects into that bucket — where you can download, share, and manage objects. You can use Cloud Storage to host websites, store data for analytics and machine learning, or back up your data for disaster recovery.
To store data in Cloud Storage, you can either use the Cloud Storage web application or the Cloud Storage API. Once your data is stored in Cloud Storage, you can access it from anywhere with an internet connection and use it in your applications or websites. You can also set permissions on your data objects to control who can access it.
Key Features
- Automatic storage class transitions With features like Object Lifecycle Management (OLM) and Autoclass you can easily optimize costs with object placement across storage classes.
- Continental-scale and SLA-backed replication Industry leading dual-region buckets support an expansive number of regions. A single, continental-scale bucket offers nine regions across three continents, providing a Recovery Time Objective (RTO) of zero.
- Fast and flexible transfer services Storage Transfer Service offers a highly performant, online pathway to Cloud Storage — both with the scalability and speed you need to simplify the data transfer process.
- Default and configurable data security Cloud Storage offers secure-by-design features to protect your data and advanced controls and capabilities to keep your data private and secure against leaks or compromises.
- Leading analytics and ML/AI tools Once your data is stored in Cloud Storage, you can easily plug into Google Cloud’s powerful tools to create your data warehouse with BigQuery, run open-source analytics with Dataproc, or build and deploy machine learning (ML) models with Vertex AI.
Use Cases
- Use Cloud Storage for backup, archives, and recovery.
- Media content storage and delivery — Stream audio or video directly to apps or websites with Cloud Storage’s geo-redundant capabilities.
- Data lakes and big data analytics — used for collaboration and version control for your code.
- Machine learning and AI — Once your data is stored in Cloud Storage, take advantage of options for training deep learning and machine learning models cost-effectively.
- Host a website.
Storage classes for Cloud Storage Buckets
The Standard storage class provides the highest level of availability and performance and is especially well-suited for frequently accessed or short-lived data.
Nearline offers fast, low-cost, and highly durable storage designed for infrequently accessed data, including long-tail content.
Coldline offers the benefits of Nearline storage while optimizing for colder data use cases, such as disaster recovery.
Archive storage Archive storage is the lowest-cost, highly durable storage service for data archiving, online backup, and disaster recovery. Unlike the “coldest” storage services offered by other Cloud providers, your data is accessible within milliseconds, not hours or days.
Cloud Storage is ideal for a wide range of storage needs — from serving hot analytics data to archiving cold regulatory compliance data — through its flexible storage classes
Pricing
All storage classes have no minimum fee and offer pay-as-you-go capacity pricing. Object lifecycle management tools allow you to reduce costs further by scheduling deletions and transitioning objects to cooler storage classes as needed.
Cloud Storage pricing is based on the following components:
- Data storage: the amount of data stored in your buckets. Storage rates vary depending on the storage class of your data and location of your buckets.
- Data processing: the processing done by Cloud Storage, which includes operations charges, any applicable retrieval fees, and inter-region replication.
- Network usage: the amount of data read from or moved between your buckets.
Filestore
Most data on-premise is stored in file systems, as applications migrate to the cloud, their need for File storage doesn’t change, so file storage is critical to enterprise lift-and-shift and infrastructure modernization. Files are a useful abstraction, and even born-in-the-cloud applications are leveraging files.
Google Filestore is a fully managed file storage service that enables you to store and access file data on Google Cloud. It is designed for use cases that require fast, scalable, and low-latency access to file data, such as shared file systems and home directories.
Filestore offers three performance tiers: Standard, High Performance, and Extreme Performance. The performance tier you choose depends on your workload requirements and the type of file data you are storing.
To use Filestore, you create a Filestore instance and then mount it to a Compute Engine virtual machine instance or a Kubernetes Engine cluster as a file system. You can then use the Filestore instance like any other file system on your instances.
Filestore is integrated with other Google Cloud services, such as Compute Engine, Kubernetes Engine, and Cloud Load Balancing, allowing you to easily scale and manage your file storage needs.
Filestore enables application migration to the cloud without requiring you to rewrite or re-architect, accelerating and simplifying your migration. Filestore supports applications that expect to use standard files and directories. If your apps are running on Windows or Linux, most likely they rely on file storage behind the scenes.
Key Features
Scales to meet the needs of high performance workloads Filestore offers low latency storage operations for applications. For workloads that are latency sensitive, like high performance computing, data analytics, or other metadata intensive applications, Filestore supports capacity up to 100 TB and throughput of 25 GB/s and 920K IOPS.
99.99% regional availability SLA supports enterprise apps Filestore Enterprise is built for critical applications (e.g., SAP) requiring regional availability to ensure the applications are unphased in a zonal outage.
Protect your data with backups and snapshots Filestore offers instantaneous backups and snapshots to help you protect your data easily. Back up data and metadata of the file share, set up a regular backup schedule, or take snapshots of your instances anytime you need.
Support GKE workloads with Filestore For apps running in GKE that require file storage, the fully managed NFS solution supports stateful and stateless applications. With an integrated and managed GKE Container Storage Interface (CSI) driver, multiple pods can have shared file system access to the same data.
Use Cases
- Enterprise application migrations (SAP) — Filestore is built for critical applications requiring regional availability and having unstructured NFS data requirements.
- Financial services and technology — It is built for high performance computing (HPC) applications to access, sort, process, model, and deliver the right information to the right decision-maker at the right time.
- Media rendering — You can easily mount Filestore file shares on Compute Engine instances, enabling visual effects artists to collaborate on the same file share.
- Data analytics — Filestore enables immediate access to data for high-performance, smart analytics without the need to lose valuable time on loading and off-loading data to clients’ drives.
- Genomics processing — Genome sequencing requires an incredible amount of raw data, on the order of billions of data points per person. This type of analysis requires speed, scalability, and security. Filestore meets the needs.
- Web content management — Web developers and large hosting providers rely on Filestore to manage and serve web content, including needs such as WordPress hosting.
Pricing
Filestore pricing is based on the following elements:
- Service tier: Whether the service tier of your instance is Basic HDD (Standard), Basic SSD (Premium), Enterprise, or High Scale SSD.
- Instance capacity: You are charged for the allocated storage capacity, even if it is unused.
- Region: The location where your instance is provisioned.
Storage Transfer Service
Moving data is a challenge, it requires a lot of time and resources.
Google Storage Transfer Service is a fully managed service that enables you to transfer data from external data sources, such as on-premises storage systems or other cloud storage providers, to Google Cloud Storage. It is designed to make it easy to move large amounts of data to the cloud, with support for scheduling and automating transfers, monitoring transfer progress, and handling errors.
Storage Transfer Service can transfer data from the following sources:
- Local file systems
- Amazon Simple Storage Service (S3)
- HTTP/HTTPS locations
- Cloud Storage buckets
You can use Storage Transfer Service to transfer data on a one-time basis or set up recurring transfers to keep your data in the cloud up to date. You can also specify filters to include or exclude certain files or directories from the transfer.
To use Storage Transfer Service, you need to create a transfer job in the Google Cloud Console or using the Storage Transfer API. The service will then handle the transfer of data from the source to your Cloud Storage bucket. Complete transfers without writing a single line of code with Centralized job management to monitor transfer status.
Key Features
Reliable and secure data Transfer Migrate petabytes of data from on-premises sources or other clouds over online networks — billions of files and 10s of Gbps. Optimize your network bandwidth and accelerate transfers with scale-out performance.
Data encryption and validation Storage Transfer Service encrypts data in transit, supports VPC Service Controls, and uses checksums to perform data integrity checks, ensuring your data arrives intact.
Incremental transfer All transfers only move files and objects that are new, updated, or deleted since the last transfer, minimizing the amount of data that needs to be transferred.
Metadata preservation Storage Transfer Service offers control over preserving object and file metadata during transfer.
Use Cases
- Data center migration
- Content storage and delivery
- Disaster recovery, backup, and archival
- Analytics and machine learning
Pricing
Transfers using agents cost $0.0125/GB , You will be charged this fee in addition to charges associated with network usage, storage and other operations performed by the service at source and destinations on your behalf.
Backup and DR Service
Backup and disaster recovery are essential components of your business continuity plan. Cloud Backup & Disaster Recovery enables GCP customers to centrally manage backups and DR at scale via policies to ensure secure, reliable and efficient backups of VMs, databases and workloads running in GCP. Protect a broad spectrum of workloads and manage them from a central dashboard. Serves critical use cases such as recovery from data corruption, data loss, ransomware recovery, or database cloning for test/dev. Rely on application-consistent, incremental-forever backups. This service offers low Recovery Time Objective (RTO) and Recovery Point Objective (RPO). To implement a disaster recovery strategy with Google Cloud, you will need to define your recovery objectives and choose the appropriate tools and services based on your needs.
Key Features
Centralized backup management across workloads With Google Cloud Backup and DR, administrators can effectively manage backups spanning multiple workloads. Admins can generate application- and crash-consistent backups for VMs on Compute Engine, VMware Engine or on-premises VMware, databases (such as SAP, MySQL and SQL Server), and file systems.
Efficient use of storage to minimize costs Space-efficient “incremental forever” storage technology to ensure that you pay only for what you truly need. With “incremental forever” backup, after Google Cloud Backup and DR takes an initial backup, subsequent backups only store data associated with changes relative to the prior backup.
Minimal recovery times Google Cloud Backup and DR stores backup data in its original, application-readable format. As a result, backup data for many workloads can be made available directly from long-term backup storage.
Use Cases
- Provide recovery times aligned with your business
- Reuse cloud backups as clones of databases for test/dev/analytics purposes
- App consistent backups in native format
Pricing
Google Cloud Backup and DR Service is billed monthly, based on usage. There is flexibility built in to allow you to strike your desired balance between storage cost and data retention time. For example, when choosing to store backups on Google Cloud Storage, you can select an appropriate Cloud Storage class in alignment with your needs.
Comparison with other popular cloud platforms
All three major cloud providers — Amazon Web Services (AWS), Google Cloud Platform (GCP), and Microsoft Azure — offer a range of storage services for storing and managing data in the cloud. These services are designed to meet different needs and use cases, such as storing and accessing data for applications, storing and processing data for analytics, or storing data for backup and disaster recovery.
Here is a high-level comparison of the storage services offered by each of these cloud providers:
AWS
- Amazon S3: Object storage service that can be used to store and retrieve any amount of data, from anywhere.
- Amazon EBS: A block storage service that provides persistent storage for Amazon Elastic Compute Cloud (EC2) instances.
- Amazon EFS: A file storage service that provides scalable file storage for Amazon EC2 instances and on-premises systems.
- Amazon Glacier: A low-cost storage service for archival data.
Azure
- Azure Blob Storage: An object storage service that allows you to store and access large amounts of unstructured data, such as images, videos, and documents.
- Azure Files: A file storage service that allows you to create file shares that can be accessed from Azure VMs or on-premises systems.
- Azure Disk Storage: A block storage service that provides persistent storage for Azure virtual machines.
Each of these cloud storage services has its own unique features and pricing model, so it’s important to carefully evaluate your storage needs and choose the service that best fits your use case and budget.
Storage Options across GCP/AWS/Azure
Conclusion
Google cloud provides with comprehensive storage options to cater to different business use cases that cloud customers would have. I hope the blog provides you with decision capability to choose from these different available options to solve storage requirements for your respective use cases.
