Various Storage Services in AWS (or Cloud based Storage Solutions)

Various Storage Services in AWS (or Cloud based Services)

Before Discussing various storage solutions in AWS , we first need to understand the various type of storage options available on cloud:

There are majorly three types of storage options in the cloud: i.e.

1. Block Storage
2. File Storage
3. Object Storage

1. Block Storage:

Block storage stores data by dividing it into fixed size chunks known as blocks. This method is commonly used for various virtual machine storage and database systems. This storage supports low-latency access and high performance.

How block Storage works?

Data slicing: Any of your data (files, databases, etc.) is chopped up into equal sized blocks. Like that of dividing a long string into equal sized chunks.

Individual Storage: Each block is stored independently with a unique identifier. In this case the order of blocks doesn’t matter.

Direct access: Unlike file storage, which relies on a file system, block storage allows direct access to each block ,which supports quick retrieval of files . It is similar to a hard drive on a PC or Laptop.

Benefits of Block Storage

• It is Low Latency with high performance , suitable for applications that require fast and consistent I/O performance, like database, transactional workloads.
• These storage solutions are highly scalable, and hence can accommodate growing storage requirements, Cloud providers offer elastic block storage services that allow users to dynamically adjust the size and performance of storage volume as needed.
• These can be attached to VMs(Virtual Machines ) or server as raw storage devices.
• Data remains persistent even when servers or VMs are terminated or show down.
• They are high available devices and hence are considered having low downtime and low risk of data loss.

What is it used for?

• Databases: Databases rely on fast data access, making block storage a popular choice.
• Virtual machines: Block storage is commonly used to provide persistent storage for virtual machines.
• High-performance computing: Applications requiring fast data processing often leverage block storage.

e.g. Amazon Elastic Block Store (EBS) , Azure Disk Storage (Azure Managed Disks) , Persistent Disk (GCP) etc.

2. File Storage

File Storage consists of files inside a hierarchical folder system. In a hierarchy of various folders, files can be stored into various folders in any level. Hence, files will reside inside a folder and these folders will reside inside other folders. It’s like a hard drive in your computer. File storage is fast and accessible, but it doesn’t offer the increased potential for complex queries . These file storage systems have no issue in performance in the case of a small number of files, sorting or organizing but when the number of files grows large , organizing and finding specific files can be a real challenge and performance can go down.

Benefits of File Storage

• File storage allows multiple users and applications to access and modify files concurrently, allowing for better collaboration and helps team to work on a shared documents and other data with regard to block and storage object
• File storage offers advanced file system features such as directories, metadata, and access controls, making data organization easier.
• File storage works well with legacy apps and workflows, easing the transition to cloud storage.
• File storage provides detailed access controls, letting admins set precise permissions for users and group

What is it used for?

• File storage is used to store and serve multimedia content, documents, and web assets for content management systems
• File storage is used to manage documents , records and files in a central repository allowing easy collaboration among teams.
• File Storage is used for backing up critical files like databases ,application data and critical other files for disaster recovery and data protection
• File storage can be used to serve as a repository for source code, documentation and project assets in a software development team.
• It is used to serve and host multimedia files like audio, images,videos and so on that are needed to be streamed.
• The parallel access capabilities of EFS allow multiple analytics jobs and big data algorithms to access the data concurrently, improving processing speed.

E.g. AWS EFS , Cloud Filestore , Azure Files

3. Object Storage:

Object storage is a type of data storage architecture that manages data as object rather than earlier systems like files or blocks. It is designed to store and manage unstructured and static data like images, videos, audio files, documents , archives and so on. Instead of storing in hierarchical systems or blocks it stores each object in a flat address space, with metadata and a unique identifier.

Benefits:

• Object Storage scales effortlessly to accommodate growing data volumes without performance degradation
• They are highly durable as they are protected against hardware failures and data loss
• They are the most cost effective solution out of all other and offer pay as you go pricing
• Low administrative overhead and complexity as they are stored with metadata
• Seamlessly integrates with cloud native architectures and easy to integrate with applications and services
• They are equipped with robust access controls and encryption to ensure data security and compliance
• They are suitable for storing diverse type of unstructured data like images,videos, files, archives , logs etc

What is it used for?

• It is used in variety of use cases across industries and applications: Majorly it is used in
• Storing Backup and Archive for long term retention and disaster recovery
• Hosting and serving multimedia , contents such as images, videos websites content and digital medias
• Storing for analyzing large datasets as data lakes for data analytics, machine learning and artificial Intelligence applications and managing Used for storage of cloud native applications, Micro services
• Distribution of downloadable files, software packages and digital assets to end users.

E.g Cloud Storage in Google Cloud Platform, Azure Blob Storage in Microsoft Azure, and Amazon S3(Amazon Simple Storage Service)

Until now, we discussed through the overview of major storage mechanisms available in Cloud, Now we will in short discuss AWS focused infrastructures.

The three Storage mechanisms are termed as the following :

1. Amazon Elastic Block Store (AWS EBS)
2. Amazon Elastic File System (AWS EFS)
3. Amazon Simple Storage Service (AWS S3)

1. Amazon Elastic Block Store

According to AWS , EBS provides highly scalable high-performance block storage resources that can be used with Amazon Elastic Compute Cloud (Amazon EC2). It has a type like Hard drive and SSD in EC2.

Following block storage resources are managed by EBS:

• EBS Storage Volumes can be used as storage volumes in EC2 instances, i.e. like a harddrive on your PC. You can install anything that you do with your PC.e.g. To store files or to install applications
• Amazon EBS snapshots can be used to take backups of EBS storage volumes at any point of time, Which can be restored at a later date.

Features of Amazon EBS

• There are two types of volumes available in EBS i.e. SSD backed and HDD. You can use any type of EBS depending on your performance need i.e. SSD-backed storage is generally used for transactional workloads, and HDD-backed storage is used for throughput intensive workloads.
• You can use EBS as per your need of volume. When you need to change your Volume needs dynamically, you can even use Elastic EBS for tuning performance as well
• You can use EBS snapshots to backup data on EBS and restore them immediately or migrate across EC2 instance, Availability zone or even regions
• You can enable encryption in EBS to ensure security of data
• Highly available – 99.8% to 99.9% for general-purpose SSDs and 99.999% for the Provisioned IOPS SSD

Usage Examples of AWS EBS

• Ecommerce and other types of applications can store databases for quickly read/write operations.
• ERPs systems that require fast input and output operations can use EBS storage.
• EBS can be used as a data warehouse to store large volumes of transactional data for Business Intelligence applications
• EBS can be used as boot volumes for EC2 instances , it can help EC2 instances to boot quickly and maintain high performance.

Note: EBS is used in case of high-performance storage services for a single instance

2. Amazon Elastic File System (AWS EFS)

EFS is a fully managed cloud based file storage solution provided by AWS. It is serverless, fully elastic file storage so that you can share file data without provisioning or managing storage capacity or performance. EFS, unlike EBS, offers automatic scalability for storage, which makes it best-suited for workloads with unpredictable or fluctuating storage requirements without disturbing your application. Another important Feature of EFS is that it can be mounted to any number of instances while ideally EBS was connected to a single instance.

Features of AWS EFS

• The service is designed to be highly scalable, highly available, and highly durable across multiple AZ
• Amazon EFS provides high throughput, IOPS, and low latency needed for a broad range of workloads
• It is elastic and hence, it can grow to petabytes without needing manual configuration beforehand.
• EFS can be mounted from different EC2 instances, but it can also cross the AWS region boundary via the use of VPC peering
• EFS is highly secure as it is equipped with authentication, authorization and encryption capabilities, required for your security compliance
• Encryption in EFS is supported in two forms, i.e. encryption at rest and encryption at transit
• AWS EFS is not available for Microsoft Windows based EC2 instances.

Usage Examples of AWS EFS

• Software development and other teams working together can use EFS to share and access code files at same time.
• Content writers, editors and reporters can use it to access and edit media files simultaneously using EFS as a content management system.
• EFS supports machine learning models to access training data from multiple instances simultaneously speeding up the training process to run models concurrently
• IT can be used to efficiently deliver web services and content, ,so can be used by blogging platforms.
• Having significant throughput , low latency file access and high read-write operations it is equally applicable for big data analytics systems and Business analytics.

3. Amazon S3(Simple Storage Services):

Amazon S3 is an object storage service which is said to be highly scalable, durable and highly available storage. It is the cheapest type of storage in AWS. S3 is good at storing long-term data due to its archiving system.It is usually used to store and retrieve data from anywhere on the web, making it a widely used and a popular choice including data storage, backup and recovery, content distribution, application hosting, and data analytics.

Features of S3

• Highly durable and 99.99% available
• Highly scalable as it can be autoscaled to petabytes and more without having to provision in upfront
• Highly secure as it supports server side and client side encryption and also
• It supports versioning of files which protects users from accidental modifications and deletions making it more reliable and safe
• It provides, inside the hood , the full access control policies and permission managements that help to maintain authority and authorization
• AWS S3 can be seamlessly integrated with other AWS services and sometimes with other service providers as well.
• It is the cheapest of all other AWS storage methods

Use Cases of S3

• S3 can be used to serve as a central repository for all types of data for building data lakes and before feeding data to big data analytics or machine learning algorithms
• It can be used to store large number media files for streaming by video streaming app like Netflix
• Static Website hosting, along with storing other media and other files for Cloud Native applications
• S3 can be used as a reliable and cost-effective solution for backing up and archiving data, including documents, databases, logs, and application backups as well as restoring in case of disaster as a disaster recovery tool

Conclusion

This article provided a summary on various data storage mechanisms available in Cloud and especially focused on Amazon Web Services. We covered AWS EBS , EFS and S3 storage. Each of the storage mechanisms has their own benefits and constraints associated with it. Choosing one or another solely depends on your use case and need.

The final Takeaways are :

• EBS is a type of block storage which is a high performance, per-instance block storage system which will mostly be associated with a single instance EC2.
• EFS is a type of file storage, a highly scalable storage mechanism designed to use in case more than an EC2 instance has to interact with the same storage.
• S3 is an object storage system which is highly durable and designed to be the cheapest of all and mostly used for data that can be served directly like application specific data ,archives and logs