Amazon S3 as Object Storage: Architectural Properties, Data Management Models, and Reliability Mechanisms

1. Overview of Amazon S3

Amazon Simple Storage Service (Amazon S3) is an object storage service designed for scalable storage, protection, and management of data within the Amazon Web Services (AWS) cloud infrastructure. Unlike file systems and block storage devices, Amazon S3 uses an object model: the basic unit of storage is an object, placed in a logical container called a bucket. An object includes user data, metadata, and a unique key by which it is addressed within the bucket.

A fundamental difference between object storage and a file system is the absence of a full hierarchical directory structure. In general-purpose Amazon S3 buckets, a flat namespace is used: a string like path/to/file.txt is not a file system path, but an object key. The display of "folders" in the AWS console is a user abstraction built on common key prefixes. Therefore, the ListObjectsV2 operation with Prefix and Delimiter parameters should be considered as filtering keys by a string prefix, rather than traversing a directory tree.

The internal physical architecture of Amazon S3 is not fully disclosed by AWS. It is more accurate to describe the service through its documented properties: regional bucket binding, distributed storage, automatic redundancy, high availability, and declared storage durability. For the S3 Standard storage class, data is redundantly stored in at least three Availability Zones (AZs) within a region. AWS states that Amazon S3 provides strong read-after-write consistency for object write and delete operations across all AWS regions.

A bucket is a logical container for objects and is created in a specific AWS region. Once created, the bucket name and its region cannot be changed. For general-purpose buckets, the Amazon S3 global namespace is used by default: the bucket name must be unique across all AWS accounts within the corresponding AWS partition. Additionally, AWS supports an account regional namespace, where the name is reserved within a specific account's regional scope. By default, an AWS account can create up to 10,000 general-purpose buckets; the number of objects within a bucket is not limited by service quotas.

An object consists of data, a key, and metadata. The maximum size of a single object is 5 TB. For uploading objects larger than 5 GB, the Multipart Upload mechanism must be used, as a single upload operation does not support larger objects. Metadata is divided into system metadata, such as Content-Type, Last-Modified, and ETag, and user-defined metadata, specified via the x-amz-meta-* prefix. Changing an object's metadata effectively requires rewriting the object, as metadata is part of the object entity.

An important distinction is the separation of storage durability and service availability. Durability characterizes the probability of data preservation without loss, while availability reflects the probability of successful data access at a given time. For example, S3 Standard is designed for 99.999999999% durability and 99.99% availability, whereas S3 One Zone-Infrequent Access provides the same declared level of durability but stores data in only one Availability Zone and has lower availability.

Amazon S3 provides data access through an HTTP/HTTPS interface and a REST API. The main object operations correspond to standard HTTP methods: PUT is used for uploading or overwriting an object, GET for reading data, HEAD for retrieving metadata without transferring the object body, DELETE for deletion, and POST for specific scenarios, including uploads via HTML forms and multipart upload. In production scenarios, HTTPS with TLS is used, and request authentication is performed using AWS Signature Version 4. AWS SDK and AWS CLI operate on top of the same API, abstracting low-level details of HTTP request formation and cryptographic signing.

2. Object Versioning

S3 Versioning is a mechanism for preserving multiple versions of an object with the same key. Its primary purpose is to protect against accidental overwrites, deletions, and incorrect data updates. When versioning is enabled, each PUT operation creates a new version of the object with a unique Version ID.

When versioning is enabled, a delete operation has special semantics. If an object is deleted without specifying a particular Version ID, Amazon S3 does not physically destroy the data but creates a Delete Marker. Upon subsequent access to the key without specifying a version, the user receives a response indicating the object is unavailable, but previous versions continue to exist. To restore the object, the Delete Marker must be deleted. For permanent deletion, a specific object version must be deleted by specifying its Version ID.

A bucket can be in one of three versioning states. In the Unversioned state, versioning is not enabled, and object versions are not saved. In the Versioning-enabled state, all new objects receive a unique Version ID. In the Versioning-suspended state, new objects receive a Version ID with a null value, and previously created versions are retained. Importantly, once versioning is enabled, it cannot be reverted to the original Unversioned state; it can only be suspended.

Versioning is a prerequisite for several high-reliability scenarios, including the use of Object Lock and the organization of secure storage for infrastructure state, such as Terraform state. However, all object versions occupy storage space and are billed. Therefore, for buckets with versioning enabled, lifecycle rules must be designed to limit the retention period of noncurrent versions.

3. Data Encryption

Amazon S3 supports several data encryption models. Server-Side Encryption (SSE) is performed by AWS after receiving the object and before writing it to physical storage. Client-Side Encryption is performed before data is transferred to Amazon S3, so AWS receives an already encrypted object.

SSE-S3 is server-side encryption with keys managed by Amazon S3. In this option, AWS fully manages the key material and encrypts data using the AES-256 algorithm. This mode is basic and does not require separate key management by the user.

SSE-KMS is server-side encryption using AWS Key Management Service (AWS KMS), which is a service for managing cryptographic keys. With this approach, keys are created and controlled through KMS. For customer managed keys, access policies, audit logs, rotation, and the ability to restrict data access by revoking key access are available. When accessing objects encrypted with SSE-KMS, additional requests to KMS may be performed, which must be considered when calculating cost and performance. The S3 Bucket Keys mechanism can be used to reduce the number of KMS calls.

SSE-C is server-side encryption with customer-provided keys. In this mode, the user provides the key with each request to write or read an object. AWS uses the key for encryption or decryption but does not store it. Responsibility for storing, rotating, and ensuring the availability of key material rests entirely with the user. This mode is supported via API, SDK, and CLI, but is not a typical option for management via the console.

Client-side encryption is used when an organization needs full control over the cryptographic perimeter before transferring data to the cloud infrastructure. In this case, Amazon S3 stores only the encrypted binary object. This model provides maximum control over keys but increases implementation complexity, especially regarding key rotation, data recovery, and application compatibility.

Starting in 2023, Amazon S3 automatically applies SSE-S3 server-side encryption to all new objects unless a different encryption mechanism is explicitly specified for them. If necessary, default encryption can be configured for a bucket using SSE-KMS and a specific KMS key. The difference between SSE-KMS and SSE-S3 is not only technical but also regulatory: SSE-KMS provides more advanced mechanisms for auditing, access control, and key lifecycle management.

4. Access Management and Security Policies

The access management model in Amazon S3 is built on a combination of identity policies and resource policies. An IAM (Identity and Access Management) policy is attached to a user, group, or role and defines what actions that identity is allowed to perform. A bucket policy (a resource policy for a bucket) is attached to the bucket itself and defines which principals are allowed access to the resource.

The final access decision is made based on the aggregate of allow and deny rules. A request may be allowed if there is an applicable permission in an IAM policy or bucket policy and no explicit deny. An explicit Deny takes precedence over any permissions and always results in the request being rejected.

In a bucket policy, the principal is specified via the Principal element. Principals can be specific IAM users or roles, other AWS accounts, AWS services, or all principals via "Principal": "*". The latter option does not necessarily mean public access: it can be used in conjunction with conditions, for example, to restrict access only from a specific organization or through a particular network endpoint.

Conditions play an important role in building secure policies. The aws:SourceVpce condition allows restricting access to a specific VPC endpoint, which reduces the risk of data transfer over the public internet. The aws:SecureTransport condition is used to prohibit HTTP requests and enforce HTTPS/TLS. The s3:prefix condition restricts object listing operations to a specific prefix, which is applicable in multi-tenant storage models. The aws:PrincipalOrgID condition allows access only to accounts within a specified AWS Organization.

It is necessary to distinguish between bucket-level operations and object-level operations. For bucket operations, such as s3:ListBucket, an ARN of the form arn:aws:s3:::example-bucket is used. For object operations, such as s3:GetObject, s3:PutObject, and s3:DeleteObject, an ARN of the form arn:aws:s3:::example-bucket/* is used. Incorrect ARN specification is a common cause of access errors, especially when configuring backend storage for infrastructure tools.

Modern access management practice in Amazon S3 primarily involves the use of IAM policies, bucket policies, and S3 Access Points. Access Control Lists (ACLs) were historically used to manage access to individual buckets and objects, but AWS recommends using policies as a more flexible and centralized mechanism whenever possible. In S3 Object Ownership Bucket owner enforced mode, ACLs are disabled, the bucket owner becomes the owner of all objects, and access is managed exclusively by policies.

5. Logging, Auditing, and Observability

Amazon S3 provides several mechanisms for observability and auditing. Server Access Logging records requests made to a bucket and saves entries in a target bucket. These logs contain information about the access time, request method, response status, amount of data transferred, user agent, and other HTTP-level parameters. It is recommended to use a separate target bucket for storing logs to avoid mixing operational data and audit data.

Server access logs are delivered on a best-effort basis, meaning without strict guarantees of completeness, order, or delivery time. Delays and duplicate entries are possible. Since the logs themselves are also Amazon S3 objects, lifecycle rules must be defined for them, otherwise they will indefinitely increase storage volume and operational costs.

AWS CloudTrail addresses a different task. It records management-level API calls, and when data events are enabled, also object-level operations. CloudTrail provides structured JSON records, integrates better with analytical and SIEM systems, and is preferred for security compliance and regulatory audit tasks. Server Access Logging is useful for HTTP-level diagnostics, analyzing download sources, referrer headers, and user agents.

Thus, Server Access Logging and CloudTrail are not interchangeable tools. The former focuses on technical analysis of object requests, while the latter focuses on auditing principal actions and API calls within the AWS infrastructure.

6. Amazon S3 Storage Classes

A storage class defines the economic and operational properties of an object's placement: storage cost, request cost, minimum storage duration, minimum billable size, access latency, and resilience to failures. The choice of storage class should be based on the data access profile, recovery requirements, criticality of data loss, and the system's economic model.

S3 Standard is designed for frequently accessed data and provides millisecond access latency, high availability, and data placement in at least three Availability Zones. S3 Standard-Infrequent Access (S3 Standard-IA) is used for infrequently accessed data that still requires rapid access. S3 One Zone-Infrequent Access (S3 One Zone-IA) also provides millisecond access but stores data in a single Availability Zone, making it suitable only for reproducible or secondary data.

S3 Glacier Instant Retrieval is for archival data that occasionally requires immediate access. S3 Glacier Flexible Retrieval is used for archives where retrieval within minutes or hours is acceptable. S3 Glacier Deep Archive is for long-term archival storage with the lowest storage cost and retrieval times of several hours. S3 Intelligent-Tiering automatically moves objects between access tiers based on actual usage patterns.

When choosing a storage class, the minimum storage duration must be considered. If an object is deleted or transitioned to another class before the specified minimum duration, billing will still be calculated for the full minimum period. For infrequent access classes, a minimum billable object size also applies, which makes them economically inefficient for a large number of small objects. Furthermore, in IA classes, the cost of read operations is higher than in S3 Standard, so for frequent access, storage savings may be offset by request costs.

S3 One Zone-IA requires separate consideration. This class stores data redundantly within a single Availability Zone but is not designed to protect against the physical loss of an entire zone. Consequently, it is applicable for data that can be restored from original sources, recomputed, or copied from another storage.

7. Object Lifecycle Rules

S3 Lifecycle is a mechanism for automatically managing objects over time. It allows moving objects between storage classes, deleting them after a specified period, managing noncurrent versions, and aborting incomplete multipart uploads.

A transition moves an object to another storage class after a specified number of days from creation. Expiration deletes an object after a specified period. For buckets with versioning enabled, there are separate actions: NoncurrentVersionTransition moves noncurrent versions to another storage class, and NoncurrentVersionExpiration deletes noncurrent versions after a specified period. The AbortIncompleteMultipartUpload action deletes incomplete multipart uploads, as already uploaded parts occupy space and are billed until the upload is completed or aborted.

Lifecycle rules can apply to all objects in a bucket, to objects with a specific prefix, to objects with specified tags, and to objects that meet size constraints. When designing rules, it is necessary to consider the limitations of transitions between storage classes and the minimum storage durations for the respective classes.

Object lifecycle is not only a cost optimization tool but also an element of data management architecture. For logs, backups, old object versions, and temporary artifacts, the absence of lifecycle rules can lead to uncontrolled growth in storage costs.

8. Data Replication

S3 Replication is a mechanism for asynchronously copying objects from a source bucket to one or more destination buckets. Replication between regions is called Cross-Region Replication (CRR), and replication within the same region is called Same-Region Replication (SRR). CRR is used for disaster recovery scenarios, geographical data distribution, compliance with jurisdictional requirements, and reducing access latency for users in different regions. SRR is used for log aggregation, copying data between accounts, separating access rights, and creating an additional managed copy of data.

Replication is performed asynchronously. Under typical conditions, latency can be seconds or minutes, but without additional mechanisms, it is not a strict guarantee. Replication Time Control (RTC) is a paid option with a service level agreement: 99.99% of objects must be replicated within 15 minutes.

To configure replication, versioning must be enabled on both the source and destination buckets. By default, only new objects created after the replication rule is enabled are replicated. For existing objects, S3 Batch Replication or a separate copying procedure, such as aws s3 sync, is required.

Not all objects and actions are replicated automatically. By default, objects created before the rule was enabled, objects encrypted with SSE-C, delete markers without corresponding configuration, and objects that are already replicas (unless a replication chain is explicitly defined) are not replicated. If the source bucket uses SSE-KMS, a destination KMS key must be specified, and the replication role must have permissions to decrypt with the source key and encrypt with the target key.

In cross-account scenarios, object ownership should be considered separately. If the destination bucket belongs to another account, object ownership transfer or appropriate access policies must be configured. In modern configurations, it is preferable to use Object Ownership mechanisms, allowing the destination account to correctly manage replicated objects.

9. Object Lock and Immutable Storage

S3 Object Lock implements a Write Once Read Many (WORM) model. This mechanism prevents the deletion or overwriting of an object version until a specified retention period expires or a legal hold is removed. Object Lock is used in systems requiring data immutability: financial reports, medical records, audit logs, evidentiary data, and archives subject to regulatory requirements.

Object Lock only works with buckets for which versioning is enabled. It can be enabled when creating a bucket or for an existing bucket if AWS requirements are met. Once enabled, Object Lock cannot be disabled; furthermore, versioning cannot be suspended for such a bucket. The lock applies to object versions, not to the key as an abstract name.

In Governance Mode, an object is protected from deletion by most users. However, principals with the s3:BypassGovernanceRetention permission can bypass the restriction if explicitly allowed by policies. This mode is suitable for protection against erroneous operations and internal administrative violations.

In Compliance Mode, an object cannot be deleted or modified by anyone, including the AWS account root user, until the retention period expires. The retention period cannot be shortened or disabled. This mode is used for strict regulatory requirements where data immutability must be technically enforced.

Legal Hold is a separate locking mechanism without a predefined end date. An object remains protected until the hold is manually removed. This mechanism is used when the retention period is unknown in advance, for example, during litigation or an internal investigation.

Retention can be set at the bucket level via default retention or at the specific object level during upload. The object level takes precedence over the default setting, allowing different storage modes to be applied to different data categories.

10. S3 Express One Zone and Directory Buckets

S3 Express One Zone is a high-performance Amazon S3 storage class designed for latency-sensitive applications. It provides single-digit millisecond latencies for PUT and GET operations and places data within a single selected Availability Zone. Directory buckets are used to work with this class.

A directory bucket differs from a general-purpose bucket in its placement and access model. It is optimized for low-latency scenarios and can be created in a specific Availability Zone, allowing object storage to be placed closer to compute resources. Unlike general-purpose buckets, directory buckets have a different endpoint model and are primarily used for high-performance workloads, rather than for universal long-term storage.

It should be noted that S3 Express One Zone is not a replacement for S3 Standard for all scenarios. Its advantages are evident in systems where access latency is critical and where data placement in a single Availability Zone is acceptable. For long-term storage of critically important data requiring cross-zone redundancy, storage classes that place data in at least three Availability Zones are more appropriate.

11. Multi-Region Access and Disaster Recovery

Various Amazon S3 mechanisms can be used for disaster recovery and geographical data distribution. CRR provides asynchronous data copying to another region, thereby reducing the Recovery Point Objective (RPO). However, switching applications to another region usually requires additional logic at the DNS level, application configuration, or routing.

S3 Multi-Region Access Points provide a single global endpoint for accessing data stored in multiple regions. Such a mechanism can route requests to the closest or available region and is used in architectures requiring higher resilience to regional failures and reduced latency for geographically distributed users.

When comparing recovery mechanisms, it is necessary to distinguish between RPO and RTO. RPO defines the acceptable amount of data loss over time, and RTO defines the acceptable time to restore service. CRR without RTC usually provides a small but not strictly guaranteed RPO. CRR with RTC adds a time guarantee for replication. Versioning and backup allow data state to be restored, but RTO depends on the recovery procedure. Multi-Region Access Points can reduce RTO through more automated request routing.

12. Performance and Operational Features

Amazon S3 is a scalable service, but the operational architecture must account for the specifics of object access. Object operations are not equivalent to file system operations. For example, there is no atomic object renaming in S3: a rename operation is typically implemented as copying the object to a new key and deleting the old object. This is important for systems that assume POSIX file system semantics.

For large objects, Multipart Upload is recommended. This mechanism divides an object into parts that can be uploaded in parallel. In case of network failures, only the failed part needs to be re-uploaded, not the entire object. This approach increases transfer resilience and can improve upload performance.

During a sudden increase in the number of requests to a bucket or a specific key range, temporary 503 Slow Down responses may occur. These indicate the need for retries with exponential backoff and proper client-side error handling. Therefore, applications working with S3 must implement retry mechanisms, consider operation idempotency, and avoid designs where system correctness depends on the immediate successful execution of every request.

13. Conclusion

Amazon S3 is not a file system, but a distributed object storage with its own model of addressing, consistency, access management, and data lifecycle. Key architectural entities are the bucket, object, key, metadata, and, when versioning is enabled, the version ID. The departure from file semantics allows the service to provide high scalability and durability, but it also requires a different approach to application design.

For correct use of Amazon S3, it is necessary to distinguish between durability and availability, versioning and backup, server-side and client-side encryption, IAM policies and bucket policies, and storage classes with different economic and operational characteristics. Incorrect selection of a storage class, absence of lifecycle rules, unaccounted object versions, incorrect access policies, and unmanaged multipart uploads can lead to increased costs and reduced system reliability.

Modern S3 architecture includes not only classic general-purpose buckets but also specialized mechanisms: Object Lock for immutable storage, S3 Replication for copying data between buckets and regions, S3 Access Points for scalable access management, S3 Express One Zone and directory buckets for low-latency scenarios. Therefore, Amazon S3 should be considered not just a simple cloud file storage, but a comprehensive platform for managing object data, requiring a formal approach to security, cost, availability, and information lifecycle.

Summary Table of Key Characteristics

Characteristic	Value
Storage Type	Object storage
Basic Storage Unit	Object: data, key, and metadata
Maximum Object Size	5 TB
Maximum Single Upload Size	5 GB
Mechanism for Large Object Uploads	Multipart Upload
General Purpose Bucket Namespace	Flat, based on keys
Folders in General Purpose Bucket	Logical visualization of prefixes
Consistency	Strong read-after-write consistency for object operations
S3 Standard Durability	99.999999999%
S3 Standard Availability	99.99%
Versioning	Can be enabled or suspended; cannot revert to unversioned
Deletion with Versioning Enabled	Creation of Delete Marker without physical deletion of versions
Default Encryption	SSE-S3 for new objects
Main Difference of SSE-KMS	Key management, audit, and KMS policies
Object Lock	WORM model for protecting object versions
Lifecycle Rules	Automatic transition, expiration, and deletion of noncurrent versions
CRR	Asynchronous cross-region replication
SRR	Asynchronous intra-region replication
S3 Express One Zone	Low-latency storage in a single Availability Zone