Post Syndicated from Bala Krishna Gangisetty original https://www.backblaze.com/blog/a-deeper-look-at-s3-compatible-lifecycle-rules-in-backblaze-b2/
As data footprints swell and multi-cloud strategies become the norm, the complexity of managing object lifecycles—from initial creation to eventual archival or deletion—can introduce significant risk and operational overhead. To help our customers solve for this, we recently announced support for lifecycle rules through S3-compatible APIs.
While the previous post focused on what the feature enables and why it matters, this follow-up will look at how lifecycle rules work at a deeper level and what to keep in mind when using them in production.
Why S3 compatible lifecycle rules matter
Let’s quickly refresh your memory about why this feature matters. Many customers already rely on lifecycle rules in Backblaze B2 to manage storage costs and data retention. These rules automate actions like deleting old objects, hiding previous versions, or cleaning up incomplete multipart uploads.
By adding S3 compatible support for lifecycle rules in Backblaze B2, it allows you:
- Lift and shift migrations from AWS S3 to Backblaze B2
- Reuse of existing tools, xml configurations, scripts, applications & infrastructure as code
- Predictable behavior across environments as part of multi-cloud strategy
API surface area and XML structure
Lifecycle rules are managed using three APIs:
PutBucketLifecycleConfigurationGetBucketLifecycleConfigurationDeleteBucketLifecycleConfiguration
These APIs accept and return XML documents that follow the AWS S3 lifecycle configuration format. From a client perspective, this behaves the same way it does on S3, including rule IDs, status flags, filters, and actions.
Example put lifecycle request
Below is a simple example that hides objects under the logs/ prefix after 30 days.
PUT /?lifecycle HTTP/1.1
Host: my-bucket.s3.us-west-004.backblazeb2.com
Content-Type: application/xml
<LifecycleConfiguration>
<Rule>
<ID>deleteOldLogs</ID>
<Status>Enabled</Status>
<Filter>
<Prefix>logs/</Prefix>
</Filter>
<Expiration>
<Days>30</Days>
</Expiration>
</Rule>
<Rule>
<ID>deleteOldLogs_marker</ID>
<Status>Enabled</Status>
<Filter>
<Prefix>logs/</Prefix>
</Filter>
<Expiration>
<ExpiredObjectDeleteMarker>true</ExpiredObjectDeleteMarker>
</Expiration>
</Rule>
</LifecycleConfiguration>
If the XML validates successfully, B2 stores this configuration and returns a 200 OK response.
Example get lifecycle response
GET /?lifecycle HTTP/1.1
Host: my-bucket.s3.us-west-004.backblazeb2.com
<LifecycleConfiguration>
<Rule>
<ID>deleteOldLogs</ID>
<Status>Enabled</Status>
<Filter>
<Prefix>logs/</Prefix>
</Filter>
<Expiration>
<Days>30</Days>
</Expiration>
</Rule>
</LifecycleConfiguration>
The response mirrors the original configuration, allowing tools like Terraform or AWS CLI to reconcile state.
Mapping from S3 rules to B2 rules
Backblaze B2 already supported lifecycle rules through the B2 Native API. The S3 compatible lifecycle implementation builds on this foundation.
Here is how the mapping works at a high level:
- Each S3 lifecycle rule is converted into one or more internal B2 lifecycle rules.
- Prefix filters in S3 map directly to B2 file name prefixes.
- Expiration actions translate into B2 hide or delete operations.
This approach allows us to reuse the same lifecycle execution engine while exposing a familiar S3 interface.
Conceptually, the flow looks like this:
S3 XML lifecycle rule
|
v
XML validation
|
v
S3 rule compilation
|
v
B2 lifecycle rule objects
|
v
Lifecycle execution engine
Expiration behavior
Lifecycle rules can target both current and non current versions.
Some important details:
- Expiring the current version results in a hide marker being created, matching S3 behavior.
- Non current version expiration deletes older file versions after the configured number of days.
- B2 keeps the same guarantees around version ordering and visibility that exist in the native API.
- To remove abandoned hidemarkers by default, an ExpiredObjectDeleteMarker rule for expiration by days is a must.
- The ExpiredObjectDeleteMarker rule ID is derived from the expiration by days rule ID with a “
_marker” suffix.
Understanding this behavior is important if you rely on object versioning for recovery or audit purposes.
Here is an example to remove older versions of objects with prefix data/ that are hidden for more than 90 days.
<Rule>
<ID>delete-old-versions</ID>
<Status>Enabled</Status>
<Filter>
<Prefix>data/</Prefix>
</Filter>
<NoncurrentVersionExpiration>
<NoncurrentDays>90</NoncurrentDays>
</NoncurrentVersionExpiration>
</Rule>
How this behaves internally
At evaluation time, B2 walks object versions in order and applies rules like this:
Pseudo code:
for each object in bucket:
for each version of object with data prefix:
if version is non current:
if age_in_days >= NoncurrentDays:
delete version
For current versions, expiration works differently.
Expiring the current version
When a lifecycle rule expires the current version, B2 creates a hide marker instead of deleting data immediately.
This matches S3 semantics and preserves version history.
current version expired
|
v
hide marker created
|
v
current version becomes non current
Overlapping and nested prefix rules
Backblaze B2 supports multiple lifecycle rules on the same bucket, including overlapping and nested prefixes.
For example, you might configure:
- A general rule for
logs/that deletes objects after 365 days - A more specific rule for
logs/audit/that deletes objects after 90 days
When rules overlap, the configuration with lowest value is applied to save cost to you. This mirrors S3 behavior and allows fine grained control without duplicating buckets.
Matching logic:
object path: logs/audit/2024/01/file.json
matching rules:
- logs/
- logs/audit/
selected rule:
- logs/audit/ because it has lowest value, 90 days
Multipart upload cleanup
Lifecycle rules can also be used to abort incomplete multipart uploads if not completed in a certain number of days. This is implemented by tracking the initiation time of multipart uploads and periodically removing uploads that exceed the configured threshold. This helps prevent abandoned uploads from consuming storage indefinitely.
Below is an example rule to delete multipart uploads that are not completed after seven days after starting the upload process.
<Rule>
<ID>abort-multipart</ID>
<Status>Enabled</Status>
<AbortIncompleteMultipartUpload>
<DaysAfterInitiation>7</DaysAfterInitiation>
</AbortIncompleteMultipartUpload>
</Rule>
Internal execution model
Multipart uploads are tracked with their initiation timestamp.
Pseudo code:
for each multipart_upload:
if now - initiation_time >= DaysAfterInitiation:
abort upload
delete uploaded parts
Lifecycle execution model
Lifecycle rules are not executed in real time. Instead, they are evaluated by background processes that scan eligible objects and apply actions asynchronously.
Important characteristics:
- Rules are not evaluated in real time
- Execution is eventually consistent
- Large buckets may take multiple passes to fully apply changes
This design allows lifecycle processing to scale across billions of objects efficiently without impacting foreground operations.
High level execution flow:
scan bucket periodically
|
v
identify eligible objects
|
v
apply lifecycle actions (hide and/or delete)
|
v
record progress and continue
Error handling and validation
Lifecycle XML is validated using S3 compatible rules:
- Invalid combinations of actions are rejected
- Missing required fields return errors
- Unsupported actions return descriptive failures
While error messages may differ slightly from AWS, the validation logic follows the same constraints.
Compatibility considerations
While the goal is strong S3 compatibility, there are still some things to keep in mind:
- Not all S3 lifecycle actions are supported yet
- XML validation follows S3 rules, but error messages may differ slightly
- The underlying storage model of B2 can affect timing and visibility
We recommend testing lifecycle rules in a non production bucket before rolling them out broadly.
Final thoughts
Adding S3 compatible support for lifecycle rules on Backblaze B2 was all about making migrations simpler and letting customers reuse existing automation with confidence. It behaves the way you expect while benefiting from B2’s internal scalability.
If you already manage lifecycle rules using S3 APIs, those same configurations can now run on Backblaze B2 with minimal or no changes.
The post A Deeper Look at S3 Compatible Lifecycle Rules in Backblaze B2 appeared first on Backblaze Blog | Cloud Storage & Cloud Backup
