Tag Archives: developer

Results from the Backblaze Generative AI Media Hackathon

Post Syndicated from Jeronimo De Leon original https://www.backblaze.com/blog/results-from-the-backblaze-generative-ai-media-hackathon/

A graphic with the Backblaze + GMI Cloud logo and the words "Hackathon Winners"

Building generative media applications on object storage: What the strongest projects have in common

Between June 22 and August 3, 2026, 1,314 people entered hundreds of projects into a single brief: build a generative media app on Backblaze B2 object storage, orchestrated through the open-source Genblaze SDK, for a share of a $10,000 prize pool. 

Almost every entry could generate an image, a video, or a soundtrack, and generate it well. What separated the strongest was the layer underneath: what gets written down, what can be proved later, and what happens when a file has to be checked, corrected, or locked against deletion.

Here’s a closer look at the five that went furthest, and the parts worth reusing. Every entry is browsable in the hackathon project gallery.

The brief, and why it was narrow on purpose

The brief was narrower than most hackathons get. Submissions couldn’t just be a working demo and a repo. Devpost required teams to explain, in writing, how their app actually used both Backblaze B2 and Genblaze, and to list every provider and model behind it. Judging ran against four criteria: real-world utility, production readiness, meaningful use of B2, and meaningful use of Genblaze.

That last pair is the constraint that did the work. “Meaningful use” rules out the default architecture, where a team generates an asset, drops the bytes in a bucket, and calls that storage. It forces a decision about what the storage layer is actually for. What follows is written the way a technical and creative director would walk five finished pipelines against exactly that rubric: who each app is actually for, whether it holds up past the demo, and how deep the storage and orchestration choices go.

The two primitives

Backblaze B2 is the ground every one of these apps had to build on: S3-compatible object storage, sized for the sheer volume of generated assets, thumbnails, metadata, and provenance records a media pipeline throws off, free to start with 10GB included.

Genblaze, Backblaze’s open-source orchestration SDK, is what feeds it: a unified Pipeline API spanning providers like OpenAI, Google, Runway, Luma, ElevenLabs, and Stability Audio, plus models served through platforms such as GMI Cloud and NVIDIA NIM, so a team can swap providers without rewriting its orchestration. Every run produces a canonical provenance manifest that can be embedded directly into the media file itself (an .mp4, a .png, an .mp3) and persisted to B2 or any S3-compatible store.

The hackathon also partnered with GMI Cloud, giving teams easy access to open-source generative models for image, video, audio, chat, reasoning, and multimodal work, which is why it turns up as a provider more than once below.

Two primitives, six weeks, hundreds of teams, and five very different examples of rigor.

Winners

1st place · firstframe

https://devpost.com/software/firstframe

github.com/migarci2/firstframe

A review room for AI video ads that doesn’t make you wait for the whole render.

firstframe builds a review room for the marketing and creative teams who commission AI-generated video ads: instead of waiting on a full multi-scene render before anyone can react, it streams the first finished scene as a live HLS playlist the moment it’s ready, appending segments as later scenes land. A reviewer starts giving notes while the ad is still being made, not after.

Every generated scene is scored by an actual vision model before a human ever sees it, so obviously broken output gets caught and retried automatically rather than shipped to a reviewer’s inbox. A failover step swaps in a backup model only on a genuine provider error, never on an ordinary slow response, and once a scene clears review its master file and manifest are locked against deletion for thirty days: a guarantee the code proves by trying to delete a locked file and catching the rejection. Automated QA paired with a tamper-evident record of what was approved is what turns a generation pipeline into something a brand could actually sign off on.

On B2, the bucket’s folder structure doubles as a workflow: a scene moves through incoming, running, provenance, approved, and rejected prefixes as it clears review. Once a scene is approved, its master file and manifest get a real thirty-day write-once hold using B2’s Object Lock in Governance mode, and the code proves that isn’t just decorative by trying to delete a locked object by its version ID and catching the rejection B2 throws back. A reviewer’s application key is scoped to the readFiles capability with a name prefix restricting it to the approved folder alone. Four separate lifecycle rules cover the bucket’s different prefixes: stalled uploads in incoming have their multipart parts cancelled after 24 hours, while rejected, in-progress, and approved objects each age out on their own separate timers. The video segments themselves land in B2 as ffmpeg finishes each one, with the playlist rewritten after every segment, which makes B2 a live broadcast target rather than an archive that fills up after the fact. Reads go out as path-style presigned URLs, working around a known issue where virtual-host-style presigning fails on a private B2 bucket. B2’s own Event Notifications, five signed webhook rules, keep the review room in sync in real time, with a fallback to plain polling if an account’s Event Notifications API isn’t enabled. The app also watches B2’s own transaction cap and backs off to local disk instead of crashing when a call gets rejected for exceeding it.

On Genblaze, generation runs through an AgentLoop scored by a ThresholdEvaluator. The judge is a real vision model, an NVIDIA NIM llama-3.2-90b-vision-instruct instance grading the actual rendered keyframes, not a fixed retry count. Two pipeline branches, audio and video, fan into a single compositor node instead of running as a straight chain, a fallback_models failover is wired in and confirmed to trigger only on a genuine model error rather than a timeout, and every run carries two layers of lineage: a shared run id across scenes and loop iterations, plus a second, custom chain id layered on top of that. The team even embedded the manifest directly inside the delivered MP4, so a separate verification command can re-download and re-hash every asset the file claims exists. Along the way they filed three pull requests and an issue against Genblaze itself.

firstframe puts B2’s feature set to work in front of the reviewer. Object Lock in Governance mode, scoped application keys, lifecycle rules, Event Notifications, and presigned URLs are all load-bearing parts of the review workflow, doing visible work on every scene that moves through it. That’s architecture built to be trusted, not just to work.

2nd place · beavous

https://devpost.com/software/beavous

github.com/AmirmLotfy/beavous

A campaign generator built to double-check its own storage, re-verifying every asset the moment it’s read back.

beavous is built for the marketers and small commerce teams who need a full paid-social campaign out of a single product photo: not one hero image, but four creative concepts, sixteen cropped aspect ratios for every placement, on-label ad copy, and a portrait video reel, packaged as a verified ZIP pulled straight from B2.

A public API hands off to a private worker behind a task queue, and every campaign is namespaced to an organization so tenants stay isolated from each other. When a generation gets rejected, the app doesn’t start over. It chains a correction onto the original attempt, which is closer to how a real creative review actually works than a one-shot retry.

On B2, keys are organized hierarchically by organization and campaign rather than by content hash, because the product is multi-tenant by design. Every upload and download goes through a presigned URL, and the database never stores a raw link, only an object key and a hash. Every time an asset is read back, beavous re-downloads it and re-hashes the bytes independently, rather than trusting a manifest check alone. B2 is treated as the single, sole system of record for every generated asset: a clean, one-source-of-truth design with nothing else to keep in sync.

On Genblaze, three custom providers handle Gemini image generation, Gemini video, and Veo image-to-video, each with its own tiered pricing registered on the model. The more interesting move is a correction chain: when a generation gets rejected, its manifest becomes the parent of the next attempt, an explicit correction lineage most one-shot generation pipelines skip entirely. Prompts are marked private so the text never lands in the public manifest, only a hashed reference to it does, and before any manifest is trusted as a correction parent it gets independently re-verified, not assumed correct just because it was the app’s own write.

beavous’s answer to “do you trust your own storage” is simple: no, never. Check it again, every time. That’s a slower design than trusting your own write, and a more honest one.

3rd place · takegraph

https://devpost.com/software/takegraph

github.com/Enoch208/takegraph

A build system for generative media that can prove its own reuse, recovery, and release integrity, live, against B2.

takegraph is built for teams running a production, not a single generation: the kind of project where a script tweak halfway through shouldn’t mean re-rendering everything from scratch. It treats the whole thing like a software build: a content-addressed dependency graph that, when a spec changes, recomputes fingerprints, rebuilds only what’s actually invalidated, and reuses everything else.

A team can also re-download and re-hash the actual bytes behind any reuse, recovery, or release straight from B2, live, checking integrity themselves instead of trusting a log. That kind of self-auditing separates a pipeline meant to run unattended for months from one built to survive a single demo.

On B2, content-addressed keys use a two-level hash split so directory listings stay fast at scale, B2’s own Event Notifications (HMAC-SHA256-signed webhooks) feed a background process, and a separate reconciler periodically re-checks everything by hand in case a webhook is ever missed, coordinated across workers with a database lock so only one reconciler runs at a time. Unvalidated uploads land in a quarantine prefix backed by a real lifecycle rule that expires it automatically, and a bad key gets rejected outright rather than silently rewritten. Two least-privilege application keys, one for day-to-day work and one for releases, are each scoped to a single bucket, and CORS rules on the work bucket exist specifically to support presigned browser uploads. Verification (re-download, re-hash, prove it) is a feature of the product, not an internal tool.

On Genblaze, takegraph builds a real pipeline around the idea: a dedicated run builder, a content-addressable storage sink, manifests, and observability events tied to every step. The team kept its media-generation side lean, calling straight through the GMI Cloud connector for image and video (one of the hackathon’s partner platforms), and pointed all of its custom engineering at the layer that makes the whole pitch work: the storage and consistency system underneath.

takegraph turns storage verification into the product itself: reuse, recovery, and release integrity are things a user can ask the system to prove, live, rather than take on faith. That’s B2 treated as a system of record in the fullest sense: self-healing, event-driven, and built to survive a missed webhook without anyone noticing.

Special mentions

Ninth

https://devpost.com/software/ninth-mkcgtv

A comic studio with a provider for every model it needed, and a manifest for every frame it drew.

Ninth is built for indie comic authors and motion-comic creators working on a small budget, who revise a scene a dozen times before it’s right and can’t afford to pay for a fresh generation on every pass. Describe a story, and it writes, draws, casts, and stages it panel by panel on an editable timeline, then bakes the result to an MP4.

An edit doesn’t force a re-generation: assets are pulled from a semantic library whenever something close enough already exists, and everything the AI produces (timing, layer position, camera, bubble placement, even the artwork itself) stays editable in place afterward. That reuse-first design is backed by 126 regression tests and a render path that shares its engine with the live editor, so what a creator previews is exactly what gets exported.

Its B2 layer favors verified durability: flat content-addressed keys and a manifest per run, plus real, working endpoints for audit, restore, and backfill that report exactly what’s on B2 versus what’s only on local disk, and rebuild the difference with a SHA-256 check on every recovered file.

On Genblaze, Ninth wrote seven or eight custom providers, covering nearly every model it reached for: Azure’s image model, Gemini’s aspect ratios, Veo keyframes, Gemini’s video interpolation model, ElevenLabs music, and two separate structured-text providers. Each one exists for the same reason: to keep that artifact inside a pipeline and give it a manifest, so a generated frame always carries a record of the run and the prompt that made it.

Ninth’s mention is for discipline: the project that took “every generated frame should be reproducible and recoverable” most literally, and built the tooling to prove it on demand.

Spatialize

https://devpost.com/software/spatialize

A spatial twin that gives a person’s spoken correction the same provenance as a model’s output.

Spatialize is built for venue and accessibility teams who need to rehearse a step-free route through a space before anyone sets foot in it: it turns a flat floor plan into a voice-navigable spatial twin, extracting validated 3D geometry from the plan image, then answering spoken questions with route guidance grounded in that geometry.

Because the underlying use case is safety-adjacent, nothing the model proposes is trusted outright. Every extracted floor plan, and every voice-driven edit to it, has to pass the same deterministic geometry validator before it’s accepted, so a plausible-looking but wrong route never quietly makes it into a rehearsal. When that validator rejects an attempt, it hands back the exact error, and the next attempt corrects specifically that, not a blind retry.

On B2, Spatialize favors resilience: every read goes through a short-lived, presigned link, and if B2 isn’t configured at all, the app quietly falls back to a local, path-safe store instead of failing to boot. It runs two storage paths side by side (its own hierarchical object store for run and scene state, and a separate Genblaze storage sink dedicated to generated-media provenance), keeping application state and generation history cleanly apart.

Under the hood, an agentic loop drives the floor-plan extraction, evaluated at each attempt by that same deterministic validator rather than an LLM’s opinion. Three custom text-to-speech providers form a genuine fallback ladder: a cloud model first, a lightweight third-party API next, and a self-hosted, zero-credential model as the last resort, so a narration request degrades gracefully rather than failing outright. The standout idea: a voice-driven edit to the scene creates a new version tagged as a human change, with the actual transcript kept as evidence, so every point in the final scene traces back to either a model’s manifest or a person’s own words.

Spatialize’s real achievement is conceptual: it gives a spoken correction the same seriousness as a model’s output, tracing every point in the final scene back to either a manifest or a transcript. That’s provenance thinking applied somewhere most generative pipelines never point it.

Same brief, different rigor

Every winning project found its own discipline. What they share is how many different, equally valid kinds of rigor a small team can bring to the same two primitives in six weeks. firstframe made B2’s write-once guarantee visible to a reviewer in real time. beavous made re-verification a habit instead of an afterthought. takegraph made its storage layer heal itself. Ninth made every one of its seven custom providers answerable to the same manifest. Spatialize made a human correction as provable as a model’s.

None of that shows up until a team stops treating storage and orchestration as plumbing and starts treating them as part of the design. firstframe’s win sits at the intersection of both: the same rigor that scores a generated frame with a vision model also locks the finished one against deletion.

What you can build on

Strip the five projects back and the same handful of moves keep appearing, none of which need a hackathon, a large team, or a novel model:

  • Object Lock in Governance mode turns “approved” from a database flag into a storage-level fact, and firstframe proves the hold is real by attempting the delete by version ID and catching the rejection B2 returns.
  • Bucket prefixes (incoming, running, approved, rejected) give you queue semantics without a queue, with a separate lifecycle rule per prefix so stalled multipart uploads and dead rejects expire on their own timers.
  • Re-download and re-hash on read, the way beavous does, because a non-deterministic pipeline produces a lot of near-identical artifacts and hashing is how you know which one you’re holding.
  • Event Notifications as the fast path, a periodic reconciler as the source of truth, so a dropped webhook costs you latency instead of correctness.
  • Content-addressed keys with a two-level hash split keep directory listings fast at scale and make reuse cheap: same fingerprint, same bytes, no second generation to pay for.
  • Application keys scoped one per role, like a reviewer key limited to the readFiles capability with a name prefix restricting it to the approved folder.
  • Correction lineage instead of retries: the rejected run’s manifest becomes the parent of the next attempt and carries the specific error forward. beavous and Spatialize arrived at this independently, which suggests it’s the general shape rather than a niche trick.

All of it is available today. B2 starts free with 10GB, and Object Lock, lifecycle rules, Event Notifications, scoped application keys, and presigned URLs are in the API from day one. Genblaze gives you one Pipeline API across OpenAI, Google, Runway, Luma, ElevenLabs, and Stability Audio, plus models served through GMI Cloud and NVIDIA NIM, and a canonical provenance manifest out of every run that embeds straight into the .mp4, .png, or .mp3 you ship.

A good first project is smaller than any of these five. Take a pipeline you already have, write a manifest for every run, persist it to B2, and add one endpoint that re-downloads and re-hashes an asset to prove the manifest is honest. That’s a weekend of work, and everything above is a variation on it. Genblaze is open source, and three of the pull requests behind firstframe are already in it.

For more ideas, the full project gallery has every entry from the hackathon, and there’s a lot in there beyond the five covered here.

The post Results from the Backblaze Generative AI Media Hackathon appeared first on Backblaze Blog | Cloud Storage & Cloud Backup

Backblaze Generative Media Hackathon: Build with B2, Genblaze, and GMI Cloud

Post Syndicated from Jeronimo De Leon original https://www.backblaze.com/blog/backblaze-generative-media-hackathon-build-with-b2-genblaze-and-gmi-cloud/

The generative AI wave is moving from text into media. ChatGPT taught a generation of developers how to wire up an LLM. The next chapter is harder and more interesting: video, image, audio, and multimodal workflows that combine them. New models for each of these are landing every few weeks, from established labs and from teams nobody had heard of a quarter ago. The quality keeps climbing, and the catalog keeps expanding.

That pace is good news, but it asks more of the apps that depend on it. A pipeline that hard-codes today’s best video model is going to feel dated in a month. The apps that come out ahead will be the ones built on reactive architectures: pipelines that stream progress as events, fan out concurrent work, fall back when a provider stalls, and let you swap in a new model with a single line of code.

That’s where the moat is now. If you’re building an AI media product, the differentiation is in the pipeline: which models you call, in what order, how you fall back when one is slow, how you keep latency tolerable, where the assets land, and how you prove what was generated. The teams winning this space treat the orchestration layer as their core IP. The frontend on top is the easy part.

We’re looking for builds that solve real-world use cases. Show us what becomes possible when the pipeline stops being a bottleneck.

What we’re running

The Backblaze Generative Media Hackathon is an online competition for developers building generative media apps. $10,000 in cash prizes, 10 GB of free Backblaze B2 storage to start with, and GMI Cloud credits for the first 270 eligible participants.

  • Grand Prize: $7,000
  • Second Place: $2,000
  • Third Place: $1,000

Registration and submissions run June 22 through August 3, 2026 (5 pm ET). Winners are announced August 12.

The stack

Backblaze B2, Genblaze, and GMI Cloud are the three pieces of a media pipeline that’s both easy to vibe code and serious enough to take to production.

Backblaze B2 is S3-compatible object storage for the generated assets, the metadata, the provenance manifests, the thumbnails, and the logs. Durable URLs that never expire. Event Notifications fire downstream work without polling. Object Lock makes your provenance records tamper-evident. Lifecycle Rules handle cleanup so intermediate artifacts don’t pile up.

Genblaze is our open-source Python SDK for orchestrating generative media workflows. One Pipeline API spans OpenAI, Google, Runway, Luma, Decart, ElevenLabs, Stability Audio, NVIDIA NIM, GMI Cloud, and others. Every run produces a SHA-256-verified provenance manifest. Swap providers with a one-line change. New models drop every couple of months, and your app doesn’t need a rewrite each time.

GMI Cloud is a unified inference platform for open source AI. Image, video, audio, chat, reasoning, and multimodal models behind a single API key. If you want to try five video models in a weekend, this is the easiest way.

What to build

Anything that turns a generative AI capability into something a real person would actually use. Some directions:

  • AI video generation or editing apps
  • Image generation, remixing, or transformation tools
  • Audio, music, or voice generation workflows
  • Multimodal apps that combine text, image, video, and audio
  • AI media libraries for storing, organizing, and searching generated assets
  • Provenance-aware workflows that track how each piece of media was generated
  • Agentic media pipelines that generate, evaluate, retry, and store outputs
  • Tools for creators, marketers, educators, entertainers, or developers

Judges will weigh real-world utility, production readiness, and how meaningfully your app uses B2 and Genblaze. We’re looking for projects where the pipeline is doing real work. Apps that handle multiple providers, recover from errors, track provenance, and store assets reliably.

How to enter

Visit the hackathon page on Devpost for registration details, eligibility requirements, prizes, submission guidelines, and important dates:

https://www.backblaze-generative-media.devpost.com

Build your generative AI media application using Backblaze B2 and Genblaze, then submit your project before the deadline.

Join the Hackathon

If you want a head start on what a Genblaze + B2 app looks like in code, two reference repos already exist. genblaze-gen-media-multi-provider-sample chains five providers into one prompt-to-MP4 pipeline. genblaze-gmicloud-pipeline is a deep multi-model composition inside one provider.

We’re looking forward to seeing what you build.

The post Backblaze Generative Media Hackathon: Build with B2, Genblaze, and GMI Cloud appeared first on Backblaze Blog | Cloud Storage & Cloud Backup

Why AI Projects Stall: Data Silos

Post Syndicated from Maddie Presland original https://www.backblaze.com/blog/why-ai-projects-stall-data-silos/

Isometric illustration of isolated data cubes on a blue gradient background, representing fragmented enterprise data silos.

Customer records live in the database. Payment activity is safely stored in your payment processor. Call recordings and transcripts live in Zoom, Teams, Webex, or another video conferencing application–or are shared to Gong for customer insights. Telemetry resides in an observability tool like Grafana or DataDog. Your own day-to-day work is in Google Drive or OneDrive. It takes hundreds of human hours to figure out what customer behavior and business continuity patterns can be extracted from all of this data.

Extracting insights from your data starts with knowing what you have. The first step is centralizing it — pulling multimodal data from across your systems into a single storage repository where your engineering team and AI agents can actually access it. From there, you can assess what’s useful, what’s usable, and what still needs to be labeled or anonymized before it’s ready to work with.

Assessing your data is like cleaning out the garage: first, you have to do a full inventory to know what you actually have before deciding on new data destinations and purposes.

The hidden data silos most organizations overlook

One of the less-discussed barriers to AI readiness is that many organizations lack a complete picture of their own data assets.

Financial assets are documented. Physical assets are tracked. But images, audio recordings, video files, email archives, documents, logs, and customer interaction histories often sit across systems with inconsistent labeling, unclear ownership, disparate tooling, and no centralized catalog.

Customer calls, support chat transcripts, QA screen captures, surveillance footage, and product images all contain operational insight that can inform AI applications, assuming they’re stored in a way that makes them accessible and usable. Most organizations haven’t done that inventory and don’t know what data they’re sitting on.

In our experience, organizations that broaden their definition of data — and build infrastructure to collect and manage it centrally — consistently find that their AI potential is larger than they initially estimated. The inverse is also true. Organizations that skip this step tend to hit the data silo problem mid-project, when data they assumed was available turns out to be fragmented, unlabeled, or simply missing.

The term “multimodal” describes this in practice: datasets that span formats—images, audio, video, text, and structured records—within the same pipeline. Managing multimodal data at a meaningful scale requires infrastructure decisions made well before an AI project kicks off.

Where the infrastructure question meets the strategy question

Here’s what aligning AI strategy with data strategy actually requires:

Inventory what you have. Before sourcing anything new, take stock of what exists. Support call recordings, usage footage, survey data, transaction histories—these are continuously generated across most organizations and rarely treated as AI assets. A governance committee (described below) is the natural owner of this inventory.

Establish governance before you deploy. Who can use which data, under what conditions, and for what purposes. When data governance is established early, teams get answers in days rather than weeks. When it’s deferred, it becomes a bottleneck mid-project.

Plan storage infrastructure for what you will have, not just what you have. A storage decision made today carries a different cost profile 18 months from now. Hyperscaler egress fees that look manageable on a pilot-scale workload become structural constraints at training scale. Archive tiers that appear to reduce costs carry retrieval latencies incompatible with active AI pipelines. Modeling these costs before committing to a provider architecture prevents the predictable trade-offs: smaller datasets, shorter retention windows, fewer training cycles.

Make the C-suite part of the conversation. IBM’s 2025 CEO Study found that 68% of AI-first organizations have mature, well-established data and governance frameworks. When the CEO is involved in AI governance decisions, the conversation stays connected to business strategy instead of fragmenting into siloed technical decisions.

The competitive advantage lives in the data (silos)

Foundation models are increasingly commoditized. The leading model today will be superseded within months, and capable alternatives are widely available from multiple providers. The latest generation from any major provider is capable, widely available, and will be superseded by something better within months. What cannot be licensed, replicated, or accessed by a competitor is the proprietary data your organization has built up over years of operation: customer patterns, process histories, institutional knowledge.

Getting that data foundation right is what separates AI programs that scale from those that stall. 

Organizations that align their AI strategy with their data strategy from the start make fundamentally different infrastructure decisions. They choose storage providers that support active data movement without penalizing it. They build governance structures that give the right people access without creating bottlenecks. And they treat data growth as a business opportunity, not a cost to manage.

For most organizations, that shift in thinking starts with a simple question: who owns AI strategy? If the answer is “it’s fragmented across different teams,” then the second question is: what would it take to bring those conversations into one room?

Everything that follows—the data readiness, the governance, the infrastructure that actually works at scale—flows from that first alignment.

Read the Backblaze ebook, Navigating Multimodal Dataset Economics, to make decisions about the AI datasets at your organization. 

The post Why AI Projects Stall: Data Silos appeared first on Backblaze Blog | Cloud Storage & Cloud Backup

Your AI Strategy Is Only as Strong as Your Data Foundation

Post Syndicated from Maddie Presland original https://www.backblaze.com/blog/your-ai-strategy-is-only-as-strong-as-your-data-foundation/

Illustration of a layered data stack with the Backblaze logo, surrounded by analytics charts and a data interface on a purple and red grid background.

Two-thirds of enterprise leaders see significant potential in integrating AI models with their proprietary data. Yet only 22% feel confident their current IT infrastructure could actually support new AI applications. That gap—between ambition and capability—is where most AI investments stall, budgets balloon, and promising projects quietly get shelved.

The problem isn’t the models, the change management, or even the cost. It’s the underlying data foundation.

The conversation that never happens

The problem is not that either conversation is wrong. It is that they happen separately.

Most organizations run two parallel conversations about AI, and they rarely meet in the same room until a deployment runs into trouble.

The first happens at the strategy level: executives identify use cases, project ROI, approve budgets, and set timelines.

AI is treated as a technology question with business outcomes.

The second happens further down the stack: infrastructure teams make decisions about where data lives, how it moves, and who can access it.

Storage is treated as a cost question, divorced from strategy.

This separation is intuitive. Different people work on it. Different timelines apply. Different success metrics matter. But the separation is a liability disguised as organizational structure.

Two-thirds of executives leading infrastructure efforts say they are excluded from key AI decision-making conversations. Every model selected, application built, and workflow redesigned depends on the same thing: a data foundation that either supports the strategy or constrains it.

That foundation has to be built alongside the strategy, not bolted on as an afterthought.

The cost of misalignment

There are consequences to this misalignment. When data infrastructure decisions get made separately from AI strategy, the result is predictable: AI teams discover too late that the data they need is fragmented, inaccessible, or poorly governed. Infrastructure teams optimize for cost without knowing what future AI workloads will require. Business leaders fund use cases without validating whether the data foundation can support them.

And more specifically, a company building an internal tool to surface insights from customer support transcripts needs audio and text data organized, labeled, and retrievable before the tool can work. A company developing an AI-powered product for external customers needs guardrails in place for data provenance, consent, and version control before model selection matters. Both depend on data that is governed and accessible before any model enters the picture.

Yet according to Gartner’s survey of data management leaders, 63% of organizations either don’t have or aren’t sure they have the right data management practices for AI. Gartner’s projection is stark: through 2026, organizations will abandon 60% of AI projects unsupported by AI-ready data.

Each abandoned project represents sunk cost, such as engineering hours and vendor contracts.

And yet, the pattern repeats because the problem is structural. Strategy teams set direction without validating that infrastructure can support it. Infrastructure teams make decisions without understanding what the strategy actually requires.

Neither group is wrong; they’re simply operating from different information, different incentives, and different success criteria.

What changes when AI strategy becomes business strategy

When organizations treat AI strategy and data strategy as the same conversation, the outcomes shift. Three things happen:

  1. Infrastructure decisions get made with strategic context. Where data lives, how it moves, what it costs—these have become capability questions rather than cost optimization questions. AI development depends on iteration: moving data between tools, environments, teams, and models. A storage provider that charges $90 per TB for egress can become architecturally limiting, penalizing the high-frequency data movement that accelerates AI development. When CFOs understand that the storage decision determines whether the organization can iterate quickly or iterate slowly, the conversation changes.
  2. AI initiatives get funded with data readiness built in. BCG’s research on future-built companies found that top performers define AI programs with ambitious cost and revenue targets set at the executive level and hold teams accountable to near-term results. What they also have in common: they’ve established governance structures, inventoried their data assets, and planned infrastructure before deployment pressure arrives. 
  3. Accountability shifts. According to McKinsey’s 2025 State of AI survey, 28% of organizations using AI report that their CEO is responsible for overseeing AI governance—the policies, processes, and controls that determine how AI is developed and deployed. McKinsey’s analysis found that CEO oversight of AI governance is one of the factors most correlated with meaningful bottom-line impact from AI use. When executive leadership owns both strategy and governance, the two stay aligned.

A companion resource to help you make data decisions

AI strategy cannot succeed as a standalone initiative. It depends on the organization’s ability to make data available, portable, governed, and cost-effective at the exact moments teams need it. For leaders, that means treating infrastructure not as a downstream implementation detail, but as part of the strategy itself.

Read the ebook, Navigating Multimodal Dataset Economics, to get the guide on making decisions about the AI datasets at your organization, and how interoperable-by-design object storage is critical for multimodal AI datasets.

The post Your AI Strategy Is Only as Strong as Your Data Foundation appeared first on Backblaze Blog | Cloud Storage & Cloud Backup

Network Stats for Q1 2026: Neocloud Traffic Trends

Post Syndicated from Brent Nowak original https://www.backblaze.com/blog/network-stats-for-q1-2026-neocloud-traffic-trends/

A decorative image with the words Q1 2026 Network Stats.

Welcome to our second quarterly Network Stats report covering Q1 of 2026. Along with Drive Stats and Performance Stats, Network Stats pulls back the curtain on real-world infrastructure data, particularly how network-level analytics reflect emerging AI industry trends and usage patterns. 

Get more Network Stats (and the details of the dataset)

If you are curious about what metrics we’re recording and how we classify data in this series, check out the details outlined in our Q3 2025 Network Stats  and Q4 2025 Network Stats report.

One of the roles of the Network Engineering (NetEng) team at Backblaze is to monitor how traffic moves into, out of, and across our platform—not just day-to-day, but over time as customer behavior and industry dynamics evolve. Right now, few forces are reshaping networks faster than AI. 

With the launch of B2 Overdrive in April 2025, we built a direct, high-performance path between our storage layers and neoclouds where processing, inference, and modeling take place. It has given us a front-row seat to the impact of AI and how network behavior is changing with it. This quarter, in addition to our regular data analysis, we’ve added some geographic heatmaps to understand where and how data is moving. 

Join us live for the webinar

Join us live for the Q1 2026 Network Stats webinarMonday, May 4, 2026 at 11:30 a.m. PT / 2:30 p.m. PT. We’ll explore where AI traffic concentrates, how high-magnitude data flows behave, and what early indicators suggest about the future of AI-native infrastructure design.

Can’t make it live, or reading this article after-the-fact? Sign up anyway and catch the recording on demand.

Get Inside Real AI Network Flows

AI: The elephant in the room

AI workflows don’t just need a place to store data, they need to be able to move it quickly, easily, and nearly constantly for short bursts. Large, multi-petabyte datasets are ingested, transformed, exported for training, pulled back for evaluation, and periodically refreshed as models evolve.  

Backblaze plays a key role at both ends of that lifecycle. We serve as a durable storage layer for the initial data ingestion, and as the high-throughput source feeding model training, evaluation, and validation to whatever best neocloud is suitable at the moment. Once that model has been trained, it needs to be stored, served, and periodically retrained, where we serve as the storage medium.

From a network perspective, this represents a meaningful shift from diffuse, internet-style traffic patterns to large, high-bandwidth flows between a smaller set of endpoints typical of AI-centric infrastructure.

Trends and spring renewal

The defining theme of the quarter is “winter freeze” as we observed what looks to be a quiet period of Neocloud and hyperscaler traffic over the winter months with an uptick in March. 

The stacked area graph below shows total traffic by network type over time updated with the most current data. Hosting and internet service provider (ISP) traffic stayed largely within historical norms reflecting steady-state usage patterns. Three slices stand out this quarter:

  • CDN traffic: Increase in CDN traffic over the winter months.
  • Neocloud traffic: After the October 2025 peak, we saw a decline in neocloud traffic heading into January. The winter months were a low volume period with an upward trend observed February into March 2026.
  • Hyperscaler traffic: Hyperscaler traffic also followed the Neocloud pattern of a low period during winter with an upward trend in March 2026.

With more than three quarters of data at our disposal, we can now start to see some trends over time. What do we think is happening? 

This could be an indication of the human element where the business cycle has a lot of downtime in the winter months—all of our code, infrastructure, computing jobs, and new innovations involve people somewhere along the chain and we all (hopefully) tend to take more time off in the winter months.

Another hypothesis is that our numbers only show bits transferred over our network links over time. If a large dataset has been stored with us, there may not need to be an update to the large stored dataset for a number of months as code and models are refined, and then suddenly a large amount of new data makes its way over our network links. Perhaps a combination of both—human and training lifecycle?

We’ll be tracking these metrics and updating in future reports, so this high-level view of network traffic segmented by network type will be interesting to watch.

Chart overview

Now let’s take a deep dive into our data and answer these questions:

Quarter over quarter heatmaps: How and where data moves

To better understand our network activity, we isolated variables like region and types of provider. Let’s look at the following dimensions: 

  1. Total traffic volume: Where did we send and receive the most traffic? 
  2. Magnitude: Where were the data transfers with the most bits per unique IP address?
  3. Uniqueness: What does the number of distinct IP addresses look like? 

Quick terminology refresher

  • Regions
    • US-West: Our largest and longest-running region
    • US-East: Region with the most observed proximity to neocloud infrastructure
    • CA-East: Our newest region in Canada. 
  • Network Types
    • CDN: Networks that use Backblaze as an origin store for content delivery 
    • Hosting: Traditional hosting providers that runs workloads like physical or virtual servers for web, database, or application tasks
    • Hyperscaler: Large, traditional cloud providers
    • ISP Regional: Local or regional ISPs, think of these as the “last mile” paths as these networks are very close to customer equipment and efficient 
    • ISP Tier1: National or international ISPs that carry our traffic long distances
    • Neocloud: AI -focused compute networks

Heatmap #1: Where did we send and receive the most traffic?

US-West ↔ ISP-Regional traffic continues to be a hotspot on the heatmap, as expected. This region has the largest footprint behind it and connections to internet exchanges (IX). This quarter, with a lull in neocloud and hyperscaler traffic over the winter period, we saw an increase in traffic to our CDN partners. The amount of traffic to CDN networks in the US-West, US-East, and EU-Central regions all increased quarter over quarter.

Heatmap #2: Where were the data transfers with the most magnitude (bits per IP address)?

Another metric we record is bits per IP or what we term “magnitude.” This combination of the amount of traffic transferred with how many actors are involved per network is a good proxy to measure how heavy or impactful individual data flows are. In short:

  • High volume, many IPs: Easier to distribute and load-balance across infrastructure. And many source and destination pairs means that we can traffic engineer at the WAN layer, sending some traffic over one provider and some over another.
  • High volume, few IPs: More difficult, but more interesting, from a NetEng perspective. 

Despite the total amount of traffic decreasing over the winter months for neocloud traffic, the magnitude of neocloud transfers still remains high. This speaks to the nature of the traffic pattern—when a GPU/compute cluster is ingesting or producing data, it does so at a high bitrate with just a few number of unique endpoints talking to each other.

As expected, the concentration of our magnitude metric is high for our US-East cluster, with an uptick in concentration in US-West and EU-Central. Where specifically is this new concentration in these regions? Spoiler: We explore neocloud traffic later in the report with geographical data!

As with last quarter, we see a high concentration of neoclouds in US-East, but our new heatmap also shows a rising activity in US-West and EU-Central. We also see more distribution in other use cases. 

Heatmap #3: How many unique addresses do we interact with?

Uniqueness—measured by the number of distinct IP addresses per network type—adds another dimension to the story. Unsurprisingly, the quarter over quarter heatmap looks almost identical. This is expected with our US-West region being the most mature and serving a large amount of ISP Regional consumers and is a good sanity check on our dataset.

  • US-West shows the highest overall uniqueness, driven by its larger number of data centers and mix of workloads.
  • Neocloud traffic, by contrast, tends to involve fewer, more persistent endpoints, consistent with AI pipelines that rely on stable, long-standing connections between storage and compute. 

This is where we can clearly see those AI networking elephant flows showing up in the data.

Summary: Seasonal change in traffic flows

With a lull in bits transferred as noted in our top level graph, we can see what other networks by percentage took over. CDN traffic increased from around 20% to 32% of our total traffic, localized ISP regional traffic also increased 21.5% to 27.8%. Neocloud and hyperscaler traffic reduced from 36.4% in Q4 2025 to 25.5% in Q1 2026. 

Quarter over quarter data

We’re tracking our usual metrics here, and we gave you both this quarter and last quarter’s charts so you can easily spot potential trends.  

First let’s take a look at where all our traffic goes from a global perspective with an updated view of last quarter.

Next, let’s take a look at the geography of our network traffic. As we’ve already seen in previous heatmaps, data gravity is concentrating traffic into specific locations.

Where in the world is the neocloud?

New for March 2026, we’ve added geographic information to our dataset, allowing us to take a look at the concentration of traffic based on network types. We’re taking a look into the following locational slices:

  1. Countries
  2. Countries excluding the United States
  3. U.S. states

Heatmap #1: What countries show the highest concentration of traffic by network type?

Neocloud, hyperscaler, and CDN traffic in our dataset all show high concentrations in the United States. Is this due to our US-West and US-East regions being the largest of our deployments or is it related to how certain traffic types are present in the emerging AI market? A quick search shows that the U.S. contains around 40-45% of all data centers globally, so it’s likely the sheer deployment size and scope of US sites aligns with what we’re seeing at the network level.

That said, excluding outliers can show us additional geographic trends, which leads to our next heatmap.

Heatmap #2: If we exclude the U.S., what countries show the highest concentration of traffic by network type?

Since the United States numbers highly skew the heatmap concentration, below is a heatmap without the US to give us better fidelity on ex-US datapoints. Interesting to note that we deliver a large amount of traffic to CDN entities in the Netherlands. This is due in part to our connectivity to AMS-IX (Amsterdam Internet Exchange). 

The network posture of Europe differs from the United States to where local IX networks are preferred over larger Tier1 ISP networks for transit (politically, cost, and preference wise). Second up is Singapore for CDN content and hosting activity in Germany. The next standout is a concentration of neocloud related activity in Finland, Brazil, France, and Canada.

Heatmap #3: What U.S. States show the highest concentration of traffic by network type?

If we dive into just the U.S. States, we can see the heavily weighted concentration of neocloud traffic to and from California. Hyperscaler activity matching up with our expectations as California and Virginia (specifically the Ashburn and Reston corridor) have a high concentration of partner networks. CDN traffic for us concentrates more for Backblaze specifically since the US-West region is our largest cluster footprint and longest running sites, so naturally the data being served out of these sites is more long-lived content lifecycle wise.

These insights into data geography help us understand, plan, and modify our growth trajectory. I say “help,” because we’re also discovering that neocloud and hyperscaler activity is very bursty and has a high magnitude of traffic flow below. From a planning perspective, understanding whether those bursts translate to a higher baseline for increased demand—and how to support bursts when they happen—is a different conversation than the predictable network trends of years past. 

Neocloud and hyperscaler traffic vs predictive patterns

This quarter we’re sharing a deeper dive into the metrics associated with neocloud and hyperscalers over time. These two traffic types are driving innovation at Backblaze and are interesting to share to the larger industry. While CDN, hosting, and ISP regional traffic patterns are easy for us to model and account for as we manage our network infrastructure, neocloud and hyperscalers growth profiling are a challenge! As we said above, they are bursty and have a high magnitude (bits per associated IP address).

And, in the spirit of that conversation, we have several new views to share:

  • Neocloud and hyperscaler magnitude (bits per IP address) over time
  • Heatmaps over time
    • Neocloud
    • Hyperscaler
    • CDN
    • Hosting
    • ISP regional

Chart #1: What’s the magnitude of neocloud and hyperscaler traffic over time?

Matching our earlier charts, we saw a burst of activity from August to December of last year with a resurgence of high magnitude neocloud traffic in March of 2026.

Heatmap #1 and #2: How dynamic are neocloud and hyperscaler traffic patterns?

Neocloud and hyperscaler traffic patterns are the most impactful to our operations due to the magnitude of their data flows. Below is a more detailed look at these concentrations over time, with a burst of activity from August to November for Neoclouds, a quiet period over the winter months, and a pickup again in March. 

As expected our US-East region remains a hotspot of Neocloud related activity. One standout from this analysis is that the Neocloud activity in March is more spread out over our US-West, US-East, and EU-Central regions. It will be interesting to see if this spread grows or contracts over time.

Over the winter months, there was a noticeable decrease in activity for hyperscale related traffic, most notable in January, but when compared to neocloud traffic, the month-over-month traffic patterns remained strongly visible in our US-East region.

Heatmap #3, #4, and #5: How dynamic are CDN, hosting, and ISP regional traffic patterns?

We’re grouping CDN, hosting, and ISP regional types together because they represent a “steady-state” for us as network operators. These patterns are predictable, spread out over time, and generally do not change month-to-month. We do see visible bursts of traffic, like with a heavy blue tile for CDN in September and a few areas of hosting related traffic in May and October of 2025, but overall these are less impactful to our operations because the magnitude (bits per unique IP address) is lower—many sources are talking to many destinations, which as operators is easy to load balance.

The most striking example of this predictability is, unsurprisingly, ISP regional traffic. This represents your more consumer driven workflows and use cases.

Neocloud business cycle and geography

We can draw a conclusion from all these charts: neocloud and hyperscaler are both different traffic patterns and far more dynamic than CDN, hosting, and ISP regional traffic. For our Network Engineering group, this means we have two different stylistic approaches towards managing our network. 

First, when planning for the neocloud and hyperscaler traffic, our solutions entail adding large amounts of additional bandwidth in increments of 100G and often 400G ports to handle burst rates, ensuring that our inter-switch links inside our datacenter can also handle bursts of traffic, and reaching out to select partners to establish private network-to-network interface (PNI) connections where appropriate for zero-settlement transit. 

Our steady state growth patterns stem from CDN, hosting, and ISP regional traffic with easy to model growth curves. Looking over monthly numbers indicates a clear pattern for us, and that’s easy to plan for.

The last factor that we’re placing more emphasis on is the geographical location of our network interconnections. We’re seeing a high concentration of demand located in the United States, specifically in regions like California, Virginia, Illinois, and Georgia.

With more datapoints, we can clearly see the magnitude of the neocloud and hyperscaler transfers when compared to other network types. As above, it’s a bit early to claim concrete quarter over quarter patterns, but we’ll keep monitoring and updating the dataset. 

What’s next?

Anything specific you want to see? Let us know in the comments or reach out to the Network Stats team. Or, keep up-to-date with the latest technical content with our Developer Newsletter. 

The post Network Stats for Q1 2026: Neocloud Traffic Trends appeared first on Backblaze Blog | Cloud Storage & Cloud Backup

Point-to-Point at 100Gbps: What AI-Grade Infrastructure Actually Requires

Post Syndicated from Brent Nowak original https://www.backblaze.com/blog/point-to-point-at-100gbps-what-ai-grade-infrastructure-actually-requires/

A decorative image that shows several cubes on a background.

Bandwidth purchased for data center connectivity surged by nearly 330% between 2020 and 2024, driven primarily by AI workloads. And in 2024, just 10 buyers accounted for nearly 62% of all purchases, according to Zayo’s Bandwidth Report. That concentration is a structural feature of how AI moves data.

Every training run that pulls data from Backblaze storage to a neocloud passes through our network, and our telemetry captures what those flows look like in real time. Our Q4 2025 Network Stats report covers a full quarter of that data. It shows AI workloads producing a distinct network signature: sustained, high-volume transfers between a small number of endpoints, with infrastructure requirements specific enough to be worth examining in detail.

This piece walks through what that signature looks like and what it means for the infrastructure decisions teams are making right now.

The AI model lifecycle and how it moves data

AI model development is a cycle. Large datasets are ingested and consolidated, exported to compute for training, pulled back for evaluation, then pushed out again as models are refined, retrained, and updated with new data. Each stage requires moving substantial volumes of data between storage and compute, repeatedly, over the life of a model.

That structure produces a specific kind of network traffic. AI training moves petabyte-scale data between storage and compute nodes in sustained, long-lived flows—what network engineers call elephant flows—with training jobs running for hours or days under continuous network load. Add frequent checkpointing, model updates, and periodic data refreshes, and the result is traffic that is high in volume and persistent across the entire training run.

This shows up clearly in the Q4 2025 Network Stats data. Neocloud traffic spiked sharply from July through November, peaking in October, then settled into a higher baseline heading into the new year. One interpretation of that shape is the AI lifecycle playing out across a concentration of large training cycles: ingestion, training egress, then a new steady state as stored models get served and periodically retrained. As we accumulate more quarters of data across a broader customer mix, we’ll be better positioned to distinguish that pattern from seasonal budget cycles or customer-specific factors.

Also visible in the Q4 data is where this traffic is going. Cloud-to-cloud traffic grew from 36.2% to 49.6% quarter-over-quarter, with hyperscaler destinations rising from 3.5% to 18%. As the report notes, it’s too early to call these statistically significant trends; the dataset reflects Backblaze’s specific customer mix and covers a single quarter. The direction is consistent with how AI teams operate in practice: moving workloads across neoclouds and hyperscalers depending on price, availability, and job requirements. The storage layer is what persists across those compute environments, which has direct implications for how it needs to be designed.

What elephant flows require from a network

Traditional cloud infrastructure is designed around a specific traffic profile: many clients, many sessions, many discrete transactions. Routing, load balancing, and edge capacity are all optimized for that pattern, distributing load broadly and handling high volumes of short-lived connections efficiently.

Elephant flows don’t fit that profile. AI training establishes persistent, high-volume connections between client and storage that sustain continuous data movement for hours or days at a time. These connections are stickier than typical cloud traffic, particularly on peered networks, and the strain they produce is concentrated rather than distributed—showing up primarily at the edge, where routers handle sustained throughput at scale. Most traditional cloud infrastructure wasn’t provisioned for that kind of sustained pressure at the edge, because most workloads don’t produce it. Internal to the network, depending on a myriad of things—object size, concurrent threads, hardware, internal routing logic—your network path being sticky doesn’t reduce the number of I/O operations for servers, and often that becomes one of the biggest bottlenecks. 

The stakes make this consequential. Training the most advanced models now costs hundreds of millions of dollars, according to Epoch AI—though we also know this is a new technology, and therefore likely the compute will get more efficient over time. That said, the storage is still storage: Data has to live somewhere.  

Infrastructure that can’t sustain throughput under continuous load doesn’t just slow training down; it adds cost to every run. The practical answer is building throughput capacity at the connection points between storage infrastructure and the internet or peer network, sized for sustained flows rather than peak bursts.

That’s what AI-grade storage infrastructure is designed around: sustained throughput to a small number of destinations, at 100Gbps to 1Tbps per transfer for the largest AI workflows. Achieving that in practice means rethinking how a network is designed, routed, and scaled. We’ve written separately about what that looked like for Backblaze’s own infrastructure. Latency matters here for a specific reason: on long-distance transfers, higher latency directly limits achievable sustained throughput, which is one reason why geographic proximity to compute infrastructure affects real-world performance. 

The Q4 Network Stats heatmaps show this pattern from two angles. The magnitude heatmap, measuring bits transferred per unique IP address, shows high-magnitude neocloud flows clustering clearly in regions serving AI-heavy compute endpoints. The uniqueness heatmap shows neocloud traffic involving fewer, more persistent endpoints than CDN or ISP traffic, consistent with AI pipelines that rely on stable, long-standing connections between storage and compute.

The geography of AI infrastructure

The heatmaps show where AI traffic concentrates geographically today. Neocloud activity in the Q4 dataset clusters in Chicago, Dallas-Houston, Denver, New York, the Northern Virginia Reston/Ashburn corridor, and Atlanta, with a clear skew toward the East Coast. This reflects where AI compute infrastructure was built first and remains densest. Keeping latency low between storage and compute is a prerequisite for sustaining the high throughput rates AI workflows require, and that constraint has historically made East Coast proximity an advantage.

That concentration is consistent with broader infrastructure dynamics. AI training workloads are driving demand for regions with available power, fiber density, and compute infrastructure, and power constraints in preferred markets are already forcing operators to explore secondary locations and invest in custom power infrastructure. Demand in markets outside traditional data center hubs is growing rapidly as a result: Metro bandwidth in Memphis grew from 0.3 terabits to 13.2 terabits between 2023 and 2024. On the flip side, building out cabling to those traditionally under-utilized locations is expensive, often adding thousands of dollars per month to data center economics. High-capacity interconnects are central to making those secondary locations viable; without them, the compute investment is stranded. 

The scale of what’s being built reflects the trajectory of the workload. AI training infrastructure is projected to grow at a 22% compound annual growth rate (CAGR) through 2030, reaching more than 60GW of capacity, while inference infrastructure is expected to grow faster still at 35% CAGR, reaching more than 90GW, according to McKinsey. Training and inference have different geographic requirements: training tolerates latency and can sit in power-rich remote locations, while inference needs to be close to users, which means the buildout will be distributed across both dense metro markets and secondary locations connected by high-capacity fiber.

Backblaze’s own infrastructure decisions reflect this pattern directly. The East Coast concentration drove the decision to double Backblaze’s US-East footprint. At 100Gbps and above, proximity to where AI compute is actually running is a determining factor in storage performance.

What AI-ready storage infrastructure actually means

Storage has traditionally been sized for capacity and evaluated on cost per terabyte. AI workflows change the calculus. When a training run is pulling petabytes of data from object storage to flash storage at sustained 100Gbps rates, the storage layer is as much a performance determinant as the compute layer. A storage system that can’t sustain those throughput rates creates a bottleneck that no amount of GPU capacity can compensate for.

Throughput capability is one requirement. Portability is the other. The multi-cloud behavior visible in the Q4 data reflects how AI teams actually operate: moving workloads to whichever compute provider offers the best price-performance for a given job. Storage that is tightly coupled to one cloud provider is structurally incompatible with that workflow. Data that can’t move freely across cloud environments becomes a constraint on the model development process.

This is the infrastructure problem Backblaze B2 Overdrive is designed to address. By building a direct, high-performance path between Backblaze’s storage layer and the neoclouds where AI processing takes place, it provides the sustained throughput that training workflows require alongside the portability to move data as compute requirements change. The connections between Backblaze and neocloud endpoints visible in the Q4 heatmaps represent that architecture in practice.

The point extends beyond any single product. As AI workloads become a larger share of overall data center activity, the criteria for evaluating storage infrastructure are shifting. Capacity and cost per terabyte remain relevant, but sustained throughput capability, interoperability with compute providers, and network proximity to AI infrastructure are becoming equally important factors. Teams that treat storage as a passive component in AI pipeline design are likely to find it becomes the active constraint.

Early signal, long trend

The Q4 2025 Network Stats data is one quarter of observations from one storage provider’s network. The patterns it shows—high-magnitude flows to a small number of endpoints, East Coast geographic concentration, rising cloud-to-cloud traffic, a higher baseline heading into the new year—are consistent with what the broader industry understands about how AI moves data. What’s new is that they’re visible in real network telemetry rather than modeled projections.

We’ll be watching how neocloud traffic concentration evolves regionally, how the training-to-inference ratio shifts as inference infrastructure scales, and whether the cloud-to-cloud growth visible this quarter continues.

The full dataset, methodology, and visualizations are in the Q4 2025 Network Stats report. For background on how we classify and measure network traffic, the Q3 2025 report covers the dataset in detail, and you can follow the whole series here.

The post Point-to-Point at 100Gbps: What AI-Grade Infrastructure Actually Requires appeared first on Backblaze Blog | Cloud Storage & Cloud Backup

Back Up Your Entire OpenClaw State to Backblaze B2

Post Syndicated from Jeronimo De Leon original https://www.backblaze.com/blog/back-up-your-entire-openclaw-state-to-backblaze-b2/

A decorative image showing a series of 0s and 1s.

There’s a new open-source plugin that snapshots your OpenClaw config, memory, and sessions to B2. It’s designed to be as simple as possible: Three fields to configure. Rollback from chat. Migrate to a new machine in one restart. 

Let’s get into how and why you might want to use it.

OpenClaw keeps everything local. That’s great until it isn’t.

Your config, sessions, memory databases, hooks, cron jobs—everything that makes your OpenClaw instance yours lives on one machine with no built-in redundancy. Compaction can rewrite session transcripts and cause memory loss. A bad config edit or an accidental deletion means rebuilding from scratch: re-onboarding channels, re-pairing devices, re-teaching your agent who you are.

openclaw-b2-backup adds automatic encrypted backups to Backblaze B2 without changing how you use OpenClaw.

Three fields and you’re done

Setup is intentionally minimal:

openclaw plugins install openclaw-b2-backup

Then open ~/.openclaw/openclaw.json and add your B2 credentials to the entry the installer created:

{
"openclaw-b2-backup": {
"enabled": true,
"config": {
"keyId": "004a...",
"applicationKey": "K004...",
"bucket": "my-openclaw-backups"
}
}
}

Restart the gateway, and you’re done. Region is auto-detected from your application key. Encryption is on by default. The first backup runs at midnight, and then daily after that. You can change the schedule to weekly or any cron expression you like.

Free tier friendly: Backblaze B2 includes 10GB of free storage. A typical OpenClaw state directory is 50–500 MB, so even with 10 encrypted snapshots retained, you’ll comfortably stay within the free tier.

Backups that actually happen

The hardest part of any backup system is remembering to run it. This plugin takes care of that with multiple automatic triggers:

  • A daily cron job (configurable) runs a full incremental push at midnight. 
  • Every time you shut down the gateway, a final push runs before exit—so you always have a snapshot of your latest state. 
  • And, before compaction fires (the thing that rewrites your session transcripts and can cause memory loss), the plugin automatically pushes a snapshot. That last one is the one you’ll be most grateful for.

There’s a 5-minute debounce on the compaction trigger, so rapid-fire compactions don’t queue up a dozen pushes.

Rolling back from chat

The plugin registers a b2_rollback tool with your agent, which means you can manage backups conversationally. Just tell your agent:

“Show me my B2 backup snapshots”

And it’ll list all available snapshots with timestamps. To restore one:

“Roll back to the snapshot from before compaction”Before any restore, the plugin automatically creates a safety snapshot of your current state. Safety snapshots are stored separately and never auto-pruned, so you can always recover from a bad rollback. It’s an undo for your undo.

Moving to a new machine

This was one of the most requested use cases: Getting your entire OpenClaw setup onto a new machine without manually copying files and hoping you got everything.

Install the plugin on your new machine, add the same B2 credentials, and restart. The plugin detects the empty state directory, finds your existing snapshots in B2, and automatically restores the latest one. Same memory, same sessions, same config, same personality. No manual file copying.

openclaw plugins install openclaw-b2-backup
# Add your B2 config to openclaw.json
openclaw gateway restart
# Plugin detects empty state + existing snapshots → auto-restores latest

Security by default

Everything is AES-256-GCM encrypted before it leaves your machine. Each file gets a random salt and IV, so identical files produce different ciphertext. The encryption key is derived from your B2 application key via scrypt—no separate key to manage or lose.

Manifests (which contain only file paths and SHA-256 hashes) stay unencrypted so incremental diffing works regardless of encryption. Credentials and auth profiles are excluded from sync by design—secrets stay per-machine, and you re-auth on new machines.

Best practice: Use a B2 application key scoped to a single bucket for least-privilege access. The plugin works perfectly with bucket-scoped keys—region is auto-detected from the authorize response.

Zero external dependencies

The plugin has no external runtime dependencies beyond croner for scheduling. The B2 client is a hand-rolled AWS Signature V4 implementation using only node:crypto. No AWS SDK, no S3 library, no heavyweight dependencies to audit or keep updated.

It runs entirely inside the gateway process—no external scripts, no separate cron daemon, no stopping the gateway to take backups.

Get started

The plugin is open source (under the MIT license) and available now:

openclaw plugins install openclaw-b2-backup

Source code and full documentation: github.com/backblaze-b2-samples/openclaw-b2-sync-backup

npm package: npmjs.com/package/openclaw-b2-backup

If you run into issues or have feature requests, open an issue on GitHub. And if this plugin saves you from a rebuild, we’d love to hear about it.

The post Back Up Your Entire OpenClaw State to Backblaze B2 appeared first on Backblaze Blog | Cloud Storage & Cloud Backup

Introducing the Backblaze Flamethrower Startup Program

Post Syndicated from Stephanie Doyle original https://www.backblaze.com/blog/introducing-the-backblaze-flamethrower-startup-program/

A decorative image showing cogs and cubes with digital lines.

The Backblaze Flameflower Startup Program is here, and it’s built by people who’ve been burned before (and that’s a good thing for you). 

Startups don’t fail because they pick the wrong cloud storage provider.

They fail because everything else is already hard enough, and infrastructure quietly becomes the thing that slows them down, surprises them, or blows up their budget at exactly the wrong moment.

That’s why we’re launching Flamethrower, the new startup program from Backblaze.

It’s not a gimmick. It’s not a lead trap. And it definitely doesn’t bring the “here’s some credits, good luck” energy.

Flamethrower exists for one simple reason: Founders deserve storage infrastructure that helps them move faster, not learn expensive lessons the hard way.

Why Flamethrower exists

Why Flamethrower? 

Startups don’t need more friction, they need a way through it. Metaphor-wise, flames have always been a part of Backblaze’s DNA, and we want this program to speak to the strongest parts of that fire: speed when you need it, reliability when it counts, and power to scale without drama. 

Flamethrower takes these ideas and applies it directly to startups, burning away the biggest blockers between you and your next milestone.

I’ve spent most of my career in and around startups, as a founder, an operator, a partner, a mentor, and occasionally as the person explaining to a CFO why last month’s cloud bill looked like a typo.

I’ve seen the pattern repeat over and over:

  • Early teams optimize for speed (correct)
  • Infrastructure decisions get made quickly (also correct)
  • Those decisions quietly become very sticky
  • And later… someone realizes storage costs are now a board-level discussion

Flamethrower is designed to meet teams before that moment, making your architecture deliberate, not accidental.

Not with abstract promises, but with:

  • Real infrastructure used in real production environments
  • Credits that actually help you get to product-market fit
  • And access to real, live humans who understand what it’s like to build under pressure

This program was built by founders, for founders, including a lot of hard-earned lessons from what doesn’t work.

What makes Flamethrower different

There are plenty of startup programs out there. Many are well-intentioned. Some are very good at marketing. A few are genuinely useful.

Flamethrower is focused on a few things we think matter more than buzzwords:

Human interaction (Yes, really)

Your application is reviewed by real people. Your emails are read by real people.

And if you want to talk to someone, you’ll talk to someone who understands startups, architectures, and workflows, not a generic support queue.

Infrastructure that scales with you

Backblaze B2 is simple, predictable, and S3 compatible. Teams use it for:

In other words: the boring stuff that absolutely has to work.

A program that respects your time

No mandatory demos. No surprise sales motions. No pressure to “convert” before you’re ready.

If Flamethrower helps you build faster—great.

If it doesn’t—that’s okay too, we’ll handle the boring bits until your rocket ship takes off.

Why Backblaze

Backblaze has always had a bit of a different personality.

We believe:

  • Pricing should be understandable
  • Infrastructure should behave predictably
  • Documentation should be written by people who actually use the product
  • And success shouldn’t be punished with surprise bills

That mindset is exactly why Flamethrower fits here.

This isn’t a side project. It’s an extension of how Backblaze already works with developers and builders, just with a little more gas on the fire (if you will).

A bit about me

If we haven’t met yet: I’m James, and I lead Startups and Developer Relations at Backblaze.

I’ve spent years building and running startup programs, developer ecosystems, partnerships, and learning, sometimes painfully, what not to do. I’ve been on the receiving end of “exciting programs” that turned out to be mostly slide decks.

Flamethrower is my attempt (with a very smart team) to build the kind of program I wish existed when I was earlier in my career: practical, honest, and actually useful.

No heroics. No silver bullets. Just support where it counts.

What happens next

The Flamethrower Program is officially live!

If you’re:

  • A founder building something data-heavy
  • An early team thinking about long-term infrastructure
  • Or someone who just wants storage to be the least cumbersome (and most strategic) part of their stack

We’d love to hear from you.

  • Learn more and apply to Flamethrower
  • Or just explore Backblaze on your own terms.
  • Sign up for the Developer newsletter to stay up-to-date with news about the platform and other peoples’ cool projects. 

And if nothing else, thanks for building things. The world needs more people who do.

The post Introducing the Backblaze Flamethrower Startup Program appeared first on Backblaze Blog | Cloud Storage & Cloud Backup

A Deeper Look at S3 Compatible Lifecycle Rules in Backblaze B2

Post Syndicated from Bala Krishna Gangisetty original https://www.backblaze.com/blog/a-deeper-look-at-s3-compatible-lifecycle-rules-in-backblaze-b2/

A decorative image showing a pattern of diamonds on a gradient background.

As data footprints swell and multi-cloud strategies become the norm, the complexity of managing object lifecycles—from initial creation to eventual archival or deletion—can introduce significant risk and operational overhead. To help our customers solve for this, we recently announced support for lifecycle rules through S3-compatible APIs.

While the previous post focused on what the feature enables and why it matters, this follow-up will look at how lifecycle rules work at a deeper level and what to keep in mind when using them in production.

Why S3 compatible lifecycle rules matter

Let’s quickly refresh your memory about why this feature matters. Many customers already rely on lifecycle rules in Backblaze B2 to manage storage costs and data retention. These rules automate actions like deleting old objects, hiding previous versions, or cleaning up incomplete multipart uploads.

By adding S3 compatible support for lifecycle rules in Backblaze B2, it allows you:

  • Lift and shift migrations from AWS S3 to Backblaze B2
  • Reuse of existing tools, xml configurations, scripts, applications & infrastructure as code
  • Predictable behavior across environments as part of multi-cloud strategy

API surface area and XML structure

Lifecycle rules are managed using three APIs:

  • PutBucketLifecycleConfiguration
  • GetBucketLifecycleConfiguration
  • DeleteBucketLifecycleConfiguration

These APIs accept and return XML documents that follow the AWS S3 lifecycle configuration format. From a client perspective, this behaves the same way it does on S3, including rule IDs, status flags, filters, and actions. 

Example put lifecycle request

Below is a simple example that hides objects under the logs/ prefix after 30 days.

PUT /?lifecycle HTTP/1.1
Host: my-bucket.s3.us-west-004.backblazeb2.com
Content-Type: application/xml
<LifecycleConfiguration>
<Rule>
<ID>deleteOldLogs</ID>
<Status>Enabled</Status>
<Filter>
<Prefix>logs/</Prefix>
</Filter>
<Expiration>
<Days>30</Days>
</Expiration>
</Rule>
 <Rule>
<ID>deleteOldLogs_marker</ID>
<Status>Enabled</Status>
<Filter>
<Prefix>logs/</Prefix>
</Filter>
<Expiration>
<ExpiredObjectDeleteMarker>true</ExpiredObjectDeleteMarker>
</Expiration>
</Rule>
</LifecycleConfiguration>

If the XML validates successfully, B2 stores this configuration and returns a 200 OK response.

Example get lifecycle response

GET /?lifecycle HTTP/1.1
Host: my-bucket.s3.us-west-004.backblazeb2.com
<LifecycleConfiguration>
<Rule>
<ID>deleteOldLogs</ID>
<Status>Enabled</Status>
<Filter>
<Prefix>logs/</Prefix>
</Filter>
<Expiration>
<Days>30</Days>
</Expiration>
</Rule>
</LifecycleConfiguration>

The response mirrors the original configuration, allowing tools like Terraform or AWS CLI to reconcile state.

Mapping from S3 rules to B2 rules

Backblaze B2 already supported lifecycle rules through the B2 Native API. The S3 compatible lifecycle implementation builds on this foundation.

Here is how the mapping works at a high level:

  • Each S3 lifecycle rule is converted into one or more internal B2 lifecycle rules.
  • Prefix filters in S3 map directly to B2 file name prefixes.
  • Expiration actions translate into B2 hide or delete operations.

This approach allows us to reuse the same lifecycle execution engine while exposing a familiar S3 interface.

Conceptually, the flow looks like this:

S3 XML lifecycle rule
|
v
XML validation
|
v
S3 rule compilation
|
v
B2 lifecycle rule objects
|
v
Lifecycle execution engine

Expiration behavior

Lifecycle rules can target both current and non current versions.

Some important details:

  • Expiring the current version results in a hide marker being created, matching S3 behavior.
  • Non current version expiration deletes older file versions after the configured number of days.
  • B2 keeps the same guarantees around version ordering and visibility that exist in the native API.
  • To remove abandoned hidemarkers by default, an ExpiredObjectDeleteMarker rule for expiration by days is a must.
  • The ExpiredObjectDeleteMarker rule ID is derived from the expiration by days rule ID with a “_marker” suffix.

Understanding this behavior is important if you rely on object versioning for recovery or audit purposes.

Here is an example to remove older versions of objects with prefix data/ that are hidden for more than 90 days.

<Rule>
<ID>delete-old-versions</ID>
<Status>Enabled</Status>
<Filter>
<Prefix>data/</Prefix>
</Filter>
<NoncurrentVersionExpiration>
<NoncurrentDays>90</NoncurrentDays>
</NoncurrentVersionExpiration>
</Rule>

How this behaves internally

At evaluation time, B2 walks object versions in order and applies rules like this:

Pseudo code:

for each object in bucket:
for each version of object with data prefix:
if version is non current:
if age_in_days >= NoncurrentDays:
delete version

For current versions, expiration works differently.

Expiring the current version

When a lifecycle rule expires the current version, B2 creates a hide marker instead of deleting data immediately.

This matches S3 semantics and preserves version history.

current version expired
|
v
hide marker created
|
v
current version becomes non current

Overlapping and nested prefix rules

Backblaze B2 supports multiple lifecycle rules on the same bucket, including overlapping and nested prefixes.

For example, you might configure:

  • A general rule for logs/ that deletes objects after 365 days
  • A more specific rule for logs/audit/ that deletes objects after 90 days

When rules overlap, the configuration with lowest value is applied to save cost to you. This mirrors S3 behavior and allows fine grained control without duplicating buckets.

Matching logic:

object path: logs/audit/2024/01/file.json

matching rules:
- logs/
- logs/audit/

selected rule:
- logs/audit/ because it has lowest value, 90 days

Multipart upload cleanup

Lifecycle rules can also be used to abort incomplete multipart uploads if not completed in a certain number of days. This is implemented by tracking the initiation time of multipart uploads and periodically removing uploads that exceed the configured threshold. This helps prevent abandoned uploads from consuming storage indefinitely. 

Below is an example rule to delete multipart uploads that are not completed after seven days after starting the upload process.

<Rule>
<ID>abort-multipart</ID>
<Status>Enabled</Status>
<AbortIncompleteMultipartUpload>
<DaysAfterInitiation>7</DaysAfterInitiation>
</AbortIncompleteMultipartUpload>
</Rule>

Internal execution model

Multipart uploads are tracked with their initiation timestamp.

Pseudo code:

for each multipart_upload:
if now - initiation_time >= DaysAfterInitiation:
abort upload
delete uploaded parts

Lifecycle execution model

Lifecycle rules are not executed in real time. Instead, they are evaluated by background processes that scan eligible objects and apply actions asynchronously.

Important characteristics:

  • Rules are not evaluated in real time
  • Execution is eventually consistent
  • Large buckets may take multiple passes to fully apply changes

This design allows lifecycle processing to scale across billions of objects efficiently without impacting foreground operations.

High level execution flow:

scan bucket periodically
|
v
identify eligible objects
|
v
apply lifecycle actions (hide and/or delete)
|
v
record progress and continue

Error handling and validation

Lifecycle XML is validated using S3 compatible rules:

  • Invalid combinations of actions are rejected
  • Missing required fields return errors
  • Unsupported actions return descriptive failures

While error messages may differ slightly from AWS, the validation logic follows the same constraints.

Compatibility considerations

While the goal is strong S3 compatibility, there are still some things to keep in mind:

  • Not all S3 lifecycle actions are supported yet
  • XML validation follows S3 rules, but error messages may differ slightly
  • The underlying storage model of B2 can affect timing and visibility

We recommend testing lifecycle rules in a non production bucket before rolling them out broadly.

Final thoughts

Adding S3 compatible support for lifecycle rules on Backblaze B2 was all about making migrations simpler and letting customers reuse existing automation with confidence. It behaves the way you expect while benefiting from B2’s internal scalability. 

If you already manage lifecycle rules using S3 APIs, those same configurations can now run on Backblaze B2 with minimal or no changes.

The post A Deeper Look at S3 Compatible Lifecycle Rules in Backblaze B2 appeared first on Backblaze Blog | Cloud Storage & Cloud Backup

Open-Sourcing Adaptive Workflows for AI-Driven Development Life Cycle (AI-DLC)

Post Syndicated from Will Matos original https://aws.amazon.com/blogs/devops/open-sourcing-adaptive-workflows-for-ai-driven-development-life-cycle-ai-dlc/

AI-Driven Development Life Cycle (AI-DLC) holds the promise of unlocking the full potential of AI in software development. By emphasizing AI-led workflows and human-centric decision-making, AI-DLC can deliver velocity and quality. However, realizing these gains hinges on how organizations effectively integrate AI into their engineering workflows.

Through our work with engineering teams across industries, we have identified three recurring challenges. These challenges consistently limit the effectiveness of AI in accelerating modern software development. The first challenge is one-size-fits-all workflows. These workflows force every project through the same rigid sequence of steps. The second challenge is the lack of flexible depth in workflow stages. This leads to over-engineering or insufficient rigor. The third challenge is tools that over-automate. These tools unintentionally divert humans away from critical validation and oversight responsibilities.

Achieving true, sustainable productivity requires the process and AI coding agents to become adaptive to context, flexible in depth, and collaborative by design. In this blog, we’ll show you how AI-DLC’s core principles address these three challenges, transforming them from productivity blockers into opportunities for adaptive, human-centered development. We’ll describe how AI-DLC enables workflows that adapt to the problem at hand by intelligently selecting stages, modulating depth, and embedding human oversight at every critical decision point.

We will also introduce our open-source Amazon Q Developer/Kiro Rules implementation, which brings AI-DLC principles to life through adaptive workflow scaffolds. This allows you to start applying these principles in your own projects and experience AI-native development that accelerates delivery without compromising engineering discipline or human judgment.

How does AI-DLC address these challenges?

Let’s explore how AI-DLC addresses these challenges.

1. The “One-Size-Fits-All” Workflow Problem

Software development has never been a linear process. In practice, different projects follow distinct pathways with their own checkpoints and deliverables. Consider these examples:

  • A simple defect fix doesn’t require elaborate requirements analysis and planning
  • A pure infrastructure porting project doesn’t warrant application design with domain modeling
  • A new feature or service addition demands different steps than applying a security patch

Yet, many modern Agentic coding tools provide hard-wired, opinionated workflows that ignore this diversity. Regardless of intent or scope, every project is forced through the same rigid sequence of steps—even when some add little or no value. This rigidity introduces friction, wastes time, and reduces productivity. The result: artificial ceremonies, unnecessary artifacts, redundant approvals, and process overhead that impede velocity.

How AI-DLC addresses this challenge:
AI-DLC addresses this challenge through the Principle 10 (No Hard-Wired, Opinionated SDLC Workflows) as defined in the AI-DLC Method Definition Paper.

AI-DLC avoids prescribing opinionated workflows for different development pathways (such as new system development, refactoring, defect fixes, or microservice scaling). Instead, it adopts a truly AI-First approach where AI recommends the Level 1 Plan based on the given pathway intention.

2. Lack of Flexible Depth Within Each Stage

True adaptivity must go beyond the breadth of a workflow and extend into its depth and intensity. This is how human experts intuitively plan software projects today.

Even when workflows are flexible, many tools fail to modulate the depth of engagement at each stage. For example, building a lightweight utility function doesn’t require full-scale Domain-Driven Design or detailed architectural modeling. When an AI coding agent compels teams to follow these steps regardless of need, the consequence is wasted effort and an over-engineered product. Developers spend cycles reviewing artifacts as the tools dictate rather than delivering business value.

How AI-DLC addresses this challenge:
Through the same principle 10, AI-DLC adapts both the breadth (choice of stages) and the depth of each stage to match the complexity of the intent and context. For example, the complexity of the requirements determines whether a conceptual design is sufficient or whether a full architectural deep dive is required in the Design stage.

Humans validate and adjust this AI-proposed breadth and depth, ensuring that each stage’s rigor matches the scope of the challenge. This elasticity—balancing breadth and depth—is essential for sustaining true velocity without sacrificing engineering discipline.

3. Tools that Reduce the Emphasis on Human Oversight

As AI tools automate more of the Software Development Life Cycle (SDLC), a new risk has emerged: process atrophy. Developers, excited by automation, often drift into passive execution—allowing AI to “decide everything.” The result is a loss of reflection, weakened oversight, and erosion of shared understanding. AI tools must not only automate work but also amplify the significance of human judgment. They should remind practitioners that “human in the loop” is not a checkbox—it is the cornerstone of trust, accountability, and correctness in AI-native development. Equally critical are the rituals and rhythms that sustain collaborative engineering.

How AI-DLC addresses this challenge:
AI-DLC addresses this challenge by requiring a collaborative human-in-the-loop cycle at every stage of the workflow. In this loop, AI generates a plan to execute a task, and relevant stakeholders assemble, review, and validate it.

These rituals, defined as Mob Elaboration and Mob Construction in AI-DLC, ensure that AI’s suggestions are not blindly accepted. Approved plans are executed, and stakeholders again review and validate the final artifacts. The AI-DLC workflow records every human action and approval, embedding reflection to ensure that humans remain the compass, guiding AI’s acceleration.

Circular workflow diagram showing AI-DLC collaboration cycle. Starting at top: Humans Provide Task (orange person icon) , arrow to AI Creates Plan and Seeks Clarification (blue brain icon), arrow to Humans Provide Clarification (orange person icon), arrow to AI Refines Plan (blue brain icon), arrow to Humans Approve Plan (orange person icon), arrow to AI Executes Plan (blue brain icon), arrow to Humans Verify Outcome (orange person icon), completing the cycle back to the start. The diagram illustrates iterative human-AI collaboration with humans making decisions and AI performing execution tasks.

Figure 1: AI-DLC workflow: Humans decide and validate, AI plans and executes.

Effective tooling must therefore emphasize:

  • Promoting for stakeholder collaboration: The system should explicitly call for collaborative rituals involving stakeholders
  • Auditability: Every AI-generated plan and artifact should surface rationale and invite review, recording every human oversight and interaction
  • Flow awareness: Tools should detect when automation races ahead of human validation and deliberately slow down to emphasize critical checkpoints

The goal is not to suppress automation but to embed critical human ownership.

From Principles to Practice

The ideas we outlined — adaptive workflows, flexible depth, and embedded human oversight — are compelling in theory and validated by all engineering teams we’ve engaged. The critical question is: How do we operationalize these ideas into practice without reintroducing the rigidity we seek to eliminate?

One approach is manual prompt engineering: crafting structured prompts that guide AI assistants through the AI-DLC workflow step by step. Each prompt encodes the role AI should assume, the task at hand, the governance requirements, and the audit trail expectations. This structured approach transforms a simple AI interaction into a disciplined workflow that embodies AI-DLC principles.

This approach, while promising, faces its own limitations. Crafting intricate prompts demands discipline and expertise, posing barriers to widespread adoption. Moreover, humans become responsible for maintaining workflow adaptability, selecting the appropriate prompt at the right moment, and ensuring collaborative checkpoints are honored. This places the burden of orchestration back on practitioners, diverging from our core principle of truly AI-native development, where AI itself drives adaptive decision-making.

The question arises: How can we embed AI-DLC principles directly into the execution layer, making adaptivity and collaboration inherent properties of the system rather than manual responsibilities?

Steering for Productivity

The answer lies in workflow scaffolds. These are Rules or Steering customizations for AI Coding Agents. They operationalize AI-DLC principles within the tools. This is done while maintaining transparency, audibility, and modifiability. Our implementation uses Rules/Steering Files. These serve as the foundation of this execution layer. It transforms AI from a passive assistant into an adaptive decision engine.

Rather than requiring developers to craft elaborate prompts, AI-Driven development begins with a simple statement of intent. From there, the workflow scaffolds evaluate context, assess complexity, and dynamically construct an appropriate development pathway. The core workflow definition, including a library of stages and decision heuristics for when and how to apply them, empowers AI to continuously tailor the development process to the nature of the work at hand.

Each AI-DLC phase (Inception, Construction, Operations) evaluates the depth at which it should execute, resulting in a process that adapts to the problem rather than forcing the problem to adapt to the process. This approach yields several critical outcomes:

  1. Adaptive decisioning: The workflow conforms to the problem’s shape, intelligently skipping or deepening stages based on contextual assessment rather than predetermined rules.
  2. Transparent checkpoints: Human approvals are embedded at every decision gate, preserving oversight while maintaining velocity. The system doesn’t just automate; it orchestrates collaboration.
  3. End-to-end traceability: Every artifact, decision, and conversation is logged, creating a continuous, inspectable trail of reasoning that supports both accountability and continuous improvement.

The result is a process that is context-aware, scalable, and self-correcting – capable of supporting everything from a single-line defect fix to a comprehensive system modernization, all while maintaining the rigor and human judgment that define engineering excellence.

Build, Test, and Evolve with Us

We’re open-sourcing the AI-DLC workflow, implemented as Amazon Q Rules and Kiro Steering Files, so organizations everywhere can experience AI-DLC in practice and build production-grade systems. We invite developers, architects, and engineering leaders to:

  1. Apply the steering rules in real-world projects, whether brownfield or greenfield. Refer to our companion AI-DLC workflow walkthrough blog for step-by-step instructions on how to build using AI-DLC in Amazon Q Developer.
  2. Observe how the process adapts to your project’s size, scope, and intent.
  3. Share your experience through our GitHub repository, where you can open issues, propose improvements, and contribute ideas.

Your feedback will help evolve this into a foundation for AI-native software development – one that accelerates delivery without sacrificing rigor or human judgment. Together, we can redefine what software engineering looks like in the age of AI: not scripted but steered.

Conclusion

AI-DLC addresses multiple challenges limiting AI’s effectiveness in software development such as rigid workflows, inflexible workflow depth, and tools that reduce human oversight. AI-DLC enables adaptive workflows that intelligently select stages, modulate depth, and embed human oversight at critical decision points. This approach, implemented through open-source tools like Amazon Q Developer Rules and Kiro Steering, accelerates delivery while maintaining engineering discipline and human judgment.

AI-DLC emphasizes human oversight and collaboration in AI-driven software development. Workflow scaffolds, embed AI-DLC principles into the execution layer, enabling adaptive decision-making, transparent checkpoints, and end-to-end traceability. Open-sourcing the AI-DLC workflow allows organizations to experience AI-DLC in practice and contribute to its evolution.

Ready to get started? Visit our GitHub repository to download the AI-DLC workflow and join the AI-Native Builders Community to contribute to the future of software development.

 

About the authors:

Raja SP

Raja is a Principal Solutions Architect at AWS, where he leads Developer Transformation Programs. He has worked with more than 100 large customers, helping them design and deliver mission critical systems built on modern architectures, platform engineering practices, and Amazon inspired operating models. As generative AI reshapes the software development landscape, Raja and his team created the AI Driven Development Lifecycle (AI-DLC) — an end to end, AI native methodology that re-imagines how large teams collaboratively build production-grade software in the AI era.

Raj Jain

Raj is a Senior Solutions Architect, Developer Specialist at AWS. Prior to this role, Raj worked as a Senior Software Development Engineer at Amazon, where he helped build the security infrastructure underlying the Amazon platform. Raj is a published author in the Bell Labs Technical Journal, and has also authored IETF standards, AWS Security blogs, and holds twelve patents

Siddhesh Jog

Siddhesh is a Senior Solutions Architect at AWS. He has worked in multiple industries in a wide variety of roles and is passionate about all things technology. At AWS Siddhesh is most excited to help customers transition to the AI Driven Development Lifecycle and enable them to build applications rapidly in a secure, complaint and cost efficient cloud environment.

Will Matos

Will Matos is a Principal Specialist Solutions Architect with AWS’s Next Generation Developer Experience (NGDE) team, revolutionizing developer productivity through Generative AI, AI-powered chat interfaces, and code generation. With 27 years of technology, AI, and software development experience, he collaborates with product teams and customers to create intelligent solutions that streamline workflows and accelerate software development cycles. A thought leader engaging early adopters, Will bridges innovation and real-world needs.

Introducing AWS Builder Center: A new home for the AWS builder community

Post Syndicated from Channy Yun (윤석찬) original https://aws.amazon.com/blogs/aws/introducing-aws-builder-center-a-new-home-for-the-aws-builder-community/

We really love builders at AWS. We’re constantly thinking of new ways to help technical communities thrive and create spaces like AWS Developer Center and community.aws where people can connect and share their knowledge and experiences.

Today, we’re announcing AWS Builder Center, a new home for builders to access all builder resources, engage with the AWS community, and provide feedback or product suggestions to AWS product teams. This new experience also integrates the previous AWS Developer Center and community.aws.

There are a variety of exciting features so let us discover some of them.

Your voice matters: Introducing Wishlist
One of the most exciting new features, in my opinion, is Wishlist. You can now submit your wishes for new features or improvements you’d like to see in AWS services. Others can discover and vote on these wishes while also creating their own.

You can influence product roadmap collectively as a community and help us shape the future of AWS services. You can share ideas, suggestions, feature proposals, or challenges while operating AWS services, with the ability for the AWS community to upvote ideas and highlight the most sought-after improvements. Our internal teams will keep an eye on these and bring the most popular wishes to the attention of our service teams, making your voice an integral part of our product development process.

Connect people in the AWS community
On the Connect page, you’ll find many opportunities to connect directly with AWS Heroes and AWS Community Builders. You can explore and join AWS User Groups and AWS Cloud Clubs near your cities around the world.

On top of that, you can bookmark this page as your centralized hub for finding upcoming community events, making it easy to find opportunities to learn and network in your local area and meet like-minded builders who share your interests.

Speaking of following people, AWS Builder Center makes it really straightforward to connect and engage with others, serving as the central hub for the AWS technical community. It brings together all the different ways that you can connect with fellow builders. For example, the Who to Follow section introduces you to AWS Heroes, Community Builders, and active community members who are sharing their knowledge and expertise in your areas of interest.

Explore our AWS hands-on resources
On the Build page, you’ll discover ways to get familiar with AWS with hands-on experience such as interactive learning resources designed for every skill level such as AWS Tutorials and AWS Workshops. You can explore generative AI and agentic AI services playground and find the AWS Free Tier to try out AWS services free of charge up to specified limits for each service.

Choose the Toolbox page and discover the latest tools, programming language resources, and Open Source projects for AWS. The Toolbox has everything you need to get your project scaffolded and up and running.

To improve the build experience for builders, we plan to expand Builder Center’s built-in offerings such as creating dedicated groups and forums for collaborating on a particular topic, run workshops for hands-on labs, and various service playgrounds where builders can freely experiment with AWS services.

Supporting your builder journey
The new Learn section serves as your gateway to skill development, bringing together everything you need to expand your AWS expertise. Here, you can explore learning and training resources, workshops, gamified experiences, and more to make your journey of building on AWS both educational and engaging.

Choose the Topics page, where you can explore and discover more content. You can explore content by topics and tags. There is a featured and trending topics section that helps you to stay connected with what’s capturing the community’s attention right now.

Built-in localization for your spoken language
AWS Builder Center breaks down language barriers with comprehensive localization support. All content published in the Builder Center is automatically available in 16 languages, and user-generated content, such as posts, comments, or wishes, can be machine-translated on demand using Translate. So, you can collaborate with builders worldwide, sharing knowledge and experiences across language boundaries.

By default, all content will be displayed in based on the language that your browser is set to. But, you can override this by visiting the settings page and choosing the language that you want AWS Builder Center to use by default.

Sign up and build your profile now
AWS Builder Center gives you a more personalized and comprehensive way to showcase your AWS journey. Your unique profile comes with a custom URL and shareable QR code, making it straightforward to connect with others and share your presence in the AWS community.

All your posts, wishes, and meaningful interactions are organized within a centralized view so you can easily check them. In the Manage profile page, you can customize your profile, add specific interests and areas of expertise, helping you connect with builders who share your passions. Profile management is seamless: it synchronizes across all AWS services using AWS Builder ID, ensuring your identity remains consistent wherever you engage with AWS offerings.

Visit builder.aws.com, sign up with AWS Builder ID, and claim your unique alias to access all features, including content creation, Wishlist, and community engagement tools.

AWS Builder Center was designed to help you connect, learn, and build with fellow AWS builders, so enjoy your journey together!

ChannyMatheus Guimaraes | @codingmatheus

Access Claude Sonnet 4 in Amazon Q Developer CLI

Post Syndicated from Kirankumar Chandrashekar original https://aws.amazon.com/blogs/devops/access-claude-sonnet-4-in-amazon-q-developer-cli/

Amazon Q Developer now supports Claude Sonnet 4 within the CLI, bringing advanced coding and reasoning capabilities to your development workflows at no additional cost. This latest model excels in coding with a state-of-the-art 72.7% for agentic coding on the SWE-bench (see Claude 4 announcement for more information). With enhanced coding and reasoning capabilities, it helps you analyze complex code, optimize everyday development tasks, implementing bug fixes, running bash commands, and developing new features with immediate feedback loops and more precise responses.

To help you leverage Claude Sonnet 4, Amazon Q Developer lets you easily select specific Claude Sonnet models, giving you increased flexibility the CLI.

  • Claude Sonnet 4: High-performance model with balanced intelligence
  • Claude Sonnet 3.7: High-performance model with extended thinking capability
  • Claude Sonnet 3.5: High-performance intelligent model

For detailed information about Claude model capabilities and comparison, refer to the Anthropic models overview.

In this blog, I will show you how to select Claude Sonnet 4 as your model within the Q Developer CLI and then walk you through a quick demo.

How to Choose Claude Sonnet 4

Make sure to update to the latest version (v1.11.0 onwards) of Amazon Q Developer CLI. Refer installing Amazon Q for command line for installation instructions. You can access Claude Sonnet 4 through these options:

  • During an active chat, use the /model command and select claude-4-sonnet
  • Start a new chat with q chat --model claude-4-sonnet
  • Set it as your default model using q settings chat.defaultModel claude-4-sonnet.

The supported model names for the --model parameter and settings are:

  • claude-3.5-sonnet
  • claude-3.7-sonnet (default)
  • claude-4-sonnet

Model Selection Priority Order

Q Developer CLI selects models in the following order:

  1. Current session model selections (via /model or --model)
  2. User-configured preferences in settings
  3. System default (Claude 3.7 Sonnet)

Key Behaviors

The Q Developer CLI agent defaults to Claude 3.7 Sonnet when no specific model is selected. During active chat sessions, you can seamlessly switch between models using the /model command. Chat continuity is maintained across sessions, with the system retaining the previously selected model when conversations are resumed. If you prefer Claude Sonnet 4, setting it as the default model in user settings will automatically apply to all new chat sessions, though this can be overridden with specific model selections as needed.

qcli-model-selection

Figure 1: Q Developer CLI showing the model loaded for the session

Claude Sonnet 4 with Q Developer CLI in Action

After switching to Claude Sonnet 4 in Q Developer CLI, let’s explore its capabilities with a practical coding example. Here’s the prompt I’ll use for this demonstration:

Create a Python command-line to-do list app with these features:
- Add tasks with descriptions and priorities (low/medium/high)
- Mark tasks as complete by index
- Display tasks sorted by priority, then insertion order
- Show completion status ([x] done, [ ] pending)
- Handle errors for empty tasks and invalid indices
- Store tasks in memory only
Please provide the code to implement this application.

qcli-model-selection-claude-sonnet-in-action

Figure 2: Q Developer CLI interface showing Claude Sonnet 4 in action

In the above demonstration, Q Developer CLI with Claude Sonnet 4 went beyond what was asked in the provided requirements in the prompt by implementing sophisticated command parsing with quoted descriptions, comprehensive error handling, and clean object-oriented design enhanced by type hints. The interface features a helpful guidance system with clear error messages, elegant enum-based priority management, and formatted output for clear task representation.

Additionally, Q Developer CLI with Claude Sonnet 4 also generated documentation in the README for the to-do application, including practical error handling examples and clear usage instructions – transforming the prompt requirements into a well-structured, user-friendly application.

Conclusion

The availability of Claude Sonnet 4 represents a significant advancement in Amazon Q Developer’s capabilities. From intricate code refactoring to streamlined documentation creation, Claude Sonnet 4 helps you accomplish both complex and routine development tasks efficiently.

Whether selecting Claude Sonnet 4 for complex tasks or using other models for specific needs, Amazon Q Developer adapts to your preferences, optimizing AI assistance while maintaining efficiency in your workflow.

The latest version(v1.11.0) of Amazon Q Developer awaits in the CLI, ready to support your development journey with enhanced model capabilities and selection options. Refer Installing Amazon Q for Command line for installation instructions.

To learn more about Amazon Q Developer’s features and pricing details, visit the Amazon Q Developer product page.

About the Author

kirankumar.jpeg

Kirankumar Chandrashekar is a Generative AI Specialist Solutions Architect at AWS, focusing on Amazon Q Developer. Bringing deep expertise in AWS cloud services, DevOps, modernization, and infrastructure as code, he helps customers enhance their development workflows using Amazon Q Developer. Kirankumar is passionate about solving complex customer challenges and enjoys music, cooking, and traveling.

Let’s Architect! Serverless developer experience in AWS

Post Syndicated from Luca Mezzalira original https://aws.amazon.com/blogs/architecture/lets-architect-serverless-developer-experience-in-aws/

Are you a developer approaching serverless for the first time, or even an experienced one looking for a better way to accelerate your feedback loop from code to production? This collection of resources is perfect for you!

There are plenty of developer goodies available on AWS to streamline your code creation and achieve a faster flow in your development lifecycle. Let us share a few examples with you.

What if I told you that you could have an assistant to create your tests? Or that you could review the schema of DynamoDB tables without logging into the AWS Console? Get ready to discover some game-changing tools and techniques that will revolutionize your serverless development process.

And if you want to know more, check out the AWS developer center for more content dedicated to your developer experience on AWS.

Enjoy the journey!

Introducing an enhanced local IDE experience for AWS Lambda developers

We’re excited to announce significant enhancements to the AWS Toolkit, designed to streamline the AWS Lambda development experience. These new features bring the power of Lambda directly to your local development environment, allowing you to work more efficiently within your preferred IDE.

With this update, you can now create, test, and debug Lambda functions locally with unprecedented ease. The toolkit supports local invocation of Lambda functions, enabling real-time testing and debugging without cloud deployment. We’ve also incorporated intelligent code completion and inline documentation for AWS SDK calls, reducing errors and accelerating your coding process.

These improvements offer substantial benefits: faster iteration cycles, deeper insights into Lambda function behavior, and the ability to deliver high-quality serverless applications more rapidly. Whether you’re new to serverless or an experienced Lambda developer, this enhanced local development experience provides a more intuitive and productive environment for building cloud-native solutions.

AWS Toolkit offers the possibility to retrieve real-time the logs of your AWS Lambda functions directly inside your IDE

Figure 1. AWS Toolkit offers the possibility to retrieve real-time the logs of your AWS Lambda functions directly inside your IDE

Take me to this blog

Test Driven Development with Amazon Q Developer

Amazon Q for developers is a versatile AI-powered assistant designed to enhance various aspects of the software development lifecycle. This innovative tool can help streamline numerous tasks, from writing code and documentation to generating unit tests, effectively reducing the time spent on common development activities. By embracing Amazon Q Developer, developers can boost their productivity and focus more on creative problem-solving, with capabilities like test generation serving as just one example of how it can accelerate the development process and improve code quality.

In this example, you will discover how Amazon Q Developer can help out to embrace test-driven development (TDD) in your projects.

Amazon Q developer in action! As you can see you can choose the right recommendation for your code

Figure 2. Amazon Q Developer in action! As you can see you can choose the right recommendation for your code

Take me to this blog

Stop guesstimating the Lambda functions memory size

Optimizing Lambda function performance is crucial for both cost efficiency and user experience, yet many developers still rely on guesswork when setting memory allocations. This approach often leads to suboptimal configurations, resulting in either wasted resources or underperforming functions. Here is where AWS Lambda Power Tuning comes in. By automatically testing your Lambda function with various memory configurations, you can identify the optimal balance between performance and cost. This data-driven approach ensures your functions run at peak efficiency, potentially reducing costs and improving response times. Moreover, as your application evolves, regular power tuning can help you adapt to changing requirements and usage patterns.

The output of running Lambda Power Tuning with your code is a diagram that shows you the best memory size based on your goals. Either optimized for cost or response time or you can choose a more balanced approach

Figure 3. The output of running Lambda Power Tuning with your code is a diagram that shows you the best memory size based on your goals. Either optimized for cost or response time or you can choose a more balanced approach

Take me to this tool

NoSQL Workbench for Amazon DynamoDB

Developers working with Amazon DynamoDB have a powerful ally in their local development toolkit: NoSQL Workbench for Amazon DynamoDB. This intuitive, graphical tool changes the way you interact with DynamoDB tables, offering a fast and efficient feedback loop right on your laptop. With NoSQL Workbench, you can visually design, create, and modify your DynamoDB table structures without the need to constantly access the AWS Console. The tool’s data modeler allows you to experiment with different schemas, ensuring optimal design before deployment. Need to populate your tables for testing? NoSQL Workbench has you covered with its data visualization and manipulation features, enabling quick data insertion and querying. Moreover, its ability to generate sample data and visualize query results in real-time accelerates the development and debugging process.

Visualizing single table design helps you to understand how to structure your serverless applications

Figure 4. Visualizing single table design helps you to understand how to structure your serverless applications

Take me to the documentation

Instrument observability for Lambda functions with Powertools

AWS Lambda Powertools is your go-to open source project when you want to instrument observability and beyond for AWS Lambda functions. Available for multiple programming languages including Python, Node.js, Java, and .NET, Powertools empowers developers to build production-ready Lambda functions with ease. At its core, it provides comprehensive observability features, enabling structured logging, creating custom metrics, and implementing distributed tracing with minimal overhead. But Powertools doesn’t stop there – it also includes utilities for parameter store and secrets management, making it simpler to handle configuration and sensitive data. The suite offers idempotency helpers to ensure reliable execution of your functions, even in the face of retries or duplicates. With its event handler functions, Powertools streamlines the processing of various AWS events, reducing boilerplate code and potential errors. By adopting Powertools, developers can significantly reduce the time spent on implementing best practices, allowing them to focus on building business logic while ensuring their Lambda functions are performant, secure, and easily maintainable.

Powertools for Python goes over and beyond just observability as you can see by the list on the left of this screenshot

Figure 5. Powertools for Python goes over and beyond just observability as you can see by the list on the left of this screenshot

Take me to this tool

AWS Serverless developer experience workshop

The AWS Serverless Developer Experience workshop is an hands-on guide that brings together all the cutting-edge tools and techniques we’ve discussed, offering developers a holistic approach to building serverless applications. This free, self-paced workshop is designed to elevate your serverless development skills, regardless of your experience level. It covers a wide range of topics, from implementing best practices with AWS Lambda Powertools, to optimizing your functions using AWS Lambda Power Tuning. The workshop also delves into CI/CD practices, showing you how to automate your deployment pipeline for faster, more reliable releases.

The serverless developer experience architecture you will work on during the workshop

Figure 6. The serverless developer experience architecture you will work on during the workshop

Take me to the workshop

See you next time!

Thanks for reading! This is the last post of the year, thank you so much for being with us for the 3rd year in a row. To revisit any of our previous posts or explore the entire series, visit the Let’s Architect! page.

Announcing Generative AI CDK Constructs

Post Syndicated from Michael Tran original https://aws.amazon.com/blogs/devops/announcing-generative-ai-cdk-constructs/

Announced by Werner Vogels in his 2023 re:Invent Keynote, Generative AI CDK Constructs, an open-source extension of the AWS Cloud Development Kit (AWS CDK), provides well-architected multi-service patterns to quickly and efficiently create repeatable infrastructure required for generative AI projects on AWS. Our initial release includes five CDK constructs enabling key generative AI capabilities like question and answering, summarization, data ingestion for Retrieval Augmented Generation (RAG), and model deployment capabilities.

Simplify and Accelerate the Development of Applications with Generative AI

Developing generative AI applications is particularly challenging due to the rapidly evolving nature of the underlying technologies. As a result, developers are faced with the challenge of keeping up with changing paradigms and best practices. This often leads to varied and disjointed patterns as developers try crafting their own solutions from scratch. For instance, there are already hundreds of implementations of patterns like retrieval augmented generation(RAG), summarization, or chatbots.

AWS introduced the Cloud Development Kit (CDK), an open-source software development framework to define cloud infrastructure in code and provision it through AWS CloudFormation. CDK empowers developers to use high-level construct libraries that encapsulate AWS best practices, allowing for the creation of cloud applications without needing to know every detail of the AWS services.

A ‘construct‘ in CDK terminology is a building block that represents an AWS resource or a combination of AWS resources configured together. These constructs can range from low-level (individual resources) to high-level (complete architectures), enabling developers to compose and share their cloud application models as code.

Our library harnesses the power of CDK, offering pre-built constructs designed to expedite the deployment of generative AI applications. These constructs use LLMs/FMs available in Amazon Bedrock facilitating easier modeling and rapid deployment of cloud architectures tailored for generative AI. With these constructs available in both Python and Typescript, developers can leverage AWS services to build industry-agnostic solutions swiftly, regardless of their expertise level in generative AI. The constructs are designed to integrate smoothly with existing CDK applications, offering a scalable approach to enhancing generative AI capabilities across various business verticals.

Getting Started with the Constructs

Prerequisites

You can install the CDK constructs in your preferred language:

  • Typescript app: npm i @cdklabs/generative-ai-cdk-constructs
  • Python app: pip install cdklabs.generative-ai-cdk-constructs

Within your existing CDK application, import the new library and get access to available constructs:

  • Typescript app: import * as genai from '@cdklabs/generative-ai-cdk-constructs';
  • Python app: import cdklabs.generative-ai-cdk-constructs as genai

Example

aws-qa-appsync-opensearch is one of the constructs available in the generative-ai-cdk-constructs library. This construct provides a question answering workflow using Amazon Bedrock and a provisioned Amazon OpenSearch cluster. Amazon Cognito is required to authenticate calls to the AWS AppSync GraphQL API created by the construct.

Typescript app:


import { Construct } from 'constructs';
import { Stack, StackProps } from 'aws-cdk-lib';
import * as os from 'aws-cdk-lib/aws-opensearchservice';
import * as cognito from 'aws-cdk-lib/aws-cognito';
import { QaAppsyncOpensearch, QaAppsyncOpensearchProps } from '@cdklabs/generative-ai-cdk-constructs';


class MyStack extends Stack {
    // get an existing OpenSearch provisioned cluster
    const osDomain = os.Domain.fromDomainAttributes(this, 'osdomain', {
        domainArn: 'arn:aws:es:us-east-1:XXXXXX',
        domainEndpoint: 'https://XXXXX.us-east-1.es.amazonaws.com'
    });
    
    // get an existing userpool 
    const cognitoPoolId = 'us-east-1_XXXXX';
    const userPoolLoaded = cognito.UserPool.fromUserPoolId(this, 'myuserpool', cognitoPoolId);
   
    // Create a QA Appsync OpenSearch construct 
    const ragSource = new QaAppsyncOpensearch(
        this,
        'QaAppsyncOpensearch',
        {
            existingOpensearchDomain: osDomain,
            openSearchIndexName: 'demoindex',
            cognitoUserPool: userPoolLoaded
        }
    )

Python app:

from aws_cdk import (
    Stack,
    aws_opensearchservice as os,
    aws_cognito as cognito
)
from constructs import Construct
from cdklabs.generative_ai_cdk_constructs import QaAppsyncOpensearch

class MyStack(Stack):
    def __init__(self, scope: Construct, id: str, **kwargs):
        super().__init__(scope, id, **kwargs)

        # Get an existing OpenSearch provisioned cluster
        os_domain = os.Domain.from_domain_attributes(self, 'osdomain',
            domain_arn='arn:aws:es:us-east-1:XXXXXX',
            domain_endpoint='https://XXXXX.us-east-1.es.amazonaws.com'
        )

        # Get an existing user pool
        cognito_pool_id = 'us-east-1_XXXXX'
        user_pool_loaded = cognito.UserPool.from_user_pool_id(self, 'myuserpool', cognito_pool_id)

        # Create a QA Appsync OpenSearch construct
        rag_source = QaAppsyncOpensearch(
            self, 
            'QaAppsyncOpensearch',
            existing_opensearch_domain=os_domain,
            open_search_index_name='demoindex',
            cognito_user_pool=user_pool_loaded
        )

Refer to the documentation for additional guidance on a particular construct: Catalog

Currently, the following constructs that are available:

Feature Description
Data ingestion pipeline Ingestion pipeline providing a RAG (retrieval augmented generation) source for storing documents in a knowledge base.
Question answering Question answering with a large language model (Anthropic Claude V2) using a RAG (retrieval augmented generation) source and/or long context.
Summarization Document summarization with a large language model (Anthropic Claude V2).
Lambda layer Python Lambda layer providing dependencies and utilities to develop generative AI applications on AWS.
SageMaker model deployment Deploy a foundation model from Amazon SageMaker JumpStart, Hugging Face, or an S3 location to an Amazon SageMaker endpoint.

Explore More Constructs

The library includes constructs for data ingestion pipelines, question answering workflows, document summarization, Lambda layer for generative AI applications, and SageMaker model deployment.

Next Steps

Visit the Generative AI CDK Constructs GitHub repository for a full list of constructs and documentation. For practical examples, check out the AWS samples repository. Your feedback and contributions are welcome on our GitHub repository to enhance the capabilities of Generative AI CDK Constructs.

Alain Krok

Alain Krok is a Senior Solutions Architect with a passion for emerging technologies. His past experience includes designing and implementing IIoT solutions for the oil and gas industry and working on robotics projects. He enjoys pushing the limits and indulging in extreme sports when he is not designing software.

Dinesh Sajwan

Dinesh Sajwan is a Senior Solutions Architect. His passion for emerging technologies allows him to stay on the cutting edge and identify new ways to apply the latest advancements to solve even the most complex business problems. His diverse expertise and enthusiasm for both technology and adventure position him as a uniquely creative problem-solver.

Michael Tran

Michael Tran is a Sr. Solutions Architect with Prototyping Acceleration team at Amazon Web Services. He provides technical guidance and helps customers innovate by showing the art of the possible on AWS. He specializes in building prototypes in the AI/ML space. You can contact him @Mike_Trann on Twitter.

 

Deploying containers to AWS using ECS and CodePipeline 

Post Syndicated from Jessica Veljanovska original https://www.anchor.com.au/blog/2022/10/deploying-containers-to-aws/

Deploying containers to AWS using ECS and CodePipeline 

In this post, it will look at deploying containers to AWS using ECS and CodePipeline. Note that this is only an overview of the process and is not a complete tutorial on how to set this up. 

Containers 101 

However, before we look at how we can deploy Containers onto AWS, it is first worth looking at why we want to use containers, so what are containers?

Containers provide environments for applications to run in isolation. Unlike virtualisation, containers do not require a full guest operating system for each container instance, instead, they share the kernel and other lower-level components of the host operating system as provided by the underlying containerization platform (the most popular of which is Docker, which we will be using in examples from here onwards). This makes containers more efficient at using the underlying hardware resources to run applications. Since Containers are isolated, they can include all their dependencies separate from other running Containers. Suppose that you have two applications that each require specific, conflicting, versions of Python to run, they will not run correctly if this requirement is not met. With containers, you can run both applications with their own environments and dependencies on the same Host Operating system without conflict. 

Containers are also portable, as a developer you can create, run, and test an image on your local workstation and then deploy the same image into your production environment. Of course, deploying into production is never that simple, but that example highlights the flexibility that containers can afford to developers in creating their applications. 

This is achieved by using images, which are read-only templates that provide the containerization platform the instructions required to run the image as a Container. Each image is built from a DockerFile that provides the specification on how to build the image and can also include other images. This can be as simple or as complicated as it needs to be to successfully run the application.

FROM ubuntu:22.04 

COPY . /app 

RUN make /app 

CMD python /app/app.py

However, it is important to know that each image is made up of different layers, which are created based on each line of instruction in the DockerFile that is used to build the image. Each layer is cached by Docker which provides performance benefits to well optimised DockerFiles and the resulting images they create. When the image is run by Docker it creates a Container that flips the layers from the image and adds a runtime Read-Write layer on top, which can be used for logging and any other activity that the application needs to perform and cannot be read from the image. You can use the same image to run as many Containers (running instances of the image) as you desire. Finally, when a Container is removed, the image is retained and only the Read-Write layer is lost. 


Elastic Container Service 

Elastic Container Service or ECS is AWS’ native Container management system. As an orchestration system it makes it easy to deploy and manage containerized applications at scale with built in scheduling that can allow you to spin up/down Containers at specific times or even configure auto-scaling. ECS has three primary modes of operation, known as launch types. Elastic Container Service or ECS is AWS’ native Container management system. As an orchestration system it makes it easy to deploy and manage containerized applications at scale with built in scheduling that can allow you to spin up/down Containers at specific times or even configure auto-scaling. ECS has three primary modes of operation, known as launch types. The first launch type is the AWS EC2 launch type, where you run the ECS agent on EC2 instances that you maintain and manage. The available resource capacity is dependent on the number and type of EC2 instances that you are using. As you are already billed for the AWS resources consumed (EC2, EBS, etc.), there are no additional charges for using ECS with this launch type. If you are using AWS Outposts, you can use also utilise that capacity instead of Cloud-based EC2 instances. 

The second launch type is the AWS Fargate launch type. This is similar to the EC2 launch type, however, the major difference is that AWS are managing the underlying infrastructure. Because of this there are no inherent capacity constraints, and the billing models is based on the amount of vCPU and memory your Containers consume. 

The last launch type is the External launch type, which is known as ECS Anywhere. This allows you to use the features of ECS with your own on-premises hardware/infrastructure. This is billed at an hourly rate per managed instance. 

ECS operates using a hierarchy that connects together the different aspects of the service and allows flexibility in exactly how the application is run. This hierarchy consists of ECS Clusters, ECS Services, ECS Tasks, and finally the running Containers. We will briefly look at each of these services. ECS Clusters 

ECS Clusters are a logical grouping of ECS Services and their associated ECS Task. A Cluster is where scheduled tasks can be defined, either using fixed intervals or cron expression. If you are using the AWS EC2 Launch Type, it is also where you can configure Auto-Scaling for the EC2 instances in the form of Capacity Providers. 

ECS Services 

ECS Services are responsible for running ECS Tasks. They will ensure that the desired number of Tasks are running and will start new Tasks if an existing Task stops or fails for any reason in order to maintain the desired number of running Tasks. Auto-Scaling can be configured here to automatically update the desired number of Tasks based on CPU and Memory. You can also associate the Service with an Elastic Load Balancer Target Group and if you do this you can also use the number of requests as a metric for Auto-Scaling. 

ECS Tasks 

ECS Tasks are running instances of an ECS Task Definition. The Task Definition describes one or more Containers that make up the Task. This is where the main configuration for the Containers is defined, including the Image/s that are used, the Ports exposed, Environment variables, and if any storage volumes (for example EFS) are required. The Task as a whole can also have Resource sizing configured, which is a hard requirement for Fargate due to its billing model. Task definitions are versioned, so each time you make a change it will create a new revision, allowing you to easily roll back to an older configuration if required.


Elastic Container Registry 

Elastic Container Registry or ECR, is not formally part of ECS but instead supports it. ECR can be used to publicly or privately store Container images and like all AWS services has granular permissions provided using IAM. The main features of ECR, besides its ability to integrate with ECS, is the built-in vulnerability scanning for images, and the lack of throttling limitations that public container registries might impose. It is not a free service though, so you will need to pay for any usage that falls outside of the included Free-Tier inclusions.

ECR is not strictly required for using ECS, you can continue to use whatever image registry you want. However, if you are using a CodePipeline to deploy your Containers we recommend using ECR purely to avoid throttling issues preventing the CodePipeline from successfully running. 


CodePipeline 

In software development, a Pipeline is a collection of multiple tools and services working together in order to achieve a common goal, most commonly building and deploying application code. AWS CodePipeline is AWS’ native form of a Pipeline that supports both AWS services as well as external services in order to help automate deployments/releases.

CodePipeline is one part of the complete set of developer tools that AWS provides, which commonly have names starting with “Code”. This is important as by itself CodePipeline can only orchestrate other tooling. For a Pipeline that will deploy a Container to ECS, we will need AWS CodeCommit, AWS CodeBuild, and AWS CodeDeploy. In addition to running the components that are configured, CodePipeline can also provide and retrieve artifacts stored in S3 for each step. For example, it will store the application code from CodeCommit into S3 and provide this to CodeBuild, Code Build will then take this and will create its own artifact files that are provided to CodeDeploy. 

AWS CodeCommit 

AWS CodeCommit is a fully managed source control service that hosts private Git repositories. While this is not strictly required for the CodePipeline, we recommend using it to ensure that AWS has a cached copy of the code at all times. External git repositories can use actions or their own pipelines to push code to CodeCommit when it has been committed. CodeCommit is used in the Source stage of the CodePipeline to both provide the code that is used and to trigger the Pipeline to run when a commit is made. Alternatively, you can use CodeStar Connections to directly use GitHub or BitBucket as the Source stage instead.

AWS CodeBuild 

AWS CodeBuild is a fully managed integration service that can be used to run commands as defined in the BuildSpec that it draws its configuration from, either in CodeBuild itself, or from the Source repository. This flexibility allows it to compile source code, run tests, make API calls, or in our case build Docker Images using the DockerFile that is part of a code repository. CodeBuild is used in the Build stage of the CodePipeline to build the Docker Image, push it to ECR, and update any Environment Variables used later in the deployment.

The following is an example of what a typical BuildSpec might look like for our purposes. 

version: 0.2
  
  
  phases:
  
    pre_build:
  
      commands:
  
        - echo Logging in to Amazon ECR...
  
        - aws --version
  
        - aws ecr get-login-password --region $AWS_DEFAULT_REGION | docker login --username AWS --password-stdin $ACCOUNTID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com
  
    build:
  
      commands:
  
        - echo Build started on `date`
  
        - echo Building the Docker image...
  
        - docker build -t "$REPOSITORY_URI:$IMAGE_TAG-$CODEBUILD_START_TIME" .
  
        - docker tag "$REPOSITORY_URI:$IMAGE_TAG-$CODEBUILD_START_TIME" "$REPOSITORY_URI:$IMAGE_TAG-$CODEBUILD_START_TIME"
  
    post_build:
  
      commands:
  
        - echo Build completed on `date`
  
        - echo Pushing the Docker images...
  
        - docker push "$REPOSITORY_URI:$IMAGE_TAG-$CODEBUILD_START_TIME"
  
        - echo Writing image definitions file...
  
        - printf '{"ImageURI":"%s"}'" $REPOSITORY_URI:$IMAGE_TAG-$CODEBUILD_START_TIME"  > imageDetail.json
  
        - echo $REPOSITORY_URI:$IMAGE_TAG-$CODEBUILD_START_TIME
  
        - sed -i 's|<APP_NAME>|'$IMAGE_REPO_NAME'|g' appspec.yaml taskdef.json
  
        - sed -i 's|<SERVICE_PORT>|'$SERVICE_PORT'|g' appspec.yaml taskdef.json
  
        - sed -i 's|<AWS_ACCOUNT_ID>|'$AWS_ACCOUNT_ID'|g' taskdef.json
  
        - sed -i 's|<MEMORY_RESV>|'$MEMORY_RESV'|g' taskdef.json
  
        - sed -i 's|<IMAGE_NAME>|'$REPOSITORY_URI'\:'$IMAGE_TAG-$CODEBUILD_START_TIME'|g' taskdef.json
  
        - sed -i 's|<IAM_EXEC_ROLE>|'$IAM_EXEC_ROLE'|g' taskdef.json
  
        - sed -i 's|<REGION>|'$REGION'|g' taskdef.json
  
  artifacts:
  
    files:
  
      - appspec.yaml
  
      - taskdef.json
  
      - imageDetail.json

Failures in this step are most likely caused by an incorrect, non-functioning DockerFile or hitting throttling from external Docker repositories. 

AWS CodeDeploy 

AWS CodeDeploy is a fully managed deployment service that integrates with several AWS services including ECS. It can also be used to deploy software to on-premises servers. The configuration of the deployment is defined using an appspec file. CodeDeploy offers several different deployment types and configurations. We tend to use the `Blue/Green` deployment type and the `CodeDeployDefault.ECSAllAtOnce` deployment configuration by default. 

The Blue/Green deployment model allows for the deployment to rollback to the previously deployed Tasks if the deployment is not successful. This makes use of Load Balancer Target Groups to determine if the created Tasks in the ECS Service are healthy. If the health checks fail, then the deployment will fail and trigger a rollback.  

In our ECS CodePipeline, CodeDeploy will run based on the following example appspec.yaml file. Note that the place holder variables in <> are updated with real configuration by the CodeBuild stage.

version: 0.0 

Resources: 

  - TargetService: 

      Type: AWS::ECS::Service 

      Properties: 

        TaskDefinition: <TASK_DEFINITION> 

        LoadBalancerInfo: 

          ContainerName: "<APP_NAME>" 

          ContainerPort: <SERVICE_PORT> 

You may have noticed a lack of configuration regarding the actual Container we will be running up to this point. As we noted earlier, ECS Tasks can use a taskdef file to configure the Task and Containers, which is exactly what we are doing here. One of the files the CodeBuild configuration above expects to be present is taskdef.json. Below is an example taskdef.json file, ss with the appspec.yaml file there are placeholder variables as indicated by <>.


{ 

  "executionRoleArn": "<IAM_EXEC_ROLE>", 

  "containerDefinitions": [ 

    { 

      "name": "<APP_NAME>", 

      "image": "<IMAGE_NAME>", 

      "essential": true, 

      "memoryReservation": <MEMORY_RESV>, 

      "portMappings": [ 

        { 

          "hostPort": 0, 

          "protocol": "tcp", 

          "containerPort": <SERVICE_PORT> 

        } 

      ], 

      "environment": [ 

        { 

          "name": "PORT", 

          "value": "<SERVICE_PORT>" 

        }, 

        { 

          "name": "APP_NAME", 

          "value": "<APP_NAME>" 

        }, 

        { 

          "name": "IMAGE_BUILD_DATE", 

          "value": "<IMAGE_BUILD_DATE>" 

        }, 

      ], 

      "mountPoints": [] 

    } 

  ], 

  "volumes": [], 

  "requiresCompatibilities": [ 

    "EC2" 

  ], 

  "networkMode": "bridge", 

  "family": "<APP_NAME>" 

Failures in this stage of the CodePipeline generally mean that there is something wrong with the Container that is preventing it from starting successfully and passing its health check. Causes of this are varied and rely heavily on having good logging in place. Failures can range from the DockerFile being configured to execute a command that does not exist, to the application itself erroring out when starting for whatever reason might be logged, such as pulling incomplete data from RDS or failing to connect to an external service. 

Summary 

In summary, we have seen what Containers are, why they are useful, and what options AWS provides with its Elastic Container Service (ECS) for running them. Additionally, we looked at what parts of AWS CodePipeline we would need to use in order to deploy our Containers to ECS using CodePipeline.

For further information on the services used, I would highly recommend looking at the documentation that AWS provides. 

For a more hands-on guided walkthrough on setting up ECS and CodePipeline, AWS provide the following resources, there is also plenty of third party material you can find online. 

The post Deploying containers to AWS using ECS and CodePipeline  appeared first on Anchor | Cloud Engineering Services.

Using CodePipeline to Host a Static Website

Post Syndicated from Jessica Veljanovska original https://www.anchor.com.au/blog/2022/10/using-codepipeline-to-host-a-static-website/

There are a variety of ways in which to host a website online. This blog (post) explores how to simply publish and automate a statically built website. Hugo is one such example of a system which can create static websites and is popularly used for blogs.

The final website itself, will consist and contain; HTML, CSS, Images and JavaScript. During this process, (Anchor’s cloud experts) have listed the AWS Services in order to achieve our goal. These include:

  • Cloudfront
  • CodeBuild
  • CodePipeline
  • IAM
  • Amazon S3
  • CodeCommit

This Solution should be well below $5 per month as most of the items are within the Always Free Tier, except the AWS S3 Storage (Which only has valid free-tier for the first 12 month) depending on the traffic.   

In order to keep this simple, the below steps are done via the console, although I have also published a similar simplified project using Terraform which can be found on Github. 

Setup the AWS Components
Setup S3 

To begin with, you will need to setup some storage for the website’s content, as well as the pipeline’s artifacts. 

Create the bucket for your Pipeline Files. Ensure that Block all public access is checked. It’s recommended you also enable Server-side encryption by default with SSE-S3. 

Create a bucket for your Website Files. Ensure that block all public access is NOT checked. This bucket will be open to the world as it will need to be configured to host a website. 

 

When the Website Files bucket is created. Go to Properties tab and Edit Static website hosting. Set it to Enable, and select the type as Host a static website. Save Changes. Note the URL under Bucket website endpoint.Go to Permissions tab and Edit the Bucket policy. Paste in a policy such as the sample below. Update ${website-bucket-name} accordingly to match the name of the bucket. 

<{ "Version": "2012-10-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Principal": { "AWS": "*" }, "Action": "s3:GetObject", "Resource": "arn:aws:s3:::${website-bucket-name}/*" } ] } >

 

Setup IAM Users

You will need to first create an IAM Role which your CodePipeline and CodeBuild will be able to assume. Some of the below permissions can be drilled down further, as they are fairly generic. In this case we are merging the CodePipeline/CodeBuild into one user.  Create a Role in IAM with a unique name based on your project. You will need to setup the below trust policy.  


<{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": [ "codebuild.amazonaws.com", "codepipeline.amazonaws.com" ] }, "Action": "sts:AssumeRole" } ] } >

Create a Policy. Paste in a policy such as the below sample. Update ${website-bucket-name} and ${pipeline-bucket-name} accordingly. Attach the policy to the role you created in the step prior. 


<{ "Version": "2012-10-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Action": "s3:*", "Resource": [ "arn:aws:s3:::${pipeline-bucket-name}/*", "arn:aws:s3:::${pipeline-bucket-name}", "arn:aws:s3:::${website-bucket-name}/*", "arn:aws:s3:::${website-bucket-name}" ] }>,

<{ "Sid": "", "Effect": "Allow", "Action": [ "codebuild:*", "codecommit:*", "cloudfront:CreateInvalidation" ], "Resource": "*" }>,

< { "Sid": "", "Effect": "Allow", "Action": [ "logs:Put*", "logs:FilterLogEvents", "logs:Create*" ], "Resource": "*" } ] } >

Setup CloudFront

  • Access Cloudfront and select Create distribution.
    Under Origin domain – select the S3 Bucket you created earlier.
  • Under Viewer protocol policy, set your desired actions. If you have a proper SSL you can set it up later and use Redirect HTTP to HTTPS.
  • Under Allowed HTTP methods, select GET, HEAD.
  • You can setup Alternate domain name here, but make sure you have an ACM Certificate to cover it, and setup Customer SSL certificate if you wish to use HTTPS.
  • Set the Default root object to index.html. This will ensure it loads the website if someone visits the root of the domain.
  • Leave everything else as default for the moment and click Create distribution.
  • Note down the Distribution ID, as you will need it for the Pipeline.

Setup the Pipeline

Now that all the components have been created, let’s setup the Pipeline to Tie them all Together.
Setup CodeCommit 
Navigate in to CodeCommit, and select Create Repository
You can use git-remote-codecommit in order to checkout the repository locally. You will need to check it out to make additional changes.  

You will need to make a sample commit, so create a directory called public and a file called index.html within it with some sample content, and push it up. 

$ cat public/index.html 
This is an S3-hosted website test 

After this is done, you should have a branch called “master” or “main” depending on your local configuration. This will need to be referenced during pipeline creation.
Setup buildspec in Repository 

Add a buildSpec.yml file to the CodeCommit Repo in order to automatically upload your files to AWS S3, and Invalidate the Cloudfront Cache. Note that ${bucketname} and ${cloudfront_id} in the examples below need to be replaced with the real values.  


version: 0.2

phases: 

  install: 

    commands: 

      - aws s3 sync ./public/ s3://${bucketname}/ 

      - aws cloudfront create-invalidation --distribution-id ${cloudfront_id} --paths "/*" 

Setup CodePipeline 

In our example, we’re going to use a very simple and straightforward pipeline. It will have a Source and Deployment phase. 

Navigate in to CodePipeline, and select Create Pipeline. Enter your Pipeline name, and select the role you created earlier under Service role. 

Under Advanced Settings, set the Artifact Store to Custom Location and update the field to reference the pipeline bucket you created earlier on. 

Click next, and move to Adding a Source Provider. Select the Source provider, Repository name and Branch name setup previously, and click next leaving everything else as default. 

On the Build section, select Build provider as AWS CodeBuild, and click Create Project under Project name 

This will open a new window. Codebuild will need to be created through this interface, otherwise it doesn’t support the artifacts, and source configuration correctly. 

Under Environment, select the latest Ubuntu Runtime for codebuild, and under Service role select the IAM role you created earlier. Once that’s all done, click Continue to CodePipeline and it will close out the window and your project will now be created. 

Click Next, and then Skip deploy stage (we’re doing it during the build stage!). Finally, click on create the pipeline and it will start running based on the work you’ve done so far.  

The website so far should now be available in the browser! Any further changes to the CodeCommit repository will result in the website being updated on S3 within minutes! 

The post Using CodePipeline to Host a Static Website appeared first on Anchor | Cloud Engineering Services.

Breaking the Bias – Women at AWS Developer Relations

Post Syndicated from Rashmi Nambiar original https://aws.amazon.com/blogs/aws/breaking-the-bias-women-at-aws-developer-relations/

Today for International Women’s Day we’re joined by a special guest writer, Rashmi Nambiar. She’s here to share her conversations with a few other members of the AWS Developer Relations team, talking about their work and experience as women in tech. Enjoy!

– The AWS News Blog Team


When I was contemplating joining AWS, many warned me about boarding the “rocket ship.” But I took the leap of faith. It has been four years since then. Now when I look back, the growth trajectory is something that I am proud of, from starting my AWS journey with a regional role in India to going global and now driving the Worldwide Developer Marketing Strategy. #HereatAWS, I get to choose the direction of my career and prioritize my time between family and work.

At AWS, we believe that the future of technology is accessible, flexible, and inclusive. So we take it very seriously when we say, “All Builders Welcome.” As a woman in tech, I have felt that strong sense of belonging with the team and acceptance for who I am.

Being part of the AWS Developer Relations (DevRel) team, I get to meet and work with awesome builders within and outside of the organization who are changing the status quo and pushing technological boundaries. This International Women’s Day, I took the opportunity to talk to some of the women at AWS DevRel about their role as tech/dev advocates.

Veliswa Boya

Headshot of Veliswa Boya

Veliswa Boya, Senior Developer Advocate

What is it that you like about being a developer advocate at AWS?
“Becoming a developer advocate is something I didn’t even dare to dream about. Some of us go through life and at some point admit that some dreams are just not meant for us. That today I am a developer advocate at AWS working with the builder community of sub-Saharan Africa and beyond is one of the most fulfilling and exciting roles I can recall throughout my entire tech career. I especially enjoy working with those new to AWS and new to tech in general, so my role spans technical content creation and delivery all the way to the mentoring of community members. I enjoy working for an organization that’s at the forefront of innovation, but at the same time not innovating for the sake of innovating, but always being customer obsessed and innovating on behalf of the customer.”

You are an icon of possibilities with many titles. How did the transition from AWS Hero to AWS employee work out for you?
“I became an AWS Hero in May 2020, and with that, I became the first woman out of Africa to ever be named an AWS Hero. I have always enjoyed sharing knowledge. Every little bit I learn, I always make sure to share. I believe that this—and more—led to my nomination. Joining AWS as a developer advocate is awesome. I continue to live the passion that led to me being a Hero, sharing knowledge with others and at the same time learning from both the community and my wonderful peers.”

Antje Barth

Headshot photo of Antje Barth

Antje Barth, Principal Developer Advocate – AI/ML

What do you like about your role as an AI/ML specialist on the AWS Developer Relations Team?
“I’ve always been excited about technology and the speed of innovation in this field. What I like most about my role as a principal developer advocate for AI/ML is that I get to share this passion and enable customers, developers, and students to build amazing things. I recently organized a hackathon asking participants to think about creative ways of using machine learning to improve disaster response. And I was simply blown away by all the ideas the teams came up with.”

You have authored books like Data Science on AWS. What is your guidance for someone planning to get on the publishing path?
“The piece of advice I would give anyone interested in becoming a book author: Find the topic you are really passionate about, dive into the topic, and start developing content—whether it’s blog posts, code samples, or videos. Become a subject matter expert and make yourself visible. Speak at meetups, submit a talk to a conference. Grow your network. Find peers, discuss your ideas, ask for feedback, make sure the topic is relevant for a large audience group. And eventually, reach out to publishers, share your content ideas and collected feedback, and put together a book proposal.”

Lena Hall

Headshot photo of Lena Hall

Lena Hall, Head of Developer Relations – North America

What excites you about AWS Developer Relations?
“I love it because AWS culture empowers anyone at AWS, including developer advocates, to always advocate for the customer and the community. While doing that, no matter how hard it is or how much friction you run into, you can be confident in doing the right thing for our customers and community. This translates to our ability to influence internally across the company, using strong data and logical narratives to support our improvement proposals.”

You have recently joined the team as the DevRel Head for North America. What does it take to lead a team of builders?
“It is important to recognize that people on your team have unique strengths and superpowers. I found it valuable to identify those early on and offer paths to develop them even more. In many cases, it leads to a bigger impact and improved motivation. It is also crucial to listen to your team, be supportive and welcoming of ideas, and protective of their time.”

Rohini Gaonkar

Headshot photo of Rohini Gaonkar

Rohini Gaonkar, Senior Developer Advocate

You have been with AWS for over eight years. What attracted you to developer advocacy?
“As a developer advocate, I love being autonomous, and I have the freedom to pick the tech of my choice. The other fun part is to work closely with the community—my efforts, however small, help someone in their career, and that is the most satisfying part of my work.”

You have worked in customer support, solutions architect, and technical evangelist roles. What’s your tip on developing multiple technical skills?
“Skills are like flowers in your bouquet; you should keep adding beautiful colors to it. Sometimes it takes months to years to develop a skill, so keep an eye on your next thing and start adding the skills for it today. Interestingly, at AWS, the ‘Learn and be curious’ leadership principle encourages us to always find ways to improve ourselves, to explore new possibilities and act on them.”

Jenna Pederson

Headshot photo of Jenna Pederson

Jenna Pederson, Senior Developer Advocate

What is your reason for taking up a developer advocate role at AWS?
“I like being a developer advocate at AWS because it lets me scale my impact. I get to work with and help so many more builders gain knowledge, level up their skills, and bring their ideas to life through technology.”

It is such a delight to watch your presentations and demo at events and other programs. What is your advice to people who want to get into public speaking?
“If you’re a new speaker, talk about what you’re learning, even if you think everyone is talking about the same thing. You will have a fresh perspective on whatever it is.”

Kris Howard

Headshot photo of Kris Howard

Kris Howard, DevRel Manager

Why did you join the Developer Relations Team?
“I joined DevRel because I love being on stage and sharing my creativity and passion for tech with others. The most rewarding part is when someone tells you that you inspired them to learn a new skill, or change their career, or stretch themselves to reach a new goal.”

Since you have worked in different geographies, what would you say to someone who is exploring working in different countries?
“The last two years have really emphasized that if you want to see the world, you should take advantage of every opportunity you get. That’s one of the benefits of Amazon: that there are so many career paths available to you in lots of different places! As a hiring manager, I was always excited to get applications from internal transfers, and in 2020 I got the chance to experience it from the other side when I moved with my partner from Sydney to Munich. It was a challenging time to relocate, but in retrospect, I’m so glad we did.”

Join Us!

Interested in working with DevRel Team? Here are some of the available opportunities.

Anchor | Cloud Engineering Services 2022-05-27 07:02:18

Post Syndicated from Gerald Bachlmayr original https://www.anchor.com.au/blog/2022/05/death-by-nodevops/

The CEO of ‘Waterfall & Silo’ walks into the meeting room and asks his three internal advisors: How are we progressing with our enterprise transformation towards DevOps, business agility and simplification? 

The well-prepared advisors, who had read at least a book and a half about organisational transformation and also watched a considerable number of Youtube videos, confidently reply: We are nearly there. We only need to get one more team on board. We have the first CI/CD pipelines established, and the containers are already up and running.

Unfortunately the advisors overlooked some details.

Two weeks later, the CEO asks the same question, and this time the response is: We only need to get two more teams on board, agree on some common tooling, the delivery methodology and relaunch our community of practice.

A month later, an executive decision is made to go back to the previous processes, tooling and perceived ‘customer focus’.

Two years later, the business closes its doors whilst other competitors achieve record revenues.

What has gone wrong, and why does this happen so often?

To answer this question, let’s have a look… 

Why do you need to transform your business?

Without transforming your business, you will run the risk of falling behind because you are potentially: 

  1. Dealing with the drag of outdated processes and ways of working. Therefore your organisation cannot react swiftly to new business opportunities and changing market trends.
  2. Wasting a lot of time and money on Undifferentiated heavy lifting (UHL). These are tasks that don’t differentiate your business from others but can be easily done better, faster and cheaper by someone else, for example, providing cloud infrastructure. Every minute you spend on UHL distracts you from focusing on your customer.
  3. Not focusing enough on what your customers need. If you don’t have sufficient data insights or experiment with new customer features, you will probably mainly focus on your competition. That makes you a follower. Customer-focused organisations will figure out earlier what works for them and what doesn’t. They will take the lead. 

How do you get started?

The biggest enablers for your transformation are the people in your business. If they work together in a collaborative way, they can leverage synergies and coach each other. This will ultimately motivate them. Delivering customer value is like in a team sport: not the team with the best player wins, but the team with the best strategy and overall team performance.  

How do we get there?

Establishing top-performing DevOps teams

Moving towards cross-functional DevOps teams, also called squads, helps to reduce manual hand-offs and waiting times in your delivery. It is also a very scalable model that is used by many modern organisations that have a good customer experience at their forefront. This applies to a variety of industries, from financial services to retail and professional services. Squad members have different skills and work together towards a shared outcome. A top-performing squad that understands the business goals will not only figure out how to deliver effectively but also how to simplify the solution and reduce Undifferentiated Heavy Lifting. A mature DevOps team will always try out new ways to solve problems. The experimental aspect is crucial for continuous improvement, and it keeps the team excited. Regular feedback in the form of metrics and retrospectives will make it easier for the team to know that they are on the right track.

Understand your customer needs and value chain

There are different methodologies to identify customer needs. Amazon has the “working backwards from the customer” methodology to come up with new ideas, and Google has the “design sprint” methodology. Identifying your actual opportunities and understanding the landscape you are operating in are big challenges. It is easy to get lost in detail and head in the wrong direction. Getting the strategy right is only one aspect of the bigger picture. You also need to get the execution right, experiment with new approaches and establish strong feedback loops between execution and strategy. 

This brings us to the next point that describes how we link those two aspects.

A bidirectional governance approach

DevOps teams operate autonomously and figure out how to best work together within their scope. They do not necessarily know what capabilities are required across the business. Hence you will need a governing working group that has complete visibility of this. That way, you can leverage synergies organisation-wide and not just within a squad. It is important that this working group gets feedback from the individual squads who are closer to specific business domains. One size does not fit all, and for some edge cases, you might need different technologies or delivery approaches. A bidirectional feedback loop will make sure you can improve customer focus and execution across the business.

Key takeaways

Establishing a mature DevOps model is a journey, and it may take some time. Each organisation and industry deals with different challenges, and therefore the journey does not always look the same. It is important to continuously tweak the approach and measure progress to make sure the customer focus can improve.

But if you don’t start the DevOps journey, you could turn into another ‘Waterfall & Silo’.

The post appeared first on Anchor | Cloud Engineering Services.

Our Top 4 Favorite Google Chrome DevTools Tips & Tricks

Post Syndicated from Andy Haine original https://www.anchor.com.au/blog/2020/10/our-top-4-favorite-google-chrome-devtools-tips-tricks/

Welcome to the final installation of our 3-part series on Google Chrome’s DevTools. In part 1 and part 2, we explored an introduction to using DevTools, as well as how you can use it to diagnose SSL and security issues on your site. In the third and final part of our DevTools series, we will be sharing our 4 favourite useful tips and tricks to help you achieve a variety of useful and efficient tasks with DevTools, without ever leaving your website!

Clearing All Site Data

Perhaps one of the most frustrating things when building your website, is the occasional menace that is browser caching. If you’ve put a lot of time into building websites, you probably know the feeling of making a change and then wondering why your site still shows the old page after you refresh. But even further to that, there can be any number of other reasons why you may need to clear all of your site data. Commonly, one might be inclined to just flush all cookies and cache settings in their browser, wiping their history from every website. This can lead to being suddenly logged out of all of your usual haunts – a frustrating inconvenience if you’re ever in a hurry to get something done.

Thankfully, DevTools has a handy little tool that allows you to clear all data related to the site that you’re currently on, without wiping anything else.

  1. Open up DevTools
  2. Click on the “Application” tab. If you can’t see it, just increase the width of your DevTools or click the arrow to view all available tabs.
  3. Click on the “Clear storage” tab under the “Application” heading.
  4. You will see how much local disk usage that specific website is taking up on your computer. To clear it all, just click on the “Clear site data” button.

That’s it! Your cache, cookies, and all other associated data for that website will be wiped out, without losing cached data for any other website.

Testing Device Responsiveness

In today’s world of websites, mobile devices make up more than half of all traffic to websites. That means that it’s more important than ever to ensure your website is fully responsive and looking sharp across all devices, not just desktop computers. Chrome DevTools has an incredibly useful tool to allow you to view your website as if you were viewing it on a mobile device.

  1. Open up DevTools.
  2. Click the “Toggle device toolbar” button on the top left corner. It looks like a tablet and mobile device. Alternatively, press Ctrl + Shift + M on Windows.
  3. At the top of your screen you should now see a dropdown of available devices to pick from, such as iPhone X. Selecting a device will adjust your screen’s ratio to that of the selected device.

Much easier than sneaking down to the Apple store to test out your site on every model of iPhone or iPad, right?

Viewing Console Errors

Sometimes you may experience an error on your site, and not know where to look for more information. This is where DevTools’ Console tab can come in very handy. If you experience any form of error on your site, you can likely follow these steps to find a lead on what to do to solve it:

  1. Open up DevTools
  2. Select the “Console” tab
  3. If your console logged any errors, you can find them here. You may see a 403 error, or a 500 error, etc. The console will generally log extra information too.

If you follow the above steps and you see a 403 error, you then know your action was not completed due to a permissions issue – which can get you started on the right track to troubleshooting the issue. Whatever the error(s) may be, there is usually a plethora of information available on potential solutions by individually researching those error codes or phrases on Google or your search engine of choice.

Edit Any Text On The Page

While you can right-click on text and choose “inspect element”, and then modify text that way, this alternative method allows you to modify any text on a website as if you were editing a regular document or a photoshop file, etc.

  1. Open up DevTools
  2. Go to the “Console” tab
  3. Copy and paste the following into the console and hit enter:
    1. document.designMode=”on”

Once that’s done, you can select any text on your page and edit it. This is actually one of the more fun DevTools features, and it can make testing text changes on your site an absolute breeze.

Conclusion

This concludes our entire DevTools series! We hope you’ve enjoyed it and maybe picked up a few new tools along the way. Our series only really scratches the surface of what DevTools can be used for, but we hope this has offered a useful introduction to some of the types of things you can accomplish. If you want to keep learning, be sure to head over to Google’s Chrome DevTools documentation for so much more!

The post Our Top 4 Favorite Google Chrome DevTools Tips & Tricks appeared first on AWS Managed Services by Anchor.