Vulnerability findings in a cloud environment can come from a variety of tools and scans depending on the underlying technology you’re using. Without processes in place to handle these findings, they can begin to mount, often leading to thousands to tens of thousands of findings in a short amount of time. We’re excited to announce the Building a scalable vulnerability management program on AWS guide, which includes how you can build a structured vulnerability management program, operationalize tooling, and scale your processes to handle a large number of findings from diverse sources.
Building a scalable vulnerability management program on AWS focuses on the fundamentals of building a cloud vulnerability management program, including traditional software and network vulnerabilities and cloud configuration risks. The guide covers how to build a successful and scalable vulnerability management program on AWS through preparation, enabling and configuring tools, triaging findings, and reporting.
Targeted outcomes
This guide can help you and your organization with the following:
Develop policies to streamline vulnerability management and maintain accountability.
Establish mechanisms to extend the responsibility of security to your application teams.
Configure relevant AWS services according to best practices for scalable vulnerability management.
Identify patterns for routing security findings to support a shared responsibility model.
Establish mechanisms to report on and iterate on your vulnerability management program.
Improve security finding visibility and help improve overall security posture.
Using the new guide
We encourage you to read the entire guide before taking action or building a list of changes to implement. After you read the guide, assess your current state compared to the action items and check off the items that you’ve already completed in the Next steps table. This will help you assess the current state of your AWS vulnerability management program. Then, plan short-term and long-term roadmaps based on your gaps, desired state, resources, and business needs. Building a cloud vulnerability management program often involves iteration, so you should prioritize key items and regularly revisit your backlog to keep up with technology changes and your business requirements.
We greatly value feedback and contributions from our community. To share your thoughts and insights about the guide, your experience using it, and what you want to see in future versions, select Provide feedback at the bottom of any page in the guide and complete the form.
If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.
Want more AWS Security news? Follow us on Twitter.
The Free Software Foundation Europe and the Software Freedom Conservancy recently released a statement that they would no longer work with Eben Moglen, chairman of the Software Freedom Law Center. Eben was the general counsel for the Free Software Foundation for over 20 years, and was centrally involved in the development of version 3 of the GNU General Public License. He’s devoted a great deal of his life to furthering free software.
But, as described in the joint statement, he’s also acted abusively towards other members of the free software community. He’s acted poorly towards his own staff. In a professional context, he’s used graphically violent rhetoric to describe people he dislikes. He’s screamed abuse at people attempting to do their job.
And, sadly, none of this comes as a surprise to me. As I wrote in 2017, after it became clear that Eben’s opinions diverged sufficiently from the FSF’s that he could no longer act as general counsel, he responded by threatening an FSF board member at an FSF-run event (various members of the board were willing to tolerate this, which is what led to me quitting the board). There’s over a decade’s evidence of Eben engaging in abusive behaviour towards members of the free software community, be they staff, colleagues, or just volunteers trying to make the world a better place.
When we build communities that tolerate abuse, we exclude anyone unwilling to tolerate being abused[1]. Nobody in the free software community should be expected to deal with being screamed at or threatened. Nobody should be afraid that they’re about to have their sexuality outed by a former boss.
But of course there are some that will defend Eben based on his past contributions. There were people who were willing to defend Hans Reiser on that basis. We need to be clear that what these people are defending is not free software – it’s the right for abusers to abuse. And in the long term, that’s bad for free software.
[1] “Why don’t people just get better at tolerating abuse?” is a terrible response to this. Why don’t abusers stop abusing? There’s fewer of them, and it should be easier.
The Civil Infrastructure Platform project has announced
that it will be maintaining the 6.1 kernel for a minimum of ten years past
its initial release (and, thus, through 2032).
CIP kernels are maintained like regular long-term-stable (LTS)
kernels, and developers of the CIP kernel are also involved in LTS
kernel review and testing. While regular LTS kernels are moving
back to 2 years maintenance, CIP kernels are set up for 10
years. In order to enable this extended lifetime, CIP kernels are
scoped-down in actively supported kernel features and target
architecture. At the same time, CIP kernels accept non-invasive
backports from newer mainline kernels that enable new hardware.
When it comes to navigating the treacherous landscape of a household’s digital ecosystem, from smartphones and laptops to smart homes and millions of subscriptions, there often emerges a silent hero—the ever-humble, quietly toiling, underappreciated Family IT Manager. This unsung role, typically filled by a tech-savvy-est member of the family, takes on the responsibility of keeping everyone’s digital lives running smoothly. Maybe you know one of these vaillant souls. Or maybe, just maybe, it’s you.
As the Family IT Manager, having one more arrow in your quiver with which to slay the dreaded data loss dragon is always helpful. And that’s what Backblaze Groups is all about—making it easier for you to keep track of everyone’s data in one place.
Today, we’re sharing some practical tips and tricks for using Groups to better manage your family’s backups.
Groups helps you manage the backups your family creates without having to log in and out of individual accounts. This makes it simple to keep track of everyone in one place. All the backup accounts are linked to the same credit card (they can Venmo you later), and you can even help someone else in your family create a backup or restore files easily with Groups. Need to help a family member with a computer emergency? Log in, access their most recent backup, and restore everything. Is your sibling unsure that you really added Backblaze to their computer? Log in, view their account, and get the screenshots to prove it to them (and everyone else).
By the way, this would be a great time to give the new Restore App, included with Backblaze Computer Backup v9.0, a spin.
One point of clarification: You might see Backblaze Groups referred to as “Business Groups,” but you don’t have to be a business to use Groups. They work equally well for businesses and personal users alike, including Family IT Managers (and, truly, running family IT is kind of like running a business, isn’t it?).
Why Use Groups?
You can already manage multiple computers on a single Backblaze account. So why use Groups instead? Well, with Groups, each user has individual access to, and control of, their account. You—as Group administrator—manage billing and, as needed, data recovery. This is a more secure and safer method than sharing the same account credentials among several computers used by different people.
Have multiple households or groupings of folks in your life that you need to manage? You can have as many Groups as you like to help you keep track of everyone and everything, and each of those Groups can have separate billing.
What Do I Need to Know About Setting Up Backblaze Groups for My Family?
The Groups feature streamlines the management of the accounts you need to monitor. As the Group administrator, you have total control over who’s included as part of your Group. You can send out email invitations, or alternatively, you can use a unique Group invitation link that allows anyone you share it with to easily join.
Here’s the visual of where you’d find everything in your account.
Being in a Group is entirely voluntary. Any member of a Group can leave any time they want, and Group administrators can also remove individuals from a Group at any time.
If you dissolve your Group for some reason or if someone chooses to leave, the removed person can decide whether they want to keep using Backblaze by establishing their own payment method. Perfect for when it’s time to wean the kiddos off of your shared accounts—whether they like it or not.
One last note: while you can set up and administer more than one Group with separate billing, you can only be a member in one Group.
Once you create a Group, you can invite members to join it. Copy the Group invite link Backblaze generates automatically for you. Give it to friends and family via email, chat, or any other means you’d like.
We promise to send the emails. You may have to remind them to check their email.
When the person you’ve invited clicks on the link, they will be prompted to either create a Backblaze account (if they don’t have one) or log in to their existing account. After completing this step, they will be prompted to download Backblaze. If they are already using Backblaze, there is no need for a reinstallation; they will seamlessly become a part of your Group.
Once an existing user successfully joins your Group, they’ll be under your billing account. Their existing credit card will automatically receive a prorated refund for the remaining portion of their previous Backblaze license. There is no need to worry about re-uploading data—their backup remains securely stored in Backblaze.
Newcomers to Backblaze can download and install the client to initiate their initial backup process. As the Group administrator, you will have the capability to monitor their backup progress. Remember that the first backup of data may take some time, but after that, everything will run smoothly in the background.
Go Forth and Conquer, Mighty IT Manager
We understand that being the go-to “tech person” for your family and friends can be challenging. We hope that Groups simplifies the process, making it easier for you to help keep your family’s data safe.
Amazon Elastic Compute Cloud (Amazon EC2) now supports replacing the root volume on a running EC2 Mac instance, enabling you to restore the root volume of an EC2 Mac instance to its initial launch state, to a specific snapshot, or to a new Amazon Machine Image (AMI).
Just like for every EC2 instance type, AWS is responsible for protecting the infrastructure that runs all of the services offered in the AWS cloud. To ensure that EC2 Mac instances provide the same security and data privacy as other Nitro-based EC2 instances, Amazon EC2 performs a scrubbing workflow on the underlying Dedicated Host as soon as you stop or terminate an instance. This scrubbing process erases the internal SSD, clears the persistent NVRAM variables, and updates the device firmware to the latest version enabling you to run the latest macOS AMIs. The documentation has more details about this process.
The scrubbing process ensures a sanitized dedicated host for each EC2 Mac instance launch and takes some time to complete. Our customers have shared two use cases where they may need to set back their instance to a previous state in a shorter time period or without the need to initiate the scrubbing workflow. The first use case is when patching an existing disk image to bring OS-level or applications-level updates to your fleet, without manually patching individual instances in-place. The second use case is during continuous integration and continuous deployment (CI/CD) when you need to restore an Amazon EC2 Mac instance to a defined well-known state at the end of a build.
To restart your EC2 Mac instance in its initial state without stopping or terminating them, we created the ability to replace the root volume of an Amazon EC2 Mac instance with another EBS volume. This new EBS volume is created either from a new AMI, an Amazon EBS Snapshot, or from the initial volume state during boot.
You just swap the root volume with a new one and initiate a reboot at OS-level. Local data, additional attached EBS volumes, networking configurations, and IAM profiles are all preserved. Additional EBS volumes attached to the instance are also preserved, as well as the instance IP addresses, IAM policies, and security groups.
Let’s see how Replace Root Volume works
To prepare and initiate an Amazon EBS root volume replacement, you can use the AWS Management Console, the AWS Command Line Interface (AWS CLI), or one of our AWS SDKs. For this demo, I used the AWS CLI to show how you can automate the entire process.
To start the demo, I first allocate a Dedicated Host and then start an EC2 Mac instance, SSH-connect to it, and install the latest version of Xcode. I use the open-source xcodeinstall CLI tool to download and install Xcode. Typically, you also download, install, and configure a build agent and additional build tools or libraries as required by your build pipelines.
Once the instance is ready, I create an Amazon Machine Image (AMI). AMIs are disk images you can reuse to launch additional and identical EC2 Mac instances. This can be done from any machine that has the credentials to make API calls on your AWS account. In the following, you can see the commands I issued from my laptop’s Terminal application.
#
# Find the instance’s ID based on the instance name tag
#
~ aws ec2 describe-instances \
--filters "Name=tag:Name,Values=RRV-Demo" \
--query "Reservations[].Instances[].InstanceId" \
--output text
i-0fb8ffd5dbfdd5384
#
# Create an AMI based on this instance
#
~ aws ec2 create-image \
--instance-id i-0fb8ffd5dbfdd5384 \
--name "macOS_13.3_Gold_AMI" \
--description "macOS 13.2 with Xcode 13.4.1"
{
"ImageId": "ami-0012e59ed047168e4"
}
It takes a few minutes to complete the AMI creation process.
After I created this AMI, I can use my instance as usual. I can use it to build, test, and distribute my application, or make any other changes on the root volume.
When I want to reset the instance to the state of my AMI, I initiate the replace root volume operation:
After a short time, the instance becomes available again, and I can connect over ssh.
~ ssh [email protected]
Warning: Permanently added '3.0.0.86' (ED25519) to the list of known hosts.
Last login: Wed May 24 18:13:42 2023 from 81.0.0.0
┌───┬──┐ | |_ )
│ ╷╭╯╷ │ _| ( /
│ └╮ │ |\ | |
│ ╰─┼╯ │ Amazon EC2
└───┴──┘ macOS Ventura 13.2.1
ec2-user@ip-172-31-58-100 ~ %
Additional thoughts
There are a couple of additional points to know before using this new capability:
By default, the old root volume is preserved. You can pass the –-delete-replaced-root-volume option to delete it automatically. Do not forget to delete old volumes and their corresponding Amazon EBS Snapshots when you don’t need them anymore to avoid being charged for them.
During the replacement, the instance will be unable to respond to health checks and hence might be marked as unhealthy if placed inside an Auto Scaled Group. You can write a custom health check to change that behavior.
When replacing the root volume with an AMI, the AMI must have the same product code, billing information, architecture type, and virtualization type as that of the instance.
When replacing the root volume with a snapshot, you must use snapshots from the same lineage as the instance’s current root volume.
The size of the new volume is the largest of the AMI’s block device mapping and the size of the old Amazon EBS root volume.
Any non-root Amazon EBS volume stays attached to the instance.
Finally, the content of the instance store (the internal SSD drive) is untouched, and all other meta-data of the instance are unmodified (the IP addresses, ENI, IAM policies etc.).
Pricing and availability
Replace Root Volume for EC2 Mac is available in all AWS Regions where Amazon EC2 Mac instances are available. There is no additional cost to use this capability. You are charged for the storage consumed by the Amazon EBS Snapshots and AMIs.
Programs running in the BPF machine can, depending on how they are
attached, perform a number of privileged operations; the ability to load
and run those programs, thus, must be a privileged operation in its own
right. Almost since the beginning of the extended-BPF era, developers have
struggled to find a way to allow users to run the programs they need
without giving away more privilege than is necessary. Earlier this year,
the idea of a BPF token ran into some
opposition from security-oriented developers. Andrii Nakryiko has since
returned with an
updated patch set that significantly increases the granularity of the
privileges that can be conferred with a BPF token.
This post is written by Rajdeep Banerjee, Sr PSA, and Brian Krygsman, Sr. Solutions Architect.
Amazon EventBridge recently announced support for wildcard filters in rule event patterns. An EventBridge event bus is a serverless event router that helps you decouple your event-driven systems. You can route events between your systems, AWS services, or third-party SaaS services. You attach a rule to your event bus to define logic for routing events from producers to consumers.
You set an event pattern on the rule to filter incoming events to specific consumers. The new wildcard filter lets you build more flexible event matching patterns to reduce rule management and optimize your event consumers. This shows how these EventBridge attributes work together.
Wildcard filters use the wildcard character (*) to match zero, single, or multiple characters in a string value. For example, a filter string like "*.png" matches strings that end with ".png".
You can also use multiple wildcard characters in a filter. For example, a filter string like "*Title*" matches string values that include "Title" in the middle. When using wildcard filters, be careful to avoid matching more events than you intend.
This blog post describes how you can use wildcard filters in example scenarios. For more information about event-driven architectures, visit Serverless Land.
Wildcard pattern matching in S3 Event Notifications
Applications must often perform an action when new data is available. One example can be to process trading data uploaded to your Amazon S3 bucket. The data may be stored in individual folders depending on the date, time, and stock symbol. Business rules may dictate that when stock XYZ receives a file, it must send a notification to a downstream system.
This is the typical folder structure in an S3 bucket:
S3 can send an event to EventBridge when an object is written to a bucket. The S3 event includes the object key (for example, 2023-10-01/T13:22:22Z/XYZ/filename.ext). When any object is uploaded to the XYZ folder, you can use an EventBridge rule to send these events to a downstream service like an Amazon SQS.
Before this launch, you would first send the event to an AWS Lambda function. Existing prefix and suffix filters alone are insufficient because of the extra date and time folders. The function would run your code to inspect the object path for the stock symbol. Your code would then forward events to SQS when they matched.
With the new wildcard patterns in EventBridge rules, the logic is simpler. You no longer need to create a Lambda function to run custom matching code. You can instead use wildcard characters in the rule’s filter pattern, matching against portions of the S3 object key.
To use this, start with creating a new rule in the EventBridge console:
Choose Next. Keep the standard parameters and move to the Event pattern section. Here you can use a JSON-based event pattern.
This pattern looks for Event Notifications from a specific bucket. The pattern then filters the events further by the object keys that match "*/XYZ/*". The rule filters out notifications from other stock symbols, listening to only “XYZ“ data, irrespective of date and time of the data feed.
To use an SQS queue for the filtered event target, you must provide resource-based policies for EventBridge to send messages to the queue.
Choose Next and review the rule details before saving.
Customers often need to perform actions when AWS Identity and Access Management (IAM) policies are added to specific roles. You can achieve this by creating custom EventBridge rules, which filter the event to match or create multiple rules to achieve the same effect. With the newly introduced wildcard filter, the task to invoke an action is simplified.
Consider an IAM role with fine-grained IAM policies attached. You may need to ensure any new policy attached to this role must be from a specific ARNs. This action can be implemented like this.
When you attach a new IAM policy to a role, it generates an event like this:
You can create a rule matching against a combination of these event properties. You can filter detail.userIdentity.arn with a wildcard to catch events that come from a particular ARN. You can then route these events to a target like an Amazon CloudWatch Logs stream to record the change. You can also route them to Amazon Simple Notification Service (SNS). You can use the SNS notification to start a review and ensure that the newly attached policies are well-crafted as part of your reconciliation and audit process. The filter looks like this:
You can use EventBridge to build your own event-driven systems with loosely coupled, scalable application services. When building event-driven applications in AWS, you can publish events to the default event bus, or create a custom event bus. You define the structure of events emitted from your services.
This structure is known as the event schema. When you attach rules to your bus to route events from producers to consumers, you match against values from properties in your event schema. Wildcard filters allow you to match property values that are unknown ahead of time, or across multiple value variants.
Consider an ecommerce application as an example. You may have several decoupled services working together, like a shopping cart service, an inventory service, and others. Each of these services emits events onto your event bus as your customers shop.
Events may include errors, to record problems customers encounter using your system. You can use a single rule with a wildcard filter to match all error events and send them to a common target. This allows you to simplify observability across your services.
This is the event flow:
Your shopping cart service may emit a timeout error event:
The detail-type property of the example event determines what type of event this is. Other services may emit error events with different prefixes in detail-type. Other error types might have different suffixes in detail-type.
For example, an inventory service may emit an out-of-stock error event like this:
{
"version": "0",
"id": "e456f480-cc1e-47fa-8399-ab2e54116958",
"detail-type": "shopping.inventory.error.outofstock",
"source": "com.mybusiness.shopping.inventory",
"account": "123456789012",
"time": "2023-10-06T03:28:44Z",
"region": "us-west-2",
"resources": [],
"detail": {
"message": "Product cannot be added to a cart. Out of stock.",
"related-entity": {
"entity-type": "product",
"id": "456"
}
// ... additional detail fields
}
}
To route these events to a common target like an Amazon CloudWatch Logs stream, you can create a rule with a wildcard filter matching against detail-type. You can combine this with a prefix filter on source that filters events down to only services from your shopping system. The filter looks like this:
Wildcard filters in EventBridge rules help simplify your event driven applications by ensuring the correct events are passed on to your targets. The new feature reduces the need for custom code, which was required previously. Try EventBridge rules with wildcard filters and experience the benefits of this new feature in your event-driven serverless applications.
For more serverless learning resources, visit Serverless Land.
The following is a guest post written by Pierre-Antoine Mills, Miguel Fernández, and Petra Donka of Prisma. Prisma provides a server-side library that helps developers read and write data to the database in an intuitive, efficient and safe way.
Prisma’s mission is to redefine how developers build data-driven applications. At its core, Prisma provides an open-source, next-generation TypeScript Object-Relational Mapping (ORM) library that unlocks a new level of developer experience thanks to its intuitive data model, migrations, type-safety, and auto-completion.
Prisma ORM has experienced remarkable growth, engaging a vibrant community of developers. And while it was a great problem to have, this growth was causing an explosion in our AWS infrastructure costs. After investigating a wide range of alternatives, we went with Cloudflare’s R2 storage — and as a result are thrilled that our engine distribution costs have decreased by 98%, while delivering top-notch performance.
It was a natural fit: Prisma is already a proud technology partner of Cloudflare’s, offering deep database integration with Cloudflare Workers. And Cloudflare products provide much of the underlying infrastructure for Prisma Accelerate and Prisma Pulse, empowering user-focused product development. In this post, we’ll dig into how we decided to extend our ongoing collaboration with Cloudflare to the Prisma ORM, and how we migrated from AWS S3 + CloudFront to Cloudflare R2, with zero downtime.
Distributing the Prisma ORM and its engines
Prisma ORM simplifies data access thanks to its type-safe Prisma Client, and enables efficient database management via the Prisma CLI, so that developers can focus on product development.
Both the Prisma Client and the Prisma CLI rely on the Prisma Engines, which are implemented in Rust and distributed as platform-specific compiled binaries. The Prisma Engines perform a variety of tasks ranging from providing information about the schema for type generation, or migrating the database, to transforming Prisma queries into SQL, and executing those queries against the database. Think of the engines as the layer in the Prisma ORM that talks to the database.
As a developer, one of the first steps to get started with Prisma is to install Prisma Client and the Prisma CLI from npm. Once installed, these packages need the Prisma Engines to be able to function. These engines have complex target-platform rules and were originally envisioned to be distributed separately from the npm package, so they can be used outside of the Node.js ecosystem. As a result, they are downloaded on demand by the Prisma CLI, only downloading what is strictly required for a given project.
As of mid-2023, the engines account for 100 million downloads a month and 250 terabytes of egress data transfer, with a continuous month-over-month increase as our user base grows. This highlights the importance of a highly available, global, and scalable infrastructure that provides low latency engine downloads to Prisma users all around the world.
Our original solution: AWS S3 & CloudFront
During the early development of the Prisma ORM, our engineering team looked for tools to build the CDN for engine distribution. With extensive AWS experience, we went with the obvious: S3 blob storage for the engine files and CloudFront to cache contents closer to the user.
A simplified representation of how the Prisma Engines flow from our CI where they are built and uploaded, to the Prisma CLI downloading the correct engine for a given environment when installing Prisma, all the way to the user being able to use it.
We were happy with AWS for the most part, and it was able to scale with our demands. However, as our user base continued to grow, so did the costs. At our scale of traffic, data transfer became a considerable cost item that we knew would only continue to grow.
The continuously increasing cost of these services prompted us to explore alternative options that could better accommodate our needs while at least maintaining the same level of performance and reliability. Prisma is committed to providing the best products and solutions to our users, and an essential part of that commitment is being intentional about the allocation of our resources, including sensible spending to enable us to serve our growing user base in the best way possible.
Exploring distribution options
We extensively explored different technologies and services that provided both reliable and fast engine distribution, while being cost-effective.
Free solutions: GitHub & npm
Because Prisma ORM is an open-source solution, we have explored various ways to distribute the engines through our existing distribution channels, at no cost. In this area, we had both GitHub Releases and npm as candidates to host and distribute our engine files. We dismissed GitHub Releases early on as the quality of service was not guaranteed, which was a requirement for us towards our users, so we can be sure to provide a good developer experience under all circumstances.
We also looked at npm, and confirmed that hosting the engine files would be in agreement with their Terms of Service. This made npm a viable option, but also meant we would have to change our engine download and upload logic to accommodate a different system. Additionally, this implied that we would have to update many past Prisma CLI versions, requiring our users to upgrade to take advantage of the new solution.
Paid solutions: CDNs & Cloudflare
We then considered only replacing CloudFront, which accounted for 97% of our distribution costs, while retaining S3 as the origin. When we evaluated different CDNs, we found that alternatives could lead to an estimated 70% cost reduction.
We also explored Cloudflare’s offerings and were impressed byCloudflare R2, an alternative to AWS S3 + CloudFront. It offers robust blob storage compatible with S3 and leverages Cloudflare’s network for global low-latency distribution. Additionally, it has no egress costs, and is solely priced based on the total volume of data stored and operations on that data. Given our reliance on Cloudflare’sproduct portfolio for ourData Platform, and extensive experience with their Workers platform, we already had high trust in the quality of Cloudflare’s products.
To finalize our decision, we implemented a test to confirm our intuitions about Cloudflare’s quality of service. We deployed a script to 50 cities across the globe, representative of our incoming traffic, to measure download latencies for our engine files (~15MB). The test was run multiple times, with latencies for the different cache statuses recorded and compared against our previous AWS-based solution. The results confirmed that Cloudflare R2's reliability and performance were at least on par with AWS S3 + CloudFront. And because R2 is compatible with S3, we wouldn’t need to make substantial changes to our software in order to move over to Cloudflare. These were great results, and we couldn’t wait to switch!
Our solution: moving to Cloudflare’s R2
In order to move our engine file distribution to Cloudflare, we needed to ensure we could make the switch without any disruption or impact to our users.
While R2 URLs match S3's format, Prisma CLI uses a fixed domain to point to the engine file distribution. This fixed domain enabled us to transition without making any changes to the code of older Prisma versions, and simply point the existing URLs to R2. However, to make the transition, we needed to change our DNS configuration to point to Cloudflare. While this seems trivial, potential issues like unexpected DNS propagation challenges, or certificate validation problems when connecting via TLS, required us to plan ahead in order to proceed confidently and safely.
We modified the Prisma ORM release pipeline to upload assets to both S3 and R2, and used the R2 Super Slurper for migrating past engine versions to R2. This ensured all Prisma releases, past and future, existed in both places. We also established Grafana monitoring checks to pull engine files from R2, using a DNS and TLS configuration similar to our desired production setup, but via an experimental domain. Those monitoring checks were later reused during the final traffic cutover to ensure that there was no service disruption.
As ensuring no impact or disruption to our users was of utmost importance, we proceeded with a gradual rollout of the DNS changes using DNS load balancing, a method where a group of alias records assigned to a domain are weighted differently. This meant that the DNS resolver directed more traffic to heavier-weighted records. We began with a load balancing configuration simulating our old setup, with one record (the control) pointing to AWS CloudFront, and the other (the candidate) pointing to R2. Initially, all weight was on the control, effectively preserving the old routing to CloudFront. We also set the lowest TTL possible, so changes in the record weights took effect as soon as possible, creating more control over DNS propagation. Additionally, we implemented a health check that would redirect all traffic to the control if download latencies were significantly higher, or if errors were detected, ensuring a stable fallback.
At this point, everything was in place and we could start the rollout.
Our DNS load balancing setup during the rollout. We assigned increasing weights to route traffic to Cloudflare R2. The health check that would fail over to AWS CloudFront never fired.
The rollout began with a gradual increase in R2's DNS weight, and our monitoring dashboards showed that Cloudflare downloads were proportional to the weight assigned to R2. With as little as 5% traffic routed to Cloudflare, cache hit ratios neared 100%, as expected. Latencies matched the control, so the health checks were all good, and our fallback never activated. Over the duration of an hour, we gradually increased R2's DNS weight to manage 25%, 50%, and finally 100% of traffic, without any issues. The cutover could not have gone any smoother.
After monitoring for an additional two days, we simplified the DNS topology and routed to Cloudflare exclusively. We were extremely satisfied with the change, and started seeing our infrastructure costs drop considerably, as expected, not to mention the zero downtime and zero reported issues from users.
A success
Transitioning to Cloudflare R2 was easy thanks to their great product and tooling, intuitive platform and supportive team. We've had an excellent experience with their service, with consistently great uptime, performance and latency. Cloudflare proved once again to be a valuable partner to help us scale.
We are thrilled that our engine distribution costs have decreased by 98%. Cloudflare's cost-effective solution has not only delivered top-notch performance but has also brought significant savings to our operations. An all around success!
Autumn in the Latvian capital of Riga is marked by a variety of traditions. The leaves fall, the rainy season arrives, the birds migrate, and IT professionals from around the world descend on the city for the annual Zabbix Summit.
On October 6 and 7, the Radisson Blu Hotel Latvija was packed with 450 delegates from 38 countries, all there for Zabbix Summit 2023, the 11th in-person version of Zabbix’s premier yearly event.
This year’s Summit was marked by presentations, partner activities, and moments of relaxation and celebration that will energize the Zabbix community and spark ideas that attendees will take home to every corner of the world.
If you couldn’t make it, here’s a little taste of how it felt to be there!
Zabbix Summit 2023 in numbers
The stage hosted 27 speakers from 17 different countries who gave 31 speeches, including both lectures and lightning talks. There were four workshops with deep dives into technical topics, conducted by the Zabbix technical team as well as our partners from Opensource ICT Solutions and IZI-IT. Summit attendees also enjoyed three parties designed to provide a relaxing experience and networking opportunities.
Zabbix Summit 2023 proudly featured 10 sponsors, all part of Zabbix’s official partner network. They included:
We’d also like to give a shout-out to our Zabbix Fans, who played a crucial role in supporting the Summit this year (as every year) with their attendance, merchandise purchases, and enthusiasm!
We’re grateful to everyone who played a role and helped us make Zabbix Summit 2023 happen!
Highlights from the main stage
This year we continued a Summit tradition and allowed our in-person audience as well those tuning in via livestream and YouTube to ask questions during live Q&A sessions – a feature that made the proceedings more interactive and helped everyone feel more involved. The speeches were all fascinating and well received, but a few in particular stood out:
What the future holds for Zabbix
Zabbix CEO and Founder Alexei Vladishev kicked off the presentations on Day 1 with a keynote speech about his current plans for Zabbix’s development, including a detailed look at enhancements requested by users.
Avoiding alert fatigue
Bringing a less technical and more conceptual approach to addressing day-to-day data monitoring issues, Rihards Olups, SaaS Architect at Nokia, discussed alert fatigue and how science explains it. During his presentation, Rihards showed how an excess of alerts can negatively affect selective attention and shared his thoughts about how professionals can intervene to prevent problems.
Making Zabbix’s latest offerings accessible to everyone
Day 2 began with Zabbix Director of Business Development Sergey Sorokin focusing on new plans and offerings, including a subscription system for technical support, consulting services, and monitoring tailored for managed service providers.
Monitoring everything (and we do mean everything!)
Janne Pikkarainen, Lead Site Reliability Engineer at Forcepoint, provided detailed and entertaining insights into how he connects Zabbix to smart accessories and uses it to monitor aspects of his home, including the location of personal items, noise levels, and even the frequency of his daughter’s naps and cries.
Implementing ideas and design in MSP environments
In tackling the topic of data collection and analysis for service providers, Brian van Baekel, Zabbix Trainer at Opensource ICT Solutions, presented details on the development of projects focused on monitoring service providers. He also highlighted best practices for data collection in Zabbix Server, data storage, and presenting on the Zabbix Frontend.
Monitoring the London transportation system
A use case presented by Nathan Liefting, Zabbix Consultant and Trainer at Opensource ICT Solutions, and Adan Mohamed, DevOps Manager at Boldyn Networks, showed how Zabbix monitors the availability of the London Underground subway system. Data is collected from 136 “tube” stations in a high-level architecture and used to assess the availability of Wi-Fi networks, emergency connections, and other services.
Bringing the Olympics and World Cup to life with Zabbix
Marianna Portela, a Tech Lead at Globo in Brazil, shared her insights into how Zabbix supports Globo’s digital transformation and helps her monitor live event infrastructure at massive events like the Olympics and World Cup.
Don’t forget the fun part!
Zabbix Summits are renowned for their friendly, informal atmosphere, which is probably most clearly on display at our famous Summit parties.
Zabbix Summit 2023’s Welcome party was held at the Stargorod Riga brewery in the heart of Riga’s old town. It featured arm wrestling, a selection of delicious foods and beverages, and plenty of opportunities for Summit participants to get to know each other.
The Main party saw live music, dancing, quizzes, and other fun events take place within the historic confines of the Latvian Railway History Museum. The atmosphere, food, drinks, and good company all combined to create an event that nobody who attended will soon forget!
Last but not least, the Closing party at the Burzma food hall was a true celebration of the diversity of the global Zabbix community, with food and music from every country with a Zabbix presence as well as plenty of opportunities for Summit attendees to swap stories and exchange contact details.
Open door, open minds
The traditional Zabbix open-door day was held on Thursday, October 5, and while past Summits have typically seen around 50 visitors, we were proud to welcome closer to 100 this time around. Attendees could have a coffee with their favorite Zabbix employees, play a friendly game of foosball or table tennis, and get a behind-the-scenes look at where the magic happens.
Testify!
One new feature that made a big splash at this year’s Summit was the testimonial booth, which allowed Summit attendees to share their thoughts and experiences about Zabbix with the rest of our community. Sharing a testimonial or leaving a review allowed attendees to collect a piece of exclusive Zabbix Summit 2023 merchandise, and we went through a lot of it – the booth provided us with 28 filmed and 17 written testimonials about Zabbix products and services, far more than we anticipated.
Where to find the presentations
If you couldn’t attend but want to stay informed about what was discussed at the event (or if you’d just like to revisit the stage presentations), both days of recordings are available on Zabbix’s YouTube channel at the following links:
The graphics and texts of the presentations are also available for reference and download on the official event website.
We hope that Zabbix Summit 2023 was a time of valuable learning, connections, and idea exchange for everyone who attended or followed along through social media. If you’ve enjoyed the photos, you can see several more on our Instagram.
If you had an amazing time at Zabbix Summit 2023 (and we certainly hope you did), registration for Zabbix Summit 2024 is already open and Early Bird tickets are available.
Миналата седмица гласувахме на второ четене много промени в Закона за общетвените поръчки. Промените са значителни, по целия закон, но нека да откроя няколко важни промени, които бяха приети и които са по наше предложения:
Публикуване на договорите, сключени без обществена поръчка (над 50 хил лв.). Има редица изключения, при които не се правят обществени поръчки и досега оставаха скрити. Освен договорите, ще трябва да се публикуват и анексите към тях, както и приложенията, тъй като през анекси и приложения понякога се крият важни детайли по сключени догвоори. Останалите (под 50 хил., което е разумен праг) плащания ще се публикуват така или иначе с данните от СЕБРА. Тук не е важна просто сумата, а условията, по които е платена.
Публикуване на договорите за телевизионно време над 10 хил. лв и пълна разбивка на разплатените средства по доставчици на медийни услуги при такива договори и при поръчки с агенции-посредници, които разпределят парите към множество медиии. С това ограничаваме възможността на властта (която и да е тя) скрито да насочва пари към медии и така покриваме критика в доклада на Европейската комисия за върховенството на закона във връзка със свободата на медиите.
Данни за поръчките ще се събират в структуриран вид – стойност при сключване, стойност при изменение, единични цени на стоки и услуги, ценовите предложения, процент участие на фирми в консорциуми, стойности на договори с подизпълнители, брой подадени оферти и др. След като се съберат, данните се публикуват като отворени данни по международен стандарт. Това ще позволи много по-детайлен анализ на обществените поръчки от активисти, журналисти, анализатори. Промяната влиза в сила след 1г, в която да бъде надградена системата за електронни обществени поръчки
Веднъж годишно действителните собственици и членовете на управителните органи на дружеств, които са получавали над половин милион лева, ще бъдат проверявани за принадлежност към Държавна сигурност от Комисията по досиетата.
Продължаваме систематично да правим държавата по-прозрачна и по-ефективна.
The NIST elliptic curves that power much of modern cryptography were generated in the late ’90s by hashing seeds provided by the NSA. How were the seeds generated? Rumor has it that they are in turn hashes of English sentences, but the person who picked them, Dr. Jerry Solinas, passed away in early 2023 leaving behind a cryptographic mystery, some conspiracy theories, and an historical password cracking challenge.
So there’s a $12K prize to recover the hash seeds.
Some of the backstory here (it’s the funniest fucking backstory ever): it’s lately been circulating—though I think this may have been somewhat common knowledge among practitioners, though definitely not to me—that the “random” seeds for the NIST P-curves, generated in the 1990s by Jerry Solinas at NSA, were simply SHA1 hashes of some variation of the string “Give Jerry a raise”.
At the time, the “pass a string through SHA1” thing was meant to increase confidence in the curve seeds; the idea was that SHA1 would destroy any possible structure in the seed, so NSA couldn’t have selected a deliberately weak seed. Of course, NIST/NSA then set about destroying its reputation in the 2000’s, and this explanation wasn’t nearly enough to quell conspiracy theories.
But when Jerry Solinas went back to reconstruct the seeds, so NIST could demonstrate that the seeds really were benign, he found that he’d forgotten the string he used!
If you’re a true conspiracist, you’re certain nobody is going to find a string that generates any of these seeds. On the flip side, if anyone does find them, that’ll be a pretty devastating blow to the theory that the NIST P-curves were maliciously generated—even for people totally unfamiliar with basic curve math.
Note that this is not the constants used in the Dual_EC_PRNG random-number generator that the NSA backdoored. This is something different.
Елате, елате ще ви покажа петното от кръв на стълбите, където простреляха крака на сина ми. Не мога да разказвам, без да плача…
С тези думи ме посреща Люба, докато влизаме в къщата ѝ в Гостомел. Градина, слънчогледи, домати, чушки и лаещо куче. Там са и две нейни съседки. Една през друга започват да разказват за дните на окупацията, припомнят си, окуражават се, плачат, замлъкват, въздъхват и продължават.
Когато колоните с танкове и БТР-и тръгнаха по нашата улица, не повярвах на очите си и много се изплаших. Спряха срещу къщите ни и първото, което направиха руснаците, беше да прострелят мъжете в краката. Знаеха, че нашите войски са някъде отпред, и явно се страхувахамъжете ни да не започнат да ги атакуват в гръб. Застреляха кмета на града. Натикаха ни в мазетата и се настаниха в домовете ни. Изпочупиха телефоните, телевизорите и радиата. Попитах един от тях докога ще останат. А той ми отговори: „Докато не убием вашия Зеленски.“ Така разбрах, че не са стигнали до Киев. Живеехме в пълно неведение какво се случва в градчетата около нас, дори в съседната махала не знаехме кой е жив и кого са застреляли. Ако се опиташ да напуснеш къщата, стреляха и убиваха. Разположиха снайперисти по покривите. Знаете ли какво е усещането да излезеш на двора и да видиш как от съседния покрив снайперист следи всяко твое движение и може да те застреля, когато си реши? Ето, вижте, на това стъпало беше синът ми, когато го простреляха в крака. Петънцата кръв не се измиват вече година и половина. Тук, пред къщата ми, избухна снаряд, още личи, нищо че запълниха дупката.
Два-три дни след като дойдоха, започна раздвижване. Попитах ги какво става. Тогава единият ми каза: „Правим смяна. Сега идват зверовете.“ И си тръгнаха. Много по-късно научихме какво се е случвало в Буча и Ирпин.
Люба и двете ѝ съседки ни изпращат до колата. Махат дълго след нас. И аз им махам, докато не ги загубвам от поглед.
В края на Ирпин виждаме голяма сграда. Прозорците ѝ са избити, покривът е разрушен. В двора, обрасъл като джунгла, се търкаля изоставена кола. Това е бившата болница на Ирпин. Пожарът и разрухата вътре не са от небрежност. Десетина дни след нахлуването на Русия в Украйна градчето е обстрелвано с руски ракети. Едната се забива в болницата. Другата – в стадиона срещу нея. Осколките и ударната вълна помитат цели етажи на жилищния блок до болница.
Пред единия от обитаемите входове на блока млада жена ми разказва:
От Буча идваха БТР-и, които помитаха и опожаряваха всичко по пътя си. Оставяха след себе си огромни дупки по пътя. Настъпваха много бързо и ги виждахме все по-близо и по-близо. Беше страшно.
Възрастна жена я допълва:
Да, те дойдоха и започнаха да стрелят направо по хората в къщите отсреща. Всичко наоколо се тресеше от взривове. Разрушаваха и палеха. Останахме без дом, без нищо.
Пресичам двете платна на пътя между блока и стадиона. Шосето е с гладък асфалт, маркировката и пешеходната пътека личат ясно. Трафикът е натоварен, но колите и камионите спират и търпеливо ме изчакват да пресека.
Сградата на стадиона е красива, боядисана в синьо. В покрива ѝ зеят огромни дупки, от които стърчат греди. Стените и прозорците са надупчени от шрапнели. Наоколо няма жива душа. Дърветата в парка са отрупани с плодове. Пейките са ремонтирани. Ще разгледам всичко това после. Тръгвам в друга посока и търся конкретни къщи. Веднага ги виждам. Залепени са до стадиона.
Малко по-рано в парка на Ирпин срещнах красиво момиче на около двайсет години. Черната ѝ коса е много късо подстригана.
Къщата ни е в най-крайната част на града, до стадиона. Видях ги как се задават откъм Буча. Видях как убиха съседите ни и труповете им останаха на улицата. Видях и как в другата къща пред нашата изнасилват приятелката ми. Всичко това беше толкова ужасно, че в душата ти не остава нищо друго освен паника. Слязох в мазето и си отрязах косата до кожа, облякох старо, широко сако на баща ми, скрих цялата си глава под вълнена шапка. Мисля, че това ме спаси, защото ме помислиха за момче. На 6 март 2022-ра разбиха вратата на дома ни. Изпочупиха всичко, включително телефоните ни. Започнаха да прелитат ракети. Покривът на стадиона беше взривен от ракета. Мислех, че сънувам някакъв кошмар. Все още ходя на психолог, защото не мога да преживея видяното и ме е страх да пусна косата си дълга, каквато беше. Мои приятелки, които бяха изнасилени и забременели от руски войници, също още ходят на психотерапия. Сега живея на друго място в Ирпин, защото, след като успяхме да избягаме с помощта на доброволци, руснаците са разрушили къщата ни.
Внимателно пристъпвам между разпадащи се къщи без покриви, точно до стадиона и вишнево дръвче. Седя вцепенена насред нещо, приличащо на стая, която има отвор, напомнящ врата към мазе или кой знае, може би врата между два двора. Сигурно това е къщата на Мария и мазето, в което се е крила две седмици, за да не я изнасилят или убият. За първи път, откакто съм в Украйна, ужасът на войната с цялата му уродливост се стоварва върху мен сред остатъците от нечий съсипан живот. Стъпвам върху разказите и спомените за паника и страх, които оцелелите от окупацията пазят. Тук, на мястото на действието, между фугите на пода вече са избуяли бурени, които се опитват да заличат всичко.
Къде са лежали труповете на съседите ѝ, в коя точно стая са изнасилили приятелката ѝ? Искам ли да знам и да си представям всичко това? То ме съсипва, ужасява, разтреперва и разбива. Съмнявам ли се в нещо? Не! Тук, насред руините, все още витае духът на насилието. Искам ли да знам? Искам! Защото разказът на тези жени, някои почти деца, след всичко преживяно в Ирпин продължава така:
Животът ни след окупацията се промени много. Заради войната всичко поскъпна, няма достатъчно работа. Някои от нас загубиха близки хора, роднини и синове, които бяха убити по време на окупацията или после на фронта. Други загубиха домовете си. Трети ходят на психотерапия заради преживяното. Полека-лека започнахме да възстановяваме всичко – къщите, градинките, парка, в който говорим с вас в момента, църквата. Сега, както виждате, в града не личи, че тук са се случили такива кошмари, защото улици и сгради се оправят лесно. Но всичко е в паметта ни, нищо не сме забравили, шокът още е в душите ни. За нас нищо не е свършило, нищо че руснаците ги няма тук в момента. Страхът и безсънието ни все още ги има. Въпреки всичко тук при нас, в Ирпин, сега е тихо в сравнение с това, което е на фронта. Страшно ми е да си мисля какво е там, и ми е много тъжно.
Гледам през задното стъкло на колата как синята сграда на стадиона с нарисувани слънчогледи по стените, дупки от снаряди и отнесен покрив бавно се отдалечава. По-навътре в градчето има красив парк, а до него – малко пазарче с отрупани сергии. Все още тук-там се виждат разрушени домове от окупацията, но не само в Ирпин, а навсякъде, където бях в Украйна. Разрушени от руското нападение здание, мост, път, къща се ремонтират колкото може по-бързо. Още не съм се отърсила от всичко чуто и преживяно, когато виждам пред себе си табела с надпис „Буча“.
Посрещат ни нови къщи, всяка с градинка отпред, безупречно чисти улици, площад с пейки от тъмно дърво и стомана, заобиколени с дървета, а в средата на всичко това – малък танк. Не е истински. Засега той е единственото, което напомня за войната тук.
Минава обед. По улиците почти няма хора. Все пак виждам две момичета. Питам ги за руската окупация от миналата година. Едното ме гледа с ужасени очи, които се наливат със сълзи. Клати глава в отказ:
Не искам да си спомням и разказвам, защото това ще ме върне в онези кошмарни дни. Не искам да преживея пак това, докато разказвам. Но беше ужасно, повярвайте ми!
Извинявам се и продължавам по равните плочки на тротоара. Срещу мен върви жена. Казвам ѝ откъде съм, и питам за окупацията. Тя избухва в плач. Говори бързо и задъхано:
Ние седяхме в мазето. Деветнайсет дни се крихме там. Нямахме никаква храна. С нас имаше бременно момиче. Който излезеше от мазето, направо го застрелваха без предупреждение. Чувахме изстрелите. Когато руснаците ни намериха в мазето, започнаха да ни измъчват – мъже, жени, няма значение, изтезаваха ни. Моята приятелка не издържа и почина. Убиха я с мъчения. Когато ни извадиха от мазето, видяхме на улицата застреляни хора. Лежаха просто така… Асфалтът беше целият в кръв, с дупки от стрелбата, а БТР-ите бяха навсякъде. Горяха къщи, някои останали без покриви, прозорци и стени. Много ми е тежко да си спомням всичко това.
След нея мъж на около шейсет години:
Това, което руснаците извършиха тук, беше ужасно. Толкова много смърт. Толкова много изтезавани хора. Толкова много насилени хора… Цялата Буча беше окупирана. Не искам да си спомням.
Стигам до кафене с маси отвън. Влизам и занемявам. Явно съм попаднала в някакъв приказен свят. Светло и уютно, стените са изрисувани на ръка в топли цветове, пердетата са вързани с панделки, витрината е пълна със сладкиши. В ъгъла на масата седи висок мъж на средна възраст. Отворил е лаптопа си и пише.
Към мен приближава младо засмяно момиче и пита какво ще поръчам. Казвам му и се представям. Питам дали някой би искал да ми разкаже за времето на окупацията. Момичето се смущава от въпроса ми и поглежда към мъжа с лаптопа. Той се изправя, протяга ръка към мен и казва: „Аз съм Серго, собственикът на кафенето. Искате ли да видите снимки от времето, когато руснаците нахлуха тук?“
Отваря папка на компютъра си. Сядам до него и невярващо гледам, докато той бавно започва разказа си.
Когато влязоха в Буча, руснаците убиваха хората по улиците. Изнасилваха в домовете и по улиците без свян – жени, мъже, деца. Навсякъде трупове. Така се държаха „руските освободители“ с нас преди година и половина. Всичко тук беше потънало в ужас. Дори си мисля, че те не са хора, а някакви луди, които явно много мразят украинците, за да проявяват такава жестокост към тях. Да разстрелваш мирни жители по улицата, докато отиват да търсят храна. Убиваха само защото не им се радваме, че са тук или защото говорим украински, а не руски. Стреляха и по мен, докато прекарвах хуманитарна помощ с колата си. Просто ми провървя, че не ме уцелиха. Но на много хора не им провървя. Убиха човек до мен, застреляха го от трийсет метра разстояние.
Серго е грузинец, женен за украинка от Буча. Щом войната започва, изпраща семейството си в Грузия, но той остава в градчето, въпреки че за него като грузинец не важи законът, който забранява мъжете да напускат Украйна. Било въпрос на дълг и чест да помага. Прекарва хуманитарна помощ между Буча и Ирпин по време на окупацията. Превежда по тайни пътища хора, които се опитват да се евакуират от двете градчета. Разказва, че според него Буча не е била толкова разрушена, колкото Ирпин, където около 80% от сградите са пострадали от обстрели.
В Буча бяха големите издевателства, изнасилвания, измъчване и насилие над хората. Тук труповете бяха толкова много, че по-трудно успяхме да ги погребем, отколкото да разчистим зарязаните по улицата танкове, след като руснаците си отидоха.
Руските войници влизат и в неговото кафене. Серго ги гледа от дома си на камерите в заведението.
Ограбиха всичко, което им се стори ценно. Изпиха всичкия алкохол. След това, извинете за думата, която ще използвам, започнаха да серат по средата на заведението, въпреки че имаме тоалетна. След това започнаха да трошат – прозорци, маси, столове. Бях в шок как е възможно въобще човешко същество да се държи по този начин.
Все пак това е нищо в сравнение с улиците на Буча, които бяха разбити от снарядите, а къщите и колите – опожарени и изтърбушени. Трупове навсякъде. Пълно с горяща военна техника. Украинците обаче са уникален народ. Когато нашите войски изгониха окупаторите, местното население се мобилизира. За няма и седмица улиците бяха разчистени от изгорелите танкове, започна ремонт на къщите, някои трябваше да построим наново, защото нищо не можеше да се възстанови от тях. Да, все още тук-там има неоправени къщи, но това е, защото собствениците им са в други страни.
Държавата предлага помощ на Серго, за да възстанови кафенето си, но той отказва
Знаех, че има много други, които се нуждаят повече от тази помощ. Щом жена ми се върна, започнахме полека-лека да ремонтираме кафенето. Тя отново аранжира и нарисува стените, оправихме всичко.
Серго е оптимист за изхода на войната. Казва, че помощта от ЕС и НАТО са безценни, но според него е лошо, че всички започват да свикват с войната.
Войната е като кошмар, който се появява, без да можеш да го прекъснеш. Те не спират, а ние няма да отстъпим. Всичките ми приятели, които сега са на фронта, казват: да, уморени сме, но няма никога да се предадем. Напротив, ще победим.
Качваме се в колата и едва тогава го виждам. Уличния стълб, до който сме спрели. От горната му част красиви червени цветя лениво се свеждат към средата на стълба, където зее огромна дупка от снаряд.
Сгушвам се в колата разбита, съкрушена, потопена в мисли. Сещам се за телефона си едва когато започва да вибрира в чантата ми. Специфична вибрация, не можеш да я сбъркаш. Знам какво е. Тревога, но оказва се, не само една. За този ден в Украйна е имало пет въздушни тревоги, които съм пропуснала, докато хората са ми разказвали…
За първи път се усмихвам днес. Личните ми страхове, тревоги и втренчен поглед в телефона ми се струват смешни и нелепи в този късен украински следобед. Да, въздушните тревоги са страшни, но все пак. Все пак… Това тук е друг свят, толкова различен от космополитния Киев. Този малък, уютен свят с китни къщи, чисти улици, равни тротоари, задружни хора и цветя на всеки ъгъл е само на трийсет километра от столицата на Украйна. Защитили са я на много висока цена, но никой не се оплаква. Дълг и любов било. Така ми казват.
Смрачава се. От колата звъня по телефона, за да уговоря среща за следващия ден. Гласът на оператора отново ми казва: „Моля, изчакайте, абонатът говори за победата.“ Усмихвам се за втори път. Магистралата е гладка като стъкло, слънцето залязва над слънчогледите пред нас. Утре, ако имам късмет, ще говоря с двама ветерани.
While the vulnerability itself is pretty run-of-the-mill, the recently disclosed
GNOME vulnerability has a number of interesting facets. The problem lies
in a library that reads files in a fairly obscure format, but it turns out
that files in that format are routinely—automatically—processed by GNOME if
they are downloaded to the local system. That turns a vulnerability in a
largely unknown library into a one-click remote-code-execution flaw for
the GNOME desktop.
In September, we experienced two incidents that resulted in degraded performance across GitHub services.
September 5 16:24 UTC (lasting 19 minutes)
On September 5, from 16:24-16:43 UTC, multiple GitHub services were down or degraded due to an outage in one of our primary databases. The primary host for a shared datastore for GitHub experienced an underlying file system write error, which affected availability for the majority of public-facing GitHub services. SAML login was affected, as was access to GitHub Actions, GitHub Issues, pull requests, GitHub Pages, GitHub API, Webhooks, GitHub Codespaces, and GitHub Packages.
The primary database suffered a partial host failure when the disk storage for the operating system became unreachable. In this case, our automatic failover was unable to detect the partial file system failure mode. We mitigated by manually failing over to a healthy host, initiated 17 minutes after our first alert and completed 2 minutes later.
With the incident mitigated, we have worked to assess more detailed impact and resilience improvements to each affected service to reduce the scope of any future incident with this shared dependency. Some of those are complete and the rest will be completed within our standard repair item SLAs. To increase the resiliency of our system, we have improved our automation that will detect and initiate a failover for this type of partial host failure. Additionally, we have identified a source of resource contention that is consistent with this type of failure and patched a fix to reduce the likelihood of recurrence.
September 19 20:36 UTC (lasting 7 hours 30 minutes)
On September 19 at 20:36 UTC, while migrating the primary datastore for GitHub Projects, an incident occurred that disrupted 95% of GitHub Projects data availability for 3.5 hours. A misconfigured index constraint on the primary GitHub Projects database table caused GitHub Projects to become fully unavailable between 20:36 UTC and 00:06 UTC. By 00:06, we restored GitHub Projects data to its state from the beginning of the incident. New project data created by users while the incident was being mitigated was fully recovered and available to users by 04:28 UTC.
In addition, a database replication interruption caused by our remediation steps created limited availability for some Git Operations, APIs, and GitHub Issues for 1.25 hours from 21:48 UTC to 23:00 UTC.
To prevent similar incidents in the future, we have improved validation of data migrations in testing and during rollout. We have evaluated and are making improvements to the constraints for any data migration to prevent the unexpected behavior that led to this data loss. To reduce the time to mitigate similar incidents, we are also in the process of rolling out improvements to reduce both the time to restore data and fix replication issues.
Please follow our status page for real-time updates on status changes. To learn more about what we’re working on, check out the GitHub Engineering Blog.
To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.
Functional
Always active
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
Preferences
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
Statistics
The technical storage or access that is used exclusively for statistical purposes.The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
Marketing
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.
comments
