“Биволъ” сваля маската на горската мафия (ПЪРВА ЧАСТ) Чадър над незаконните сечи разпънаха горски в Софийска област

Post Syndicated from Екип на Биволъ original https://bivol.bg/nezakonna-sech-sofia-oblast.html

сряда 13 декември 2023


Липсата на реален контрол в горите и кражби за над 1 млн. лв. в Софийска област разкрива първото от серията нови разследвания на “Биволъ” за незаконния дърводобив в партньорство с…

Our role in supporting the nonprofit ecosystem

Post Syndicated from Let's Encrypt original https://letsencrypt.org/2023/12/13/ngos/

For more than ten years, we at the nonprofit Internet Security Research Group (ISRG) have been focused on our mission of building a more secure and privacy-respecting Internet for everyone, everywhere. As we touch on in our 2023 Annual Report, we now serve more than 360 million domains with free TLS certificates.

Beyond being a big number, what does that signify? What’s the importance of having TLS being widely adopted anyways? We’ll take a closer look at these questions through the lens of one group of Subscribers we can relate to particularly well: nonprofits.

Serving .org at Internet scale

Let’s Encrypt serves 57% of all websites using the .org top level domain (TLD), which is commonly used by nonprofits. In the US alone there are 1.8M registered nonprofit organizations. And while the focus of these organizations are varied, all of them rely on the Internet in some capacity.

When a nonprofit uses a TLS certificate on their website, it protects their visitors and stakeholders from snoopers, MITM attacks, and surveillance. Without TLS, nonprofits’ content could be changed without their knowledge or their visitors’ private information could be compromised. Access to free and automated TLS via Let’s Encrypt means these nonprofits face as few barriers as possible to adopting TLS.

In short, something as fundamental as security and privacy should be as easy to access as possible. For nonprofits both large and small, Let’s Encrypt makes it easy to provide security and privacy for users of their websites, enabling them to remain focused on their missions.

Zooming in on three nonprofits we serve

The American Civil Liberties Union (ACLU) uses Let’s Encrypt as it works to realize its focus of being a “guardian of liberty” for US citizens. Using Let’s Encrypt protects ACLU’s constituents when they’re trying to know their rights or take action. With more than 4 million page views per month, ACLU’s website is a critical part of their mission.

Human Rights Watch (HRW) is an international nonprofit organization. With more than 500 individuals on staff around the world, HRW’s website is a trove of information empowering individuals and organizations alike to be informed and take action with a global perspective. Nearly 70% of HRW’s web traffic comes from people outside of the United States; that’s millions of page views per month secured by Let’s Encrypt—and by extension, millions of people around the world benefitting from a more secure and privacy-respecting Web.

The Center for Democracy & Technology (CDT) uses Let’s Encrypt to advance its mission to promote democratic values by shaping technology policy and architecture, with a focus on the rights of the individual. The CDT website offers updated and insightful information into the ways policy and innovation impact the digital space. Without a TLS certificate, the content of these pages could be intercepted and changed. What’s more, for those looking to financially support CDT, using TLS on their donation page encrypts the transaction protecting user details such as credit card and other personal information. Mallory Knodel, CTO at CDT and longtime digital rights defender and advocate commented, “Billions of people in over 60 countries access the internet with less censorship and surveillance because Let’s Encrypt hastened the adoption of web security measures by making certificates easy to obtain.”

Serving philanthropic foundations

In the United States, the work of nonprofits is made possible in large part through philanthropic foundations and organizations. When it comes to philanthropy’s web presence, Let’s Encrypt is there, too.

We provide TLS to billion dollar philanthropic organizations like the Hewlett Foundation, the Silicon Valley Community Foundation, Yield Giving, and many others. Taking a look at the top 50 philanthropic organizations around the world, Let’s Encrypt serves 36% of them. For large philanthropies, their website is the primary tool they have to communicate their focus areas for future funding as well as the impact they’ve made with past giving.

One of the leading philanthropists in the US, Craig Newmark, uses Let’s Encrypt and Digital Ocean for his website, craig newmark philanthropies. Commenting on our work, Craig recently shared, “The people at ISRG have been helping protect the Internet for over ten years, and continue to protect us all. They’re a necessary part of Cyber Civil Defense and national security.”

Overall, while Let’s Encrypt aims to build a better Internet, we’re particularly proud that our impact protects those seeking to build a better world.

Internet Security Research Group (ISRG) is the parent organization of Prossimo, Let’s Encrypt, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

Our role in supporting the nonprofit ecosystem

Post Syndicated from Let's Encrypt original https://letsencrypt.org/2023/12/13/ngos.html

For more than ten years, we at the nonprofit Internet Security Research Group (ISRG) have been focused on our mission of building a more secure and privacy-respecting Internet for everyone, everywhere. As we touch on in our 2023 Annual Report, we now serve more than 360 million domains with free TLS certificates.

Beyond being a big number, what does that signify? What’s the importance of having TLS being widely adopted anyways? We’ll take a closer look at these questions through the lens of one group of Subscribers we can relate to particularly well: nonprofits.

Serving .org at Internet scale

Let’s Encrypt serves 51% of all websites using the .org top level domain (TLD), which is commonly used by nonprofits. In the US alone there are 1.8M registered nonprofit organizations. And while the focus of these organizations are varied, all of them rely on the Internet in some capacity.

When a nonprofit uses a TLS certificate on their website, it protects their visitors and stakeholders from snoopers, MITM attacks, and surveillance. Without TLS, nonprofits’ content could be changed without their knowledge or their visitors’ private information could be compromised. Access to free and automated TLS via Let’s Encrypt means these nonprofits face as few barriers as possible to adopting TLS.

In short, something as fundamental as security and privacy should be as easy to access as possible. For nonprofits both large and small, Let’s Encrypt makes it easy to provide security and privacy for users of their websites, enabling them to remain focused on their missions.

Zooming in on four nonprofits we serve

The American Civil Liberties Union (ACLU) uses Let’s Encrypt as it works to realize its focus of being a “guardian of liberty” for US citizens. Using Let’s Encrypt protects ACLU’s constituents when they’re trying to know their rights or take action. With more than 4 million page views per month, ACLU’s website is a critical part of their mission.

Human Rights Watch (HRW) is an international nonprofit organization. With more than 500 individuals on staff around the world, HRW’s website is a trove of information empowering individuals and organizations alike to be informed and take action with a global perspective. Nearly 70% of HRW’s web traffic comes from people outside of the United States; that’s millions of page views per month secured by Let’s Encrypt—and by extension, millions of people around the world benefitting from a more secure and privacy-respecting Web.

The Center for Democracy & Technology (CDT) uses Let’s Encrypt to advance its mission to promote democratic values by shaping technology policy and architecture, with a focus on the rights of the individual. The CDT website offers updated and insightful information into the ways policy and innovation impact the digital space. Without a TLS certificate, the content of these pages could be intercepted and changed. What’s more, for those looking to financially support CDT, using TLS on their donation page encrypts the transaction protecting user details such as credit card and other personal information. Mallory Knodel, CTO at CDT and longtime digital rights defender and advocate commented, “Billions of people in over 60 countries access the internet with less censorship and surveillance because Let’s Encrypt hastened the adoption of web security measures by making certificates easy to obtain.”

Serving philanthropic foundations

In the United States, the work of nonprofits is made possible in large part through philanthropic foundations and organizations. When it comes to philanthropy’s web presence, Let’s Encrypt is there, too.

We provide TLS to billion dollar philanthropic organizations like the Hewlett Foundation, the Silicon Valley Community Foundation, and many others. Taking a look at the top 50 philanthropic organizations around the world, Let’s Encrypt serves 36% of them. For large philanthropies, their website is the primary tool they have to communicate their focus areas for future funding as well as the impact they’ve made with past giving.

One of the leading philanthropists in the US, Craig Newmark, uses Let’s Encrypt and Digital Ocean for his website, craig newmark philanthropies. Commenting on our work, Craig recently shared, “The people at ISRG have been helping protect the Internet for over ten years, and continue to protect us all. They’re a necessary part of Cyber Civil Defense and national security.”

Overall, while Let’s Encrypt aims to build a better Internet, we’re particularly proud that our impact protects those seeking to build a better world.

Internet Security Research Group (ISRG) is the parent organization of Prossimo, Let’s Encrypt, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

The end of vger.kernel.org

Post Syndicated from corbet original https://lwn.net/Articles/954783/

Konstantin Ryabitsev has announced
that the movement of kernel mailing lists away from the venerable
vger.kernel.org system is nearly complete:

Over the past few months we’ve migrated all of the vger.kernel.org
mailing lists, with the exception of the Big One (linux-kernel, aka
LKML). This list alone is responsible for about 80% of all vger
mailing list traffic, so we left it for the last.

This Thursday, December 14, at 11AM Pacific (19:00 UTC), we will
switch the MX record for vger to point to the new location
(subspace.kernel.org), which will complete the mailing list
migration from the legacy vger server to the new infrastructure.

Graber: LXD now re-licensed and under a CLA

Post Syndicated from corbet original https://lwn.net/Articles/954777/

The story of Canonical’s takeover of the LXD container manager, and the
subsequent creation of the Incus fork, has been
simmering for a while. Now Incus developer Stéphane Graber reports
that Canonical has changed the license and contribution terms for LXD:

Per the commit message performing the re-licensing, all further
contributions will be under the AGPLv3 license and all
contributions from Canonical employees have been re-licensed to
AGPLv3.

However, Canonical does not own the copyright on any contribution
from non-employees, such as the many changes they have imported
from Incus over the past few months. Those therefore remain under
the Apache2 license that they were contributed under.

As a result, Canonical cannot release LXD under the AGPLv3 license
and likely never will be able to. LXD is now under a weird mix of
Apache2 and AGPLv3 with no clear metadata indicating what file or
what part of each file is under one license or the other.

He also notes that this change will put an end to the flow of patches — in
either direction — between the two projects.

Patch Tuesday – December 2023

Post Syndicated from Adam Barnett original https://blog.rapid7.com/2023/12/12/patch-tuesday-december-2023/

Patch Tuesday - December 2023

Microsoft is addressing 34 vulnerabilities this December Patch Tuesday, including a single zero-day vulnerability and three critical remote code execution (RCE) vulnerabilities. December Patch Tuesday has historically seen fewer patches than a typical month, and this trend continues in 2023. This total does not include eight browser vulnerabilities published earlier this month. At time of writing, none of the vulnerabilities patched today are yet added to the CISA KEV list.

Certain AMD processors: zero-day information disclosure

This month’s lone zero-day vulnerability is CVE-2023-20588, which describes a potential information disclosure due to a flaw in certain AMD processor models as listed on the AMD advisory. AMD states that a divide-by-zero on these processor models could potentially return speculative data. AMD believes the potential impact of the vulnerability is low since local access is required; however, Microsoft ranks severity as important under its own proprietary severity scale. The vulnerability is patched at the OS level in all supported versions of Windows, even as far back as Windows Server 2008 for Azure-hosted assets participating in the Extended Security Update (ESU) program.

Outlook: no-interaction critical RCE

CVE-2023-35628 describes a critical RCE vulnerability in the MSHTML proprietary browser engine still used by Outlook, among others, to render HTML content. Of particular note: the most concerning exploitation scenario leads to exploitation as soon as Outlook retrieves and processes the specially crafted malicious email. This means that exploitation could occur before the user interacts with the email in any way; not even the Preview Pane is required in this scenario. Other attack vectors exist: the user could also click a malicious link received via email, instant message, or other medium. Assets where Internet Explorer 11 has been fully disabled are still vulnerable until patched; the MSHTML engine remains installed within Windows regardless of the status of IE11.

Internet Connection Sharing: critical RCE

This month also brings patches for a pair of critical RCE vulnerabilities in Internet Connection Sharing. CVE-2023-35630 and CVE-2023-35641 share a number of similarities: a base CVSS v3.1 score of 8.8, Microsoft critical severity ranking, low attack complexity, and presumably execution in SYSTEM context on the target machine, although the advisories do not specify execution context. Description of the exploitation method does differ between the two, however. CVE-2023-35630 requires the attacker to modify an option->length field in a DHCPv6 DHCPV6_MESSAGE_INFORMATION_REQUEST input message. Exploitation of CVE-2023-35641 is also via a maliciously crafted DHCP message to an ICS server, but the advisory gives no further clues. A broadly similar ICS vulnerability in September 2023 led to RCE in a SYSTEM context on the ICS server. In all three cases, a mitigating factor is the requirement for the attack to be launched from the same network segment as the ICS server. It seems improbable that either of this month’s ICS vulnerabilities are exploitable against a target on which ICS is not running, although Microsoft does not explicitly deny the possibility.

Holiday season update

Notable by their absence this month: no security patches for Exchange, SharePoint, Visual Studio/.NET, or SQL Server. There are also no lifecycle transitions for Microsoft products this month, although a number of Windows Server 2019 editions and Office components will transition out of mainstream support and into extended support from January 2024.

Summary Charts

Patch Tuesday - December 2023
Sharing is caring, unless it’s exploitative.
Patch Tuesday - December 2023
A rare occurence: Remote Code Execution not in the top spot.
Patch Tuesday - December 2023
Fewer vulns this month overall means less variation in the heatmap.

Summary Tables

Azure vulnerabilities

CVE Title Exploited? Publicly disclosed? CVSSv3 base score
CVE-2023-35624 Azure Connected Machine Agent Elevation of Privilege Vulnerability No No 7.3
CVE-2023-35625 Azure Machine Learning Compute Instance for SDK Users Information Disclosure Vulnerability No No 4.7

Browser vulnerabilities

CVE Title Exploited? Publicly disclosed? CVSSv3 base score
CVE-2023-35618 Microsoft Edge (Chromium-based) Elevation of Privilege Vulnerability No No 9.6
CVE-2023-36880 Microsoft Edge (Chromium-based) Information Disclosure Vulnerability No No 4.8
CVE-2023-38174 Microsoft Edge (Chromium-based) Information Disclosure Vulnerability No No 4.3
CVE-2023-6512 Chromium: CVE-2023-6512 Inappropriate implementation in Web Browser UI No No N/A
CVE-2023-6511 Chromium: CVE-2023-6511 Inappropriate implementation in Autofill No No N/A
CVE-2023-6510 Chromium: CVE-2023-6510 Use after free in Media Capture No No N/A
CVE-2023-6509 Chromium: CVE-2023-6509 Use after free in Side Panel Search No No N/A
CVE-2023-6508 Chromium: CVE-2023-6508 Use after free in Media Stream No No N/A

ESU Windows vulnerabilities

CVE Title Exploited? Publicly disclosed? CVSSv3 base score
CVE-2023-36006 Microsoft WDAC OLE DB provider for SQL Server Remote Code Execution Vulnerability No No 8.8
CVE-2023-35639 Microsoft ODBC Driver Remote Code Execution Vulnerability No No 8.8
CVE-2023-35641 Internet Connection Sharing (ICS) Remote Code Execution Vulnerability No No 8.8
CVE-2023-35630 Internet Connection Sharing (ICS) Remote Code Execution Vulnerability No No 8.8
CVE-2023-35628 Windows MSHTML Platform Remote Code Execution Vulnerability No No 8.1
CVE-2023-21740 Windows Media Remote Code Execution Vulnerability No No 7.8
CVE-2023-35633 Windows Kernel Elevation of Privilege Vulnerability No No 7.8
CVE-2023-35632 Windows Ancillary Function Driver for WinSock Elevation of Privilege Vulnerability No No 7.8
CVE-2023-36011 Win32k Elevation of Privilege Vulnerability No No 7.8
CVE-2023-36005 Windows Telephony Server Elevation of Privilege Vulnerability No No 7.5
CVE-2023-36004 Windows DPAPI (Data Protection Application Programming Interface) Spoofing Vulnerability No No 7.5
CVE-2023-35622 Windows DNS Spoofing Vulnerability No No 7.5
CVE-2023-35643 DHCP Server Service Information Disclosure Vulnerability No No 7.5
CVE-2023-35638 DHCP Server Service Denial of Service Vulnerability No No 7.5
CVE-2023-35629 Microsoft USBHUB 3.0 Device Driver Remote Code Execution Vulnerability No No 6.8
CVE-2023-35642 Internet Connection Sharing (ICS) Denial of Service Vulnerability No No 6.5
CVE-2023-36012 DHCP Server Service Information Disclosure Vulnerability No No 5.3
CVE-2023-20588 AMD: CVE-2023-20588 AMD Speculative Leaks Security Notice No Yes N/A

Microsoft Dynamics vulnerabilities

CVE Title Exploited? Publicly disclosed? CVSSv3 base score
CVE-2023-36020 Microsoft Dynamics 365 (on-premises) Cross-site Scripting Vulnerability No No 7.6
CVE-2023-35621 Microsoft Dynamics 365 Finance and Operations Denial of Service Vulnerability No No 7.5

Microsoft Dynamics Azure vulnerabilities

CVE Title Exploited? Publicly disclosed? CVSSv3 base score
CVE-2023-36019 Microsoft Power Platform Connector Spoofing Vulnerability No No 9.6

Microsoft Office vulnerabilities

CVE Title Exploited? Publicly disclosed? CVSSv3 base score
CVE-2023-35636 Microsoft Outlook Information Disclosure Vulnerability No No 6.5
CVE-2023-36009 Microsoft Word Information Disclosure Vulnerability No No 5.5
CVE-2023-35619 Microsoft Outlook for Mac Spoofing Vulnerability No No 5.3

System Center vulnerabilities

CVE Title Exploited? Publicly disclosed? CVSSv3 base score
CVE-2023-36010 Microsoft Defender Denial of Service Vulnerability No No 7.5

Windows vulnerabilities

CVE Title Exploited? Publicly disclosed? CVSSv3 base score
CVE-2023-35634 Windows Bluetooth Driver Remote Code Execution Vulnerability No No 8
CVE-2023-35644 Windows Sysmain Service Elevation of Privilege No No 7.8
CVE-2023-36696 Windows Cloud Files Mini Filter Driver Elevation of Privilege Vulnerability No No 7.8
CVE-2023-35631 Win32k Elevation of Privilege Vulnerability No No 7.8
CVE-2023-36391 Local Security Authority Subsystem Service Elevation of Privilege Vulnerability No No 7.8
CVE-2023-36003 XAML Diagnostics Elevation of Privilege Vulnerability No No 6.7
CVE-2023-35635 Windows Kernel Denial of Service Vulnerability No No 5.5

Federate IAM-based single sign-on to Amazon Redshift role-based access control with Okta

Post Syndicated from Debu Panda original https://aws.amazon.com/blogs/big-data/federate-iam-based-single-sign-on-to-amazon-redshift-role-based-access-control-with-okta/

Amazon Redshift accelerates your time to insights with fast, easy, and secure cloud data warehousing at scale. Tens of thousands of customers rely on Amazon Redshift to analyze exabytes of data and run complex analytical queries.

You can use your preferred SQL clients to analyze your data in an Amazon Redshift data warehouse. Connect seamlessly by leveraging an identity provider (IdP) or single sign-on (SSO) credentials to connect to the Redshift data warehouse to reuse existing user credentials and avoid additional user setup and configuration. Using role-based access control (RBAC), you can simplify managing user privileges, database permissions and management of security permissions in Amazon Redshift. You can also use redshift database roles to define a set of elevated permissions, such as for a system monitor or database administrator.

Using AWS Identity and Access Management (IAM) with RBAC, organizations can simplify user management because you no longer need to create users and map them to database roles manually. You can define the mapped database roles as a principal tag for the IdP groups or IAM role, so Redshift database roles and users who are members of those IdP groups are granted to the database roles automatically.

Earlier in 2023, we launched support for Okta integration with Amazon Redshift Serverless using database roles. In this post, we focus on Okta as the IdP and provide step-by-step guidance to integrate a Redshift provisioned cluster with Okta using the Redshift Query Editor v2 and with SQL clients like SQL Workbench/J. You can use this mechanism with other IdP providers such as Azure Active Directory or Ping with any applications or tools using Amazon’s JDBC, ODBC, or Python driver.

Recently we also announced Amazon Redshift integration with AWS IAM Identity Center, supporting trusted identity propagation, allowing you to use third-party Identity Providers (IdP) such as Microsoft Entra ID (Azure AD), Okta, Ping, and OneLogin. This integration simplifies the authentication and authorization process for Amazon Redshift users using Query Editor V2 or Amazon Quicksight, making it easier for them to securely access your data warehouse. AWS IAM Identity Center offers automatic user and group provisioning from Okta to itself by utilizing the System for Cross-domain Identity Management (SCIM) 2.0 protocol. This integration allows for seamless synchronization of information between two services, ensuring accurate and up-to-date information in AWS IAM Identity Center. Refer to Integrate Okta with Amazon Redshift Query Editor V2 using AWS IAM Identity Center for seamless Single Sign-On blog post to learn more about setting up single sign-on (SSO) to Amazon Redshift using integration with IdC and Okta as the Identity Provider.

If you are interested in using IAM-based single sign-on with Amazon Redshift database roles then you can continue reading this blog.

Solution overview

The following diagram illustrates the authentication flow of Okta with a Redshift provisioned cluster using federated IAM roles and automatic database role mapping.

Architecture Diagram

The workflow contains the following steps:

  1. Either the user chooses an IdP app in their browser, or the SQL client initiates a user authentication request to the IdP (Okta).
  2. Upon a successful authentication, Okta submits a request to the AWS federation endpoint with a SAML assertion containing the principal tags.
  3. The AWS federation endpoint validates the SAML assertion and invokes the AWS Security Token Service (AWS STS) API AssumeRoleWithSAML. The SAML assertion contains the IdP user and group information that is stored in the RedshiftDbUser and RedshiftDbRoles principal tags, respectively. Temporary IAM credentials are returned to the SQL client or, if using the Query Editor v2, the user’s browser is redirected to the Query Editor v2 console using the temporary IAM credentials.
  4. The temporary IAM credentials are used by the SQL client or Query Editor v2 to call the Redshift API GetClusterCredentialsWithIAM. This API uses the principal tags to determine the user and database roles that the user belongs to. An associated database user is created if the user is signing in for the first time and is granted the matching database roles automatically. A temporary password is returned to the SQL client.
  5. Using the database user and temporary password, the SQL client or Query Editor v2 connects to Amazon Redshift. Upon login, the user is authorized based on the Redshift database roles that were assigned in Step 4.

Prerequisites

You need the following prerequisites to set up this solution:

Connect with a Redshift provisioned cluster as a federated user using Query Editor v2

To connect using Query Editor v2, complete the following steps:

  1. Follow all the steps described in the sections Set up your Okta application and Set up AWS configuration in the following post.
  2. For the Amazon Redshift access IAM policy, replace the policy with the following JSON to use the GetClusterCredentialsWithIAM API:
    {
    	"Version": "2012-10-17",
    	"Statement": [
    					{
    						"Sid": "VisualEditor0",
    						"Effect": "Allow",
    						"Action": "redshift:GetClusterCredentialsWithIAM",
    						"Resource": "arn:aws:redshift:us-west-2:123456789012:dbname:redshift-cluster-1/dev"
    					}
    				]
    }

Now you’re ready to connect to your Redshift provisioned cluster using Query Editor v2 and federated login.

  1. Use the SSO URL from Okta and log in to your Okta account with your user credentials. For this demo, we log in with user Ethan.
  2. In Query Editor v2, choose your Redshift provisioned cluster (right-click) and choose Create connection.
  3. For Authentication, select Temporary credentials using your IAM identity.
  4. For Database, enter the database name you want to connect to.
  5. Choose Create connection.
  6. Run the following command to validate that you are logged in as a federated user and also to get the list of roles associated with that user for the current session:
SELECT current_user,* FROM pg_get_session_roles() eff_ro(name name, roleid integer);

Because Ethan is part of the sales group and has been granted permissions to access tables in the sales_schema, he should be able to access those tables without any issues. However, if he tries to access tables in the finance_schema, he would receive a permission denied error because Ethan is not part of the finance group in Okta.

Okta-QEV2-Federation

Connect with a Redshift provisioned cluster as a federated user via a third-party client

To connect as a federated user via a third-party client, complete the following steps:

  1. Follow steps 1 and 2 which are described in above section (Connect with a Redshift provisioned cluster as a federated user using Query Editor v2).
  2. Use the Redshift JDBC driver v2.1.0.18 and above because it supports authentication with IAM group federation. For the URL, enter jdbc:redshift:iam://<cluster endpoint>:<port>:<databasename>?groupfederation=true. For example, jdbc:redshift:iam://redshift-cluster-1.abdef0abc0ab.us-west-2.redshift.amazonaws.com:5439/dev?groupfederation=true

In the preceding URL, groupfederation is a mandatory parameter that allows you to authenticate with the IAM credentials for the Redshift provisioned cluster. Without the groupfederation parameter, it will not use Redshift database roles.

  1. For Username and Password, enter your Okta credentials.

SQL Workbench/J - Connection

  1. To set up extended properties, follow Steps 4–9 in the section Configure the SQL client (SQL Workbench/J) in the following post.

User Ethan will be able to access the sales_schema tables. If Ethan tries to access the tables in the finance_schema, he will get a permission denied error.

SQL Workbench/J Demo

Troubleshooting

If your connection didn’t work, consider the following:

  • Enable logging in the driver. For instructions, see Configure logging.
  • Make sure to use the latest Amazon Redshift JDBC driver version.
  • If you’re getting errors while setting up the application on Okta, make sure you have admin access.
  • If you can authenticate via the SQL client but get a permission issue or can’t see objects, grant the relevant permission to the role.

Clean up

When you’re done testing the solution, clean up the resources to avoid incurring future charges:

  1. Delete the Redshift provisioned cluster.
  2. Delete the IAM roles, IAM IdPs, and IAM policies.

Conclusion

In this post, we provided step-by-step instructions to integrate a Redshift provisioned cluster with Okta using the Redshift Query Editor v2 and SQL Workbench/J with the help of federated IAM roles and automatic database role mapping. You can use a similar setup with other SQL clients (such as DBeaver or DataGrip). We also showed how Okta group membership is mapped automatically with Redshift provisioned cluster roles to use role-based authentication seamlessly.

If you have any feedback or questions, please leave them in the comments.


About the Authors

Debu-PandaDebu Panda is a Senior Manager, Product Management at AWS. He is an industry leader in analytics, application platform, and database technologies, and has more than 25 years of experience in the IT world.

Ranjan Burman is an Analytics Specialist Solutions Architect at AWS. He specializes in Amazon Redshift and helps customers build scalable analytical solutions. He has more than 16 years of experience in different database and data warehousing technologies. He is passionate about automating and solving customer problems with cloud solutions.

Maneesh Sharma is a Senior Database Engineer at AWS with more than a decade of experience designing and implementing large-scale data warehouse and analytics solutions. He collaborates with various Amazon Redshift Partners and customers to drive better integration.

Orchestrate Amazon EMR Serverless Spark jobs with Amazon MWAA, and data validation using Amazon Athena

Post Syndicated from Gaurav Parekh original https://aws.amazon.com/blogs/big-data/orchestrate-amazon-emr-serverless-spark-jobs-with-amazon-mwaa-and-data-validation-using-amazon-athena/

As data engineering becomes increasingly complex, organizations are looking for new ways to streamline their data processing workflows. Many data engineers today use Apache Airflow to build, schedule, and monitor their data pipelines.

However, as the volume of data grows, managing and scaling these pipelines can become a daunting task. Amazon Managed Workflows for Apache Airflow (Amazon MWAA) can help simplify the process of building, running, and managing data pipelines. By providing Apache Airflow as a fully managed platform, Amazon MWAA allows data engineers to focus on building data workflows instead of worrying about infrastructure.

Today, businesses and organizations require cost-effective and efficient ways to process large amounts of data. Amazon EMR Serverless is a cost-effective and scalable solution for big data processing that can handle large volumes of data. The Amazon Provider in Apache Airflow comes with EMR Serverless operators and is already included in Amazon MWAA, making it easy for data engineers to build scalable and reliable data processing pipelines. You can use EMR Serverless to run Spark jobs on the data, and use Amazon MWAA to manage the workflows and dependencies between these jobs. This integration can also help reduce costs by automatically scaling the resources needed to process data.

Amazon Athena is a serverless, interactive analytics service built on open-source frameworks, supporting open-table and file formats. You can use standard SQL to interact with data. Athena, a serverless and interactive analytics service, makes this possible without the need to manage complex infrastructure.

In this post, we use Amazon MWAA, EMR Serverless, and Athena to build a complete end-to-end data processing pipeline.

Solution overview

The following diagram illustrates the solution architecture.

The workflow includes the following steps:

  1. Create an Amazon MWAA workflow that retrieves data from your input Amazon Simple Storage Service (Amazon S3) bucket.
  2. Use EMR Serverless to process the data stored in Amazon S3. EMR Serverless automatically scales up or down based on the workload, so you don’t need to worry about provisioning or managing any infrastructure.
  3. Use EMR Serverless to transform the data using PySpark code and then store the transformed data back in your S3 bucket.
  4. Use Athena to create an external table based on the S3 dataset and run queries to analyze the transformed data. Athena uses the AWS Glue Data Catalog to store the table metadata.

Prerequisites

You should have the following prerequisites:

Data preparation

To illustrate using EMR Serverless jobs with Apache Spark via Amazon MWAA and data validation using Athena, we use the publicly available NYC taxi dataset. Download the following datasets to your local machine:

  • Green taxi and Yellow taxi trip records – Trip records for yellow and green taxis, which include information such as pick-up and drop-off dates and times, locations, trip distances, and payment types. In our example, we use the latest Parquet files for 2022.
  • Dataset for Taxi zone lookup – A dataset that provides location IDs and corresponding zone details for taxis.

In later steps, we upload these datasets to Amazon S3.

Create solution resources

This section outlines the steps for setting up data processing and transformation.

Create an EMR Serverless application

You can create one or more EMR Serverless applications that use open source analytics frameworks like Apache Spark or Apache Hive. Unlike EMR on EC2, you do not need to delete or terminate EMR Serverless applications. EMR Serverless application is only a definition and once created, can be re-used as long as needed. This makes the MWAA pipeline simpler as now you just have to submit jobs to a pre-created EMR Serverless application.

By default, EMR Serverless application will auto-start on job submission and auto-stop when idle for 15 minutes by default to ensure cost efficiency. You can modify the amount of idle time or choose to turn the feature off.

To create an application using EMR Serverless console, follow the instructions in “Create an EMR Serverless application”. Note down the application ID as we will use it in following steps.

Create an S3 bucket and folders

Complete the following steps to set up your S3 bucket and folders:

  1. On the Amazon S3 console, create an S3 bucket to store the dataset.
  2. Note the name of the S3 bucket to use in later steps.
  3. Create an input_data folder for storing input data.
  4. Within that folder, create three separate folders, one for each dataset: green, yellow, and zone_lookup.

You can download and work with the latest datasets available. For our testing, we use the following files:

  • The green/ folder has the file green_tripdata_2022-06.parquet
  • The yellow/ folder has the file yellow_tripdata_2022-06.parquet
  • The zone_lookup/ folder has the file taxi_zone_lookup.csv

Set up the Amazon MWAA DAG scripts

Complete the following steps to set up your DAG scripts:

  1. Download the following scripts to your local machine:
    1. requirements.txt – A Python dependency is any package or distribution that is not included in the Apache Airflow base install for your Apache Airflow version on your Amazon MWAA environment. For this post, we use Boto3 version >=1.23.9.
    2. blog_dag_mwaa_emrs_ny_taxi.py – This script is a part of the Amazon MWAA DAG and consists of the following tasks: yellow_taxi_zone_lookup, green_taxi_zone_lookup, and ny_taxi_summary,. These tasks involve running Spark jobs to lookup taxi zones, and generating a data summary .
    3. green_zone.py – This PySpark script reads data files for green taxi rides and zone lookup, performs a join operation to combine them, and generates an output file containing green taxi rides with zone information. It utilizes temporary views for the df_green and df_zone data frames, performs column-based joins, and aggregates data like passenger count, trip distance, and fare amount. Lastly, it creates the output_data folder in the specified S3 bucket to write the resulting data frame, df_green_zone, as Parquet files.
    4. yellow_zone.py – This PySpark script processes yellow taxi ride and zone lookup data files by joining them to generate an output file containing yellow taxi rides with zone information. The script accepts a user-provided S3 bucket name and initiates a Spark session with the application name yellow_zone. It reads the yellow taxi files and zone lookup file from the specified S3 bucket, creates temporary views, performs a join based on location ID, and calculates statistics such as passenger count, trip distance, and fare amount. Lastly, it creates the output_data folder in the specified S3 bucket to write the resulting data frame, df_yellow_zone, as Parquet files.
    5. ny_taxi_summary.py – This PySpark script processes the green_zone and yellow_zone files to aggregate statistics on taxi rides, grouping data by service zones and location IDs. It requires an S3 bucket name as a command line argument, creates a SparkSession named ny_taxi_summary, reads the files from S3, performs a join, and generates a new data frame named ny_taxi_summary. It creates an output_data folder in the specified S3 bucket to write the resulting data frame to new Parquet files.
  2. On your local machine, update the blog_dag_mwaa_emrs_ny_taxi.py script with the following information:
    • Update your S3 bucket name in the following two lines:
      S3_LOGS_BUCKET = "<<bucket_name_here>>"
      S3_BASE_BUCKET = "<<bucket_name_here>>"

    • Update your role name ARN:
      JOB_ROLE_ARN = “<<emr_serverless_execution_role ARN here>>”
      e.g. arn:aws:iam::<<ACCOUNT_ID>>:role/<<ROLE_NAME>>

    • Update EMR Serverless Application ID. Use the Application ID created earlier.
      EMR_SERVERLESS_APPLICATION_ID  = “<<emr serverless application ID here>>”

  3. Upload the requirements.txt file to the S3 bucket created earlier
  4. In the S3 bucket, create a folder named dags and upload the updated blog_dag_mwaa_emrs_ny_taxi.py file from your local machine.
  5. On the Amazon S3 console, create a new folder named scripts inside the S3 bucket and upload the scripts to this folder from your local machine.

Create an Amazon MWAA environment

To create an Airflow environment, complete the following steps:

  1. On the Amazon MWAA console, choose Create environment.
  2. For Name, enter mwaa_emrs_athena_pipeline.
  3. For Airflow version, choose the latest version (for this post, 2.5.1).
  4. For S3 Bucket, enter the path to your S3 bucket.
  5. For DAGs folder, enter the path to your dags folder.
  6. For Requirements file, enter the path to the requirements.txt file.
  7. Choose Next.
  8. For Virtual private cloud (VPC), choose a VPC that has a minimum of two private subnets.

This will populate two of the private subnets in your VPC.

  1. Under Web server access, select Public network.

This allows the Apache Airflow UI to be accessed over the internet by users granted access to the IAM policy for your environment.

  1. For Security group(s), select Create new security group.
  2. For Environment class, select mw1.small.
  3. For Execution role, choose Create a new role.
  4. For Role name, enter a name.
  5. Leave the other configurations as default and choose Next.
  6. On the next page, choose Create environment.

It may take about 20–30 minutes to create your Amazon MWAA environment.

  1. When the Amazon MWAA environment status changes to Available, navigate to the IAM console and update cluster execution role to add pass role privileges to emr_serverless_execution_role.

Trigger the Amazon MWAA DAG

To trigger the DAG, complete the following steps:

  1. On the Amazon MWAA console, choose Environments in the navigation pane.
  2. Open your environment and choose Open Airflow UI.
  3. Select blog_dag_mwaa_emr_ny_taxi, choose the play icon, and choose Trigger DAG.
  4. When the DAG is running, choose the DAG blog_dag_mwaa_emrs_ny_taxi and choose Graph to locate your DAG run workflow.

The DAG will take approximately 4–6 minutes to run all the scripts. You will see all the complete tasks and the overall status of the DAG will show as success.

To rerun the DAG, remove s3://<<your_s3_bucket here >>/output_data/.

Optionally, to understand how Amazon MWAA runs these tasks, choose the task you want to inspect.

Choose Run to view the task run details.

The following screenshot shows an example of the task logs.

If you like to dive deep in the execution logs, then on the EMR Serverless console, navigate to “Applications”. The Apache Spark driver logs will indicate the initiation of your job along with the details for executors, stages and tasks that were created by EMR Serverless. These logs can be helpful to monitor your job progress and troubleshoot failures.

By default, EMR Serverless will store application logs securely in Amazon EMR managed storage for a period of 30 days. However, you can also specify Amazon S3 or Amazon CloudWatch as your log delivery options during job submission.

Validate the final result set with Athena

Let’s validate the data loaded by the process using Athena SQL queries.

  1. On the Athena console, choose Query editor in the navigation pane.
  2. If you’re using Athena for the first time, under Settings, choose Manage and enter the S3 bucket location that you created earlier (<S3_BUCKET_NAME>/athena), then choose Save.
  3. In the query editor, enter the following query to create an external table:
CREATE EXTERNAL TABLE default.ny_taxi_summary(
  pu_service_zone string, 
  pulocationid bigint, 
  do_service_zone string, 
  dolocationid bigint, 
  passenger_count bigint, 
  trip_distance double, 
  fare_amount double, 
  extra double, 
  mta_tax double, 
  tip_amount double, 
  tolls_amount double, 
  improvement_surcharge double, 
  total_amount double, 
  congestion_surcharge double, 
  airport_fee double)
ROW FORMAT SERDE 
  'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' 
STORED AS INPUTFORMAT 
  'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' 
OUTPUTFORMAT 
  'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
LOCATION
  's3://<<YOUR-S3-BUCKET Here>>/output_data/ny_taxi_summary/' -- *** Change bucket name to your bucket***
TBLPROPERTIES (
  'classification'='parquet', 
  'compressionType'='none');


Run the following query on the recently created ny_taxi_summary table to retrieve the first 10 rows to validate the data:

select * from default.ny_taxi_summary limit 10;

Clean up

To prevent future charges, complete the following steps:

  1. On the Amazon S3 console, delete the S3 bucket you created to store the Amazon MWAA DAG, scripts, and logs.
  2. On the Athena console, drop the table you created:
    drop table default.ny_taxi_summary;

  3. On the Amazon MWAA console, navigate to the environment that you created and choose Delete.
  4. On the EMR Studio console, delete the application.

To delete the application, navigate to the List applications page. Select the application that you created and choose Actions → Stop to stop the application. After the application is in the STOPPED state, select the same application and choose Actions → Delete.

Conclusion

Data engineering is a critical component of many organizations, and as data volumes continue to grow, it’s essential to find ways to streamline data processing workflows. The combination of Amazon MWAA, EMR Serverless, and Athena provides a powerful solution to build, run, and manage data pipelines efficiently. With this end-to-end data processing pipeline, data engineers can easily process and analyze large amounts of data quickly and cost-effectively without the need to manage complex infrastructure. The integration of these AWS services provides a robust and scalable solution for data processing, helping organizations make informed decisions based on their data insights.

Now that you’ve seen how to submit Spark jobs on EMR Serverless via Amazon MWAA, we encourage you to use Amazon MWAA to create a workflow that will run PySpark jobs via EMR Serverless.

We welcome your feedback and inquiries. Please feel free to reach out to us if you have any questions or comments.


About the authors

Rahul Sonawane is a Principal Analytics Solutions Architect at AWS with AI/ML and Analytics as his area of specialty.

Gaurav Parekh is a Solutions Architect helping AWS customers build large scale modern architecture. He specializes in data analytics and networking. Outside of work, Gaurav enjoys playing cricket, soccer and volleyball.


Audit History

December 2023: This post was reviewed for technical accuracy by Santosh Gantaram, Sr. Technical Account Manager.

New for AWS Amplify – Query MySQL and PostgreSQL database for AWS CDK

Post Syndicated from Channy Yun original https://aws.amazon.com/blogs/aws/new-for-aws-amplify-query-mysql-and-postgresql-database-for-aws-cdk/

Today we are announcing the general availability to connect and query your existing MySQL and PostgreSQL databases with support for AWS Cloud Development Kit (AWS CDK), a new feature to create a real-time, secure GraphQL API for your relational database within or outside Amazon Web Services (AWS). You can now generate the entire API for all relational database operations with just your database endpoint and credentials. When your database schema changes, you can run a command to apply the latest table schema changes.

In 2021, we announced AWS Amplify GraphQL Transformer version 2, enabling developers to develop more feature-rich, flexible, and extensible GraphQL-based app backends even with minimal cloud expertise. This new GraphQL Transformer was redesigned from the ground up to generate extensible pipeline resolvers to route a GraphQL API request, apply business logic, such as authorization, and communicate with the underlying data source, such as Amazon DynamoDB.

However, customers wanted to use relational database sources for their GraphQL APIs such as their Amazon RDS or Amazon Aurora databases in addition to Amazon DynamoDB. You can now use @model types of Amplify GraphQL APIs for both relational database and DynamoDB data sources. Relational database information is generated to a separate schema.sql.graphql file. You can continue to use the regular schema.graphql files to create and manage DynamoDB-backed types.

When you simply provide any MySQL or PostgreSQL database information, whether behind a virtual private cloud (VPC) or publicly accessible on the internet, AWS Amplify automatically generates a modifiable GraphQL API that securely connects to your database tables and exposes create, read, update, or delete (CRUD) queries and mutations. You can also rename your data models to be more idiomatic for the frontend. For example, a database table is called “todos” (plural, lowercase) but is exposed as “ToDo” (singular, PascalCase) to the client.

With one line of code, you can add any of the existing Amplify GraphQL authorization rules to your API, making it seamless to build use cases such as owner-based authorization or public read-only patterns. Because the generated API is built on AWS AppSync‘ GraphQL capabilities, secure real-time subscriptions are available out of the box. You can subscribe to any CRUD events from any data model with a few lines of code.

Getting started with your MySQL database in AWS CDK
The AWS CDK lets you build reliable, scalable, cost-effective applications in the cloud with the considerable expressive power of a programming language. To get started, install the AWS CDK on your local machine.

$ npm install -g aws-cdk

Run the following command to verify the installation is correct and print the version number of the AWS CDK.

$ cdk –version

Next, create a new directory for your app:

$ mkdir amplify-api-cdk
$ cd amplify-api-cdk

Initialize a CDK app by using the cdk init command.

$ cdk init app --language typescript

Install Amplify’s GraphQL API construct in the new CDK project:

$ npm install @aws-amplify/graphql-api-construct

Open the main stack file in your CDK project (usually located in lib/<your-project-name>-stack.ts). Import the necessary constructs at the top of the file:

import {
    AmplifyGraphqlApi,
    AmplifyGraphqlDefinition
} from '@aws-amplify/graphql-api-construct';

Generate a GraphQL schema for a new relational database API by executing the following SQL statement on your MySQL database. Make sure to output the results to a .csv file, including column headers, and replace <database-name> with the name of your database, schema, or both.

SELECT
  INFORMATION_SCHEMA.COLUMNS.TABLE_NAME,
  INFORMATION_SCHEMA.COLUMNS.COLUMN_NAME,
  INFORMATION_SCHEMA.COLUMNS.COLUMN_DEFAULT,
  INFORMATION_SCHEMA.COLUMNS.ORDINAL_POSITION,
  INFORMATION_SCHEMA.COLUMNS.DATA_TYPE,
  INFORMATION_SCHEMA.COLUMNS.COLUMN_TYPE,
  INFORMATION_SCHEMA.COLUMNS.IS_NULLABLE,
  INFORMATION_SCHEMA.COLUMNS.CHARACTER_MAXIMUM_LENGTH,
  INFORMATION_SCHEMA.STATISTICS.INDEX_NAME,
  INFORMATION_SCHEMA.STATISTICS.NON_UNIQUE,
  INFORMATION_SCHEMA.STATISTICS.SEQ_IN_INDEX,
  INFORMATION_SCHEMA.STATISTICS.NULLABLE
      FROM INFORMATION_SCHEMA.COLUMNS
      LEFT JOIN INFORMATION_SCHEMA.STATISTICS ON INFORMATION_SCHEMA.COLUMNS.TABLE_NAME=INFORMATION_SCHEMA.STATISTICS.TABLE_NAME AND INFORMATION_SCHEMA.COLUMNS.COLUMN_NAME=INFORMATION_SCHEMA.STATISTICS.COLUMN_NAME
      WHERE INFORMATION_SCHEMA.COLUMNS.TABLE_SCHEMA = '<database-name>';

Run the following command, replacing <path-schema.csv> with the path to the .csv file created in the previous step.

$ npx @aws-amplify/cli api generate-schema \
    --sql-schema <path-to-schema.csv> \
    --engine-type mysql –out lib/schema.sql.graphql

You can open schema.sql.graphql file to see the imported data model from your MySQL database schema.

input AMPLIFY {
     engine: String = "mysql"
     globalAuthRule: AuthRule = {allow: public}
}

type Meals @model {
     id: Int! @primaryKey
     name: String!
}

type Restaurants @model {
     restaurant_id: Int! @primaryKey
     address: String!
     city: String!
     name: String!
     phone_number: String!
     postal_code: String!
     ...
}

If you haven’t already done so, go to the Parameter Store in the AWS Systems Manager console and create a parameter for the connection details of your database, such as hostname/url, database name, port, username, and password. These will be required in the next step for Amplify to successfully connect to your database and perform GraphQL queries or mutations against it.

In the main stack class, add the following code to define a new GraphQL API. Replace the dbConnectionConfg options with the parameter paths created in the previous step.

new AmplifyGraphqlApi(this, "MyAmplifyGraphQLApi", {
  apiName: "MySQLApi",
  definition: AmplifyGraphqlDefinition.fromFilesAndStrategy(
    [path.join(__dirname, "schema.sql.graphql")],
    {
      name: "MyAmplifyGraphQLSchema",
      dbType: "MYSQL",
      dbConnectionConfig: {
        hostnameSsmPath: "/amplify-cdk-app/hostname",
        portSsmPath: "/amplify-cdk-app/port",
        databaseNameSsmPath: "/amplify-cdk-app/database",
        usernameSsmPath: "/amplify-cdk-app/username",
        passwordSsmPath: "/amplify-cdk-app/password",
      },
    }
  ),
  authorizationModes: { apiKeyConfig: { expires: cdk.Duration.days(7) } },
  translationBehavior: { sandboxModeEnabled: true },
});

This configuration assums that your database is accessible from the internet. Also, the default authorization mode is set to Api Key for AWS AppSync and the sandbox mode is enabled to allow public access on all models. This is useful for testing your API before adding more fine-grained authorization rules.

Finally, deploy your GraphQL API to AWS Cloud.

$ cdk deploy

You can now go to the AWS AppSync console and find your created GraphQL API.

Choose your project and the Queries menu. You can see newly created GraphQL APIs compatible with your tables of MySQL database, such as getMeals to get one item or listRestaurants to list all items.

For example, when you select items with fields of address, city, name, phone_number, and so on, you can see a new GraphQL query. Choose the Run button and you can see the query results from your MySQL database.

When you query your MySQL database, you can see the same results.

How to customize your GraphQL schema for your database
To add a custom query or mutation in your SQL, open the generated schema.sql.graphql file and use the @sql(statement: "") pass in parameters using the :<variable> notation.

type Query {
     listRestaurantsInState(state: String): Restaurants @sql("SELECT * FROM Restaurants WHERE state = :state;”)
}

For longer, more complex SQL queries, you can reference SQL statements in the customSqlStatements config option. The reference value must match the name of a property mapped to a SQL statement. In the following example, a searchPosts property on customSqlStatements is being referenced:

type Query {
      searchPosts(searchTerm: String): [Post]
      @sql(reference: "searchPosts")
}

Here is how the SQL statement is mapped in the API definition.

new AmplifyGraphqlApi(this, "MyAmplifyGraphQLApi", { 
    apiName: "MySQLApi",
    definition: AmplifyGraphqlDefinition.fromFilesAndStrategy( [path.join(__dirname, "schema.sql.graphql")],
    {
        name: "MyAmplifyGraphQLSchema",
        dbType: "MYSQL",
        dbConnectionConfig: {
        //	...ssmPaths,
     }, customSqlStatements: {
        searchPosts: // property name matches the reference value in schema.sql.graphql 
        "SELECT * FROM posts WHERE content LIKE CONCAT('%', :searchTerm, '%');",
     },
    }
  ),
//...
});

The SQL statement will be executed as if it were defined inline in the schema. The same rules apply in terms of using parameters, ensuring valid SQL syntax, and matching return types. Using a reference file keeps your schema clean and allows the reuse of SQL statements across fields. It is best practice for longer, more complicated SQL queries.

Or you can change a field and model name using the @refersTo directive. If you don’t provide the @refersTo directive, AWS Amplify assumes that the model name and field name exactly match the database table and column names.

type Todo @model @refersTo(name: "todos") {
     content: String
     done: Boolean
}

When you want to create relationships between two database tables, use the @hasOne and @hasMany directives to establish a 1:1 or 1:M relationship. Use the @belongsTo directive to create a bidirectional relationship back to the relationship parent. For example, you can make a 1:M relationship between a restaurant and its meals menus.

type Meals @model {
     id: Int! @primaryKey
     name: String!
     menus: [Restaurants] @hasMany(references: ["restaurant_id"])
}

type Restaurants @model {
     restaurant_id: Int! @primaryKey
     address: String!
     city: String!
     name: String!
     phone_number: String!
     postal_code: String!
     meals: Meals @belongsTo(references: ["restaurant_id"])
     ...
}

Whenever you make any change to your GraphQL schema or database schema in your DB instances, you should deploy your changes to the cloud:

Whenever you make any change to your GraphQL schema or database schema in your DB instances, you should re-run the SQL script and export to .csv step mentioned earlier in this guide to re-generate your schema.sql.graphql file and then deploy your changes to the cloud:

$ cdk deploy

To learn more, see Connect API to existing MySQL or PostgreSQL database in the AWS Amplify documentation.

Now available
The relational database support for AWS Amplify now works with any MySQL and PostgreSQL databases hosted anywhere within Amazon VPC or even outside of AWS Cloud.

Give it a try and send feedback to AWS re:Post for AWS Amplify, the GitHub repository of Amplify GraphQL API, or through your usual AWS Support contacts.

— Channy

P.S. Specially thanks to René Huangtian Brandel, a principal product manager at AWS for his contribution to write sample codes.

[$] Project Bluefin: A customized Fedora Silverblue desktop image

Post Syndicated from jake original https://lwn.net/Articles/954059/

So-called “immutable” Linux distributions have been in development for
some time, but (unless you count Chrome OS) haven’t gained much traction. Project Bluefin, is a heavily
customized set of Fedora
Silverblue
images coming from the Universal Blue community; they are
designed to deliver a reliable Linux desktop that’s as easy to use as a
Chromebook but more customizable. Bluefin’s mission is to change up
the desktop experience and attract a new generation of open-source
contributors with a “cloud-native”
take
on developing and delivering the operating system.

ASRock Industrial 4×4 BOX-7840U Review AMD Powered NUC Form Factor

Post Syndicated from Patrick Kennedy original https://www.servethehome.com/asrock-industrial-4x4-box-7840u-review-amd-powered-nuc-form-factor/

In our ASRock Industrial 4×4 BOX-7840U review, we see how this AMD Ryzen 7040 series mini PC compares to others we have tested

The post ASRock Industrial 4×4 BOX-7840U Review AMD Powered NUC Form Factor appeared first on ServeTheHome.

Peeking into the crystal ball: What 2023 cyber threats told us about 2024

Post Syndicated from sabeen malik original https://blog.rapid7.com/2023/12/12/peeking-into-the-crystal-ball-what-2023-cyber-threats-told-us-about-2024/

Peeking into the crystal ball: What 2023 cyber threats told us about 2024

Stepping into 2024 feels like opening the latest best-selling mystery novel – you know there’s adventure ahead, but the plot is still up in the air.

In the twist-riddled world of cybersecurity, we can’t help but ask, What’s next on the digital horizon? Sure, every business would like 2024 to be “less surprising” in terms of cyber threats, but let’s be honest, that’s not going to happen.

Even though we’re surely in for more than a few surprises in the coming year, there are ways we can be better prepared. So sit back and relax as we venture through some insights we’ve gained in 2023 and offer ways you can put them into practice in the coming year.

Prediction 1: Ransomware actors burning through zero-days

RDP (remote desktop protocol) has long been the initial entry vector of choice for ransomware groups, closely followed by the less-so sophisticated email. However the MOVEit and SysAid campaigns show change is brewing.

Rapid7 has observed an increasing number of zero-day vulnerabilities being exploited by ransomware groups, and it’s unlikely this trend will abate. Forget the mindset that ransomware actors just go after “the low hanging fruit”; they are now exploiting zero-day vulnerabilities at mass scale.

This trend is seeing criminal groups that to date have not demonstrated any real capable skills in gaining access to previously unidentified vulnerabilities, exploit them and gain a foothold into victim networks. This demonstrates that potentially something is afoot in the ransomware ecosystem. For organizations, the message is simple: get your vulnerability management and patching procedures in place and do it now. Being proactive when it comes to dealing with vulnerabilities that are being exploited in the wild is imperative.

Prediction 2:  Cyber Risk and Vulnerability disclosures will lead to consolidation around better risk management practices.

With the growing number of regulatory disclosures for cyber risk management practices and incidents, the emergence of GenAI as a potent tool for cyber attacks, more ransomware hijacks, and the lack of common lexicon around cyber risk, businesses are truly going to have to spend more time than ever determining their risk profile, and subsequently thinking about the tools and services that they will need to address the risks.

This means that more leaders will be deciding between whether to deal with compliance risk mitigation and/or creating agile cyber risk management strategies. The leaders that understand this moment as a rallying call to uplevel the conversation about systemic risks will set their business up for success by not getting sidetracked by playing compliance whack-a-mole, but by investing in a strategic vision for dealing with cyber business risks. Those businesses will also be able to withstand the scrutiny related to more global requirements for disclosure of both cyber risk management and cyber incident response and procedures.  More disclosure may not necessarily lead to clarity in the short term on what are best practices, but over the long term we will see more consolidation on best practices on cyber disclosures and risk management practices.

At the same time, governments will also be struggling to find the right balance on how to incentivize risk management rather than compliance risk mitigation whack-a-mole if they continue to introduce regulations that are not driven by harmonization around best practices and product security instead of first to market on regulations.

Here’s the catch: as regulations become more comprehensive, they may inadvertently nudge the industry towards a more consolidated structure – a double edged sword.

Prediction 3: Growth of real-time information sharing within global public-private cyber partnerships

The regulatory dance floor will definitely become more crowded in 2024, especially with AI cutting in.

This new dance partner will be adding to the complexity of tools needed to deal with cyber risk mitigation and will lead to more robust and global public-private partnerships. We might see something like a global cybersecurity flash mob in 2024. Instead of just sharing the usual threat intelligence of cyber threats and cyber risks, governments and businesses will join hands to share threat intel, resources and bolster defenses in concentrated ways to deal with specific threats. Ultimately, moving beyond the historical PPP’s of quarterly meetings, to a more real-time sharing approach in order to deal with the diminishing timelines between initial entry vectors to final stage payloads.

It will be interesting to see if more action oriented partnerships bolster capacity and cyber defenses. In order for such an approach to be successful, a mentality of “information sharing” and an open door of communication must be developed.

Prediction 4: Cloudy with a chance of threats

The cloud will continue to be a critical cyber battleground. And in the coming year, an emerging concern will likely be the misuse of commercial cloud service providers (CSPs). That’s because cybercriminals are no longer relying on known command-and-control servers; instead, they’re turning to commercial CSPs for cover to host malicious content.

It’s a clever trend, and it comes back to the game of hide-and-seek, with attackers exploiting the cloud’s anonymity and legitimacy, and blending their activities with legitimate services. Combatting this threat requires more innovative solutions, such as those leveraging AI and advanced automation techniques — as well as heightened vigilance — in the cloud. Organizations need advanced risk scoring across cloud environments, so security teams get complete visibility that eliminates blind spots and enables them to effectively prioritize remediation actions.

Prediction 5: AI and automation will be table stakes

As mentioned in prediction 4, innovations in AI and automation promise to effectively address an ever-increasing volume of attacks. Seeing threat intelligence is one thing, but it’s a completely different ball game to be doing something about it. This is where more automated responses come into play. With AI coming and more advanced automation techniques, the majority of detection and remediation or prevention work will occur automatically.

But, let’s not get ahead of ourselves. The inevitable rush to market for some solutions means that some AI capabilities will miss the mark. Therefore, organizations that adopt AI solutions must ensure that they truly improve cyber resilience without presenting new cyber risks.

Over the next year, a growing AI use case will be the use of AI synthetic media (i.e., deep fakes) and identity management. Governments will have the challenge of navigating the tricky space between the problematic use case of biometric technology and synthetic media, while businesses will have to understand how to manage the risks with identity and access management.

Be ready for 2024

So, there’s our bird’s eye view of what the cyber landscape could look like in 2024. But as always, there will be many shifts, evolutions, and transformations in the new year, some unprecedented and some expected. Regardless, practitioners must stay on their toes, remain vigilant, and aim for resilience. Here’s to a more prepared, secure — and less stressful — 2024.

For more thoughts from our team on what 2024 could bring, watch the Top Cybersecurity Predictions webinar on-demand.

Security updates for Tuesday

Post Syndicated from corbet original https://lwn.net/Articles/954706/

Security updates have been issued by Debian (libreoffice and webkit2gtk), Fedora (java-1.8.0-openjdk and seamonkey), Oracle (apr, edk2, kernel, and squid:4), Red Hat (postgresql:12, tracker-miners, and webkit2gtk3), SUSE (curl, go1.20, go1.21, hplip, openvswitch, opera, squid, and xerces-c), and Ubuntu (binutils, ghostscript, libreoffice, linux, linux-aws, linux-aws-5.15, linux-azure, linux-azure-5.15,
linux-azure-fde, linux-azure-fde-5.15, linux-gcp, linux-gke,
linux-hwe-5.15, linux-ibm, linux-ibm-5.15, linux-kvm, linux-nvidia,
linux-oracle, linux-oracle-5.15, linux-raspi, linux, linux-aws, linux-aws-5.4, linux-azure, linux-azure-5.4,
linux-bluefield, linux-gcp, linux-gcp-5.4, linux-hwe-5.4, linux-ibm,
linux-ibm-5.4, linux-kvm, linux-xilinx-zynqmp, postfixadmin, python3.11, and webkit2gtk).

From Google to Generative AI: Ranking top Internet services in 2023

Post Syndicated from João Tomé original http://blog.cloudflare.com/radar-2023-year-in-review-internet-services/


From Google to Generative AI: Ranking top Internet services in 2023

Ask nearly any Internet user, and they are bound to have their own personal list of favorite sites, applications, and Internet services for news, messaging, video, AI chatbots, music, and more. Sum that question up across a lot of users in a lot of different countries, and you end up with a sense of the most popular websites and services in the world. In a nutshell, that’s what this blog post is about: how humans interacted with the online world in 2023 from what Cloudflare observed.

Building on similar reports we’ve done over the past two years, we have compiled a ranking of the top Internet properties of 2023. In addition to our overall ranking, we chose 9 categories to focus on. One of these is a new addition in 2023: Generative AI. Here are the 9 categories we’ll be digging into:

1. Generative AI
2. Social Media
3. E-commerce
4. Video Streaming
5. News
6. Messaging
7. Metaverse & Gaming
8. Financial Services
9. Cryptocurrency Services

Our method for calculating the results is the same as in 2022: we analyze anonymized DNS query data from our 1.1.1.1 public DNS resolver, used by millions of people around the world. To build the lists of Internet services, we use two additional methods. First, we aggregate domains that belong to one online service. For instance, for Twitter/X, we include twitter.com, t.co, and x.com among others. Second, we reference our large source list of domains and identify the sites that provide services to humans. That means that our rankings do not include every domain seen in the data (for example, we exclude domains such as root-servers.net and cloudflare-dns.com). A site’s overall ranking is relative to other sites that meet these criteria; its rank within a category is relative to other sites in the same category. That’s important to note: just because a site has gone down in the rankings, it doesn’t necessarily mean its traffic has declined — it could just be that other sites’ traffic increased. Similarly, the inverse is true. What we’re doing here isn’t tracking absolute traffic, but rather, relative popularity.

With that, we can begin our analysis. Following the success of OpenAI’s ChatGPT launch on November 30, 2022, Generative AI has captivated both the world and the news. OpenAI is now close to ranking among our top 100 most popular Internet services overall, rising from #200 in January.

In e-commerce, low price and fast fashion marketplace Temu experienced similar growth, becoming the year’s surprise by surpassing Shein and other major players, reaching #7 in its category. And Black Friday was the best day for several other e-commerce services, as well as payment services such as PayPal, Stripe and Klarna.

In social media, X/Twitter lost some ground in our Overall ranking, but maintained its status as a significant online discussion platform. New additions like Threads by Instagram, launched in July, are still gaining their footing in the category. Additionally, we observed notable news-related trends linked to events such as the March 2023 United States banking crisis, the Titan submersible implosion, the Wagner Group rebellion, and the October 7 Hamas attack on Israel.

We close our report out with a discussion of a range of trends that don’t fit neatly into our other categories, including a look at Taylor Swift’s and Beyoncé websites — both of which appeared in our overall traffic rankings.

Keep reading for a detailed look at the evolution of trends throughout the year. For more, visit our 2023 Cloudflare Radar Year in Review microsite. Along with the lists of most popular Internet services, the Year in Review site and its associated blog post explore a number of additional metrics.

Google is #1. Facebook, Apple and TikTok follow

Since we began reporting these rankings in 2021, we always start with an Overall Top 10 list. These are the services that are the top Internet properties globally in 2023, based on DNS traffic through our 1.1.1.1 resolver. Unsurprisingly, Google (we’re including here services like Google Maps and Google Flights that use google.com, for example) remained the #1 most popular Internet service in our Overall ranking. Since we implemented our new ranking method last year, no other service has challenged Google’s position as the #1 leader in our rankings.

  1. Google
  2. Facebook
  3. Apple
  4. TikTok
  5. Microsoft
  6. YouTube
  7. AWS
  8. Instagram
  9. Amazon
  10. iCloud

Beyond Google, Facebook was consistently #2, while Apple (that uses, for example, apple.com for several use cases, including on iPhone related services) was generally #3, except when TikTok took the spot in April. Microsoft mostly held the #5 ranking, although it sometimes traded places with YouTube at #6.

AWS — we’re separating it from Amazon by using domains like amazonaws.com — held a firm position at #7, and Instagram was clearly #8 through the year, with a few exceptions involving Amazon. The #10 position changed more frequently, alternating between iCloud (for which we use domains such as icloud.com, distinct from apple.com), Netflix (mainly on weekends), and Microsoft Office. In the chart below, you can follow the evolution of the top Internet services in our Overall ranking throughout the year.

In 2022, X/Twitter ranked as high as #10 in our Overall ranking, but never reached this spot in 2023. We’ll talk more about X/Twitter’s performance in the Social Media category below.

Ready to face the Generative AI era?

The Generative AI category became a global phenomenon in 2023, though it started gaining attention in late November 2022 with the launch of ChatGPT from OpenAI. We tested it on our ‘This Week in NET’ show on December 2, 2022, and were impressed. It’s not surprising that OpenAI was popular early in 2023 and topped this list. Other generative AI services also emerged during the year.

Top 10 — Generative AI services 2023

  1. OpenAI
  2. character.ai
  3. QuillBot
  4. Hugging Face
  5. Poe
  6. Perplexity
  7. Wordtune
  8. Bard
  9. ProWritingAid
  10. Voicemod

Significant changes occurred below Open AI’s #1 spot throughout the year. After mid-year, we noticed a shift, when character.ai took the #2 spot from Quillbot. Quora’s Poe AI, which combines several AI chatbots including ChatGPT, entered the top 10 in late March, holding the #3 spot until late April before settling at #4.

This figure shows movement among the Generative AI services that were more popular later in the year:

AI model platform Hugging Face was typically #5, but Google’s Bard also reached #5 in November. Bard, launched in limited regions in March, made our top 10 within the category after its broader release in Europe and Brazil. It was #7 from July to September before peaking later in the year.

Other Generative AI services that became important in our list later in 2023 include Anthropic’s Claude (whose new model, Claude 2, launched in July, though it is not yet available in the European Union). It was #5 in August before dropping to #6 after September. Perplexity AI, a ChatGPT competitor, was #8 after September.

Midjourney, which relies on Discord bot commands to create artwork, peaked at #3 in late March but started declining, dropping to #5 in April and May, and then to #10 in September.

Other AI-inclusive services like Prowritingaid, Voicemod, and Wordtune were more popular earlier in the year but became less so later on. Bardeen was in the top 10 in May and June, while Descript appeared at #9 between March and May.

Notable trends that we observed when looking at trends for Generative AI services in our larger Overall ranking include:

  • OpenAI, which was around #200 in our Overall ranking in early January, saw a significant rise between March and April (OpenAI launched GPT-4 and plugins during that time) and is now near the top 100. It peaked early in November (#104 on November 9), right after OpenAI’s first developer conference on November 6 in San Francisco. The brief removal of Sam Altman in late November didn’t have a noticeable impact.
  • character.ai‘s growth was similar to OpenAI, rising from the top 500 in early January to around #200 recently. Quillbot remained steady throughout the year, around #258.
  • Poe AI peaked at #276 on June 18 and is now around #290. Claude AI was around #380 in November after a late August peak at #337.

Social media: The Facebook (not X) effect

An analysis from Kepios estimates that there are 4.95 billion social media users around the world in 2023, comprising 61.4% of the world population, so this category plays a big role in our everyday life as a central stage for communication, information, and general attention.

Unsurprisingly, social media platforms such as Facebook, TikTok, and Instagram rank high in our most popular Internet properties and are featured in our Top 10 Overall Internet services list.

Within our Social Media category list, the top five remained the same as last year. Facebook was #1, followed by TikTok (#2), Instagram (#3), X/Twitter (#4), and Snapchat (#5).

Top 10 — Social Media services 2023

  1. Facebook
  2. TikTok
  3. Instagram
  4. X/Twitter
  5. Snapchat
  6. LinkedIn
  7. Discord
  8. Reddit
  9. Pinterest
  10. Kwai

In contrast to 2022, when Twitter (renamed X in July 2023) and Instagram often swapped places at #3, this year X/Twitter never challenged Instagram in our rankings.

LinkedIn held a strong #6, with Discord challenging it during a few days in January and April.

Reddit, usually at #8, competed with Discord for the #7 spot in February and March but fell back to #8 in April, concurrent with the controversy around Reddit API changes. Reddit often challenged Discord on weekdays, while Discord was more popular on weekends. Following these are Pinterest (#9) and Kwai, the Chinese video app popular in Brazil, Indonesia, and other countries (#10). Here’s the top 10 chart across 2023:

A Twitter (or X) drop after a possible football (or soccer) peak

Let’s continue on social media, but leave its specific category and examine how social media services fared in our Overall ranking where bigger shifts between services are seen.

The graph below depicts the position of Instagram and Twitter in the Overall ranking. Instagram consistently ranked in the top 10, typically holding positions #8 or #9 with brief dips to #10 around May 2023.

X/Twitter, which reached as high as #8 overall last year following the start of the war in Ukraine, ended 2022 at #12 and continued its downward trend in 2023. It started between #12 and #16, lower than last year, and ended between #13 and #19. Here’s how these two evolved in 2023:

We noticed that X/Twitter was particularly popular on weekends, peaking at #11 or #12 between April 15 and June 10. ​​Specifically, it ranked at #11 in the weekends before May 14 and then at #12 from May 14 to June 10. This coincides with key moments in European football competitions. However, after the European Champions League final on June 10, X/Twitter never reached those heights again.

Let’s explore this possible football (or soccer) Twitter trend. X/Twitter’s rankings peaked around significant moments in the English Premier League (arguably the most-watched sports league in the world), particularly when Manchester City made crucial advances to their title after April 15. The trend lasted until the European Champions League final on June 10, where City’s victory and historic treble coincided with X/Twitter’s final peak ranking of the year — the weekend of June 3, when there were already no Premier League games, and X/Twitter dropped to #13.

Alternatives to X/Twitter: Mastodon, Threads, and others

Tumblr, a more established platform than other recent alternatives, fluctuated between #125 and #153 in our Overall ranking, showing a downward trend. Close behind was an aggregation of several hundred Mastodon servers, ranking between #160 and #200.

Threads from Instagram/Meta (seen in the next chart), peaked at #227 on July 6 in our Overall ranking, then dropped but recovered to around #300 after late August, and did not make it into the top 10 social media services ranking for 2023. For context, Kwai (#10 in the Social Media ranking) usually ranks around our Overall top 50.

Bluesky, a newer entrant, first appeared in our Top 500 in late August, with a first spike at #432 on September 19 and reaching the top 400 in November, peaking at #397 on November 19. Truth Social had a peak at #318 on August 23 but then dropped, averaging around #450 in November.

Other social apps including Hive Social, Counter Social, Post.News, T2/Pebble, Parler, etc., didn’t appear in our Overall 500 ranking.

Here are some other trends we observed among social media apps, and how they did in our Overall ranking:

  • Snapchat was more popular on weekends, fluctuating between #18 and #21 overall, with improved performance after October, with several days at #18, close to X/Twitter.
  • Discord was also more popular on weekends, varying between #24 and #35, peaking in April at #24. It had a similar trend to Midjourney, the generative AI image service that generates images for its users on Discord. It ended the year around #32.
  • Reddit, which was more popular on weekdays, dropped in our ranking during the summer in the Northern Hemisphere, in contrast with other social media services, and moved between #30 and #38.
  • Quora was also more popular on weekdays, and fluctuated between #116 and #146, dropping in summer and settling around #130 in November.
  • Tinder, the social dating app, fell from a peak of #124 to #133 in November, and is much more popular on Sundays.
  • OnlyFans showed steady growth, peaking at #148 in early July and settling around #175. The adult oriented social content subscription service was more popular on the weekends, particularly on Sundays.
  • BeReal, a French social media app for daily photo sharing, approached but did not enter the Social Media top 10 ranking, and grew to a peak of #141 in our Overall ranking in September but fell in November.

E-commerce: Temu means growth

E-commerce remains as relevant as ever, something that is especially evident in our recent Cyber Week blog post. Amazon consistently tops the category, with Taobao as a solid #2. In 2022, eBay was mostly #2 but has now dropped to #3, although it had several days in early 2023 in the #2 spot, as well as on June 19.

Top 10 — E-commerce services 2023

  1. Amazon
  2. Taobao
  3. eBay
  4. Shopify
  5. Alibaba
  6. AliExpress
  7. Temu
  8. Rakuten
  9. Mercado Libre
  10. Walmart

Throughout the year, Shopify was a solid #4, overtaking eBay at #3 on Black Friday. Alibaba was #5, peaking on Singles’ Day, November 11. AliExpress followed at #6.

Rakuten and Temu battled for the #7 ranking in 2023. Temu, the Chinese-owned and Boston-based low price and fast fashion marketplace (launched in the US in September 2022) was definitely one of the surprises of the year. It expanded to Canada, Australia and New Zealand in February, and ended 2023 as #7 in front of Rakuten. It ranked #6 at times during the year, including Black Friday. Temu wasn’t in our top 10 in 2022.

The South American e-commerce platform Mercado Libre, the American retail giant Walmart, and the fast fashion brand Shein, ‘battled’ during the year for the 9th and 10th place rankings in the E-commerce category.

Looking at how e-commerce sites did in our Overall ranking, we observed the following trends:

  • Amazon finished the year at #9 overall, reaching #8 on its Prime Day shopping event (July 11-12) and Black Friday.
  • Shein, the Chinese fast fashion brand, showed growth, starting at around #130 in January and ending around #120. It’s a similar trend to the one we observed in 2022.
  • Temu was not in the top 200 in January but finished in the top 100, peaking on Black Friday at #84 — a similar trend was seen in OpenAI performance. Temu also overtook Shein in May 2023.
  • Best Buy also peaked on Black Friday in 2023, as did Nike, Adidas, Victoria’s Secret, and H&M. Zara’s best day was November 23.
  • Target first peaked on November 12 at #134 and again on Black Friday at #135.
  • Ikea didn’t perform as well during Black Friday week in our ranking, and had its 2023 peak on June 14-15 (#172), when it launched its own summer sale and a remote interior design AI related strategy.
  • Alibaba’s highest position in 2023 was on Single’s Day, November 11, at #67.

Video streaming: YouTube and Netflix remain uncontested

Video streaming platforms continue to play a central role in entertaining us. YouTube remains the top service for all video streaming, with Netflix as a close second. However, within our rankings, Netflix is the clear leader among paid streaming services, followed by Disney Plus, Amazon Prime Video, Hulu, and HBO/HBO Max.

Top 10 — Video streaming services 2023

  1. YouTube
  2. Netflix
  3. Twitch
  4. Roku
  5. Disney Plus
  6. Prime Video
  7. Hulu
  8. HBO/Max
  9. Vimeo
  10. Pluto TV

Twitch, which has a significant amount of video game live-streaming content, secured the #3 spot, as it did in 2022. Roku, a digital media player that also offers streaming services, ranked #4. Following are Disney Plus, Prime Video, Hulu, and HBO/Max (now known as Max), with Vimeo surpassing Pluto TV later in the year. Dailymotion also made a top 10 appearance in late February for a few days.

Throughout the year, Disney+ occasionally challenged Roku, especially on weekends following the premiere of the Star Wars-related miniseries Ahsoka on August 23.

Looking at how video streaming services performed in our Overall ranking, we found:

  • Netflix peaked at #9 on Sundays in April and May, coinciding with the release of ‘The Diplomat‘ starring Keri Russell — it was the most watched show on the platform at that time. Netflix, Prime Video, HBO/Max, were more popular on weekends.
  • Prime Video was more popular earlier in the year, peaking at #50 on March 26, the weekend ‘Top Gun: Maverick‘ premiered.
  • HBO/Max has been dropping in our Overall ranking since September but remains in the top 100.
  • Peacock, NBC’s streaming service, peaked at #111 in October and was more popular on Sundays.
  • Paramount Plus has been falling in our Overall ranking, down to around #156 in November, and is also more popular on Sundays.

The News: Globo and BBC global perspectives

News organizations worldwide play a crucial role in keeping the public informed, especially during times of crisis like pandemics, wars, or financial uncertainty. With that in mind, the ranking of news organizations also highlights a few newsworthy trends.

Top 10 — News services 2023

  1. Globo
  2. BBC
  3. Fox News
  4. CNN
  5. NY Times
  6. Daily Mail
  7. Washington Post
  8. The Guardian
  9. NPR
  10. Wall Street Journal

Last year, BBC and Globo tied for #1 in this category. In 2023, Globo — a Brazilian media conglomerate popular in South America, was consistently #1, followed by the BBC, the UK’s national broadcaster. The only exception was on June 20, when the BBC was #1. This coincided with the Titan submersible implosion on June 18, which remained in global news for most of the following week.

Fox News and CNN also played a key role in our list. Fox News was overtaken by CNN for the #3 spot from July onwards, though Fox led again in November. The New York Times held a steady #5, followed by the Daily Mail and NPR, with the Washington Post overtaking NPR in June.

The ranking lower in the top 10 fluctuated more, with the Wall Street Journal mostly at #9, reaching #8 in September. The Guardian was #8 in October, coinciding with the start of the Hamas-Israel conflict on October 7.

Titan submersible implosion & the Hamas-Israel conflict

Notable news trends we identified in our larger Overall ranking include:

  • CNN had its 2023 peak on June 22, reaching #73, and was already higher than usual in the previous days. That was the day the United States Coast Guard announced that they had discovered debris from the Titan submersible implosion, confirming that the five people aboard were dead. The other peak of the year for CNN was February 14, Valentine’s Day.
  • The Daily Mail (#127) and The Indian Express (#389) also peaked on June 22, with Fox News also reaching one of its highest ranks on that day, although it was higher in early January.
  • German news outlet Bild also peaked on June 22, while Der Spiegel‘s highest point was the previous day, June 21.
  • BBC, had a significant peak in our Overall list on October 8, following the Hamas attack on Israel, reaching #76. It also moved higher in the news category ranking on that day, as noted above.
  • RT, the Russian news organization, showed a decline throughout the year but peaked on June 24 (#234) during the Wagner Group rebellion in Russia.
  • Israeli newspapers Times of Israel and Haaretz jumped into our Overall ranking on October 7, the day of the Hamas attack on Israel. The former had its peak on October 9 at #275, the latter on October 8 at #393.
  • The Washington Post peaked on April 4-5 (#117), coinciding with Finland joining NATO and Donald Trump’s not guilty plea after becoming the first US president to be indicted.
  • China Daily appeared in our top 500 in January, peaking on January 20 at #361, ahead of the Chinese New Year on January 22. A similar rise was seen in the South China Morning Post.
  • TMZ‘s highest rank was on October 29 (#243), following the announcement of the TV star Matthew Perry‘s death.
  • BuzzFeed, combining buzzfeed.com and buzzfeednews.com, declined in our Overall ranking, especially after the summer, falling to around #300 in November. In April 2023, it was announced that BuzzFeed News would be shutting down, and it did so in May.

Messaging: WhatsApp rules & Telegram rises

Messaging is seen as a type of social media and remains as relevant as ever including for specific communication purposes. Apple’s iMessage is not included in this category, because it doesn’t have a unique domain name whose traffic can be analyzed. Keeping that in mind, WhatsApp remained the top messaging service in 2023, consistent with its position in 2022.

Top — Messaging services 2023

  1. WhatsApp
  2. QQ
  3. Viber
  4. Telegram
  5. LINE
  6. Signal
  7. WeChat
  8. Messenger
  9. GroupMe
  10. Kik

Following WhatsApp is the Chinese service QQ, also known as Tencent QQ, which includes games and mobile payments and is popular in Asia, at #2. Viber, popular in Eastern Europe, Asia, and the Middle East, comes next. The top three are unchanged from 2022. Telegram, widely used in Eastern Europe and Asia, holds the #4 spot. LINE from Japan briefly contested this position early in 2023. Signal follows, and the Chinese app WeChat is at #7, ahead of Facebook’s Messenger.

The list concludes with Microsoft’s GroupMe and the Canadian service Kik Messenger. The standings are similar to 2022, but Telegram, WeChat, and Signal have shown improvements.

Here are other messaging trends from our Overall ranking:

  • WhatsApp‘s was generally most popular between late May and early September, though its peak came on November 14th, when it reached #13 in our Overall ranking (a rank it shared that month with X/Twitter).
  • Telegram rose to #79 overall on June 24, following the Wagner Group rebellion in Russia, which occurred June 23-24.
  • WeChat saw a significant spike the first quarter of the year on January 21-22, Chinese New Year‘s Eve and respective New Year day. WeChat peaked in 2023 at #122 on August 19, coinciding with news regarding China’s military drills around Taiwan.

Metaverse & Gaming: Roblox leads, Oculus grows

Is gaming part of the metaverse? In a sense, one could argue that it depends on the game, given that it is all about being immersed in another world. Concepts aside, we’ve included both in the same category since last year. Roblox was the uncontested winner of this category in 2023, followed by two services that are now much more than just popular gaming consoles, but also popular online gaming services: Microsoft’s Xbox and Sony’s PlayStation.

Top 10 — Metaverse & Gaming services 2023

  1. Roblox
  2. Xbox/Xbox Live
  3. Epic Games/Fortnite
  4. PlayStation
  5. Oculus
  6. Steam
  7. Electronic Arts
  8. Blizzard
  9. Nintendo
  10. Riot Games/League of Legends

Xbox and PlayStation were at the podium of the list for most of the year. The top spots were similar to 2022, but PlayStation, which was #3 last year, was surpassed by Epic Games (known for Fortnite) during the last part of 2023.

Oculus, a VR headset and also metaverse experience service owned by Meta (we’re considering domains related to those two aspects provided by Meta’s Reality Labs), rose in the rankings to #5, continuing a trend from late 2022. It reached as high as #2 on June 29, and again on October 3-4 after the announcement of Quest 3 at Meta Connect 2023.

Steam was more popular than Oculus from July to September, reaching #5. Electronic Arts was a solid #7 throughout the year, and Blizzard (famous for World of Warcraft and an Activision Blizzard subsidiary), was mostly in #8. The top 10 list ends with Nintendo and Riot Games/League of Legends ‘battling’ for the #9 and #10 spots, with the latter overtaking Nintendo since September.

Here’s the top 10 chart across 2023:

Financial services: Stripe takes the lead, Black Friday impact

The financial services sector is diverse, ranging from traditional banking to cryptocurrency-only services to tax-related services. This year, Stripe, an Irish-American payment platform, dethroned PayPal as the top service in this category.

Top 10 — Financial Services 2023

  1. Stripe
  2. PayPal
  3. Alipay
  4. TradingView
  5. Nubank (BR)
  6. Intuit
  7. American Express
  8. Binance
  9. Bradesco Bank
  10. CoinGecko

PayPal started the year at #1 but was overtaken by Stripe on most days since March. PayPal still led on some weekends during the summer and on Black Friday. TradingView lost its #3 spot to Alipay in late July. Intuit was more popular early in the year, peaking on April 18, the Tax Day in the United States, but ended mostly at #5 on weekdays (see our 2022 blog post on how Tax Day impacts related sites for more on this trend).

The Brazilian Nubank, an online-only bank or neobank, and the largest of its kind in Latin America, was a surprise at #6, often reaching higher ranks on weekends, especially on Saturdays, as it peaked on June 3 and July 1.

Focusing on specific crypto services in the Financial category (more on crypto below), we can see that Binance lost ground throughout 2023, especially after August moving between #8 and #9. This is a change from its trend in 2022, when Binance was on a growth trajectory and ranked #6. CoinGecko, a cryptocurrency data aggregation and tracking website, made a late appearance in 2023 with several days as #6 in November, and Coinbase appeared at #10 after late October. Here’s the crypto perspective in this Financial services category:

Here are other financial services trends from our Overall ranking

  • Investing.com‘s peak at #199 occurred on March 13, three days after the collapse and seizure of Silicon Valley Bank (SVB), and the same day that bank shares plunged on contagion fears — a period known as the 2023 US banking crisis. It was the same day MarketWatch also peaked (#293).
  • Online-only financial services such as PayPal (#71) and Klarna (#211) reached their annual peak in our Overall ranking on Black Friday, November 24. Stripe, however, had a clear spike at #77 on that day, but its best day was on November 10, at #68.
  • Venmo, an American mobile payment service owned by PayPal, had its best month in September.

Crypto: Binance declines and CoinGecko rises

In addition to our Financial Services category, we also evaluated cryptocurrency-related services in particular. Despite the disappearance of Sam Bankman-Fried’s FTX from our rankings after its bankruptcy in November 2022, the crypto sector continues to show several changes this year. Binance, Coinbase, and CoinGecko (a provider of crypto data tools) remain at the top of the list, a trend similar to 2022. While Binance held on to its #1 position across most of 2023, in the last month it was overtaken by CoinGecko.

Top 10 — Cryptocurrency services 2023

  1. Binance
  2. CoinGecko
  3. Coinbase
  4. CoinMarketCap
  5. NiceHash
  6. OKX
  7. MEXC
  8. CryptoCompare
  9. Kraken
  10. Crypto.com

Throughout the year, CoinGecko gained momentum, surpassing the cryptocurrency exchange platforms Binance and Coinbase in November. CoinMarketCap and NiceHash were also prominent, with CoinMarketCap reaching #4 by October. OKX, MEXC, and Crypto.com were already in the top 10 in 2022, and CryptoCompare, Kraken, and Trust: Crypto & Bitcoin, which also competed for the #10 position, were new inclusions in 2023. Kucoin and Etherscan fell out of the 2023 top 10 after being there in 2022.

What happened to Binance? In both the Financial Services and Cryptocurrency categories, we noticed Binance, a key cryptocurrency player, lost its leading position in 2023, dropping to #2 in November. The company faced challenges in July, with several top executives leaving. Then, in November, US authorities filed a lawsuit against Binance, resulting in multiple charges and fines. In our Overall ranking, Binance’s highest point was on April 19, the day after US Tax Day, ranking at #122. However, its rank fell later, though it slightly improved to around #140 by November.

Outside the categories we reviewed in the Year in Review, several notable trends emerged in our Overall ranking:

  • Beyoncé’s official site went even higher on our overall ranking than Swift’s. It appeared only on June 15, but reached #346. In the news that day (and the day before), the Financial Times reported that economists at Danske Bank believed that Beyoncé’s decision to start her world tour in Stockholm led to a surge in local hotel prices that resulted in inflation in Sweden, calling it “astonishing for a single event”. At the time, June 15, Beyoncé had a Cologne, Germany, concert, where a fan gender reveal also made the news.
  • GitHub is a top 50 site in our Overall ranking, and it showed clear growth in 2023, moving from #49 to #42 in November. It reached its highest point at #36 on January 19, when it announced reaching 100 million developers, and had another peak on May 12 at #38. Have any guesses about what contributed to these peaks (or any of the others you see in our report)? Let us know at @CloudflareRadar.
  • Spotify’s best day in 2023 was on Black Friday, November 24, when it reached #57, after showing significant growth throughout November. However, our list ends on November 25, so we couldn’t capture the impact of the recently launched Spotify Wrapped.
  • NASA. This year, NASA continued to showcase images from the James Webb Space Telescope. The NASA website peaked in our ranking on October 12 at #160, the day before the scheduled launch of NASA’s Psyche mission, aimed at exploring a unique metal-rich asteroid. Another peak occurred on April 28 at #172, coinciding with a broadcasted spacewalk at the International Space Station.
  • SpaceX. SpaceX had its best and only notable days in our ranking (within the overall top 500) on April 17 (#412), followed by April 20 (#416). April 17 marked SpaceX’s Starship orbital flight‘s first attempt, which was aborted just before launch and then resumed on April 20.
  • Craigslist. The American classified ads website saw a decline in our ranking this year, with its lowest point in November and its worst day on Black Friday, when it fell to #268.
  • DHL. The courier service improved its ranking in 2023, with its best performance in November, peaking on Thanksgiving Day, November 23,  in the US at #211.
  • NFL (National Football League). The NFL site had its first significant peak on April 29 at #189, the day of the NFL Draft, surpassing even the Super Bowl’s popularity. This peak was matched only by weekends after September 10, when it climbed as high as #160.
  • Flightradar24’s most-trafficked day was April 23, when an American Airlines flight made an emergency landing in Ohio due to an engine fire. It rose to #176 on that day.
  • Waze. The traffic app had lower rankings between June and early September (summer in the Northern Hemisphere), peaking on March 19 at #142 and on October 2 at #145.
  • Tides & Currents. The US Weather Service peaked on August 29-30 at #215 during Hurricane Idalia, described as “an unprecedented event,” as the storm approached Florida’s Gulf Coast.

Wrap up: 2023, shifting AI and e-commerce tides

The Internet plays a role in socializing, entertaining, working, communicating, learning, and staying informed when you most need it. In our popular Internet services rankings, the dominance of giants like Google and Facebook, and the relevance of TikTok and others, underscore the continued influence of established players in shaping online interactions and content consumption. However, the rise of generative AI services, notably OpenAI’s ChatGPT, signals an exciting sector that is rapidly gaining traction. Let’s see where generative AI services can go in 2024.

In the social media realm, X/Twitter seems to be losing some influence in our ranking but continues to be highly influential, and much higher than the direct competition. Mastodon, Threads, and others still have a long way to go to compete. Although not seen as direct microblogging competition, Discord and Reddit continue to show growth.

An emerging player, Temu, made significant strides in the E-commerce realm. In the cryptocurrency space, Binance lost momentum as CoinGecko gained traction. In the gaming and metaverse sectors, the highlights included Roblox’s consistency and Oculus’s growth.

Looking ahead, the trends observed in 2023 set the stage for an even more interconnected and technologically advanced future. The growing importance of AI, the steadfast popularity of social media, and the evolving dynamics in e-commerce and financial services suggest a future where humans will have to continue to adapt to the opportunities and challenges that lie ahead.

Creating rankings is a team effort that comes with its own challenges and requires careful attention and frequent updates. If you want to help us make these categorical rankings better, you can. Feedback is appreciated, including regarding other categories to include in the 2024 Year in Review.

(Our data scientist, Sabina Zejnilovic, played a crucial role in accurately gathering the Internet services data and contributed to this blog post, as did David Belson with his guidance, along with many others.)

Cloudflare 2023 Year in Review

Post Syndicated from David Belson original http://blog.cloudflare.com/radar-2023-year-in-review/


Cloudflare 2023 Year in Review

The 2023 Cloudflare Radar Year in Review is our fourth annual review of Internet trends and patterns observed throughout the year at both a global and country/region level across a variety of metrics. Below, we present a summary of key findings, and then explore them in more detail in subsequent sections.

Key findings

  • Global Internet traffic grew 25%, in line with peak 2022 growth. Major holidays, severe weather, and intentional shutdowns clearly impacted Internet traffic. 🔗
  • Google was again the most popular general Internet service, with 2021 leader TikTok falling to fourth place. OpenAI was the most popular service in the emerging Generative AI category, and Binance remained the most popular Cryptocurrency service. 🔗
  • Globally, over two-thirds of mobile device traffic was from Android devices. Android had a >90% share of mobile device traffic in over 25 countries/regions; peak iOS mobile device traffic share was 66%. 🔗
  • Global traffic from Starlink nearly tripled in 2023. After initiating service in Brazil in mid-2022, Starlink traffic from that country was up over 17x in 2023. 🔗
  • Google Analytics, React, and HubSpot were among the most popular technologies found on top websites. 🔗
  • Globally, nearly half of web requests used HTTP/2, with 20% using HTTP/3. 🔗
  • NodeJS was the most popular language used for making automated API requests. 🔗
  • Googlebot was responsible for the highest volume of request traffic to Cloudflare in 2023. 🔗

Connectivity & Speed

  • Over 180 Internet outages were observed around the world in 2023, with many due to government-directed regional and national shutdowns of Internet connectivity. 🔗
  • Aggregated across 2023, only a third of IPv6-capable requests worldwide were made over IPv6. In India, however, that share reached 70%. 🔗
  • The top 10 countries all had measured average download speeds above 200 Mbps, with Iceland showing the best results across all four measured Internet quality metrics. 🔗
  • Over 40% of global traffic comes from mobile devices. In more than 80 countries/regions, the majority of traffic comes from mobile devices. 🔗

Security

  • Just under 6% of global traffic was mitigated by Cloudflare’s systems as being potentially malicious or for customer-defined reasons. In the United States, 3.65% of traffic was mitigated, while in South Korea, it was 8.36%. 🔗
  • A third of global bot traffic comes from the United States, and over 11% of global bot traffic comes from Amazon Web Services. 🔗
  • Globally, Finance was the most attacked industry, but the timing of spikes in mitigated traffic and the target industries varied widely throughout the year and around the world. 🔗
  • Even as an older vulnerability, Log4j remained a top target for attacks during 2023. However, HTTP/2 Rapid Reset emerged as a significant new vulnerability, beginning with a flurry of record-breaking attacks. 🔗
  • 1.7% of TLS 1.3 traffic is using post-quantum encryption. 🔗
  • Deceptive links and extortion attempts were two of the most common types of threats found in malicious email messages. 🔗
  • Routing security, measured as the share of RPKI valid routes, improved globally during 2023. Significant growth was observed in countries including Saudi Arabia, the United Arab Emirates, and Vietnam. 🔗

Introduction

Cloudflare Radar launched in September 2020, and in the blog post that announced its availability, we talked about how its intent was to “shine a light on the Internet’s patterns”. Cloudflare’s network currently spans more than 310 cities in over 120 countries/regions, serving an average of over 50 million HTTP(S) requests per second for millions of Internet properties, in addition to handling over 70 million DNS requests per second on average. The data generated by this massive global footprint and scale, combined with data from complementary Cloudflare tools, enables Radar to provide unique near-real time perspectives on the patterns and trends we observe across the Internet. For the last several years (2020, 2021, 2022), we’ve been aggregating these insights into an annual Year In Review, shining a light on the Internet’s patterns over the course of that year. The new Cloudflare Radar 2023 Year In Review continues that tradition, featuring interactive charts, graphs, and maps you can use to explore notable Internet trends observed throughout this past year.

The 2023 Year In Review is organized into three sections: Traffic Insights & Trends, Connectivity & Speed, and Security. We have incorporated several new metrics this year, and have endeavored to keep underlying methodologies consistent with last year wherever possible. Website visualizations shown at a weekly granularity cover the period from January 2 through November 26, 2023. Trends for over 180 countries/regions are available on the website, with some smaller or less populated locations excluded due to insufficient data. Note that some of the metrics are presented only as a worldwide view, and will not be shown if a country/region is selected. Because of the control plane and analytics outage that occurred November 2-4, traffic data for relevant metrics has been interpolated for that three-day period.

Below, we provide an overview of the content contained within the major Year In Review sections (Traffic Insights & Trends, Connectivity & Speed, and Security), along with notable observations and key findings. In addition, we have also published a companion blog post that specifically explores trends seen across Top Internet Services.

However, the notable observations and key findings contained within this post only skim the surface of the unique insights that can be found in the Year in Review website, which we strongly encourage you to visit to explore the data in more detail and look at trends for your country/region. As you do so, we encourage you to consider how the trends presented within these blog posts and the website’s various sections impact your business or organization, and to think about how these insights can inform actions that you can take to improve user experience or enhance your security posture in the future.

Traffic Insights & Trends

Global Internet traffic grew 25%, in line with peak 2022 growth. Major holidays, severe weather, and intentional shutdowns clearly impacted Internet traffic.

Twenty-five years ago, Worldcom executives claimed that Internet traffic was doubling every 100 days (3.5 months). A quarter-century later, we know that these claims were unrealistically aggressive, but it is clear that the Internet is growing quickly as more and more devices are connected, consuming content from a growing universe of websites, applications, and services.

To determine the traffic trends over time, we first established a baseline, calculated as the average daily traffic volume (excluding bot traffic) over the second full calendar week (January 8-14) of 2023. We chose the second calendar week to allow time for people to get back into their “normal” routines (school, work, etc.) after the winter holidays and New Year’s Day. The percent change shown in our traffic trends chart is calculated relative to the baseline value, and represents a seven-day trailing average — it does not represent absolute traffic volume for a country/region. The seven-day averaging is done to smooth the sharp changes seen with a daily granularity. A trend line for 2022 is shown for comparison purposes.

Our data shows that globally, Internet traffic grew 25% in 2023, with nominal initial growth accelerating during the second half of the year. Overall, the pattern is similar to that observed in 2022 (excepting last year’s late February spike), and peak growth for the year is just slightly above the peak growth level seen in 2022. Traffic patterns in Canada were also rather consistent year-over-year, exhibiting similar seasonality, and peak growth above 30% in both 2022 and 2023. In many countries, the 2022 trend line shows a clear drop in traffic heading into the Christmas holiday, with a slight rebound ahead of New Year’s Day. It will be interesting to see if traffic follows this pattern in 2023 as well.

Global Internet traffic growth in 2023, compared with 2022
Internet traffic growth in Canada in 2023, compared with 2022 

Comparisons with 2022 traffic trends helps make the impact of major holidays on Internet traffic more visible. For example, in Muslim countries including Indonesia, Turkey, and the United Arab Emirates, the celebration of Eid-Ul-Fitr, the festival marking the end of the fast of Ramadan, is visible as a noticeable drop in traffic around April 21-23, 2023, just before a similar drop visible in the 2022 trend line during last year’s celebration on May 2-3. In Italy, a drop in traffic is clearly visible around Pasqua di Resurrezione and Lunedì dell’Angelo (Easter Sunday and Monday) on April 9-10, one week ahead of a similar drop in traffic in 2022

Internet traffic growth in Indonesia in 2023, compared with 2022 

In addition, extended disruptions to Internet connectivity are also clearly visible within the traffic trend charts. Examples include Mauritania, where government-directed shutdowns occurred from March 6-12 and May 30 – June 6, and Gabon, where a shutdown was in place from August 26-30, as well as Guam, where Super Typhoon Mawar caused a multi-week drop in traffic starting on May 24.

Internet traffic growth in Mauritania in 2023, compared with 2022 
Internet traffic growth in Guam in 2023, compared with 2022 

Google was again the most popular general Internet service, with 2021 leader TikTok falling to fourth place. OpenAI was the most popular service in the emerging Generative AI category, and Binance remained the most popular Cryptocurrency service.

One of the most popular sections of the Year In Review over the last several years has been the exploration of the most popular Internet services, both generally and across a number of categories. These rankings of service popularity are based on analysis of anonymized query data of traffic to our 1.1.1.1 public DNS resolver from millions of users around the world. Although DNS resolution operates at a domain level, domains that belong to a single Internet service are grouped together for the purposes of these rankings.

In the overall category, Google once again held the top spot, owing in part to its broad portfolio of services as well as the popularity of the Android mobile operating system. In addition to perennial categories like e-commerce, video streaming, and messaging, this year we also looked at Generative AI, which has been on a meteoric rise in 2023. In this category, OpenAI held the top spot, building on the success and popularity of ChatGPT, which it launched only a year ago. And despite the turmoil seen in the cryptocurrency space this year, Binance remained the most popular service in that category.

We explore these categorical rankings, as well as trends seen by specific services, in more detail in a separate blog post.

Globally, over two-thirds of mobile device traffic was from Android devices. Android had a >90% share of mobile device traffic in over 25 countries/regions; peak iOS mobile device traffic share was 66%.

Apple’s iOS and Google’s Android are the two leading operating systems used on mobile devices, and analysis of information in the user agent reported with each request allows us to gain insight into the distribution of traffic by client operating system throughout the year. Given the wide range of both devices and price points for Android devices, it is not surprising that Android is responsible for the majority of mobile device traffic when aggregated globally.

Globally, over two-thirds of mobile device traffic was from Android devices. The split is in line with Android/iOS usage observed in 2022. When looking at the countries/regions with the highest levels of Android usage, we find Bangladesh and Papua New Guinea at the top of the list, both with over 95% of mobile device traffic coming from Android devices. Looking more closely at other countries that see particularly high levels of Android usage, it is interesting to note that they are largely in Africa, Oceania/Asia, and South America, and that many have lower levels of gross national income per capita. This is presumably where the availability of lower priced “budget” phones plays to Android’s advantage from an adoption perspective.

In contrast, while the share of mobile device traffic from iOS at a country/region level never tops 70%, many of the countries with an iOS share over 50%, including Denmark, Australia, Japan, and Canada, have comparatively higher gross national income per capita, which likely speaks to a greater ability to afford higher priced devices.

Mobile device traffic operating system distribution across selected countries

SpaceX’s Starlink high-speed satellite Internet service has continued to rapidly grow its footprint since launching in 2019, making high performance Internet connections available in many countries/regions that were previously unserved or underserved by traditional wired or wireless broadband. The current leader in the space, in the future it will be joined by Amazon’s Project Kuiper service, which launched its first two test satellites this year, as well as Eutelsat OneWeb, which grew its satellite constellation in 2023 as well.

To track the growth in usage and availability of Starlink’s service, we analyzed aggregate Cloudflare traffic volumes associated with the service’s autonomous system (AS14593) throughout 2023. Although Starlink is not yet available globally, we did see traffic growth across a number of countries/regions. The request volume shown on the trend line in the chart represents a seven-day trailing average. A trend line for 2022 is shown for comparison purposes, and is scaled to the maximum value across 2022 and 2023.

Globally, we saw Starlink traffic more than triple this year. In the United States, traffic from Starlink was up over 2.5x, and grew over 17x in Brazil. In countries where Starlink turned up service in 2023, including Kenya, the Philippines, and Zambia, we saw traffic grow rapidly once the service became available.

Starlink traffic growth in Brazil, compared with 2022

Google Analytics, React, and HubSpot were among the most popular technologies found on top websites.

Modern websites are complex productions, relying on a mix of frameworks, platforms, services, and tools, and the developer community is responsible for making them coexist with one another to deliver a seamless experience. Using the Cloudflare Radar URL Scanner, which we launched in March 2023, we scanned websites associated with the top 5000 domains to identify the most popular technologies and services used across a dozen different categories, including (but not limited to) Analytics, where Google Analytics was by far the most widely used; JavaScript Frameworks, where React had a commanding lead; and Marketing Automation providers, where leader HubSpot was closely followed by several competitors.

Top website technologies, JavaScript frameworks category

Globally, nearly half of web requests used HTTP/2, with 20% using HTTP/3.

HTTP (HyperText Transfer Protocol) is the core protocol that the web relies upon. HTTP/1.0 was first standardized in 1996, HTTP/1.1 in 1999, and HTTP/2 in 2015. The most recent version, HTTP/3, was completed in 2022, and runs on top of QUIC, a new transport protocol. On the client side, HTTP/3 support is enabled by default in the latest versions of desktop and mobile Google Chrome and Mozilla Firefox, and for a portion of Apple Safari users. HTTP/3 is available for free for all Cloudflare customers, though not every customer chooses to enable it.

Using QUIC allows HTTP/3 to deliver improved performance by mitigating the effects of packet loss and network changes, as well as establishing connections more quickly. It also provides encryption by default, mitigating the risk of attacks. Websites and applications that remain on older versions of HTTP miss out on these benefits.

Analysis of the HTTP version negotiated for each request allows us to gain insight into the distribution of traffic by the various versions of the protocol aggregated throughout the year. (“HTTP/1.x” aggregates requests made over HTTP/1.0 and HTTP/1.1.) At a global level, 20% of requests were made over the latest version, HTTP/3. Another third of requests were made over the comparatively ancient HTTP/1.x versions, while HTTP/2 remained dominant, and accounted for the 47% balance.

Global HTTP version traffic distribution

Looking at the version distribution geographically, we found a number of Asian countries, including Nepal, Thailand, Malaysia, and Sri Lanka among those with highest rates of HTTP/3 usage, although these rates did not exceed 35%. In contrast, more than half of the requests from ten countries, including Ireland, Albania, Finland, and China, were made over HTTP/1.x during 2023.

NodeJS was the most popular language used for making automated API requests.

In addition, as developers increasingly use automated API calls to power dynamic websites and applications, we can use our unique visibility into Web traffic to identify the top languages these API clients are written in. Looking at API-related requests determined to not be coming from a person using a browser or native mobile application, we applied heuristics to help identify the language used to build the client.

Our analysis found that almost 15% of automated API requests are made by NodeJS clients, with Go, Java, Python, and .NET holding smaller shares.

Top languages used to make automated API calls

Googlebot was responsible for the highest volume of request traffic to Cloudflare in 2023.

Cloudflare Radar enables users to see Internet traffic trends at a country/region or network level over a selected period of time. However, we wanted to zoom out a bit, and look at the traffic Cloudflare saw from the entire IPv4 Internet over the course of the entire year. Hilbert curves, as “continuous space-filling curves”, have properties that are useful for visualizing the Internet’s IPv4 address space.

Using a Hilbert curve visualization, we can visualize aggregated request traffic (over IPv4) to Cloudflare from January 1st through November 26th, 2023. In order to make the amount of data used for the visualization manageable, IP addresses are aggregated at a /20 level, meaning that at the highest zoom level, each cell represents traffic from 4096 IPv4 addresses. (The sheer size of the IPv6 address space would make associated traffic very hard to see in such a visualization, especially as such a small amount has been allocated for assignment by the Regional Internet Registries.)

Within the visualization, IP addresses are grouped by ownership, and for much of the IP address space shown there, a mouseover at the default zoom level will show the Regional Internet Registry (RIR) that the address block belongs to. However, there are also a number of blocks that were assigned prior to the existence of the RIR system, and for these, they are labeled with the name of the organization that owns them. Progressive zooming ultimately shows the autonomous system and country/region that the IP address block is associated with, as well as its share of traffic relative to the maximum. (If a country/region is selected, only the IP address blocks associated with that location are visible.) Overall traffic shares are indicated by shading based on a color scale, and although a number of large unshaded blocks are visible, this does not necessarily mean that the associated address space is unused, but rather that it may be used in a way that does not generate traffic to Cloudflare.

Hilbert curve showing aggregated 2023 traffic to Cloudflare across the IPv4 Internet

Areas of higher request volume, indicated by warmer orange/red shading, are visibly scattered throughout the plot, but the IP address block that had the maximum request volume to Cloudflare during 2023 was 66.249.64.0/20, which belongs to Google. This IP address block is one of several used by the Googlebot web crawler, which is a likely explanation for the high request volume, given the number of web properties on Cloudflare’s network.

Zoomed view of the Hilbert curve showing the IPv4 address block that generated the highest volume of requests

It is hard to do this visualization justice with a short summary and static screenshot. To explore it in more detail, we encourage you to go to the Year in Review website and explore it by dragging and zooming to move around the IPv4 Internet.

Connectivity & Speed

Over 180 Internet outages were observed around the world in 2023, with many due to government-directed regional and national shutdowns of Internet connectivity.

During 2023, we have written frequently about Internet outages, whether due to technical issues, government-directed shutdowns, or geopolitical conflict, as well as infrastructure resilience issues (including fiber cuts, power outages, and severe weather) highlighted in our quarterly summaries. The impacts of these outages can be significant, including significant economic losses and severely limited communications. The Cloudflare Radar Outage Center tracks these Internet outages, and uses Cloudflare traffic data for insights into their scope and duration.

Some of these outages seen through the year were short-lived, lasting just a couple of hours, while others have stretched on for multiple months. In the latter category, localized government-directed shutdowns in Manipur, India and Amhara, Ethiopia have lasted over seven and four months respectively (as of early December). In the former category, Iraq frequently experienced multi-hour nationwide Internet shutdowns intended to prevent cheating on academic exams — these contribute to the clustering visible in the timeline during June, July, and August.

Within the timeline on the Year in Review website, mousing over a dot will display metadata about that outage, and clicking on it will open a page with additional information. If a country/region is selected, only outages for that country will be displayed.

Internet outages were observed around the world during 2023

Aggregated across 2023, only a third of IPv6-capable requests worldwide were made over IPv6. In India, however, that share reached 70%.

IPv6 has been around in some fashion since 1998, with an expanded address space that better supports the universe of Internet-connected devices that has grown exponentially over the last quarter-century. And over that time, available IPv4 space has been exhausted, leading connectivity providers to resort to solutions like Network Address Translation, and cloud and hosting providers to acquire blocks of IPv4 address space for as much as $50 per address. IPv6 also brings a number of other benefits to network providers, and if implemented correctly, adoption should be transparent from an end user perspective.

Cloudflare has been a vocal and active advocate for IPv6 stretching all the way back to our first birthday in 2011, when we announced our Automatic IPv6 Gateway, which enabled free IPv6 support for all of our customers. Just a few years later, we enabled IPv6 support by default for all of our customers. (Although it is enabled by default, not all customers choose to keep it enabled for a variety of reasons.) However, this support is only half of the equation for driving IPv6 adoption, as end user connections need to support it as well. (Technically, it is a bit more complex than that, but those are the two foundational requirements.) Analysis of the IP version used for each request made to Cloudflare allows us to gain insight into the distribution of traffic by the various versions of the protocol, aggregated throughout the year.

Thanks to near-complete IPv6 adoption by Indian telecommunications provider Reliance Jio, 70% of dual-stacked requests from Indian users were made via IPv6. India was followed closely by Malaysia, where 66% of dual-stacked requests were made over IPv6 during 2023, thanks to strong IPv6 adoption rates across leading Internet providers within the country. Other countries that saw more than half of dual-stacked requests, on average, made over IPv6 include Saudi Arabia, Vietnam, Greece, France, Uruguay, and Thailand. In contrast, there were on the order of 40 countries/regions where less than 1% of dual-stacked requests were made over IPv6 during 2023. Lagging adoption across such a large cohort of countries/regions 25 years after IPv6 was first published as a draft standard is quite surprising.

IPv4/IPv6 traffic distribution in India

The top 10 countries all had measured average download speeds above 200 Mbps, with Iceland showing the best results across all four measured Internet quality metrics.

Even when they are not facing Internet outages, users around the world are often contending with poor performance on their Internet connections, whether due to low speeds, high latency, or a combination of these factors. Although Internet providers continue to evolve their service portfolios to offer increased connection speeds and reduced latency in order to support growth in use cases like online gaming and videoconferencing, consumer adoption is often mixed due to cost, availability, or other issues. By aggregating the results of speed.cloudflare.com tests taken during 2023, we can get a geographic perspective on connection quality metrics including average download and upload speeds, and average idle and loaded latencies, as well as the distribution of the measurements.

In Iceland, over 85% of all Internet connections are over fiber, and this is reflected in its ranking as the country with the best overall Internet quality metrics, as speed test results show that providers there deliver the highest average speeds (282.5 Mbps download, 179.9 Mbps upload) and lowest average latencies (9.6 ms idle, 77.1 ms loaded). The histogram below shows that while there is a large cluster of download speeds between 0–100 Mbps, there were also a significant number of tests that measured even higher speeds, including some in excess of 1 Gbps.

Western European countries including Spain, Portugal, and Denmark also ranked among the top 10 across multiple Internet quality metrics.

Download and upload speed test result distribution in Iceland

Over 40% of global traffic comes from mobile devices. In more than 80 countries/regions, the majority of traffic comes from mobile devices.

Over the last 15 years or so, mobile devices have become increasingly ubiquitous, becoming indispensable in both our personal and professional lives, thanks in large part to their ability to enable us to access the Internet from nearly anywhere at any time. In some countries/regions, mobile devices primarily connect to the Internet via Wi-Fi, while others are “mobile first”, where Internet access is primarily through 4G/5G services.

Analysis of information contained with the user agent reported with each request to Cloudflare enables us to categorize it as coming from a mobile, desktop, or other type of device. Aggregating this categorization throughout the year at a global level, we found that 42% of traffic came from mobile devices, with 58% coming from desktop devices such as laptops and “classic” PCs. These traffic shares were in line with those measured in 2022. 79% of traffic came from mobile devices in Zambia, making it the country with the largest mobile device traffic share in 2023. Other countries/regions that had more than 50% of traffic come from mobile devices were concentrated in the Middle East/Africa, the Asia Pacific Region, and South/Central America. In contrast, Finland had one of the highest shares of desktop device traffic, at 80%.

Desktop and mobile device traffic distribution across selected countries

Security

Security

Just under 6% of global traffic was mitigated by Cloudflare’s systems as being potentially malicious or for customer-defined reasons. In the United States, 3.65% of traffic was mitigated, while in South Korea, it was 8.36%.

Malicious bots are often used to attack websites and applications. To protect customers from these threats, Cloudflare mitigates (blocks) this attack traffic using DDoS mitigation techniques or Web Application Firewall (WAF) Managed Rules. However, customers may also choose to have Cloudflare mitigate traffic using other techniques for a variety of other reasons, such as rate-limiting requests, or blocking all traffic from a given location, even if it isn’t malicious. Analyzing traffic to Cloudflare’s network seen throughout 2023, we looked at the overall share that was mitigated (for any reason), as well as the share that was mitigated as a DDoS attack or by WAF Managed Rules.

Overall, just under 6% of global traffic was mitigated by Cloudflare’s systems as being potentially malicious or for customer-defined reasons, while only around 2% of it saw DDoS/Managed WAF mitigations. Some countries, such as Bermuda, saw the percentages for the two metrics track very closely, while other countries, like Pakistan and South Africa showed much larger gaps between their trend lines.

Mitigated traffic trends in Pakistan

A third of global bot traffic comes from the United States, and over 11% of global bot traffic comes from Amazon Web Services.

Bot traffic describes any non-human Internet traffic, and monitoring bot traffic levels can help site and application owners spot potentially malicious activity. Of course, bots can be helpful too, and Cloudflare maintains a list of verified bots to help keep the Internet healthy. Verified bots include those used for things like search engine indexing, performance testing, and availability monitoring. Regardless of intent, we wanted to look at where bot traffic was coming from, and we can use the IP address of a request to identify the network (autonomous system) and country/region associated with the bot making the request. Perhaps unsurprisingly, we found that cloud platforms were among the leading sources of bot traffic. This is likely due to the ease of automating the provisioning/teardown of compute resources and the relatively low cost of doing so, the distributed geographic footprint of cloud platforms, and the availability of high-bandwidth connections.

Globally, nearly 12% of bot traffic comes from Amazon Web Services, and over 7% from Google. Some of it comes from consumer ISPs as well, with U.S. broadband provider Comcast originating over 1.5% of global bot traffic. A disproportionate amount of bot traffic originates from the United States, responsible for nearly a third of global bot traffic, four times that of Germany, which originates just 8%. Within the United States, Amazon’s total share of bot traffic just edges out Google’s.

Global bot traffic distribution by source network
Global bot traffic distribution by source country

Globally, Finance was the most attacked industry, but the timing of spikes in mitigated traffic and the target industries varied widely throughout the year and around the world.

The industries targeted by attacks often shift over time, depending on the intent of the attackers. They may be trying to cause financial harm by attacking ecommerce sites during a busy shopping period, or they may be trying to make a political statement by attacking government-related sites. To identify industry-targeted attack activity during 2023, we analyzed mitigated traffic for customers that had an associated industry and vertical within their customer record. Mitigated traffic was aggregated weekly by source country/region across 18 target industries.

At a global level, Finance organizations were the most attacked over the course of the year, though we saw a significant amount of volatility from week-to-week. Interestingly, some clustering was evident, as Finance, which includes organizations that provide websites and applications for mobile payments, investments/trading, and cryptocurrency, was also a top target for a number of European countries, including Austria, Switzerland, France, the United Kingdom, Ireland, Italy, and the Netherlands, as well as in North America, for Canada, the United States, and Mexico. The Health industry, which includes companies that make exercise equipment, as well medical testing device manufacturers, was a top target across multiple African countries, including Benin, Côte d’Ivoire, Cameroon, Ethiopia, Senegal, and Somalia.

Overall, however, the year started slowly, with no industry seeing more than 8% of traffic being mitigated. As the first quarter progressed, Professional Services and News/Media/Publications organizations saw spikes in the share of mitigated traffic later in January, with Health jumping in mid-February and Law & Government organizations seeing a sharp increase in mitigated traffic in early March. Customers in the Arts/Entertainment/Recreation industry classification were apparently targeted by a multi-week attack campaign, with more than 20% of traffic mitigated during the weeks of March 26, April 2, and April 9. The overall peak during the year was experienced by the Professional Services industry, which saw a mitigated traffic share of 38.4% for the week of August 6, nearly twice its January spike. The timing of spikes and the industries experiencing those spikes varied widely across countries/regions.

Global mitigated traffic share by industry, week of August 6, 2023

Even as an older vulnerability, Log4j remained a top target for attacks during 2023. However, HTTP/2 Rapid Reset emerged as a significant new vulnerability, beginning with a flurry of record-breaking attacks.

In August 2023, we published a blog post that explored traffic seen by Cloudflare for the most commonly exploited vulnerabilities of 2022, as listed in a joint Cybersecurity Advisory. These included vulnerabilities in the Log4j Java-based logging utility, Microsoft Exchange, Atlassian’s Confluence platform, VMWare, and F5’s BIG-IP traffic management system. Although these are older vulnerabilities, attackers continued to actively target and exploit them throughout 2023, in part because organizations are frequently slow to follow the recommendations outlined in the Cybersecurity Advisory. We updated the analysis done for our blog post to include just the attack activity seen in 2023.

Attack activity by vulnerability varied by location, and in some, attacks targeted only a subset of the vulnerabilities. Aggregated worldwide, attack volume targeting Log4j consistently dwarfed that seen for the other vulnerabilities, and saw spikes during the last week of October and mid-late November; attack activity targeting Atlassian vulnerabilities increased in late July and trended slowly higher through the rest of the year. At a country/region level, Log4j was generally the most targeted vulnerability. In countries including France, Germany, India, and the United States, associated attack volume remained at a significant level throughout the year, while in other countries/regions, these attacks are most visible as infrequent, short-lived spikes within a country/region’s graphs, punctuating otherwise low levels of attack volume.

Global attack activity trends for commonly exploited vulnerabilities

We also expect that through 2024, attackers will continue to target the HTTP/2 Rapid Reset vulnerability disclosed in October. The vulnerability (see CVE-2023-44487 for details) abuses an underlying weakness in the request cancellation feature of the HTTP/2 protocol, leading to resource exhaustion on the target web/proxy server. Between the end of August and the beginning of October, we saw a number of attacks targeting this vulnerability. Across this set of attacks, the average attack rate was 30M requests per second (rps), with nearly 90 peaking above 100M rps, and the largest one hitting 201M rps. This largest attack was nearly 3x bigger than our previous biggest attack on record.

One notable concern about this vulnerability is that the attacker was able to generate such a large attack with a botnet consisting of just 20,000 compromised systems. This is much smaller than some of the largest botnets today, which comprise hundreds of thousands or millions of hosts. With average web traffic estimated to be between 1–3 billion requests per second, attacks using this method could conceivably focus an entire web’s worth of requests on a few unsuspecting targets.

HTTP/2 Rapid Reset campaign of hyper-volumetric DDoS attacks

1.7% of TLS 1.3 traffic is using post-quantum encryption

Post-quantum refers to a new set of cryptographic techniques that can protect data from adversaries with the ability to capture and store today’s data for decryption by sufficiently powerful quantum computers in the future. The Cloudflare Research team has been exploring post-quantum cryptography since 2017.

In October 2022, we enabled post-quantum key agreement at our edge by default, but use of it requires that the browser support it as well. Google’s Chrome browser started to slowly enable support in August 2023, and we expect its support will continue to grow in 2024, and that other browsers will add support over time as well. In September 2023, we announced general availability of post-quantum cryptography for both inbound and outbound connections and for many internal services, and expect to finish upgrading all internal services by the end of 2024.

After first enabling support in August, Chrome began ramping the number of browsers (version 116 and later) that use post-quantum cryptography, resulting in gradual growth leading to the significant increase seen on November 8. These actions helped push the share of TLS 1.3 traffic using post-quantum encryption to 1.7% at the end of November. As this ramp continues with future Chrome updates, and as other browsers add support for post-quantum encryption, we expect this share to continue to grow rapidly in 2024.

Growth trends in post-quantum encrypted TLS 1.3 traffic

Deceptive links and extortion attempts were two of the most common types of threats found in malicious email messages.

As the #1 business application, email represents a very attractive entry point into enterprise networks for attackers. Targeted malicious emails may attempt to impersonate an otherwise legitimate sender, try to get the user to click on a deceptive link, or contain a dangerous attachment, among other types of threats. Cloudflare Area 1 Email Security protects customers from email-based attacks, including those carried out through targeted malicious email messages. Over the course of 2023, an average of 2.65% of emails analyzed by Cloudflare Area 1 were found to be malicious. Aggregated at a weekly level, spikes to over 3.5%, 4.5%, and over 5% were seen in early February, early September, and late October respectively.

Global malicious email share trends

When carrying out attacks using malicious email messages, attackers use a variety of techniques, which we refer to as threat categories. These categories are defined and explored in detail in Cloudflare’s 2023 phishing threats report. Analysis of malicious emails shows that messages may contain multiple types of threats, highlighting the need for a comprehensive email security solution. Exploring threat activity trends for these categories, aggregated weekly across the year, we found that as much as 80% of them contained deceptive links.

However, it appears that attackers may have started to shift strategies in August, as the percentage of emails containing deceptive links began to fall while the share proposing to extort the recipient began to increase. By the end of October, and into November, the two threat categories had traded places, with nearly 80% of analyzed malicious emails containing an extortion threat, while only 20% contained deceptive links, as seen towards the right side of the graph below. However, this extortion campaign may have been short-lived, as its percentage fell almost as quickly as it rose. Identity deception and credential harvesting were also commonly identified threats, though the share of emails they were found in gradually declined over the course of the year.

Global threat category trends for malicious emails

Routing security, measured as the share of RPKI valid routes, improved globally during 2023. Significant growth was observed in countries including Saudi Arabia, the United Arab Emirates, and Vietnam.

Border Gateway Protocol (BGP) is the routing protocol for the Internet, communicating routes between networks, enabling traffic to flow between source and destination. However, because it relies on trust between networks, incorrect information shared between peers, whether done so intentionally or not, can send traffic to the wrong place, potentially with malicious results. Resource Public Key Infrastructure (RPKI) is a cryptographic method of signing records that associate a BGP route announcement with the correct originating AS number. In simple terms, it provides a way of ensuring that the information being shared originally came from a network that is allowed to do so. (Note that this is only half of the challenge of implementing routing security, as network providers also need to validate these signatures and filter out invalid announcements.) In the United States, the federal government recognizes the importance of routing security, with the Federal Communications Commission holding a “Border Gateway Protocol Security Workshop” on July 31.

Cloudflare has been a strong proponent of routing security, from being a founding participant in the MANRS CDN and Cloud Programme, to releasing an RPKI toolkit for network operators, to providing a public tool that enables users to test whether their Internet provider has implemented BGP safely, to presenting at this summer’s FCC workshop.

Building on the July release of the new Routing page on Cloudflare Radar, we analyzed data from RIPE NCC’s RPKI daily archive to determine the share of RPKI valid routes (as opposed to those route announcements that are invalid or whose status is unknown)  and how that share has changed over the course of 2023. Since the start of the year, the global share of RPKI valid routes grew to nearly 45%, up six percentage points from the end of 2022. At a country/region level, we are looking at routes announced by autonomous systems associated with the given country/region. In the United States, the increased FCC attention on routing security is arguably warranted, as less than a third of the routes are RPKI valid. Although this is significantly better than South Korea, where less than 1% of announced routes are RPKI valid, it trails Vietnam significantly, where the share increased 35 percentage points during the first half of the year to 90%.

RPKI valid route growth in Vietnam

Conclusion

In the Cloudflare Radar 2023 Year In Review, we have attempted to provide a snapshot of the Internet, as dynamic as it is, through trend graphs and summary statistics, providing unique perspectives on Internet traffic, Internet quality, and Internet security, and how key metrics across these areas vary around the world.

As we said in the introduction, we strongly encourage you to visit the Cloudflare Radar 2023 Year In Review website and explore the trends relevant to metrics, countries/regions, and industries of interest, and to consider how they impact your organization so that you are appropriately prepared for 2024.

If you have any questions, you can contact the Cloudflare Radar team at [email protected] or on social media at @CloudflareRadar (X/Twitter), cloudflare.social/@radar (Mastodon), and radar.cloudflare.com (Bluesky).

Acknowledgements

As we noted last year, it truly is a team effort to produce the data, website, and content for our annual Year in Review, and I’d like to acknowledge those team members that contributed to this year’s effort. Thank you to: Sabina Zejnilovic, Jorge Pacheco, Carlos Azevedo (Data Science); Arun Chintalapati, Reza Mohammady (Design); Vasco Asturiano, Nuno Pereira, Tiago Dias (Front End Development); João Tomé (Most popular Internet services); and Davide Marquês, Paula Tavares, Celso Martinho (Project/Engineering Management) as well as countless other colleagues for their answers, edits, and ideas.

The collective thoughts of the interwebz