Detect Stripe keys in S3 buckets with Amazon Macie

Post Syndicated from Koulick Ghosh original

Many customers building applications on Amazon Web Services (AWS) use Stripe global payment services to help get their product out faster and grow revenue, especially in the internet economy. It’s critical for customers to securely and properly handle the credentials used to authenticate with Stripe services. Much like your AWS API keys, which enable access to your AWS resources, Stripe API keys grant access to the Stripe account, which allows for the movement of real money. Therefore, you must keep Stripe’s API keys secret and well-controlled. And, much like AWS keys, it’s important to invalidate and re-issue Stripe API keys that have been inadvertently committed to GitHub, emitted in logs, or uploaded to Amazon Simple Storage Service (Amazon S3).

Customers have asked us for ways to reduce the risk of unintentionally exposing Stripe API keys, especially when code files and repositories are stored in Amazon S3. To help meet this need, we collaborated with Stripe to develop a new managed data identifier that you can use to help discover and protect Stripe API keys.

“I’m really glad we could collaborate with AWS to introduce a new managed data identifier in Amazon Macie. Mutual customers of AWS and Stripe can now scan S3 buckets to detect exposed Stripe API keys.”
Martin Pool, Staff Engineer in Cloud Security at Stripe

In this post, we will show you how to use the new managed data identifier in Amazon Macie to discover and protect copies of your Stripe API keys.

About Stripe API keys

Stripe provides payment processing software and services for businesses. Using Stripe’s technology, businesses can accept online payments from customers around the globe.

Stripe authenticates API requests by using API keys, which are included in the request. Stripe takes various measures to help customers keep their secret keys safe and secure. Stripe users can generate test-mode keys, which can only access simulated test data, and which doesn’t move real money. Stripe encourages its customers to use only test API keys for testing and development purposes to reduce the risk of inadvertent disclosure of live keys or of accidentally generating real charges.

Stripe also supports publishable keys, which you can make publicly accessible in your web or mobile app’s client-side code to collect payment information.

In this blog post, we focus on live-mode keys, which are the primary security concern because they can access your real data and cause money movement. These keys should be closely held within the production services that need to use them. Stripe allows keys to be restricted to read or write specific API resources, or used only from certain IP ranges, but even with these restrictions, you should still handle live mode keys with caution.

Stripe keys have distinctive prefixes to help you detect them such as sk_live_ for secret keys, and rk_live_ for restricted keys (which are also secret).

Amazon Macie

Amazon Macie is a fully managed service that uses machine learning (ML) and pattern matching to discover and help protect your sensitive data, such as personally identifiable information. Macie can also provide detailed visibility into your data and help you align with compliance requirements by identifying data that needs to be protected under various regulations, such as the General Data Protection Regulation (GDPR) and the Health Insurance Portability and Accountability Act (HIPAA).

Macie supports a suite of managed data identifiers to make it simpler for you to configure and adopt. Managed data identifiers are prebuilt, customizable patterns that help automatically identify sensitive data, such as credit card numbers, social security numbers, and email addresses.

Now, Macie has a new managed data identifier STRIPE_CREDENTIALS that you can use to identify Stripe API secret keys.

Configure Amazon Macie to detect Stripe credentials

In this section, we show you how to use the managed data identifier STRIPE_CREDENTIALS to detect Stripe API secret keys. We recommend that you carry out these tutorial steps in an AWS account dedicated to experimentation and exploration before you move forward with detection in a production environment.


To follow along with this walkthrough, complete the following prerequisites.

Create example data

The first step is to create some example objects in an S3 bucket in the AWS account. The objects contain strings that resemble Stripe secret keys. You will use the example data later to demonstrate how Macie can detect Stripe secret keys.

To create the example data

  1. Open the S3 console and create an S3 bucket.
  2. Create four files locally, paste the following mock sensitive data into those files, and upload them to the bucket.
     stripe publishable key sk_live_cpegcLxKILlrXYNIuqYhGXoy
     stripe payment sk_live_ijklcLxKILlrXYNIuqYhGXoy
     stripe api key sk_live_abcdcLxKILlrXYNIuqYhGXoy
     stripe secret key sk_live_cpegcLxKILlrXYNIuqYhGXoy

Note: The keys mentioned in the preceding files are mock data and aren’t related to actual live Stripe keys.

Create a Macie job with the STRIPE_CREDENTIALS managed data identifier

Using Macie, you can scan your S3 buckets for sensitive data and security risks. In this step, you run a one-time Macie job to scan an S3 bucket and review the findings.

To create a Macie job with STRIPE_CREDENTIALS

  1. Open the Amazon Macie console, and in the left navigation pane, choose Jobs. On the top right, choose Create job.
    Figure 1: Create Macie Job

    Figure 1: Create Macie Job

  2. Select the bucket that you want Macie to scan or specify bucket criteria, and then choose Next.
    Figure 2: Select S3 bucket

    Figure 2: Select S3 bucket

  3. Review the details of the S3 bucket, such as estimated cost, and then choose Next.
    Figure 3: Review S3 bucket

    Figure 3: Review S3 bucket

  4. On the Refine the scope page, choose One-time job, and then choose Next.

    Note: After you successfully test, you can schedule the job to scan S3 buckets at the frequency that you choose.

    Figure 4: Select one-time job

    Figure 4: Select one-time job

  5. For Managed data identifier options, select Custom and then select Use specific managed data identifiers. For Select managed data identifiers, search for STRIPE_CREDENTIALS and then select it. Choose Next.
    Figure 5: Select managed data identifier

    Figure 5: Select managed data identifier

  6. Enter a name and an optional description for the job, and then choose Next.
    Figure 6: Enter job name

    Figure 6: Enter job name

  7. Review the job details and choose Submit. Macie will create and start the job immediately, and the job will run one time.
  8. When the Status of the job shows Complete, select the job, and from the Show results dropdown, select Show findings.
    Figure 7: Select the job and then select Show findings

    Figure 7: Select the job and then select Show findings

  9. You can now review the findings for sensitive data in your S3 bucket. As shown in Figure 8, Macie detected Stripe keys in each of the four files, and categorized the findings as High severity. You can review and manage the findings in the Macie console, retrieve them through the Macie API for further analysis, send them to Amazon EventBridge for automated processing, or publish them to AWS Security Hub for a comprehensive view of your security state.
    Figure 8: Review the findings

    Figure 8: Review the findings

Respond to unintended disclosure of Stripe API keys

If you discover Stripe live-mode keys (or other sensitive data) in an S3 bucket, then through the Stripe dashboard, you can roll your API keys to revoke access to the compromised key and generate a new one. This helps ensure that the key can’t be used to make malicious API requests. Make sure that you install the replacement key into the production services that need it. In the longer term, you can take steps to understand the path by which the key was disclosed and help prevent a recurrence.


In this post, you learned about the importance of safeguarding Stripe API keys on AWS. By using Amazon Macie with managed data identifiers, setting up regular reviews and restricted access to S3 buckets, training developers in security best practices, and monitoring logs and repositories, you can help mitigate the risk of key exposure and potential security breaches. By adhering to these practices, you can help ensure a robust security posture for your sensitive data on AWS.

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, start a new thread on Amazon Macie re:Post.

Koulick Ghosh

Koulick Ghosh

Koulick is a Senior Product Manager in AWS Security based in Seattle, WA. He loves speaking with customers about how AWS Security services can help improve their security. In his free time, he enjoys playing the guitar, reading, and exploring the Pacific Northwest.

Sagar Gandha

Sagar Gandha

Sagar is an experienced Senior Technical Account Manager at AWS adept at assisting large customers in enterprise support. He offers expert guidance on best practices, facilitates access to subject matter experts, and delivers actionable insights on optimizing AWS spend, workloads, and events. Outside of work, Sagar loves spending time with his kids.

Mohan Musti

Mohan Musti

Mohan is a Senior Technical Account Manager at AWS based in Dallas. Mohan helps customers architect and optimize applications on AWS. In his spare time, he enjoys spending time with his family and camping.

AWS Weekly Roundup — AWS Control Tower new API, TLS 1.3 with API Gateway, Private Marketplace Catalogs, and more — February 19, 2024

Post Syndicated from Irshad Buchh original

Over the past week, our service teams have continued to innovate on your behalf, and a lot has happened in the Amazon Web Services (AWS) universe that I want to tell you about. I’ll also share about all the AWS Community events and initiatives that are happening around the world.

Let’s dive in!

Last week’s launches
Here are some launches that got my attention during the previous week.

AWS Control Tower introduces APIs to register organizational units – With these new APIs, you can extend governance to organizational units (OUs) using APIs and automate your OU provisioning workflow. The APIs can also be used for OUs that are already under AWS Control Tower governance to re-register OUs after landing zone updates. These APIs include AWS CloudFormation support, allowing customers to manage their OUs with infrastructure as code (IaC).

API Gateway now supports TLS 1.3 – By using TLS 1.3 with API Gateway as the centralized point of control, developers can secure communication between the client and the gateway; uphold the confidentiality, integrity, and authenticity of their API traffic; and benefit from API Gateway’s integration with AWS Certificate Manager (ACM) for centralized deployment of SSL certificates using TLS.

Amazon OpenSearch Service now lets you update cluster volume without blue/green – While blue/green deployments are meant to avoid any disruption to your clusters because the deployment uses additional resources on the domain, it is recommended that you perform them during low traffic periods. Now, you can update volume-related cluster configuration without requiring a blue/green deployment, ensuring minimal performance impact on your online traffic and avoiding any potential disruption to your cluster operations.

Amazon GuardDuty Runtime Monitoring protects clusters running in shared VPC – With this launch, customers who are already opted into automated agent management in GuardDuty will benefit from a renewed 30-day trial of GuardDuty Runtime Monitoring, where we will automatically start monitoring the resources (clusters) deployed in a shared VPC setup. Customers also have the option to manually manage the agent and provision the virtual private cloud (VPC) endpoint in their shared VPC environment.

AWS Marketplace now supports managing Private Marketplace catalogs for OUs – This capability supports distinct product catalogs per business unit or development environment, empowering organizations to align software procurement with specific needs. Additionally, customers can designate a trusted member account as a delegated administrator for Private Marketplace administration, reducing the operational burden on management account administrators. With this launch, organizations can procure more quickly by providing administrators with the agile controls they need to scale their procurement governance across distinct business and user needs.

For a full list of AWS announcements, be sure to keep an eye on the What’s New at AWS page.

Other AWS news

Join AWS Cloud Clubs Captains – The C3 cohort of AWS Cloud Club Captains is open for applications from February 5–23, 2024, at 5:00 PM EST.

AWS open source news and updates – Our colleague Ricardo writes this weekly open source newsletter highlighting new open source projects, tools, and demos from the AWS Community.

Upcoming AWS events

Check your calendars and sign up for upcoming AWS events:

Building with Generative AI on AWS using PartyRock, Amazon Bedrock and Amazon Q – You will gain skills in prompt engineering and using the Amazon Bedrock API. We will also explore how to “chat with your documents” through knowledge bases, Retrieval Augmented Generation (RAG), embeddings, and agents. We will also use next-generation developer tools Amazon Q and Amazon CodeWhisperer to assist in coding and debugging.

Location: AWS Skills Center, 1550-G Crystal Drive, Arlington, VA

AI/ML security – Artificial intelligence and machine learning (AI/ML) and especially generative AI  have become top of mind for many organizations, but even the companies who want to move forward with this new and transformative technology are hesitating. They don’t necessarily understand how they can ensure that what they build will be secure. This webinar explains how they can do that.

AWS Jam Session – Canada Edition – AWS JAM is a gamified learning platform where you come to play, learn, and validate your AWS skills. The morning will include a mix of challenges across various technical domains – security, serverless, AI/ML, analytics, and more. The afternoon will be focused on a different specialty domain each month. You can form teams of up to four people to solve the challenges. There will be prizes for the top three winning teams.

Whether you’re in the Americas, Asia Pacific and Japan, or the EMEA region, there’s an upcoming AWS Innovate Online event that fits your time zone. Innovate Online events are free, online, and designed to inspire and educate you about AWS.

AWS Summits are a series of free online and in-person events that bring the cloud computing community together to connect, collaborate, and learn about AWS. These events are designed to educate you about AWS products and services and help you develop the skills needed to build, deploy, and operate your infrastructure and applications. Find an AWS Summit near you and register or set a notification to know when registration opens for a Summit that interests you.

AWS Community re:Invent re:Caps – Join a Community re:Cap event organized by volunteers from AWS User Groups and AWS Cloud Clubs around the world to learn about the latest announcements from AWS re:Invent.

You can browse all upcoming in-person and virtual events.

That’s all for this week. Check back next Monday for another Weekly Roundup!

– Irshad

This post is part of our Weekly Roundup series. Check back each week for a quick roundup of interesting news and announcements from AWS!

“Значителни като обем недвижими имоти и парични влогове” БОЕЦ сезира Сарафов за гъстия с горската мафия ексдепутат от ГЕРБ

Post Syndicated from Николай Марченко original

понеделник 19 февруари 2024

Гражданското движение “България обединена с една цел” (БОЕЦ) внесе сигнал до и.ф. главен прокурор на Република България Борислав Сарафов по повод разследването на “Биволъ” за чадър над горската мафия от…

Zabbix Conference Latam: living real connections

Post Syndicated from Fernanda Moraes original

It’s official! Registrations for Zabbix Conference Latam 2024 are now open.

Of all the events that our Zabbix team in Latin America organizes and participates in (over 50 in 2023 alone), we’re confident that this is the most impressive.

The 2024 conference is the third one organized directly by Zabbix since the beginning of our operations in Latin America. It has become a key reference point for topics related to data monitoring and Zabbix.

When our team participated in the last edition of Zabbix Summit, a global Zabbix event, I remember a partner asking me what was so special about an event like Zabbix Conference Latam. The answer is easy – the strength and vitality of the Latin American community!

A few days ago, I read an excerpt from a book by Brazilian sociologist Muniz Sodré, where he addressed the concept of “community.” Etymologically, the word “community” originates from the Latin “communitas,” composed of two radicals: “cum” (together with) and “munus” (obligation to the Other).

In essence, the sense of community is related to a collective dimension that allows us to be with and be together. There is a bond, something that makes us stay together. A point of similarity amidst differences, if you will.

Indeed, it’s not a very didactic concept, precisely because it needs to be lived – and felt. It is the strength of a community that produces possibilities and changes. And this is extremely present in open-source communities like the one we have at Zabbix.

The union of totally different people around a common point (Zabbix) is impressive – and captivating.

One of the greatest advantages of participating in a community like the one we’ve built at Zabbix is the fact that there is a direct relationship with collaborative culture. This makes users feel like protagonists and active subjects in the product’s development.

In communities like this, a collaborative strength exists among members, along with an open and genuine spirit of sharing and support. And that’s exactly what we experience at an event like Zabbix Conference Latam.

Every year, Zabbix warmly welcomes users, partners, clients, and enthusiasts. We receive fans who are excited to check out news about the tool, meet friends again, share knowledge, interact with experts, and even chat with Zabbix Founder and CEO Alexei Vladishev.

We hear amazing stories about how people came to know the tool, developed incredible projects, and transformed businesses – and how many other members also started their own businesses with Zabbix.

Zabbix Conference Latam is a space where there are real connections, dialogue, and (very) happy (re)encounters. In other words, it’s an experience that every member of the Zabbix Community should have.

Checking out news straight from the manufacturer

The event provides technical immersion through lectures, real-life case presentations, and technical workshops with the Zabbix team, official partners, clients, and experts in the field over both days of the event (June 7 and 8, 2024).

In other words, you can expect plenty of knowledge directly from the source – Alexei Vladishev, Founder and CEO of Zabbix! For those who use Zabbix or are interested in using it, you won’t want to miss the chance to participate, either through lectures or workshops.

Expanding networking

We plan to welcome over 250 participants, including technical leaders, analysts, infrastructure architects, engineers, and other professionals. It’s a great opportunity to meet colleagues in the field and make professional contacts.

Understanding a bit more about business

The open-source movement democratizes the use of technology, allowing companies of different sizes and segments to have freedom of use for powerful tools like Zabbix. At the Conference, we provide a space for discussion on open-source and business-related topics.

In 2024, we will feature the second edition of the Open Source and Business panel, where we will bring together leaders and companies to share views and perspectives on the relationship between the open-source theme and business development.

Get ready for lots of inspiration!

Talking to our official business partners and visiting sponsor booths at the event while enjoying a nice cup of coffee is a fascinating experience.

These interactions teach us a little more about their experiences and their relationship with Zabbix. From brand connections and integrations, simple implementations, or even extremely complex and creative projects, it’s possible to understand the real power of Zabbix and how it can positively impact different businesses.

A room full of opportunities

The speakers at Zabbix Conference Latam include our team of experts, official business partners, clients, and our community.

Among technical immersions and updated topics on functionalities, roadmaps, and all Zabbix news, community members can submit presentations and, if approved, participate in the event as speakers.

This allows them to share insights, discoveries, projects, and use cases in different industries, inspiring everyone with creative ways to solve real problems with Zabbix.

Living the Zabbix Conference Latam is a beautiful experience that allows us to understand the meaning and real strength of a community. Participating is also actively contributing to the growth and strengthening of the tool.

It truly is one of the best ways to evangelize Zabbix, and we look forward to gathering our community again in June 2024!

About Zabbix Conference Latam 2024

Zabbix Conference Latam 2024 is the largest Zabbix and monitoring event in Latin America. It takes place in São Paulo on June 7 and 8.

Interested parties can purchase tickets at the lowest price of the season, starting at R$999.00.

You can check out package information on the official event website.

The post Zabbix Conference Latam: living real connections appeared first on Zabbix Blog.

VMware VCSP Customers Seeing 10x or More Cost Increases Under Broadcom

Post Syndicated from Patrick Kennedy original

We discuss how some VMware customers are facing 10x or more cost increases under Broadcom with smaller MSPs hit particularly hard

The post VMware VCSP Customers Seeing 10x or More Cost Increases Under Broadcom appeared first on ServeTheHome.

Добро ченге, лошо ченге

Post Syndicated from Антония original

Чавето гледа филм. Спира го и идва да ме пита:

– Каква е тази игра на добро ченге и лошо ченге?
– Не е точно игра. Когато полицаите хванат заподозрян – обикновено го разпитват по двойки. Единият полицай се прави, че е приятел на заподозрения и иска да му помогне. Държи се мило с него, казва му неща като “За твое добро е, за да се измъкнеш от тази ситуация, влизам ти в положението, вината не е твоя, разбирам те”. Другият полицай се държи заплашително и кряска как ще вкара заподозрения в затвора и ще го държи там докато изгние. Така двамата полицаи ошашкват заподозрения и той може да си признае всичко.
– Аме заподозреният не знае ли, че трябва да говори само в присъствието на адвокат?!?

Научили сме го добре

[$] A Spritely distributed-computing library

Post Syndicated from daroc original

Spritely is a project seeking to
build a platform for sovereign distributed applications — applications where
users run their own nodes in order to control their own data — as the basis of a
new social internet.
While there are many such existing
projects, Spritely takes an unusual approach based on a new
interoperable protocol for
efficient, secure remote procedure calls (RPC). The project is in its early stages,
with many additional features planned, but it is already possible to play around
with Goblins, the distributed
actor library that Spritely intends to build on.

Improve your ETL performance using multiple Redshift warehouses for writes

Post Syndicated from Ryan Waldorf original

Amazon Redshift is a fast, petabyte-scale, cloud data warehouse that tens of thousands of customers rely on to power their analytics workloads. Thousands of customers use Amazon Redshift read data sharing to enable instant, granular, and fast data access across Redshift provisioned clusters and serverless workgroups. This allows you to scale your read workloads to thousands of concurrent users without having to move or copy the data.

Now, at Amazon Redshift we are announcing multi-data warehouse writes through data sharing in public preview. This allows you to achieve better performance for extract, transform, and load (ETL) workloads by using different warehouses of different types and sizes based on your workload needs. Additionally, this allows you to easily keep your ETL jobs running more predictably as you can split them between warehouses in a few clicks, monitor and control costs as each warehouse has its own monitoring and cost controls, and foster collaboration as you can enable different teams to write to another team’s databases in just a few clicks.

The data is live and available across all warehouses as soon as it is committed, even when it’s written to cross-account or cross-region. For preview you can use a combination of ra3.4xl clusters, ra3.16xl clusters, or serverless workgroups.

In this post, we discuss when you should consider using multiple warehouses to write to the same databases, explain how multi-warehouse writes through data sharing works, and walk you through an example on how to use multiple warehouses to write to the same database.

Reasons for using multiple warehouses to write to the same databases

In this section, we discuss some of the reasons why you should consider using multiple warehouses to write to the same database.

Better performance and predictability for mixed workloads

Customers often start with a warehouse sized to fit their initial workload needs. For example, if you need to support occasional user queries and nightly ingestion of 10 million rows of purchase data, a 32 RPU workgroup may be perfectly suited for your needs. However, adding a new hourly ingestion of 400 million rows of user website and app interactions could slow existing users’ response times as the new workload consumes significant resources. You could resize to a larger workgroup so read and write workloads complete quickly without fighting over resources. However, this may provide unneeded power and cost for existing workloads. Also, because workloads share compute, a spike in one workload can affect the ability of other workloads to meet their SLAs.

The following diagram illustrates a single-warehouse architecture.

Single-Warehouse ETL Architecture. Three separate workloads--a Purchase History ETL job ingesting 10M rows nightly, Users running 25 read queries per hour, and a Web Interactions ETL job ingesting 400M rows/hour--all using the same 256 RPU Amazon Redshift serverless workgroup to read and write from the database called Customer DB.

With the ability to write through datashares, you can now separate the new user website and app interactions ETL into a separate, larger workgroup so that it completes quickly with the performance you need without impacting the cost or completion time of your existing workloads. The following diagram illustrates this multi-warehouse architecture.

Multi-Warehouse ETL Architecture. Two workloads--a Purchase History ETL job ingesting 10M rows nightly and users running 25 read queries per hour--using a 32 RPU serverless workgroup to read from and write to the database Customer DB. It shows a separate workload--a Web Interactions ETL job ingesting 400M rows/hour--using a separate 128 RPU serverless workgroup to write to the database Customer DB.

The multi-warehouse architecture enables you to have all write workloads complete on time with less combined compute, and subsequently lower cost, than a single warehouse supporting all workloads.

Control and monitor costs

When you use a single warehouse for all your ETL jobs, it can be difficult to understand which workloads are contributing to your costs. For instance, you may have one team running an ETL workload ingesting data from a CRM system while another team is ingesting data from internal operational systems. It’s hard for you to monitor and control the costs for the workloads because queries are running together using the same compute in the warehouse. By splitting the write workloads into separate warehouses, you can separately monitor and control costs while ensuring the workloads can progress independently without resource conflict.

Collaborate on live data with ease

The are times when two teams use different warehouses for data governance, compute performance, or cost reasons, but also at times need to write to the same shared data. For instance, you may have a set of customer 360 tables that need to be updated live as customers interact with your marketing, sales, and customer service teams. When these teams use different warehouses, keeping this data live can be difficult because you may have to build a multi-service ETL pipeline using tools like Amazon Simple Storage Service (Amazon S3), Amazon Simple Notification Service (Amazon SNS), Amazon Simple Queue Service (Amazon SQS), and AWS Lambda to track live changes in each team’s data and ingest it into a single source.

With the ability to write through datashares, you can grant granular permissions on your database objects (for example, SELECT on one table, and SELECT, INSERT, and TRUNCATE on another) to different teams using different warehouses in a few clicks. This enables teams to start writing to the shared objects using their own warehouses. The data is live and available to all warehouses as soon as it is committed, and this even works if the warehouses are using different accounts and regions.

In the following sections, we walk you through how to use multiple warehouses to write to the same databases via data sharing.

Solution overview

We use the following terminology in this solution:

  • Namespace – A logical container for database objects, users and roles, their permissions on database objects, and compute (serverless workgroups and provisioned clusters).
  • Datashare – The unit of sharing for data sharing. You grant permissions on objects to datashares.
  • Producer – The warehouse that creates the datashare, grants permissions on objects to datashares, and grants other warehouses and accounts access to the datashare.
  • Consumer – The warehouse that is granted access to the datashare. You can think of consumers as datashare tenants.

This use case involves a customer with two warehouses: a primary warehouse used for attached to the primary namespace for most read and write queries, and a secondary warehouse attached to a secondary namespace that is primarily used to write to the primary namespace. We use the publicly available 10 GB TPCH dataset from AWS Labs, hosted in an S3 bucket. You can copy and paste many of the commands to follow along. Although it’s small for a data warehouse, this dataset allows easy functional testing of this feature.

The following diagram illustrates our solution architecture.

Architecture Diagram showing Two Warehouses for ETL

We set up the primary namespace by connecting to it via its warehouse, creating a marketing database in it with a prod and staging schema, and creating three tables in the prod schema called region, nation, and af_customer. We then load data into the region and nation tables using the warehouse. We do not ingest data into the af_customer table.

We then create a datashare in the primary namespace. We grant the datashare the ability to create objects in the staging schema and the ability to select, insert, update, and delete from objects in the prod schema. We then grant usage on the schema to another namespace in the account.

At that point, we connect to the secondary warehouse. We create a database from a datashare in that warehouse as well as a new user. We then grant permissions on the datashare object to the new user. Then we reconnect to the secondary warehouse as the new user.

We then create a customer table in the datashare’s staging schema and copy data from the TPCH 10 customer dataset into the staging table. We insert staging customer table data into the shared af_customer production table, and then truncate the table.

At this point, the ETL is complete and you are able to read the data in the primary namespace, inserted by the secondary ETL warehouse, from both the primary warehouse and the secondary ETL warehouse.


To follow along with this post, you should have the following prerequisites:

  • Two warehouses created with the PREVIEW_2023 track. The warehouses can be a mix of serverless workgroups, ra3.4xl clusters, and ra3.16xl clusters.
  • Access to a superuser in both warehouses.
  • An AWS Identity and Access Management (IAM) role that is able to ingest data from Amazon Redshift to Amazon S3 (Amazon Redshift creates one by default when you create a cluster or serverless workgroup).
  • For cross-account only, you need access to an IAM user or role that is allowed to authorize datashares. For the IAM policy, refer to Sharing datashares.

Refer to Sharing both read and write data within an AWS account or across accounts (preview) for the most up-to-date information.

Set up the primary namespace (producer)

In this section, we show how to set up the primary (producer) namespace we will use to store our data.

Connect to producer

Complete the following steps to connect to the producer:

  1. On the Amazon Redshift console, choose Query editor v2 in the navigation pane.

In the query editor v2, you can see all the warehouses you have access to in the left pane. You can expand them to see their databases.

  1. Connect to your primary warehouse using a superuser.
  2. Run the following command to create the marketing database:

Create the database objects to share

Complete the following steps to create your database objects to share:

  1. After you create the marketing database, switch your database connection to the marketing database.

You may need to refresh your page to be able to see it.

  1. Run the following commands to create the two schemas you intend to share:
  1. Create the tables to share with the following code. These are standard DDL statements coming from the AWS Labs DDL file with modified table names.
create table prod.region (
  r_regionkey int4 not null,
  r_name char(25) not null ,
  r_comment varchar(152) not null,
  Primary Key(R_REGIONKEY)

create table prod.nation (
  n_nationkey int4 not null,
  n_name char(25) not null ,
  n_regionkey int4 not null,
  n_comment varchar(152) not null,
  Primary Key(N_NATIONKEY)

create table prod.af_customer (
  c_custkey int8 not null ,
  c_name varchar(25) not null,
  c_address varchar(40) not null,
  c_nationkey int4 not null,
  c_phone char(15) not null,
  c_acctbal numeric(12,2) not null,
  c_mktsegment char(10) not null,
  c_comment varchar(117) not null,
  Primary Key(C_CUSTKEY)
) distkey(c_custkey) sortkey(c_custkey);

Copy data into the region and nation tables

Run the following commands to copy data from the AWS Labs S3 bucket into the region and nation tables. If you created a cluster while keeping the default created IAM role, you can copy and paste the following commands to load data into your tables:

copy prod.nation from 's3://redshift-downloads/TPC-H/2.18/10GB/nation.tbl' iam_role default delimiter '|' region 'us-east-1';
copy prod.region from 's3://redshift-downloads/TPC-H/2.18/10GB/region.tbl' iam_role default delimiter '|' region 'us-east-1';

Create the datashare

Create the datashare using the following command:

create datashare marketing publicaccessible true;

The publicaccessible setting specifies whether or not a datashare can be used by consumers with publicly accessible provisioned clusters and serverless workgroups. If your warehouses are not publicly accessible, you can ignore that field.

Grant permissions on schemas to the datashare

To add objects with permissions to the datashare, use the grant syntax, specifying the datashare you’d like to grant the permissions to:

grant usage on schema prod to datashare marketing;
grant usage, create on schema staging to datashare marketing;

This allows the datashare consumers to use objects added to the prod schema and use and create objects added to the staging schema. To maintain backward compatibility, if you use the alter datashare command to add a schema, it will be the equivalent of granting usage on the schema.

Grant permissions on tables to the datashare

Now you can grant access to tables to the datashare using the grant syntax, specifying the permissions and the datashare. The following code grants all privileges on the af_customer table to the datashare:

grant all on table prod.af_customer to datashare marketing;

To maintain backward compatibility, if you use the alter datashare command to add a table, it will be the equivalent of granting select on the table.

Additionally, we’ve added scoped permissions that allow you to grant the same permission to all current and future objects within the datashare. We add the scoped select permission on the prod schema tables to the datashare:

grant select for tables in schema prod to datashare marketing;

After this grant, the customer will have select permissions on all current and future tables in the prod schema. This gives them select access on the region and nation tables.

View permissions granted to the datashare

You can view permissions granted to the datashare by running the following command:

show access for datashare marketing;

Grant permissions to the secondary ETL namespace

You can grant permissions to the secondary ETL namespace using the existing syntax. You do this by specifying the namespace ID. You can find the namespace on the namespace details page if your secondary ETL namespace is serverless, as part of the namespace ID in the cluster details page if your secondary ETL namespace is provisioned, or by connecting to the secondary ETL warehouse in the query editor v2 and running select current_namespace. You can then grant access to the other namespace with the following command (change the consumer namespace to the namespace UID of your own secondary ETL warehouse):

grant usage on datashare marketing to namespace '<consumer_namespace>';

Set up the secondary ETL namespace (consumer)

At this point, you’re ready to set up your secondary (consumer) ETL warehouse to start writing to the shared data.

Create a database from the datashare

Complete the following steps to create your database:

  1. In the query editor v2, switch to the secondary ETL warehouse.
  2. Run the command show datashares to see the marketing datashare as well as the datashare producer’s namespace.
  3. Use that namespace to create a database from the datashare, as shown in the following code:
create database marketing_ds_db with permissions from datashare marketing of namespace '&lt;producer_namespace&gt;';

Specifying with permissions allows you to grant granular permissions to individual database users and roles. Without this, if you grant usage permissions on the datashare database, users and roles get all permissions on all objects within the datashare database.

Create a user and grant permissions to that user

Create a user using the CREATE USER command:

create user data_engineer password '[choose a secure password]';
grant usage on database marketing_ds_db to data_engineer;
grant all on schema to data_engineer;
grant all on schema marketing_ds_db.staging to data_engineer;
grant all on all tables in schema marketing_ds_db.staging to data_engineer;
grant all on all tables in schema to data_engineer;

With these grants, you’ve given the user data_engineer all permissions on all objects in the datashare. Additionally, you’ve granted all permissions available in the schemas as scoped permissions for data_engineer. Any permissions on any objects added to those schemas will be automatically granted to data_engineer.

At this point, you can continue the steps using either the admin user you’re currently signed in as or the data_engineer.

Options for writing to the datashare database

You can write data to the datashare database three ways.

Use three-part notation while connected to a local database

Like with read data sharing, you can use three-part notation to reference the datashare database objects. For instance, insert into Note that you can’t use multi-statement transactions to write to objects in the datashare database like this.

Connect directly to the datashare database

You can connect directly to the datashare database via the Redshift JDBC, ODBC, or Python driver, in addition to the Amazon Redshift Data API (new). To connect like this, specify the datashare database name in the connection string. This allows you to write to the datashare database using two-part notation and use multi-statement transactions to write to the datashare database. Note that some system and catalog tables are not available this way.

Run the use command

You can now specify that you want to use another database with the command use <database_name>. This allows you to write to the datashare database using two-part notation and use multi-statement transactions to write to the datashare database. Note that some system and catalog tables are not available this way. Also, when querying system and catalog tables, you will be querying the system and catalog tables of the database you are connected to, not the database you are using.

To try this method, run the following command:

use marketing_ds_db;

Start writing to the datashare database

In this section, we show how to write to the datashare database using the second and third options we discussed (direct connection or use command). We use the AWS Labs provided SQL to write to the datashare database.

Create a staging table

Create a table within the staging schema, because you’ve been granted create privileges. We create a table within the datashare’s staging schema with the following DDL statement:

create table staging.customer (
  c_custkey int8 not null ,
  c_name varchar(25) not null,
  c_address varchar(40) not null,
  c_nationkey int4 not null,
  c_phone char(15) not null,
  c_acctbal numeric(12,2) not null,
  c_mktsegment char(10) not null,
  c_comment varchar(117) not null,
  Primary Key(C_CUSTKEY)
) distkey(c_nationkey) sortkey(c_nationkey);

You can use two-part notation because you used the USE command or directly connected to the datashare database. If not, you need to specify the datashare database names as well.

Copy data into the staging table

Copy the customer TPCH 10 data from the AWS Labs public S3 bucket into the table using the following command:

copy staging.customer from 's3://redshift-downloads/TPC-H/2.18/10GB/customer.tbl' iam_role default delimiter '|' region 'us-east-1';

As before, this requires you to have set up the default IAM role when creating this warehouse.

Ingest African customer data to the table prod.af_customer

Run the following command to ingest only the African customer data to the table prod.af_customer:

insert into prod.af_customer
select c.* from staging.customer c
  join prod.nation n on c.c_nationkey = n.n_nationkey
  join prod.region r on n.n_regionkey = r.r_regionkey
  where r.r_regionkey = 0; --0 is the region key for Africa

This requires you to join on the nation and region tables you have select permission for.

Truncate the staging table

You can truncate the staging table so that you can write to it without recreating it in a future job. The truncate action will run transactionally and can be rolled back if you are connected directly to the datashare database or you are using the use command (even if you’re not using a datashare database). Use the following code:

truncate staging.customer;

At this point, you’ve completed ingesting the data to the primary namespace. You can query the af_customer table from both the primary warehouse and secondary ETL warehouse and see the same data.


In this post, we showed how to use multiple warehouses to write to the same database. This solution has the following benefits:

  • You can use provisioned clusters and serverless workgroups of different sizes to write to the same databases
  • You can write across accounts and regions
  • Data is live and available to all warehouses as soon as it is committed
  • Writes work even if the producer warehouse (the warehouse that owns the database) is paused

To learn more about this feature, see Sharing both read and write data within an AWS account or across accounts (preview). Additionally, if you have any feedback, please email us at [email protected].

About the authors

Ryan Waldorf is a Senior Product Manager at Amazon Redshift. Ryan focuses on features that enable customers to define and scale compute including data sharing and concurrency scaling.

Harshida Patel is a Analytics Specialist Principal Solutions Architect, with Amazon Web Services (AWS).

Sudipto Das is a Senior Principal Engineer at Amazon Web Services (AWS). He leads the technical architecture and strategy of multiple database and analytics services in AWS with special focus on Amazon Redshift and Amazon Aurora.

EU Court of Human Rights Rejects Encryption Backdoors

Post Syndicated from Bruce Schneier original

The European Court of Human Rights has ruled that breaking end-to-end encryption by adding backdoors violates human rights:

Seemingly most critically, the [Russian] government told the ECHR that any intrusion on private lives resulting from decrypting messages was “necessary” to combat terrorism in a democratic society. To back up this claim, the government pointed to a 2017 terrorist attack that was “coordinated from abroad through secret chats via Telegram.” The government claimed that a second terrorist attack that year was prevented after the government discovered it was being coordinated through Telegram chats.

However, privacy advocates backed up Telegram’s claims that the messaging services couldn’t technically build a backdoor for governments without impacting all its users. They also argued that the threat of mass surveillance could be enough to infringe on human rights. The European Information Society Institute (EISI) and Privacy International told the ECHR that even if governments never used required disclosures to mass surveil citizens, it could have a chilling effect on users’ speech or prompt service providers to issue radical software updates weakening encryption for all users.

In the end, the ECHR concluded that the Telegram user’s rights had been violated, partly due to privacy advocates and international reports that corroborated Telegram’s position that complying with the FSB’s disclosure order would force changes impacting all its users.

The “confidentiality of communications is an essential element of the right to respect for private life and correspondence,” the ECHR’s ruling said. Thus, requiring messages to be decrypted by law enforcement “cannot be regarded as necessary in a democratic society.”

Security updates for Monday

Post Syndicated from jake original

Security updates have been issued by Debian (engrampa, openvswitch, pdns-recursor, and runc), Fedora (caddy, expat, freerdp, libgit2, libgit2_1.6, mbedtls, python-cryptography, qt5-qtbase, and sudo), Gentoo (Apache Log4j, Chromium, Google Chrome, Microsoft Edge, CUPS, e2fsprogs, Exim, firefox, Glade, GNU Tar, intel-microcode, libcaca, QtNetwork, QtWebEngine, Samba, Seamonkey, TACACS+, Thunar, and thunderbird), Mageia (dnsmasq, unbound, and vim), Oracle (container-tools:4.0, container-tools:ol8, dotnet6.0, dotnet7.0, kernel, nss, openssh, and sudo), Red Hat (python-pillow), and SUSE (bitcoin, dpdk, libssh, openvswitch, postgresql12, and postgresql13).

Hello World #23 out now: Global exchange of computing education ideas

Post Syndicated from Meg Wang original

How is computing taught around the globe? Our brand-new, free issue of Hello World, out today, paints a picture for you. It features stories from over 20 countries, where educators, researchers, and volunteers share their work and their personal challenges and joys in bringing computing education to their part of the world.

The Hello World Global Exchange magazine cover on a plain background.

Global exchange in a worldwide community

In Hello World issue 23, you’ll hear about countries where computing is an official school subject and how it was set up that way, and you’ll hear about countries that are newer to computing education and working to fast-track their students’ learning.

  • Ethel Tshukudu’s article on her research using the CAPE framework is a fascinating comparison of computer science education in four African countries
  • Iliana Ramirez describes how volunteers are at the heart of Ciberistas, a technology training programme for young people in Mexico
  • Matthew Griffin’s article highlights how computing education works in Canada, a large country with two official languages
  • Dana Rensi’s article about a solar-powered Raspberry Pi computing lab in the middle of the Peruvian rainforest will surprise and delight you
  • Randal Rousseau, a librarian in Cape Town, South Africa, shares how he teaches children to code through unplugged activities

And there is lots more for you to discover in issue 23.

Sue Sentance, director of the Raspberry Pi Computing Education Research Centre at the University of Cambridge, says in her article:

“Our own experience of implementing computing education in England since 2014 has shown the importance of teachers supporting each other, and how various networks … are instrumental in bringing computing teachers together to share knowledge and experiences. With so many countries introducing computing education, and teachers around the globe facing similar challenges, maybe we need to extend this to a global teacher network, where teachers and policymakers can share good practice and learn from each other. “

We aim for Hello World magazine to be one of the places where this sharing, exchange, and learning can take place. Subscribe for free to never miss an issue, and find out how you can write for the magazine.

Download Hello World issue 23 for free

Research highlights the importance of computing education to young people’s futures, whether or not they pursue a degree or career in the area. From teaching computing in schools where the electricity cuts out, to incorporating artificial intelligence into curricula in different countries, and to teaming up with local governments when there isn’t a national computing curriculum, educators are doing wonderful things around the globe to make sure the young people they support have the opportunity to learn. Read their stories today.

Also in issue 23:

  • Research on culturally adapted resources 
  • How community building enhances computing education
  • Tips for hosting a STEM event in school

And much, much more.

Send us a message or tag us on social media to let us know which articles have made you think, and most importantly, which will help you with your teaching. And to hear monthly news about Hello World and the whole Raspberry Pi Foundation, sign up to the Hello World newsletter.

The post Hello World #23 out now: Global exchange of computing education ideas appeared first on Raspberry Pi Foundation.

All you need to know about the Digital Services Act

Post Syndicated from Petra Arts original

February 17th, 2024 marked the entry into force of a landmark piece of European Union (EU) legislation, affecting European users who create and disseminate online content as well as tech companies who act as “intermediaries” on the Internet. I am talking of course about the EU Digital Services Act, or DSA for short. The DSA was first proposed in December 2020, and is meant to update a 20-year-old law called the EU e-commerce Directive, which provides important safeguards and legal certainty for all businesses operating online. The principles of that legal framework, most notably the introduction of EU-wide rules on intermediary liability, are still of major importance today. The DSA is a landmark piece of European legislation because it also sets out, for the first time, enhanced regulatory requirements for (large) digital platforms, thus affecting the entire Internet ecosystem.

At Cloudflare, we are supportive of the longstanding legal frameworks both in Europe and other parts of the world that protect Internet companies from liability for the content that is uploaded or sent through their networks by their users, subscribers or customers. These frameworks are indispensable for the growth of online services, and have been essential in the growth of online applications, marketplaces and social networks.

What’s the Digital Services Act all about?

The EU Digital Services Act consists of two main parts: First, the DSA maintains the strong liability protections for intermediary services that have existed in Europe for over 20 years, and modernizes them, including by giving explicit recognition of supporting Internet services. Services which perform important roles in the functioning of the Internet, such as CDNs, reverse proxies and technical services at the DNS level were not explicitly mentioned in the EU e-commerce Directive at the time. The DSA, in recital 28, recognises that those services, along with many others, are part of the fundamental fabric of the Internet and deserve protection against liability for any illegal or infringing content. This marks an important clarification milestone in EU law.

Secondly, the DSA establishes varying degrees of due diligence and transparency obligations for intermediary services that offer services in the EU. The DSA follows a ‘staggered’ or ‘cumulative’ approach to those obligations and the different services it applies to. This ranges from a number of detailed obligations for the largest platforms (so-called “Very Large Online Platforms” or VLOPs, such as the Apple App Store, Facebook, TikTok, and YouTube), down to less extensive but still impactful rules for smaller platforms, hosting services and Internet intermediaries. What is really important to note with regard to the different service providers that are impacted is that the DSA clearly distinguishes between (technical) intermediary services, “mere” hosting services, and “online platforms”, with the latter category having a number of additional obligations under the new law. Online platform services are considered as hosting services which store information at the request of the recipients of the service, with the important additional role of also disseminating that information to the public.

This proportionate approach is in line with Cloudflare’s view of the Internet stack and the idea that infrastructure services are distinct from social media and search services that are designed to curate and recommend Internet content. This principle of a targeted, proportionate response to the matter is also embedded in the DSA itself. Recital 27 states that “(…) any requests or orders for [such] involvement should, as a general rule, be directed to the specific provider that has the technical and operational ability to act against specific items of illegal content, so as to prevent and minimise (sic) any possible negative effects on the availability and accessibility of information that is not illegal content”. This is an important provision, as principles of proportionality, freedom of speech, and access to information should be safeguarded at all times when it relates to online content.

What do the new rules mean for Cloudflare?

As a provider of intermediary services, Cloudflare has engaged with European policymakers on the topic of intermediary liability for a number of years. From the start of the legislative process on the proposed DSA in 2020 we have contributed extensively to public consultations, and have shared our views on the proposed DSA with lawmakers in Brussels.

In many ways, the final version of the law reflects our existing practices. We have long taken the position, for example, that our intermediary services should have different rules than our hosting services, as is anticipated under the DSA. We have taken a few additional measures to ensure compliance with DSA requirements. For instance, we’ve announced a new legal representative in the EU and point of contact for the purposes of the DSA.

Cloudflare has strongly believed in transparency reporting for a long time, and we have issued transparency reports twice a year since 2013. We recognize that the DSA includes some new requirements around transparency reporting, some of which match with our current reports and processes, and others that do not. We will be revising our transparency reporting, to reflect the DSA’s requirements, beyond our existing documentation. We have also taken steps to confirm that our limited hosting services comply with DSA requirements.

The EU Digital Services Act, because of its enhanced regulatory requirements for (large) digital platforms, represents a significant change to the Internet ecosystem. Cloudflare feels nonetheless well-prepared to address the different requirements that came into force on February 17, 2024, and we look forward to having positive and constructive conversations with relevant European regulators as they start to work on the enforcement of the new law.

How to Start Stop and List Proxmox VE Virtual Machines via the CLI

Post Syndicated from Eric Smith original

We have a quick guide if you need to use the CLI shell in Proxmox VE to list, start, stop, restart, shutdown, suspend, or resume VMs

The post How to Start Stop and List Proxmox VE Virtual Machines via the CLI appeared first on ServeTheHome.

The collective thoughts of the interwebz

By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.