OpenSSH 8.7 has been released. Changes include
steps toward deprecating scp and
using the SFTP protocol for file transfers instead, changes to
remote-to-remote copies (they go through the local host by default now), a
stricter configuration-file parser, and more.
Unix-like systems — and their users — tend to expect all filesystems to
behave in the same way. But those users are also often interested in fancy
new filesystems offering features that were never envisioned by the
developers of the Unix filesystem model; that has led to a number of
interesting incompatibilities over time. Btrfs is certainly one of those
filesystems; it provides a long list of features that are found in few
other systems, and some of those features interact poorly with the
traditional view of how filesystems work. Recently, Neil Brown has been
trying to resolve a specific source of confusion relating to how Btrfs
handles inode numbers.
Second, these new and increasingly specialized requirements
primarily benefit a specific class of open source users—large
enterprises. That isn’t necessarily a bad thing—big enterprises are
essential in many ways, and indeed, the risks to them deserve to be
taken seriously.
But in a world where hundreds of billions of dollars in enterprise
value have been created by open source, and where small
educational/hobby projects (and even many small companies) don’t
really benefit from these new unfunded mandates, developers will
likely focus on other things, since few of them got into open
source primarily to benefit the Fortune 500.
In this post, I’ll collect links on Apple’s iPhone backdoor for scanning CSAM images. Previous links are here and here.
Apple says that hash collisions in its CSAM detection system were expected, and not a concern. I’m not convinced that this secondary system was originally part of the design, since it wasn’t discussed in the original specification.
Good op-ed from a group of Princeton researchers who developed a similar system:
Our system could be easily repurposed for surveillance and censorship. The design wasn’t restricted to a specific category of content; a service could simply swap in any content-matching database, and the person using that service would be none the wiser.
Starting today, your team can use Cloudflare’s Browser Isolation service to protect sensitive data inside the web browser. Administrators can define Zero Trust policies to control who can copy, paste, and print data in any web based application.
In March 2021, for Security Week, we announced the general availability of Cloudflare Browser Isolation as an add-on within the Cloudflare for Teams suite of Zero Trust application access and browsing services. Browser Isolation protects users from browser-borne malware and zero-day threats by shifting the risk of executing untrusted website code from their local browser to a secure browser hosted on our edge.
And currently, we’re democratizing browser isolation for any business by including it with our Teams Enterprise Plan at no additional charge.1
A different approach to zero trust browsing
Web browsers, the same tool that connects users to critical business applications, is one of the most common attack vectors and hardest to control.
Browsers started as simple tools intended to share academic documents over the Internet and over time have become sophisticated platforms that replaced virtually every desktop application in the workplace. The dominance of web-based applications in the workplace has created a challenge for security teams who race to stay patch zero-day vulnerabilities and protect sensitive data stored in self-hosted and SaaS based applications.
In an attempt to protect users and applications from web based attacks, administrators have historically relied on DNS or HTTP inspection to prevent threats from reaching the browser. These tools, while useful for protecting against known threats, are difficult to tune without false-positives (negatively impacting user productivity and increasing IT support burden) and ineffective against zero day vulnerabilities.
Browser isolation technologies mitigate risk by shifting the risk of executing foreign code from the endpoint to a secure environment. Historically administrators have had to make a compromise between performance and security when adopting such a solution. They could either:
Prioritizesecurity by choosing a solution that relies on pixel pushing techniques to serve a visual representation to users. This comes at the cost of performance by introducing latency, graphical artifacts and heavy bandwidth usage.
OR
Prioritize performance by choosing a solution that relies on code scrubbing techniques to unpack, inspect and repack the webpage. This model is fragile (often failing to repack leading to a broken webpage) and insecure by allowing undetected threats to compromise users.
At Cloudflare, we know that security products do not need to come at the expense of performance. We developed a third option that delivers a remote browsing experience without needing to compromise on performance and security for users.
Prioritize security by never sending foreign code to the endpoint and executing it in a secure remote environment.
Prioritizeperformance sending light-weight vector instructions (rather than pixels) over the wire and minimize remote latency on our global edge network.
This unique approach delivers an isolated browser without the security or performance challenges faced by legacy solutions.
Data control through the browser
Malware and zero-day threats are not the only security challenges administrators face with web browsers. The mass adoption of SaaS products has made the web browser the primary tool used to access data. Lack of control over both the application and the browser has left administrators little control over their data once it is delivered to an endpoint.
Data loss prevention tools typically rely on pattern recognition to partially or completely redact the transmission of sensitive data values. This model is useful for protecting against an unexpected breach of PII and PCI data, such as locations and financial information but comes at the loss of visibility.
The redaction model falls short when sensitive data does not fit into easily recognizable patterns, and the end-users require visibility to do their job. In industries such as health care, redacting sensitive data is not feasible as medical professions require visibility of patient notes and appointment data.
Once data lands in the web browser it is trivial for a user to copy-paste and print sensitive data into another website, application, or physical location. These seemingly innocent actions can lead to data being misplaced by naive users leading to a data breach. Administrators have had limited options to protect data in the browser, some even going so far as to deploy virtual desktop services to control access to a SaaS based customer relationship management (CRM) tool. This increased operating costs, and frustrated users who had to learn how to use computer-in-a-computer just to use a website.
One-click to isolate data in the browser
Cloudflare Browser Isolation executes all website code (including HTML) in the remote browser. Since page content remains on the remote browser and draw instructions are only sent to the browser, Cloudflare Browser Isolation is in a powerful position to protect sensitive data on any website or SaaS application.
Administrators can now control copy-paste, and printing functionality with per-rule granularity with one click in the Cloudflare for Teams Dashboard. For example, now administrators can build rules that prevent users from copying information from your CRM or that stop team members from printing data from your ERP—without blocking their attempts to print from external websites where printing does not present a data loss risk.
From the user’s perspective websites look and behave normally until the user performs a restricted action.
Copy-paste and printing control can be configured for both new and existing HTTP policies in the Teams Dashboard.
Navigate to the Cloudflare for Teams dashboard.
Navigate to Gateway → Policies → HTTP.
Create/update an HTTP policy with an Isolate action (docs).
Configure policy settings.
Administrators have flexibility with data protection controls and can enable/disable browser behaviours based on application, hostname, user identity and security risk.
What’s next?
We’re just getting started with zero trust browsing controls. We’re hard at work building controls to protect against phishing attacks, further protect data by controlling file uploading and downloading without needing to craft complex network policies as well as support for a fully clientless browser isolation experience.
Democratizing browser isolation for any business
Historically, only large enterprises had justified the cost to add on remote browser isolation to their existing security deployments. And the resulting loosely-integrated solution fell short of achieving Zero Trust due to poor end-user experiences. Cloudflare has already solved these challenges, so businesses achieve full Zero Trust security including browser-based data protection controls without performance tradeoffs.
Yet it’s not always enough to democratize Zero Trust browser isolation for any business, so we’re currently including it with our Teams Enterprise Plan at no additional charge.1Get started here.
As any job seeker knows, a lot of thought goes into accepting a new role at a new company — even more so during a pandemic. For sales professionals, this decision includes considering company growth and trajectory, industry leadership, and company culture, all of which had the potential of being majorly impacted by the effects of COVID-19.
We talked with a few of our North America Account Executives to hear firsthand about why they chose to join Rapid7 (even during a pandemic), how they learned about the company, and why they’d recommend Rapid7 as a great place to work.
Where did you hear about Rapid7?
“Prior to joining the team, I’d long been familiar with Rapid7 as a high-profile, publicly traded company here in Boston. Around June of 2019, the word had spread about what an engaging and rewarding culture existed at Rapid7.” – Nicholas Lennek
“I heard about Rapid7 because of the tech scene here in Boston; Rapid7 is a big name, and as I was searching for a career change. I knew of a few friends who had been working here for a few years at the time.” – Devonne Skinner
What attracted you to work for Rapid7?
“The energy! My role at Rapid7 gives me the privilege of tackling unique and often nuanced challenges on a routine basis. The team mentality here at Rapid7 is one of rigor and dedication, which is an attitude I crave. We set our aims high, work hard to achieve them, and recognize a job well done here. To me, that’s what it’s all about.” – Nicholas Lennek
“I started in the beginning of the pandemic, which sounds a bit crazy, I know. At the time, I felt like I needed a challenge, and switching industries alone was a mountain, but adding in starting remote was a whole other ball game. I was very confident after my interviews that Rapid7 was going to be invested in me through learning the security industry, along with helping me continue my growth here. So the clear path to where I could go in this role, along with training, was something that attracted me to Rapid7. Additionally, the team atmosphere was big to me. Collaborating with not just sales but other departments is important to not just individual success but team and company success.” – Devonne Skinner
“I realized that the company had a vision for the future; as the industry grew and progressed, so did Rapid7. I wanted to work for a company that was looking to be a leader in the space and not just another player. Rapid7 has exceeded my expectations and continues to grow rapidly.” – Stephen Hislop
Why would you recommend Rapid7 as a great place to work for your next opportunity?
“Speaking to my own personal path, Rapid7 offers you a chance to skip the local train and join the expressway. My role at Rapid7 isn’t a job, it’s a career — and that same opportunity is afforded to everybody. Your peers and your leadership personally invest in your growth and in your success. To fulfill my professional ambitions, I need the chance to make an impact with my work. Rapid7 has provided me with the platform to do that every day that I’ve been here.” – Nicholas Lennek
“I would recommend Rapid7 as a great place to work because of the attention to their employees. As mentioned, it was important to me when I took this role that I had a clear path to how I was going to achieve my goals and continue down a path of success. Rapid7 has always asked me how they can help me achieve my goals, and individually, that is going to be different for everyone. But they follow through. Whether that be additional training, collaboration with teams, listening to my ideas, etc., they take the time to listen and respond appropriately. Rapid7 is a great company to work for regardless of the industry you’re coming from, because they are invested in YOU just as much as the bigger picture.” – Devonne Skinner
“I would recommend Rapid7 because it is a company that will challenge you but also train you. I have learned so much during my time here and have grown as an Account Executive. I love working with everyone, and it truly is a collaborative environment.” – Stephen Hislop
The tools we use at work are starting to look like the apps we use in our personal lives. We send emails for our jobs using Google Workspace and respond to personal notes in Gmail. We download PDFs from our team’s Dropbox and then upload images to our personal account. This can lead to confusion and mistakes—made worse by remote work when we forget to log off for the day.
Today, we’re excited to announce Tenant Control in Cloudflare Gateway, a new feature that helps keep our work at work. Organizations can deploy Cloudflare Gateway to their corporate devices and apply rules ensuring that employees can only log in to the corporate version of the tools they need. Now, teams can prevent users from logging in to the wrong instance of popular applications. What’s more, they can make sure corporate data stays within corporate accounts.
Controlling the application, alone, isn’t sufficient
Cloudflare Gateway provides security from threats on the Internet by sending all traffic leaving a device to Cloudflare’s network where it can be filtered. Organizations send traffic to Cloudflare by deploying the WARP agent, a WireGuard-based client built on feedback from our popular consumer app.
Cloudflare Gateway can be deployed in several modes, but most customers start with DNS filtering which only sends DNS queries to Cloudflare. Cloudflare runs the world’s fastest DNS resolver, 1.1.1.1, and on top of that we’ve built a DNS filtering solution where we help prevent users from visiting sites that contain malware or serve phishing attacks.
When organizations are ready to add more security to their deployment, they go beyond DNS filtering by adding HTTP filtering as well. Cloudflare inspects the HTTP traffic leaving the device which provides more granular control than just DNS filtering over destinations and events that happen inside the traffic, like blocking file uploads to certain destinations.
Customers use the HTTP filtering to filter and control SaaS application usage. For example, if your team uses OneDrive, you can block all file uploads to Google Drive to avoid data leaving the tenants you control. Cloudflare provides the classification of what hostnames and URLs constitute an application and make it possible to build rules with just two clicks. However, what happens when you aren’t using two different applications — you’re using two different instances of the same one?
Applying control to the SaaS tenant
Today, you can enable tenant control using Gateway HTTP policies in Cloudflare Gateway. Administrators can begin by adding a new type of rule in Gateway that prompts them to input a specific value provided by the SaaS application. For example, an administrator can gather the tenant ID for their Microsoft 365 deployment.
Once the rule is enabled, Cloudflare Gateway will append a specific header and, if enabled, the specific tenant ID as part of the appended header to your request. Depending on the SaaS application, these will either block all consumer or personal usage or block all logins to accounts that are not part of that tenant ID. The SaaS application is aware of the specific header it relies on to enforce this rule and, when received, responds accordingly.
Traditionally, these headers are injected by corporate VPNs or proxy servers maintained on-premises and accessed by backhauling user traffic. Cloudflare Gateway provides customers with filtering and inspection in our data centers closer to your users and, combined with our ability to accelerate traffic, delivers your users to their destination without the performance consequences of legacy backhaul approaches.
Enforcing Corporate Tenant Access
You can begin configuring these rules today in the Cloudflare for Teams dashboard. To enforce tenant control with Gateway, you can configure an HTTP policy in the Teams Dashboard. For example, you can prevent users from authenticating to GSuite with their personal account and uploading documents to Google Drive account by using the following policy (GSuite uses the “X-GooGApps-Allowed-Domains” header):
As requests get filtered by Gateway’s firewall, allowed requests are proxied to their respective upstream servers. Before sending them upstream, we preprocess the request and append our own trace headers — these include things that are useful for debugging, like request ID headers. Now you can specify your own custom headers to be added onto these requests, which is what enables customers to enforce tenant control for their organizations.
What’s Next
Controlling data usage in your organization is a multistep process. Today, Cloudflare Gateway gives your teams control of what applications you use, where you can upload or download files, and when to block copy-paste and printing in our isolated browser. We’re excited to introduce tenant control into that portfolio to add another layer of security.
That said, we’re just getting started. We’ll be introducing new scanning features on top of this existing functionality as we continue to build Cloudflare’s data control features. If you want to be the first to know about the next wave of these features, follow this link to sign up today.
We’re not going to lie — the thing we like most about this automated plant watering project is the timelapse at the very end of the build video. But we also thought now might be a good time to show you another Raspberry Pi project for keeping an eye on your plants, since some of us are getting back to something more like our usual routines and our houseplants are no longer our best friends, so they might need a little extra automated attention.
Maker Christopher Barnatt chose Raspberry Pi Zero for this project because although Raspberry Pi Pico could handle it, he needed a camera connector to record timelapse video of his plants’ growth.
Hardware
Christopher is a gem and has included links to all the hardware he used. There are also some cheaper, smaller alternatives listed in the info section of his build video.
The moisture sensor checks every half hour to determine whether or not the plant has enough water, and communicates with the Raspberry Pi. Water flow is controlled by the solenoid valve, and if the Raspberry Pi finds the soil is too dry, it opens the valve for a set amount of time to let water out.
Code your own plant watering machine
Christopher has shared all the code you need to make your own plant watering system:
Watering.py — the final watering system and timelapse code
Check out Christopher’s YouTube channel Explaining Computers where he posts new videos every week on topics like PC hardware, single board computers such as Raspberry Pi, AI, Big Data, and quantum computing.
През май 2017-та година поискан по ЗДОИ информация за изплатените детски надбавки от социалното министерство. Интересуваше ме за колко семейства и колко деца са изплатени те за всяка община. На тази база направих анализ и карта с различни изводи. Един от тях беше, че броят деца следва почти икономически признак, а далеч не етнически такъв. С последното се спекулира доста, особено що се отнася до използването на този вид помощи. Цялата статия ще намерите тук: Данните за детските надбавки и какво може да научим от тях
Онзи анализ се базираше единствено на данните за 2016-та година. Затова наскоро реших да искам обновените данни включвайки отказите за помощи, както и разбивка не само по общини, но и за всеки месец от януари 2016-та до декември 2020. Включих също така справка за получените в натура помощи от непълнолетни майки. За последните писах подробно през февруари 2016-та използвайки данните на НСИ от тогава.
За справката използвах портала по ЗДОИ и получих доста бързо подробен отговор. Интересното беше, че освен по мейл, както исках, ми изпратиха справките и по пощата на дискове. Това несъответствие с вътрешните правила и въобще как се предоставят данни е нещо, по което ще работим.
За съжаление, както и в други случаи, не ми остава все още време да обработя данните, да ги визуализирам смислено и да направя анализ подобен на този от преди 4 години. Може би нещо се е променило от тогава. Може би има динамика между годините или по месеци, за която не знаем. Може би има условности в данните, които виждаме.
Затова пускам таблицата свободно за всеки, който иска да я използва. Ако направите нещо с тях, ще се радвам да пуснете линк в коментарите. Направих си труда да ги събера в две таблици, тъй като ми ги бяха предоставили в повече. Стигнах само до там да отбележа общините по NUTS4 кодове, за да може да се покажат по-лесно на карта като тези тук.
Забелязват се обаче някои подробности. Има записи с квартали в градове като София, Пловдив и Бургас. Тях съм ги отбелязал с пореден номер след NUTS4 кода. Например VAR0601 до VAR0605 за Варна. При данните за помощи в натура има и запис за София-област, т.е. момиче, за което се знае, че живее в областта, но не е ясно точно къде. Поне така предполагам. За да се покажат тези на карта, трябва или да се съберат на ниво общини, или да се нареже geojson-а на големите градове по квартали. В някои месеци, особено през 2016-та, има данни за общината, но не за кварталите в нея. Вижда се също, че липсват някои числа за част от 2016-та. Та трябва известно чистене трябва преди да се направи нещо.
Както с данните на НЦОЗА моля за внимание към условностите на данните, кой ги е предоставил и какво показват. Лесно се правят сензации и за това не са ни нужни данни, а и може просто да си ги измислим, както предостатъчно хора правят. С тези тук бихме могли да научим нещо.
In any business intelligence system, securing and restricting access to the data is important. For example, you might want a particular dashboard to only be viewed by the users with whom the dashboard has been shared, yet customize the data displayed on that dashboard per user by implementing row-level security. With row-level security, you can grant and restrict access to only specific rows, all rows, or no rows of any dataset.
Organizations typically want to display different data to users within different business units. To do this, we need to traverse the organizational hierarchy to find the ultimate business owner for each employee. This post goes into the details of how we can use Lightweight Directory Access Protocol (LDAP) data to find what business unit each employee belongs to, and how we can apply data restriction using row-level security at an Amazon QuickSight group level rather than at an individual user level.
Additionally, this post explains how to automate user and group management within QuickSight to add or remove a QuickSight group or a user to a group.
Architectural overview
The following diagram illustrates the solution architecture.
The solution has two parts:
We first move data from an LDAP data store to Amazon Redshift or any other QuickSight supported data source, flatten it and integrate it with existing analytical data, and create a joined result set matching the organizational hierarchy
Then we automate the user and group management, which we apply to the permission file in QuickSight to dictate access to a particular dataset
We demonstrate this with a fictitious IT support ticketing system. This system tracks who is creating a particular support ticket and who is working on that ticket. We use QuickSight to visualize trends like number of tickets worked on by a particular assignment group, number of open vs. closed tickets, and so on for a particular business unit. We use row-level security to demonstrate different levels of access and how the same dataset changes for a super user like a president of a company overseeing multiple business units vs. a manager of a particular business or a direct report working on the actual tickets.
Solution overview
We extract and dump the LDAP dataset into Amazon Simple Storage Service (Amazon S3). This dataset can be a direct database dump. The LDAP dataset is usually not flattened and can’t be directly queried to find the reporting hierarchy of a particular employee. To flatten the directory structure, we load the data in an Amazon Relational Database Service (Amazon RDS) instance that supports recursive SQLs. Optionally, we can also flatten the LDAP data using AWS Glue with Apache Spark. The flattened LDAP data is written back into Amazon S3 and then loaded into Amazon Redshift or a QuickSight supported data store. This portion of loading to Amazon S3 and then to Amazon Redshift is optional if the QuickSight data source you’re using supports running recursive SQLs.
The flattened LDAP structure table should now consist of a user column, their manager, and who they report to, up to a vice president in the organization hierarchy. This is then joined with the ticketing system dataset and assignment group or business unit ownership table, which tells which manager or vice president is responsible for a particular business unit. The final joined and aggregated table provides ticketing metadata, ownership, and the business unit hierarchy for each of the tickets. This can be directly loaded into SPICE either as a query or a dataset. You can apply a permission file to this new dataset that dictates which group has access to which datasets.
The second component of the solution goes into the details of how to automate user management, which is done by uploading separate CSV files to Amazon S3 for adding new groups, adding new users, and removing users. When a particular object is uploaded, an event triggers an AWS Lambda function, which makes API calls for the QuickSight client to add or remove the users or add a group.
When you combine these two components, you get an automated way of incorporating your LDAP structure in QuickSight and managing the users in a similar fashion to how you manage users in your corporate active directory.
Prerequisites
To follow along with this post, clone the GitHub repo, deploy the infrastructure, and download the scripts.
Prepare the data
To get started, create a new S3 bucket or use an existing bucket to upload the provided scripts. On the Amazon S3 console, the prefix structure of the bucket should look like the following screenshot.
The CloudFormation template requires several parameters, including the following:
InboundTraffic – We recommend restricting access to IP addresses within your network or to the IP address of the client. Entering 0.0.0.0/0 allows incoming traffic from all IP addresses.
S3BucketForRedshiftIAMRole – The name of the S3 bucket where the scripts are uploaded. This is used to create an IAM role that is assumed by the cluster.
To perform the next steps, you can use the Amazon Redshift query editor. The scripts that need to be run are under the scripts folder. In the provided queries, replace bucket-name with the bucket where the scripts and data are uploaded, replace the iam_role ARN (arn:aws:iam::111122223333:role/quicksight-row-level-demo-redshift-role) with your actual account number, and replace the Region us-east-1 with the Region where the S3 bucket is created. When the queries are complete, several tables are now in the cluster (see the following screenshot).
Ticketing data
For this post, we assume that the IT support ticketing data is already loaded into Amazon Redshift. The important columns of interest in this dataset are ticket_no, which can be a primary key for this table, and assigned_user_email_id, which we use as a foreign key to join with the flattened LDAP dataset. We refer to this table as ticket_data. You can query this table in your Amazon Redshift cluster, as shown in the following screenshot.
In this post, we take the organizational structure where different business units are managed by different vice presidents.
We want the ability to categorize our ticketing data based on user assignment into business units. Our ticket data has assigned users for each ticket. We use assigned_user_email_id to identify owners for each of these tickets. Based on the LDAP dataset, we identify the vice president in the hierarchy for each assigned user. We also have an assignment group and assignment_group_manager_email mapping in a separate table, which gives us relationships between business units (or assignment group) and the respective vice presidents.
LDAP data
Our primary key for this data in this example dataset is user_email_id, and reports_to is the foreign key that refers to user_email_id in the same dataset. The following screenshot shows the employees table in an RDS instance. This table can also exist in any database that supports recursive queries.
We’re interested in taking a particular user, finding their manager, and moving up the hierarchy until we find the vice president of a particular business unit. For example, in the preceding LDAP table, Todd V reports to Jimmy P, Jimmy P reports to Steve W, and Steve W reports to President Thomas J. This hierarchy is difficult to query in the current table structure because Amazon Redshift doesn’t support recursive queries. As an alternative, we decided to load the data in an RDS instance that supports recursive queries first and then load it into Amazon Redshift.
After we load the data into the RDS instance, we can use a SQL query like the following to find the hierarchy order we discussed:
SELECT USER_EMAIL_ID, NAME, DESIGNATION, REPORTS_TO,
SYS_CONNECT_BY_PATH(USER_EMAIL_ID, '/') "Path",
SUBSTR(SYS_CONNECT_BY_PATH(USER_EMAIL_ID, '/'),
Instr(SYS_CONNECT_BY_PATH(USER_EMAIL_ID, '/'), '/', 2)-2) vp_roll_up
FROM employees
WHERE designation != 'Vice President'
START WITH designation = 'Vice President'
CONNECT BY PRIOR USER_EMAIL_ID = REPORTS_TO
ORDER BY REPORTS_TO;
The following screenshot shows the flattened sample LDAP dataset for the preceding example. Let’s call this LDAP_flattened_data. You can query this table in your Amazon Redshift cluster.
Note that for this post, the dataset is already flattened.
Assignment groups and manager mapping
The next step is to identify the business unit or assignment group each vice president belongs to.
For simplicity, we assume the following mapping between assignment groups (business units) and their respective vice presidents (or assignment group manager). This can be simply stored in a table. Let’s call the table assignment_group_manager_mapping. After we join LDAP_flattened_data with the assignment_group_manager_mapping table, we can identify which assignment group each user belongs to. Carrying forward the previous example, Todd V rolls up to vice president Steve W and therefore belongs to the IT Repairs assignment group. You can query this table in your Amazon Redshift cluster.
Now that we have the three base tables ready, all we need to do is join them to form a flattened table that is suitable for visualizations in QuickSight. We store this data into SPICE, which is an in-memory optimized calculation engine designed specifically to support low-latency, ad hoc data visualization. You can create this table by performing a join between ticket_data and LDAP_flattened_data on assigned_user_email_id and USER_EMAIL_ID, which gives us VP_ROLL_UP, which we can then use to join with the assignment_group_manager_mapping table on assignment_group_manager_email. The final dataset created from the sample data is shown in the following table.
For this post, we use the following sample SQL statement, but you can achieve this in multiple ways.
SELECT *
FROM ticket_data td, LDAP_flattened_data ld, assignment_group_manager_mapping ag
WHERE td.assigned_user_email_id = ld.USER_EMAIL_ID and
ld.VP_ROLL_UP = ag.assignment_group_manager_email;
You can directly load this table and query into SPICE from Amazon S3, Amazon Athena, Amazon Redshift, or any other supported data sources. For more information, see Working with Datasets.
With the preceding examples, the users that are added to a particular assignment group get the same permissions as the group, and the users that belong to the superuser group have elevated access. In QuickSight, we perform the following steps.
Create a new dataset with the row_level_security table.
Both datasets are visible in QuickSight.
In the details for the ticketing-data-final dataset, and choose Row-level security.
Select the row_level_security dataset that we created.
Select Apply dataset and confirm.
When the row-level security has been successfully applied, the lock icon is visible next to the ticketing dataset.
Automate user management
To tie it all together, we automate user and group management. When a particular user is added to a group, they get access as defined by the preceding permissions file. For this setup, we use the following Amazon S3 prefix structure, essentially a separate path for adding groups, adding users, and removing users.
We upload CSV files under each of these Amazon S3 paths.
Filters – created_date should be current month and status is closed
When user Todd V (the QuickSight user name is the same as the users email ID) logs in, he sees a dashboard like the following screenshot.
When Steve W logs in, his dashboard shows more information.
When a superuser logs in, they see the following dashboard.
Conclusion
We demonstrated one of the many ways we can use LDAP data for organizational hierarchy-based visualizations in QuickSight. In this post, we talked about how we can find organizational ownership for ticket data. You can further generalize this solution to fit any data within an organization that needs business unit-based grouping. Another use case for this can be for visualizing security threats or sales data across different business units.
This post covers only one level of organizational hierarchy, but in many enterprises, the organizational structure can be much more complicated. You can use a similar approach to deal with these nested organizational hierarchies, where we can report on different levels of business units. If you are interested in implementing row-level security using organizational LDAP hierarchical structure refer to Implement row-level security using a complete LDAP hierarchical organization structure in Amazon QuickSight.
About the Author
Anand Sakhare is a Big Data Architect with AWS. He helps customers build big data, analytics, and machine learning capabilities using a mix of technologies. He is passionate about innovation and solving complex problems.
Rohan Jamadagni is a Sr. Data Architect, working with AWS for the past 5 years. He works closely with customers to implement data and analytics solutions on AWS. He enjoys understanding the meaning behind data and helping customers visualize their data to provide meaningful insights.
Umair Nawaz is a DevOps Engineer at Amazon Web Services in New York City. He works on building secure architectures and advises enterprises on agile software delivery. He is motivated to solve problems strategically by utilizing modern technologies.
In this blog post, I demonstrate how to implement service-to-service authorization using OAuth 2.0 access tokens for microservice APIs hosted on Amazon Elastic Kubernetes Service (Amazon EKS). A common use case for OAuth 2.0 access tokens is to facilitate user authorization to a public facing application. Access tokens can also be used to identify and authorize programmatic access to services with a system identity instead of a user identity. In service-to-service authorization, OAuth 2.0 access tokens can be used to help protect your microservice API for the entire development lifecycle and for every application layer. AWS Well Architected recommends that you validate security at all layers, and by incorporating access tokens validated by the microservice, you can minimize the potential impact if your application gateway allows unintended access. The solution sample application in this post includes access token security at the outset. Access tokens are validated in unit tests, local deployment, and remote cluster deployment on Amazon EKS. Amazon Cognito is used as the OAuth 2.0 token issuer.
Benefits of using access token security with microservice APIs
Some of the reasons you should consider using access token security with microservices include the following:
Access tokens provide production grade security for microservices in non-production environments, and are designed to ensure consistent authentication and authorization and protect the application developer from changes to security controls at a cluster level.
They enable service-to-service applications to identify the caller and their permissions.
Access tokens are short-lived credentials that expire, which makes them preferable to traditional API gateway long-lived API keys.
You get better system integration with a web or mobile interface, or application gateway, when you include token validation in the microservice at the outset.
Overview of solution
In the solution described in this post, the sample microservice API is deployed to Amazon EKS, with an Application Load Balancer (ALB) for incoming traffic. Figure 1 shows the application architecture on Amazon Web Services (AWS).
Figure 1: Application architecture
The application client shown in Figure 1 represents a service-to-service workflow on Amazon EKS, and shows the following three steps:
The application client requests an access token from the Amazon Cognito user pool token endpoint.
The access token is forwarded to the ALB endpoint over HTTPS when requesting the microservice API, in the bearer token authorization header. The ALB is configured to use IP Classless Inter-Domain Routing (CIDR) range filtering.
The microservice deployed to Amazon EKS validates the access token using JSON Web Key Sets (JWKS), and enforces the authorization claims.
Walkthrough
The walkthrough in this post has the following steps:
Amazon EKS cluster setup
Amazon Cognito configuration
Microservice OAuth 2.0 integration
Unit test the access token claims
Deployment of microservice on Amazon EKS
Integration tests for local and remote deployments
Prerequisites
For this walkthrough, you should have the following prerequisites in place:
A basic familiarity with Kotlin or Java, and frameworks such as Quarkus or Spring
A Unix terminal
Set up
Amazon EKS is the target for your microservices deployment in the sample application. Use the following steps to create an EKS cluster. If you already have an EKS cluster, you can skip to the next section: To set up the AWS Load Balancer Controller. The following example creates an EKS cluster in the Asia Pacific (Singapore) ap-southeast-1 AWS Region. Be sure to update the Region to use your value.
To create an EKS cluster with eksctl
In your Unix editor, create a file named eks-cluster-config.yaml, with the following cluster configuration:
Create the cluster by using the following eksctl command:
eksctl create cluster -f eks-cluster-config.yaml
Allow 10–15 minutes for the EKS control plane and managed nodes creation. eksctl will automatically add the cluster details in your kubeconfig for use with kubectl.
Validate your cluster node status as “ready” with the following command
kubectl get nodes
Create the demo namespace to host the sample application by using the following command:
kubectl create namespace demo
With the EKS cluster now up and running, there is one final setup step. The ALB for inbound HTTPS traffic is created by the AWS Load Balancer Controller directly from the EKS cluster using a Kubernetes Ingress resource.
To set up the AWS Load Balancer Controller
Follow the installation steps to deploy the AWS Load Balancer Controller to Amazon EKS.
An Ingress resource defines the ALB configuration. You customize the ALB by using annotations. Create a file named alb.yml, and add resource definition as follows, replacing the inbound IP CIDR with your values:
Deploy the Ingress resource with kubectl to create the ALB by using the following command:
kubectl apply -f alb.yml
After a few moments, you should see the ALB move from status provisioning to active, with an auto-generated public DNS name.
Validate the ALB DNS name and the ALB is in active status by using the following command:
kubectl -n demo describe ingress alb-ingress
To alias your host, in this case gateway.example.com with the ALB, create a Route 53 alias record. The remote API is now accessible using your Route 53 alias, for example: https://gateway.example.com/api/demo/*
The ALB that you created will only allow incoming HTTPS traffic on port 443, and restricts incoming traffic to known source IP addresses. If you want to share the ALB across multiple microservices, you can add the alb.ingress.kubernetes.io/group.name annotation. To help protect the application from common exploits, you should add an annotation to bind AWS Web Application Firewall (WAFv2) ACLs, including rate-limiting options for the microservice.
Configure the Amazon Cognito user pool
To manage the OAuth 2.0 client credential flow, you create an Amazon Cognito user pool. Use the following procedure to create the Amazon Cognito user pool in the console.
In the top-right corner of the page, choose Create a user pool.
Provide a name for your user pool, and choose Review defaults to save the name.
Review the user pool information and make any necessary changes. Scroll down and choose Create pool.
Note down your created Pool Id, because you will need this for the microservice configuration.
Next, to simulate the client in subsequent tests, you will create three app clients: one for read permission, one for write permission, and one for the microservice.
To create Amazon Cognito app clients
In the left navigation pane, under General settings, choose App clients.
On the right pane, choose Add an app client.
Enter the App client name as readClient.
Leave all other options unchanged.
Choose Create app client to save.
Choose Add another app client, and add an app client with the name writeClient, then repeat step 5 to save.
Choose Add another app client, and add an app client with the name microService. Clear Generate Client Secret, as this isn’t required for the microservice. Leave all other options unchanged. Repeat step 5 to save.
Note down the App client id created for the microService app client, because you will need it to configure the microservice.
You now have three app clients: readClient, writeClient, and microService.
With the read and write clients created, the next step is to create the permission scope (role), which will be subsequently assigned.
To create read and write permission scopes (roles) for use with the app clients
In the left navigation pane, under App integration, choose Resource servers.
On the right pane, choose Add a resource server.
Enter the name Gateway for the resource server.
For the Identifier enter your host name, in this case https://gateway.example.com.Figure 2 shows the resource identifier and custom scopes for read and write role permission.
Figure 2: Resource identifier and custom scopes
In the first row under Scopes, for Name enter demo.read, and for Description enter Demo Read role.
In the second row under Scopes, for Name enter demo.write, and for Description enter Demo Write role.
Choose Save changes.
You have now completed configuring the custom role scopes that will be bound to the app clients. To complete the app client configuration, you will now bind the role scopes and configure the OAuth2.0 flow.
To configure app clients for client credential flow
In the left navigation pane, under App Integration, select App client settings.
On the right pane, the first of three app clients will be visible.
Scroll to the readClient app client and make the following selections:
For Enabled Identity Providers, select Cognito User Pool.
Under OAuth 2.0, for Allowed OAuth Flows, select Client credentials.
Under OAuth 2.0, under Allowed Custom Scopes, select the demo.read scope.
Leave all other options blank.
Scroll to the writeClient app client and make the following selections:
For Enabled Identity Providers, select Cognito User Pool.
Under OAuth 2.0, for Allowed OAuth Flows, select Client credentials.
Under OAuth 2.0, under Allowed Custom Scopes, select the demo.write scope.
Leave all other options blank.
Scroll to the microService app client and make the following selections:
For Enabled Identity Providers, select Cognito User Pool.
Under OAuth 2.0, for Allowed OAuth Flows, select Client credentials.
Under OAuth 2.0, under Allowed Custom Scopes, select the demo.read scope.
Leave all other options blank.
Figure 3 shows the app client configured with the client credentials flow and custom scope—all other options remain blank
Figure 3: App client configuration
Your Amazon Cognito configuration is now complete. Next you will integrate the microservice with OAuth 2.0.
Microservice OAuth 2.0 integration
For the server-side microservice, you will use Quarkus with Kotlin. Quarkus is a cloud-native microservice framework with strong Kubernetes and AWS integration, for the Java Virtual Machine (JVM) and GraalVM. GraalVM native-image can be used to create native executables, for fast startup and low memory usage, which is important for microservice applications.
On the top left, you can modify the Group, Artifact and Build Tool to your preference, or accept the defaults.
In the Pick your extensions search box, select each of the following extensions:
RESTEasy JAX-RS
RESTEasy Jackson
Kubernetes
Container Image Jib
OpenID Connect
Choose Generate your application to download your application as a .zip file.
Quarkus permits low-code integration with an identity provider such as Amazon Cognito, and is configured by the project application.properties file.
To configure application properties to use the Amazon Cognito IDP
Edit the application.properties file in your quick start project:
src/main/resources/application.properties
Add the following properties, replacing the variables with your values. Use the cognito-pool-id and microservice App client id that you noted down when creating these Amazon Cognito resources in the previous sections, along with your Region.
The Kotlin code sample that follows verifies the authenticated principle by using the @Authenticated annotation filter, which performs JSON Web Key Set (JWKS) token validation. The JWKS details are cached, adding nominal latency to the application performance.
The access token claims are auto-filtered by the @RolesAllowed annotation for the custom scopes, read and write. The protected methods are illustrations of a microservice API and how to integrate this with one to two lines of code.
import io.quarkus.security.Authenticated
import javax.annotation.security.RolesAllowed
import javax.enterprise.context.RequestScoped
import javax.ws.rs.*
@Authenticated
@RequestScoped
@Path("/api/demo")
class DemoResource {
@GET
@Path("protectedRole/{name}")
@RolesAllowed("https://gateway.example.com/demo.read")
fun protectedRole(@PathParam(value = "name") name: String) = mapOf("protectedAPI" to "true", "paramName" to name)
@POST
@Path("protectedUpload")
@RolesAllowed("https://gateway.example.com/demo.write")
fun protectedDataUpload(values: Map<String, String>) = "Received: $values"
}
Unit test the access token claims
For the unit tests you will test three scenarios: unauthorized, forbidden, and ok. The @TestSecurity annotation injects an access token with the specified role claim using the Quarkus test security library. To include access token security in your unit test only requires one line of code, the @TestSecurity annotation, which is a strong reason to include access token security validation upfront in your development. The unit test code in the following example maps to the protectedRole method for the microservice via the uri /api/demo/protectedRole, with an additional path parameter sample-username to be returned by the method for confirmation.
Deploying the microservice to Amazon EKS is the same as deploying to any upstream Kubernetes-compliant installation. You declare your application resources in a manifest file, and you deploy a container image of your application to your container registry. You can do this in a similar low-code manner with the Quarkus Kubernetes extension, which automatically generates the Kubernetes deployment and service resources at build time. The Quarkus Container Image Jib extension to automatically build the container image and deploys the container image to Amazon Elastic Container Registry (ECR), without the need for a Dockerfile.
Amazon ECR setup
Your microservice container image created during the build process will be published to Amazon Elastic Container Registry (Amazon ECR) in the same Region as the target Amazon EKS cluster deployment. Container images are stored in a repository in Amazon ECR, and in the following example uses a convention for the repository name of project name and microservice name. The first command that follows creates the Amazon ECR repository to host the microservice container image, and the second command obtains login credentials to publish the container image to Amazon ECR.
To set up the application for Amazon ECR integration
In the AWS CLI, create an Amazon ECR repository by using the following command. Replace the project name variable with your parent project name, and replace the microservice name with the microservice name.
Obtain an ECR authorization token, by using your IAM principal with the following command. Replace the variables with your values for the AWS account ID and Region.
After the application re-build, you should now have a container image deployed to Amazon ECR in your region with the following name [project-group]/[project-name]. The Quarkus build will give an error if the push to Amazon ECR failed.
Now, you can deploy your application to Amazon EKS, with kubectl from the following build path:
kubectl apply -f build/kubernetes/kubernetes.yml
Integration tests for local and remote deployments
The following environment assumes a Unix shell: either MacOS, Linux, or Windows Subsystem for Linux (WSL 2).
How to obtain the access token from the token endpoint
Obtain the access token for the application client by using the Amazon Cognito OAuth 2.0 token endpoint, and export an environment variable for re-use. Replace the variables with your Amazon Cognito pool name, and AWS Region respectively.
To generate the client credentials in the required format, you need the Base64 representation of the app client client-id:client-secret. There are many tools online to help you generate a Base64 encoded string. Export the following environment variables, to avoid hard-coding in configuration or scripts.
You can use curl to post to the token endpoint, and obtain an access token for the read and write app client respectively. You can pass grant_type=client_credentials and the custom scopes as appropriate. If you pass an incorrect scope, you will receive an invalid_grant error. The Unix jq tool extracts the access token from the JSON string. If you do not have the jq tool installed, you can use your relevant package manager (such as apt-get, yum, or brew), to install using sudo [package manager] install jq.
The following shell commands obtain the access token associated with the read or write scope. The client credentials are used to authorize the generation of the access token. An environment variable stores the read or write access token for future use. Update the scope URL to your host, in this case gateway.example.com.
If the curl commands are successful, you should see the access tokens in the environment variables by using the following echo commands:
echo $access_token_read
echo $access_token_write
For more information or troubleshooting, see TOKEN Endpoint in the Amazon Cognito Developer Guide.
Test scope with automation script
Now that you have saved the read and write access tokens, you can test the API. The endpoint can be local or on a remote cluster. The process is the same, all that changes is the target URL. The simplicity of toggling the target URL between local and remote is one of the reasons why access token security can be integrated into the full development lifecycle.
To perform integration tests in bulk, use a shell script that validates the response code. The example script that follows validates the API call under three test scenarios, the same as the unit tests:
If no valid access token is passed: 401 (unauthorized) response is expected.
A valid access token is passed, but with an incorrect role claim: 403 (forbidden) response is expected.
A valid access token and valid role-claim is passed: 200 (ok) response with content-type of application/json expected.
Name the following script, demo-api.sh. For each API method in the microservice, you duplicate these three tests, but for the sake of brevity in this post, I’m only showing you one API method here, protectedRole.
Test the microservice API against the access token claims
Run the script for a local host deployment on http://localhost:8080, and on the remote EKS cluster, in this case https://gateway.example.com.
If everything works as expected, you will have demonstrated the same test process for local and remote deployments of your microservice. Another advantage of creating a security test automation process like the one demonstrated, is that you can also include it as part of your continuous integration/continuous delivery (CI/CD) test automation.
The test automation script accepts the microservice host URL as a parameter (the default is local), referencing the stored access tokens from the environment variables. Upon error, the script will exit with the error code. To test the remote EKS cluster, use the following command, with your host URL, in this case gateway.example.com.
./demo-api.sh https://<gateway.example.com>
Expected output:
Test 401-unauthorized: No access token for /api/demo/protectedRole/{name}
Test 403-forbidden: Incorrect role/custom-scope for /api/demo/protectedRole/{name}
Test 200-ok: Correct role for /api/demo/protectedRole/{name}
Output: {"protectedAPI":"true","paramName":"sample-username"}!200!application/json
Best practices for a well architected production service-to-service client
For elevated security in alignment with AWS Well Architected, it is recommend to use AWS Secrets Manager to hold the client credentials. Separating your credentials from the application permits credential rotation without the requirement to release a new version of the application or modify environment variables used by the service. Access to secrets must be tightly controlled because the secrets contain extremely sensitive information. Secrets Manager uses AWS Identity and Access Management (IAM) to secure access to the secrets. By using the permissions capabilities of IAM permissions policies, you can control which users or services have access to your secrets. Secrets Manager uses envelope encryption with AWS KMS customer master keys (CMKs) and data key to protect each secret value. When you create a secret, you can choose any symmetric customer managed CMK in the AWS account and Region, or you can use the AWS managed CMK for Secrets Manager aws/secretsmanager.
Access tokens can be configured on Amazon Cognito to expire in as little as 5 minutes or as long as 24 hours. To avoid unnecessary calls to the token endpoint, the application client should cache the access token and refresh close to expiry. In the Quarkus framework used for the microservice, this can be automatically performed for a client service by adding the quarkus-oidc-client extension to the application.
Cleaning up
To avoid incurring future charges, delete all the resources created.
To delete the EKS cluster, and the associated Application Load Balancer, follow the steps described in Deleting a cluster in the Amazon EKS User Guide.
This post has focused on the last line of defense, the microservice, and the importance of a layered security approach throughout the development lifecycle. Access token security should be validated both at the application gateway and microservice for end-to-end API protection.
As an additional layer of security at the application gateway, you should consider using Amazon API Gateway, and the inbuilt JWT authorizer to perform the same API access token validation for public facing APIs. For more advanced business-to-business solutions, Amazon API Gateway provides integrated mutual TLS authentication.
If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, start a new thread on the Amazon Cognito forum or contact AWS Support.
Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.
През 2016-та и 2019-та направих карти проследяващи активните българи в чужбина. Повече по темата ще намерите в първата ми статия по темата. Накратко казано това е опит за картографиране на придвижването на хора съдейки по заявленията им за гласуване в чужбина. Последните, за съжаление, са публични и макар трите имена на някои да са често срещани, лесно може да се изключат или да се разпознаят дубликати. За целта използвах криптирани и анонимизирани хешове на имената, които събирах за картата ми със заявленията за гласуване.
Така през 2016-та бях картографирал движението на 1268 лица и през 2019-та – на 2633. Пускайки същия алгоритъм през юли 2021 събрах пътя на 9473 души. Голямата разлика идва от рекордната активност на последните два вота. Тук може да проследите тези пътища. Натискайки някой град ще откриете кой си е тръгвал и от къде е минавал.
За разлика от миналия път, сега анализа не отчита гласуване между два града на разстояние 80 км. вместо предишните 50 км. Важно е тук, че не се отчита реално местене на хора, а само там, където са подавали заявления за гласуване. Доколкото последното е някаква индикация за местоположение, дали всъщност става дума за местене, почивка, грешка или просто подкрепа на секции в други държави може само да гадаем. Все пак с тези мащаби с добра доза увереност може да кажем, че виждаме пътищата на почти десет хиляди българи по света.
Interactive applications need to process requests and respond very quickly, and this requirement extends to all the components of their architecture. That is even more important when you adopt microservices and your architecture is composed of many small independent services that communicate with each other.
For this reason, database performance is critical to the success of applications. To reduce read latency to microseconds, you can put an in-memory cache in front of a durable database. For caching, many developers use Redis, an open-source in-memory data structure store. In fact, according to Stack Overflow’s 2021 Developer Survey, Redis has been the most loved database for five years.
To implement this setup on AWS, you can use Amazon ElastiCache for Redis, a fully managed in-memory caching service, as a low latency cache in front of a durable database service such as Amazon Aurora or Amazon DynamoDB to minimize data loss. However, this setup requires you to introduce custom code in your applications to keep the cache in sync with the database. You’ll also incur costs for running both a cache and a database.
Introducing Amazon MemoryDB for Redis Today, I am excited to announce the general availability of Amazon MemoryDB for Redis, a new Redis-compatible, durable, in-memory database. MemoryDB makes it easy and cost-effective to build applications that require microsecond read and single-digit millisecond write performance with data durability and high availability.
Instead of using a low-latency cache in front of a durable database, you can now simplify your architecture and use MemoryDB as a single, primary database. With MemoryDB, all your data is stored in memory, enabling low latency and high throughput data access. MemoryDB uses a distributed transactional log that stores data across multiple Availability Zones (AZs) to enable fast failover, database recovery, and node restarts with high durability.
MemoryDB maintains compatibility with open-source Redis and supports the same set of Redis data types, parameters, and commands that you are familiar with. This means that the code, applications, drivers, and tools you already use today with open-source Redis can be used with MemoryDB. As a developer, you get immediate access to many data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs, geospatial indexes, and streams. You also get access to advanced features such as built-in replication, least recently used (LRU) eviction, transactions, and automatic partitioning. MemoryDB is compatible with Redis 6.2 and will support newer versions as they are released in open source.
One question you might have at this point is how MemoryDB compares to ElastiCache because both services give access to Redis data structures and API:
MemoryDB can safely be the primary database for your applications because it provides data durability and microsecond read and single-digit millisecond write latencies. With MemoryDB, you don’t need to add a cache in front of the database to achieve the low latency you need for your interactive applications and microservices architectures.
On the other hand, ElastiCache provides microsecond latencies for both reads and writes. It is ideal for caching workloads where you want to accelerate data access from your existing databases. ElastiCache can also be used as a primary datastore for use cases where data loss might be acceptable (for example, because you can quickly rebuild the database from another source).
Creating an Amazon MemoryDB Cluster In the MemoryDB console, I follow the link on the left navigation pane to the Clusters section and choose Create cluster. This opens Cluster settings where I enter a name and a description for the cluster.
All MemoryDB clusters run in a virtual private cloud (VPC). In Subnet groups I create a subnet group by selecting one of my VPCs and providing a list of subnets that the cluster will use to distribute its nodes.
In Cluster settings, I can change the network port, the parameter group that controls the runtime properties of my nodes and clusters, the node type, the number of shards, and the number of replicas per shard. Data stored in the cluster is partitioned across shards. The number of shards and the number of replicas per shard determine the number of nodes in my cluster. Considering that for each shard there is a primary node plus the replicas, I expect this cluster to have eight nodes.
For Redis version compatibility, I choose 6.2. I leave all other options to their default and choose Next.
In the Security section of Advanced settings I add the default security group for the VPC I used for the subnet group and choose an access control list (ACL) that I created before. MemoryDB ACLs are based on Redis ACLs and provide user credentials and permissions to connect to the cluster.
In the Snapshot section, I leave the default to have MemoryDB automatically create a daily snapshot and select a retention period of 7 days.
For Maintenance, I leave the defaults and then choose Create. In this section I can also provide an Amazon Simple Notification Service (SNS) topic to be notified of important cluster events.
Using Amazon MemoryDB as Your Primary Database Managing customer data is a critical component of many business processes. To test the durability of my new Amazon MemoryDB cluster, I want to use it as a customer database. For simplicity, let’s build a simple microservice in Python that allows me to create, update, delete, and get one or all customer data from a Redis cluster using a REST API.
This is the requirements.txt file, which lists the Python modules required by the application:
redis-py-cluster
Flask
Flask-RESTful
The same code works with MemoryDB, ElastiCache, or any Redis Cluster database.
I start a Linux Amazon Elastic Compute Cloud (Amazon EC2) instance in the same VPC as the MemoryDB cluster. To be able to connect to the MemoryDB cluster, I assign the default security group. I also add another security group that gives me SSH access to the instance.
I copy the server.py and requirements.txt files onto the instance and then install the dependencies:
pip3 install --user -r requirements.txt
Now, I start the microservice:
python3 server.py
In another terminal connection, I use curl to create a customer in my database with an HTTP POST on the /customers resource:
One of the customers has just spent all his balance. I update the field with an HTTP PUT on the URL of the customer resource that includes the ID (/customers/<id>):
The code is updating the fields of the Redis Hash with the data of the request. In this case, it’s setting the balance to zero. I verify the update by getting the customer data by ID:
That’s the power of Redis! I was able to create the skeleton of a microservice with just a few lines of code. On top of that, MemoryDB gives me the durability and the high availability I need in production without the need to add another database in the backend.
Depending on my workload, I can scale my MemoryDB cluster horizontally, by adding or removing nodes, or vertically, by moving to larger or smaller node types. MemoryDB supports write scaling with sharding and read scaling by adding replicas. My cluster continues to stay online and support read and write operations during resizing operations.
Availability and Pricing Amazon MemoryDB for Redis is available today in US East (N. Virginia), EU (Ireland), Asia Pacific (Mumbai), and South America (Sao Paulo) with more AWS Regions coming soon.
With MemoryDB, you pay for what you use based on on-demand instance hours per node, volume of data written to your cluster, and snapshot storage. For more information, see the MemoryDB pricing page.
Learn More Check out the video below for a quick overview.
By continuing to use the site, you agree to the use of cookies. more information
The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.