Estimating scoring probabilities by preparing soccer matches data with AWS Glue DataBrew

Post Syndicated from Arash Rowshan original https://aws.amazon.com/blogs/big-data/estimating-scoring-probabilities-by-preparing-soccer-matches-data-with-aws-glue-databrew/

In soccer (or football outside of the US), players decide to take shots when they think they can score. But how do they make that determination vs. when to pass or dribble? In a fraction of a second, in motion, while chased from multiple directions by other professional athletes, they think about their distance from the goal, the speed they’re running, the ball movement, the number of defenders, the goalkeeper’s position, their own shot power, accuracy, angle, and more. For a moment, time stands still. A decision is made. To shoot or not to shoot.

This post was inspired by AWS’s collaboration with Germany’s Bundesliga, which lead to Match Fact xGoals. We use representative sample datasets and walk through using AWS Glue DataBrew to prepare data prior to training a model that can predict the probability of a player scoring at any given moment in the game.

We start with two datasets that have the information about the events in the game and the players. We describe how a problem in this domain could be framed to benefit from data preparation and how that can lead us to making predictions.

No prior knowledge is required to enjoy this post. However, basic familiarity with data preparation and machine learning (ML) is beneficial. By the end of this post, you should have a good sense regarding what DataBrew offers as well as how you can apply the approaches offered here to other use cases.

Sample datasets

Let’s assume we have collected a lot of data from video records of soccer matches. We extracted the following dataset by taking a snapshot of every shot taken in every match. For each shot, we also recorded if it resulted in a goal or if it did not. The dataset is fictionalized and not based on historic soccer games.

The dataset is fictionalized and not based on historic soccer games.

The following table summarizes what each column contains.

Column Name Description
event_id Unique identifier for each record
game_minute Minute of the game between 0 to approximately 90
player_id Unique identifier for each player
player_name Name of the player who took the shot
defenders Number of defenders between the player and the opponent’s goalkeeper
player_position [x, y] coordinate of the player taking the shot
player_angle Angle of the player taking the shot
player_speed Speed of the player at the moment taking the shot in km/h
goalkeeper_position [x, y] coordinate of the opponent’s goalkeeper
situation Open play, free kick, or penalty kick
result Goal or no goal

We also use the FIFA 20 complete player dataset, a dataset on soccer players across the globe that attempts to estimate their qualities quantitatively. The following screenshot shows some of the columns this dataset contains.

The following screenshot shows some of the columns this dataset contains.

We’re interested in the following columns:

  • age
  • height
  • weight
  • overall
  • preferred_foot
  • weak_foot
  • pace
  • shooting
  • attacking_finishing
  • skill_fk_accuracy
  • movement_acceleration
  • movement_sprint_speed
  • power_shot_power
  • mentality_penalties 

We talk more about how each of these columns can be relevant later in this post.

As you can imagine, these datasets aren’t quite ready to train an ML model. We need to combine, clean, and extract values in numeric forms so we can start creating our training set. Normally that can take a lot of time and is somewhat tedious work. Data scientists prefer to focus on training their model, but they have to spend most of their time wrangling data to the shape that can be used for their ML flow.

But fear not! Enter AWS Glue DataBrew, your friendly neighborhood visual data prep tool. As businesswire puts it, “

“…AWS Glue DataBrew offers customers over 250 pre-built transformations to automate data preparation tasks that would otherwise require days or weeks writing hand-coded transformations.”

Let’s get to work and see how we can prepare a dataset for training an ML model using DataBrew.

Preparing the data with DataBrew

Let’s start with the shots dataset. I first create a project on the DataBrew console and upload this dataset.

I first create a project on the DataBrew console and upload this dataset.

When my session is ready, I should see all the columns and some analytics that give me useful insights about my dataset.

When my session is ready, I should see all the columns and some analytics that give me useful insights about my dataset.

The player_position column is recorded as [x,y] coordinates with x between 0–1000 and y between 0–500. The following image shows how a player is positioned based on this data.

The following image shows how a player is positioned based on this data.

In CSV, this is recorded as a string and has two values. We want to extract the values and combine them into a single value that can be represented as a number. This can help our model draw better conclusions from this feature. We can start preparing our data by completing the following steps:

  1. Using the Remove values transform, I remove the opening and closing brackets from the entries of the player_position column.

Using the Remove values transform, I remove the opening and closing brackets from the entries of the player_position column

  1. I split this column by a comma to generate two columns that hold the x and y coordinates.

I split this column by a comma to generate two columns that hold the x and y coordinates.

  1. We also need to convert these newly generated columns from strings to numbers.

We also need to convert these newly generated columns from strings to numbers.

Now we can generate a single value using the two columns that we generated. Imagine we want to break down the whole soccer field into 1×1 squares. If we think about the field similar to rows and columns, we can calculate the number for each square as such: x * 500 + y.

The benefit of this approach is that squares with higher numbers tend to be closer to the opponent’s goal, and this single feature can help our model draw good correlations between a player’s location and event outcomes. Note that the squares in the following image aren’t drawn to scale.

Note that the squares in the following image aren’t drawn to scale.

We can calculate the square numbers by using the the Functions transform Multiply and then Sum.

  1. First I multiple the x coordinate by 500.

First I multiple the x coordinate by 500.

  1. Using the ADD function, I sum this generated column with the y coordinate.

Using the ADD function, I sum this generated column with the y coordinate.

  1. We apply the same transforms to the goal_keeper position to achieve a single number for that feature as well.

We apply the same transforms to the goal_keeper position to achieve a single number for that feature as well.

Next, let’s take a look at the player_angle column.

Next, let’s take a look at the player_angle column.

When positioned on the lower half of the field, a positive angle likely presents a better opportunity to score, and when on the top half, a negative angle faces the players more toward the goal. Another point to consider is the player’s strong or weak foot. The bottom half presents a wide angle for a left-footed player and the top half does so for a right-footed player. Angle in combination with strong foot can help our model draw good conclusions. We add the information on players’ strong or weak foot later in this post.

We have three different situations recorded in our dataset.

  1. We apply one-hot-encoding to this column to refactor the situations as 1 (True) or 0 (False) values suitable for training our model.

We apply one-hot-encoding to this column to refactor the situations as 1 (True) or 0 (False) values suitable for training our model.

  1. For the shots dataset, we change the results column to 0s and 1s using the custom flag transform.
  2. Because this is what we want to predict, I name the column y.

We apply one-hot-encoding to this column to refactor the situations as 1 (True) or 0 (False) values suitable for training our model.

We can enrich the data further by joining this dataset with the player dataset to take each player’s qualities into consideration. You may have noticed that we generated a lot of extra columns while applying the previous transforms. We can address that in one step while we do the join and clean things up a bit.

  1. When applying the join, we can connect the players dataset.

  1. I use an inner join and choose player_id as my key.
  2. I also only select the columns that I’m interested in.

You can select more or fewer columns depending on what features you want to feed into your model. For instance, I’m not selecting the player’s nationality, but you may want your model to take that into consideration. That’s really up to you, so feel free to play around with your options.

You can select more or fewer columns depending on what features you want to feed into your model.

  1. I deselect the extra columns from the shots dataset and only select the following from the players dataset:
    1. age
    2. overall
    3. preferred_foot
    4. weak_foot
    5. shooting
    6. attacking_finishing
    7. skill_fk_accuracy
    8. movement_acceleration
    9. movement_sprint_speed
    10. power_shot_power
    11. mentality_penalties 

We’re almost done. We just need to apply a few transforms to the newly added player columns.

  1. I one-hot-encode preferred foot.

I one-hot-encode preferred foot.

We can normalize some of the columns depending on the ML model that we want to run on this dataset afterwards. I want to train a basic logistic regression model.

  1. I use min-max normalization on most columns to scale values between 0–1.

Depending on your model, it may make more sense to center around 0, use a custom range, or apply z-score for your normalization.

  1. I also apply mean normalization to the player angle.

I also apply mean normalization to the player angle.

  1. Now that I have the normalized columns, I can delete all their source columns.

Now that I have the normalized columns, I can delete all their source columns.

  1. Lastly, I move the result column all the way to the end because this is my output column (what I intend to predict).

Lastly, I move the result column all the way to the end because this is my output column (what I intend to predict).

  1. Now we have a complete recipe and it’s time to run a job to apply the steps on the full dataset and generate an output file to use to train our model.

Now we have a complete recipe and it’s time to run a job to apply the steps on the full dataset and generate an output file to use to train our model.

Training the model

When the job is finished, I retrieve the output from my Amazon Simple Storage Service (Amazon S3) bucket. This rich dataset is now ready to be fed into a model. Logistic regression or Support Vector Machines (SVM) could be good candidates for our dataset. You could use Amazon SageMaker to train a model and generate a probability of scoring per event. The following screenshot shows a basic logistic regression model created using scikit-learn.

The following screenshot shows a basic logistic regression model created using scikit-learn.

We see an approximately 80% probability that this model correctly predicts a scoring opportunity. You may get even higher accuracy using SVM. Feel free to try those or edit one of your data preparation steps and see how it affects your model accuracy.

Conclusion

In this post, we started with some raw fictionalized data of soccer shots and players. We framed the problem based on our domain knowledge and the data available. We used DataBrew to rapidly and visually connect the dots and forge the original datasets into an enriched form that could be used to train an ML model.

I encourage you to apply the same methodology to a problem domain that interests you and see how DataBrew can speed up your workflow.


About the Author

Arash RowshanArash Rowshan is a Software Engineer at Amazon Web Services. He’s passionate about big data and applications of ML. Most recently he was part of the team that launched AWS Glue DataBrew. Any time he’s not at his desk, he’s likely playing soccer somewhere. You can follow him on Twitter @OSO_Arash.

 

 

Metasploit Wrap-Up

Post Syndicated from Sonny Gonzalez original https://blog.rapid7.com/2021/01/08/metasploit-wrap-up-93/

Struts2 Multi Eval OGNL RCE

Metasploit Wrap-Up

Our very own zeroSteiner added exploit/multi/http/struts2_multi_eval_ognl, which exploits Struts2 evaluating OGNL expressions in HTML attributes multiple times (CVE-2019-0230 and CVE-2020-17530). The CVE-2019-0230 OGNL chain for remote code execution requires a one-time chain to enable the RCE gadget, which is handled automatically by the module. The OGNL gadget chain for CVE-2020-17530 will echo the command output. Both chains use a simple mathematical expression to ensure that evaluation occurs. These vulnerabilities are application dependent, and the user does need to know which CVE they are targeting. Setting the NAME parameter appropriately and using the check method to ensure evaluation takes place inside an HTML attribute are key to successful exploitation.

JuicyPotato-like Windows privilege escalation exploit

Exploit module exploits/windows/local/bits_ntlm_token_impersonation was added by Metasploit contributor C4ssandre. It exploits BITS connecting to a local Windows Remote Management server (WinRM) at startup time. A fake WinRM server listening on port 5985 is started by a DLL loaded from a previous unprivileged meterpreter session. The fake server triggers BITS and then steals a SYSTEM token from the subsequent authentication request. The token is then used to start a new process and launch powershell.exe as the SYSTEM user. It downloads a malicious PowerShell script and executes it on a second local HTTP server, not writing any files to disk. The exploit is based on decoder’s PoC. It has been successfully tested on Windows 10 (10.0 Build 19041) 32 bits.

Pulse Connect Secure Gzip RCE

Metasploit contributor h00die added an exploit that targets Pulse Connect Secure server version 9.1R8 and earlier. The vulnerability was originally discovered by the NCC Group. It achieves authenticated remote code execution as root by uploading an encrypted config that contains an overwrite for a Perl template file. This module was made possible by rxwx, who shared the encryption code with the author. Admin credentials are required for successful root access. The module has been tested against server version 9.1R8.

New modules (8)

Enhancements and features

  • PR 14566 from zeroSteiner Module auxiliary/server/socks_proxy replaces modules/auxiliary/server/socks4a.rb and modules/auxiliary/server/socks5.rb.
  • PR 14538 from jmartin-r7 Improves Metasploit’s XML importer error messages when data is not Base64 encoded.
  • PR 14528 from zeroSteiner Clarifies Windows Meterpreter payloads description support of XP SP2 or newer.
  • PR 14522 from axxop Replaces the hardcoded default Shiro encryption key with a new datastore option that allows users to specify rememberMe cookie encryption key.
  • PR 14517 from timwr Changes the osx/x64/shell_reverse_tcp payload to be generated with Metasm and captures and sends STDERR to msfconsole.
  • PR 14509 from egypt This adds a Java target to the Apache Solr RCE exploit module and fixes several payload issues.
  • PR 14444 from dwelch-r7 Adds a couple of missing methods from the remote data services for adding and deleting routes.

Bugs fixed

  • PR 14589 from timwr Fixes a file download issue with the Android Meterpreter’s download command.
  • PR 14532 from bcoles Fixes a NoMethodError exception caused by the Msf::Post::Common mixin not being included in post/android/capture/screen.
  • PR 14530 from jmartin-r7 Fixes a failing test on macOS caused by IPv6 vs IPv4 result precedence.
  • PR 14475 from dwelch-r7 Fixes the EICAR canary check.
  • PR 14334 from Summus-git Fixes a x86 linux bind shell payloads socket closing bug.

Get it

As always, you can update to the latest Metasploit Framework with msfupdate
and you can get more details on the changes since the last blog post from
GitHub:

If you are a git user, you can clone the Metasploit Framework repo (master branch) for the latest.
To install fresh without using git, you can use the open-source-only Nightly Installers or the
binary installers (which also include the commercial edition).

[$] A possible step toward integrity measurement for Fedora

Post Syndicated from original https://lwn.net/Articles/842002/rss

The Fedora 34 release is planned
for April 20 — a plan that may well come to fruition, given that the
Fedora project appears to have abandoned its tradition of delayed
releases. As part of that schedule, any proposals for system-wide changes
were supposed to be posted by December 29. That has not stopped the
arrival of a
late proposal
to add file signatures to Fedora’s RPM packages, though.
This proposal, meant to support the use of the integrity measurement
architecture
(IMA) in Fedora, has not been met with universal acclaim.

Security updates for Friday

Post Syndicated from original https://lwn.net/Articles/842093/rss

Security updates have been issued by Debian (firefox-esr and libxstream-java), Fedora (awstats and dia), Mageia (c-ares, dash, and dovecot), openSUSE (dovecot23, gimp, kitty, and python-notebook), Oracle (kernel), SUSE (python-paramiko and tomcat), and Ubuntu (edk2, firefox, ghostscript, and openjpeg2).

What’s New in InsightAppSec and tCell: Q4 2020 in Review

Post Syndicated from Bria Grangard original https://blog.rapid7.com/2021/01/08/whats-new-in-insightappsec-and-tcell-q4-2020-in-review/

What’s New in InsightAppSec and tCell: Q4 2020 in Review

It’s crazy to believe 2020 has come to an end, and we’re sure we’re not alone in our excitement for 2021! Without a doubt, 2020 has presented some challenges for us all in the security world, as many companies quickly adopted a work-from-home model and pivoted from an in-store experience quickly to a digital one. This accelerated digital transformation encouraged us at Rapid7 to create new programs and think about how we can continuously improve the customer experience.

For application security in particular, we saw restaurants creating new apps to facilitate takeout and delivery orders, fast-growing platforms like Instacart and DoorDash developing internal apps to keep in touch with their employees, and small-business owners creating web apps to continue selling their products and services. With the rapid increase in web application development came the need to make sure these applications were as secure as possible.

Here at Rapid7, we view your application security program as a key component of your vulnerability risk management (VRM) program. Considering the challenges of 2020, we wanted to make sure we not only continued to support our existing customers through their challenges, but that we also provided new ways for our customers to get visibility into their application security program while helping them to scale with the pressures of 2020.

We’ve previously recapped some of our product enhancements from this year, such as this blog covering Q2 and this one covering Q3 for 2020, but now we will cover the highlights for Q4. Below, we’ll recap some of the new and exciting features we have released as a part of our application security portfolio (inclusive of our industry-leading testing and monitoring solutions).

Increase your visibility

We continue to hear the desire to gain more visibility into application security programs, which is why we have released:

New ‘All Apps’ report in InsightAppSec

What’s New in InsightAppSec and tCell: Q4 2020 in Review

The New “All Apps” report in InsightAppSec is now available for companies that are looking to get a single view into risk activity across all of their applications and communicate this up to their leadership teams. Want to check it out? Click here to see how you can create your own All Apps report in InsightAppSec today!

New joint ‘All Apps’ and ‘All Assets’ report (between InsightAppSec and InsightVM)

What’s New in InsightAppSec and tCell: Q4 2020 in Review

Are you currently using InsightAppSec and InsightVM and looking for a view into the risk across your vulnerability risk management portfolio? Check out this new joint report, where you can get a single view into your full-stack vulnerability risk management activity across both InsightVM and InsightAppSec. You can find more information about this here!

Scale up with ease

While visibility is a key component to a successful VRM program, many teams were challenged this year with the need to scale their application security programs and activities. We wanted to make it easier on these teams, so we released the following features to help security teams save time and effort when it came to these scaling activities:

Application tagging in InsightAppSec

What’s New in InsightAppSec and tCell: Q4 2020 in Review

You can now easily create and manage tags across one or multiple apps based on what matters to you, such as criticality, tech stack, environment, or business unit. This helps you manage your application portfolio by filtering both apps and vulnerabilities based on these tags.

New pages in InsightAppSec

What’s New in InsightAppSec and tCell: Q4 2020 in Review

We have launched a new global schedule page that allows you to create and manage scan schedules and blackouts in a single view, and we have created a new manage files tab that saves you time when it comes to edits or updates that need to be made to macros for scan authentication (you can now download the macro file and make edits, rather than having to re-record the entire macro!).

What’s New in InsightAppSec and tCell: Q4 2020 in Review

tCell now available in Europe

We understand the importance of data sovereignty, which is why we wanted to make sure we made tCell available globally, with new data centers in Europe and new ones to be added in Q1 of 2021.

AppFirewall filter on IP CIDR ranges and Groups

Looking to reduce the noise and number of events in the AppFirewall dashboard in tCell? We have added filtering on IP Groups and CIDR ranges so you can get faster, more actionable insights.

Keep up with constant change

While we are only highlighting some of our updates above, we recognize application development is ever-changing and we want to be able to support our customers to build secure software. For that reason we wanted to share one more update with you from this quarter:

New Envoy agent in tCell

If you are currently (or looking to explore) leveraging the Envoy Proxy for your cloud-native apps, tCell now has a dedicated Envoy agent that plugs directly into the proxy layer to provide monitoring and protection capabilities for modern architectures. You can find more information on this here!

As always, many of our releases this quarter went through early access programs with our customers, and if you were one of our customers who participated and gave us feedback, we just want to take a moment to say thank you! We appreciate your feedback and always look for ways to incorporate it to make our solutions provide the maximum value to our customers. Want to participate in an on-going or upcoming early access program to have your voice heard on areas where we can continue to improve our products? Reach out to your CSM, who can tell you about ongoing early access programs and get you signed up!

Thank you for your loyalty and support through 2020. We look forward to 2021 and our continued partnership!

NEVER MISS A BLOG

Get the latest stories, expertise, and news about security today.

Smart Fairy Tale

Post Syndicated from Ashley Whittaker original https://www.raspberrypi.org/blog/smart-fairy-tale/

This is creepy, and we love it. OK, it’s not REALLY creepy, it’s just that some people have an aversion to dolls that appear to move of their own accord, due to a disturbing childhood experience — but enough about me.

Smart Fairy Tale is a whimsical, unique community project created by Berlin-based installation artist Niklas Roy and interaction designer Felix Fisgus.

Using a smartphone app, viewers determine which way a ball travels through transparent pipes, and depending on which light barriers the ball interrupts on its journey, various toys are animated to tell different stories.

The server of the installation is a Raspberry Pi 4. Via its GPIO pins, it controls the track switches and releases the ball.

Raspberry Pi 4 mounted onto plastic with the installation's servo and all the microcontrollers
Raspberry Pi 4 tucked in the top right-hand corner, mounted together with the router. Photo courtesy of Niklas’ project page

The apparatus is full of toys donated by residents of Wolfsburg, Germany. The artists wanted local people to not only be able to operate the mechanical piece, but also to have a hand in creating it. Each animatronic toy is made as a separate module, controlled by its own Arduino Nano.

Smart Fairy Tale can be remotely controlled by viewers who want to check in on the toys they gifted to the installation, and by any other curious people elsewhere in the world.

A phone using the app to control the installation. The installation is out of focus in the background
The app in action. Photo from Felix’s project page.

Better yet, the stories the toys tell were devised by local school students. The artists showed the gifted toys to a few elementary school classes, and the students drew several stories featuring toys they liked. The makers then programmed the toys to match what the drawings said they could do. A servo here, a couple of LEDs there, and the students’ stories were brought to life.

Some drawings local children made suggesting storylines for each of the gifted toys
Some of the storylines drawn by local children. Photo courtesy of Felix’s project page.

So what kind of stories did Wolfsburg’s finest come up with? One of the creators explains:

“There were a lot of scenes to interpret, like the blow-up love story, the chemtrail conspiracy, and the fossil fuel disaster, which culminates in a major traffic jam. The latter one even involved a laboratory for breeding synthetic dinosaurs by the use of renewable energies.”

Felix Fisgus

We LOVE it. Don’t tell me this isn’t creepy though…

WHY DO YOU HAUNT MY DREAMS???

You’ll find tonnes of extra technical specs and images in the project posts on both Felix and Niklas‘ websites.

The post Smart Fairy Tale appeared first on Raspberry Pi.

Thank you 2020: Results, Achievements, and Plans

Post Syndicated from Jekaterina Petruhina original https://blog.zabbix.com/thank-you-2020-results-achievements-and-plans/13184/

2020 was not an easy year, but it challenged and taught us a lot. Let’s sum up the past year’s results together and make some plans for the next one.

Going online

Despite the worldwide lockdown and the entire team’s inability to work from the office, Zabbix continued to evolve and get better. The Zabbix 5.0 release was the first release to be done remotely – without the usual cake and “family” celebration on release day in our cozy office kitchen. Version 5.2 also followed remotely, with the already familiar online celebration in Zoom.

Unexpectedly we had to change the vector of planned activities. In January 2020, we expected to hold ten conferences in different countries of the world. But in reality, due to the pandemic and the restrictions it caused, we were only able to have a few: Zabbix Conference Benelux 2020, Zabbix Conference China 2020, and Zabbix Conference Japan 2020.

In 2020, we were going to have the biggest Zabbix Summit ever – the 10th-anniversary event. But plans changed, and instead, we held our first online Zabbix Summit. Given the unusual format of a traditional Zabbix Summit – it turned out very successful and proved the online version also could be sustainable.  We are grateful to all the speakers and guests who supported us and joined the event.

We can say that 2020 was all about online events. We start a good tradition to hold Zabbix meetups online and received positive feedback from the community. Thanks to our partners’ support, online meetups have been held not only in English and Russian but also in Czech, Italian, French, Portuguese, Spanish, German, and Polish. We will continue this tradition for sure, providing our users worldwide to learn Zabbix, share their experience, and meet co-thinkers. Remember that we always record the meetups for you to have access to presentations when it is required. Recordings are available on our website’s event section under each event.

Conquering the world

One of the most important events of 2020 for Zabbix was the opening of the office in Brazil. The software and professional services are now even more accessible to Latin American users thanks to language localization and the branch office’s closeness. We are happy to be part of one of Zabbix’s most active communities worldwide and being able to cooperate on a much greater level. We have expanded our Latin American branch with new Zabbix professionals, partners, and initiatives during last year.

We are also proud of making Zabbix more reachable for different language users – with our Latin American office, we opened Zabbix Spanish and Portuguese web page and thanks to our active partners – also German site. Now we are working on more languages to make our open source solution even more open.

As for the overall results, we can share with you some statistics

Our integration team has been very active over the past year and has released many useful templates. In total, their efforts resulted in 42 new templates and 20 new Webhook integrations.

Zabbix partnership network has also grown. Zabbix Partnership program creates a worldwide network of trustful and highly skilled companies ready to provide immediate technical assistance and become an accessible intermediary between you and the Zabbix team. This network has now grown by 40+ partners worldwide, resulting in around 200 partners for now.

In the past year, we rethought the professional training program and expanded it with extra training – one-day courses for in-depth study of one specific topic. These courses differ from the rest of the program by having no requirements. You can get extra knowledge about monitoring with Zabbix without Zabbix certification. Four additional training courses are available now, but we are working on the next topics to provide you.

Note that we have already published many training sessions for this year, so now is an excellent time to choose and book the courses and timing that suit you best. Don’t forget to use the filter – you can quickly sort courses by language, level, and location (online or on-site).

Speaking of statistics for last year, thanks to the Zabbix team and partners’ efforts, there were 203 training sessions in total. Altogether, 1,214 official Zabbix certificates were awarded to the training attendees who successfully mastered the material and passed the exam.

But education doesn’t end with training. Webinars are an excellent opportunity to learn different aspects of Zabbix for free. Note that we have also added the ability to watch recorded sessions at your convenience. And we also want to remind you not to forget to use the filter to navigate between sessions. We cover many topics in many different languages, and we want to make sure you don’t miss out on learning the material. Last year we hosted 245 webinar sessions (including 55 sessions held by Zabbix partners worldwide), and we hope it was beneficial for our community.

A sneak peek to 2021

So what can we expect from Zabbix in the new year of 2021? Lots of things! We are not going to stop and continue to work hard. The Zabbix software development roadmap is available on our website. Feel free to explore the features we are working on right now! We will continue to hold meetups, covering complex technical issues. As for the Zabbix Summit 2021, we are also planning to have it online in October 2021. The first experience was a success, so why not strengthen the achieved result. And as for other news, keep an eye on our updates here on the blog, social media, and newsletter. We have an exciting year ahead of us, so let’s make the most of it!

 

Russia’s SolarWinds Attack and Software Security

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2021/01/russias-solarwinds-attack-and-software-security.html

The information that is emerging about Russia’s extensive cyberintelligence operation against the United States and other countries should be increasingly alarming to the public. The magnitude of the hacking, now believed to have affected more than 250 federal agencies and businesses — ­primarily through a malicious update of the SolarWinds network management software — ­may have slipped under most people’s radar during the holiday season, but its implications are stunning.

According to a Washington Post report, this is a massive intelligence coup by Russia’s foreign intelligence service (SVR). And a massive security failure on the part of the United States is also to blame. Our insecure Internet infrastructure has become a critical national security risk­ — one that we need to take seriously and spend money to reduce.

President-elect Joe Biden’s initial response spoke of retaliation, but there really isn’t much the United States can do beyond what it already does. Cyberespionage is business as usual among countries and governments, and the United States is aggressively offensive in this regard. We benefit from the lack of norms in this area and are unlikely to push back too hard because we don’t want to limit our own offensive actions.

Biden took a more realistic tone last week when he spoke of the need to improve US defenses. The initial focus will likely be on how to clean the hackers out of our networks, why the National Security Agency and US Cyber Command failed to detect this intrusion and whether the 2-year-old Cybersecurity and Infrastructure Security Agency has the resources necessary to defend the United States against attacks of this caliber. These are important discussions to have, but we also need to address the economic incentives that led to SolarWinds being breached and how that insecure software ended up in so many critical US government networks.

Software has become incredibly complicated. Most of us almost don’t know all of the software running on our laptops and what it’s doing. We don’t know where it’s connecting to on the Internet­ — not even which countries it’s connecting to­ — and what data it’s sending. We typically don’t know what third party libraries are in the software we install. We don’t know what software any of our cloud services are running. And we’re rarely alone in our ignorance. Finding all of this out is incredibly difficult.

This is even more true for software that runs our large government networks, or even the Internet backbone. Government software comes from large companies, small suppliers, open source projects and everything in between. Obscure software packages can have hidden vulnerabilities that affect the security of these networks, and sometimes the entire Internet. Russia’s SVR leveraged one of those vulnerabilities when it gained access to SolarWinds’ update server, tricking thousands of customers into downloading a malicious software update that gave the Russians access to those networks.

The fundamental problem is one of economic incentives. The market rewards quick development of products. It rewards new features. It rewards spying on customers and users: collecting and selling individual data. The market does not reward security, safety or transparency. It doesn’t reward reliability past a bare minimum, and it doesn’t reward resilience at all.

This is what happened at SolarWinds. A New York Times report noted the company ignored basic security practices. It moved software development to Eastern Europe, where Russia has more influence and could potentially subvert programmers, because it’s cheaper.

Short-term profit was seemingly prioritized over product security.

Companies have the right to make decisions like this. The real question is why the US government bought such shoddy software for its critical networks. This is a problem that Biden can fix, and he needs to do so immediately.

The United States needs to improve government software procurement. Software is now critical to national security. Any system for acquiring software needs to evaluate the security of the software and the security practices of the company, in detail, to ensure they are sufficient to meet the security needs of the network they’re being installed in. Procurement contracts need to include security controls of the software development process. They need security attestations on the part of the vendors, with substantial penalties for misrepresentation or failure to comply. The government needs detailed best practices for government and other companies.

Some of the groundwork for an approach like this has already been laid by the federal government, which has sponsored the development of a “Software Bill of Materials” that would set out a process for software makers to identify the components used to assemble their software.

This scrutiny can’t end with purchase. These security requirements need to be monitored throughout the software’s life cycle, along with what software is being used in government networks.

None of this is cheap, and we should be prepared to pay substantially more for secure software. But there’s a benefit to these practices. If the government evaluations are public, along with the list of companies that meet them, all network buyers can benefit from them. The US government acting purely in the realm of procurement can improve the security of nongovernmental networks worldwide.

This is important, but it isn’t enough. We need to set minimum safety and security standards for all software: from the code in that Internet of Things appliance you just bought to the code running our critical national infrastructure. It’s all one network, and a vulnerability in your refrigerator’s software can be used to attack the national power grid.

The IOT Cybersecurity Improvement Act, signed into law last month, is a start in this direction.

The Biden administration should prioritize minimum security standards for all software sold in the United States, not just to the government but to everyone. Long gone are the days when we can let the software industry decide how much emphasis to place on security. Software security is now a matter of personal safety: whether it’s ensuring your car isn’t hacked over the Internet or that the national power grid isn’t hacked by the Russians.

This regulation is the only way to force companies to provide safety and security features for customers — just as legislation was necessary to mandate food safety measures and require auto manufacturers to install life-saving features such as seat belts and air bags. Smart regulations that incentivize innovation create a market for security features. And they improve security for everyone.

It’s true that creating software in this sort of regulatory environment is more expensive. But if we truly value our personal and national security, we need to be prepared to pay for it.

The truth is that we’re already paying for it. Today, software companies increase their profits by secretly pushing risk onto their customers. We pay the cost of insecure personal computers, just as the government is now paying the cost to clean up after the SolarWinds hack. Fixing this requires both transparency and regulation. And while the industry will resist both, they are essential for national security in our increasingly computer-dependent worlds.

This essay previously appeared on CNN.com.

Use AWS Secrets Manager to simplify the management of private certificates

Post Syndicated from Maitreya Ranganath original https://aws.amazon.com/blogs/security/use-aws-secrets-manager-to-simplify-the-management-of-private-certificates/

AWS Certificate Manager (ACM) lets you easily provision, manage, and deploy public and private Secure Sockets Layer/Transport Layer Security (SSL/TLS) certificates for use with Amazon Web Services (AWS) services and your internal connected resources. For private certificates, AWS Certificate Manager Private Certificate Authority (ACM PCA) can be used to create private CA hierarchies, including root and subordinate CAs, without the investment and maintenance costs of operating an on-premises CA. With these CAs, you can issue custom end-entity certificates or use the ACM defaults.

When you manage the lifecycle of certificates, it’s important to follow best practices. You can think of a certificate as an identity of a service you’re connecting to. You have to ensure that these identities are secure and up to date, ideally with the least amount of manual intervention. AWS Secrets Manager provides a mechanism for managing certificates, and other secrets, at scale. Specifically, you can configure secrets to automatically rotate on a scheduled basis by using pre-built or custom AWS Lambda functions, encrypt them by using AWS Key Management Service (AWS KMS) keys, and automatically retrieve or distribute them for use in applications and services across an AWS environment. This reduces the overhead of manually managing the deployment, creation, and secure storage of these certificates.

In this post, you’ll learn how to use Secrets Manager to manage and distribute certificates created by ACM PCA across AWS Regions and accounts.

We present two use cases in this blog post to demonstrate the difference between issuing private certificates with ACM and with ACM PCA. For the first use case, you will create a certificate by using the ACM defaults for private certificates. You will then deploy the ACM default certificate to an Amazon Elastic Compute Cloud (Amazon EC2) instance that is launched in the same account as the secret and private CA. In the second scenario, you will create a custom certificate by using ACM PCA templates and parameters. This custom certificate will be deployed to an EC2 instance in a different account to demonstrate cross-account sharing of secrets.

Solution overview

Figure 1 shows the architecture of our solution.

Figure 1: Solution architecture

Figure 1: Solution architecture

This architecture includes resources that you will create during the blog walkthrough and by using AWS CloudFormation templates. This architecture outlines how these services can be used in a multi-account environment. As shown in the diagram:

  1. You create a certificate authority (CA) in ACM PCA to generate end-entity certificates.
  2. In the account where the issuing CA was created, you create secrets in Secrets Manager.
    1. There are several required parameters that you must provide when creating secrets, based on whether you want to create an ACM or ACM PCA issued certificate. These parameters will be passed to our Lambda function to make sure that the certificate is generated and stored properly.
    2. The Lambda rotation function created by the CloudFormation template is attached when configuring secrets rotation. Initially, the function generates two Privacy-Enhanced Mail (PEM) encoded files containing the certificate and private key, based on the provided parameters, and stores those in the secret. Subsequent calls to the function are made when the secret needs to be rotated, and then the function stores the resulting Certificate PEM and Private Key PEM in the desired secret. The function is written using Python, the AWS SDK for Python (Boto3), and OpenSSL. The flow of the function follows the requirements for rotating secrets in Secrets Manager.
  3. The first CloudFormation template creates a Systems Manager Run Command document that can be invoked to install the certificate and private key from the secret on an Apache Server running on EC2 in Account A.
  4. The second CloudFormation template deploys the same Run Command document and EC2 environment in Account B.
    1. To make sure that the account has the ability to pull down the certificate and private key from Secrets Manager, you need to update the key policy in Account A to give Account B access to decrypt the secret.
    2. You also need to add a resource-based policy to the secret that gives Account B access to retrieve the secret from Account A.
    3. Once the proper access is set up in Account A, you can use the Run Command document to install the certificate and private key on the Apache Server.

In a multi-account scenario, it’s common to have a central or shared AWS account that owns the ACM PCA resource, while workloads that are deployed in other AWS accounts use certificates issued by the ACM PCA. This can be achieved in two ways:

  1. Secrets in Secrets Manager can be shared with other AWS accounts by using resource-based policies. Once shared, the secrets can be deployed to resources, such as EC2 instances.
  2. You can share the central ACM PCA with other AWS accounts by using AWS Resource Access Manager or ACM PCA resource-based policies. These two options allow the receiving AWS account or accounts to issue private certificates by using the shared ACM PCA. These issued certificates can then use Secrets Manager to manage the secret in the child account and leverage features like rotation.

We will focus on first case for sharing secrets.

Solution cost

The cost for running this solution comes from the following services:

  • AWS Certificate Manager Private Certificate Authority (ACM PCA)
    Referring to the pricing page for ACM PCA, this solution incurs a prorated monthly charge of $400 for each CA that is created. A CA can be deleted the same day it’s created, leading to a charge of around $13/day (400 * 12 / 365.25). In addition, there is a cost for issuing certificates using ACM PCA. For the first 1000 certificates, this cost is $0.75. For this demonstration, you only need two certificates, resulting in a total charge of $1.50 for issuing certificates using ACM PCA. In all, the use of ACM PCA in this solution results in a charge of $14.50.
  • Amazon EC2
    The CloudFormation templates create t2.micro instances that cost $0.0116/hour, if they’re not eligible for Free Tier.
  • Secrets Manager
    There is a 30-day free trial for Secrets Manager, which is initiated when the first secret is created. After the free trial has completed, it costs $0.40 per secret stored per month. You will use two secrets for this solution and can schedule these for deletion after seven days, resulting in a prorated charge of $0.20.
  • Lambda
    Lambda has a free usage tier that allows for 1 million free requests per month and 400,000 GB-seconds of compute time per month. This fits within the usage for this blog, making the cost $0.
  • AWS KMS
    A single key created by one of the CloudFormation templates costs $1/month. The first 20,000 requests to AWS KMS are free, which fits within the usage of the test environment. In a production scenario, AWS KMS would charge $0.03 per 10,000 requests involving this key.

There are no charges for Systems Manager Run Command.

See the “Clean up resources” section of this blog post to get information on how to delete the resources that you create for this environment.

Deploy the solution

Now we’ll walk through the steps to deploy the solution. The CloudFormation templates and Lambda function code can be found in the AWS GitHub repository.

Create a CA to issue certificates

First, you’ll create an ACM PCA to issue private certificates. A common practice we see with customers is using a subordinate CA in AWS that is used to issue end-entity certificates for applications and workloads in the cloud. This subordinate can either point to a root CA in ACM PCA that is maintained by a central team, or to an existing on-premises public key infrastructure (PKI). There are some considerations when creating a CA hierarchy in ACM.

For demonstration purposes, you need to create a CA that can issue end-entity certificates. If you have an existing PKI that you want to use, you can create a subordinate CA that is signed by an external CA that can issue certificates. Otherwise, you can create a root CA and begin building a PKI on AWS. During creation of the CA, make sure that ACM has permissions to automatically renew certificates, because this feature will be used in later steps.

You should have one or more private CAs in the ACM console, as shown in Figure 2.

Figure 2: A private CA in the ACM PCA console

Figure 2: A private CA in the ACM PCA console

You will use two CloudFormation templates for this architecture. The first is launched in the same account where your private CA lives, and the second is launched in a different account. The first template generates the following: a Lambda function used for Secrets Manager rotation, an AWS KMS key to encrypt secrets, and a Systems Manager Run Command document to install the certificate on an Apache Server running on EC2 in Amazon Virtual Private Cloud (Amazon VPC). The second template launches the same Systems Manager Run Command document and EC2 environment.

To deploy the resources for the first template, select the following Launch Stack button. Make sure you’re in the N. Virginia (us-east-1) Region.

Select the Launch Stack button to launch the template

The template takes a few minutes to launch.

Use case #1: Create and deploy an ACM certificate

For the first use case, you’ll create a certificate by using the ACM defaults for private certificates, and then deploy it.

Create a Secrets Manager secret

To begin, create your first secret in Secrets Manager. You will create these secrets in the console to see how the service can be set up and used, but all these actions can be done through the AWS Command Line Interface (AWS CLI) or AWS SDKs.

To create a secret

  1. Navigate to the Secrets Manager console.
  2. Choose Store a new secret.
  3. For the secret type, select Other type of secrets.
  4. The Lambda rotation function has a set of required parameters in the secret type depending on what kind of certificate needs to be generated.For this first secret, you’re going to create an ACM_ISSUED certificate. Provide the following parameters.

    Key Value
    CERTIFICATE_TYPE ACM_ISSUED
    CA_ARN The Amazon Resource Name (ARN) of your certificate-issuing CA in ACM PCA
    COMMON_NAME The end-entity name for your certificate (for example, server1.example)
    ENVIRONMENT TEST (You need this later on to test the renewal of certificates. If using this outside of the blog walkthrough, set it to something like DEV or PROD.)
  5. For Encryption key, select CAKey, and then choose Next.
  6. Give the secret a name and optionally add tags or a description. Choose Next.
  7. Select Enable automatic rotation and choose the Lambda function that starts with <CloudFormation Stack Name>-SecretsRotateFunction. Because you’re creating an ACM-issued certificate, the rotation will be handled 60 days before the certificate expires. The validity is set to 365 days, so any value higher than 305 would work. Choose Next.
  8. Review the configuration, and then choose Store.
  9. This will take you back to a list of your secrets, and you will see your new secret, as shown in Figure 3. Select the new secret.

    Figure 3: The new secret in the Secrets Manager console

    Figure 3: The new secret in the Secrets Manager console

  10. Choose Retrieve secret value to confirm that CERTIFICATE_PEM, PRIVATE_KEY_PEM, CERTIFICATE_CHAIN_PEM, and CERTIFICATE_ARN are set in the secret value.

You now have an ACM-issued certificate that can be deployed to an end entity.

Deploy to an end entity

For testing purposes, you will now deploy the certificate that you just created to an Apache Server.

To deploy the certificate to the Apache Server

  1. In a new tab, navigate to the Systems Manager console.
  2. Choose Documents at the bottom left, and then choose the Owned by me tab.
  3. Choose RunUpdateTLS.
  4. Choose Run command at the top right.
  5. Copy and paste the secret ARN from Secrets Manager and make sure there are no leading or trailing spaces.
  6. Select Choose instances manually, and then choose ApacheServer.
  7. Select CloudWatch output to track progress.
  8. Choose Run.

The certificate and private key are now installed on the server, and it has been restarted.

To verify that the certificate was installed

  1. Navigate to the EC2 console.
  2. In the dashboard, choose Running Instances.
  3. Select ApacheServer, and choose Connect.
  4. Select Session Manager, and choose Connect.
  5. When you’re logged in to the instance, enter the following command.
    openssl s_client -connect localhost:443 | openssl x509 -text -noout
    

    This will display the certificate that the server is using, along with other metadata like the certificate chain and validity period. For the validity period, note the Not Before and Not After dates and times, as shown in figure 4.

    Figure 4: Server certificate

    Figure 4: Server certificate

Now, test the rotation of the certificate manually. In a production scenario, this process would be automated by using maintenance windows. Maintenance windows allow for the least amount of disruption to the applications that are using certificates, because you can determine when the server will update its certificate.

To test the rotation of the certificate

  1. Navigate back to your secret in Secrets Manager.
  2. Choose Rotate secret immediately. Because you set the ENVIRONMENT key to TEST in the secret, this rotation will renew the certificate. When the key isn’t set to TEST, the rotation function pulls down the renewed certificate based on its rotation schedule, because ACM is managing the renewal for you. In a couple of minutes, you’ll receive an email from ACM stating that your certificate was rotated.
  3. Pull the renewed certificate down to the server, following the same steps that you used to deploy the certificate to the Apache Server.
  4. Follow the steps that you used to verify that the certificate was installed to make sure that the validity date and time has changed.

Use case #2: Create and deploy an ACM PCA certificate by using custom templates

Next, use the second CloudFormation template to create a certificate, issued by ACM PCA, which will be deployed to an Apache Server in a different account. Sign in to your other account and select the following Launch Stack button to launch the CloudFormation template.

Select the Launch Stack button to launch the template

This creates the same Run Command document you used previously, as well as the EC2 and Amazon VPC environment running an Apache Server. This template takes in a parameter for the KMS key ARN; this can be found in the first template’s output section, shown in figure 5.

Figure 5: CloudFormation outputs

Figure 5: CloudFormation outputs

While that’s completing, sign in to your original account so that you can create the new secret.

To create the new secret

  1. Follow the same steps you used to create a secret, but change the secret values passed in to the following.

    Key Value
    CA_ARN The ARN of your certificate-issuing CA in ACM PCA
    COMMON_NAME You can use any name you want, such as server2.example
    TEMPLATE_ARN

    For testing purposes, use arn:aws:acm-pca:::template/EndEntityCertificate/V1

    This template ARN determines what type of certificate is being created and your desired path length. For more information, see Understanding Certificate Templates.

    KEY_ALGORITHM TYPE_RSA
    (You can also use TYPE_DSA)
    KEY_SIZE 2048
    (You can also use 1024 or 4096)
    SIGNING_HASH sha256
    (You can also use sha384 or sha512)
    SIGNING_ALGORITHM RSA
    (You can also use ECDSA if the key type for your issuing CA is set to ECDSA P256 or ECDSA P384)
    CERTIFICATE_TYPE ACM_PCA_ISSUED
  2. Add the following resource policy during the name and description step. This gives your other account access to pull this secret down to install the certificate on its Apache Server.
    {
      "Version" : "2012-10-17",
      "Statement" : [ {
        "Effect" : "Allow",
        "Principal" : {
          "AWS" : "<ARN in output of second CloudFormation Template>"
        },
        "Action" : "secretsmanager:GetSecretValue",
        "Resource" : "*"
      } ]
    }
    

  3. Finish creating the secret.

After the secret has been created, the last thing you need to do is add permissions to the KMS key policy so that your other account can decrypt the secret when installing the certificate on your server.

To add AWS KMS permissions

  1. Navigate to the AWS KMS console, and choose CAKey.
  2. Next to the key policy name, choose Edit.
  3. For the Statement ID (SID) Allow use of the key, add the ARN of the EC2 instance role in the other account. This can be found in the CloudFormation templates as an output called ApacheServerInstanceRole, as shown in Figure 5. The Statement should look something like this:
    {
                "Sid": "Allow use of the key",
                "Effect": "Allow",
                "Principal": {
                    "AWS": [
                        "arn:aws:iam::<AccountID with CA>:role/<Apache Server Instance Role>",
                        "arn:aws:iam:<Second AccountID>:role/<Apache Server Instance Role>"
                    ]
                },
                "Action": [
                    "kms:Encrypt",
                    "kms:Decrypt",
                    "kms:ReEncrypt*",
                    "kms:GenerateDataKey*",
                    "kms:DescribeKey"
                ],
                "Resource": "*"
    }
    

Your second account now has permissions to pull down the secret and certificate to the Apache Server. Follow the same steps described in the earlier section, “Deploy to an end entity.” Test rotating the secret the same way, and make sure the validity period has changed. You may notice that you didn’t get an email notifying you of renewal. This is because the certificate isn’t issued by ACM.

In this demonstration, you may have noticed you didn’t create resources that pull down the secret in different Regions, just in different accounts. If you want to deploy certificates in different Regions from the one where you create the secret, the process is exactly the same as what we described here. You don’t need to do anything else to accomplish provisioning and deploying in different Regions.

Clean up resources

Finally, delete the resources you created in the earlier steps, in order to avoid additional charges described in the section, “Solution cost.”

To delete all the resources created:

  1. Navigate to the CloudFormation console in both accounts, and select the stack that you created.
  2. Choose Actions, and then choose Delete Stack. This will take a few minutes to complete.
  3. Navigate to the Secrets Manager console in the CA account, and select the secrets you created.
  4. Choose Actions, and then choose Delete secret. This won’t automatically delete the secret, because you need to set a waiting period that allows for the secret to be restored, if needed. The minimum time is 7 days.
  5. Navigate to the Certificate Manager console in the CA account.
  6. Select the certificates that were created as part of this blog walkthrough, choose Actions, and then choose Delete.
  7. Choose Private CAs.
  8. Select the subordinate CA you created at the beginning of this process, choose Actions, and then choose Disable.
  9. After the CA is disabled, choose Actions, and then Delete. Similar to the secrets, this doesn’t automatically delete the CA but marks it for deletion, and the CA can be recovered during the specified period. The minimum waiting period is also 7 days.

Conclusion

In this blog post, we demonstrated how you could use Secrets Manager to rotate, store, and distribute private certificates issued by ACM and ACM PCA to end entities. Secrets Manager uses AWS KMS to secure these secrets during storage and delivery. You can introduce additional automation for deploying the certificates by using Systems Manager Maintenance Windows. This allows you to define a schedule for when to deploy potentially disruptive changes to EC2 instances.

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, start a new thread on the AWS Secrets Manager forum or contact AWS Support.

Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.

Author

Maitreya Ranganath

Maitreya is an AWS Security Solutions Architect. He enjoys helping customers solve security and compliance challenges and architect scalable and cost-effective solutions on AWS.

Author

Blake Franzen

Blake is a Security Solutions Architect with AWS in Seattle. His passion is driving customers to a more secure AWS environment while ensuring they can innovate and move fast. Outside of work, he is an avid movie buff and enjoys recreational sports.

The collective thoughts of the interwebz

By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.

Close