Irish News has recognized Rapid7 in its Workplace and Employment Awards, where we’ve taken home the trophy for Best Company Connection. Reflecting on the past two years, this award recognizes the organization that best demonstrates how it has adapted its workplace well-being strategy to the challenges of remote working influenced by the COVID-19 pandemic. Specifically, this includes how the company has remained committed to providing excellent support to its staff throughout, and maintaining contact and connection with workers during periods of uncertainty and isolation.
Rapid7 has been part of Belfast’s booming technology scene since 2014 and is home to a growing team of engineers, developers, and customer advisors. From 2020 to 2022, the office population nearly doubled in size to support the increasing demand from customers around the world for streamlined and accessible cybersecurity solutions. Maintaining Rapid7’s commitment to the core values of “Be an Advocate,” “Never Done,” “Impact Together,” “Challenge Convention,” and “Bring You” was a critical focal point for our local leadership as they scaled their teams in the midst of an unprecedented global pandemic.
The judges were very impressed by Rapid7’s holistic response to this new way of working, and how the company recognised the importance of maintaining contact, culture, and connection during such unprecedented times. Programs that stood out included leadership engagement through weekly Town Halls, engagement with mental well-being experts, and several grassroots community initiatives, including an Academy group designed to support parents in homeschooling their children.
In addition to taking home the winning title, Rapid7 was also recognised as a finalist in two other categories this year: Best People Development Programme and Best Place to Work. Rapid7’s global commitment to its employees has been recognized in other recent designations, including the #1 spot on the Boston Business Journal Best Places to Work list in June and landing at #2 on Comparably’s list of Best Workplaces in Boston in March. Expanding our winning track record into the United Kingdom speaks to how we support employees in creating the career experience of a lifetime while positively impacting our customers and the greater cybersecurity community.
NEVER MISS A BLOG
Get the latest stories, expertise, and news about security today.
The CPU scheduler’s job has never been easy; it must find a way to allocate
CPU time to all tasks in the system that is fair, allows all tasks to
progress, and maximizes the throughput of the system as a whole. More
recently, it has been called upon to satisfy another constraint: minimizing
the system’s energy consumption. There is currently a
patch set in circulation, posted by Vincent Donnefort with work from
Dietmar Eggemann as well, that changes how this constraint is met. The
actual change is small, but it illustrates how hard it can be to get the
needed heuristics right.
Many individuals are creating their own ecommerce and online stores in order to sell their products and services. This simplifies and speeds the process of getting products out to your selected markets. This is a critical key indicator for the success of your business.
Artificial Intelligence/Machine Learning (AI/ML) and automation can offer you an improved and seamless process for image manipulation. You can take a picture identifying your products. You can then remove the background in order to publish high quality and clean product images. These images can be added to your online stores for consumers to view and purchase. This automated process will drastically decrease the manual effort required, though some manual quality review will be necessary. It will increase your time-to-market (TTM) and quickly get your products out to customers.
This blog post explains how you can automate the removal of image backgrounds by combining semantic segmentation inferences using Amazon SageMaker JumpStart. You can automate image processing using AWS Lambda. We will walk you through how you can set up an Amazon SageMaker JumpStart semantic segmentation inference endpoint using curated training data.
Amazon SageMaker JumpStart solution overview
Figure 1. Architecture for automatically processing new images and outputting isolated labels identified through semantic segmentation.
The example architecture in Figure 1 shows a serverless architecture that uses SageMaker to perform semantic segmentation on images. Image processing takes place within a Lambda function, which extracts the identified (product) content from the background content in the image.
In this event driven architecture, Amazon Simple Storage Service (Amazon S3) invokes a Lambda function each time a new product image lands in the Uploaded Image Bucket. That Lambda function calls out to a semantic segmentation endpoint in Amazon SageMaker. The function then receives a segmentation mask that identifies the pixels that are part of the segment we are identifying. Then, the Lambda function processes the image to isolate the identified segment from the rest of the image, outputting the result to our Processed Image Bucket.
Semantic segmentation model
The semantic segmentation algorithm provides a fine-grained, pixel-level approach to developing computer vision applications. It tags every pixel in an image with a class label from a predefined set of classes. Because the semantic segmentation algorithm classifies every pixel in an image, it also provides information about the shapes of the objects contained in the image. The segmentation output is represented as a grayscale image, called a segmentation mask. A segmentation mask is a grayscale image with the same shape as the input image.
You can use the segmentation mask and replace the pixels that correspond to the class that is identified with the pixels from the original image. You can use the Python library PIL to do pixel manipulation on the image. The following images show how the image in Figure 1 will result in the image shown in Figure 2, when passed through semantic segmentation. When you use the Figure 2 mask and replace it with pixels from Figure 1, the end result is the image from Figure 3. Due to minor quality issues of the final image, you will need to do manual cleanup after automation.
Figure 2. Car image with background
Figure 3. Car mask image
Figure 4. Final image, background removed
SageMaker JumpStart streamlines the deployment of the prebuilt model on SageMaker, which supports the semantic segmentation algorithm. You can test this using the sample Jupyter notebook available at Extract Image using Semantic Segmentation, which demonstrates how to extract an individual form from the surrounding background.
Learn more about SageMaker JumpStart
SageMaker JumpStart is a quick way to learn about SageMaker features and capabilities through curated one-step solutions, example notebooks, and deployable pre-trained models. You can also fine-tune the models and then deploy them. You can access JumpStart using Amazon SageMaker Studio or programmatically through the SageMaker APIs.
SageMaker JumpStart provides lot of different semantic segmentation models that are pre-trained with class of objects it can identify. These models are fine-tuned for a sample dataset. You can tune the model with your dataset to get an effective mask for the class of object you want to retrieve from the image. When you fine-tune a model, you can use the default dataset or choose your own data, which is located in an Amazon S3 bucket. You can customize the hyperparameters of the training job that are used to fine-tune the model.
When the fine-tuning process is complete, JumpStart provides information about the model: parent model, training job name, training job Amazon Resource Name (ARN), training time, and output path. We retrieve the deploy_image_uri, deploy_source_uri, and base_model_uri for the pre-trained model. You can host the pre-trained base-model by creating an instance of sagemaker.model.Model and deploying it.
Conclusion
In this blog, we review the steps to use Amazon SageMaker JumpStart and AWS Lambda for automation and processing of images. It uses pre-trained machine learning models and inference. The solution ingests the product images, identifies your products, and then removes the image background. After some review and QA, you can then publish your products to your ecommerce store or other medium.
Further resources:
For more hands-on experience with SageMaker, explore our SageMaker Workshop
Look through our available SageMaker JumpStarts (SageMaker JumpStarts include paths for data scientists, business analysts, and MLOps engineers)
Security updates have been issued by Debian (firefox-esr, isync, kernel, and systemd), Fedora (chromium, curl, firefox, golang-github-vultr-govultr-2, and xen), Mageia (openssl, python-bottle, and python-pyjwt), Red Hat (compat-openssl10, curl, expat, firefox, go-toolset-1.17 and go-toolset-1.17-golang, go-toolset:rhel8, kernel, kpatch-patch, libarchive, libgcrypt, libinput, libxml2, pcre2, php:7.4, php:8.0, qemu-kvm, ruby:2.6, thunderbird, and vim), and Ubuntu (curl, libjpeg6b, and vim).
Here at Cloudflare we’re constantly working on improving our service. Our engineers are looking at hundreds of parameters of our traffic, making sure that we get better all the time.
One of the core numbers we keep a close eye on is HTTP request latency, which is important for many of our products. We regard latency spikes as bugs to be fixed. One example is the 2017 story of “Why does one NGINX worker take all the load?”, where we optimized our TCP Accept queues to improve overall latency of TCP sockets waiting for accept().
Performance tuning is a holistic endeavor, and we monitor and continuously improve a range of other performance metrics as well, including throughput. Sometimes, tradeoffs have to be made. Such a case occurred in 2015, when a latency spike was discovered in our processing of HTTP requests. The solution at the time was to set tcp_rmem to 4 MiB, which minimizes the amount of time the kernel spends on TCP collapse processing. It was this collapse processing that was causing the latency spikes. Later in this post we discuss TCP collapse processing in more detail.
The tradeoff is that using a low value for tcp_rmem limits TCP throughput over high latency links. The following graph shows the maximum throughput as a function of network latency for a window size of 2 MiB. Note that the 2 MiB corresponds to a tcp_rmem value of 4 MiB due to the tcp_adv_win_scale setting in effect at the time.
For the Cloudflare products then in existence, this was not a major problem, as connections terminate and content is served from nearby servers due to our BGP anycast routing.
Since then, we have added new products, such as Magic WAN, WARP, Spectrum, Gateway, and others. These represent new types of use cases and traffic flows.
For example, imagine you’re a typical Magic WAN customer. You have connected all of your worldwide offices together using the Cloudflare global network. While Time to First Byte still matters, Magic WAN office-to-office traffic also needs good throughput. For example, a lot of traffic over these corporate connections will be file sharing using protocols such as SMB. These are elephant flows over long fat networks. Throughput is the metric every eyeball watches as they are downloading files.
We need to continue to provide world-class low latency while simultaneously providing high throughput over high-latency connections.
Before we begin, let’s introduce the players in our game.
TCP receive window is the maximum number of unacknowledged user payload bytes the sender should transmit (bytes-in-flight) at any point in time. The size of the receive window can and does go up and down during the course of a TCP session. It is a mechanism whereby the receiver can tell the sender to stop sending if the sent packets cannot be successfully received because the receive buffers are full. It is this receive window that often limits throughput over high-latency networks.
net.ipv4.tcp_adv_win_scale is a (non-intuitive) number used to account for the overhead needed by Linux to process packets. The receive window is specified in terms of user payload bytes. Linux needs additional memory beyond that to track other data associated with packets it is processing.
The value of the receive window changes during the lifetime of a TCP session, depending on a number of factors. The maximum value that the receive window can be is limited by the amount of free memory available in the receive buffer, according to this table:
tcp_adv_win_scale
TCP window size
4
15/16 * available memory in receive buffer
3
⅞ * available memory in receive buffer
2
¾ * available memory in receive buffer
1
½ * available memory in receive buffer
0
available memory in receive buffer
-1
½ * available memory in receive buffer
-2
¼ * available memory in receive buffer
-3
⅛ * available memory in receive buffer
We can intuitively (and correctly) understand that the amount of available memory in the receive buffer is the difference between the used memory and the maximum limit. But what is the maximum size a receive buffer can be? The answer is sk_rcvbuf.
sk_rcvbuf is a per-socket field that specifies the maximum amount of memory that a receive buffer can allocate. This can be set programmatically with the socket option SO_RCVBUF. This can sometimes be useful to do, for localhost TCP sessions, for example, but in general the use of SO_RCVBUF is not recommended.
So how is sk_rcvbuf set? The most appropriate value for that depends on the latency of the TCP session and other factors. This makes it difficult for L7 applications to know how to set these values correctly, as they will be different for every TCP session. The solution to this problem is Linux autotuning.
Linux autotuning
Linux autotuning is logic in the Linux kernel that adjusts the buffer size limits and the receive window based on actual packet processing. It takes into consideration a number of things including TCP session RTT, L7 read rates, and the amount of available host memory.
Autotuning can sometimes seem mysterious, but it is actually fairly straightforward.
The central idea is that Linux can track the rate at which the local application is reading data off of the receive queue. It also knows the session RTT. Because Linux knows these things, it can automatically increase the buffers and receive window until it reaches the point at which the application layer or network bottleneck links are the constraint on throughput (and not host buffer settings). At the same time, autotuning prevents slow local readers from having excessively large receive queues. The way autotuning does that is by limiting the receive window and its corresponding receive buffer to an appropriate size for each socket.
The values set by autotuning can be seen via the Linux “ss” command from the iproute package (e.g. “ss -tmi”). The relevant output fields from that command are:
Recv-Q is the number of user payload bytes not yet read by the local application.
rcv_ssthresh is the window clamp, a.k.a. the maximum receive window size. This value is not known to the sender. The sender receives only the current window size, via the TCP header field. A closely-related field in the kernel, tp->window_clamp, is the maximum window size allowable based on the amount of available memory. rcv_sshthresh is the receiver-side slow-start threshold value.
skmem_r is the actual amount of memory that is allocated, which includes not only user payload (Recv-Q) but also additional memory needed by Linux to process the packet (packet metadata). This is known within the kernel as sk_rmem_alloc.
Note that there are other buffers associated with a socket, so skmem_r does not represent the total memory that a socket might have allocated. Those other buffers are not involved in the issues presented in this post.
skmem_rb is the maximum amount of memory that could be allocated by the socket for the receive buffer. This is higher than rcv_ssthresh to account for memory needed for packet processing that is not packet data. Autotuning can increase this value (up to tcp_rmem max) based on how fast the L7 application is able to read data from the socket and the RTT of the session. This is known within the kernel as sk_rcvbuf.
rcv_space is the high water mark of the rate of the local application reading from the receive buffer during any RTT. This is used internally within the kernel to adjust sk_rcvbuf.
Earlier we mentioned a setting called tcp_rmem. net.ipv4.tcp_rmem consists of three values, but in this document we are always referring to the third value (except where noted). It is a global setting that specifies the maximum amount of memory that any TCP receive buffer can allocate, i.e. the maximum permissible value that autotuning can use for sk_rcvbuf. This is essentially just a failsafe for autotuning, and under normal circumstances should play only a minor role in TCP memory management.
It’s worth mentioning that receive buffer memory is not preallocated. Memory is allocated based on actual packets arriving and sitting in the receive queue. It’s also important to realize that filling up a receive queue is not one of the criteria that autotuning uses to increase sk_rcvbuf. Indeed, preventing this type of excessive buffering (bufferbloat) is one of the benefits of autotuning.
What’s the problem?
The problem is that we must have a large TCP receive window for high BDP sessions. This is directly at odds with the latency spike problem mentioned above.
Something has to give. The laws of physics (speed of light in glass, etc.) dictate that we must use large window sizes. There is no way to get around that. So we are forced to solve the latency spikes differently.
A brief recap of the latency spike problem
Sometimes a TCP session will fill up its receive buffers. When that happens, the Linux kernel will attempt to reduce the amount of memory the receive queue is using by performing what amounts to a “defragmentation” of memory. This is called collapsing the queue. Collapsing the queue takes time, which is what drives up HTTP request latency.
We do not want to spend time collapsing TCP queues.
Why do receive queues fill up to the point where they hit the maximum memory limit? The usual situation is when the local application starts out reading data from the receive queue at one rate (triggering autotuning to raise the max receive window), followed by the local application slowing down its reading from the receive queue. This is valid behavior, and we need to handle it correctly.
Selecting sysctl values
Before exploring solutions, let’s first decide what we need as the maximum TCP window size.
As we have seen above in the discussion about BDP, the window size is determined based upon the RTT and desired throughput of the connection.
Because Linux autotuning will adjust correctly for sessions with lower RTTs and bottleneck links with lower throughput, all we need to be concerned about are the maximums.
For latency, we have chosen 300 ms as the maximum expected latency, as that is the measured latency between our Zurich and Sydney facilities. It seems reasonable enough as a worst-case latency under normal circumstances.
For throughput, although we have very fast and modern hardware on the Cloudflare global network, we don’t expect a single TCP session to saturate the hardware. We have arbitrarily chosen 3500 mbps as the highest supported throughput for our highest latency TCP sessions.
The calculation for those numbers results in a BDP of 131MB, which we round to the more aesthetic value of 128 MiB.
Recall that allocation of TCP memory includes metadata overhead in addition to packet data. The ratio of actual amount of memory allocated to user payload size varies, depending on NIC driver settings, packet size, and other factors. For full-sized packets on some of our hardware, we have measured average allocations up to 3 times the packet data size. In order to reduce the frequency of TCP collapse on our servers, we set tcp_adv_win_scale to -2. From the table above, we know that the max window size will be ¼ of the max buffer space.
A tcp_rmem of 512MiB and tcp_adv_win_scale of -2 results in a maximum window size that autotuning can set of 128 MiB, our desired value.
Disabling TCP collapse
Patient: Doctor, it hurts when we collapse the TCP receive queue.
Doctor: Then don’t do that!
Generally speaking, when a packet arrives at a buffer when the buffer is full, the packet gets dropped. In the case of these receive buffers, Linux tries to “save the packet” when the buffer is full by collapsing the receive queue. Frequently this is successful, but it is not guaranteed to be, and it takes time.
There are no problems created by immediately just dropping the packet instead of trying to save it. The receive queue is full anyway, so the local receiver application still has data to read. The sender’s congestion control will notice the drop and/or ZeroWindow and will respond appropriately. Everything will continue working as designed.
At present, there is no setting provided by Linux to disable the TCP collapse. We developed an in-house patch to the kernel to disable the TCP collapse logic.
Kernel patch – Attempt #1
The kernel patch for our first attempt was straightforward. At the top of tcp_try_rmem_schedule(), if the memory allocation fails, we simply return (after pred_flag = 0 and tcp_sack_reset()), thus completely skipping the tcp_collapse and related logic.
It didn’t work.
Although we eliminated the latency spikes while using large buffer limits, we did not observe the throughput we expected.
One of the realizations we made as we investigated the situation was that standard network benchmarking tools such as iperf3 and similar do not expose the problem we are trying to solve. iperf3 does not fill the receive queue. Linux autotuning does not open the TCP window large enough. Autotuning is working perfectly for our well-behaved benchmarking program.
We need application-layer software that is slightly less well-behaved, one that exercises the autotuning logic under test. So we wrote one.
A new benchmarking tool
Anomalies were seen during our “Attempt #1” that negatively impacted throughput. The anomalies were seen only under certain specific conditions, and we realized we needed a better benchmarking tool to detect and measure the performance impact of those anomalies.
This tool has turned into an invaluable resource during the development of this patch and raised confidence in our solution.
It consists of two Python programs. The reader opens a TCP session to the daemon, at which point the daemon starts sending user payload as fast as it can, and never stops sending.
The reader, on the other hand, starts and stops reading in a way to open up the TCP receive window wide open and then repeatedly causes the buffers to fill up completely. More specifically, the reader implemented this logic:
reads as fast as it can, for five seconds
this is called fast mode
opens up the window
calculates 5% of the high watermark of the bytes reader during any previous one second
for each second of the next 15 seconds:
this is called slow mode
reads that 5% number of bytes, then stops reading
sleeps for the remainder of that particular second
most of the second consists of no reading at all
steps 1-3 are repeated in a loop three times, so the entire run is 60 seconds
This has the effect of highlighting any issues in the handling of packets when the buffers repeatedly hit the limit.
Revisiting default Linux behavior
Taking a step back, let’s look at the default Linux behavior. The following is kernel v5.15.16.
The Linux kernel is effective at freeing up space in order to make room for incoming packets when the receive buffer memory limit is hit. As documented previously, the cost for saving these packets (i.e. not dropping them) is latency.
However, the latency spikes, in milliseconds, for tcp_try_rmem_schedule(), are:
These are the latency spikes we cannot have on the Cloudflare global network.
Kernel patch – Attempt #2
So the “something” that was not working in Attempt #1 was that the receive queue memory limit was hit early on as the flow was just ramping up (when the values for sk_rmem_alloc and sk_rcvbuf were small, ~800KB). This occurred at about the two second mark for 137p4 test (about 2.25 seconds for 170p2).
In hindsight, we should have noticed that tcp_prune_queue() actually raises sk_rcvbuf when it can. So we modified the patch in response to that, added a guard to allow the collapse to execute when sk_rmem_alloc is less than the threshold value.
net.ipv4.tcp_collapse_max_bytes = 6291456
The next section discusses how we arrived at this value for tcp_collapse_max_bytes.
The tcp_notsent_lowat setting is discussed in the last section of this post.
The middle value of tcp_rmem was changed as a result of separate work that found that Linux autotuning was setting receive buffers too high for localhost sessions. This updated setting reduces TCP memory usage for those sessions, but does not change anything about the type of TCP sessions that is the focus of this post.
For the following benchmarks, we used non-Cloudflare host machines in Iowa, US, and Melbourne, Australia performing data transfers to the Cloudflare data center in Marseille, France. In Marseille, we have some hosts configured with the existing production settings, and others with the system settings described in this post. Software used is perf3 version 3.9, kernel 5.15.32.
Throughput results
RTT (ms)
Throughput with Current Settings (mbps)
Throughput with New Settings (mbps)
Increase Factor
Iowa to Marseille
121
276
6600
24x
Melbourne to Marseille
282
120
3800
32x
Iowa-Marseille throughput
Iowa-Marseille receive window and bytes-in-flight
Melbourne-Marseille throughput
Melbourne-Marseille receive window and bytes-in-flight
Even with the new settings in place, the Melbourne to Marseille performance is limited by the receive window on the Cloudflare host. This means that further adjustments to these settings yield even higher throughput.
Latency results
The Y-axis on these charts are the 99th percentile time for TCP collapse in seconds.
Cloudflare hosts in Marseille running the current production settings
Cloudflare hosts in Marseille running the new settings
The takeaway in looking at these graphs is that maximum TCP collapse time for the new settings is no worse than with the current production settings. This is the desired result.
Send Buffers
What we have shown so far is that the receiver side seems to be working well, but what about the sender side?
As part of this work, we are setting tcp_wmem max to 512 MiB. For oscillating reader flows, this can cause the send buffer to become quite large. This represents bufferbloat and wasted kernel memory, both things that nobody likes or wants.
Fortunately, there is already a solution: tcp_notsent_lowat. This setting limits the size of unsent bytes in the write queue. More details can be found at https://lwn.net/Articles/560082.
The results are significant:
The RTT for these tests was 466ms. Throughput is not negatively affected. Throughput is at full wire speed in all cases (1 Gbps). Memory usage is as reported by /proc/net/sockstat, TCP mem.
Our web servers already set tcp_notsent_lowat to 131072 for its sockets. All other senders are using 4 GiB, the default value. We are changing the sysctl so that 131072 is in effect for all senders running on the server.
Conclusion
The goal of this work is to open the throughput floodgates for high BDP connections while simultaneously ensuring very low HTTP request latency.
Can you monitor a Philips OneBlade shaver with Zabbix? Of course, you can! But why do that and how to monitor a dumb device with zero IoT capabilities?
Welcome to my weekly blog: I get my bread and butter by being a monitoring tech lead in a global cyber security company, but I monitor my home for fun with Zabbix & Grafana and do some weird experiments.
Staying Alive
We all know how the battery-operated shavers, toothbrushes and similar devices sound very energetic and trustworthy immediately after you have charged their battery to full. Over time (over not so long time) they start to sound tired, but technically you can still use them. Or, you think you can still use them, but instead, they will betray you and die in the middle of the operation. Zabbix to the rescue!
Sing to me, bad boy
To get an idea about the battery runtime left, I needed to somehow capture the sound frequency and analyze it. The recording part was easy — after I had charged my razor to full level, I did leave it running and recorded the sound with my iPhone Voice Memos.
But how to get the sound frequency? This is the part where the audio engineers of the world can laugh at me in unison.
At first, I tried with Audacity as traditionally it has done all the tricks I possibly need to do with audio. Unfortunately, I could not find a way to accomplish my dream with it, and even if I would have, I fear I would have to manually do something with it, instead of the automated fashion I’m wishing for.
I could see all kinds of frequencies with Audacity, but was not able to isolate the humming sound of Philips OneBlade, at least not to a format I could use with Zabbix. Yes, Audacity has macros and some functionality remotely from the command line, but I interrupted my attempts with it. If you can do stuff like this with Audacity, drop me a note, I’m definitely interested!
Here come the numbers
Then, after a bit of searching, I found out aubiopitch. It analyzes the sample and returns a proper heckton of numbers back to you.
Those are not GPS coordinates or lottery numbers. That’s a timestamp in seconds and the sound frequency in Hz. And, just by peeking at the file manually, I found out that the values around 100, plus-minus something, were constantly present in the file. Yes, my brains have developed a very good pattern matching algorithm when it comes to log files, as that’s what I have been staring at for the last 20+ years.
As my 30+ minutes sample contained over 300,000 lines of these numbers, I did not want to bother my poor little home Zabbix with this kind of data volume for my initial analysis. I hate spreadsheet programs, especially with data that spans to hundreds of thousands of rows or more, so how to analyze my data? I possibly could have utilized Grafana’s CSV plugin, but to make things more interesting (for me, anyway), I called to my old friend gnuplot instead. Well, a friend in a sense that I know that it exists and that I occasionally used it two decades ago for simple plotting.
There it is, my big long needle in a haystack! Among some other environmental sounds, aubiopitch did recognize the Philips soundtrack as well! What if I filter out those higher frequencies? Or at least attempt to, my gnuplot-fu is not strong.
Yes, there it is, the upper line steadily coming down. After my first recording, it looks like that with a full battery the captured frequency starts from about 115 Hz, and everything goes well until about 93 Hz, but if I would start to shave around that time, I would better be quick, as I would only have two to three minutes left before the frequency quickly spirals down.
Production show-stoppers
This thing is not in “production” yet, because
I need to do more recordings to see if I get similar frequencies each time
I need to fiddle with iPhone Shortcuts to make this as automated as possible.
Anyway, I did start building a preliminary Zabbix template with some macros already filled in…
… and I have a connection established between my dear Siri and Zabbix, too; this will be a topic for another blog entry in the future.
I am hoping that I could get Siri to upload the Voice Memo automatically to my Zabbix Raspberry Pi, which then would immediately analyze the data with aubiopitch maybe with a simple incron hook, and Zabbix would parse the values. That part is yet to be implemented, but I am getting there. It’s just numbers, and in the end, I will just point Zabbix to a simple text file to gather its numbers or make zabbix_sender to send in the values. Been there, done that.
I have been working at Forcepoint since 2014 and for this post to happen I needed to use some razor-sharp thinking. — Janne Pikkarainen
Welcome to the 18th edition of the AWS Serverless ICYMI (in case you missed it) quarterly recap. Every quarter, we share all the most recent product launches, feature enhancements, blog posts, webinars, Twitch live streams, and other interesting things that you might have missed!
In case you missed our last ICYMI, check out what happened last quarter here.
Customers are rapidly adopting the new runtime version by updating to Node.js 16.x. To help keep Lambda functions secure, AWS continually updates Node.js 16 with all minor updates released by the Node.js community when using the zip archive format. Read the release blog post to learn more about building Lambda functions with Node.js 16.x.
A new Lambda custom runtime is now available for PowerShell. It makes it even easier to run Lambda functions written in PowerShell. Although Lambda has supported PowerShell since 2018, this new version simplifies the process and reduces the additional steps required during the development process.
To get started, see the GitHub repository which contains the code, examples and installation instructions.
PowerShell code in Lambda console
AWS Lambda Powertools is an open-source library to help customers discover and incorporate serverless best practices more easily. Powertools for Python went GA in July 2020, followed by Java in 2021, TypeScript in 2022, and .NET is coming soon. AWS Lambda Powertools crossed the 10M download milestone and TypeScript support has now moved from beta to a release candidate.
AWS Step Functions launched a new opt-in console experience to help builders analyze, debug, and optimize Step Functions Standard Workflows. This allows you to debug workflow executions and analyze the payload as it passes through each state. To opt in to the new console experience and get started, follow these detailed instructions.
Events tab in Step Functions workflow
Amazon EventBridge
Amazon EventBridge released support for global endpoints in April 2022. Global endpoints provide a reliable way for you to improve availability and reliability of event-driven applications. Using global endpoints, you can fail over event ingestion automatically to another Region during service disruptions.
The new IngestionToInvocationStartLatency metric exposes the time to process events from the point at which they are ingested by EventBridge to the point of the first invocation. Amazon Route 53 uses this information to failover event ingestion automatically to a secondary Region if the metric exceeds a configured threshold of 30 seconds, consecutively for 5 minutes.
Serverlesspresso is a serverless event-driven application that allows you to order coffee from your phone.
Since building Serverlesspresso for reinvent 2021, the Developer Advocate team have put in approximately 100 additional development hours to improve the application to make it a multi-tenant event-driven serverless app.
This allowed us to run Serverlesspresso concurrently at five separate events across Europe on a single day in June, serving over 5,000 coffees. Each order is orchestrated by a single Step Functions workflow. To read more about how this application is built:
The AWS Heroes program recognizes talented experts whose enthusiasm for knowledge-sharing has a real impact within the community. The EMEA-based Heroes gathered for a Summit on June 28 to share their thoughts, providing valuable feedback on topics such as containers, serverless and machine learning.
Serverless workflow collection added to Serverless Land
Serverless Land is a website that is maintained by the Serverless Developer Advocate team to help you learn with workshops, patterns, blogs and videos.
Using the new collection you can explore common patterns built with Step Functions and use the 1-click deploy button to deploy straight into your AWS account.
Serverless Workflows Collection on Serverless Land
Videos
Serverless Office Hours – Tues 10AM PT
ServerlessLand YouTube Channel
Weekly live virtual office hours. In each session we talk about a specific topic or technology related to serverless and open it up to helping you with your real serverless challenges and issues. Ask us anything you want about serverless technologies and applications.
Marcia Villalba frequently publishes new videos on her popular serverless YouTube channel. You can view all of Marcia’s videos at https://www.youtube.com/c/FooBar_codes.
The Serverless landing page has more information. The Lambda resources page contains case studies, webinars, whitepapers, customer stories, reference architectures, and even more Getting Started tutorials.
Кой, ако не президентът Румен Радев, е насреща, за да нагласи един служебен кабинет и да извади България от тресавището? Да качи на гърба си локомотива и да издърпа цялата композиция като бурлак на Волга, за да тръгне влакът. Защото политиците… ами eто, видяхме ги колко могат – минаха шест месеца, коалицията се разбута и пак избори, четвърти за година и нещо.
Ако партиите в парламента не успеят да се договорят за правителство, упованието в президентската институция ще нарасне като упованието в мощите на св. Йоан Кръстител от Созопол. Такова явление не е наблюдавано досега в българската политика. По време на Виденовата зима, когато имаше хиперинфлация и бунтове, президентът Петър Стоянов (1997–2002) беше олицетворение на просветения разум и институционализъм, а назначеният от него служебен кабинет започна стабилизацията на държавата с подготовката на валутния борд. Но служебните правителства на Румен Радев наброяват три още в първия му мандат, а по всичко личи, че и във втория ще са няколко. Чрез тях обаче Радев акумулира власт и трупа рейтинг за сметка на Народното събрание в парламентарната република България.
Улеснен е – конституционно разтоварен от отговорности, но със собствени стратегически назначения в регулатори и съдебна система, с назначения на посланици и шефове на спецслужби. Нещо повече, служебно правителство на Радев подмени миналата година шефовете на ДАНС, ДАР, военното разузнаване, съгласувано с президента, вместо да назначи временно изпълняващи и да остави на редовен кабинет да избере титулярите. Така че в момента в ръцете на президента е националната сигурност (заедно с НСО, чийто шеф той така и не смени, въпреки че премиерът Кирил Петков поиска оставката на бригаден генерал Емил Тонев).
Капитулация на един главнокомандващ
Предвид тези факти е странно, когато президентът обяснява как не е бил информиран за изгонването на 70-те руски дипломати и служители. Държавният глава призова правителството в оставка да преосмисли действията си предвид „последствията от ескалацията на кризата в двустранните отношения“, а премиера Кирил Петков – да свика заседание на кабинета. След изявлението му Русия заплаши със закриване на посолството си с вербална нота, връчена от посланичката на страната у нас Митрофанова. Тя даде ултиматум до 12 часа в петък да бъдат оттеглени нотите за обявените за персона нон грата служители на руските задгранични представителства – в противен случай ще говори с президента Путин.
А петъчният ден бе открит с публичната подкрепа, заявена от лидера на ГЕРБ Бойко Борисов, за решението за изгонването на руските дипломати, и с настойчивостта на ГЕРБ, ДПС и „Възраждане“ да извикат в парламента премиера за изслушване по темата. „Сега има война. Ние сме на страната на Украйна, на евро-атлантическия свят. Ако влезем в детайлите и в глупостта им, ще излезе, че не сме от тази страна на чертата, а ние сме. И дотогава, макар и аматьорски, дилетантски, абсолютно непрофесионални, тези действия трябва да бъдат подкрепени, за да стигнат до края“, заяви Борисов. За първи път бившият премиер, който в трите си мандата показа съобразяване с руските зависимости и не направи сериозен опит да ги прекрати, в опозиция действа като 100-процентов евро-атлантик. И тук съвпадения с държавния глава няма.
Откакто влезе във втория си мандат, спечелен заради противостоенето му на ГЕРБ и на главния прокурор Иван Гешев, президентът заговори в един глас с ГЕРБ и изпадна в амнезия за Гешев, чиято оставка допреди две години искаше наравно с други. Макар да критикува правителството заради несъстоялата се съдебна реформа, той удобно забрави за собствения си проект за промени в Конституцията. Какво от това, че през ноември м.г. обяви, че му „идва времето“.
В едно обаче Румен Радев е бил винаги праволинеен – нито по времето, когато беше активен борец срещу „бойковизма“, а още повече сега не е произнасял и дума срещу влиянието на Кремъл, камо ли да го дефинира като заплаха за националната сигурност заради генерирането на корупция, на страх и недоверие в ЕС и НАТО. По време на протестите през 2020 г., а и през 2021 г., когато се самономинира за втори президентски мандат, демократичната общност се правеше, че не забелязва това… опущение.
Радев обаче винаги е разбирал ядреното бъдеще на България като бъдеще за АЕЦ „Белене“; тръбата на „Турски поток“ – като злоупотреби със средства, но не и като още по-тежка зависимост от „Газпром“; военна помощ за Украйна – като въвличане на България във война; а за спрените от „Газпром“ доставки на руски газ обвини правителството. Като натовски генерал и бивш началник на ВВС той дори поиска България сама да пази въздушното си пространство, въпреки че е наясно колко несъстоятелно е това заради състоянието на бойната авиация. Разбиранията му за „руския“ Крим са константни още от първия мандат.
Затова и президентът няма да застане на страната на премиера Петков и правителството в оставка и да го подкрепи срещу шантажа на Кремъл. Настроенията в българското общество са колебливи. Според национално представително изследване на общественото мнение, проведено на терен в периода 6–16 юни 2022 г. от „Алфа Рисърч“ по поръчка на Институт „Отворено общество – София“, над 39% от анкетираните заявяват, че България трябва да се позиционира в съюз със страните от НАТО и ЕС в случай на ново разделение в Европа, подобно на Студената война. Но макар и с най-голям дял във всички възрастови групи, подкрепящите нямат мнозинство в нито една.
За съюз със страни като Русия и Беларус се обявяват 23% от запитаните. Близо 7% биха предпочели друг вариант на поведение на страната. Но е твърде голям делът на хората, които не могат да преценят или не отговарят на въпроса – близо 31%. За спечелването им на следващи избори ще се конкурират и БСП, и „Възраждане“, и „Български възход“ на Стефан Янев, защо не и „Изправи се, България“ на Мая Манолова.
Президентът ще помогне с каквото може. Откакто се позиционира като критик на правителството (прави го почти ежедневно), канонадата от жълти медии по Радев спря. А неговата представителка в СЕМ Габриела Наплатанова пък спомогна настоящият директор на БНТ Емил Кошлуков, известен с близостта си и до ГЕРБ, и до ДПС, да запази поста си. Удобното оправдание беше принципност, в името на която тя се въздържа от гласуване, макар да оцени високо професионализма на един от кандидатите и неин бивш шеф в bTV Венелин Петков.
Залезът на Борисов
Докато звездата на Румен Радев се вдига все по-нависоко, Борисов залязва. Повече от година бившият премиер и все още лидер на ГЕРБ обикаля из България, говори по партийни сбирки и отмаря в Банкя с внуците и кучетата. Оттам „вдига“ ветото за Северна Македония с призива си към премиера Кирил Петков да внесе т.нар. френско предложение в парламента още на 22 юни, след което тутакси изригнаха похвали от председателя на ЕНП Манфред Вебер, еврокомисаря по разширяването Оливер Вархеи и албанския премиер Еди Рама. Макар че именно правителството на Борисов блокира европейския път на Скопие през 2019 г.
Но въпреки някои ситуационни победи, в т.ч. успешния вот на недоверие, Борисов е наясно, че времето му е изтекло и е най-добре да се оттегли – само че не си е изработил пътя за отстъпление. Той е твърде обременен от десетте години управление – политическо людоедство, корупционни схеми, чадър от главния прокурор, укрепване на руското влияние. Докато президентът няма как да бъде уличен в корупция, тъй като пряко не управлява.
Въпреки някогашното си противоборство обаче, днес те като че ли играят в един отбор – на противниците на правителството на Кирил Петков, независимо от мнозинството в 47-мото НС, подкрепило френското предложение за ветото за Северна Македония. ГЕРБ и ДПС, гласували заедно с „Продължаваме промяната“ и „Демократична България“, изпълниха волята на европейските политически семейства, към които принадлежат. А президентът така и не успя да изиграе „македонската карта“, въпреки упорството си да издига бариери пред разрешаването на проблема.
Ако миналата година служебните правителства бяха добър инструмент за рейтинга на Радев, също и ускорител за политическата кариера на Кирил Петков и Асен Василев, сега ситуацията е различна. Кризите са прекалено много, а напрежението е твърде високо – растящи цени, скъпи горива, нови страхове заради нови щамове на COVID-19, война.
Този път Румен Радев ще съставя с неохота четвъртия си служебен кабинет. Какво да се прави – кой, ако не той…
Три български дружества са внасяли фосфати от Сирийската арабска република, чийто президент Башар Асад, правителството и ресорното Министерство на петрола са под санкциите на САЩ от 2011 г., както и…
The Software Freedom Conservancy (SFC) has issued a strong call for free software projects to give up GitHub and to move their repositories elsewhere. There are a number of problems that SFC has identified with the GitHub code-hosting service and, in particular, with its Copilot AI-based code-writing tool that was trained on the community’s code stored in the company’s repositories. Moving away from GitHub will not be easy, SFC said, but it is important to do so lest the free-software community repeat the SourceForge mistake.
Specifically, we at Software Freedom Conservancy have been actively communicating with Microsoft and their GitHub subsidiary about our concerns with “Copilot” since they first launched it almost exactly a year ago. Our initial video chat call (in July 2021) with Microsoft and GitHub representatives resulted in several questions which they said they could not answer at that time, but would “answer soon”. […] Last week, after we reminded GitHub of (a) the pending questions that we’d waited a year for them to answer and (b) of their refusal to join public discussion on the topic, they responded a week later, saying they would not join any public nor private discussion on this matter because “a broader conversation [about the ethics of AI-assisted software] seemed unlikely to alter your [SFC’s] stance, which is why we [GitHub] have not responded to your [SFC’s] detailed questions”. In other words, GitHub’s final position on Copilot is: if you disagree with GitHub about policy matters related to Copilot, then you don’t deserve a reply from Microsoft or GitHub. They only will bother to reply if they think they can immediately change your policy position to theirs. But, Microsoft and GitHub will leave you hanging for a year before they’ll tell you that!
Wired is reporting on a new remote-access Trojan that is able to infect at least eighty different targets:
So far, researchers from Lumen Technologies’ Black Lotus Labs say they’ve identified at least 80 targets infected by the stealthy malware, including routers made by Cisco, Netgear, Asus, and DrayTek. Dubbed ZuoRAT, the remote access Trojan is part of a broader hacking campaign that has existed since at least the fourth quarter of 2020 and continues to operate.
The discovery of custom-built malware written for the MIPS architecture and compiled for small-office and home-office routers is significant, particularly given its range of capabilities. Its ability to enumerate all devices connected to an infected router and collect the DNS lookups and network traffic they send and receive and remain undetected is the hallmark of a highly sophisticated threat actor.
Amazon Keyspaces (for Apache Cassandra) is a scalable, highly available, and managed Apache Cassandra–compatible database service. With Amazon Keyspaces you don’t have to provision, patch, or manage servers, and you don’t have to install, maintain, or operate software. Amazon Keyspaces is serverless, so you only pay for the resources that you use and the service can automatically scale tables up and down in response to application traffic. You can use Amazon Keyspaces to store large volumes of data, such as entries in a log file or the message history for a chat application as Amazon Keyspaces offers virtually unlimited throughput and storage. You can also use Amazon Keyspaces to store information about devices for Internet of Things (IoT) applications or player profiles for games.
A popular use case in the wind energy sector is to protect wind turbines from wind speed. Engineers and analysts often want to see real-time aggregated wind turbine speed data to analyze the current situation out in the field. Furthermore, they need access to historical aggregated wind turbine speed data to build machine learning (ML) models which can help them take preventative actions on wind turbines. Customers often ingest high-velocity IoT data into Amazon Kinesis Data Streams and use Amazon Kinesis Data Analytics, AWS Lambda, or Amazon Kinesis Client Library (KCL) applications to aggregate IoT data in real-time and store it in Amazon Keyspaces, Amazon DynamoDB, or Amazon Timestream.
In this post, we demonstrate how to aggregate sensor data using Amazon Kinesis Data Analytics and persist aggregated sensor data in to Amazon Keyspaces using Apache Flink’s Apache Cassandra Connector.
Architecture
In the architecture diagram above, Lambda simulates wind speed sensor data and ingests sensor data into Amazon Kinesis Data Stream. Amazon Kinesis Data Analytics Apache Flink application reads wind speed sensor data from Amazon Kinesis Data Stream in real-time and aggregates wind speed sensor data using a five minutes tumbling window and storing aggregated wind speed sensor data into Amazon Keyspaces table. Aggregated wind speed sensor data stored in Amazon Keyspaces can be used by engineers and analysts to review real-time dashboards or to perform historical analysis on specific wind turbine.
Deploying resources using AWS CloudFormation
After you sign in to your AWS account, launch the AWS CloudFormation template by choosing Launch Stack:
The CloudFormation template configures the following resources in your account:
One Lambda function which simulates wind turbine data
One Amazon Kinesis Data Stream
One Amazon Kinesis Data Analytics Apache Flink application
One Amazon Keyspaces Table: turbine_aggregated_sensor_data
After you complete the setup, sign in to the Kinesis Data Analytics console. On the Kinesis Data Analytics applications page, choose the Streaming applications tab, where you can see the Streaming application in the ready status. Select the Streaming application, choose Run, and wait until the Streaming application is in running status. It can take a couple of minutes for the Streaming application to get into running status.
Now that we have deployed all of the resources using CloudFormation template, let’s review deployed resources and how they function.
Format of wind speed sensor data
Lambda simulates wind turbine speed data every one minute and ingests it into Amazon Kinesis Data Stream. Each wind turbine sensor data message consists of two attributes: turbineId and speed.
{
"turbineId": "turbine-0001",
"speed": 60
}
Schema of destination Amazon Keyspaces table
We’ll store aggregated sensor data in to destination turbine_aggregated_sensor_data Amazon Keyspaces table. turbine_aggregated_sensor_data table has on-demand capacity mode enabled. Amazon Keyspaces (for Apache Cassandra) on-demand capacity mode is a flexible billing option capable of serving thousands of requests per second without capacity planning. This option offers pay-per-request pricing for read and write requests so that you pay only for what you use. When you choose on-demand mode, Amazon Keyspaces can scale the throughput capacity for your table up to any previously reached traffic level instantly, and then back down when application traffic decreases. If a workload’s traffic level hits a new peak, then the service adapts rapidly to increase throughput capacity for your table.
Apache Flink code to aggregate and persist data in Amazon Keyspaces Table
The following code snippet demonstrates how incoming wind turbine messages are getting aggregated using a five-minute tumbling window and produces a DataStream of TurbineAggregatedRecord records.
The following code snippet demonstrates how Amazon Keyspaces table name and column names are annotated on the TurbineAggregatedRecord class.
@Table(keyspace = "sensor_data", name = "turbine_aggregated_sensor_data", readConsistency = "LOCAL_QUORUM", writeConsistency = "LOCAL_QUORUM")
public class TurbineAggregatedRecord {
@Column(name = "turbineid")
@PartitionKey(0)
private String turbineid = "";
@Column(name = "reported_time")
private long reported_time = 0;
@Column(name = "max_speed")
private long max_speed = 0;
@Column(name = "min_speed")
private long min_speed = 0;
@Column(name = "avg_speed")
private long avg_speed = 0;
The following code snippet demonstrates the implementation of Apache Cassandra Connector to sink aggregated wind speed sensor data TurbineAggregatedRecord into Amazon Keyspaces table. We’re using SigV4AuthProvider with Apache Cassandra Connector. The SigV4 authentication plugin lets you use IAM credentials for users or roles when connecting to Amazon Keyspaces. Instead of requiring a user name and password, this plugin signs API requests using access keys.
CassandraSink.addSink(result)
.setClusterBuilder(
new ClusterBuilder() {
private static final long serialVersionUID = 2793938419775311824L;
@Override
public Cluster buildCluster(Cluster.Builder builder) {
return builder
.addContactPoint("cassandra."+ region +".amazonaws.com")
.withPort(9142)
.withSSL()
.withAuthProvider(new SigV4AuthProvider(region))
.withLoadBalancingPolicy(
DCAwareRoundRobinPolicy
.builder()
.withLocalDc(region)
.build())
.withQueryOptions(queryOptions)
.build();
}
})
.setMapperOptions(() -> new Mapper.Option[] {Mapper.Option.saveNullFields(true)})
.setDefaultKeyspace("sensor_data")
.build();
Review output in Amazon Keyspaces Table
Once Amazon Kinesis Data Analytics Apache Flink application aggregates wind turbine sensor data and persists aggregated data in Amazon Keyspaces table, we can query and review aggregated data using Amazon Keyspaces CQL editor as illustrated in the following.
select * from sensor_data.turbine_aggregated_sensor_data
Clean up
To avoid incurring future charges, complete the following steps:
Empty Amazon S3 bucket created by AWS CloudFormation stack.
Delete AWS CloudFormation stack.
Conclusion
As you’ve learned in this post, you can build Amazon Kinesis Data Analytics Apache Flink application to read sensor data from Amazon Kinesis Data Streams, perform aggregations, and persist aggregated sensor data in Amazon Keyspaces using Apache Cassandra Connector. There are several use cases in IoT and Application development to move data quickly through the analytics pipeline and persist data in Amazon Keyspaces.
We look forward to hearing from you about your experience. If you have questions or suggestions, please leave a comment.
About the Author
Pratik Patel is a Sr Technical Account Manager and streaming analytics specialist. He works with AWS customers and provides ongoing support and technical guidance to help plan and build solutions using best practices and proactively helps in keeping customer’s AWS environments operationally healthy.
Is this comment out-of-date? I don’t think it describes what I’m seeing.
This pull request is massive, where do I start?
Where did this bug come from?
These questions all reflect the limitations of collaboratively-developed source code as a communication medium. While there are ways to mitigate these issues (code comments, style guides, documentation requirements), we still inevitably find ourselves spending hours on just trying to understand code. Luckily, the tools needed to solve these problems have been here all along!
Commits in Git repositories are more than just save points or logs of incremental progress in a larger project. In the words of GitHub’s “Git Guides“:
[Commits] are snapshots of your entire repository at specific times…based around logical units of change. Over time, commits should tell a story of the history of your repository and how it came to be the way that it currently is.
Commits are a firsthand historical record of exactly how and why each line of code came to be. They even come with human-readable messages! As a result, a repository’s commit history is the best tool a developer can use to explain and understand code.
It’s been my experience that commits are most effective when they’re tweaked and polished to deliberately convey a message to their audiences: reviewers, other contributors, and even your future self. This post will:
Introduce some guidelines for organizing and revising commits.
Outline pragmatic approaches for applying those guidelines.
Describe some of the practical applications of a well-crafted commit history.
Writing better commits
Software development involves a lot of creativity, so your commits will reflect the context of your changes, your goals, and your personal style. The guidelines below are presented to help you utilize that creative voice to make your commits effective tools of communication.
As you read these guidelines, don’t worry about how you’ll be able to utilize all of this advice in the midst of writing code. Although you may naturally incorporate them into your development process with practice, each can be applied iteratively after you’ve written all of your code.
Structure the narrative
Like your favorite novel, a series of commits has a narrative structure that contextualizes the “plot” of your change with the code. Before any polishing, the narrative of a branch typically reflects an improvised stream of consciousness. It might contain:
A commit working on component A, followed by one on component B, followed by one finishing component A
A multi-commit detour into trying again and again to get the right CI syntax
A commit fixing a typo from an earlier commit
A commit with a mixed bag of all of the review feedback
A merge commit resolving conflicts with the main branch
Although an accurate retelling of your journey, a branch like this tells a “story” that is neither coherent nor memorable.
The problem
Disorganized commits that eschew a clear narrative will affect two people: the reviewer, and the developer themself.
Reviewing commit-by-commit is the easiest way to avoid being overwhelmed by the changes in a sufficiently large pull request. If those commits do not tell a singular, easy-to-follow story, the reviewer will need to context-switch as the author’s commits jump from topic to topic. To ensure earlier commits properly set up later ones (for example, verifying a newly-created function is used properly), the reviewer ultimately needs to piece together the narrative on their own; for each commit, figure out which earlier changes establish the relevant background context and tediously click back and forth between them. Alternatively, they’ll remember some vague details and simply assume earlier commits properly set up later ones, failing to identify potential issues.
But how does a scatterbrained narrative hurt the developer? A developer’s first instinct when working on a new project is often to hack on it until they get something functional. Fluctuating between “fun” and “frustrating,” this approach eventually yields good results, but it’s far from efficient. Jumping in without a plan – the mindset of following a narrative – makes that process slower than it needs to be.
The solution
Outline your narrative, and reorganize your commits to match it.
The narrative told by your commits is the vehicle by which you convey the meaning of your changes. Also, like a story, it can take on many structures or forms.
Your branch is your story to tell. While the narrative is up to you, here are some editorial tips on how to keep it organized:
DO
DON’T
Write an outline and include it in the pull request description.
Wait until the end to form the outline – try using it to guide your work!
Stick to one high-level concept per branch.
Go down a tangentially-related “rabbit hole”.
Add your “implement feature” commit immediately after the refactoring that sets it up.
Jump back and forth between topics throughout your branch.
Treat commits as “building blocks” of different types: bugfix, refactor, stylistic change, feature, etc.
Mix multiple building block types in a single commit.
Resize and stabilize the commits
Although the structure of a commit series can tell the high-level story of an author’s feature, it’s the code within each commit that creates software. Code itself can be complex, dense, and cryptic but in order to collaborate, others need to understand it.
The problem
The cognitive burden of parsing code is exacerbated by having either too much or not enough information presented at once. Too much, and your reader will need to read and understand multiple conceptually-different topics that could get jumbled, misinterpreted, or simply missed; too little, and your reader will develop an incomplete mental model of a change.
For a reviewer, one of the big benefits of a commit-by-commit review is – like individual lectures in a semester-long course – pacing the development of their mental model with small, easy-to-digest changes. When a large commit doesn’t provide that sustainable learning pace, the reviewer may fail fail to identify questionable architectural decisions because they conflate unrelated topics, or even miss a bug because it’s in a section seemingly irrelevant to the impacted feature.
You might think reviewers’ problems would be solved with commits as small as possible, but an incomplete change leaves them unable to evaluate it fully as they read it. When a later commit “completes” the change, a reviewer may not easily draw connections to the earlier context. This is made worse when a later commit undoes something from the earlier, partial commit. The “churn” in these situations leads to the same weakened mental model – and same consequences – as when dealing with too-large commits.
Poorly-sized commits present more tangible issues as well. Most apparent is the inability to roll back your repository to a commit (for example, when debugging a strange feature interaction). Incomplete changes often fail to build, so a developer will be stuck searching nearby commits for a fix. Similarly, a bug narrowed down to a massive commit requires teasing apart its intermixed changes, a potentially more difficult task than it was during the initial review due to loss of institutional project knowledge over time.
The solution
Make each commit both “small” and “atomic.”
To best convey your story, commits should minimize the effort needed to build a mental model of the changes they introduce. With effort tied to having a “just right” amount of information, the key to a good commit is fitting into quantified upper and lower bounds on that information.
A small commit is one with minimal scope; it does one “thing.” This often correlates to minimizing the modified lines of cone, but that isn’t a firm requirement. For example, changing the name of a commonly-used function may modify hundreds of lines of code, but its constrained scope makes it simple to both explain and review.
A commit is atomic when it is a stable, independent unit of change. In concrete terms, a repository should still build, pass tests, and generally function if rolled back to that exact commit without needing other changes. In an atomic commit, your reader will have everything they need to evaluate the change in the commit itself.
Explain the context
Commits are more than just the code they contain. Despite there being no shortage of jokes about them, commit messages are an extremely valuable – but often overlooked – component of a commit. Most importantly, they’re an opportunity to speak directly to your audience and explain a change in your own terms.
The problem
Even with a clear narrative and appropriately-sized commits, a niche change can still leave readers confused. This is especially true in large or open-source projects, where a reviewer or other future reader (even yourself!) likely won’t be clued into the implementation details or nuances of the code you’ve changed.
Code is rarely as self-evident as the author may believe, and even simple changes can be prone to misinterpretation. For example, what may appear to be a bug may instead be a feature implemented to solve an unrelated problem. Without understanding the intent of the original change, a developer may inadvertently modify an expected user-facing behavior. Conversely, something that appears intentional may have been a bug in the first place. A misinterpretation could cause a developer to enshrine a small mistake as a “feature” that hurts user experience for years.
Even in a best-case scenario, poorly explained changes will slow down reviewers and contributors as they attempt to interpret code, unnecessarily wasting everyone’s time and energy.
The solution
Describe what you’re doing and why you’re doing it in the commit message.
Because you’re writing for an audience, the content of a commit message should clearly communicate what readers need to understand. As the developer, you should already know the background and implementation well enough to explain them. Rather than write excessively long (and prone to obsoletion) code comments or put everything into a monolithic pull request description, you can use commit messages to provide piecewise clarification to each change.
“What” and “why” break down further into high- and low-level details, all of which can be framed as four questions to answer in each commit message:
What you’re doing
Why you’re doing it
High-level (strategic)
Intent (what does this accomplish?)
Context (why does the code do what it does now?)
Low-level (tactical)
Implementation (what did you do to accomplish your goal?)
Justification (why is this change being made?)
Building better projects
Using the guidelines established above, you can mitigate the challenges of common software development tasks including code review, finding bugs, and root cause analysis.
Code review
Reviewing even the largest pull requests can be a manageable, straightforward process if you are able to evaluate changes on a commit-by-commit basis. Each of the guidelines detailed earlier focuses on making the commits readable; to extract information from commits, you can use the guidelines as a template.
Determine the narrative by reading the pull request description and list of commits. If the commits seem to jump between topics or address, leave a comment asking for clarification or changes.
Lightly scan the message and contents of each commit, starting from the beginning of the branch. Verify smallness and atomicity by checking that the commit does one thing and that doesn’t include any incomplete implementations. Recommend splitting or combining commits that are incorrectly scoped.
Thoroughly read each commit. Ensure the commit message sufficiently explains the code by first checking that implementation matches the intent, then that the code matches the stated implementation. Use the context and justification to guide your understanding of the code. If any of the requisite information is missing, ask for clarification from the author.
Finally, with a complete mental model of the commit’s changes and the overarching narrative, confirm the code is efficient and bug-free.
Finding bugs with git bisect
If you’ve ever found yourself with a broken deployment and no idea when the breakage was introduced, git bisect is the tool for you. Specifically, git bisect is a tool built into Git that, when given a known-good commit (for example, your last stable deployment) and a known-bad commit (the broken one), will perform a binary search of the commits in the middle to find which one introduced the error.
As useful as git bisect is, it absolutely requires that each commit it traverses is both atomic and small. If not atomic, you will be unable to test for repository stability at each commit; if not small, the source commit of your bug may be so large that you end up inefficiently reading code line-by-line to find the bug anyway.
Root cause analysis
Suppose you’ve used something like git bisect to isolate the source commit of a bug. If you’re lucky, the underlying problem is obvious and you can fix it immediately. More often than not, things aren’t so simple; the bug-causing code might be necessary for another feature, or does not make sense as the source of the error you’re seeing. You need to understand why the code was written, and to do that, you can again use the commit history to investigate.
There are two main tools to help you search through commits: git blame and git log.
git blame annotates the file with the commit that last changed it:
This can be particularly helpful in finding which commits modify the same area of code, which you can then read to determine if they interact poorly.
For a more generalized commit search, you can use git log. In its simplest form, git log will display a list of commits in reverse chronological order, starting at HEAD:
$ git log --oneline
09823ba09de1 README.md: update project title
abd52642da46 my-file.py: add README stat printout
7392d7dbb9ae my-file.py: rename from oldname.py
5ad823d1bc48 test.py: commonize test setup
603ab927a0dd oldname.py: create printout script
...
The displayed list of commits can also be filtered by file(s), by function name, by line range in a file, by commit message text, and more. As with git blame, these filtered lists of commits can help you build a complete mental model of the changes that comprise a particular file or function, ultimately guiding you to the root cause of your bug.
Final thoughts
Although subjective and sometimes difficult to quantify, commit quality can make a massive difference in developer quality-of-life on any project: old or new, large or small, open- or closed-source. To make commit refinement part of your own development process, some guidelines to follow are:
Organize your commits into a narrative.
Make each commit both small and atomic.
Explain the “what” and “why” of your change in the commit message.
These guidelines, as well as their practical applications, demonstrate how powerful commits can be when used to contextualize code. Regardless of what you do with them, commits will tell your project’s story; these strategies will help you make it a good one.
Version
1.62.0 of the Rust language has been released. Changes include a new cargo add command, default enum variants, an improved Linux
mutex implementation, a number of stabilized APIs, and more.
Amazon Web Services (AWS) is the first cloud service provider to produce an Open Security Control Assessment Language (OSCAL)–formatted system security plan (SSP) for the FedRAMP Project Management Office (PMO). OSCAL is the first step in the AWS effort to automate security documentation to simplify our customers’ journey through cloud adoption and accelerate the authorization to operate (ATO) process.
AWS continues its commitment to innovation and customer obsession. Our incorporation of the OSCAL format will improve the customer experience of reviewing and assessing security documentation. It can take an estimated 4,200 workforce hours for companies to receive an ATO, with much of the effort due to manual review and transcription of documentation. Automating this process through a machine-translatable language gives our customers the ability to ingest security documentation into a governance, risk management, and compliance (GRC) tool to automate much of this time-consuming task. AWS worked with an AWS Partner, to ingest the AWS SSP through their tool, Xacta.
This is a first step in several initiatives AWS has planned to automate the security assurance process across multiple compliance frameworks. We continue to look for ways to earn trust with our customers, and over the next year we will continue to release new solutions that customers can use to rapidly deploy secure and innovative services.
“Providing the SSP packages in OSCAL is a great milestone in security automation marking the beginning of a new era in cybersecurity. We appreciate the leadership in this area and look forward to working with all cyber professionals, in particular with the visionary cloud service providers, to help deliver secure innovation faster to the people they serve.”
– Dr. Michaela Iorga, OSCAL Strategic Outreach Director, NIST
To learn more about OSCAL, visit the NIST OSCAL website. To learn more about FedRAMP’s plans for OSCAL, visit the FedRAMP Blog.
Want more AWS Security news? Follow us on Twitter.
The collective thoughts of the interwebz
By continuing to use the site, you agree to the use of cookies. more information
The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.
Pratik Patel is a Sr Technical Account Manager and streaming analytics specialist. He works with AWS customers and provides ongoing support and technical guidance to help plan and build solutions using best practices and proactively helps in keeping customer’s AWS environments operationally healthy.
Structure the narrative
Resize and stabilize the commits
Explain the context