Post Syndicated from Patrick Kennedy original https://www.servethehome.com/pch-less-intel-granite-rapids-motherboard-and-mr-dimm-shown-at-computex-2023/
At Computex 2023, we saw a PCH-less Intel server motherboard, likely for Granite Rapids, and a next-gen MR-DIMM
The post PCH-Less Intel Granite Rapids Motherboard and MR-DIMM Shown at Computex 2023 appeared first on ServeTheHome.
Post Syndicated from original https://lwn.net/Articles/933480/
Version
1.70.0 of the Rust language is out. Changes include enabling the
“sparse” protocol for Cargo, a couple of new types for the initialization
of shared data, and more. “You should see substantially improved
”
performance when fetching information from the crates.io index.
Post Syndicated from Jeremy Ber original https://aws.amazon.com/blogs/big-data/real-time-inference-using-deep-learning-within-amazon-kinesis-data-analytics-for-apache-flink/
Apache Flink is a framework and distributed processing engine for stateful computations over data streams. Amazon Kinesis Data Analytics for Apache Flink is a fully managed service that enables you to use an Apache Flink application to process streaming data. The Deep Java Library (DJL) is an open-source, high-level, engine-agnostic Java framework for deep learning.
In this blog post, we demonstrate how you can use DJL within Kinesis Data Analytics for Apache Flink for real-time machine learning inference. Real-time inference can be valuable in a variety of applications and industries where it is essential to make predictions or take actions based on new data as quickly as possible with low latencies. We show how to load a pre-trained deep learning model from the DJL model zoo into a Flink job and apply the model to classify data objects in a continuous data stream. The DJL model zoo includes a wide variety of pre-trained models for image classification, semantic segmentation, speech recognition, text embedding generation, question answering, and more. It supports HuggingFace, Pytorch, MXNet, and TensorFlow model frameworks and also helps developers create and publish their own models. We will focus on image classification and use a popular classifier called ResNet-18 to produce predictions in real time. The model has been pre-trained on ImageNet with 1.2 million images belonging to 1,000 class labels.
We provide sample code, architecture diagrams, and an AWS CloudFormation template so you can follow along and employ ResNet-18 as your classifier to make real-time predictions. The solution we provide here is a powerful design pattern for continuously producing ML-based predictions on streaming data within Kinesis Data Analytics for Apache Flink. You can adapt the provided solution for your use case and choose an alternative model from the model zoo or even provide your own model.
Image classification is a classic problem that takes an image and selects the best-fitting class, such as whether the image from an autonomous driving system is that of a bicycle or a pedestrian. Common use cases for real-time inference on streams of images include classifying images from vehicle cameras and license plate recognition systems, and classifying images uploaded to social media and ecommerce websites. The use cases typically need low latency while handling high throughput and potentially bursty streams. For example, in ecommerce websites, real-time classification of uploaded images can help in marking pictures of banned goods or hazardous materials that have been supplied by sellers. Immediate determination through streaming inference is needed to trigger alerts and follow-up actions to prevent these images from being part of the catalog. This enables faster decision-making compared to batch jobs that run on a periodic basis. The data stream pipeline can involve multiple models for different purposes, such as classifying uploaded images into ecommerce categories of electronics, toys, fashion, and so on.
The following diagram illustrates the workflow of our solution.
The application performs the following steps:
Images are recommended to be of reasonable size so that they may fit into a Kinesis Processing Unit. Images larger than 50MB in size may result in latency in processing and classification.
The main class for this Apache Flink job is located at src/main/java/com.amazon.embeddedmodelinference/EMI.java. Here you can find the main() method and entry point to our Flink job.
To get started, configure the following prerequisites on your local machine:
Once this is set up, you can clone the code base to access the source code for this solution. The Java application code for this example is available on GitHub. To download the application code, clone the remote repository using the following command:
Find and navigate to the folder of the image classification example, called image-classification.
An example set of images to stream and test the code is available in the imagenet-sample-images folder.
Let’s walk through the code step by step.
If you would like to test this application locally on your machine, ensure you have AWS credentials set up locally on your machine. Additionally, download the Flink S3 Filesystem Hadoop JAR to use with your Apache Flink installation and place it in a folder named plugins/s3 in the root of your project. Then configure the following environment variables either on your IDE or in your machine’s local variable scope:
Replace these values with your own.
After configuring the environment variables and downloading the necessary plugin JAR, let’s look at the code.
In the main method, after setting up our StreamExecutionEnvironment, we define our FileSource to read files from Amazon S3. By default, this source operator reads from a sample bucket. You can replace this bucket name with your own by changing the variable called bucket, or setting the application property on Kinesis Data Analytics for Apache Flink once deployed.
The FileSource is configured to read in files in the ImageReaderFormat, and will check Amazon S3 for new images every 10 seconds. This can be configured as well.
After we have read in our images, we convert our FileSource into a stream that can be processed:
Next, we create a tumbling window of a variable time window duration, specified in the configuration, defaulting to 60 seconds. Every window close creates a batch (list) of images to be classified using a ProcessWindowFunction.
This ProcessWindowFunction calls the classifier predict function on the list of images and returns the best probability of classification from each image. This result is then sent back to the Flink operator, where it’s promptly written out to the S3 bucket path of your configuration.
In Classifier.java, we read the image and apply crop, transpose, reshape, and finally convert to an N-dimensional array that can be processed by the deep learning model. Then we feed the array to the model and apply a forward pass. During the forward pass, the model computes the neural network layer by layer. At last, the output object contains the probabilities for each image object that the model is being trained on. We map the probabilities with the object name and return to the map function.
To run this code base on Kinesis Data Analytics for Apache Flink, we have a helpful CloudFormation template that will spin up the necessary resources. Simply open AWS CloudShell or your local machine’s terminal and enter the following commands. Complete the following steps to deploy the solution:
The script will clone a GitHub repo of images to classify and upload them to your source S3 bucket. Then it will launch the CloudFormation stack given your input parameters. 
This CloudFormation stack creates the following resources:
blog-DJL-flink-ImageClassification-application and choose Run.outputBucketPath variable.If you have readable images in the source bucket listed, you should see classifications of those images within the checkpoint interval of the running application.
If you prefer to use your own code base, you can follow the manual steps in this section:
pom.xml and running the following command:This builds your application JAR in the target/ directory called embedded-model-inference-1.0-SNAPSHOT.jar.
You can open the Apache Flink Dashboard to check for application exceptions or to see data flowing through the tasks defined in the code.
To validate our results, let’s check the results in Amazon S3 by navigating to the Amazon S3 console and finding our S3 bucket. We can find the output in a folder called output-kda.
When we choose one of the data-partitioned folders, we can see partition files. Ensure that there is no underscore in front of your part file, because this indicates that the results are still being finalized according to the rollover interval defined in Apache Flink’s FileSink connector. After the underscores have disappeared, we can use Amazon S3 Select to view our data.
We now have a solution that continuously performs classification on incoming images in real time using Kinesis Data Analytics for Apache Flink. It extracts a pre-trained classification model (ResNet-18) from the DJL model zoo, applies some preprocessing, loads the model into a Flink operator’s memory, and continuously applies the model for online predictions on streaming images.
Although we used ResNet-18 in this post, you can choose another model by modifying the classifier. The DJL model zoo provides many other models, both for classification and other applications, that can be used out of the box. You can also load your custom model by providing an S3 link or URL to the criteria. DJL supports models in a large number of engines such as PyTorch, ONNX, TensorFlow, and MXNet. Using a model in the solution is relatively simple. All of the preprocessing and postprocessing code is encapsulated in the (built-in) translator, so all we have to do is load the model, create a predictor, and call predict(). This is done within the data source operator, which processes the stream of input data and sends the links to the data to the inference operator where the model you selected produces the prediction. Then the sink operator writes the results.
The CloudFormation template in this example focused on a simple 1 KPU application. You could extend the solution to further scale out to large models and high-throughput streams, and support multiple models within the pipeline.
To clean up the CloudFormation script you launched, complete the following steps:
KDAImageClassification.CDKToolkit, which you launched earlier as well.In this post, we presented a solution for real-time classification using the Deep Java Library within Kinesis Data Analytics for Apache Flink. We shared a working example and discussed how you can adapt the solution for your specific ML use case. If you have any feedback or questions, please leave them in the comments section.
Jeremy Ber has been working in the telemetry data space for the past 9 years as a Software Engineer, Machine Learning Engineer, and most recently a Data Engineer. At AWS, he is a Streaming Specialist Solutions Architect, supporting both Amazon Managed Streaming for Apache Kafka (Amazon MSK) and AWS’s managed offering for Apache Flink.
Deepthi Mohan is a Principal Product Manager for Amazon Kinesis Data Analytics, AWS’s managed offering for Apache Flink.
Gaurav Rele is a Data Scientist at the Amazon ML Solution Lab, where he works with AWS customers across different verticals to accelerate their use of machine learning and AWS Cloud services to solve their business challenges.
Post Syndicated from Jeremy Ware original https://aws.amazon.com/blogs/security/announcing-the-aws-blueprint-for-ransomware-defense/
In this post, Amazon Web Services (AWS) introduces the AWS Blueprint for Ransomware Defense, a new resource that both enterprise and public sector organizations can use to implement preventative measures to protect data from ransomware events. The AWS Blueprint for Ransomware Defense provides a mapping of AWS services and features as they align to aspects of the Center for Internet Security (CIS) Critical Security Controls (CIS Controls). This information can be used to help customers assess and protect their data from ransomware events.
The following is background on ransomware, CIS, and the initiatives that led to the publication of this new blueprint.
In April of 2021, the U.S. government launched the Ransomware Task Force (RTF), which has the mission of uniting key stakeholders across industry, government, and civil society to create new solutions, break down silos, and find effective new methods of countering the ransomware threat. The RTF has since launched several progress reports with specific recommendations, including the development of the RTF Blueprint for Ransomware Defense, which provides a framework with practical steps to mitigate, respond to, and recover from ransomware. AWS is a member of the RTF, and we have taken action to create our own AWS Blueprint for Ransomware Defense that maps actionable and foundational security controls to AWS services and features that customers can use to implement those controls. The AWS Blueprint for Ransomware Defense is based on the CIS Controls framework.
The Center for Internet Security (CIS) is a community-driven nonprofit, globally recognized for establishing best practices for securing IT systems and data. To help establish foundational defense mechanisms, a subset of the CIS Critical Security Controls (CIS Controls) have been identified as important first steps in the implementation of a robust program to prevent, respond to, and recover from ransomware events. This list of controls was established to provide safeguards against the most impactful and well-known internet security issues. The controls have been further prioritized into three implementation groups (IGs), to help guide their implementation. IG1, considered “essential cyber hygiene,” provides foundational safeguards. IG2 builds on IG1 by including the controls in IG1 plus a number of additional considerations. Finally, IG3 includes the controls in IG1 and IG2, with an additional layer of controls that protect against more sophisticated security issues.
CIS recommends that organizations use the CIS IG1 controls as basic preventative steps against ransomware events. We’ve produced a mapping of AWS services that can help you implement aspects of these controls in your AWS environment. Ransomware is a complex event, and the best course of action to mitigate risk is to apply a thoughtful strategy of defense in depth. The mitigations and controls outlined in this mapping document are general security best practices, but are a non-exhaustive list.
Because data is often vital to the operation of mission-critical services, ransomware can severely disrupt business processes and applications that depend on this data. For this reason, many organizations are looking for effective security controls that will improve their security posture against these types of events. We hope you find the information in the AWS Blueprint for Ransomware Defense helpful and incorporate it as a tool to provide additional layers of security to help keep your data safe.
Let us know if you have any feedback through the AWS Security Contact Us page. Please reach out if there is anything we can do to add to the usefulness of the blueprint or if you have any additional questions on security and compliance. You can find more information from the IST (Institute for Security and Technology) describing ransomware and how to protect yourself on the IST website.
If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.
Want more AWS Security news? Follow us on Twitter.
Post Syndicated from Netflix Technology Blog original https://netflixtechblog.com/ensuring-the-successful-launch-of-ads-on-netflix-f99490fdf1ba
By Jose Fernandez, Ed Barker, Hank Jacobs
In November 2022, we introduced a brand new tier — Basic with ads. This tier extended existing infrastructure by adding new backend components and a new remote call to our ads partner on the playback path. As we were gearing up for launch, we wanted to ensure it would go as smoothly as possible. To do this, we devised a novel way to simulate the projected traffic weeks ahead of launch by building upon the traffic migration framework described here. We used this simulation to help us surface problems of scale and validate our Ads algorithms.
Basic with ads was launched worldwide on November 3rd. In this blog post, we’ll discuss the methods we used to ensure a successful launch, including:
Netflix traffic ebbs and flows throughout the day in a sinusoidal pattern. New content or national events may drive brief spikes, but, by and large, traffic is usually smoothly increasing or decreasing. An exception to this trend is when we redirect traffic between AWS data centers during regional evacuations, which leads to sudden spikes in traffic in multiple regions. Region evacuations can occur at any time, for a variety of reasons.
While evaluating options to test anticipated load and evaluate our ad selection algorithms at scale, we realized that mimicking member viewing behavior in combination with the seasonality of our organic traffic with abrupt regional shifts were important requirements. Replaying real traffic and making it appear as Basic with ads traffic was a better solution than artificially simulating Netflix traffic. Replay traffic enabled us to test our new systems and algorithms at scale before launch, while also making the traffic as realistic as possible.
A key objective of this initiative was to ensure that our customers were not impacted. We used member viewing habits to drive the simulation, but customers did not see any ads as a result. Achieving this goal required extensive planning and implementation of measures to isolate the replay traffic environment from the production environment.
Netflix’s data science team provided projections of what the Basic with ads subscriber count would look like a month after launch. We used this information to simulate a subscriber population through our AB testing platform. When traffic matching our AB test criteria arrived at our playback services, we stored copies of those requests in a Mantis stream.
Next, we launched a Mantis job that processed all requests in the stream and replayed them in a duplicate production environment created for replay traffic. We set the services in this environment to “replay traffic” mode, which meant that they did not alter state and were programmed to treat the request as being on the ads plan, which activated the components of the ads system.
The replay traffic environment generated responses containing a standard playback manifest, a JSON document containing all the necessary information for a Netflix device to start playback. It also included metadata about ads, such as ad placement and impression-tracking events. We stored these responses in a Keystone stream with outputs for Kafka and Elasticsearch. A Kafka consumer retrieved the playback manifests with ad metadata and simulated a device playing the content and triggering the impression-tracking events. We used Elasticsearch dashboards to analyze results.
Ultimately, we accurately simulated the projected Basic with ads traffic weeks ahead of the launch date.
To fully replay the traffic, we first validated the idea with a small percentage of traffic. The Mantis query language allowed us to set the percentage of replay traffic to process. We informed our engineering and business partners, including customer support, about the experiment and ramped up traffic incrementally while monitoring the success and error metrics through Lumen dashboards. We continued ramping up and eventually reached 100% replay. At this point we felt confident to run the replay traffic 24/7.
To validate handling traffic spikes caused by regional evacuations, we utilized Netflix’s region evacuation exercises which are scheduled regularly. By coordinating with the team in charge of region evacuations and aligning with their calendar, we validated our system and third-party touchpoints at 100% replay traffic during these exercises.
We also constructed and checked our ad monitoring and alerting system during this period. Having representative data allowed us to be more confident in our alerting thresholds. The ads team also made necessary modifications to the algorithms to achieve the desired business outcomes for launch.
Finally, we conducted chaos experiments using the ChAP experimentation platform. This allowed us to validate our fallback logic and our new systems under failure scenarios. By intentionally introducing failure into the simulation, we were able to identify points of weakness and make the necessary improvements to ensure that our ads systems were resilient and able to handle unexpected events.
The availability of replay traffic 24/7 enabled us to refine our systems and boost our launch confidence, reducing stress levels for the team.
The above summarizes three months of hard work by a tiger team consisting of representatives from various backend teams and Netflix’s centralized SRE team. This work helped ensure a successful launch of the Basic with ads tier on November 3rd.
To briefly recap, here are a few of the things that we took away from this journey:
Replay traffic at Netflix has numerous applications, one of which has proven to be a valuable tool for development and launch readiness. The Resilience team is streamlining this simulation strategy by integrating it into the CHAP experimentation platform, making it accessible for all development teams without the need for extensive infrastructure setup. Keep an eye out for updates on this.
Ensuring the Successful Launch of Ads on Netflix was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.
Post Syndicated from Ted Tanner original https://aws.amazon.com/blogs/security/updated-whitepaper-available-architecting-for-pci-dss-segmentation-and-scoping-on-aws/
Amazon Web Services (AWS) has re-published the whitepaper Architecting for PCI DSS Scoping and Segmentation on AWS to provide guidance on how to properly define the scope of your Payment Card Industry (PCI) Data Security Standard (DSS) workloads that are running in the AWS Cloud. The whitepaper has been refreshed to include updated AWS best practices and technologies, and updates that are applicable to the new PCI DSS v4.0 requirements. The whitepaper looks at how to define segmentation boundaries between your in-scope and out-of-scope resources by using cloud-based AWS services.
The whitepaper is intended for engineers and solution builders, but it also serves as a guide for Qualified Security Assessors (QSAs) and internal security assessors (ISAs) to better understand the different segmentation controls that are available within AWS products and services, along with associated scoping considerations.
Compared to on-premises environments, software-defined networking on AWS transforms the scoping process for applications by providing additional segmentation controls beyond network segmentation. Thoughtful design of your applications and selection of security-impacting services for implementing your required controls can reduce the number of systems and services in your cardholder data environment (CDE).
The whitepaper is based on the PCI Council’s Information Supplement: Guidance for PCI DSS Scoping and Network Segmentation.
If you have questions or want to learn more, contact your account representative, or leave a comment below.
Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.
Post Syndicated from Taylor Blau original https://github.blog/2023-06-01-highlights-from-git-2-41/
The open source Git project just released Git 2.41 with features and bug fixes from over 95 contributors, 29 of them new. We last caught up with you on the latest in Git back when 2.40 was released.
To celebrate this most recent release, here’s GitHub’s look at some of the most interesting features and changes introduced since last time.
At the heart of every Git repository lies a set of objects. For the unfamiliar, you can learn about the intricacies of Git’s object model in this post. In general, objects are the building blocks of your repository. Blobs represent the contents of an individual file, and trees group many blobs (and other trees!) together, representing a directory. Commits tie everything together by pointing at a specific tree, representing the state of your repository at the time when the commit was written.
Git objects can be in one of two states, either “reachable” or “unreachable.” An object is reachable when you can start at some branch or tag in your repository and “walk” along history, eventually ending up at that object. Walking merely means looking at an individual object, and seeing what other objects are immediately related to it. A commit has zero or more other commits which it refers to as parents. Conversely, trees point to many blobs or other trees that make up their contents.
Objects are in the “unreachable” state when there is no branch or tag you could pick as a starting point where a walk like the one above would end up at that object. Every so often, Git decides to remove some of these unreachable objects in order to compress the size of your repository. If you’ve ever seen this message:
Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.
or run git gc directly, then you have almost certainly removed unreachable objects from your repository.
But Git does not necessarily remove unreachable objects from your repository the first time git gc is run. Since removing objects from a live repository is inherently risky1, Git imposes a delay. An unreachable object won’t be eligible for deletion until it has not been written since a given (via the –prune argument) cutoff point. In other words, if you ran git gc --prune=2.weeks.ago, then:
Until Git 2.37, Git kept track of the last write time of unreachable objects by storing them as loose copies of themselves, and using the object file’s mtime as a proxy for when the object was last written. However, storing unreachable objects as loose until they age out can have a number of negative side-effects. If there are many unreachable objects, they could cause your repository to balloon in size, and/or exhaust the available inodes on your system.
Git 2.37 introduced “cruft packs,” which store unreachable objects together in a packfile, and use an auxiliary *.mtimes file stored alongside the pack to keep track of object ages. By storing unreachable objects together, Git prevents inode exhaustion, and allows unreachable objects to be stored as deltas.
The figure above shows a cruft pack, along with its corresponding *.idx and *.mtimes file. Storing unreachable objects together allows Git to store your unreachable data more efficiently, without worry that it will put strain on your system’s resources.
In Git 2.41, cruft pack generation is now on by default, meaning that a normal git gc will generate a cruft pack in your repository. To learn more about cruft packs, you can check out our previous post, “Scaling Git’s garbage collection.”
[source]
Starting in Git 2.41, you may notice a new kind of file in your repository’s .git/objects/pack directory: the *.rev file.
This new file stores information similar to what’s in a packfile index. If you’ve seen a file in the pack directory above ending in *.idx, that is where the pack index is stored.
Pack indexes map between the positions of all objects in the corresponding pack among two orders. The first is name order, or the index at which you’d find a given object if you sorted those objects according to their object ID (OID). The other is pack order, or the index of a given object when sorting by its position within the packfile itself.
Git needs to translate between these two orders frequently. For example, say you want Git to print out the contents of a particular object, maybe with git cat-file -p. To do this, Git will look at all *.idx files it knows about, and use a binary search to find the position of the given object in each packfile’s name order. When it finds a match, it uses the *.idx to quickly locate the object within the packfile itself, at which point it can dump its contents.
But what about going the other way? How does Git take a position within a packfile and ask, “What object is this”? For this, it uses the reverse index, which maps objects from their pack order into the name order. True to its name, this data structure is the inverse of the packfile index mentioned above.
The figure above shows a representation of the reverse index. To discover the lexical (index) position of, say, the yellow object, Git reads the corresponding entry in the reverse index, whose value is the lexical position. In this example, the yellow object is assumed to be the fourth object in the pack, so Git reads the fourth entry in the .rev file, whose value is 1. Reading the corresponding value in the *.idx file gives us back the yellow object.
In previous versions of Git, this reverse index was built on-the-fly by storing a list of pairs (one for each object, each pair contains that object’s position in name and packfile order). This approach has a couple of drawbacks, most notably that it takes time and memory in order to materialize and store this structure.
In Git 2.31, the on-disk reverse index was introduced. It stores the same contents as above, but generates it once and stores the result on disk alongside its corresponding packfile as a *.rev file. Pre-computing and storing reverse indexes can dramatically speed-up performance in large repositories, particularly for operations like pushing, or determining the on-disk size of an object.
In Git 2.41, Git will now generate these reverse indexes by default. This means that the next time you run git gc on your repository after upgrading, you should notice things get a little faster. When testing the new default behavior, the CPU-intensive portion of a git push operation saw a 1.49x speed-up when pushing the last 30 commits in torvalds/linux. Trivial operations, like computing the size of a single object with git cat-file --batch='%(objectsize:disk)' saw an even greater speed-up of nearly 77x.
To learn more about on-disk reverse indexes, you can check out another previous post, “Scaling monorepo maintenance,” which has a section on reverse indexes.
[source]
Credential helpers haven’t had a mechanism to understand additional information necessary to generate a credential, like OAuth scopes, which are typically passed over the WWW-Authenticate header.
In Git 2.41, the credential helper protocol is extended to support passing WWW-Authenticate headers between credential helpers and the services that they are trying to authenticate with. This can be used to allow services to support more fine-grained access to Git repositories by letting users scope their requests.
[source]
Previous versions of Git allowed this comparison by running two reachability queries: git rev-list --count main..my-feature (to count the number of commits unique to my-feature) and git rev-list --count my-feature..main (the opposite). This works fine, but involves two separate queries, which can be awkward. If comparing many branches against a common base (like on the /branches page above), Git may end up walking over the same commits many times.
In Git 2.41, you can now ask for this information directly via a new for-each-ref formatting atom, %(ahead-behind:<base>). Git will compute its output using only a single walk, making it far more efficient than in previous versions.
For example, suppose I wanted to list my unmerged topic branches along with how far ahead and behind they are relative to upstream’s mainline. Before, I would have had to write something like:
$ git for-each-ref --format='%(refname:short)' --no-merged=origin/HEAD \
refs/heads/tb |
while read ref
do
ahead="$(git rev-list --count origin/HEAD..$ref)"
behind="$(git rev-list --count $ref..origin/HEAD)"
printf "%s %d %d\n" "$ref" "$ahead" "$behind"
done | column -t
tb/cruft-extra-tips 2 96
tb/for-each-ref--exclude 16 96
tb/roaring-bitmaps 47 3
which takes more than 500 milliseconds to produce its results. Above, I first ask git for-each-ref to list all of my unmerged branches. Then, I loop over the results, computing their ahead and behind values manually, and finally format the output.
In Git 2.41, the same can be accomplished using a much simpler invocation:
$ git for-each-ref --no-merged=origin/HEAD \
--format='%(refname:short) %(ahead-behind:origin/HEAD)' \
refs/heads/tb/ | column -t
tb/cruft-extra-tips 2 96
tb/for-each-ref--exclude 16 96
tb/roaring-bitmaps 47 3
[...]
That produces the same output (with far less scripting!), and performs a single walk instead of many. By contrast to earlier versions, the above takes only 28 milliseconds to produce output, a more than 17-fold improvement.
[source]
git fetch, Git’s output will contain information about which references were updated from the remote, like:
+ 4aaf690730..8cebd90810 my-feature -> origin/my-feature (forced update)
While convenient for a human to read, it can be much more difficult for a machine to parse. Git will shorten the reference names included in the update, doesn’t print the full before and after values of the reference being updated, and columnates its output, all of which make it more difficult to script around.
In Git 2.41, git fetch can now take a new --porcelain option, which changes its output to a form that is much easier to script around. In general, the --porcelain output looks like:
<flag> <old-object-id> <new-object-id> <local-reference>
When invoked with --porcelain, git fetch does away with the conveniences of its default human readable output, and instead emits data that is much easier to parse. There are four fields, each separated by a single space character. This should make it much easier to script around the output of git fetch.
git fetch, Git 2.41 has another new feature that can improve its performance: fetch.hideRefs. Before we get into it, it’s helpful to recall our previous coverage of git rev-list’s --exclude-hidden option. If you’re new around here, don’t worry: this option was originally introduced to improve the performance of Git’s connectivity check, the process that checks that an incoming push is fully connected, and doesn’t reference any objects that the remote doesn’t already have, or are included in the push itself.
Git 2.39 sped-up the connectivity check by ignoring parts of the repository that weren’t advertised to the pusher: its hidden references. Since these references weren’t advertised to the pusher, it’s unlikely that any of these objects will terminate the connectivity check, so keeping track of them is usually just extra bookkeeping.
Git 2.41 introduces a similar option for git fetch on the client side. By setting fetch.hideRefs appropriately, you can exclude parts of the references in your local repository from the connectivity check that your client performs to make sure the server didn’t send you an incomplete set of objects.
When checking the connectedness of a fetch, the search terminates at the branches and tags from any remote, not just the one you’re fetching from. If you have a large number of remotes, this can take a significant amount of time, especially on resource-constrained systems.
In Git 2.41, you can narrow the endpoints of the connectivity check to focus just on the remote you’re fetching from. (Note that transfer.hideRefs values that start with ! are interpreted as un-hiding those references, and are applied in reverse order.) If you’re fetching from a remote called $remote, you can do this like so:
$ git -c fetch.hideRefs=refs -c fetch.hideRefs=!refs/remotes/$remote \
fetch $remote
The above first hides every reference from the connectivity check (fetch.hideRefs=refs) and then un-hides just the ones pertaining to that specific remote (fetch.hideRefs=!refs/remotes/$remote). On a resource constrained machine with repositories that have many remote tracking references, this takes the time to complete a no-op fetch from 20 minutes to roughly 30 seconds.
[source]
git fsck. This tool is used to check that the objects in your repository are intact and connected. In other words, that your repository doesn’t have any corrupt or missing objects.git fsck can also check for more subtle forms of repository corruption, like malicious looking .gitattributes or .gitmodules files, along with malformed objects (like trees that are out of order, or commits with a missing author). The full suite of checks it performs can be found under the fsck. configuration.
In Git 2.41, git fsck learned how to check for corruption in reachability bitmaps and on-disk reverse indexes. These checks detect and warn about incorrect trailing checksums, which indicate that the preceding data has been mangled. When examining on-disk reverse indexes, git fsck will also check that the *.rev file holds the correct values.
To learn more about the new kinds of fsck checks implemented, see the git fsck documentation.
That’s just a sample of changes from the latest release. For more, check out the release notes for 2.41, or any previous version in the Git repository.
git gc is about to remove. If a new object is written which references the deleted one, the repository can become corrupt. If you’re curious to learn more, this section is a good place to start. ↩
Post Syndicated from original https://www.backblaze.com/blog/2023-state-of-the-backup-as-data-needs-grow-backups-need-to-fill-the-gaps/
Each passing year brings with it a June, and with that comes a Backup Awareness Month. For those that are new to the blog, each June we partner with The Harris Poll to gauge the state of backups in the United States, by asking the simple question: “How often do you back up all the data on your computer?” (And a handful of other questions, too.) This post marks our survey’s 15th anniversary, and as you can imagine, the backup world has changed a lot in that time.
If you’ve followed our previous State of the Backup articles, you’ll know that we usually have a burning question that we want to answer. In 2021 and 2022, we were interested in the “best backer upper.” This year’s focus touches on the different “cloud” services that respondents use, and their confidence level that those services are providing them with the protection they crave—nay, deserve.
And that’s not just our (totally) normal love of backup coming out. It comes as news to no one that the world has shifted to a more digital environment, the ways we store, use, and manage data have changed. We have our data at more touchpoints in our lives, and cloud-based sync services are readily available. Being able to capture a full backup for all those various uses—business and personal—means that the need for automatic, unlimited solutions that you can access from anywhere is no longer a “nice to have.” It’s essential.
This is one of my favorite graphs. Now with 15 years of data, we look at our headline question: How often do you back up all the data on your computer? This year we’re pleased that daily backups have not decreased and remain at similar levels to last year. In 2023, 11% of Americans who own a computer backed up their data at least once a day compared to 10% in 2022 and 11% in 2021. Weekly (8% in 2023 and 7% for both previous years) and monthly (15% in 2023 and 13% and 14% in the two previous years) backups among those who own a computer are similar to the past years as well.
The number of people who have never backed up data also remains at similar levels, with 18% of Americans who own a computer saying they’ve never backed up data in 2023 compared to 20% saying the same in the prior two years.
If you’ve hung around the Backblaze blog before, you likely have heard me (Yev) saying that the main competitor we have to our computer backup service is apathy. While we’re not (yet) seeing a statistically significant drop in those “never” numbers, we love to see that this category isn’t growing. And, we’ve got big plans to get more folks backing up in the future—we love using polls like this to learn more about how to move the needle. You can see from the graph below that, if you compare to 2008, we’ve certainly seen change over time.
Since some people prefer the raw data, and you can find it here, lovingly copied from Google Sheets:
While we’re past March 14, Pi Day is one of my favorite holidays to celebrate and also one of my favorite chart types. Here we have pie charts comparing the 2008 to 2023 data:
We love seeing the daily section growing while the never section shrinks—that’s progress. And, like we stated above, we view it as a big indicator of the massive shift in the ways people are using data.
Of course, we would love to see the daily backups skyrocketing year over year, because at the end of the day, if people aren’t backing up frequently, they are at risk of losing at least that day’s worth of data, which is one of the things that Backblaze Computer Backup is trying to help people avoid.
A few years ago we got curious about who is “best” at backing up. In 2021, we saw some statistical significance that indicated women between 35-44 years of age (21% likely to backup versus 9% of those 18-34 and 6% of those 55-64), and those who live in the Western United States (17% more likely to back up vs. the South and Midwest at 9% and 7%, respectively) were more likely to be computer owners who backed up once a day or more. When we checked in 2022, we found no statistical difference between the categories—but, we’ll explain why we keep checking in on this profile.
It’s worth taking a time-out for a stats mini-lesson, because we Backblaze plebeians did see some interesting fluctuations in the data, and the awesome experts at The Harris Poll let us know that these fluctuations are to be expected. Here’s why: When you’re talking about surveys rather than a census of a group, there is a sampling error that is expected just because you get a different group of people responding over time. We do statistical testing to see if the fluctuation could be due to chance (we just randomly got a different group of people) or if something has really changed in the group over time. In our survey, performed with 95% confidence, that means if we collected the survey 100 times we would report differences that would only occur in five of those surveys by chance assuming no change in the group. (Here’s an article that explains in more detail, if you’d like to dig in.)
The TL:DR of all that is that for a category to show statistical significance, it needs to be 5% greater or less than the total average of the parent category. In our parent category of computer owners who back up once a day or more,” we have a total average of 11% for 2023. Women between the ages of 35–44—clock in at 15%. So, while that’s not statistically significant, it’s certainly worth monitoring in our future surveys.
And that means that this year, the title is up for grabs! Get geared up for next year, folks: We love healthy competition about better backups.
When we started Backblaze 16 years ago, Amazon AWS had only recently started marketing Amazon Elastic Cloud Compute (in 2006, folks). Now, “the cloud” is a household name although it’s something of an amorphous concept for many—but there’s no denying that the cloud is accessible to consumers and companies alike.
Here are some indicative stats:
Some of these facts reveal very important questions about backup and sync in relation to our backup questions. We’ve talked about the differences in the past, and here we see folks who are saying that they use a drive or sync service as their backup method.
So, when those Americans using cloud-based systems think they’re backing up each day, in all likelihood, at least some of them are describing another type of data storage—syncing data across devices or storing data on a cloud drive. But, without an additional copy of your data, you aren’t truly backing up.
Still, even with more people backing up, overall data confidence is low, with fewer than one in five Americans (17%) stating they are absolutely certain that their most important computer files are safely backed up somewhere. Even more telling? 12% of Americans who use a cloud service, external hard drive, or a network attached storage (NAS) as a primary method to back up data say they are not confident the method they use is set up to protect all of the data on their computer.
That number means we need to get more people not only backing up but testing their restores. If you’ve never tested your restores, we highly recommend it. Not only does it let you see exactly where you’re storing your files and what it takes to restore them, but it also shows you how long it will take to get your files back online in the event of data loss. It’s also one of the main differentiators between “cloud” services—the ease with which you can get all your data back is a good selection criteria.
The whole point of a backup is to make sure that you never lose critical data. We certainly want and value this for personal recovery—check out our article about how a digital go bag can help you prepare for natural disasters and the like.
But, if you’re a business owner, any disruption in data or file loss can mean that your business is off-line. Business continuity plans help you understand how to get back online with minimal stress and as soon as possible. Especially if you’re a small or medium-sized business, you know how important that is! Services like Instant Business Recovery are a great tool in that respect: They help your business prepare for scenarios in which you might have outages and to get back online as quickly as possible.
When looking at computer owners who use a cloud backup service like Backblaze as their primary backup we find that:
Why is this important? Well, different types of backups mean that you can recover things in different ways. For instance, a bare metal recovery is designed to take you from a computer with nothing on it—not even an operating system—to a fully-functional computer with all your files. Does your backup service support those file types?
Another example: a grandfather-father-son or a full vs. incremental backup may help you save valuable storage space when it comes to backing up (especially when your backup service may have limits or charge you based on storage space, as some folks indicate above).
Finally, our old pal, the 3-2-1 backup strategy, recommends both an on-site and an off-site copy of your data, which means that whichever method you use to backup, you’re going to want to store those backups in different and accessible ways.
It’s not only a question of knowing if your backup service is automatically backing up all the data on your computer—you also want to know where that data is being stored, what file types are supported, if or how you need to set “rules,” and more. There are positives and negatives to each backup strategy, of course, but the numbers above show us that we have some work to do to help people know where to start when it comes to building an effective plan. We see a detailed education campaign about what Backblaze backs up brewing in our future!
When looking at the data loss statistics of Americans who own a computer:
This year over half of Americans who own a computer reported being affected by a security incident, and that number is in line with last year (54% this year vs. 53% last year). Over the years, ransomware has increasingly become a “when, not if” situation, which means that seemingly simple things, like making strong passwords, using virtual private networks (VPNs), and knowing how to recognize a phishing attack are more important than ever. Prevention is only half the battle of course—and a good backup and recovery plan is the other.
What’s more, these things become even more interesting when you think about data loss as it comes to our working environment. As we see modern work patterns change, we see more people working on their home networks and using mobile phones. As the workplace has shifted to include more remote and hybrid environments, employers are having to think about what it looks like to secure data across dispersed locations. Not only do you have lots of folks working on software as a service (SaaS) tools that include synced or shared drives (and have their own backup strategy demands), but you also need to capture folks saving on their local drives—which means you need individual workstation backup to ensure that you have a complete disaster recovery (DR) plan in place.
Fifteen years into this Harris Poll (and 16 years into being Backblaze!), we are still driven to understand the world of backup to support our ultimate goal: making storing and using data astonishingly easy. The work of taking these disparate data points (70% of Americans who own a computer have accidentally deleted something) and turning them into solutions (hey, an automatic backup means that you can get it back) is something we can’t do without understanding the world and how it changes.
And, this is true on an individual level, too. When you’re building your backup solution, what you need for your personal backups is likely to be different from what you’ll need for your job. Maybe a network attached storage (NAS) device lets you bring all of your family’s household devices to a single place to manage and backup data, keeping you away from those pesky iCloud data storage limits. If you own a business, maybe you want to consider how long to keep your backups and how much it costs to store data in different formats like on hard drives or in the cloud.
We hope you enjoy seeing the big picture as much as we do, and we hope this information ultimately helps you to find the best backup service for your data. Jump into the comments and let us know what you think!
This year’s survey was conducted online within the United States by The Harris Poll on behalf of Backblaze from April 25–27, 2023, among 2,050 adults ages 18+, among whom 1,857 own a computer. The sampling precision of Harris online polls is measured by using a Bayesian credible interval. For this study, the sample data is accurate to within +/- 2.8 percentage points using a 95% confidence level.
Prior year’s surveys were conducted online by The Harris Poll on behalf of Backblaze among U.S. adults ages 18+ who own a computer in May 19–23, 2022 (n=1,861); May 12–14, 2021 (n=1,870); June 1–3, 2020 (n=1,913); June 6–10, 2019 (n=1,858); June 5–7, 2018 (n=1,871); May 19–23, 2017 (n=1,954); May 13–17, 2016 (n=1,920); May 15–19, 2015 (n=2,009); June 2-4, 2014 (n=1,991); June 13–17, 2013 (n=1,952); May 31–June 4, 2012 (n=2,176); June 28–30, 2011 (n=2,209); June 3–7, 2010 (n=2,051); May 13–14, 2009 (n=2,154); and May 27–29, 2008 (n=2,723).
For complete survey methodologies, including weighting variables and subgroup sample sizes, please contact Backblaze.
The post 2023 State of the Backup: As Data Needs Grow, Backups Need to Fill the Gaps appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.
Post Syndicated from BeardedTinker original https://www.youtube.com/watch?v=WmyImS1vR1g
Post Syndicated from original https://lwn.net/Articles/933043/
Like most other distributions, the Debian project decided to end the
separation between the root and /usr filesystems years ago.
Unlike most others, though, Debian is still working on the implementation
of this decision. The upcoming Debian 12 (“bookworm”) release will
feature a merged /usr in most respects, but there are a couple of
nagging issues that threaten to stretch this transition out for some time
yet.
Post Syndicated from Caitlin Condon original https://blog.rapid7.com/2023/06/01/rapid7-observed-exploitation-of-critical-moveit-transfer-vulnerability/
Rapid7 managed services teams are observing exploitation of a critical vulnerability in Progress Software’s MOVEit Transfer solution across multiple customer environments. We have observed an uptick in related cases since the vulnerability was disclosed publicly yesterday (May 31, 2023); file transfer solutions have been popular targets for attackers, including ransomware groups, in recent years. We strongly recommend that MOVEit Transfer customers prioritize mitigation on an emergency basis.
Progress Software published an advisory on Wednesday, May 31, 2023 warning of a critical SQL injection vulnerability in their MOVEit Transfer solution. The vulnerability, which currently does not have a CVE, is a SQL injection flaw that allows for “escalated privileges and potential unauthorized access” on target systems. While the advisory does not explicitly confirm the vulnerability was exploited by threat actors as a zero-day, Progress Software is advising MOVEit customers to check for indicators of unauthorized access over “at least the past 30 days,” which implies that attacker activity was detected before the vulnerability was disclosed.
As of May 31, there were roughly 2,500 instances of MOVEit Transfer exposed to the public internet, the majority of which look to be in the United States. Rapid7 has previously analyzed similar SQLi-to-RCE flaws in network edge systems; these types of vulnerabilities can provide threat actors with initial access to corporate networks.
Our teams have so far observed the same webshell name in multiple customer environments, which may indicate automated exploitation. Rapid7 analyzed a sample webshell payload associated with successful exploitation. The webshell code would first determine if the inbound request contained a header named X-siLock-Comment, and would return a 404 "Not Found" error if the header was not populated with a specific password-like value. As of June 1, 2023, all instances of Rapid7-observed MOVEit Transfer exploitation involve the presence of the file human2.aspx in the wwwroot folder of the MOVEit install directory.
We will update this section as our investigations progress.
The MOVEit Transfer advisory has contradictory wording on patch availability, but as of June 1, it does appear that fixed versions of the software are available. Patches should be applied on an emergency basis. Per the MOVEit advisory published on May 31, 2023, organizations should look for indicators of compromise dating back at least a month.
The advisory also advises customers to modify firewall rules to deny HTTP and HTTPs traffic to MOVEit Transfer on ports 80 and 443.
For InsightVM and Nexpose customers, an authenticated vulnerability check is expected to ship in the June 1, 2023 content release.
Post Syndicated from original https://lwn.net/Articles/933466/
A new version of NixOS, which is a Linux
distribution based on the Nix package
manager, has been released: NixOS 23.05
is now available. The release
notes list numerous updates, including Nix 2.13, Linux 6.1,
glibc 2.37,
Cinnamon 5.6, GNOME 44, and KDE Plasma 5.27.
The 23.05 release was made possible due to the efforts of 1867
contributors, who authored 36566 commits since the previous
release.
Our thanks go the contributors who also take care of the continued
stability and security of our stable release.NixOS is already known as the most up to date distribution while also being
the
distribution with the most packages. This release saw 16240 new
packages and 13524 updated packages in Nixpkgs. We also
removed 13466
packages in an effort to keep the package set maintainable and secure.
In addition to packages the NixOS distribution also features modules and
tests that make it what it is. This release brought 282 new modules and
removed 183. In that process we added 2882 options and removed 728.
Post Syndicated from original https://lwn.net/Articles/933465/
Security updates have been issued by Debian (libwebp, openssl, sssd, and texlive-bin), Fedora (bitcoin-core, editorconfig, edk2, mod_auth_openidc, pypy, pypy3.9, python3.10, and python3.8), Red Hat (kernel, openssl, pcs, pki-core:10.6, and qatzip), SUSE (chromium, ImageMagick, openssl-1_1, and tiff), and Ubuntu (cups, libvirt, and linux, linux-aws, linux-aws-5.15, linux-azure, linux-azure-5.15,
linux-gcp, linux-gcp-5.15, linux-gke, linux-gke-5.15, linux-gkeop,
linux-hwe-5.15, linux-hwe-5.19, linux-ibm, linux-kvm, linux-lowlatency,
linux-lowlatency-hwe-5.15, linux-oracle, linux-oracle-5.15, linux-raspi).
Post Syndicated from The Hook Up original https://www.youtube.com/watch?v=4ldZCNkksgw
Post Syndicated from Rapid7 original https://blog.rapid7.com/2023/06/01/rapid7-sales-director-devin-poulter-on-building-a-career-as-an-account-executive/
Devin Poulter is a Sales Director with over 20 years of experience in the tech industry. Recently, we asked him a few questions to learn more about how he built his career, what it’s like to lead a team at Rapid7, and more.
You’ve been in software sales for most of your career, what can you tell us about being successful as an Account Executive?
I’d say building your personal brand is really important for success as an Account Executive. Being able to have open and honest conversations with customers and really focus on where you can (or sometimes, where you can not) add value is something people will remember you by.
Years down the line, when you are either introducing a new product, or representing a different company, that decision maker you are reaching out to will probably remember how you treated them previously. If you took the time to create a positive relationship, they are more likely to take your call and listen because you’ve built that foundation of trust. I’ve seen that play out time and time again, and I think it’s important for people building their career to equally focus on their personal brand as well as the product or company brand you’re representing.
What stands out to you about Rapid7 when it comes to building a sales career?
Well, the cybersecurity market is white hot as threats and bad actors continue to find new ways of exposing vulnerable organizations. It’s a market that is attracting a lot of competition as new companies and startups emerge—but where I see the most value and sustainability is with organizations like Rapid7 that can meet customers where they are—and grow with them.
When we think about our customers, particularly the CISO, their role has become increasingly challenging, and will continue to be so. Between an evolving threat landscape, a struggle to find talent for in-house security teams, and a volatile economic landscape that puts pressure on budgets—they need partners who can help them do more with less resources. At Rapid7, we’re making it everyone’s mission to think about these customer challenges and ensure we’re bringing the best and most impactful solutions to the table so we can help customers build a more secure digital future.
Additionally, Rapid7 isn’t a one trick pony for a CISO. Our Account Executives can sell across the full platform of our offerings, enabling them to be a true partner to customers and add value based on their unique needs and challenges. When it comes to building a successful career in sales (and especially in building your personal brand), this is something that helps our reps deliver the utmost value to customers with solutions that can grow and scale with them.
Finally, Rapid7’s mission is inspiring. The need for our products is something that is never going to go away. All around the world, there are bad actors who are constantly trying to take advantage of individuals and businesses to make a quick profit. When you think of the impact a hacker or vulnerability can have, it’s everything from financial losses to entire infrastructure systems and communities getting disrupted. We’re giving organizations the tools and resources to protect themselves from the bad guys, and helping their security teams work more efficiently and keep their data—and more importantly, people, safe from harm.
What’s it like to lead a team at Rapid7?
As a leader, I’m excited for the opportunity to help mature my team and see them succeed. That’s always been a big motivator for me—and it still is today. As I’ve progressed through my career, I’d say it’s equally rewarding to see the growth and success of the business, and the impact we’re making on our customers.
Today, my motivation goes far beyond hitting our numbers and going to Club. Being part of a company that’s creating real impact every day, whether it’s through our products, our open source communities, or our work in public policy and governance, it’s fulfilling to be part of a journey that’s being recognized and giving us a reputation in the market for being really innovative.
When people are proactively approaching you at a conference or a show, and commenting on the work Rapid7 is doing, it instills a lot of pride in your work and reminds you of how important our mission is. A company that gives you that kind of fulfillment, along with opportunities to grow, is the real difference between finding your next job and finding a career home.
What do you look for in an Account Executive?
What I look for is genuine curiosity, and the ability to critically think about challenges. I don’t need someone to tell me what they think I want to hear or give a generic answer. In order to uncover this in an interview, I’ll often share a real world situation and ask their opinion on how they would approach it. This gives me an opportunity to see what kind of follow up questions they have, how they think, and how they might approach challenges on the job. As far as their answer, I’m not looking for a right or wrong response, it’s more about learning how they think. Sometimes, I’ll even help coach or give the person feedback to help them understand the context or maybe frame it in a way they weren’t thinking about originally. The candidates that stand out the most are the ones who are able to ask the right questions, can articulate their opinion, and who are also coachable and open to feedback. Knowledge of our products and industry can always be taught, but having the right mindset is critical to be able to help them grow and be successful in the role.
At Rapid7, when we say we are open to ideas and encourage people to challenge convention—we really do mean it, and we respect the different perspectives that are brought to the table. This kind of culture is what helps sustain the business and enables you to do some really cool things in your field.
Looking to find your career home as an Account Executive?
Explore what it’s like to work as part of our sales team here: https://careers.rapid7.com/sales
Post Syndicated from Sam Rhea original http://blog.cloudflare.com/deprecating-railgun/
Cloudflare will deprecate the Railgun product on January 31, 2024. At that time, existing Railgun deployments and connections will stop functioning. Customers have the next eight months to migrate to a supported Cloudflare alternative which will vary based on use case.
Cloudflare first launched Railgun more than ten years ago. Since then, we have released several products in different areas that better address the problems that Railgun set out to solve. However, we shied away from the work to formally deprecate Railgun.
That reluctance led to Railgun stagnating and customers suffered the consequences. We did not invest time in better support for Railgun. Feature requests never moved. Maintenance work needed to occur and that stole resources away from improving the Railgun replacements. We allowed customers to deploy a zombie product and, starting with this deprecation, we are excited to correct that by helping teams move to significantly better alternatives that are now available in Cloudflare’s network.
We know that this will require migration effort from Railgun customers over the next eight months. We want to make that as smooth as possible. Today’s announcement features recommendations on how to choose a replacement, how to get started, and guidance on where you can reach us for help.
Cloudflare’s reverse proxy secures and accelerates your applications by placing a Cloudflare data center in over 285+ cities between your infrastructure and your audience. Bad actors attempting to attack your applications hit our network first where products like our WAF and DDoS mitigation service stop them. Your visitors and users connect to our data centers where our cache can serve them content without the need to reach all the way back to your origin server.
For some customers, your infrastructure also runs on Cloudflare’s network in the form of Cloudflare Workers. Others maintain origin servers running on anything from a Raspberry Pi to a hyperscale public cloud. In those cases, Cloudflare needs to connect to that infrastructure to grab new content that our network can serve from our cache to your audience.
However, some content cannot be cached. Dynamically-generated or personalized pages can change for every visitor and every session. Cloudflare Railgun aimed to solve that by determining what was the minimum amount of content that changed and attempting to only send that difference in an efficient transfer – a form of delta compression. By reducing the amount of content that needed to be sent to Cloudflare’s network, we could accelerate page loads for end users.
Railgun accomplishes this goal by running a piece of software inside the customer’s environment, the Railgun listener, and a corresponding service running in Cloudflare’s network, the Railgun sender. The pair establish a permanent TCP connection. The listener keeps track of the most recent version of a page that was requested. When a request arrives for a known page, the listener sends an HTTP request to the origin server, determines what content changed, and then compresses and sends only the delta to the sender in Cloudflare’s network.
The last major release of Railgun took place eight years ago in 2015. However, products should not be deprecated just because active development stops. We believe that a company should retire a product only when:
Hundreds of customers still use Railgun today and the service has continued to run over the last decade without too much involvement from our team. That relative stability deterred us from pushing customers to adopt newer technologies that solved the same problems. As a result, we kept Railgun in a sort of maintenance mode for the last few years.
Cloudflare’s network has evolved in the eight years since the last Railgun release. We deploy hardware and run services in more than 285 cities around the world, nearly tripling the number of cities since Railgun was last updated. The hardware itself also advanced, becoming more efficient and capable.
The software platform of Cloudflare’s network developed just as fast. Every data center in Cloudflare’s network can run every service that we provide to our customers. These services range from our traditional reverse proxy products to forward proxy services like Zero Trust to our compute and storage platform Cloudflare Workers. Supporting such a broad range of services requires a platform that can adapt to the requirements of the evolving needs of these products.
Maintaining Railgun, despite having better alternatives, creates a burden on our ability to continue investing in new solutions. Some of these tools that power Railgun are themselves approaching an end of life state. Others will likely present security risks that we are not comfortable accepting in the next few years.
We considered several options before deciding on deprecation. First, we could accept the consequences of inaction, leaving our network in a worse state and our Railgun customers in purgatory. Second, we could run Railgun on dedicated infrastructure and silo it from the rest of our network. However, that would violate our principle that every piece of hardware in Cloudflare runs every service.
Third, we could spin up a new engineering team and rebuild Railgun from scratch in a modern way. Doing so would take away from resources we could otherwise invest in newer technologies. We also believe that existing, newer products from Cloudflare solve the same problems that Railgun set out to address. Rebuilding Railgun would take away from our ability to keep shipping and would duplicate better features already released in other products. As a result, we have decided to deprecate Railgun.
Railgun addressed a number of problems for our customers at launch. Today, we have solutions available that solve the same range of challenges in significantly improved ways.
We do not have an exact like-for-like successor for Railgun. The solutions that solve the same set of problems have also evolved with our customers. Different use cases that customers deploy Railgun to address will map to different solutions available in Cloudflare today. We have broken out some of the most common reasons that customers used Railgun and where we recommend they consider migrating.
“I use Railgun to maintain a persistent, secure connection to Cloudflare’s network without the need for a static publicly available IP address.”
Customers can deploy Cloudflare Tunnel to connect their infrastructure to Cloudflare’s network without the need to expose a public IP address. Cloudflare Tunnel software runs in your environment, similar to the Railgun listener, and creates an outbound-only connection to Cloudflare’s network. Cloudflare Tunnel is available at no cost.
“I use Railgun to front multiple services running in my infrastructure.”
Cloudflare Tunnel can be deployed in this type of bastion mode to support multiple services running behind it in your infrastructure. You can use Tunnel to support services beyond just HTTP servers, and you can deploy replicas of the Cloudflare Tunnel connector for high availability.
“I use Railgun for performance improvements.”
Cloudflare has invested significantly in performance upgrades in the eight years since the last release of Railgun. This list is not comprehensive, but highlights some areas where performance can be significantly improved by adopting newer services relative to using Railgun.
“I use Railgun to reduce the amount of data that egresses from my infrastructure to Cloudflare.”
Certain public cloud providers charge egregious egress fees for you to move your own data outside their environment. We believe that degrades an open Internet and locks in customers. We have spent the last several years investing in ways to reduce or eliminate these altogether.
From the time of this announcement, customers have eight months available to migrate away from Railgun. January 31, 2024, will be the last day that Railgun connections will be supported. Starting on February 1, 2024, existing Railgun connections will stop functioning.
Over the next few days we will prevent new Railgun deployments from being created. Zones with Railgun connections already established will continue to function during the migration window.
Contract customers can reach out to their Customer Success team to discuss additional questions or migration plans. Each of Cloudflare’s regions has a specialist available to help guide teams who need additional help during the migration.
Customers can also raise questions and provide commentary in this dedicated forum room. We will continue to staff that discussion and respond to questions as customers share them.
Railgun customers will also receive an email notice later today about the deprecation plan and timeline. We will continue sending email notices multiple times over the next eight months leading up to the deprecation.
We are grateful to the Railgun customers who first selected Cloudflare to accelerate the applications and websites that power their business. We are excited to share the latest Cloudflare features with them that will continue to make them faster as they reach their audience.
Post Syndicated from Alex Krivit original http://blog.cloudflare.com/introducing-regional-tiered-cache/
Today we’re excited to announce an update to our Tiered Cache offering: Regional Tiered Cache.
Tiered Cache allows customers to organize Cloudflare data centers into tiers so that only some “upper-tier” data centers can request content from an origin server, and then send content to “lower-tiers” closer to visitors. Tiered Cache helps content load faster for visitors, makes it cheaper to serve, and reduces origin resource consumption.
Regional Tiered Cache provides an additional layer of caching for Enterprise customers who have a global traffic footprint and want to serve content faster by avoiding network latency when there is a cache miss in a lower-tier, resulting in an upper-tier fetch in a data center located far away. In our trials, customers who have enabled Regional Tiered Cache have seen a 50-100ms improvement in tail cache hit response times from Cloudflare’s CDN.
First, a quick refresher on caching: a request for content is initiated from a visitor on their phone or computer. This request is generally routed to the closest Cloudflare data center. When the request arrives, we look to see if we have the content cached to respond to that request with. If it’s not in cache (it’s a miss), Cloudflare data centers must contact the origin server to get a new copy of the content.
Getting content from an origin server suffers from two issues: latency and increased origin egress and load.
Origin servers, where content is hosted, can be far away from visitors. This is especially true the more global of an audience a particular piece of content has relative to where the origin is located. This means that content hosted in New York can be served in dramatically different amounts of time for visitors in London, Tokyo, and Cape Town. The farther away from New York a visitor is, the longer they must wait before the content is returned. Serving content from cache helps provide a uniform experience to all of these visitors because the content is served from a data center that’s close.
Even when using a CDN, many different visitors can be interacting with different data centers around the world and each data center, without the content visitors are requesting, will need to reach out to the origin for a copy. This can cost customers money because of egress fees origins charge for sending traffic to Cloudflare, and it places needless load on the origin by opening multiple connections for the same content, just headed to different data centers.
Performance improvements and origin load reductions are the promise of tiered cache.
Tiered Caching means that instead of every data center reaching out to the origin when there is a cache miss, the lower-tier data center that is closest to the visitor will reach out to a larger upper-tier data center to see if it has the requested content cached before the upper-tier asks the origin for the content. Organizing Cloudflare’s data centers into tiers means that fewer requests will make it back to the origin for the same content, preserving origin resources, reducing load, and saving the customer money in egress fees.
Cloudflare customers are given access to different Tiered Cache topologies based on their plan level. There are currently two predefined Tiered Cache topologies to select from – Smart and Generic Global. If either of those don’t work for a particular customer’s traffic profile, Enterprise customers can also work with us to define a custom topology.
In 2021, we announced that we’d allow all plans to access Smart Tiered Cache. Smart Tiered Cache dynamically finds the single closest data center to a customer’s origin server and chooses that as the upper-tier that all lower-tier data centers reach out to in the event of a cache miss. All other data centers go through that single upper-tier for content and that data center is the only one that can reach out to the origin. This helps to drastically boost cache hit ratios and reduces the connections to the origin. However, this topology can come at the cost of increased latency for visitors that are farther away from that single upper-tier.
Enterprise customers may select additional tiered cache topologies like the Generic Global topology which allows all of Cloudflare’s large data centers on our network (about 40 data centers) to serve as upper-tiers. While this topology may help reduce the long tail latencies for far-away visitors, it does so at the cost of increased connections and load on a customer's origin.
To describe the latency problem with Smart Tiered Cache in more detail let’s use an example. Suppose the upper-tier data center is selected to be in New York using Smart Tiered Cache. The traffic profile for the website with the New York upper-tier is relatively global. Visitors are coming from London, Tokyo, and Cape Town. For every cache miss in a lower-tier it will need to reach out to the New York upper-tier for content. This means these requests from Tokyo will need to traverse the Pacific Ocean and most of the continental United States to check the New York upper-tier cache. Then turn around and go all the way back to Tokyo. This is a giant performance hit for visitors outside the US for the sake of improving origin resource load.
With Regional Tiered Cache we introduce a middle-tier in each region around the world. When a lower-tier fetches on a cache miss it tries the regional-tier first if the upper-tier is in a different region. If the regional-tier does not have the asset then it asks the upper-tier for it. On the response the regional-tier writes to its cache so other lower-tiers in the same region benefit.
By putting an additional tier in the same region as the lower-tier, there’s an increased chance that the content will be available in the region before heading to a far-away upper-tier. This can drastically improve the performance of assets while still reducing the number of connections that will eventually need to be made to the customer’s origin.
Regional Tiered Cache helps customers with Smart Tiered Cache or a Custom Tiered Cache topology with upper-tiers in one or two regions. Regional Tiered Cache is not beneficial for customers with many upper-tiers in many regions like Generic Global Tiered Cache .
Enterprise customers can enable Regional Tiered Cache via the Cloudflare Dashboard or the API:
Please see the documentation for detailed information about how to configure Regional Tiered Cache from the API.
GET
curl --request GET \
--url https://api.cloudflare.com/client/v4/zones/zone_identifier/cache/regional_tiered_cache \
--header 'Content-Type: application/json' \
--header 'X-Auth-Email: '
PATCH
curl --request PATCH \
--url https://api.cloudflare.com/client/v4/zones/zone_identifier/cache/regional_tiered_cache \
--header 'Content-Type: application/json' \
--header 'X-Auth-Email: ' \
--data '{
"value": "on"
}'
Regional Tiered Cache is the first of many planned improvements to Cloudflare’s Tiered Cache offering which are currently in development. We look forward to hearing what you think about Regional Tiered Cache, and if you’re interested in helping us improve our CDN, we’re hiring.
Post Syndicated from Evgeny Karasik original https://aws.amazon.com/blogs/devops/cdk-v1-end-of-support/
Since its introduction in 2018, AWS CDK has gained significant traction among developers for building Infrastructure as Code solutions. As technology advances and new features emerge, it is inevitable that older versions of tools must reach their end-of-support.
As of June 1, 2023, AWS CDK v1 has officially reached its end-of-support. Consequently, AWS will no longer provide maintenance, updates, patches, or technical support for this version of the CDK. While the existing releases will remain accessible through public package managers and the code will still be available on GitHub, using the older AWS CDK v1 could expose your code to security vulnerabilities and unaddressed bugs. Moreover, as the technology landscape evolves, new AWS services and features may not be compatible with the outdated version. This could result in compatibility issues, lack of support for newer services, and missed opportunities to leverage the latest innovations from AWS.
To mitigate the risks associated with using an unsupported AWS CDK version and to benefit from the inclusion of new features and capabilities, we strongly recommends that you transition to AWS CDK v2 without delay.
AWS CDK version 2 (v2) provides improved features, enhanced performance, and continued support from AWS. By embracing the latest version of AWS CDK, developers can ensure the security, compatibility, and stability of their infrastructure as code solutions on AWS. Moving to AWS CDK v2 is typically a straightforward process for most projects. It involves a one-time, secure re-bootstrapping of your AWS accounts, along with updating dependencies and making changes to import statements.
To learn more, refer to the following resources:
Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2023/06/on-the-catastrophic-risk-of-ai.html
Earlier this week, I signed on to a short group statement, coordinated by the Center for AI Safety:
Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.
The press coverage has been extensive, and surprising to me. The New York Times headline is “A.I. Poses ‘Risk of Extinction,’ Industry Leaders Warn.” BBC: “Artificial intelligence could lead to extinction, experts warn.” Other headlines are similar.
I actually don’t think that AI poses a risk to human extinction. I think it poses a similar risk to pandemics and nuclear war—which is to say, a risk worth taking seriously, but not something to panic over. Which is what I thought the statement said.
In my talk at the RSA Conference last month, I talked about the power level of our species becoming too great for our systems of governance. Talking about those systems, I said:
Now, add into this mix the risks that arise from new and dangerous technologies such as the internet or AI or synthetic biology. Or molecular nanotechnology, or nuclear weapons. Here, misaligned incentives and hacking can have catastrophic consequences for society.
That was what I was thinking about when I agreed to sign on to the statement: “Pandemics, nuclear weapons, AI—yeah, I would put those three in the same bucket. Surely we can spend the same effort on AI risk as we do on future pandemics. That’s a really low bar.” Clearly I should have focused on the word “extinction,” and not the relative comparisons.
Seth Lazar, Jeremy Howard, and Arvind Narayanan wrote:
We think that, in fact, most signatories to the statement believe that runaway AI is a way off yet, and that it will take a significant scientific advance to get there—ne that we cannot anticipate, even if we are confident that it will someday occur. If this is so, then at least two things follow.
I agree with that, and with their follow up:
First, we should give more weight to serious risks from AI that are more urgent. Even if existing AI systems and their plausible extensions won’t wipe us out, they are already causing much more concentrated harm, they are sure to exacerbate inequality and, in the hands of power-hungry governments and unscrupulous corporations, will undermine individual and collective freedom.
This is what I wrote in Click Here to Kill Everybody (2018):
I am less worried about AI; I regard fear of AI more as a mirror of our own society than as a harbinger of the future. AI and intelligent robotics are the culmination of several precursor technologies, like machine learning algorithms, automation, and autonomy. The security risks from those precursor technologies are already with us, and they’re increasing as the technologies become more powerful and more prevalent. So, while I am worried about intelligent and even driverless cars, most of the risks arealready prevalent in Internet-connected drivered cars. And while I am worried about robot soldiers, most of the risks are already prevalent in autonomous weapons systems.
Also, as roboticist Rodney Brooks pointed out, “Long before we see such machines arising there will be the somewhat less intelligent and belligerent machines. Before that there will be the really grumpy machines. Before that the quite annoying machines. And before them the arrogant unpleasant machines.” I think we’ll see any new security risks coming long before they get here.
I do think we should worry about catastrophic AI and robotics risk. It’s the fact that they affect the world in a direct, physical manner—and that they’re vulnerable to class breaks.
(Other things to read: David Chapman is good on scary AI. And Kieran Healy is good on the statement.)
Okay, enough. I should also learn not to sign on to group statements.
Post Syndicated from Михаил Ангелов original https://www.toest.bg/nauchni-novini-transplantatsiya-na-mitohondrii/
Митохондриите са органели, отговорни за енергийните нужди на клетките, и се намират във всички човешки клетки, освен в червените кръвни телца. Поради произхода си (ендосимбионтни бактерии) те са запазили част от своя генетичен материал и имат хромозома, която при хората носи 37 гена, 13 от които вземат участие в енергийния метаболизъм. Бактериалният произход и някои други техни особености са причина геномът им да има значително по-високи нива на мутации в сравнение с ядрената ДНК. Това има два основни ефекта: понякога тези мутации водят до повреди в гените, а също и до получаване на хетерогенна популация в различните тъкани и клетки.
Децата получават митохондриите си почти ексклузивно от майките (има няколко описани случая на преобладаващи бащини митохондрии, но за момента се смята, че те са рядкост). Поради случайното разпределение на митохондриите създаването на потомство от майки с установени мутации поражда риск, тъй като има вероятност в яйцеклетките им да попаднат увредени митохондрии и децата да развият заболявания с митохондриална етиология. Обикновено те са свързани с органи с висока потребност от енергия, като мозък, сърце, мускулатура.
Потенциално решение за преодоляване на този проблем е т.нар. трансплантация на митохондрии. В момента има два основни подхода за извършване на процедурата. При първия се използват неоплодени яйцеклетки от здрав донор и майката. Ядрото от донорската клетка се премахва и в нея се прехвърля ядрото от майчината, където е останала цитоплазмата с митохондриите. След това се извършва оплождането със сперматозоиди от бащата. При втория подход и двете изходни яйцеклетки първо се оплождат, след което отново ядрото от едната зигота се прехвърля в другата. Въпреки че в крайна сметка децата наистина носят генетичен материал от трима „родители“, приносът на донора е около 1%, така че той не би следвало да има правни задължения (или права) към тях.
Методът все още има сравнително ограничено приложение, защото законовият му статус е неуточнен в повечето страни. Едно от първите съобщения е от американски лекар, който е извършил процедурата в Мексико. Известни са и екипи, които работят в Гърция и Украйна, но може би най-добре уредена е ситуацията в Обединеното кралство, където процедурата е одобрена през 2015 г., като всеки случай се разглежда от регулаторния орган Human Fertilisation and Embryology Authority (HFEA). След искане за достъп до информация журналисти от Guardian са получили отговор, че към края на април 2023 г. са родени по-малко от пет бебета с донорски митохондрии. Точният брой не е посочен, защото според организацията това би дало възможност децата да бъдат идентифицирани.
Въпреки че като цяло резултатите са обещаващи, не всички учени са съгласни с прилагането на терапията в този ѝ вид. И при двата метода не е възможно да се изключат напълно митохондриите от майката. Обикновено те са под 2% от всички налични в зиготата, което е малък процент, но все пак крие рискове. Вече има съобщения за бебета, при които е настъпила т.нар. реверсия – митохондриите на майката се намножават и достигат нива, носещи риск за активиране на заболяването.
В публикуваните данни се посочва, че в началните стадии, непосредствено след трансплантацията, пренесените майчини митохондрии са били под 0,8%, от общото съдържание, но при раждането те вече са били между 30 и 60%. И при двата известни случая трансплантацията на митохондрии е използвана за преодоляване на репродуктивни проблеми, така че повишението не представлява опасност за здравето на децата, но тези случаи са индикация, че преди масовото прилагане на терапията има технически недостатъци, които трябва да се преодолеят.
Новите подходи, особено в медицината, крият своите рискове, но дават и възможност за справяне с проблеми, които само допреди години са се считали за непреодолими. Темпът на развитие на технологиите предполага, че в недалечно бъдеще методът ще бъде прецизиран и ще може да се прилага рутинно без тревога за потенциални реверсии.
В тази връзка е интересно скорошното развитие в създаването на яйцеклетки от мъжки клетки при мишки. Първо кожни клетки са третирани по специфичен начин, за да се индуцира превръщането им в структури, сходни на стволови клетки. След това Y хромозомата в тях е премахната и е вмъкната още една X хромозома от друга клетка. Получените XX стволови клетки са имплантирани в органоид със структура на миши яйчник, след което развитите яйцеклетки са оплодени и имплантирани в сурогатни майки. Родените мишки са били жизнени и с нормална продължителност на живота, като някои дори са оставили потомство. За момента процедурата е сложна и с ниска успеваемост, но учените се надяват в рамките на няколко години да я усъвършенстват и да я приложат при хора, които имат репродуктивни проблеми.
С напредването на науката най-вероятно ще има възможност и за създаване на яйцеклетки от соматични клетки на майката, след като се проведе скрининг и се подберат само клетки, в които всички митохондрии са здрави. Така ще отпадне необходимостта от използване на донори и целият генетичен материал на децата ще бъде принос само на двамата им преки родители или на единия от тях.
Листата на сарепската горчица имат кулинарно приложение, но поради горчивия си вкус, сходен с този на другите ѝ зелеви роднини, обикновено минават през термична обработка преди консумация, за да се намали силата му. Това е пречка пред използването им в суров вид въпреки по-високата хранителна стойност в сравнение с другите листни зеленчуци, от които обичайно се правят салати.
Биотехнологичната компания Pairwise предлага решение за този проблем с първия CRISPR редактиран хранителен продукт в САЩ. С помощта на технологията те са ограничили активността на ензима мирозиназа, който е свързан със защитната система на зелевите растения. Когато са изложени на опасност (например от насекоми), този ензим освобождава съединения, съдържащи сяра в структурата си (тиоли). Те бързо образуват нови молекули с циано- или нитрилна група, които са силно токсични за неприятелите. Именно тиолите (и сярата в тях) са причина за горчивия вкус и острия мирис на сарепската горчица. Понякога те са проблем и за тревопасните животни, поради което рапицата от същото семейство се използва за фураж с повишено внимание.
Компанията е получила разрешение от USDA през 2020 г., като сега смята да предлага салатен микс, в който има листа от сарепска горчица, на подбрани ресторанти и някои магазини. Освен по-високата хранителна стойност, друго предимство на тази култура пред останалите листни зеленчуци е и по-високият добив.
Това не е първият генетично редактиран продукт на пазара в глобален мащаб. През 2021 г. в Япония започна продажбата на домати със завишено съдържание на гама-аминомаслена киселина (GABA), която инхибира предаването на импулси между нервните клетки. Според компанията, създала тези домати, приемът ѝ помага за понижаване на кръвното налягане и намаляване на стреса.
За интересно приложение на CRISPR технологията съобщава и друг американски екип, съставен от учени от USDA, университетски изследователи и частни компании. Те успешно са създали говедо, което има повишена толерантност към вируса, причиняващ вирусна диария (мукозна болест). Това е заболяване, за което има множество ваксини, но въпреки това те невинаги успяват да го предотвратят, а предаването е много лесно, особено във ферми с по-голяма плътност на отглеждане. Болестта е особено опасна за бременните крави, защото намалява процента успешни раждания, а родените телета са носители на причинителя и в тях се наблюдава много висок вирусен товар.
За да проникне успешно в клетките на животните, вирусът се прикачва към един от протеините на тяхната мембрана (CD46). Използвайки тази информация, известна ни от почти 20 години, екипът е направил промяна в шест аминокиселини (от общо 361) на този протеин с помощта на Crispr-Cas9. Първо моделът е изпитан в клетъчни култури, а след потвърждаването му е било създадено генетично редактирано женско теле, наречено Ginger.
Редакцията не е променила съществено функционалността на протеина, така че той успешно продължава да работи нормално в клетките на животните, но вирусът вече не може да го използва за проникване в тях. След като е станала на десет месеца, Ginger е изложена на вируса чрез поставяне в съжителство с теле, което е негов носител и го отделя активно. Въпреки че той е открит в кръвта ѝ, вирусният товар е бил значително по-нисък и тя не е проявила симптомите, характерни за заболяването.
Тази разработка показва нов хоризонт пред генните редакции под формата на своеобразни перманентни ваксини, които могат да бъдат вмъкнати при създаването на нови породи. Възможността за прецизни редакции е изключително полезен инструмент, тъй като дори малка разлика в един ген може да доведе до големи промени в организма. Това е видно и от работата на учени от Женевския университет.
Биохимичните процеси в организмите протичат като последователност от химични реакции, наречена сигнален път (или биохимична каскада). В повечето случаи тези реакции са контролирани от активацията или спирането на действието на различни гени. В най-простия вариант са линейни и се активират от един ген вследствие на някакъв стимул, като всяка стъпка започва след края на предходната, но има и случаи, в които са разклонени, с много стъпки, които се влияят една от друга и са с различни активиращи гени.
Един от особено важните гени, контролиращи такива сигнални пътища, е Shh (Sonic hedgehog, Таралежът Соник), който взема участие в морфогенезата на централната нервна система, дробовете, крайниците, зъбите – симетрични части на тялото. Той е отговорен и за правилното развитие на различни образувания по кожата – косми, пера и люспи. Тези структури са хомолози и се предполага, че произхождат от общ предшественик на гръбначните животни.
Установено е, че при активиране на Shh по време на развитието на ембриона люспите, които обичайно покриват краката на кокошките, се трансформират в пера. Те имат идентична структура като тези по другите части и с възрастта преминават от пух към същински пера. Интересното е, че в случая дори не е имало генетична манипулация в класическия ѝ смисъл. Активацията на гена е постигната чрез инжектиране на малка молекула (SAG) в 11-тия ден от развитието на ембриона, която активира сигналния път на Shh, без да се правят намеси в генома на животните.
Според учените самата трансформация дава повече информация за процесите на образуване на различните структури. Но по-интересното е, че тя е следствие от временна промяна в експресията на гена, което може да има голямо значение за земеделската продукция, където генните манипулации подлежат на строг контрол и ако могат да бъдат избегнати, пускането на нови продукти на пазара ще бъде значително по-лесно.
ДНК е навсякъде – всички организми я отделят в околната среда с биологичните си дейности. Благодарение на това се развива и цяла нова научна област, в която се работи със сборни проби от най-различни източници, като водоеми, въздух, почва и др. Събраната ДНК от околната среда (eDNA, environmental DNA) може да бъде секвенирана и след това с помощта на биоинформатични методи (компютърен анализ на данните) да се правят най-различни проучвания за обитателите на даден ареал – видов състав, здравен статус. Два актуални примера са откриването на нови вируси, за което разказахме наскоро, и употребата на сходен подход за проследяване и прогноза на потенциални огнища на COVID-19 чрез проби от пречиствателни станции.
В това отношение човекът не се различава от останалите организми. Ние отделяме ежедневно своята ДНК чрез аерозоли (говор, кашляне, кихане), частици кожа, при отделителните процеси. Доскоро това не беше от ключово значение, защото при повечето изследвания се използва количествен PCR (qPCR, quantitative PCR), при който е необходима информация за търсения организъм. Но с напредването на технологиите за изолация и секвениране на ДНК и последващата обработка на получените данни за учените става достъпна информация за целия генетичен материал в събраната проба.
Ново изследване описва потенциалните проблеми и възможности, които произтичат от това. Основният обект на учените са костенурки, размножаващи се по плажовете на Флорида. Изолирайки ДНК от следите, които оставят в пясъка, те могат да проследяват популациите им и да наблюдават появата на заболявания, причинени от вируси. След като откриват човешки генетични последователности в един такъв набор, изследователите решават да се фокусират върху това и освен от плажен пясък събират и водни проби от океана и река във Флорида.
Откриването на човешка ДНК в тях не е особена изненада, но качеството ѝ е било значително по-високо от очакваното. Успешно са открити митохондриални мутации, свързани с аутизъм, диабет, очни и сърдечни заболявания. Наред с това е направен и популационно-генетичен анализ, при който е установено, че данните отговарят на демографското разпределение в района. Експериментът е повторен и с вода от река в Ирландия със сходни резултати.
Може би най-впечатляващо е отчитането на човешка ДНК в проби от въздух, взет от стаите на ветеринарна клиника. Отново са засечени мутации, свързани със заболявания, а също така успешно е определено потеклото на работещите в клиниката. Открити са и папиломен вирус, причиняващ туморни заболявания по костенурки (пациентите на клиниката), както и тяхната ДНК.
Това повдига редица етични въпроси, свързани с неприкосновеността и даването на съгласие за обработка на генетични данни, възможността за проследяване и дискриминацията на цели групи на база белези, които вече могат да бъдат открити, без да се вземат проби непосредствено от хората. Предвид практиката данните от секвениране при всички изследвания, финансирани от държавата, да се качват в публично достъпни бази данни, това би могло да даде достъп до подобна информация на компании и други организации, без реално те да извършват работата по обработка на биологичните проби.
Но това не означава, че подходът не може да се използва и за добро. Потенциални приложения са следенето на популационната динамика, както и на появата на зловредни мутации, нови заболявания или техни щамове и варианти. Друга авангардна идея, предложена от авторите, е търсенето на загубени хора чрез събиране на проби от въздуха с помощта на дрон. Въпреки че има потенциално приложение и в криминалистиката, според професорката по право Натали Рам, специализираща в сигурността на генетичните данни, технологията крие риск да ни постави под постоянно наблюдение и заключения да се правят въз основа на случайно попаднал генетичен материал.
Както при всяко ново откритие, предстои период на изясняване на пълните му възможности и поставяне на законови и етични рамки, които да насочат употребата му така, че да бъде полезно за обществото и да носи основно позитиви.