Post Syndicated from Darknet original http://feedproxy.google.com/~r/darknethackers/~3/A3ozOmH4BDE/
TheFatRat is an easy-to-use Exploitation Tool that can help you to generate backdoors and post exploitation attacks like browser attack DLL files. This tool compiles malware with popular payloads and then the compiled malware can be executed on Windows, Linux, Mac OS X and Android. The malware that is created with this tool also has […]
The…
Read the full post at darknet.org.uk
Post Syndicated from Simon Long original https://www.raspberrypi.org/blog/a-raspbian-desktop-update-with-some-new-programming-tools/
Today we’ve released another update to the Raspbian desktop. In addition to the usual small tweaks and bug fixes, the big new changes are the inclusion of an offline version of Scratch 2.0, and of Thonny (a user-friendly IDE for Python which is excellent for beginners). We’ll look at all the changes in this post, but let’s start with the biggest…
Scratch is one of the most popular pieces of software on Raspberry Pi. This is largely due to the way it makes programming accessible – while it is simple to learn, it covers many of the concepts that are used in more advanced languages. Scratch really does provide a great introduction to programming for all ages.
Raspbian ships with the original version of Scratch, which is now at version 1.4. A few years ago, though, the Scratch team at the MIT Media Lab introduced the new and improved Scratch version 2.0, and ever since we’ve had numerous requests to offer it on the Pi.
There was, however, a problem with this. The original version of Scratch was written in a language called Squeak, which could run on the Pi in a Squeak interpreter. Scratch 2.0, however, was written in Flash, and was designed to run from a remote site in a web browser. While this made Scratch 2.0 a cross-platform application, which you could run without installing any Scratch software, it also meant that you had to be able to run Flash on your computer, and that you needed to be connected to the internet to program in Scratch.
We worked with Adobe to include the Pepper Flash plugin in Raspbian, which enables Flash sites to run in the Chromium browser. This addressed the first of these problems, so the Scratch 2.0 website has been available on Pi for a while. However, it still needed an internet connection to run, which wasn’t ideal in many circumstances. We’ve been working with the Scratch team to get an offline version of Scratch 2.0 running on Pi.
The Scratch team had created a website to enable developers to create hardware and software extensions for Scratch 2.0; this provided a version of the Flash code for the Scratch editor which could be modified to run locally rather than over the internet. We combined this with a program called Electron, which effectively wraps up a local web page into a standalone application. We ended up with the Scratch 2.0 application that you can find in the Programming section of the main menu.
We didn’t stop there though. We know that people want to use Scratch for physical computing, and it has always been a bit awkward to access GPIO pins from Scratch. In our Scratch 2.0 application, therefore, there is a custom extension which allows the user to control the Pi’s GPIO pins without difficulty. Simply click on ‘More Blocks’, choose ‘Add an Extension’, and select ‘Pi GPIO’. This loads two new blocks, one to read and one to write the state of a GPIO pin.
The Scratch team kindly allowed us to include all the sprites, backdrops, and sounds from the online version of Scratch 2.0. You can also use the Raspberry Pi Camera Module to create new sprites and backgrounds.
This first release works well, although it can be slow for some operations; this is largely unavoidable for Flash code running under Electron. Bear in mind that you will need to have the Pepper Flash plugin installed (which it is by default on standard Raspbian images). As Pepper Flash is only compatible with the processor in the Pi 2.0 and Pi 3, it is unfortunately not possible to run Scratch 2.0 on the Pi Zero or the original models of the Pi.
We hope that this makes Scratch 2.0 a more practical proposition for many users than it has been to date. Do let us know if you hit any problems, though!
One of the paths from Scratch to ‘real’ programming is through Python. We know that the transition can be awkward, and this isn’t helped by the tools available for learning Python. It’s fair to say that IDLE, the Python IDE, isn’t the most popular piece of software ever written…
Earlier this year, we reviewed every Python IDE that we could find that would run on a Raspberry Pi, in an attempt to see if there was something better out there than IDLE. We wanted to find something that was easier for beginners to use but still useful for experienced Python programmers. We found one program, Thonny, which stood head and shoulders above all the rest. It’s a really user-friendly IDE, which still offers useful professional features like single-stepping of code and inspection of variables.
Thonny was created at the University of Tartu in Estonia; we’ve been working with Aivar Annamaa, the lead developer, on getting it into Raspbian. The original version of Thonny works well on the Pi, but because the GUI is written using Python’s default GUI toolkit, Tkinter, the appearance clashes with the rest of the Raspbian desktop, most of which is written using the GTK toolkit. We made some changes to bring things like fonts and graphics into line with the appearance of our other apps, and Aivar very kindly took that work and converted it into a theme package that could be applied to Thonny.
Due to the limitations of working within Tkinter, the result isn’t exactly like a native GTK application, but it’s pretty close. It’s probably good enough for anyone who isn’t a picky UI obsessive like me, anyway! Have a look at the Thonny webpage to see some more details of all the cool features it offers. We hope that having a more usable environment will help to ease the transition from graphical languages like Scratch into ‘proper’ languages like Python.
Other than these two new packages, this release is mostly bug fixes and small version bumps. One thing you might notice, though, is that we’ve made some tweaks to our custom icon set. We wondered if the icons might look better with slightly thinner outlines. We tried it, and they did: we hope you prefer them too.
You can either download a new image from the Downloads page, or you can use apt to update:
sudo apt-get update sudo apt-get dist-upgrade
To install Scratch 2.0:
sudo apt-get install scratch2
To install Thonny:
sudo apt-get install python3-thonny
Before Christmas, we released an experimental version of the desktop running on Debian for x86-based computers. We were slightly taken aback by how popular it turned out to be! This made us realise that this was something we were going to need to support going forward. We’ve decided we’re going to try to make all new desktop releases for both Pi and x86 from now on.
The version of this we released last year was a live image that could run from a USB stick. Many people asked if we could make it permanently installable, so this version includes an installer. This uses the standard Debian install process, so it ought to work on most machines. I should stress, though, that we haven’t been able to test on every type of hardware, so there may be issues on some computers. Please be sure to back up your hard drive before installing it. Unlike the live image, this will erase and reformat your hard drive, and you will lose anything that is already on it!
You can still boot the image as a live image if you don’t want to install it, and it will create a persistence partition on the USB stick so you can save data. Just select ‘Run with persistence’ from the boot menu. To install, choose either ‘Install’ or ‘Graphical install’ from the same menu. The Debian installer will then walk you through the install process.
You can download the latest x86 image (which includes both Scratch 2.0 and Thonny) from here or here for a torrent file.
This version of the desktop is based on Debian Jessie. Some of you will be aware that a new stable version of Debian (called Stretch) was released last week. Rest assured – we have been working on porting everything across to Stretch for some time now, and we will have a Stretch release ready some time over the summer.
The post A Raspbian desktop update with some new programming tools appeared first on Raspberry Pi.
Post Syndicated from Ernesto original https://torrentfreak.com/kim-dotcom-opposes-uss-fugitive-claims-supreme-court-170622/
When Megaupload and Kim Dotcom were raided five years ago, the authorities seized millions of dollars in cash and other property.
The US government claimed the assets were obtained through copyright crimes so went after the bank accounts, cars, and other seized possessions of the Megaupload defendants.
Kim Dotcom and his colleagues were branded as “fugitives” and the Government won its case. Dotcom’s legal team quickly appealed this verdict, but lost once more at the Fourth Circuit appeals court.
A few weeks ago Dotcom and his former colleagues petitioned the Supreme Court to take on the case.
They don’t see themselves as “fugitives” and want the assets returned. The US Government opposed the request, but according to a new reply filed by Megaupload’s legal team, the US Government ignores critical questions.
The Government has a “vested financial stake” in maintaining the current situation, they write, which allows the authorities to use their “fugitive” claims as an offensive weapon.
“Far from being directed towards persons who have fled or avoided our country while claiming assets in it, fugitive disentitlement is being used offensively to strip foreigners of their assets abroad,” the reply brief (pdf) reads.
According to Dotcom’s lawyers there are several conflicting opinions from lower courts, which should be clarified by the Supreme Court. That Dotcom and his colleagues have decided to fight their extradition in New Zealand, doesn’t warrant the seizure of their assets.
“Absent review, forfeiture of tens of millions of dollars will be a fait accompli without the merits being reached,” they write, adding that this is all the more concerning because the US Government’s criminal case may not be as strong as claimed.
“This is especially disconcerting because the Government’s criminal case is so dubious. When the Government characterizes Petitioners as ‘designing and profiting from a system that facilitated wide-scale copyright infringement,’ it continues to paint a portrait of secondary copyright infringement, which is not a crime.”
The defense team cites several issues that warrant review and urges the Supreme Court to hear the case. If not, the Government will effectively be able to use assets seizures as a pressure tool to urge foreign defendants to come to the US.
“If this stands, the Government can weaponize fugitive disentitlement in order to claim assets abroad,” the reply brief reads.
“It is time for the Court to speak to the Questions Presented. Over the past two decades it has never had a better vehicle to do so, nor is any such vehicle elsewhere in sight,” Dotcom’s lawyers add.
Whether the Supreme Court accepts or denies the case will likely be decided in the weeks to come.
Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.
Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2017/06/nsa_insider_sec.html
According to a recently declassified report obtained under FOIA, the NSA’s attempts to protect itself against insider attacks aren’t going very well:
The N.S.A. failed to consistently lock racks of servers storing highly classified data and to secure data center machine rooms, according to the report, an investigation by the Defense Department’s inspector general completed in 2016.
[…]
The agency also failed to meaningfully reduce the number of officials and contractors who were empowered to download and transfer data classified as top secret, as well as the number of “privileged” users, who have greater power to access the N.S.A.’s most sensitive computer systems. And it did not fully implement software to monitor what those users were doing.
In all, the report concluded, while the post-Snowden initiative — called “Secure the Net” by the N.S.A. — had some successes, it “did not fully meet the intent of decreasing the risk of insider threats to N.S.A. operations and the ability of insiders to exfiltrate data.”
Marcy Wheeler comments:
The IG report examined seven of the most important out of 40 “Secure the Net” initiatives rolled out since Snowden began leaking classified information. Two of the initiatives aspired to reduce the number of people who had the kind of access Snowden did: those who have privileged access to maintain, configure, and operate the NSA’s computer systems (what the report calls PRIVACs), and those who are authorized to use removable media to transfer data to or from an NSA system (what the report calls DTAs).
But when DOD’s inspectors went to assess whether NSA had succeeded in doing this, they found something disturbing. In both cases, the NSA did not have solid documentation about how many such users existed at the time of the Snowden leak. With respect to PRIVACs, in June 2013 (the start of the Snowden leak), “NSA officials stated that they used a manually kept spreadsheet, which they no longer had, to identify the initial number of privileged users.” The report offered no explanation for how NSA came to no longer have that spreadsheet just as an investigation into the biggest breach thus far at NSA started. With respect to DTAs, “NSA did not know how many DTAs it had because the manually kept list was corrupted during the months leading up to the security breach.”
There seem to be two possible explanations for the fact that the NSA couldn’t track who had the same kind of access that Snowden exploited to steal so many documents. Either the dog ate their homework: Someone at NSA made the documents unavailable (or they never really existed). Or someone fed the dog their homework: Some adversary made these lists unusable. The former would suggest the NSA had something to hide as it prepared to explain why Snowden had been able to walk away with NSA’s crown jewels. The latter would suggest that someone deliberately obscured who else in the building might walk away with the crown jewels. Obscuring that list would be of particular value if you were a foreign adversary planning on walking away with a bunch of files, such as the set of hacking tools the Shadow Brokers have since released, which are believed to have originated at NSA.
Read the whole thing. Securing against insiders, especially those with technical access, is difficult, but I had assumed the NSA did more post-Snowden.
Post Syndicated from Andy original https://torrentfreak.com/three-men-sentenced-following-2-5m-internet-piracy-case-170622/
While legal action against low-level individual file-sharers is extremely rare in the UK, the country continues to pose a risk for those engaged in larger-scale infringement.
That is largely due to the activities of the Police Intellectual Property Crime Unit and private anti-piracy outfits such as the Federation Against Copyright Theft (FACT). Investigations are often a joint effort which can take many years to complete, but the outcomes can often involve criminal sentences.
That was the profile of another Internet piracy case that concluded in London this week. It involved three men from the UK, Eric Brooks, 43, from Bolton, Mark Valentine, 44, from Manchester, and Craig Lloyd, 33, from Wolverhampton.
The case began when FACT became aware of potentially infringing activity back in February 2011. The anti-piracy group then investigated for more than a year before handing the case to police in March 2012.
On July 4, 2012, officers from City of London Police arrested Eric Brooks’ at his home in Bolton following a joint raid with FACT. Computer equipment was seized containing evidence that Brooks had been running a Netherlands-based server hosting more than £100,000 worth of pirated films, music, games, software and ebooks.
According to police, a spreadsheet on Brooks’ computer revealed he had hundreds of paying customers, all recruited from online forums. Using PayPal or utilizing bank transfers, each paid money to access the server. Police mentioned no group or site names in information released this week.
“Enquiries with PayPal later revealed that [Brooks] had made in excess of £500,000 in the last eight years from his criminal business and had in turn defrauded the film and TV industry alone of more than £2.5 million,” police said.
“As his criminal enterprise affected not only the film and TV but the wider entertainment industry including music, games, books and software it is thought that he cost the wider industry an amount much higher than £2.5 million.”
On the same day police arrested Brooks, Mark Valentine’s home in Manchester had a similar unwelcome visit. A day later, Craig Lloyd’s home in Wolverhampton become the third target for police.
Computer equipment was seized from both addresses which revealed that the pair had been paying for access to Brooks’ servers in order to service their own customers.
“They too had used PayPal as a means of taking payment and had earned thousands of pounds from their criminal actions; Valentine gaining £34,000 and Lloyd making over £70,000,” police revealed.
But after raiding the trio in 2012, it took more than four years to charge the men. In a feature common to many FACT cases, all three were charged with Conspiracy to Defraud rather than copyright infringement offenses. All three men pleaded guilty before trial.
On Monday, the men were sentenced at Inner London Crown Court. Brooks was sentenced to 24 months in prison, suspended for 12 months and ordered to complete 140 hours of unpaid work.
Valentine and Lloyd were each given 18 months in prison, suspended for 12 months. Each was ordered to complete 80 hours unpaid work.
Detective Constable Chris Glover, who led the investigation for the City of London Police, welcomed the sentencing.
“The success of this investigation is a result of co-ordinated joint working between the City of London Police and FACT. Brooks, Valentine and Lloyd all thought that they were operating under the radar and doing something which they thought was beyond the controls of law enforcement,” Glover said.
“Brooks, Valentine and Lloyd will now have time in prison to reflect on their actions and the result should act as deterrent for anyone else who is enticed by abusing the internet to the detriment of the entertainment industry.”
While even suspended sentences are a serious matter, none of the men will see the inside of a cell if they meet the conditions of their sentence for the next 12 months. For a case lasting four years involving such large sums of money, that is probably a disappointing result for FACT and the police.
Nevertheless, the men won’t be allowed to enjoy the financial proceeds of their piracy, if indeed any money is left. City of London Police say the trio will be subject to a future confiscation hearing to seize any proceeds of crime.
Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.
Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/dynamodb-accelerator-dax-now-generally-available/
Earlier this year I told you about Amazon DynamoDB Accelerator (DAX), a fully-managed caching service that sits in front of (logically speaking) your Amazon DynamoDB tables. DAX returns cached responses in microseconds, making it a great fit for eventually-consistent read-intensive workloads. DAX supports the DynamoDB API, and is seamless and easy to use. As a managed service, you simply create your DAX cluster and use it as the target for your existing reads and writes. You don’t have to worry about patching, cluster maintenance, replication, or fault management.
Now Generally Available
Today I am pleased to announce that DAX is now generally available. We have expanded DAX into additional AWS Regions and used the preview time to fine-tune performance and availability:
Now in Five Regions – DAX is now available in the US East (Northern Virginia), EU (Ireland), US West (Oregon), Asia Pacific (Tokyo), and US West (Northern California) Regions.
In Production – Our preview customers are reporting that they are using DAX in production, that they loved how easy it was to add DAX to their application, and have told us that their apps are now running 10x faster.
Getting Started with DAX
As I outlined in my earlier post, it is easy to use DAX to accelerate your existing DynamoDB applications. You simply create a DAX cluster in the desired region, update your application to reference the DAX SDK for Java (the calls are the same; this is a drop-in replacement), and configure the SDK to use the endpoint to your cluster. As a read-through/write-through cache, DAX seamlessly handles all of the DynamoDB read/write APIs.
We are working on SDK support for other languages, and I will share additional information as it becomes available.
DAX Pricing
You pay for each node in the cluster (see the DynamoDB Pricing page for more information) on a per-hour basis, with prices starting at $0.269 per hour in the US East (Northern Virginia) and US West (Oregon) regions. With DAX, each of the nodes in your cluster serves as a read target and as a failover target for high availability. The DAX SDK is cluster aware and will issue round-robin requests to all nodes in the cluster so that you get to make full use of the cluster’s cache resources.
Because DAX can easily handle sudden spikes in read traffic, you may be able to reduce the amount of provisioned throughput for your tables, resulting in an overall cost savings while still returning results in microseconds.
— Jeff;
Post Syndicated from Tara Van Unen original https://aws.amazon.com/blogs/compute/building-loosely-coupled-scalable-c-applications-with-amazon-sqs-and-amazon-sns/
One of the many challenges professional software architects and developers face is how to make cloud-native applications scalable, fault-tolerant, and highly available.
Fundamental to your project success is understanding the importance of making systems highly cohesive and loosely coupled. That means considering the multi-dimensional facets of system coupling to support the distributed nature of the applications that you are building for the cloud.
By that, I mean addressing not only the application-level coupling (managing incoming and outgoing dependencies), but also considering the impacts of of platform, spatial, and temporal coupling of your systems. Platform coupling relates to the interoperability, or lack thereof, of heterogeneous systems components. Spatial coupling deals with managing components at a network topology level or protocol level. Temporal, or runtime coupling, refers to the ability of a component within your system to do any kind of meaningful work while it is performing a synchronous, blocking operation.
The AWS messaging services, Amazon SQS and Amazon SNS, help you deal with these forms of coupling by providing mechanisms for:
Following on the recent topic, Building Scalable Applications and Microservices: Adding Messaging to Your Toolbox, in this post, I look at some of the ways you can introduce SQS and SNS into your architectures to decouple your components, and show how you can implement them using C#.
To illustrate some of these concepts, consider a web application that processes customer orders. As good architects and developers, you have followed best practices and made your application scalable and highly available. Your solution included implementing load balancing, dynamic scaling across multiple Availability Zones, and persisting orders in a Multi-AZ Amazon RDS database instance, as in the following diagram.
In this example, the application is responsible for handling and persisting the order data, as well as dealing with increases in traffic for popular items.
One potential point of vulnerability in the order processing workflow is in saving the order in the database. The business expects that every order has been persisted into the database. However, any potential deadlock, race condition, or network issue could cause the persistence of the order to fail. Then, the order is lost with no recourse to restore the order.
With good logging capability, you may be able to identify when an error occurred and which customer’s order failed. This wouldn’t allow you to “restore” the transaction, and by that stage, your customer is no longer your customer.
As illustrated in the following diagram, introducing an SQS queue helps improve your ordering application. Using the queue isolates the processing logic into its own component and runs it in a separate process from the web application. This, in turn, allows the system to be more resilient to spikes in traffic, while allowing work to be performed only as fast as necessary in order to manage costs.
In addition, you now have a mechanism for persisting orders as messages (with the queue acting as a temporary database), and have moved the scope of your transaction with your database further down the stack. In the event of an application exception or transaction failure, this ensures that the order processing can be retired or redirected to the Amazon SQS Dead Letter Queue (DLQ), for re-processing at a later stage. (See the recent post, Using Amazon SQS Dead-Letter Queues to Control Message Failure, for more information on dead-letter queues.)
This change allows you now to scale the web application frontend independently from the processing nodes. The frontend application can continue to scale based on metrics such as CPU usage, or the number of requests hitting the load balancer. Processing nodes can scale based on the number of orders in the queue. Here is an example of scale-in and scale-out alarms that you would associate with the scaling policy.
Scale-out Alarm
On top of scaling at an infrastructure level using Auto Scaling, make sure to take advantage of the processing power of your Amazon EC2 instances by using as many of the available threads as possible. There are several ways to implement this. In this post, we build a Windows service that uses the BackgroundWorker class to process the messages from the queue.
Here’s a closer look at the implementation. In the first section of the consuming application, use a loop to continually poll the queue for new messages, and construct a ReceiveMessageRequest variable.
public static void PollQueue()
{
while (_running)
{
Task<ReceiveMessageResponse> receiveMessageResponse;
// Pull messages off the queue
using (var sqs = new AmazonSQSClient())
{
const int maxMessages = 10; // 1-10
//Receiving a message
var receiveMessageRequest = new ReceiveMessageRequest
{
// Get URL from Configuration
QueueUrl = _queueUrl,
// The maximum number of messages to return.
// Fewer messages might be returned.
MaxNumberOfMessages = maxMessages,
// A list of attributes that need to be returned with message.
AttributeNames = new List<string> { "All" },
// Enable long polling.
// Time to wait for message to arrive on queue.
WaitTimeSeconds = 5
};
receiveMessageResponse = sqs.ReceiveMessageAsync(receiveMessageRequest);
}
The WaitTimeSeconds property of the ReceiveMessageRequest specifies the duration (in seconds) that the call waits for a message to arrive in the queue before returning a response to the calling application. There are a few benefits to using long polling:
For more information, see Amazon SQS Long Polling.
After you have returned messages from the queue, you can start to process them by looping through each message in the response and invoking a new BackgroundWorker thread.
// Process messages
if (receiveMessageResponse.Result.Messages != null)
{
foreach (var message in receiveMessageResponse.Result.Messages)
{
Console.WriteLine("Received SQS message, starting worker thread");
// Create background worker to process message
BackgroundWorker worker = new BackgroundWorker();
worker.DoWork += (obj, e) => ProcessMessage(message);
worker.RunWorkerAsync();
}
}
else
{
Console.WriteLine("No messages on queue");
}
The event handler, ProcessMessage, is where you implement business logic for processing orders. It is important to have a good understanding of how long a typical transaction takes so you can set a message VisibilityTimeout that is long enough to complete your operation. If order processing takes longer than the specified timeout period, the message becomes visible on the queue. Other nodes may pick it and process the same order twice, leading to unintended consequences.
In order to manage duplicate messages, seek to make your processing application idempotent. In mathematics, idempotent describes a function that produces the same result if it is applied to itself:
f(x) = f(f(x))
No matter how many times you process the same message, the end result is the same (definition from Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions, Hohpe and Wolf, 2004).
There are several strategies you could apply to achieve this:
var sendMessageRequest = new SendMessageRequest
{
QueueUrl = _queueUrl,
MessageBody = JsonConvert.SerializeObject(order),
MessageGroupId = Guid.NewGuid().ToString("N"),
MessageDeduplicationId = Guid.NewGuid().ToString("N")
};
Because of the distributed nature of SQS queues, it does not automatically delete the message. Therefore, you must explicitly delete the message from the queue after processing it, using the message ReceiptHandle property (see the following code example).
However, if at any stage you have an exception, avoid handling it as you normally would. The intention is to make sure that the message ends back on the queue, so that you can gracefully deal with intermittent failures. Instead, log the exception to capture diagnostic information, and swallow it.
By not explicitly deleting the message from the queue, you can take advantage of the VisibilityTimeout behavior described earlier. Gracefully handle the message processing failure and make the unprocessed message available to other nodes to process.
In the event that subsequent retries fail, SQS automatically moves the message to the configured DLQ after the configured number of receives has been reached. You can further investigate why the order process failed. Most importantly, the order has not been lost, and your customer is still your customer.
private static void ProcessMessage(Message message)
{
using (var sqs = new AmazonSQSClient())
{
try
{
Console.WriteLine("Processing message id: {0}", message.MessageId);
// Implement messaging processing here
// Ensure no downstream resource contention (parallel processing)
// <your order processing logic in here…>
Console.WriteLine("{0} Thread {1}: {2}", DateTime.Now.ToString("s"), Thread.CurrentThread.ManagedThreadId, message.MessageId);
// Delete the message off the queue.
// Receipt handle is the identifier you must provide
// when deleting the message.
var deleteRequest = new DeleteMessageRequest(_queueName, message.ReceiptHandle);
sqs.DeleteMessageAsync(deleteRequest);
Console.WriteLine("Processed message id: {0}", message.MessageId);
}
catch (Exception ex)
{
// Do nothing.
// Swallow exception, message will return to the queue when
// visibility timeout has been exceeded.
Console.WriteLine("Could not process message due to error. Exception: {0}", ex.Message);
}
}
}
One of the benefits of introducing a message queue is that you can accommodate new business requirements without dramatically affecting your application.
If, for example, the business decided that all orders placed over $5000 are to be handled as a priority, you could introduce a new “priority order” queue. The way the orders are processed does not change. The only significant change to the processing application is to ensure that messages from the “priority order” queue are processed before the “standard order” queue.
The following diagram shows how this logic could be isolated in an “order dispatcher,” whose only purpose is to route order messages to the appropriate queue based on whether the order exceeds $5000. Nothing on the web application or the processing nodes changes other than the target queue to which the order is sent. The rates at which orders are processed can be achieved by modifying the poll rates and scalability settings that I have already discussed.
Amazon SNS supports reliable publish-subscribe (pub-sub) scenarios and push notifications to known endpoints across a wide variety of protocols. It eliminates the need to periodically check or poll for new information and updates. SNS supports:
With these capabilities, you can provide parallel asynchronous processing of orders in the system and extend it to support any number of different business use cases without affecting the production environment. This is commonly referred to as a “fanout” scenario.
Rather than your web application pushing orders to a queue for processing, send a notification via SNS. The SNS messages are sent to a topic and then replicated and pushed to multiple SQS queues and Lambda functions for processing.
As the diagram above shows, you have the development team consuming “live” data as they work on the next version of the processing application, or potentially using the messages to troubleshoot issues in production.
Marketing is consuming all order information, via a Lambda function that has subscribed to the SNS topic, inserting the records into an Amazon Redshift warehouse for analysis.
All of this, of course, is happening without affecting your order processing application.
While I haven’t dived deep into the specifics of each service, I have discussed how these services can be applied at an architectural level to build loosely coupled systems that facilitate multiple business use cases. I’ve also shown you how to use infrastructure and application-level scaling techniques, so you can get the most out of your EC2 instances.
One of the many benefits of using these managed services is how quickly and easily you can implement powerful messaging capabilities in your systems, and lower the capital and operational costs of managing your own messaging middleware.
Using Amazon SQS and Amazon SNS together can provide you with a powerful mechanism for decoupling application components. This should be part of design considerations as you architect for the cloud.
For more information, see the Amazon SQS Developer Guide and Amazon SNS Developer Guide. You’ll find tutorials on all the concepts covered in this post, and more. To can get started using the AWS console or SDK of your choice visit:
Happy messaging!
Post Syndicated from Andy original https://torrentfreak.com/us-embassy-threatens-to-close-domain-registry-over-pirate-bay-domain-170620/
Domains have become an integral part of the piracy wars and no one knows this better than The Pirate Bay.
The site has burned through numerous domains over the years, with copyright holders and authorities successfully pressurizing registries to destabilize the site.
The latest news on this front comes from the Central American country of Costa Rica, where the local domain registry is having problems with the United States government.
The drama is detailed in a letter to ICANN penned by Dr. Pedro León Azofeifa, President of the Costa Rican Academy of Science, which operates NIC Costa Rica, the registry in charge of local .CR domain names.
Azofeifa’s letter is addressed to ICANN board member Thomas Schneider and pulls no punches. It claims that for the past two years the United States Embassy in Costa Rica has been pressuring NIC Costa Rica to take action against a particular domain.
“Since 2015, the United Estates Embassy in Costa Rica, who represents the interests of the United States Department of Commerce, has frequently contacted our organization regarding the domain name thepiratebay.cr,” the letter to ICANN reads.
“These interactions with the United States Embassy have escalated with time and include great pressure since 2016 that is exemplified by several phone calls, emails, and meetings urging our ccTLD to take down the domain, even though this would go against our domain name policies.”
The letter states that following pressure from the US, the Costa Rican Ministry of Commerce carried out an investigation which concluded that not taking down the domain was in line with best practices that only require suspensions following a local court order. That didn’t satisfy the United States though, far from it.
“The representative of the United States Embassy, Mr. Kevin Ludeke, Economic Specialist, who claims to represent the interests of the US Department of
Commerce, has mentioned threats to close our registry, with repeated harassment
regarding our practices and operation policies,” the letter to ICANN reads.
Ludeke is indeed listed on the US Embassy site for Costa Rica. He’s also referenced in a 2008 diplomatic cable leaked previously by Wikileaks. Contacted via email, Ludeke did not immediately respond to TorrentFreak’s request for comment.
Surprisingly, Azofeifa says the US representative then got personal, making negative comments towards his Executive Director, “based on no clear evidence or statistical data to support his claims, as a way to pressure our organization to take down the domain name without following our current policies.”
Citing the Tunis Agenda for the Information Society of 2005, Azofeifa asserts that “policy authority for Internet-related public policy issues is the sovereign right of the States,” which in Costa Rica’s case means that there must be “a final judgment from the Courts of Justice of the Republic of Costa Rica” before the registry will suspend a domain.
But it seems legal action was not the preferred route of the US Embassy. Demanding that NIC Costa Rica take unilateral action, Mr. Ludeke continued with “pressure and harassment to take down the domain name without its proper process and local court order.”
Azofeifa’s letter to ICANN, which is cc’d to Stafford Fitzgerald Haney, United States Ambassador to Costa Rica and various people in the Costa Rican Ministry of Commerce, concludes with a request for suggestions on how to deal with the matter.
While the response should prove very interesting, none of the parties involved appear to have noticed that ThePirateBay.cr isn’t officially connected to The Pirate Bay
The domain and associated site appeared in the wake of the December 2014 shut down of The Pirate Bay, claiming to be the real deal and even going as far as making fake accounts in the names of famous ‘pirate’ groups including ettv and YIFY.
Today it acts as an unofficial and unaffiliated reverse proxy to The Pirate Bay while presenting the site’s content as its own. It’s also affiliated with a fake KickassTorrents site, Kickass.cd, which to this day claims that it’s a reincarnation of the defunct torrent giant.
But perhaps the most glaring issue in this worrying case is the apparent willingness of the United States to call out Costa Rica for not doing anything about a .CR domain run by third parties, when the real Pirate Bay’s .org domain is under United States’ jurisdiction.
Registered by the Public Interest Registry in Reston, Virginia, ThePirateBay.org is the famous site’s main domain. TorrentFreak asked PIR if anyone from the US government had ever requested action against the domain but at the time of publication, we had received no response.
Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.
Post Syndicated from Lennart Poettering original http://0pointer.net/blog/casync-a-tool-for-distributing-file-system-images.html
In the past months I have been working on a new project:
casync. casync takes
inspiration from the popular rsync file
synchronization tool as well as the probably even more popular
git revision control system. It combines the
idea of the rsync algorithm with the idea of git-style
content-addressable file systems, and creates a new system for
efficiently storing and delivering file system images, optimized for
high-frequency update cycles over the Internet. Its current focus is
on delivering IoT, container, VM, application, portable service or OS
images, but I hope to extend it later in a generic fashion to become
useful for backups and home directory synchronization as well (but
more about that later).
The basic technological building blocks casync is built from are
neither new nor particularly innovative (at least not anymore),
however the way casync combines them is different from existing tools,
and that’s what makes it useful for a variety of usecases that other
tools can’t cover that well.
I created casync after studying how today’s popular tools store and
deliver file system images. To very incomprehensively and briefly name
a few: Docker has a layered tarball approach,
OSTree serves the
individual files directly via HTTP and maintains packed deltas to
speed up updates, while other systems operate on the block layer and
place raw squashfs images (or other archival file systems, such as
IS09660) for download on HTTP shares (in the better cases combined
with zsync data).
Neither of these approaches appeared fully convincing to me when used
in high-frequency update cycle systems. In such systems, it is
important to optimize towards a couple of goals:
I don’t think any of the tools mentioned above are really good on more
than a small subset of these points.
Specifically: Docker’s layered tarball approach dumps the “delta”
question onto the feet of the image creators: the best way to make
your image downloads minimal is basing your work on an existing image
clients might already have, and inherit its resources, maintaing full
history. Here, revision control (a tool for the developer) is
intermingled with update management (a concept for optimizing
production delivery). As container histories grow individual deltas
are likely to stay small, but on the other hand a brand-new deployment
usually requires downloading the full history onto the deployment
system, even though there’s no use for it there, and likely requires
substantially more disk space and download sizes.
OSTree’s serving of individual files is unfriendly to CDNs (as many
small files in file trees cause an explosion of HTTP GET
requests). To counter that OSTree supports placing pre-calculated
delta images between selected revisions on the delivery servers, which
means a certain amount of revision management, that leaks into the
clients.
Delivering direct squashfs (or other file system) images is almost
beautifully simple, but of course means every update requires a full
download of the newest image, which is both bad for disk usage and
generated traffic. Enhancing it with zsync makes this a much better
option, as it can reduce generated traffic substantially at very
little cost of history/metadata (no explicit deltas between a large
number of versions need to be prepared server side). On the other hand
server requirements in disk space and functionality (HTTP Range
requests) are minus points for the usecase I am interested in.
(Note: all the mentioned systems have great properties, and it’s not
my intention to badmouth them. They only point I am trying to make is
that for the use case I care about — file system image delivery with
high high frequency update-cycles — each system comes with certain
drawbacks.)
Besides the issues pointed out above I wasn’t happy with the security
and reproducability properties of these systems. In today’s world
where security breaches involving hacking and breaking into connected
systems happen every day, an image delivery system that cannot make
strong guarantees regarding data integrity is out of
date. Specifically, the tarball format is famously undeterministic:
the very same file tree can result in any number of different
valid serializations depending on the tool used, its version and the
underlying OS and file system. Some tar implementations attempt to
correct that by guaranteeing that each file tree maps to exactly
one valid serialization, but such a property is always only specific
to the tool used. I strongly believe that any good update system must
guarantee on every single link of the chain that there’s only one
valid representatin of the data to deliver, that can easily be
verified.
So much about the background why I created casync. Now, let’s have a
look what casync actually is like, and what it does. Here’s the brief
technical overview:
Encoding: Let’s take a large linear data stream, split it into
variable-sized chunks (the size of each being a function of the
chunk’s contents), and store these chunks in individual, compressed
files in some directory, each file named after a strong hash value of
its contents, so that the hash value may be used to as key for
retrieving the full chunk data. Let’s call this directory a “chunk
store”. At the same time, generate a “chunk index” file that lists
these chunk hash values plus their respective chunk sizes in a simple
linear array. The chunking algorithm is supposed to create variable,
but similarly sized chunks from the data stream, and do so in a way
that the same data results in the same chunks even if placed at
varying offsets. For more information see this blog
story.
Decoding: Let’s take the chunk index file, and reassemble the large
linear data stream by concatenating the uncompressed chunks retrieved
from the chunk store, keyed by the listed chunk hash values.
As an extra twist, we introduce a well-defined, reproducible,
random-access serialization format for file trees (think: a more
modern tar), to permit efficient, stable storage of complete file
trees in the system, simply by serializing them and then passing them
into the encoding step explained above.
Finally, let’s put all this on the network: for each image you want to
deliver, generate a chunk index file and place it on an HTTP
server. Do the same with the chunk store, and share it between the
various index files you intend to deliver.
Why bother with all of this? Streams with similar contents will result
in mostly the same chunk files in the chunk store. This means it is
very efficient to store many related versions of a data stream in the
same chunk store, thus minimizing disk usage. Moreover, when
transferring linear data streams chunks already known on the receiving
side can be made use of, thus minimizing network traffic.
Why is this different from rsync or OSTree, or similar tools? Well,
one major difference between casync and those tools is that we
remove file boundaries before chunking things up. This means that
small files are lumped together with their siblings and large files
are chopped into pieces, which permits us to recognize similarities in
files and directories beyond file boundaries, and makes sure our chunk
sizes are pretty evenly distributed, without the file boundaries
affecting them.
The “chunking” algorithm is based on a the buzhash rolling hash
function. SHA256 is used as strong hash function to generate digests
of the chunks. xz is used to compress the individual chunks.
Here’s a diagram, hopefully explaining a bit how the encoding process
works, wasn’t it for my crappy drawing skills:
The diagram shows the encoding process from top to bottom. It starts
with a block device or a file tree, which is then serialized and
chunked up into variable sized blocks. The compressed chunks are then
placed in the chunk store, while a chunk index file is written listing
the chunk hashes in order. (The original SVG of this graphic may be
found here.
Note that casync operates on two different layers, depending on the
usecase of the user:
You may use it on the block layer. In this case the raw block data
on disk is taken as-is, read directly from the block device, split
into chunks as described above, compressed, stored and delivered.
You may use it on the file system layer. In this case, the
file tree serialization format mentioned above comes into play:
the file tree is serialized depth-first (much like tar would do
it) and then split into chunks, compressed, stored and delivered.
The fact that it may be used on both the block and file system layer
opens it up for a variety of different usecases. In the VM and IoT
ecosystems shipping images as block-level serializations is more
common, while in the container and application world file-system-level
serializations are more typically used.
Chunk index files referring to block-layer serializations carry the
.caibx suffix, while chunk index files referring to file system
serializations carry the .caidx suffix. Note that you may also use
casync as direct tar replacement, i.e. without the chunking, just
generating the plain linear file tree serialization. Such files
carry the .catar suffix. Internally .caibx are identical to
.caidx files, the only difference is semantical: .caidx files
describe a .catar file, while .caibx files may describe any other
blob. Finally, chunk stores are directories carrying the .castr
suffix.
Here are a couple of other features casync has:
When downloading a new image you may use casync‘s --seed=
feature: each block device, file, or directory specified is processed
using the same chunking logic described above, and is used as
preferred source when putting together the downloaded image locally,
avoiding network transfer of it. This of course is useful whenever
updating an image: simply specify one or more old versions as seed and
only download the chunks that truly changed since then. Note that
using seeds requires no history relationship between seed and the new
image to download. This has major benefits: you can even use it to
speed up downloads of relatively foreign and unrelated data. For
example, when downloading a container image built using Ubuntu you can
use your Fedora host OS tree in /usr as seed, and casync will
automatically use whatever it can from that tree, for example timezone
and locale data that tends to be identical between
distributions. Example: casync extract. This
http://example.com/myimage.caibx --seed=/dev/sda1 /dev/sda2
will place the block-layer image described by the indicated URL in the
/dev/sda2 partition, using the existing /dev/sda1 data as seeding
source. An invocation like this could be typically used by IoT systems
with an A/B partition setup. Example 2: casync extract, is very similar but
http://example.com/mycontainer-v3.caidx --seed=/srv/container-v1
--seed=/srv/container-v2 /src/container-v3
operates on the file system layer, and uses two old container versions
to seed the new version.
When operating on the file system level, the user has fine-grained
control on the metadata included in the serialization. This is
relevant since different usecases tend to require a different set of
saved/restored metadata. For example, when shipping OS images, file
access bits/ACLs and ownership matter, while file modification times
hurt. When doing personal backups OTOH file ownership matters little
but file modification times are important. Moreover different backing
file systems support different feature sets, and storing more
information than necessary might make it impossible to validate a tree
against an image if the metadata cannot be replayed in full. Due to
this, casync provides a set of --with= and --without= parameters
that allow fine-grained control of the data stored in the file tree
serialization, including the granularity of modification times and
more. The precise set of selected metadata features is also always
part of the serialization, so that seeding can work correctly and
automatically.
casync tries to be as accurate as possible when storing file
system metadata. This means that besides the usual baseline of file
metadata (file ownership and access bits), and more advanced features
(extended attributes, ACLs, file capabilities) a number of more exotic
data is stored as well, including Linux
chattr(1) file attributes, as
well as FAT file
attributes
(you may wonder why the latter? — EFI is FAT, and /efi is part of
the comprehensive serialization of any host). In the future I intend
to extend this further, for example storing btrfs subvolume
information where available. Note that as described above every single
type of metadata may be turned off and on individually, hence if you
don’t need FAT file bits (and I figure it’s pretty likely you don’t),
then they won’t be stored.
The user creating .caidx or .caibx files may control the desired
average chunk length (before compression) freely, using the
--chunk-size= parameter. Smaller chunks increase the number of
generated files in the chunk store and increase HTTP GET load on the
server, but also ensure that sharing between similar images is
improved, as identical patterns in the images stored are more likely
to be recognized. By default casync will use a 64K average chunk
size. Tweaking this can be particularly useful when adapting the
system to specific CDNs, or when delivering compressed disk images
such as squashfs (see below).
Emphasis is placed on making all invocations reproducible,
well-defined and strictly deterministic. As mentioned above this is a
requirement to reach the intended security guarantees, but is also
useful for many other usecases. For example, the casync digest
command may be used to calculate a hash value identifying a specific
directory in all desired detail (use --with= and --without to pick
the desired detail). Moreover the casync mtree command may be used
to generate a BSD mtree(5) compatible manifest of a directory tree,
.caidx or .catar file.
The file system serialization format is nicely composable. By this
I mean that the serialization of a file tree is the concatenation of
the serializations of all files and file subtrees located at the
top of the tree, with zero metadata references from any of these
serializations into the others. This property is essential to ensure
maximum reuse of chunks when similar trees are serialized.
When extracting file trees or disk image files, casync
will automatically create
reflinks
from any specified seeds if the underlying file system supports it
(such as btrfs, ocfs, and future xfs). After all, instead of
copying the desired data from the seed, we can just tell the file
system to link up the relevant blocks. This works both when extracting
.caidx and .caibx files — the latter of course only when the
extracted disk image is placed in a regular raw image file on disk,
rather than directly on a plain block device, as plain block devices
do not know the concept of reflinks.
Optionally, when extracting file trees, casync can
create traditional UNIX hardlinks for identical files in specified
seeds (--hardlink=yes). This works on all UNIX file systems, and can
save substantial amounts of disk space. However, this only works for
very specific usecases where disk images are considered read-only
after extraction, as any changes made to one tree will propagate to
all other trees sharing the same hardlinked files, as that’s the
nature of hardlinks. In this mode, casync exposes OSTree-like
behaviour, which is built heavily around read-only hardlink trees.
casync tries to be smart when choosing what to include in file
system images. Implicitly, file systems such as procfs and sysfs are
exluded from serialization, as they expose API objects, not real
files. Moreover, the “nodump” (+d)
chattr(1) flag is honoured by
default, permitting users to mark files to exclude from serialization.
When creating and extracting file trees casync may apply am
automatic or explicit UID/GID shift. This is particularly useful when
transferring container image for use with Linux user namespacing.
In addition to local operation, casync currently supports HTTP,
HTTPS, FTP and ssh natively for downloading chunk index files and
chunks (the ssh mode requires installing casync on the remote host,
though, but an sftp mode not requiring that should be easy to
add). When creating index files or chunks, only ssh is supported as
remote backend.
When operating on block-layer images, you may expose locally or
remotely stored images as local block devices. Example: casync mkdev exposes the disk image described by
http://example.com/myimage.caibx
the indicated URL as local block device in /dev, which you then may
use the usual block device tools on, such as mount or fdisk (only
read-only though). Chunks are downloaded on access with high priority,
and at low priority when idle in the background. Note that in this
mode, casync also plays a role similar to “dm-verity”, as all blocks
are validated against the strong digests in the chunk index file
before passing them on to the kernel’s block layer. This feature is
implemented though Linux’ NBD kernel facility.
Similar, when operating on file-system-layer images, you may mount
locally or remotely stored images as regular file systems. Example:
casync mount http://example.com/mytree.caidx /srv/mytree mounts the
file tree image described by the indicated URL as a local directory
/srv/mytree. This feature is implemented though Linux’ FUSE kernel
facility. Note that special care is taken that the images exposed this
way can be packed up again with casync make and are guaranteed to
return the bit-by-bit exact same serialization again that it was
mounted from. No data is lost or changed while passing thrings through
FUSE (OK, strictly speaking this is a lie, we do lose ACLs, but that’s
hopefully just a temporary gap to be fixed soon).
In IoT A/B fixed size partition setups the file systems placed in
the two partitions are usually much shorter than the partition size,
in order to keep some room for later, larger updates. casync is able
to analyze the superblock of a number of common file systems in order
to determine the actual size of a file system stored on a block
device, so that writing a file system to such a partition and reading
it back again will result in reproducible data. Moreover this speeds
up the seeding process, as there’s little point in seeding the
whitespace after the file system within the partition.
Here’s how to use casync, explained with a few examples:
$ casync make foobar.caidx /some/directory
This will create a chunk index file foobar.caidx in the local
directory, and populate the chunk store directory default.castr
located next to it with the chunks of the serialization (you can
change the name for the store directory with --store= if you
like). This command operates on the file-system level. A similar
command operating on the block level:
$ casync make foobar.caibx /dev/sda1
This command creates a chunk index file foobar.caibx in the local
directory describing the current contents of the /dev/sda1 block
device, and populates default.castr in the same way as above. Note
that you may as well read a raw disk image from a file instead of a
block device:
$ casync make foobar.caibx myimage.raw
To reconstruct the original file tree from the .caidx file and
the chunk store of the first command, use:
$ casync extract foobar.caidx /some/other/directory
And similar for the block-layer version:
$ casync extract foobar.caibx /dev/sdb1
or, to extract the block-layer version into a raw disk image:
$ casync extract foobar.caibx myotherimage.raw
The above are the most basic commands, operating on local data
only. Now let’s make this more interesting, and reference remote
resources:
$ casync extract http://example.com/images/foobar.caidx /some/other/directory
This extracts the specified .caidx onto a local directory. This of
course assumes that foobar.caidx was uploaded to the HTTP server in
the first place, along with the chunk store. You can use any command
you like to accomplish that, for example scp or
rsync. Alternatively, you can let casync do this directly when
generating the chunk index:
$ casync make ssh.example.com:images/foobar.caidx /some/directory
This will use ssh to connect to the ssh.example.com server, and then
places the .caidx file and the chunks on it. Note that this mode of
operation is “smart”: this scheme will only upload chunks currently
missing on the server side, and not retransmit what already is
available.
Note that you can always configure the precise path or URL of the
chunk store via the --store= option. If you do not do that, then the
store path is automatically derived from the path or URL: the last
component of the path or URL is replaced by default.castr.
Of course, when extracting .caidx or .caibx files from remote sources,
using a local seed is advisable:
$ casync extract http://example.com/images/foobar.caidx --seed=/some/exising/directory /some/other/directory
Or on the block layer:
$ casync extract http://example.com/images/foobar.caibx --seed=/dev/sda1 /dev/sdb2
When creating chunk indexes on the file system layer casync will by
default store metadata as accurately as possible. Let’s create a chunk
index with reduced metadata:
$ casync make foobar.caidx --with=sec-time --with=symlinks --with=read-only /some/dir
This command will create a chunk index for a file tree serialization
that has three features above the absolute baseline supported: 1s
granularity timestamps, symbolic links and a single read-only bit. In
this mode, all the other metadata bits are not stored, including
nanosecond timestamps, full unix permission bits, file ownership or
even ACLs or extended attributes.
Now let’s make a .caidx file available locally as a mounted file
system, without extracting it:
$ casync mount http://example.comf/images/foobar.caidx /mnt/foobar
And similar, let’s make a .caibx file available locally as a block device:
$ casync mkdev http://example.comf/images/foobar.caidx
This will create a block device in /dev and print the used device
node path to STDOUT.
As mentioned, casync is big about reproducability. Let’s make use of
that to calculate the a digest identifying a very specific version of
a file tree:
$ casync digest .
This digest will include all metadata bits casync and the underlying
file system know about. Usually, to make this useful you want to
configure exactly what metadata to include:
$ casync digest --with=unix .
This makes use of the --with=unix shortcut for selecting metadata
fields. Specifying --with-unix= selects all metadata that
traditional UNIX file systems support. It is a shortcut for writing out:
--with=16bit-uids --with=permissions --with=sec-time --with=symlinks.
--with=device-nodes --with=fifos --with=sockets
Note that when calculating digests or creating chunk indexes you may
also use the negative --without= option to remove specific features
but start from the most precise:
$ casync digest --without=flag-immutable
This generates a digest with the most accurate metadata, but leaves
one feature out: chattr(1)‘s
immutable (+i) file flag.
To list the contents of a .caidx file use a command like the following:
$ casync list http://example.com/images/foobar.caidx
or
$ casync mtree http://example.com/images/foobar.caidx
The former command will generate a brief list of files and
directories, not too different from tar t or ls -al in its
output. The latter command will generate a BSD
mtree(5) compatible
manifest. Note that casync actually stores substantially more file
metadata than mtree files can express, though.
casync is not an attempt to minimize serialization and downloaded
deltas to the extreme. Instead, the tool is supposed to find a good
middle ground, that is good on traffic and disk space, but not at the
price of convenience or requiring explicit revision control. If you
care about updates that are absolutely minimal, there are binary delta
systems around that might be an option for you, such as Google’s
Courgette.
casync is not a replacement for rsync, or git or zsync or
anything like that. They have very different usecases and
semantics. For example, rsync permits you to directly synchronize two
file trees remotely. casync just cannot do that, and it is unlikely
it every will.
casync is supposed to be a generic synchronization tool. Its primary
focus for now is delivery of OS images, but I’d like to make it useful
for a couple other usecases, too. Specifically:
To make the tool useful for backups, encryption is missing. I have
pretty concrete plans how to add that. When implemented, the tool
would might become an alternative to
restic or
tarsnap.
Right now, if you want to deploy casync in real-life, you still
need to validate the downloaded .caidx or .caibx file yourself, for
example with some gpg signature. It is my intention to integrate with
gpg in a minimal way so that signing and verifying chunk index files
is done automatically.
In the longer run, I’d like to build an automatic synchronizer for
$HOME between systems from this. Each $HOME instance would be
stored automatically in regular intervals in the cloud using casync,
and conflicts would be resolved locally.
casync is written in a shared library style, but it is not yet
built as one. Specifically this means that almost all of casync‘s
functionality is supposed to be available as C API soon, and
applications can process casync files on every level. It is my
intention to make this library useful enough so that it will be easy
to write a module for GNOME’s gvfs subsystem in order to make remote
or local .caidx files directly available to applications (as an
alternative to casync mount). In fact the idea is to make this all
flexible enough that even the remoting backends can be replaced
easily, for example to replace casync‘s default HTTP/HTTPS backends
built on CURL with GNOME’s own HTTP implementation, in order to share
cookies, certificates, … There’s also an alternative method to
integrate with casync in place already: simply invoke casync as a
subprocess. casync will inform you about a certain set of state
changes using a mechanism compatible with
sd_notify(3). In
future it will also propagate progress data this way and more.
I intend to a add a new seeding back-end that sources chunks from
the local network. After downloading the new .caidx file off the
Internet casync would then search for the listed chunks on the local
network first before retrieving them from the Internet. This should
speed things up on all installations that have multiple similar
systems deployed in the same network.
Further plans are listed tersely in the
TODO file.
Is this a systemd project? — casync is hosted under the
github systemd umbrella, and the
projects share the same coding style. However, the codebases are
distinct and without interdependencies, and casync works fine both
on systemd systems and systems without it.
Is casync portable? — At the moment: no. I only run Linux and
that’s what I code for. That said, I am open to accepting portability
patches (unlike for systemd, which doesn’t really make sense on
non-Linux systems), as long as they don’t interfere too much with the
way casync works. Specifically this means that I am not too
enthusiastic about merging portability patches for OSes lacking the
openat(2) family
of APIs.
Does casync require reflink-capable file systems to work, such
as btrfs? No it doesn’t. The reflink magic in casync is
employed when the file system permits it, and it’s good to have it,
but it’s not a requirement, and casync will implicitly fall back to
copying when it isn’t available. Note that casync supports a number
of file system features on a variety of file systems that aren’t
available everywhere, for example FAT’s system/hidden file flags or
xfs‘s projinherit file flag.
Is casync stable? — I just tagged the first, initial
release. While I have been working on it since quite some time and it
is quite featureful, this is the first time I advertise it publicly,
and it hence received very little testing outside of its own test
suite. I am also not fully ready to commit to the stability of the
current serialization or chunk index format. I don’t see any breakages
coming for it though. casync is pretty light on documentation right
now, and does not even have a man page. I also intend to correct that
soon.
Are the .caidx/.caibx and .catar file formats open and
documented? — casync is Open Source, so if you want to know the
precise format, have a look at the sources for now. It’s definitely my
intention to add comprehensive docs for both formats however. Don’t
forget this is just the initial version right now.
casync is just like $SOMEOTHERTOOL! Why are you reinventing
the wheel (again)? — Well, because casync isn’t “just like” some
other tool. I am pretty sure I did my homework, and that there is no
tool just like casync right now. The tools coming closest are probably
rsync, zsync, tarsnap, restic, but they are quite different beasts
each.
Why did you invent your own serialization format for file trees?
Why don’t you just use tar? That’s a good question, and other
systems — most prominently tarsnap — do that. However, as mentioned
above tar doesn’t enforce reproducability. It also doesn’t really do
random access: if you want to access some specific file you need to
read every single byte stored before it in the tar archive to find
it, which is of course very expensive. The serialization casync
implements places a focus on reproducability, random access, and
metadata control. Much like traditional tar it can still be
generated and extracted in a stream fashion though.
Does casync save/restore SELinux/SMACK file labels? At the
moment not. That’s not because I wouldn’t want it to, but simply
because I am not a guru of either of these systems, and didn’t want to
implement something I do not fully grok nor can test. If you look at
the sources you’ll find that there’s already some definitions in place
that keep room for them though. I’d be delighted to accept a patch
implementing this fully.
What about delivering squashfs images? How well does chunking
work on compressed serializations? – That’s a very good point!
Usually, if you apply the a chunking algorithm to a compressed data
stream (let’s say a tar.gz file), then changing a single bit at the
front will propagate into the entire remainder of the file, so that
minimal changes will explode into major changes. Thankfully this
doesn’t apply that strictly to squashfs images, as it provides
random access to files and directories and thus breaks up the
compression streams in regular intervals to make seeking easy. This
fact is beneficial for systems employing chunking, such as casync as
this means single bit changes might affect their vicinity but will not
explode in an unbounded fashion. In order achieve best results when
delivering squashfs images through casync the block sizes of
squashfs and the chunks sizes of casync should be matched up (using
casync‘s --chunk-size= option). How precisely to choose both
values is left to reasearch by the user, for now.
What does the name casync mean? – It’s a synchronizing
tool, hence the -sync suffix, following rsync‘s naming. It makes
use of the content-addressable concept of git hence the ca-
prefix.
Where can I get this stuff? Is it already packaged? – Check
out the sources on GitHub. I
just tagged the first
version. Martin
Pitt has packaged casync for
Ubuntu. There
is also an ArchLinux
package.
Well, that’s up to you really. If you are involved with projects that
need to deliver IoT, VM, container, application or OS images, then
maybe this is a great tool for you — but other options exist, some of
which are linked above.
Note that casync is an Open Source project: if it doesn’t do exactly
what you need, prepare a patch that adds what you need, and we’ll
consider it.
If you are interested in the project and would like to talk about this
in person, I’ll be presenting casync soon at Kinvolk’s Linux
Technologies
Meetup
in Berlin, Germany. You are invited. I also intend to talk about it at
All Systems Go!, also in Berlin.
Post Syndicated from Andy original https://torrentfreak.com/the-pirate-bay-isnt-affected-by-adverse-court-rulings-everyone-else-is-170618/
For more than a decade The Pirate Bay has been the world’s most controversial site. Delivering huge quantities of copyrighted content to the masses, the platform is revered and reviled across the copyright spectrum.
Its reputation is one of a defiant Internet swashbuckler, but due to changes in how the site has been run in more recent times, its current philosophy is more difficult to gauge. What has never been in doubt, however, is the site’s original intent to be as provocative as possible.
Through endless publicity stunts, some real, some just for the ‘lulz’, The Pirate Bay managed to attract a massive audience, all while incurring the wrath of every major copyright holder in the world.
Make no mistake, they all queued up to strike back, but every subsequent rightsholder action was met by a Pirate Bay middle finger, two fingers, or chin flick, depending on the mood of the day. This only served to further delight the masses, who happily spread the word while keeping their torrents flowing.
This vicious circle of being targeted by the entertainment industries, mocking them, and then reaping the traffic benefits, developed into the cheapest long-term marketing campaign the Internet had ever seen. But nothing is ever truly for free and there have been consequences.
After taunting Hollywood and the music industry with its refusals to capitulate, endless legal action that the site would have ordinarily been forced to participate in largely took place without The Pirate Bay being present. It doesn’t take a law degree to work out what happened in each and every one of those cases, whatever complex route they took through the legal system. No defense, no win.
For example, the web-blocking phenomenon across the UK, Europe, Asia and Australia was driven by the site’s absolute resilience and although there would clearly have been other scapegoats had The Pirate Bay disappeared, the site was the ideal bogeyman the copyright lobby required to move forward.
Filing blocking lawsuits while bringing hosts, advertisers, and ISPs on board for anti-piracy initiatives were also made easier with the ‘evil’ Pirate Bay still online. Immune from every anti-piracy technique under the sun, the existence of the platform in the face of all onslaughts only strengthened the cases of those arguing for even more drastic measures.
Over a decade, this has meant a significant tightening of the sharing and streaming climate. Without any big legislative changes but plenty of case law against The Pirate Bay, web-blocking is now a walk in the park, ad hoc domain seizures are a fairly regular occurrence, and few companies want to host sharing sites. Advertisers and brands are also hesitant over where they place their ads. It’s a very different world to the one of 10 years ago.
While it would be wrong to attribute every tightening of the noose to the actions of The Pirate Bay, there’s little doubt that the site and its chaotic image played a huge role in where copyright enforcement is today. The platform set out to provoke and succeeded in every way possible, gaining supporters in their millions. It could also be argued it kicked a hole in a hornets’ nest, releasing the hell inside.
But perhaps the site’s most amazing achievement is the way it has managed to stay online, despite all the turmoil.
This week yet another ruling, this time from the powerful European Court of Justice, found that by offering links in the manner it does, The Pirate Bay and other sites are liable for communicating copyright works to the public. Of course, this prompted the usual swathe of articles claiming that this could be the final nail in the site’s coffin.
Wrong.
In common with every ruling, legal defeat, and legislative restriction put in place due to the site’s activities, this week’s decision from the ECJ will have zero effect on the Pirate Bay’s availability. For right or wrong, the site was breaking the law long before this ruling and will continue to do so until it decides otherwise.
What we have instead is a further tightened legal landscape that will have a lasting effect on everything BUT the site, including weaker torrent sites, Internet users, and user-uploaded content sites such as YouTube.
With The Pirate Bay carrying on regardless, that is nothing short of remarkable.
Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.
Post Syndicated from Ernesto original https://torrentfreak.com/comodo-dns-blocks-torrentfreak-over-hacking-and-warez-170617/
Website blocking has become one of the go-to methods for reducing online copyright infringement.
In addition to court-ordered blockades, various commercial vendors also offer a broad range of blocking tools. This includes Comodo, which offers a free DNS service that keeps people away from dangerous sites.
The service labeled SecureDNS is part of the Comodo Internet Security bundle but can be used by the general public as well, without charge. Just change the DNS settings on your computer or any other device, and you’re ready to go.
“As a leading provider of computer security solutions, Comodo is keenly aware of the dangers that plague the Internet today. SecureDNS helps users keep safe online with its malware domain filtering feature,” the company explains.
Aside from malware and spyware, Comodo also blocks access to sites that offer access to pirated content. Or put differently, they try to do this. But it’s easier said than done.
This week we were alerted to the fact that Comodo blocks direct access to TorrentFreak. Those who try to access our news site get an ominous warning instead, suggesting that we might share pirated content.
“This website has been blocked temporarily because of the following reason(s): Hacking/Warez: Site may offer illegal sharing of copyrighted software or media,” the warning reads, adding that several users also reported the site to be unsafe.
People can still access the site by clicking on a big red cross, although that’s something Comodo doesn’t recommend. However, it is quite clear that new readers will be pretty spooked by the alarming message.
We assume that TorrentFreak was added to Comodo’s blocklist by mistake. And while mistakes can happen everywhere, this once again show that overblocking is a serious concern.
We are lucky enough that readers alerted us to the problem, but in other cases, it could easily go unnoticed.
Interestingly, the ‘piracy’ blocklist is not as stringent as the above would suggest. While we replicated the issue, we also checked several other known ‘pirate’ sites including The Pirate Bay, RARBG, GoMovies, and Pubfilm. These could all be accessed through SecureDNS without any warning.
TorrentFreak contacted Comodo for a comment on their curious blocking efforts, but we have yet to hear back from the company. In the meantime, Comodo SecureDNS users may want to consider switching to a more open DNS provider.
Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.
Post Syndicated from Andy original https://torrentfreak.com/kodi-boxes-are-a-fire-risk-awful-timing-or-opportunism-170618/
Anyone who saw the pictures this week couldn’t have failed to be moved by the plight of Londoners caught up in the Grenfell Tower inferno. The apocalyptic images are likely to stay with people for years to come and the scars for those involved may never heal.
As the building continued to smolder and the death toll increased, UK tabloids provided wall-to-wall coverage of the disaster. On Thursday, however, The Sun took a short break to put out yet another sensationalized story about Kodi. Given the week’s events, it was bound to raise eyebrows.
“HOT GOODS: Kodi boxes are a fire hazard because thousands of IPTV devices nabbed by customs ‘failed UK electrical standards’,” the headline reads.
“It’s estimated that thousands of Brits have bought so-called Kodi boxes which can be connected to telly sets to stream pay-per-view sport and films for free,” the piece continued.
“But they could be a fire hazard, according to the Federation Against Copyright Theft (FACT), which has been nabbing huge deliveries of the devices as they arrive in the UK.”
As the image below shows, “Kodi box” fire hazard claims appeared next to images from other news articles about the huge London fire. While all separate stories, the pairing is not a great look.
FACT chief executive Kieron Sharp told The Sun that his group had uncovered two parcels of 2,000 ‘Kodi’ boxes and found that they “failed electrical safety standards”, making them potentially dangerous. While that may well be the case, the big question is all about timing.
It’s FACT’s job to reduce copyright infringement on behalf of clients such as The Premier League so it’s no surprise that they’re making a sustained effort to deter the public from buying these devices. That being said, it can’t have escaped FACT or The Sun that fire and death are extremely sensitive topics this week.
That leaves us with a few options including unfortunate opportunism or perhaps terrible timing, but let’s give the benefit of the doubt for a moment.
There’s a good argument that FACT and The Sun brought a valid issue to the public’s attention at a time when fire safety is on everyone’s lips. So, to give credit where it’s due, providing people with a heads-up about potentially dangerous devices is something that most people would welcome.
However, it’s difficult to offer congratulations on the PSA when the story as it appears in The Sun does nothing – absolutely nothing – to help people stay safe.
If some boxes are a risk (and that’s certainly likely given the level of Far East imports coming into the UK) which ones are dangerous? Where were they manufactured? Who sold them? What are the serial numbers? Which devices do people need to get out of their houses?
Sadly, none of these questions were answered or even addressed in the article, making it little more than scaremongering. Only making matters worse, the piece notes that it isn’t even clear how many of the seized devices are indeed a fire risk and that more tests need to be done. Is this how we should tackle such an important issue during an extremely sensitive week?
Timing and lack of useful information aside, one then has to question the terminology employed in the article.
As a piece of computer software, Kodi cannot catch fire. So, what we’re actually talking about here is small computers coming into the country without passing safety checks. The presence of Kodi on the devices – if indeed Kodi was even installed pre-import – is absolutely irrelevant.
Anti-piracy groups warning people of the dangers associated with their piracy habits is nothing new. For years, Internet users have been told that their computers will become malware infested if they share files or stream infringing content. While in some cases that may be true, there’s rarely any effort by those delivering the warnings to inform people on how to stay safe.
A classic example can be found in the numerous reports put out by the Digital Citizens Alliance in the United States. The DCA has produced several and no doubt expensive reports which claim to highlight the risks Internet users are exposed to on ‘pirate’ sites.
The DCA claims to do this in the interests of consumers but the group offers no practical advice on staying safe nor does it provide consumers with risk reduction strategies. Like many high-level ‘drug prevention’ documents shuffled around government, it could be argued that on a ‘street’ level their reports are next to useless.
Demonizing piracy is a well-worn and well-understood strategy but if warnings are to be interpreted as representing genuine concern for the welfare of people, they have to be a lot more substantial than mere scaremongering.
Anyone concerned about potentially dangerous devices can check out these useful guides from Electrical Safety First (pdf) and the Electrical Safety Council (pdf)
Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.
Post Syndicated from Andy original https://torrentfreak.com/pirates-cost-australias-ten-network-hundreds-of-millions-of-dollars-170616/
In 2016, Australia’s Ten Network posted losses of AUS$157 million. This April, the broadcaster showed signs of continuing distress when it posted a half-year loss of AUS$232 million.
In a statement to the stock exchange, Ten said it was trying to secure new terms for a AUS$200 million debt financing guarantee. According to ABC, the company had lost more than 60% of its value in the preceding 12 months and almost 98% over the previous five years.
More bad news arrived this week when Ten’s board decided to put the company into voluntary administration after failing to secure a guarantee for a AUS$250 million loan that could’ve kept the ship afloat into the new year. As moves get underway to secure the company’s future, fingers of blame are being raised.
According to Village Roadshow co-chief executive Graham Burke, Internet pirates cost Ten “hundreds of millions of dollars” in advertising revenue due to their tendency to obtain movies and TV shows from the web rather than via legitimate means.
Burke told The Australian (paywall) that movies supplied to Ten by 21st Century Fox (including The Revenant and The Peanuts Movie which were both leaked) had received lower broadcast ratings due to people viewing them online in advance.
“Piracy is a much bigger channel and an illicit economy than the three main commercial networks combined,” Burke told the publication.
“Movies from Fox arrive with several million people having seen them through piracy. If it wasn’t for piracy, the ratings would be stronger and the product would not be arriving clapped out.”
But leaked or not, content doesn’t come cheap. As part of efforts to remain afloat, Ten Network recently tried to re-negotiate content supply deals with Fox and CBS. Together they reportedly cost the broadcaster more than AUS$900 million over the previous six years.
Despite this massive price tag and numerous other problems engulfing the troubled company, Burke suggests it is pirates that are to blame for Ten’s demise.
“A large part of Ten’s expenditure is on movies and they are being seen by millions of people illegitimately on websites supported by rogue advertising for drugs, prostitution and even legitimate advertising. The cumulative effect of all the pirated product out there has brought down Ten,” Burke said.
While piracy has certainly been blamed for a lot of things over the years, it is extremely rare for a senior industry figure to link it so closely with the potential demise of a major broadcaster.
Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.
Post Syndicated from mikesefanov original https://yahooeng.tumblr.com/post/161855616651
By Michael Natkovich, Akshai Sarma, Nathan Speidel, Marcus Svedman, and Cat Utah
Big Data is no longer just Apache server logs. Nowadays, the data may be user engagement data, performance metrics, IoT (Internet of Things) data, or something else completely atypical. Regardless of the size of the data, or the type of querying patterns on it (exploratory, ad-hoc, periodic, long-term, etc.), everyone wants queries to be as fast as possible and cheap to run in terms of resources. Data can be broadly split into two kinds: the streaming (generally real-time) kind or the batched-up-over-a-time-interval (e.g., hourly or daily) kind. The batch version is typically easier to query since it is stored somewhere like a data warehouse that has nice SQL-like interfaces or an easy to use UI provided by tools such as Tableau, Looker, or Superset. Running arbitrary queries on streaming data quickly and cheaply though, is generally much harder… until now. Today, we are pleased to share our newly open sourced, forward-looking general purpose query engine, called Bullet, with the community on GitHub.
With Bullet, you can:
One of the key differences between how Bullet queries data and the standard querying paradigm is that Bullet does not store any data. In most other systems where you have a persistence layer (including in-memory storage), you are doing a look-back when you query the layer. Instead, Bullet operates on data flowing through the system after the query is started – it’s a look-forward system that doesn’t need persistence. On a real-time data stream, this means that Bullet is querying data after the query is submitted. This also means that Bullet does not query any data that has already passed through the stream. The fact that Bullet does not rely on a persistence layer is exactly what makes it extremely lightweight and cheap to run.
To see why this is better for the kinds of use cases Bullet is meant for – such as quickly looking at some metric, checking some assumption, iterating on a query, checking the status of something right now, etc. – consider the following: if you had a 1000 queries in a traditional query system that operated on the same data, these query systems would most likely scan the data 1000 times each. By the very virtue of it being forward looking, 1000 queries in Bullet scan the data only once because the arrival of the query determines and fixes the data that it will see. Essentially, the data is coming to the queries instead of the queries being farmed out to where the data is. When the conditions of the query are satisfied (usually a time window or a number of events), the query terminates and returns you the result.
A Brief Architecture Overview
The Bullet architecture is multi-tenant, can scale linearly for more queries and/or more data, and has been tested to handle 700+ simultaneous queries on a data stream that had up to 1.5 million records per second, or 5-6 GB/s. Bullet is currently implemented on top of Storm and can be extended to support other stream processing engines as well, like Spark Streaming or Flink. Bullet is pluggable, so you can plug in any source of data that can be read in Storm by implementing a simple data container interface to let Bullet work with it.
The UI, web service, and the backend layers constitute your standard three-tier architecture. The Bullet backend can be split into three main subsystems:
The web service can be deployed on any servlet container, like Jetty. The UI is a Node-based Ember application that runs in the client browser. Our full documentation contains all the details on exactly how we perform computationally-intractable queries like Count Distincts on fields with cardinality in the millions, etc. (DataSketches).
Usage at Yahoo
An instance of Bullet is currently running at Yahoo in production against a small subset of Yahoo’s user engagement data stream. This data is roughly 100,000 records per second and is about 130 MB/s compressed. Bullet queries this with about 100 CPU Virtual Cores and 120 GB of RAM. This fits on less than 2 of our (64 Virtual Cores, 256 GB RAM each) test Storm cluster machines.
One of the most popular use cases at Yahoo is to use Bullet to manually validate the instrumentation of an app or web application. Instrumentation produces user engagement data like clicks, views, swipes, etc. Since this data powers everything we do from analytics to personalization to targeting, it is absolutely critical that the data is correct. The usage pattern is generally to:
In addition, Bullet is also used programmatically in continuous delivery pipelines for functional testing instrumentation on product releases. Product usage is simulated, then data is generated and validated in seconds using Bullet. Bullet is orders of magnitude faster to use for this kind of validation and for general data exploration use cases, as opposed to waiting for the data to be available in Hive or other systems. The Bullet UI supports pivot tables and a multitude of charting options that may speed up analysis further compared to other querying options.
We also use Bullet to do a bunch of other interesting things, including instances where we dynamically compute cardinalities (using a Count Distinct Bullet query) of fields as a check to protect systems that can’t support extremely high cardinalities for fields like Druid.
What you do with Bullet is entirely determined by the data you put it on. If you put it on data that is essentially some set of performance metrics (data center statistics for example), you could be running a lot of queries that find the 95th and 99th percentile of a metric. If you put it on user engagement data, you could be validating instrumentation and mostly looking at raw data.
We hope you will find Bullet interesting and tell us how you use it. If you find something you want to change, improve, or fix, your contributions and ideas are always welcome! You can contact us here.
Helpful Links
Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/new-auto-scaling-for-amazon-dynamodb/
Amazon DynamoDB has more than one hundred thousand customers, spanning a wide range of industries and use cases. These customers depend on DynamoDB’s consistent performance at any scale and presence in 16 geographic regions around the world. A recent trend we’ve been observing is customers using DynamoDB to power their serverless applications. This is a good match: with DynamoDB, you don’t have to think about things like provisioning servers, performing OS and database software patching, or configuring replication across availability zones to ensure high availability – you can simply create tables and start adding data, and let DynamoDB handle the rest.
DynamoDB provides a provisioned capacity model that lets you set the amount of read and write capacity required by your applications. While this frees you from thinking about servers and enables you to change provisioning for your table with a simple API call or button click in the AWS Management Console, customers have asked us how we can make managing capacity for DynamoDB even easier.
Today we are introducing Auto Scaling for DynamoDB to help automate capacity management for your tables and global secondary indexes. You simply specify the desired target utilization and provide upper and lower bounds for read and write capacity. DynamoDB will then monitor throughput consumption using Amazon CloudWatch alarms and then will adjust provisioned capacity up or down as needed. Auto Scaling will be on by default for all new tables and indexes, and you can also configure it for existing ones.
Even if you’re not around, DynamoDB Auto Scaling will be monitoring your tables and indexes to automatically adjust throughput in response to changes in application traffic. This can make it easier to administer your DynamoDB data, help you maximize availability for your applications, and help you reduce your DynamoDB costs.
Let’s see how it works…
Using Auto Scaling
The DynamoDB Console now proposes a comfortable set of default parameters when you create a new table. You can accept them as-is or you can uncheck Use default settings and enter your own parameters:
Here’s how you enter your own parameters:
Target utilization is expressed in terms of the ratio of consumed capacity to provisioned capacity. The parameters above would allow for sufficient headroom to allow consumed capacity to double due to a burst in read or write requests (read Capacity Unit Calculations to learn more about the relationship between DynamoDB read and write operations and provisioned capacity). Changes in provisioned capacity take place in the background.
Auto Scaling in Action
In order to see this important new feature in action, I followed the directions in the Getting Started Guide. I launched a fresh EC2 instance, installed (sudo pip install boto3) and configured (aws configure) the AWS SDK for Python. Then I used the code in the Python and DynamoDB section to create and populate a table with some data, and manually configured the table for 5 units each of read and write capacity.
I took a quick break in order to have clean, straight lines for the CloudWatch metrics so that I could show the effect of Auto Scaling. Here’s what the metrics look like before I started to apply a load:
I modified the code in Step 3 to continually issue queries for random years in the range of 1920 to 2007, ran a single copy of the code, and checked the read metrics a minute or two later:
The consumed capacity is higher than the provisioned capacity, resulting in a large number of throttled reads. Time for Auto Scaling!
I returned to the console and clicked on the Capacity tab for my table. Then I clicked on Read capacity, accepted the default values, and clicked on Save:
DynamoDB created a new IAM role (DynamoDBAutoscaleRole) and a pair of CloudWatch alarms to manage the Auto Scaling of read capacity:
DynamoDB Auto Scaling will manage the thresholds for the alarms, moving them up and down as part of the scaling process. The first alarm was triggered and the table state changed to Updating while additional read capacity was provisioned:
The change was visible in the read metrics within minutes:
I started a couple of additional copies of my modified query script and watched as additional capacity was provisioned, as indicated by the red line:
I killed all of the scripts and turned my attention to other things while waiting for the scale-down alarm to trigger. Here’s what I saw when I came back:
The next morning I checked my Scaling activities and saw that the alarm had triggered several more times overnight:
This was also visible in the metrics:
Until now, you would prepare for this situation by setting your read capacity well about your expected usage, and pay for the excess capacity (the space between the blue line and the red line). Or, you might set it too low, forget to monitor it, and run out of capacity when traffic picked up. With Auto Scaling you can get the best of both worlds: an automatic response when an increase in demand suggests that more capacity is needed, and another automated response when the capacity is no longer needed.
Things to Know
DynamoDB Auto Scaling is designed to accommodate request rates that vary in a somewhat predictable, generally periodic fashion. If you need to accommodate unpredictable bursts of read activity, you should use Auto Scaling in combination with DAX (read Amazon DynamoDB Accelerator (DAX) – In-Memory Caching for Read-Intensive Workloads to learn more). Also, the AWS SDKs will detect throttled read and write requests and retry them after a suitable delay.
I mentioned the DynamoDBAutoscaleRole earlier. This role provides Auto Scaling with the privileges that it needs to have in order for it to be able to scale your tables and indexes up and down. To learn more about this role and the permissions that it uses, read Grant User Permissions for DynamoDB Auto Scaling.
Auto Scaling has complete CLI and API support, including the ability to enable and disable the Auto Scaling policies. If you have some predictable, time-bound spikes in traffic, you can programmatically disable an Auto Scaling policy, provision higher throughput for a set period of time, and then enable Auto Scaling again later.
As noted on the Limits in DynamoDB page, you can increase provisioned capacity as often as you would like and as high as you need (subject to per-account limits that we can increase on request). You can decrease capacity up to nine times per day for each table or global secondary index.
You pay for the capacity that you provision, at the regular DynamoDB prices. You can also purchase DynamoDB Reserved Capacity to further savings.
Available Now
This feature is available now in all regions and you can start using it today!
— Jeff;
Post Syndicated from Andy original https://torrentfreak.com/uk-police-claim-success-in-keeping-gambling-ads-off-pirate-sites-170614/
Over the past several years, there has been a major effort by entertainment industry groups to cut off revenue streams to ‘pirate’ sites. The theory is that if sites cannot generate funds, their operators will eventually lose interest.
Since advertising is a key money earner for any website, significant resources have been expended trying to keep ads off sites that directly or indirectly profit from infringement. It’s been a multi-pronged affair, with agencies being encouraged to do the right thing and brands warned that their ads appearing on pirate sites does nothing for their image.
One sector that has trailed behind most is the gambling industry. Up until fairly recently, ads for some of the UK’s largest bookmakers have been a regular feature on many large pirate sites, either embedded in pages or more often than not, appearing via popup or pop-under spreads. Now, however, a significant change is being reported.
According to the City of London Police’s Intellectual Property Crime Unit (PIPCU), over the past 12 months there has been an 87% drop in adverts for licensed gambling operators being displayed on infringing websites.
The research was carried out by whiteBULLET, a brand safety and advertising solutions company which helps advertisers to assess whether placing an advert on a particular URL will cause it to appear on a pirate site.
PIPCU says that licensed gambling operators have an obligation to “keep crime out of gambling” due to their commitments under the Gambling Act 2005. However, the Gambling Commission, the UK’s gambling regulatory body, has recently been taking additional steps to tackle the problem.
In September 2015, the Commission consulted on amendments (pdf) to licensing conditions that would compel licensees to ensure that advertisements “placed by themselves and others” do not appear on websites providing unauthorized access to copyrighted content.
After the consultation was published in May 2016 (pdf), all respondents agreed in principle that gambling operators should not advertise on pirate sites. A month later, the Commission said it would ban the placement of gambling ads on such platforms.
When the new rules came into play last October, 40 gambling companies (including Bet365, Coral and Sky Bet, who had previously been called out for displaying ads on pirate sites) were making use of PIPCU’s ‘Infringing Website List‘, a database of sites that police claim are actively involved in piracy.
Speaking yesterday, acting Detective Superintendent Peter Ratcliffe, Head of the Police Intellectual Property Crime Unit (PIPCU), welcomed the ensuing reduction in ad placement on ‘pirate’ domains.
“The success of a strong relationship built between PIPCU and The Gambling Commission can be seen by these figures. This is a fantastic example of a joint working initiative between police and an industry regulator,” Ratcliffe said.
“We commend the 40 gambling companies who are already using the Infringing Website List and encourage others to sign up. We will continue to encourage all UK advertisers to become a member of the Infringing Website List to ensure they’re not inadvertently funding criminal websites.”
Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.
Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/aws-govcloud-us-heads-east-new-region-in-the-works-for-2018/
AWS GovCloud (US) gives AWS customers a place to host sensitive data and regulated workloads in the AWS Cloud. The first AWS GovCloud (US) Region was launched in 2011 and is located on the west coast of the US.
I’m happy to announce that we are working on a second Region that we expect to open in 2018. The upcoming AWS GovCloud (US-East) Region will provide customers with added redundancy, data durability, and resiliency, and will also provide additional options for disaster recovery.
Like the existing region, which we now call AWS GovCloud (US-West), the new region will be isolated and meet top US government compliance requirements including International Traffic in Arms Regulations (ITAR), NIST standards, Federal Risk and Authorization Management Program (FedRAMP) Moderate and High, Department of Defense Impact Levels 2-4, DFARs, IRS1075, and Criminal Justice Information Services (CJIS) requirements. Visit the GovCloud (US) page to learn more about the compliance regimes that we support.
Government agencies and the IT contactors that serve them were early adopters of AWS GovCloud (US), as were companies in regulated industries. These organizations are able to enjoy the flexibility and cost-effectiveness of public cloud while benefiting from the isolation and data protection offered by a region designed and built to meet their regulatory needs and to help them to meet their compliance requirements. Here’s a small sample from our customer base:
Federal (US) Government – Department of Veterans Affairs, General Services Administration 18F (Digital Services Delivery), NASA JPL, Defense Digital Service, United States Air Force, United States Department of Justice.
Regulated Industries – CSRA, Talen Energy, Cobham Electronics.
SaaS and Solution Providers – FIGmd, Blackboard, Splunk, GitHub, Motorola.
Federal, state, and local agencies that want to move their existing applications to the AWS Cloud can take advantage of the AWS Cloud Adoption Framework (CAF) offered by AWS Professional Services.
— Jeff;
Post Syndicated from Andy original https://torrentfreak.com/global-entertainment-giants-form-massive-anti-piracy-coalition-170613/
It’s not unusual for companies within the same area of business to collaborate in order to combat piracy. The studios and labels that form the MPAA and RIAA, for example, have doing just that for decades.
Today, however, an unprecedented number of global content creators and distribution platforms have announced the formation of a brand new coalition to collaboratively fight Internet piracy on a global scale.
The Alliance for Creativity and Entertainment (ACE) is a coalition of 30 companies that reads like a who’s who of the global entertainment market. In alphabetical order the members are:
Amazon, AMC Networks, BBC Worldwide, Bell Canada and Bell Media, Canal+ Group, CBS Corporation, Constantin Film, Foxtel, Grupo Globo, HBO, Hulu, Lionsgate, Metro-Goldwyn-Mayer (MGM), Millennium Media, NBCUniversal, Netflix, Paramount Pictures, SF Studios, Sky, Sony Pictures Entertainment, Star India, Studio Babelsberg, STX Entertainment, Telemundo, Televisa, Twentieth Century Fox, Univision Communications Inc., Village Roadshow, The Walt Disney Company, and Warner Bros. Entertainment Inc.
In a joint announcement today, ACE notes that there are now more than 480 services available for consumers to watch films and TV programs online. However, despite that abundance of content, piracy continues to pose a threat to creators and the economy.
“Films and television shows can often be found on pirate sites within days – and in many cases hours – of release,” ACE said in a statement.
“Last year, there were an estimated 5.4 billion downloads of pirated wide release films and primetime television and VOD shows using peer-to-peer protocols worldwide. There were also an estimated 21.4 billion total visits to streaming piracy sites worldwide across both desktops and mobile devices in 2016.”
Rather than the somewhat fragmented anti-piracy approach currently employed by ACE members separately, the coalition will present a united front of all major content creators and distributors, with a mission to cooperate and expand in order to minimize the threat.
At the center of the alliance appears to be the MPAA. ACE reports that the anti-piracy resources of the Hollywood group will be used “in concert” with the existing anti-piracy departments of the member companies.
Unprecedented scale aside, ACE’s modus operandi will be a familiar one.
The coalition says it will work closely with law enforcement to shut down pirate sites and services, file civil litigation, and forge new relationships with other content protection groups. It will also strive to reach voluntary anti-piracy agreements with other interested parties across the Internet.
MPAA chief Chris Dodd, whose group will play a major role in ACE, welcomed the birth of the alliance.
“ACE, with its broad coalition of creators from around the world, is designed, specifically, to leverage the best possible resources to reduce piracy,” Dodd said.
“For decades, the MPAA has been the gold standard for antipiracy enforcement. We are proud to provide the MPAA’s worldwide antipiracy resources and the deep expertise of our antipiracy unit to support ACE and all its initiatives.”
The traditionally non-aggressive BBC described ACE as “hugely important” in the fight against “theft and illegal distribution”, with Netflix noting that even its creative strategies for dealing with piracy are in need of assistance.
“While we’re focused on providing a great consumer experience that ultimately discourages piracy, there are still bad players around the world trying to profit off the hard work of others,” said Netflix General Counsel, David Hyman.
“By joining ACE, we will work together, share knowledge, and leverage the group’s combined anti-piracy resources to address the global online piracy problem.”
It’s likely that the creation of ACE will go down as a landmark moment in the fight against piracy. Never before has such a broad coalition promised to pool resources on such a grand and global scale. That being said, with great diversity comes the potential for greatly diverging opinions, so only time will tell if this coalition can really hold together.
Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.
Post Syndicated from Anuj Sharma original https://aws.amazon.com/blogs/devops/aws-service-catalog-sync-code/
AWS Service Catalog enables organizations to create and manage catalogs of IT services that are approved for use on AWS. These IT services can include everything from virtual machine images, servers, software, and databases to complete multitier application architectures. You can use AWS Service Catalog to centrally manage commonly deployed IT services. It also helps you achieve consistent governance and meet your compliance requirements, while enabling users to quickly deploy only the approved IT services they need.
However, as the number of Service Catalog portfolios and products increases across an organization, centralized management and scaling can become a challenge. In this blog post, I walk you through a solution that simplifies management of AWS Service Catalog portfolios and related products. This solution also enables portfolio sharing with other accounts, portfolio tagging, and granting access to users. Finally, the solution delivers updates to the products using a continuous delivery in AWS CodePipeline. This enables you to maintain them in version control, thereby adopting “Infrastructure as Code” practices.
The authors also define a mapping.yaml file, which includes the list of products that you want to include in the portfolio and related metadata. The mapping.yaml file is the core configuration component of this solution. This file defines your portfolio and its associated permissions and products. This configuration file determines how your portfolio will look in AWS Service Catalog, after the solution deploys it. A sample mapping.yaml is described here. Configuration properties of this mapping.yaml are explained here.
The name of the folder must start with portfolio- because the AWS Lambda function iterates through all folders whose names start with portfolio-, and syncs them with AWS Service Catalog.
Checking in any code in the repository triggers an AWS CodePipeline orchestration and invokes the Lambda function.
Get started using this solution, which is available in this AWSLabs GitHub repository.
aws cloudformation deploy --template-file pipeline-to-service-catalog.yaml \
--stack-name service-catalog-sync-pipeline --capabilities CAPABILITY_NAMED_IAM \
--parameter-overrides RepositoryName=blogs-pipeline-to-service-catalog
This creates the following resources.
You can deploy the Lambda function as we explained in this post to sync AWS Service Catalog products, portfolios, and permissions across multiple accounts that you own with version control. You can create a secure cross-account continuous delivery pipeline, as explained here. To do this:
aws cloudformation delete-stack -- stack-name service-catalog-sync-pipelineYou can use AWS Lambda to make API calls to AWS Service Catalog to keep portfolios and products in sync with a mapping file. The code includes the CloudFormation templates and the mapping file and folder structure, which resembles the portfolios in AWS Service Catalog. When checked in to an AWS CodeCommit repository, it invokes the Lambda function, orchestrated by AWS CodePipeline.
Post Syndicated from Eevee original https://eev.ee/dev/2017/06/12/weekly-roundup-potpourri/
potluck: It’s about time I actually started on this! I spent a couple days squabbling with JavaScript frameworks, but in the end I gave up and decided to just build it on my existing LÖVE code. Trying out a new thing is nice, but maybe not when I have a somewhat complicated game in mind that I want to build as soon as possible.
fox flux: I made a font! Also drew a ton of player sprites and made my grass much prettier. I’m going to need to start thinking about environmental art soon, and I’m kinda dreading it because I have no idea what I’m doing there.
art: I did some (unfinished) modelling and painted some character art (warning: this pic is fine, but the rest of the account is NSFW).
blog: I wrote some thoughts about teaching technical subjects.
For the first time in possibly my entire life, I feel like I’m a little ahead of the game! The month isn’t even half over and I’ve already done some obligatory stuff, finished off one languishing task, and made some decent inroads into several other things. Nice.
By continuing to use the site, you agree to the use of cookies. more information
The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.