Tag Archives: Network

Banning VPNs and Proxies is Dangerous, IT Experts Warn

Post Syndicated from Andy original https://torrentfreak.com/banning-vpns-and-proxies-is-dangerous-it-experts-warn-170623/

In April, draft legislation was developed to crack down on systems and software that allow Russian Internet users to bypass website blockades approved by telecoms watchdog Roskomnadzor.

Earlier this month the draft bill was submitted to the State Duma, the lower house of the Russian parliament. If passed, the law will make it illegal for services to circumvent web blockades by “routing traffic of Russian Internet users through foreign servers, anonymous proxy servers, virtual private networks and other means.”

As the plans currently stand, anonymization services that fail to restrict access to sites listed by telecoms watchdog Rozcomnadzor face being blocked themselves. Sites offering circumvention software for download also face potential blacklisting.

This week the State Duma discussed the proposals with experts from the local Internet industry. In addition to the head of Rozcomnadzor, representatives from service providers, search engines and even anonymization services were in attendance. Novaya Gazeta has published comments (Russian) from some of the key people at the meeting and it’s fair to say there’s not a lot of support.

VimpelCom, the sixth largest mobile network operator in the world with more than 240 million subscribers, sent along Director for Relations with Government, Sergey Malyanov. He wondered where all this blocking will end up.

“First we banned certain information. Then this information was blocked with the responsibility placed on both owners of resources and services. Now there are blocks on top of blocks – so we already have a triple effort,” he said.

“It is now possible that there will be a fourth iteration: the block on the block to block those that were not blocked. And with that, we have significantly complicated the law and the activities of all the people affected by it.”

Malyanov said that these kinds of actions have the potential to close down the entire Internet by ruining what was once an open network running standard protocols. But amid all of this, will it even be effective?

“The question is not even about the losses that will be incurred by network operators, the owners of the resources and the search engines. The question is whether this bill addresses the goal its creators have set for themselves. In my opinion, it will not.”

Group-IB, one of the world’s leading cyber-security and threat intelligence providers, was represented CEO Ilya Sachkov. He told parliament that “ordinary respectable people” who use the Internet should always use a VPN for security. Nevertheless, he also believes that such services should be forced to filter sites deemed illegal by the state.

But in a warning about blocks in general, he warned that people who want to circumvent them will always be one step ahead.

“We have to understand that by the time the law is adopted the perpetrators will already find it very easy to circumvent,” he said.

Mobile operator giant MTS, which turns over billions of dollars and employs 50,000+ people, had their Vice-President of Corporate and Legal Affairs in attendance. Ruslan Ibragimov said that in dealing with a problem, the government should be cautious of not causing more problems, including disruption of a growing VPN market.

“We have an understanding that evil must be fought, but it’s not necessary to create a new evil, even more so – for those who are involved in this struggle,” he said.

“Broad wording of this law may pose a threat to our network, which could be affected by the new restrictive measures, as well as the VPN market, which we are currently developing, and whose potential market is estimated at 50 billion rubles a year.”

In its goal to maintain control of the Internet, it’s clear that Russia is determined to press ahead with legislative change. Unfortunately, it’s far from clear that there’s a technical solution to the problem, but if one is pursued regardless, there could be serious fallout.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

Kotlin and Groovy JVM Languages with AWS Lambda

Post Syndicated from Juan Villa original https://aws.amazon.com/blogs/compute/kotlin-and-groovy-jvm-languages-with-aws-lambda/


Juan Villa – Partner Solutions Architect

 

When most people hear “Java” they think of Java the programming language. Java is a lot more than a programming language, it also implies a larger ecosystem including the Java Virtual Machine (JVM). Java, the programming language, is just one of the many languages that can be compiled to run on the JVM. Some of the most popular JVM languages, other than Java, are Clojure, Groovy, Scala, Kotlin, JRuby, and Jython (see this link for a list of more JVM languages).

Did you know that you can compile and subsequently run all these languages on AWS Lambda?

AWS Lambda supports the Java 8 runtime, but this does not mean you are limited to the Java language. The Java 8 runtime is capable of running JVM languages such as Kotlin and Groovy once they have been compiled and packaged as a “fat” JAR (a JAR file containing all necessary dependencies and classes bundled in).

In this blog post we’ll work through building AWS Lambda functions in both Kotlin and Groovy programming languages. To compile and package our projects we will use Gradle build tool.

To follow along, please clone the Git repository available at GitHub here. Also, I recommend using an Integrated Development Environment (IDE) such as JetBrain’s IntelliJ IDEA, this is the IDE I used while working on these projects.

Kotlin

Kotlin is a statically-typed JVM language designed and developed by JetBrains (one of our Amazon Partner Network Technology partners) and the open source community. Compared to Java the programming language, Kotlin has additional powerful language features such as: Data Classes, Default Arguments, Extensions, Elvis Operator, and Destructuring Declarations. This is a just a short list of Kotlin’s powerful language features. For a more thorough list of features, and how to use them, refer to the full documentation of the Kotlin language.

Let’s jump right into the code and see what an AWS Lambda function looks like in Kotlin.

package com.aws.blog.jvmlangs.kotlin

import java.io.*
import com.fasterxml.jackson.module.kotlin.*

data class HandlerInput(val who: String)
data class HandlerOutput(val message: String)

class Main {
    val mapper = jacksonObjectMapper()

    fun handler(input: InputStream, output: OutputStream): Unit {
        val inputObj = mapper.readValue<HandlerInput>(input)
        mapper.writeValue(output, HandlerOutput("Hello ${inputObj.who}"))
    }
}

The above example is a very simple Hello World application that accepts as an input a JSON object containing a key called “who” and returns a JSON object containing a key called “message” with a value of “Hello {who}”.

AWS Lambda does not support serializing JSON objects into Kotlin data classes, but don’t worry! AWS Lambda supports passing an input object as a Stream, and also supports an output Stream for returning a result (see this link for more information). Combined with the Input/Output Stream form of the handler function, we are using the Jackson library with a Kotlin extension function to support serialization and deserialization of Kotlin data class types.

To get started with this example, let’s first compile and package the Kotlin project.

git clone https://github.com/awslabs/lambda-kotlin-groovy-example
cd lambda-kotlin-groovy-example/kotlin
./gradlew shadowJar

Once packaged, a JAR file containing all necessary dependencies will be available at “build/libs/ jvmlangs-kotlin-1.0-SNAPSHOT-all.jar”. Now let’s deploy this package to AWS Lambda.

To deploy the lambda function, we will be using the AWS Command Line Interface (CLI). You can find information on how to set up the AWS CLI here. This tool allows you to set up and manage AWS services via the command line.

aws lambda create-function --region us-east-1 --function-name kotlin-hello \
--zip-file fileb://build/libs/jvmlangs-kotlin-1.0-SNAPSHOT-all.jar \
--role arn:aws:iam::<account_id>:role/lambda_basic_execution \
--handler com.aws.blog.jvmlangs.kotlin.Main::handler --runtime java8 \
--timeout 15 --memory-size 128

Once deployed, we can test the function by invoking the lambda function from the CLI.

aws lambda invoke --function-name kotlin-hello --payload '{"who": "AWS Fan"}' output.txt
cat output.txt

If successful, you’ll see an output of “{"message":"Hello AWS Fan"}”.

Groovy

Groovy is an optionally typed JVM language with both dynamic and static typing capabilities. Groovy is currently being supported by the Apache Software Foundation. Like Kotlin, Groovy also packs a lot of powerful features such as: Closures, Dynamic Typing, Collection Literals, String Interpolation, and Elvis Operator. This is just a short list, see the full documentation for a list of features and how to use them.

Once again, let’s jump right into the code.

package com.aws.blog.jvmlangs.groovy

class HandlerInput {
    String who
}
class HandlerOutput {
    String message
}

class Main {
    def handler(HandlerInput input) {
        return new HandlerOutput(message: "Hello ${input.who}")
    }
}

Just like the Kotlin example, we have defined a function that takes a simple JSON object containing a “who” key value and build a response containing a “message” key. Note that in this case we are not using the Input/Output Stream form of the handler function, but rather we are letting AWS Lambda serialize the input JSON object into the type HandlerInput. To accomplish this, AWS Lambda uses the Jackson library and handles the serialization for us.

Let’s go ahead and compile and package this Groovy example.

git clone https://github.com/awslabs/lambda-kotlin-groovy-example
cd lambda-kotlin-groovy-example/groovy
./gradlew shadowJar

Once packaged, a JAR file containing all necessary dependencies will be available at “build/libs/ jvmlangs-groovy-1.0-SNAPSHOT-all.jar”. Now let’s deploy this package to AWS Lambda.

aws lambda create-function --region us-east-1 --function-name groovy-hello \
--zip-file fileb://build/libs/jvmlangs-groovy-1.0-SNAPSHOT-all.jar \
--role arn:aws:iam::<account_id>:role/lambda_basic_execution \
--handler com.aws.blog.jvmlangs.groovy.Main::handler --runtime java8 \
--timeout 15 --memory-size 128

Once deployed, we can test the function by invoking the lambda function from the CLI.

aws lambda invoke --function-name groovy-hello --payload '{"who": "AWS Fan"}' output.txt
cat output.txt

If successful, you’ll see an output of “{"message":"Hello AWS Fan"}”.

Gradle Build Tool

Finally, let’s touch up on how we built the JAR package from the Kotlin and Groovy sources above. To build the JARs we used the Gradle build tool. Gradle builds a project by reading instructions from a file called “build.gradle”. This is a file written in Gradle’s Groovy Domain Specific Langauge (DSL). You can find more information on the gradle build file by looking at their documentation. Let’s take a look at the Gradle build files we used for this post.

For the Kotlin example, this is the build file we used.

buildscript {
    repositories {
        mavenCentral()
        jcenter()
    }
    dependencies {
        classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version"
        classpath "com.github.jengelman.gradle.plugins:shadow:1.2.3"
    }
}

group 'com.aws.blog.jvmlangs.kotlin'
version '1.0-SNAPSHOT'

apply plugin: 'kotlin'
apply plugin: 'com.github.johnrengelman.shadow'

repositories {
    mavenCentral()
}

dependencies {
    compile "org.jetbrains.kotlin:kotlin-stdlib:$kotlin_version"
    compile "com.fasterxml.jackson.module:jackson-module-kotlin:2.8.2"
}

For the Groovy example this is the build file we used.

buildscript {
    repositories {
        jcenter()
    }
    dependencies {
        classpath 'com.github.jengelman.gradle.plugins:shadow:1.2.3'
    }
}

group 'com.aws.blog.jvmlangs.groovy'
version '1.0-SNAPSHOT'

apply plugin: 'groovy'
apply plugin: 'com.github.johnrengelman.shadow'

repositories {
    mavenCentral()
}

dependencies {
    compile 'org.codehaus.groovy:groovy-all:2.3.11'
    testCompile group: 'junit', name: 'junit', version: '4.11'
}

As you can see, the build files for both Kotlin and Groovy files are very similar. For the Kotlin project we define a dependency on the Jackson Kotlin module. Also, for each respective language we include the language supporting libraries (kotlin-stdlib and groovy-all respectively).

In addition, you will notice that we are using a plugin called “shadow”. We use this plugin to package all the project dependencies into one JAR by using the Gradle task “shadowJar”. You can find more information on Shadow in their documentation.

Final Words

Don’t stop here though! Take a look at other JVM languages and get them running on AWS Lambda with the Java 8 runtime. Maybe start with Clojure? or Scala?

Also take a look AWS Lambda Java libraries provided by AWS. They provide interfaces and models to make handling events from event sources easier to handle.

Court Grants Subpoenas to Unmask ‘TVAddons’ and ‘ZemTV’ Operators

Post Syndicated from Ernesto original https://torrentfreak.com/court-grants-subpoenas-to-unmask-tvaddons-and-zemtv-operators-170621/

Earlier this month we broke the news that third-party Kodi add-on ZemTV and the TVAddons library were being sued in a federal court in Texas.

In a complaint filed by American satellite and broadcast provider Dish Network, both stand accused of copyright infringement, facing up to $150,000 for each offense.

While the allegations are serious, Dish doesn’t know the full identities of the defendants.

To find out more, the company requested a broad range of subpoenas from the court, targeting Amazon, Github, Google, Twitter, Facebook, PayPal, and several hosting providers.

From Dish’s request

This week the court granted the subpoenas, which means that they can be forwarded to the companies in question. Whether that will be enough to identify the people behind ‘TVAddons’ and ‘ZemTV’ remains to be seen, but Dish has cast its net wide.

For example, the subpoena directed at Google covers any type of information that can be used to identify the account holder of [email protected], which is believed to be tied to ZemTV.

The information requested from Google includes IP address logs with session date and timestamps, but also covers “all communications,” including GChat messages from 2014 onwards.

Similarly, Twitter is required to hand over information tied to the accounts of the users “TV Addons” and “shani_08_kodi” as well as other accounts linked to tvaddons.ag and streamingboxes.com. This also applies the various tweets that were sent through the account.

The subpoena specifically mentions “all communications, including ‘tweets’, Twitter sent to or received from each Twitter Account during the time period of February 1, 2014 to present.”

From the Twitter subpoena

Similar subpoenas were granted for the other services, tailored towards the information Dish hopes to find there. For example, the broadcast provider also requests details of each transaction from PayPal, as well as all debits and credits to the accounts.

In some parts, the subpoenas appear to be quite broad. PayPal is asked to reveal information on any account with the credit card statement “Shani,” for example. Similarly, Github is required to hand over information on accounts that are ‘associated’ with the tvaddons.ag domain, which is referenced by many people who are not directly connected to the site.

The service providers in question still have the option to challenge the subpoenas or ask the court for further clarification. A full overview of all the subpoena requests is available here (Exhibit 2 and onwards), including all the relevant details. This also includes several letters to foreign hosting providers.

While Dish still appears to be keen to find out who is behind ‘TVAddons’ and ‘ZemTV,’ not much has been heard from the defendants in question.

ZemTV developer “Shani” shut down his addon soon after the lawsuit was announced, without mentioning it specifically. TVAddons, meanwhile, has been offline for well over a week, without any notice in public about the reason for the prolonged downtime.

The court’s order granting the subpoenas and letters of request is available here (pdf).

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

Building Loosely Coupled, Scalable, C# Applications with Amazon SQS and Amazon SNS

Post Syndicated from Tara Van Unen original https://aws.amazon.com/blogs/compute/building-loosely-coupled-scalable-c-applications-with-amazon-sqs-and-amazon-sns/

 
Stephen Liedig, Solutions Architect

 

One of the many challenges professional software architects and developers face is how to make cloud-native applications scalable, fault-tolerant, and highly available.

Fundamental to your project success is understanding the importance of making systems highly cohesive and loosely coupled. That means considering the multi-dimensional facets of system coupling to support the distributed nature of the applications that you are building for the cloud.

By that, I mean addressing not only the application-level coupling (managing incoming and outgoing dependencies), but also considering the impacts of of platform, spatial, and temporal coupling of your systems. Platform coupling relates to the interoperability, or lack thereof, of heterogeneous systems components. Spatial coupling deals with managing components at a network topology level or protocol level. Temporal, or runtime coupling, refers to the ability of a component within your system to do any kind of meaningful work while it is performing a synchronous, blocking operation.

The AWS messaging services, Amazon SQS and Amazon SNS, help you deal with these forms of coupling by providing mechanisms for:

  • Reliable, durable, and fault-tolerant delivery of messages between application components
  • Logical decomposition of systems and increased autonomy of components
  • Creating unidirectional, non-blocking operations, temporarily decoupling system components at runtime
  • Decreasing the dependencies that components have on each other through standard communication and network channels

Following on the recent topic, Building Scalable Applications and Microservices: Adding Messaging to Your Toolbox, in this post, I look at some of the ways you can introduce SQS and SNS into your architectures to decouple your components, and show how you can implement them using C#.

Walkthrough

To illustrate some of these concepts, consider a web application that processes customer orders. As good architects and developers, you have followed best practices and made your application scalable and highly available. Your solution included implementing load balancing, dynamic scaling across multiple Availability Zones, and persisting orders in a Multi-AZ Amazon RDS database instance, as in the following diagram.


In this example, the application is responsible for handling and persisting the order data, as well as dealing with increases in traffic for popular items.

One potential point of vulnerability in the order processing workflow is in saving the order in the database. The business expects that every order has been persisted into the database. However, any potential deadlock, race condition, or network issue could cause the persistence of the order to fail. Then, the order is lost with no recourse to restore the order.

With good logging capability, you may be able to identify when an error occurred and which customer’s order failed. This wouldn’t allow you to “restore” the transaction, and by that stage, your customer is no longer your customer.

As illustrated in the following diagram, introducing an SQS queue helps improve your ordering application. Using the queue isolates the processing logic into its own component and runs it in a separate process from the web application. This, in turn, allows the system to be more resilient to spikes in traffic, while allowing work to be performed only as fast as necessary in order to manage costs.


In addition, you now have a mechanism for persisting orders as messages (with the queue acting as a temporary database), and have moved the scope of your transaction with your database further down the stack. In the event of an application exception or transaction failure, this ensures that the order processing can be retired or redirected to the Amazon SQS Dead Letter Queue (DLQ), for re-processing at a later stage. (See the recent post, Using Amazon SQS Dead-Letter Queues to Control Message Failure, for more information on dead-letter queues.)

Scaling the order processing nodes

This change allows you now to scale the web application frontend independently from the processing nodes. The frontend application can continue to scale based on metrics such as CPU usage, or the number of requests hitting the load balancer. Processing nodes can scale based on the number of orders in the queue. Here is an example of scale-in and scale-out alarms that you would associate with the scaling policy.

Scale-out Alarm

aws cloudwatch put-metric-alarm --alarm-name AddCapacityToCustomerOrderQueue --metric-name ApproximateNumberOfMessagesVisible --namespace "AWS/SQS" 
--statistic Average --period 300 --threshold 3 --comparison-operator GreaterThanOrEqualToThreshold --dimensions Name=QueueName,Value=customer-orders
--evaluation-periods 2 --alarm-actions <arn of the scale-out autoscaling policy>

Scale-in Alarm

aws cloudwatch put-metric-alarm --alarm-name RemoveCapacityFromCustomerOrderQueue --metric-name ApproximateNumberOfMessagesVisible --namespace "AWS/SQS" 
 --statistic Average --period 300 --threshold 1 --comparison-operator LessThanOrEqualToThreshold --dimensions Name=QueueName,Value=customer-orders
 --evaluation-periods 2 --alarm-actions <arn of the scale-in autoscaling policy>

In the above example, use the ApproximateNumberOfMessagesVisible metric to discover the queue length and drive the scaling policy of the Auto Scaling group. Another useful metric is ApproximateAgeOfOldestMessage, when applications have time-sensitive messages and developers need to ensure that messages are processed within a specific time period.

Scaling the order processing implementation

On top of scaling at an infrastructure level using Auto Scaling, make sure to take advantage of the processing power of your Amazon EC2 instances by using as many of the available threads as possible. There are several ways to implement this. In this post, we build a Windows service that uses the BackgroundWorker class to process the messages from the queue.

Here’s a closer look at the implementation. In the first section of the consuming application, use a loop to continually poll the queue for new messages, and construct a ReceiveMessageRequest variable.

public static void PollQueue()
{
    while (_running)
    {
        Task<ReceiveMessageResponse> receiveMessageResponse;

        // Pull messages off the queue
        using (var sqs = new AmazonSQSClient())
        {
            const int maxMessages = 10;  // 1-10

            //Receiving a message
            var receiveMessageRequest = new ReceiveMessageRequest
            {
                // Get URL from Configuration
                QueueUrl = _queueUrl, 
                // The maximum number of messages to return. 
                // Fewer messages might be returned. 
                MaxNumberOfMessages = maxMessages, 
                // A list of attributes that need to be returned with message.
                AttributeNames = new List<string> { "All" },
                // Enable long polling. 
                // Time to wait for message to arrive on queue.
                WaitTimeSeconds = 5 
            };

            receiveMessageResponse = sqs.ReceiveMessageAsync(receiveMessageRequest);
        }

The WaitTimeSeconds property of the ReceiveMessageRequest specifies the duration (in seconds) that the call waits for a message to arrive in the queue before returning a response to the calling application. There are a few benefits to using long polling:

  • It reduces the number of empty responses by allowing SQS to wait until a message is available in the queue before sending a response.
  • It eliminates false empty responses by querying all (rather than a limited number) of the servers.
  • It returns messages as soon any message becomes available.

For more information, see Amazon SQS Long Polling.

After you have returned messages from the queue, you can start to process them by looping through each message in the response and invoking a new BackgroundWorker thread.

// Process messages
if (receiveMessageResponse.Result.Messages != null)
{
    foreach (var message in receiveMessageResponse.Result.Messages)
    {
        Console.WriteLine("Received SQS message, starting worker thread");

        // Create background worker to process message
        BackgroundWorker worker = new BackgroundWorker();
        worker.DoWork += (obj, e) => ProcessMessage(message);
        worker.RunWorkerAsync();
    }
}
else
{
    Console.WriteLine("No messages on queue");
}

The event handler, ProcessMessage, is where you implement business logic for processing orders. It is important to have a good understanding of how long a typical transaction takes so you can set a message VisibilityTimeout that is long enough to complete your operation. If order processing takes longer than the specified timeout period, the message becomes visible on the queue. Other nodes may pick it and process the same order twice, leading to unintended consequences.

Handling Duplicate Messages

In order to manage duplicate messages, seek to make your processing application idempotent. In mathematics, idempotent describes a function that produces the same result if it is applied to itself:

f(x) = f(f(x))

No matter how many times you process the same message, the end result is the same (definition from Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions, Hohpe and Wolf, 2004).

There are several strategies you could apply to achieve this:

  • Create messages that have inherent idempotent characteristics. That is, they are non-transactional in nature and are unique at a specified point in time. Rather than saying “place new order for Customer A,” which adds a duplicate order to the customer, use “place order <orderid> on <timestamp> for Customer A,” which creates a single order no matter how often it is persisted.
  • Deliver your messages via an Amazon SQS FIFO queue, which provides the benefits of message sequencing, but also mechanisms for content-based deduplication. You can deduplicate using the MessageDeduplicationId property on the SendMessage request or by enabling content-based deduplication on the queue, which generates a hash for MessageDeduplicationId, based on the content of the message, not the attributes.
var sendMessageRequest = new SendMessageRequest
{
    QueueUrl = _queueUrl,
    MessageBody = JsonConvert.SerializeObject(order),
    MessageGroupId = Guid.NewGuid().ToString("N"),
    MessageDeduplicationId = Guid.NewGuid().ToString("N")
};
  • If using SQS FIFO queues is not an option, keep a message log of all messages attributes processed for a specified period of time, as an alternative to message deduplication on the receiving end. Verifying the existence of the message in the log before processing the message adds additional computational overhead to your processing. This can be minimized through low latency persistence solutions such as Amazon DynamoDB. Bear in mind that this solution is dependent on the successful, distributed transaction of the message and the message log.

Handling exceptions

Because of the distributed nature of SQS queues, it does not automatically delete the message. Therefore, you must explicitly delete the message from the queue after processing it, using the message ReceiptHandle property (see the following code example).

However, if at any stage you have an exception, avoid handling it as you normally would. The intention is to make sure that the message ends back on the queue, so that you can gracefully deal with intermittent failures. Instead, log the exception to capture diagnostic information, and swallow it.

By not explicitly deleting the message from the queue, you can take advantage of the VisibilityTimeout behavior described earlier. Gracefully handle the message processing failure and make the unprocessed message available to other nodes to process.

In the event that subsequent retries fail, SQS automatically moves the message to the configured DLQ after the configured number of receives has been reached. You can further investigate why the order process failed. Most importantly, the order has not been lost, and your customer is still your customer.

private static void ProcessMessage(Message message)
{
    using (var sqs = new AmazonSQSClient())
    {
        try
        {
            Console.WriteLine("Processing message id: {0}", message.MessageId);

            // Implement messaging processing here
            // Ensure no downstream resource contention (parallel processing)
            // <your order processing logic in here…>
            Console.WriteLine("{0} Thread {1}: {2}", DateTime.Now.ToString("s"), Thread.CurrentThread.ManagedThreadId, message.MessageId);
            
            // Delete the message off the queue. 
            // Receipt handle is the identifier you must provide 
            // when deleting the message.
            var deleteRequest = new DeleteMessageRequest(_queueName, message.ReceiptHandle);
            sqs.DeleteMessageAsync(deleteRequest);
            Console.WriteLine("Processed message id: {0}", message.MessageId);

        }
        catch (Exception ex)
        {
            // Do nothing.
            // Swallow exception, message will return to the queue when 
            // visibility timeout has been exceeded.
            Console.WriteLine("Could not process message due to error. Exception: {0}", ex.Message);
        }
    }
}

Using SQS to adapt to changing business requirements

One of the benefits of introducing a message queue is that you can accommodate new business requirements without dramatically affecting your application.

If, for example, the business decided that all orders placed over $5000 are to be handled as a priority, you could introduce a new “priority order” queue. The way the orders are processed does not change. The only significant change to the processing application is to ensure that messages from the “priority order” queue are processed before the “standard order” queue.

The following diagram shows how this logic could be isolated in an “order dispatcher,” whose only purpose is to route order messages to the appropriate queue based on whether the order exceeds $5000. Nothing on the web application or the processing nodes changes other than the target queue to which the order is sent. The rates at which orders are processed can be achieved by modifying the poll rates and scalability settings that I have already discussed.

Extending the design pattern with Amazon SNS

Amazon SNS supports reliable publish-subscribe (pub-sub) scenarios and push notifications to known endpoints across a wide variety of protocols. It eliminates the need to periodically check or poll for new information and updates. SNS supports:

  • Reliable storage of messages for immediate or delayed processing
  • Publish / subscribe – direct, broadcast, targeted “push” messaging
  • Multiple subscriber protocols
  • Amazon SQS, HTTP, HTTPS, email, SMS, mobile push, AWS Lambda

With these capabilities, you can provide parallel asynchronous processing of orders in the system and extend it to support any number of different business use cases without affecting the production environment. This is commonly referred to as a “fanout” scenario.

Rather than your web application pushing orders to a queue for processing, send a notification via SNS. The SNS messages are sent to a topic and then replicated and pushed to multiple SQS queues and Lambda functions for processing.

As the diagram above shows, you have the development team consuming “live” data as they work on the next version of the processing application, or potentially using the messages to troubleshoot issues in production.

Marketing is consuming all order information, via a Lambda function that has subscribed to the SNS topic, inserting the records into an Amazon Redshift warehouse for analysis.

All of this, of course, is happening without affecting your order processing application.

Summary

While I haven’t dived deep into the specifics of each service, I have discussed how these services can be applied at an architectural level to build loosely coupled systems that facilitate multiple business use cases. I’ve also shown you how to use infrastructure and application-level scaling techniques, so you can get the most out of your EC2 instances.

One of the many benefits of using these managed services is how quickly and easily you can implement powerful messaging capabilities in your systems, and lower the capital and operational costs of managing your own messaging middleware.

Using Amazon SQS and Amazon SNS together can provide you with a powerful mechanism for decoupling application components. This should be part of design considerations as you architect for the cloud.

For more information, see the Amazon SQS Developer Guide and Amazon SNS Developer Guide. You’ll find tutorials on all the concepts covered in this post, and more. To can get started using the AWS console or SDK of your choice visit:

Happy messaging!

Sync vs. Backup vs. Storage

Post Syndicated from Yev original https://www.backblaze.com/blog/sync-vs-backup-vs-storage/

Cloud Sync vs. Cloud Backup vs. Cloud Storage

Google Drive recently announced their new Backup and Sync feature for Google Drive, which allows users to select folders on their computer that they want to back up to their Google Drive account (note: these files count against your Google Drive storage limit). Whenever new backup services are announced, we get a lot of questions so I thought we should take a minute to review the differences in cloud based services.

What is the Cloud? Sync Vs Backup Vs Storage

There is still a lot of confusion in the space about what exactly the “cloud” is and how different services interact with it. When folks use a syncing and sharing service like Dropbox, Box, Google Drive, OneDrive or any of the others, they often assume those are acting as a cloud backup solution as well. Adding to the confusion, cloud storage services are often the backend for backup and sync services as well as standalone services. To help sort this out, we’ll define some of the terms below as they apply to a traditional computer set-up with a bunch of apps and data.

Cloud Sync (ex. Dropbox, iCloud Drive, OneDrive, Box, Google Drive) – these services sync folders on your computer to folders on other machines or to the cloud – allowing users to work from a folder or directory across devices. Typically these services have tiered pricing, meaning you pay for the amount of data you store with the service. If there is data loss, sometimes these services even have a rollback feature, of course only files that are in the synced folders are available to be recovered.

Cloud Backup (ex. Backblaze Cloud Backup, Mozy, Carbonite) – these services work in the background automatically. The user does not need to take any action like setting up specific folders. Backup services typically back up any new or changed data on your computer to another location. Before the cloud took off, that location was primarily a CD or an external hard drive – but as cloud storage became more readily available it became the most popular storage medium. Typically these services have fixed pricing, and if there is a system crash or data loss, all backed up data is available for restore. In addition, these services have rollback features in case there is data loss / accidental file deletion.

Cloud Storage (ex. Backblaze B2, Amazon S3, Microsoft Azure) – these services are where many online backup and syncing and sharing services store data. Cloud storage providers typically serve as the endpoint for data storage. These services typically provide APIs, CLIs, and access points for individuals and developers to tie in their cloud storage offerings directly. These services are priced “per GB” meaning you pay for the amount of storage that you use. Since these services are designed for high-availability and durability, data can live solely on these services – though we still recommend having multiple copies of your data, just in case.

What Should You Use?

Backblaze strongly believes in a 3-2-1 Backup Strategy. A 3-2-1 strategy means having at least 3 total copies of your data, 2 of which are local but on different mediums (e.g. an external hard drive in addition to your computer’s local drive), and at least 1 copy offsite. The best setup is data on your computer, a copy on a hard drive that lives somewhere not inside your computer, and another copy with a cloud backup provider. Backblaze Cloud Backup is a great compliment to other services, like Time Machine, Dropbox, and even the free-tiers of cloud storage services.

What is The Difference Between Cloud Sync and Backup?

Let’s take a look at some sync setups that we see fairly frequently.

Example 1) Users have one folder on their computer that is designated for Dropbox, Google Drive, OneDrive, or one of the other syncing/sharing services. Users save or place data into those directories when they want them to appear on other devices. Often these users are using the free-tier of those syncing and sharing services and only have a few GB of data uploaded in them.

Example 2) Users are paying for extended storage for Dropbox, Google Drive, OneDrive, etc… and use those folders as the “Documents” folder – essentially working out of those directories. Files in that folder are available across devices, however, files outside of that folder (e.g. living on the computer’s desktop or anywhere else) are not synced or stored by the service.

What both examples are missing however is the backup of photos, movies, videos, and the rest of the data on their computer. That’s where cloud backup providers excel, by automatically backing up user data with little or no set-up, and no need for the dragging-and-dropping of files. Backblaze actually scans your hard drive to find all the data, regardless of where it might be hiding. The results are, all the user’s data is kept in the Backblaze cloud and the portion of the data that is synced is also kept in that provider’s cloud – giving the user another layer of redundancy. Best of all, Backblaze will actually back up your Dropbox, iCloud Drive, Google Drive, and OneDrive folders.

Data Recovery

The most important feature to think about is how easy it is to get your data back from all of these services. With sync and share services, retrieving a lot of data, especially if you are in a high-data tier, can be cumbersome and take awhile. Generally, the sync and share services only allow customers to download files over the Internet. If you are trying to download more than a couple gigabytes of data, the process can take time and can be fraught with errors.

With cloud storage services, you can usually only retrieve data over the Internet as well, and you pay for both the storage and the egress of the data, so retrieving a large amount of data can be both expensive and time consuming.

Cloud backup services will enable you to download files over the internet too and can also suffer from long download times. At Backblaze we never want our customers to feel like we’re holding their data hostage, which is why we have a lot of restore options, including our Restore Return Refund policy, which allows people to restore their data via a USB Hard Drive, and then return that drive to us for a refund. Cloud sync providers do not provide this capability.

One popular data recovery use case we’ve seen when a person has a lot of data to restore is to download just the files that are needed immediately, and then order a USB Hard Drive restore for the remaining files that are not as time sensitive. The user gets all their files back in a few days, and their network is spared the download charges.

The bottom line is that all of these services have merit for different use-cases. Have questions about which is best for you? Sound off in the comments below!

The post Sync vs. Backup vs. Storage appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

AWS Marketplace Update – SaaS Contracts in Action

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/aws-marketplace-update-saas-contracts-in-action/

AWS Marketplace lets AWS customers find and use products and services offered by members of the AWS Partner Network (APN). Some marketplace offerings are billed on an hourly basis, many with a cost-saving annual option designed to line up with the procurement cycles of our enterprise customers. Other offerings are available in SaaS (Software as a Service) form and are billed based on consumption units specified by the seller. The SaaS model (described in New – SaaS subscriptions on AWS Marketplace) give sellers the flexibility to bill for actual usage: number of active hosts, number of requests, GB of log files processed, and so forth.

Recently we extended the SaaS model with the addition of SaaS contracts, which my colleague Brad Lyman introduced in his post, Announcing SaaS Contracts, a Feature to Simplify SaaS Procurement on AWS Marketplace. The contracts give our customers the opportunity save money by setting up monthly subscriptions that can be expanded to cover a one, two, or three year contract term, with automatic, configurable renewals. Sellers can provide services that require up-front payment or that offer discounts in exchange for a usage commitment.

Since Brad has already covered the seller side of this powerful and flexible new model, I would like to show you what it is like to purchase a SaaS contract. Let’s say that I want to use Splunk Cloud. I simply search for it as usual:

I click on Splunk Cloud and see that it is available in SaaS Contract form:

I can also see and review the pricing options, noting that pricing varies by location, index volume, and subscription duration:

I click on Continue. Since I do not have a contract with Splunk for this software, I’ll be redirected to the vendor’s site to create one as part of the process. I choose my location, index volume, and contract duration, and opt for automatic renewal, and then click on Create Contract:

This sets up my subscription, and I need only set up my account with Splunk:

I click on Set Up Your Account and I am ready to move forward by setting up my custom URL on the Splunk site:

This feature is available now and you can start using it today.

Jeff;

 

Internet Provider Refutes RIAA’s Piracy Allegations

Post Syndicated from Ernesto original https://torrentfreak.com/internet-provider-refutes-riaas-piracy-allegations-170620/

For more than a decade copyright holders have been sending ISPs takedown notices to alert them that their subscribers are sharing copyrighted material.

Under US law, providers have to terminate the accounts of repeat infringers “in appropriate circumstances” and increasingly they are being held to this standard.

Earlier this year several major record labels, represented by the RIAA, filed a lawsuit in a Texas District Court, accusing ISP Grande Communications of failing to take action against its pirating subscribers.

“Despite their knowledge of repeat infringements, Defendants have permitted repeat infringers to use the Grande service to continue to infringe Plaintiffs’ copyrights without consequence,” the RIAA’s complaint read.

Grande and its management consulting firm Patriot, which was also sued, both disagree and have filed a motion to dismiss at the court this week. Grande argues that it doesn’t encourage any of its customers to download copyrighted works, and that it has no control over the content subscribers access.

The Internet provider doesn’t deny that it has received millions of takedown notices through the piracy tracking company Rightscorp. However, it believes that these notices are flawed as Rightscorp is incapable of monitoring actual copyright infringements.

“These notices are so numerous and so lacking in specificity, that it is infeasible for Grande to devote the time and resources required to meaningfully investigate them. Moreover, the system that Rightscorp employs to generate its notices is incapable of detecting actual infringement and, therefore, is incapable of generating notices that reflect real infringement,” Grande writes.

Grande says that if they acted on these notices without additional proof, its subscribers could lose their Internet access even though they are using it for legal purposes.

“To merely treat these allegations as true without investigation would be a disservice to Grande’s subscribers, who would run the risk of having their Internet service permanently terminated despite using Grande’s services for completely legitimate purposes.”

Even if the notices were able to prove actual infringement, they would still fail to identify the infringer, according to the ISP. The notices identify IP-addresses which may have been used by complete strangers, who connected to the network without permission.

The Internet provider admits that online copyright infringement is a real problem. But, they see themselves as a victim of this problem, not a perpetrator, as the record labels suggest.

“Grande does not profit or receive any benefit from subscribers that may engage in such infringing activity using its network. To the contrary, Grande suffers demonstrable losses as a direct result of purported copyright infringement conducted on its network.

“To hold Grande liable for copyright infringement simply because ‘something must be done’ to address this growing problem is to hold the wrong party accountable,” Grande adds.

In common with the previous case against Cox Communications, Rightscorp’s copyright infringement notices are once again at the center of a prominent lawsuit. According to Grande, Rightscorp’s system can’t prove that infringing content was actually downloaded by third parties, only that it was made available.

The Internet provider sees the lacking infringement notices as a linchpin that, if pulled, will take the entire case down.

It’s expected that, if the case moves forward, both parties will do all they can to show that the evidence is sufficient, or not. In the Cox lawsuit, this was the case, but that verdict is currently being appealed.

Grande Communication’s full motion to dismiss is avalaible here (pdf).

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

All Systems Go! 2017 CfP Open

Post Syndicated from Lennart Poettering original http://0pointer.net/blog/all-systems-go-2017-cfp-open.html

The All Systems Go! 2017 Call for Participation is Now Open!

We’d like to invite presentation proposals for All Systems Go! 2017!

All Systems Go! is an Open Source community conference focused on the projects and technologies at the foundation of modern Linux systems — specifically low-level user-space technologies. Its goal is to provide a friendly and collaborative gathering place for individuals and communities working to push these technologies forward.

All Systems Go! 2017 takes place in Berlin, Germany on October 21st+22nd.

All Systems Go! is a 2-day event with 2-3 talks happening in parallel. Full presentation slots are 30-45 minutes in length and lightning talk slots are 5-10 minutes.

We are now accepting submissions for presentation proposals. In particular, we are looking for sessions including, but not limited to, the following topics:

  • Low-level container executors and infrastructure
  • IoT and embedded OS infrastructure
  • OS, container, IoT image delivery and updating
  • Building Linux devices and applications
  • Low-level desktop technologies
  • Networking
  • System and service management
  • Tracing and performance measuring
  • IPC and RPC systems
  • Security and Sandboxing

While our focus is definitely more on the user-space side of things, talks about kernel projects are welcome too, as long as they have a clear and direct relevance for user-space.

Please submit your proposals by September 3rd. Notification of acceptance will be sent out 1-2 weeks later.

To submit your proposal now please visit our CFP submission web site.

For further information about All Systems Go! visit our conference web site.

systemd.conf will not take place this year in lieu of All Systems Go!. All Systems Go! welcomes all projects that contribute to Linux user space, which, of course, includes systemd. Thus, anything you think was appropriate for submission to systemd.conf is also fitting for All Systems Go!

Debian Edu / Skolelinux Stretch released

Post Syndicated from ris original https://lwn.net/Articles/725828/rss

Debian Edu, also known as Skolelinux, is a Debian derivative aimed at
making it easy to administrate a computer lab or a whole school network.
Version 9 “Stretch” has been released. “Would you like to install
servers, workstations and laptops which will then work together? Do you want the
stability of Debian with network services already preconfigured? Do you
wish to have a web-based tool to manage systems and several hundred or even
more user accounts? Have you asked yourself if and how older computers
could be used? Then Debian Edu is for you.

AIMS Desktop 2017.1 released

Post Syndicated from corbet original https://lwn.net/Articles/725712/rss

The AIMS desktop is a
Debian-derived distribution aimed at mathematical and scientific use. This
project’s first public release, based on Debian 9, is now available.
It is a GNOME-based distribution with a bunch of add-on software.
It is maintained by AIMS (The African Institute for Mathematical
Sciences), a pan-African network of centres of excellence enabling Africa’s
talented students to become innovators driving the continent’s scientific,
educational and economic self-sufficiency.

Pirates Cost Australia’s Ten Network “Hundreds of Millions of Dollars”

Post Syndicated from Andy original https://torrentfreak.com/pirates-cost-australias-ten-network-hundreds-of-millions-of-dollars-170616/

In 2016, Australia’s Ten Network posted losses of AUS$157 million. This April, the broadcaster showed signs of continuing distress when it posted a half-year loss of AUS$232 million.

In a statement to the stock exchange, Ten said it was trying to secure new terms for a AUS$200 million debt financing guarantee. According to ABC, the company had lost more than 60% of its value in the preceding 12 months and almost 98% over the previous five years.

More bad news arrived this week when Ten’s board decided to put the company into voluntary administration after failing to secure a guarantee for a AUS$250 million loan that could’ve kept the ship afloat into the new year. As moves get underway to secure the company’s future, fingers of blame are being raised.

According to Village Roadshow co-chief executive Graham Burke, Internet pirates cost Ten “hundreds of millions of dollars” in advertising revenue due to their tendency to obtain movies and TV shows from the web rather than via legitimate means.

Burke told The Australian (paywall) that movies supplied to Ten by 21st Century Fox (including The Revenant and The Peanuts Movie which were both leaked) had received lower broadcast ratings due to people viewing them online in advance.

“Piracy is a much bigger channel and an illicit economy than the three main commercial networks combined,” Burke told the publication.

“Movies from Fox arrive with several million people having seen them through piracy. If it wasn’t for piracy, the ratings would be stronger and the product would not be arriving clapped out.”

But leaked or not, content doesn’t come cheap. As part of efforts to remain afloat, Ten Network recently tried to re-negotiate content supply deals with Fox and CBS. Together they reportedly cost the broadcaster more than AUS$900 million over the previous six years.

Despite this massive price tag and numerous other problems engulfing the troubled company, Burke suggests it is pirates that are to blame for Ten’s demise.

“A large part of Ten’s expenditure is on movies and they are being seen by millions of people ­illegitimately on websites supported by rogue ­advertising for drugs, prostitution and even legitimate advertising. The cumulative effect of all the ­pirated product out there has brought down Ten,” Burke said.

While piracy has certainly been blamed for a lot of things over the years, it is extremely rare for a senior industry figure to link it so closely with the potential demise of a major broadcaster.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

“Top ISPs” Are Discussing Fines & Browsing Hijacking For Pirates

Post Syndicated from Andy original https://torrentfreak.com/top-isps-are-discussing-fines-browsing-hijacking-for-pirates-170614/

For the past several years, anti-piracy outfit Rightscorp has been moderately successful in forcing smaller fringe ISPs in the United States to collaborate in a low-tier copyright trolling operation.

The way it works is relatively simple. Rightscorp monitors BitTorrent networks, captures the IP addresses of alleged infringers, and sends DMCA notices to their ISPs. Rightscorp expects ISPs to forward these to their customers along with an attached cash settlement demand.

These demands are usually for small amounts ($20 or $30) but most of the larger ISPs don’t forward them to their customers. This deprives Rightscorp (and clients such as BMG) of the opportunity to generate revenue, a situation that the anti-piracy outfit is desperate to remedy.

One of the problems is that when people who receive Rightscorp ‘fines’ refuse to pay them, the company does nothing, leading to a lack of respect for the company. With this in mind, Rightscorp has been trying to get ISPs involved in forcing people to pay up.

In 2014, Rightscorp said that its goal was to have ISPs place a redirect page in front of ‘pirate’ subscribers until they pay a cash fine.

“[What] we really want to do is move away from termination and move to what’s called a hard redirect, like, when you go into a hotel and you have to put your room number in order to get past the browser and get on to browsing the web,” the company said.

In the three years since that statement, the company has raised the issue again but nothing concrete has come to fruition. However, there are now signs of fresh movement which could be significant, if Rightscorp is to be believed.

“An ISP Good Corporate Citizenship Program is what we feel will drive revenue associated with our primary revenue model. This program is an attempt to garner the attention and ultimately inspire a behavior shift in any ISP that elects to embrace our suggestions to be DMCA-compliant,” the company told shareholders yesterday.

“In this program, we ask for the ISPs to forward our notices referencing the infringement and the settlement offer. We ask that ISPs take action against repeat infringers through suspensions or a redirect screen. A redirect screen will guide the infringer to our payment screen while limiting all but essential internet access.”

At first view, this sounds like a straightforward replay of Rightscorp’s wishlist of three years ago, but it’s worth noting that the legal landscape has shifted fairly significantly since then.

Perhaps the most important development is the BMG v Cox Communications case, in which the ISP was sued for not doing enough to tackle repeat infringers. In that case (for which Rightscorp provided the evidence), Cox was held liable for third-party infringement and ordered to pay damages of $25 million alongside $8 million in legal fees.

All along, the suggestion has been that if Cox had taken action against infringing subscribers (primarily by passing on Rightscorp ‘fines’ and/or disconnecting repeat infringers) the ISP wouldn’t have ended up in court. Instead, it chose to sweat it out to a highly unfavorable decision.

The BMG decision is a potentially powerful ruling for Rightscorp, particularly when it comes to seeking ‘cooperation’ from other ISPs who might not want a similar legal battle on their hands. But are other ISPs interested in getting involved?

According to the Rightscorp, preliminary negotiations are already underway with some big players.

“We are now beginning to have some initial and very thorough discussions with a handful of the top ISPs to create and implement such a program that others can follow. We have every reason to believe that the litigations referred to above are directly responsible for the beginning of a change in thinking of ISPs,” the company says.

Rightscorp didn’t identify these “top ISPs” but by implication, these could include companies such as Comcast, AT&T, Time Warner Cable, CenturyLink, Charter, Verizon, and/or even Cox Communications.

With cooperation from these companies, Rightscorp predicts that a “cultural shift” could be brought about which would significantly increase the numbers of subscribers paying cash demands. It’s also clear that while it may be seeking cooperation from ISPs, a gun is being held under the table too, in case any feel hesitant about putting up a redirect screen.

“This is the preferred approach that we advocate for any willing ISP as an alternative to becoming a defendant in a litigation and facing potential liability and significantly larger statutory damages,” Rightscorp says.

A recent development suggests the company may not be bluffing. Back in April the RIAA sued ISP Grande Communcations for failing to disconnect persistent pirates. Yet again, Rightscorp is deeply involved in the case, having provided the infringement data to the labels for a considerable sum.

Whether the “top ISPs” in the United States will cave into the pressure and implied threats remains to be seen but there’s no doubting the rising confidence at Rightscorp.

“We have demonstrated the tenacity to support two major litigation efforts initiated by two of our clients, which we feel will set a precedent for the entire anti-piracy industry led by Rightscorp. If you can predict the law, you can set the competition,” the company concludes.

Meanwhile, Rightscorp appears to continue its use of disingenuous tactics to extract money from alleged file-sharers.

In the wake of several similar reports, this week a Reddit user reported that Rightscorp asked him to pay a single $20 fine for pirating a song. After paying up, the next day the company allegedly called the user back and demanded payment for a further 200 notices.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

Latency Distribution Graph in AWS X-Ray

Post Syndicated from Randall Hunt original https://aws.amazon.com/blogs/aws/latency-distribution-graph-in-aws-x-ray/

We’re continuing to iterate on the AWS X-Ray service based on customer feedback and today we’re excited to release a set of tools to help you quickly dive deep on latencies in your applications. Visual Node and Edge latency distribution graphs are shown in a handy new “Service Details” side bar in your X-Ray Service Map.

The X-Ray service graph gives you a visual representation of services and their interactions over a period of time that you select. The nodes represent services and the edges between the nodes represent calls between the services. The nodes and edges each have a set of statistics associated with them. While the visualizations provided in the service map are useful for estimating the average latency in an application they don’t help you to dive deep on specific issues. Most of the time issues occur at statistical outliers. To alleviate this X-Ray computes histograms like the one above help you solve those 99th percentile bugs.

To see a Response Distribution for a Node just click on it in the service graph. You can also click on the edges between the nodes to see the Response Distribution from the viewpoint of the calling service.

The team had a few interesting problems to solve while building out this feature and I wanted to share a bit of that with you now! Given the large number of traces an app can produce it’s not a great idea (for your browser) to plot every single trace client side. Instead most plotting libraries, when dealing with many points, use approximations and bucketing to get a network and performance friendly histogram. If you’ve used monitoring software in the past you’ve probably seen as you zoom in on the data you get higher fidelity. The interesting thing about the latencies coming in from X-Ray is that they vary by several orders of magnitude.

If the latencies were distributed between strictly 0s and 1s you could easily just create 10 buckets of 100 milliseconds. If your apps are anything like mine there’s a lot of interesting stuff happening in the outliers, so it’s beneficial to have more fidelity at 1% and 99% than it is at 50%. The problem with fixed bucket sizes is that they’re not necessarily giving you an accurate summary of data. So X-Ray, for now, uses dynamic bucket sizing based on the t-digests algorithm by Ted Dunning and Otmar Ertl. One of the distinct advantages of this algorithm over other approximation algorithms is its accuracy and precision at extremes (where most errors typically are).

An additional advantage of X-Ray over other monitoring software is the ability to measure two perspectives of latency simultaneously. Developers almost always have some view into the server side latency from their application logs but with X-Ray you can examine latency from the view of each of the clients, services, and microservices that you’re interacting with. You can even dive deeper by adding additional restrictions and queries on your selection. You can identify the specific users and clients that are having issues at that 99th percentile.

This info has already been available in API calls to GetServiceGraph as ResponseTimeHistogram but now we’re exposing it in the console as well to make it easier for customers to consume. For more information check out the documentation here.

Randall

Man Faces Prison For Sharing Pirated Deadpool Movie on Facebook

Post Syndicated from Ernesto original https://torrentfreak.com/man-faces-prison-for-sharing-pirated-deadpool-movie-on-facebook-170614/

With roughly two billion active users per month, Facebook is by far the largest social networking site around.

While most of the content posted to the site is relatively harmless, some people use it to share things they are not supposed to.

This is also what 21-year-old Trevon Maurice Franklin from Fresno, California, did early last year. Just a week after the box-office hit Deadpool premiered in theaters, he shared a pirated copy of the movie on the social network.

Franklin, who used the screen name “Tre-Von M. King,” saw his post go viral as it allegedly reached five million views. This didn’t go unnoticed by Twentieth Century Fox, and soon after the feds were involved as well.

The FBI began to investigate the possibly criminal Facebook post and decided to build a case. This eventually led to an indictment, and the alleged “pirate” was arrested soon after.

Facebook post from early 2016

The U.S. Attorney’s Office for the Central District of California, which released the news a few hours ago, states that Franklin faces up to three years in prison for the alleged copyright infringement.

“Franklin is charged in a one-count indictment returned by a federal grand jury on April 7 with reproducing and distributing a copyrighted work, a felony offense that carries a statutory maximum penalty of three years in federal prison,” the office wrote in a press release.

According to comments on Facebook, posted last year, several people warned “Tre-Von M. King” that it wasn’t wise to post copyright-infringing material on Facebook. However, Franklin said he wasn’t worried that he would get in trouble.

Comment from early 2016

While the case is significant, there are also plenty of questions that remain unanswered.

Was the defendant involved in recording the copyright infringing copy? Was it already widely available elsewhere? Are the reported five million “views” people who watched a large part of the movie, or is this just the number of people who might have seen it in their feeds?

Thus far we have not seen a copy of the indictment in the court records, but a follow-up may be warranted when it becomes available.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

Manage Instances at Scale without SSH Access Using EC2 Run Command

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/manage-instances-at-scale-without-ssh-access-using-ec2-run-command/

The guest post below, written by Ananth Vaidyanathan (Senior Product Manager for EC2 Systems Manager) and Rich Urmston (Senior Director of Cloud Architecture at Pegasystems) shows you how to use EC2 Run Command to manage a large collection of EC2 instances without having to resort to SSH.

Jeff;


Enterprises often have several managed environments and thousands of Amazon EC2 instances. It’s important to manage systems securely, without the headaches of Secure Shell (SSH). Run Command, part of Amazon EC2 Systems Manager, allows you to run remote commands on instances (or groups of instances using tags) in a controlled and auditable manner. It’s been a nice added productivity boost for Pega Cloud operations, which rely daily on Run Command services.

You can control Run Command access through standard IAM roles and policies, define documents to take input parameters, control the S3 bucket used to return command output. You can also share your documents with other AWS accounts, or with the public. All in all, Run Command provides a nice set of remote management features.

Better than SSH
Here’s why Run Command is a better option than SSH and why Pegasystems has adopted it as their primary remote management tool:

Run Command Takes Less Time –  Securely connecting to an instance requires a few steps e.g. jumpboxes to connect to or IP addresses to whitelist etc. With Run Command, cloud ops engineers can invoke commands directly from their laptop, and never have to find keys or even instance IDs. Instead, system security relies on AWS auth, IAM roles and policies.

Run Command Operations are Fully Audited – With SSH, there is no real control over what they can do, nor is there an audit trail. With Run Command, every invoked operation is audited in CloudTrail, including information on the invoking user, instances on which command was run, parameters, and operation status. You have full control and ability to restrict what functions engineers can perform on a system.

Run Command has no SSH keys to Manage – Run Command leverages standard AWS credentials, API keys, and IAM policies. Through integration with a corporate auth system, engineers can interact with systems based on their corporate credentials and identity.

Run Command can Manage Multiple Systems at the Same Time – Simple tasks such as looking at the status of a Linux service or retrieving a log file across a fleet of managed instances is cumbersome using SSH. Run Command allows you to specify a list of instances by IDs or tags, and invokes your command, in parallel, across the specified fleet. This provides great leverage when troubleshooting or managing more than the smallest Pega clusters.

Run Command Makes Automating Complex Tasks Easier – Standardizing operational tasks requires detailed procedure documents or scripts describing the exact commands. Managing or deploying these scripts across the fleet is cumbersome. Run Command documents provide an easy way to encapsulate complex functions, and handle document management and access controls. When combined with AWS Lambda, documents provide a powerful automation platform to handle any complex task.

Example – Restarting a Docker Container
Here is an example of a simple document used to restart a Docker container. It takes one parameter; the name of the Docker container to restart. It uses the AWS-RunShellScript method to invoke the command. The output is collected automatically by the service and returned to the caller. For an example of the latest document schema, see Creating Systems Manager Documents.

{
  "schemaVersion":"1.2",
  "description":"Restart the specified docker container.",
  "parameters":{
    "param":{
      "type":"String",
      "description":"(Required) name of the container to restart.",
      "maxChars":1024
    }
  },
  "runtimeConfig":{
    "aws:runShellScript":{
      "properties":[
        {
          "id":"0.aws:runShellScript",
          "runCommand":[
            "docker restart {{param}}"
          ]
        }
      ]
    }
  }
}

Putting Run Command into practice at Pegasystems
The Pegasystems provisioning system sits on AWS CloudFormation, which is used to deploy and update Pega Cloud resources. Layered on top of it is the Pega Provisioning Engine, a serverless, Lambda-based service that manages a library of CloudFormation templates and Ansible playbooks.

A Configuration Management Database (CMDB) tracks all the configurations details and history of every deployment and update, and lays out its data using a hierarchical directory naming convention. The following diagram shows how the various systems are integrated:

For cloud system management, Pega operations uses a command line version called cuttysh and a graphical version based on the Pega 7 platform, called the Pega Operations Portal. Both tools allow you to browse the CMDB of deployed environments, view configuration settings, and interact with deployed EC2 instances through Run Command.

CLI Walkthrough
Here is a CLI walkthrough for looking into a customer deployment and interacting with instances using Run Command.

Launching the cuttysh tool brings you to the root of the CMDB and a list of the provisioned customers:

% cuttysh
d CUSTA
d CUSTB
d CUSTC
d CUSTD

You interact with the CMDB using standard Linux shell commands, such as cd, ls, cat, and grep. Items prefixed with s are services that have viewable properties. Items prefixed with d are navigable subdirectories in the CMDB hierarchy.

In this example, change directories into customer CUSTB’s portion of the CMDB hierarchy, and then further into a provisioned Pega environment called env1, under the Dev network. The tool displays the artifacts that are provisioned for that environment. These entries map to provisioned CloudFormation templates.

> cd CUSTB
/ROOT/CUSTB/us-east-1 > cd DEV/env1

The ls –l command shows the version of the provisioned resources. These version numbers map back to source control–managed artifacts for the CloudFormation, Ansible, and other components that compose a version of the Pega Cloud.

/ROOT/CUSTB/us-east-1/DEV/env1 > ls -l
s 1.2.5 RDSDatabase 
s 1.2.5 PegaAppTier 
s 7.2.1 Pega7 

Now, use Run Command to interact with the deployed environments. To do this, use the attach command and specify the service with which to interact. In the following example, you attach to the Pega Web Tier. Using the information in the CMDB and instance tags, the CLI finds the corresponding EC2 instances and displays some basic information about them. This deployment has three instances.

/ROOT/CUSTB/us-east-1/DEV/env1 > attach PegaWebTier
 # ID         State  Public Ip    Private Ip  Launch Time
 0 i-0cf0e84 running 52.63.216.42 10.96.15.70 2017-01-16 
 1 i-0043c1d running 53.47.191.22 10.96.15.43 2017-01-16 
 2 i-09b879e running 55.93.118.27 10.96.15.19 2017-01-16 

From here, you can use the run command to invoke Run Command documents. In the following example, you run the docker-ps document against instance 0 (the first one on the list). EC2 executes the command and returns the output to the CLI, which in turn shows it.

/ROOT/CUSTB/us-east-1/DEV/env1 > run 0 docker-ps
. . 
CONTAINER ID IMAGE             CREATED      STATUS        NAMES
2f187cc38c1  pega-7.2         10 weeks ago  Up 8 weeks    pega-web

Using the same command and some of the other documents that have been defined, you can restart a Docker container or even pull back the contents of a file to your local system. When you get a file, Run Command also leaves a copy in an S3 bucket in case you want to pass the link along to a colleague.

/ROOT/CUSTB/us-east-1/DEV/env1 > run 0 docker-restart pega-web
..
pega-web

/ROOT/CUSTB/us-east-1/DEV/env1 > run 0 get-file /var/log/cfn-init-cmd.log
. . . . . 
get-file

Data has been copied locally to: /tmp/get-file/i-0563c9e/data
Data is also available in S3 at: s3://my-bucket/CUSTB/cuttysh/get-file/data

Now, leverage the Run Command ability to do more than one thing at a time. In the following example, you attach to a deployment with three running instances and want to see the uptime for each instance. Using the par (parallel) option for run, the CLI tells Run Command to execute the uptime document on all instances in parallel.

/ROOT/CUSTB/us-east-1/DEV/env1 > run par uptime
 …
Output for: i-006bdc991385c33
 20:39:12 up 15 days, 3:54, 0 users, load average: 0.42, 0.32, 0.30

Output for: i-09390dbff062618
 20:39:12 up 15 days, 3:54, 0 users, load average: 0.08, 0.19, 0.22

Output for: i-08367d0114c94f1
 20:39:12 up 15 days, 3:54, 0 users, load average: 0.36, 0.40, 0.40

Commands are complete.
/ROOT/PEGACLOUD/CUSTB/us-east-1/PROD/prod1 > 

Summary
Run Command improves productivity by giving you faster access to systems and the ability to run operations across a group of instances. Pega Cloud operations has integrated Run Command with other operational tools to provide a clean and secure method for managing systems. This greatly improves operational efficiency, and gives greater control over who can do what in managed deployments. The Pega continual improvement process regularly assesses why operators need access, and turns those operations into new Run Command documents to be added to the library. In fact, their long-term goal is to stop deploying cloud systems with SSH enabled.

If you have any questions or suggestions, please leave a comment for us!

— Ananth and Rich

Global Entertainment Giants Form Massive Anti-Piracy Coalition

Post Syndicated from Andy original https://torrentfreak.com/global-entertainment-giants-form-massive-anti-piracy-coalition-170613/

It’s not unusual for companies within the same area of business to collaborate in order to combat piracy. The studios and labels that form the MPAA and RIAA, for example, have doing just that for decades.

Today, however, an unprecedented number of global content creators and distribution platforms have announced the formation of a brand new coalition to collaboratively fight Internet piracy on a global scale.

The Alliance for Creativity and Entertainment (ACE) is a coalition of 30 companies that reads like a who’s who of the global entertainment market. In alphabetical order the members are:

Amazon, AMC Networks, BBC Worldwide, Bell Canada and Bell Media, Canal+ Group, CBS Corporation, Constantin Film, Foxtel, Grupo Globo, HBO, Hulu, Lionsgate, Metro-Goldwyn-Mayer (MGM), Millennium Media, NBCUniversal, Netflix, Paramount Pictures, SF Studios, Sky, Sony Pictures Entertainment, Star India, Studio Babelsberg, STX Entertainment, Telemundo, Televisa, Twentieth Century Fox, Univision Communications Inc., Village Roadshow, The Walt Disney Company, and Warner Bros. Entertainment Inc.

In a joint announcement today, ACE notes that there are now more than 480 services available for consumers to watch films and TV programs online. However, despite that abundance of content, piracy continues to pose a threat to creators and the economy.

“Films and television shows can often be found on pirate sites within days – and in many cases hours – of release,” ACE said in a statement.

“Last year, there were an estimated 5.4 billion downloads of pirated wide release films and primetime television and VOD shows using peer-to-peer protocols worldwide. There were also an estimated 21.4 billion total visits to streaming piracy sites worldwide across both desktops and mobile devices in 2016.”

Rather than the somewhat fragmented anti-piracy approach currently employed by ACE members separately, the coalition will present a united front of all major content creators and distributors, with a mission to cooperate and expand in order to minimize the threat.

At the center of the alliance appears to be the MPAA. ACE reports that the anti-piracy resources of the Hollywood group will be used “in concert” with the existing anti-piracy departments of the member companies.

Unprecedented scale aside, ACE’s modus operandi will be a familiar one.

The coalition says it will work closely with law enforcement to shut down pirate sites and services, file civil litigation, and forge new relationships with other content protection groups. It will also strive to reach voluntary anti-piracy agreements with other interested parties across the Internet.

MPAA chief Chris Dodd, whose group will play a major role in ACE, welcomed the birth of the alliance.

“ACE, with its broad coalition of creators from around the world, is designed, specifically, to leverage the best possible resources to reduce piracy,” Dodd said.

“For decades, the MPAA has been the gold standard for antipiracy enforcement. We are proud to provide the MPAA’s worldwide antipiracy resources and the deep expertise of our antipiracy unit to support ACE and all its initiatives.”

The traditionally non-aggressive BBC described ACE as “hugely important” in the fight against “theft and illegal distribution”, with Netflix noting that even its creative strategies for dealing with piracy are in need of assistance.

“While we’re focused on providing a great consumer experience that ultimately discourages piracy, there are still bad players around the world trying to profit off the hard work of others,” said Netflix General Counsel, David Hyman.

“By joining ACE, we will work together, share knowledge, and leverage the group’s combined anti-piracy resources to address the global online piracy problem.”

It’s likely that the creation of ACE will go down as a landmark moment in the fight against piracy. Never before has such a broad coalition promised to pool resources on such a grand and global scale. That being said, with great diversity comes the potential for greatly diverging opinions, so only time will tell if this coalition can really hold together.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

Security Flaws in 4G VoLTE

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2017/06/security_flaws_1.html

Research paper: “Subscribers remote geolocation and tracking using 4G VoLTE enabled Android phone,” by Patrick Ventuzelo, Olivier Le Moal, and Thomas Coudray.

Abstract: VoLTE (Voice over LTE) is a technology implemented by many operators over the world. Unlike previous 2G/3G technologies, VoLTE offers the possibility to use the end-to-end IP networks to handle voice communications. This technology uses VoIP (Voice over IP) standards over IMS (IP Multimedia Subsystem) networks. In this paper, we will first introduce the basics of VoLTE technology. We will then demonstrate how to use an Android phone to communicate with VoLTE networks and what normal VoLTE communications look like. Finally, we will describe different issues and implementations’ problems. We will present vulnerabilities, both passive and active, and attacks that can be done using VoLTE Android smartphones to attack subscribers and operators’ infrastructures. Some of these vulnerabilities are new and not previously disclosed: they may allow an attacker to silently retrieve private pieces of information on targeted subscribers, such as their geolocation.

News article. Slashdot thread.

More notes on US-CERTs IOCs

Post Syndicated from Robert Graham original http://blog.erratasec.com/2017/06/more-notes-on-us-certs-iocs.html

Yet another Russian attack against the power grid, and yet more bad IOCs from the DHS US-CERT.

IOCs are “indicators of compromise“, things you can look for in order to order to see if you, too, have been hacked by the same perpetrators. There are several types of IOCs, ranging from the highly specific to the uselessly generic.

A uselessly generic IOC would be like trying to identify bank robbers by the fact that their getaway car was “white” in color. It’s worth documenting, so that if the police ever show up in a suspected cabin in the woods, they can note that there’s a “white” car parked in front.

But if you work bank security, that doesn’t mean you should be on the lookout for “white” cars. That would be silly.

This is what happens with US-CERT’s IOCs. They list some potentially useful things, but they also list a lot of junk that waste’s people’s times, with little ability to distinguish between the useful and the useless.

An example: a few months ago was the GRIZZLEYBEAR report published by US-CERT. Among other things, it listed IP addresses used by hackers. There was no description which would be useful IP addresses to watch for, and which would be useless.

Some of these IP addresses were useful, pointing to servers the group has been using a long time as command-and-control servers. Other IP addresses are more dubious, such as Tor exit nodes. You aren’t concerned about any specific Tor exit IP address, because it changes randomly, so has no relationship to the attackers. Instead, if you cared about those Tor IP addresses, what you should be looking for is a dynamically updated list of Tor nodes updated daily.

And finally, they listed IP addresses of Yahoo, because attackers passed data through Yahoo servers. No, it wasn’t because those Yahoo servers had been compromised, it’s just that everyone passes things though them, like email.

A Vermont power-plant blindly dumped all those IP addresses into their sensors. As a consequence, the next morning when an employee checked their Yahoo email, the sensors triggered. This resulted in national headlines about the Russians hacking the Vermont power grid.

Today, the US-CERT made similar mistakes with CRASHOVERRIDE. They took a report from Dragos Security, then mutilated it. Dragos’s own IOCs focused on things like hostile strings and file hashes of the hostile files. They also included filenames, but similar to the reason you’d noticed a white car — because it happened, not because you should be on the lookout for it. In context, there’s nothing wrong with noting the file name.

But the US-CERT pulled the filenames out of context. One of those filenames was, humorously, “svchost.exe”. It’s the name of an essential Windows service. Every Windows computer is running multiple copies of “svchost.exe”. It’s like saying “be on the lookout for Windows”.

Yes, it’s true that viruses use the same filenames as essential Windows files like “svchost.exe”. That’s, generally, something you should be aware of. But that CRASHOVERRIDE did this is wholly meaningless.

What Dragos Security was actually reporting was that a “svchost.exe” with the file hash of 79ca89711cdaedb16b0ccccfdcfbd6aa7e57120a was the virus — it’s the hash that’s the important IOC. Pulling the filename out of context is just silly.

Luckily, the DHS also provides some of the raw information provided by Dragos. But even then, there’s problems: they provide it in formatted form, for HTML, PDF, or Excel documents. This corrupts the original data so that it’s no longer machine readable. For example, from their webpage, they have the following:

import “pe”
import “hash”

Among the problems are the fact that the quote marks have been altered, probably by Word’s “smart quotes” feature. In other cases, I’ve seen PDF documents get confused by the number 0 and the letter O, as if the raw data had been scanned in from a printed document and OCRed.

If this were a “threat intel” company,  we’d call this snake oil. The US-CERT is using Dragos Security’s reports to promote itself, but ultimate providing negative value, mutilating the content.

This, ultimately, causes a lot of harm. The press trusted their content. So does the network of downstream entities, like municipal power grids. There are tens of thousands of such consumers of these reports, often with less expertise than even US-CERT. There are sprinklings of smart people in these organizations, I meet them at hacker cons, and am fascinated by their stories. But institutionally, they are dumbed down the same level as these US-CERT reports, with the smart people marginalized.

There are two solutions to this problem. The first is that when the stupidity of what you do causes everyone to laugh at you, stop doing it. The second is to value technical expertise, empowering those who know what they are doing. Examples of what not to do are giving power to people like Obama’s cyberczar, Michael Daniels, who once claimed his lack of technical knowledge was a bonus, because it allowed him to see the strategic picture instead of getting distracted by details.

Who’s To Blame For The Kodi Crackdown?

Post Syndicated from Andy original https://torrentfreak.com/whos-to-blame-for-the-kodi-crackdown-170611/

Perfectly legal as standard, the Kodi media player can be easily modified to turn it into the ultimate streaming piracy machine.

Uptake by users has been nothing short of phenomenal. Millions of people are now consuming illicit media through third-party Kodi addons. With free movies, TV shows, sports, live TV and more on tap, it’s not difficult to see why the system is so popular.

As a result, barely a day goes by without Kodi making headlines and this week was no exception. On Monday, TorrentFreak broke the news that the ZEMTV addon and TV Addons, one of the most popular addon communities, were being sued by Dish Network for copyright infringement.

Within hours of the announcement and apparently as a direct result, several addons (including the massively popular Phoenix) decided to throw in the towel. Quite understandably, users of the platforms were disappointed, and that predictably resulted in people attempting to apportion blame.

The first comment to catch the eye was posted directly beneath our article. Interestingly, it placed the blame squarely on our shoulders.

“Thanks Torrentfreak, for ruining Kodi,” it read.

While shooting the messenger is an option, it’s historically problematic. Town criers were the original newsreaders, delivering important messages to the public. Killing a town crier was considered treason, but it was also pointless – it didn’t change the facts on the ground.

So if we can’t kill those who read about a lawsuit in the public PACER system and reported it, who’s left to blame? Unsurprisingly, there’s no shortage of targets, but most of them fall short.

The underlying theme is that most people voicing a negative opinion about the profile of Kodi do not appreciate their previously niche piracy system being in the spotlight. Everything was just great when just a few people knew about the marvelous hidden world of ‘secret’ XBMC/Kodi addons, many insist, but seeing it in the mainstream press is a disaster. It’s difficult to disagree.

However, the point where this all falls down is when people are asked when the discussion about Kodi should’ve stopped. We haven’t questioned them all, of course, but it’s almost guaranteed that while most with a grievance didn’t want Kodi getting too big, they absolutely appreciate the fact that someone told them about it. Piracy and piracy techniques spread by word of mouth so unfortunately, people can’t have it both ways.

Interestingly, some people placed the blame on TV Addons, the site that hosts the addons themselves. They argued that the addon scene didn’t need such a high profile target and that the popularity of the site only brought unwanted attention. However, for every critic, there are apparently thousands who love what the site does to raise the profile of Kodi. Without that, it’s clear that there would be fewer users and indeed, fewer addons.

For TV Addons’ part, they’re extremely clear who’s responsible for bringing the heat. On numerous occasions in emails to TF, the operators of the repository have blamed those who have attempted to commercialize the Kodi scene. For them, the responsibility must be placed squarely on the shoulders of people selling ‘Kodi boxes’ on places like eBay and Amazon. Once big money got involved, that attracted the authorities, they argue.

With this statement in mind, TF spoke with a box seller who previously backed down from selling on eBay due to issues over Kodi’s trademark. He didn’t want to speak on the record but admitted to selling “a couple of thousand” boxes over the past two years, noting that all he did was respond to demand with supply.

And this brings us full circle and a bit closer to apportioning blame for the Kodi crackdown.

The bottom line is that when it comes to piracy, Kodi and its third-party ‘pirate’ addons are so good at what they do, it’s no surprise they’ve been a smash hit with Internet users. All of the content that anyone could want – and more – accessible in one package, on almost any platform? That’s what consumers have been demanding for more than a decade and a half.

That brings us to the unavoidable conclusion that modified Kodi simply got too good at delivering content outside controlled channels, and that success was impossible to moderate or calm. Quite simply, every user that added to the Kodi phenomenon by installing the software with ‘pirate’ addons has to shoulder some of the blame for the crackdown.

That might sound harsh but in the piracy world it’s never been any different. Without millions of users, The Pirate Bay raid would never have happened. Without users, KickassTorrents might still be rocking today. But of course, what would be the point?

Users might break sites and services, but they also make them. That’s the piracy paradox.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.