Tag Archives: reports

Announcing the Winners of the AWS Chatbot Challenge – Conversational, Intelligent Chatbots using Amazon Lex and AWS Lambda

Post Syndicated from Tara Walker original https://aws.amazon.com/blogs/aws/announcing-the-winners-of-the-aws-chatbot-challenge-conversational-intelligent-chatbots-using-amazon-lex-and-aws-lambda/

A couple of months ago on the blog, I announced the AWS Chatbot Challenge in conjunction with Slack. The AWS Chatbot Challenge was an opportunity to build a unique chatbot that helped to solve a problem or that would add value for its prospective users. The mission was to build a conversational, natural language chatbot using Amazon Lex and leverage Lex’s integration with AWS Lambda to execute logic or data processing on the backend.

I know that you all have been anxiously waiting to hear announcements of who were the winners of the AWS Chatbot Challenge as much as I was. Well wait no longer, the winners of the AWS Chatbot Challenge have been decided.

May I have the Envelope Please? (The Trumpets sound)

The winners of the AWS Chatbot Challenge are:

  • First Place: BuildFax Counts by Joe Emison
  • Second Place: Hubsy by Andrew Riess, Andrew Puch, and John Wetzel
  • Third Place: PFMBot by Benny Leong and his team from MoneyLion.
  • Large Organization Winner: ADP Payroll Innovation Bot by Eric Liu, Jiaxing Yan, and Fan Yang

 

Diving into the Winning Chatbot Projects

Let’s take a walkthrough of the details for each of the winning projects to get a view of what made these chatbots distinctive, as well as, learn more about the technologies used to implement the chatbot solution.

 

BuildFax Counts by Joe Emison

The BuildFax Counts bot was created as a real solution for the BuildFax company to decrease the amount the time that sales and marketing teams can get answers on permits or properties with permits meet certain criteria.

BuildFax, a company co-founded by bot developer Joe Emison, has the only national database of building permits, which updates data from approximately half of the United States on a monthly basis. In order to accommodate the many requests that come in from the sales and marketing team regarding permit information, BuildFax has a technical sales support team that fulfills these requests sent to a ticketing system by manually writing SQL queries that run across the shards of the BuildFax databases. Since there are a large number of requests received by the internal sales support team and due to the manual nature of setting up the queries, it may take several days for getting the sales and marketing teams to receive an answer.

The BuildFax Counts chatbot solves this problem by taking the permit inquiry that would normally be sent into a ticket from the sales and marketing team, as input from Slack to the chatbot. Once the inquiry is submitted into Slack, a query executes and the inquiry results are returned immediately.

Joe built this solution by first creating a nightly export of the data in their BuildFax MySQL RDS database to CSV files that are stored in Amazon S3. From the exported CSV files, an Amazon Athena table was created in order to run quick and efficient queries on the data. He then used Amazon Lex to create a bot to handle the common questions and criteria that may be asked by the sales and marketing teams when seeking data from the BuildFax database by modeling the language used from the BuildFax ticketing system. He added several different sample utterances and slot types; both custom and Lex provided, in order to correctly parse every question and criteria combination that could be received from an inquiry.  Using Lambda, Joe created a Javascript Lambda function that receives information from the Lex intent and used it to build a SQL statement that runs against the aforementioned Athena database using the AWS SDK for JavaScript in Node.js library to return inquiry count result and SQL statement used.

The BuildFax Counts bot is used today for the BuildFax sales and marketing team to get back data on inquiries immediately that previously took up to a week to receive results.

Not only is BuildFax Counts bot our 1st place winner and wonderful solution, but its creator, Joe Emison, is a great guy.  Joe has opted to donate his prize; the $5,000 cash, the $2,500 in AWS Credits, and one re:Invent ticket to the Black Girls Code organization. I must say, you rock Joe for helping these kids get access and exposure to technology.

 

Hubsy by Andrew Riess, Andrew Puch, and John Wetzel

Hubsy bot was created to redefine and personalize the way users traditionally manage their HubSpot account. HubSpot is a SaaS system providing marketing, sales, and CRM software. Hubsy allows users of HubSpot to create engagements and log engagements with customers, provide sales teams with deals status, and retrieves client contact information quickly. Hubsy uses Amazon Lex’s conversational interface to execute commands from the HubSpot API so that users can gain insights, store and retrieve data, and manage tasks directly from Facebook, Slack, or Alexa.

In order to implement the Hubsy chatbot, Andrew and the team members used AWS Lambda to create a Lambda function with Node.js to parse the users request and call the HubSpot API, which will fulfill the initial request or return back to the user asking for more information. Terraform was used to automatically setup and update Lambda, CloudWatch logs, as well as, IAM profiles. Amazon Lex was used to build the conversational piece of the bot, which creates the utterances that a person on a sales team would likely say when seeking information from HubSpot. To integrate with Alexa, the Amazon Alexa skill builder was used to create an Alexa skill which was tested on an Echo Dot. Cloudwatch Logs are used to log the Lambda function information to CloudWatch in order to debug different parts of the Lex intents. In order to validate the code before the Terraform deployment, ESLint was additionally used to ensure the code was linted and proper development standards were followed.

 

PFMBot by Benny Leong and his team from MoneyLion

PFMBot, Personal Finance Management Bot,  is a bot to be used with the MoneyLion finance group which offers customers online financial products; loans, credit monitoring, and free credit score service to improve the financial health of their customers. Once a user signs up an account on the MoneyLion app or website, the user has the option to link their bank accounts with the MoneyLion APIs. Once the bank account is linked to the APIs, the user will be able to login to their MoneyLion account and start having a conversation with the PFMBot based on their bank account information.

The PFMBot UI has a web interface built with using Javascript integration. The chatbot was created using Amazon Lex to build utterances based on the possible inquiries about the user’s MoneyLion bank account. PFMBot uses the Lex built-in AMAZON slots and parsed and converted the values from the built-in slots to pass to AWS Lambda. The AWS Lambda functions interacting with Amazon Lex are Java-based Lambda functions which call the MoneyLion Java-based internal APIs running on Spring Boot. These APIs obtain account data and related bank account information from the MoneyLion MySQL Database.

 

ADP Payroll Innovation Bot by Eric Liu, Jiaxing Yan, and Fan Yang

ADP PI (Payroll Innovation) bot is designed to help employees of ADP customers easily review their own payroll details and compare different payroll data by just asking the bot for results. The ADP PI Bot additionally offers issue reporting functionality for employees to report payroll issues and aids HR managers in quickly receiving and organizing any reported payroll issues.

The ADP Payroll Innovation bot is an ecosystem for the ADP payroll consisting of two chatbots, which includes ADP PI Bot for external clients (employees and HR managers), and ADP PI DevOps Bot for internal ADP DevOps team.


The architecture for the ADP PI DevOps bot is different architecture from the ADP PI bot shown above as it is deployed internally to ADP. The ADP PI DevOps bot allows input from both Slack and Alexa. When input comes into Slack, Slack sends the request to Lex for it to process the utterance. Lex then calls the Lambda backend, which obtains ADP data sitting in the ADP VPC running within an Amazon VPC. When input comes in from Alexa, a Lambda function is called that also obtains data from the ADP VPC running on AWS.

The architecture for the ADP PI bot consists of users entering in requests and/or entering issues via Slack. When requests/issues are entered via Slack, the Slack APIs communicate via Amazon API Gateway to AWS Lambda. The Lambda function either writes data into one of the Amazon DynamoDB databases for recording issues and/or sending issues or it sends the request to Lex. When sending issues, DynamoDB integrates with Trello to keep HR Managers abreast of the escalated issues. Once the request data is sent from Lambda to Lex, Lex processes the utterance and calls another Lambda function that integrates with the ADP API and it calls ADP data from within the ADP VPC, which runs on Amazon Virtual Private Cloud (VPC).

Python and Node.js were the chosen languages for the development of the bots.

The ADP PI bot ecosystem has the following functional groupings:

Employee Functionality

  • Summarize Payrolls
  • Compare Payrolls
  • Escalate Issues
  • Evolve PI Bot

HR Manager Functionality

  • Bot Management
  • Audit and Feedback

DevOps Functionality

  • Reduce call volume in service centers (ADP PI Bot).
  • Track issues and generate reports (ADP PI Bot).
  • Monitor jobs for various environment (ADP PI DevOps Bot)
  • View job dashboards (ADP PI DevOps Bot)
  • Query job details (ADP PI DevOps Bot)

 

Summary

Let’s all wish all the winners of the AWS Chatbot Challenge hearty congratulations on their excellent projects.

You can review more details on the winning projects, as well as, all of the submissions to the AWS Chatbot Challenge at: https://awschatbot2017.devpost.com/submissions. If you are curious on the details of Chatbot challenge contest including resources, rules, prizes, and judges, you can review the original challenge website here:  https://awschatbot2017.devpost.com/.

Hopefully, you are just as inspired as I am to build your own chatbot using Lex and Lambda. For more information, take a look at the Amazon Lex developer guide or the AWS AI blog on Building Better Bots Using Amazon Lex (Part 1)

Chat with you soon!

Tara

Analyzing AWS Cost and Usage Reports with Looker and Amazon Athena

Post Syndicated from Dillon Morrison original https://aws.amazon.com/blogs/big-data/analyzing-aws-cost-and-usage-reports-with-looker-and-amazon-athena/

This is a guest post by Dillon Morrison at Looker. Looker is, in their own words, “a new kind of analytics platform–letting everyone in your business make better decisions by getting reliable answers from a tool they can use.” 

As the breadth of AWS products and services continues to grow, customers are able to more easily move their technology stack and core infrastructure to AWS. One of the attractive benefits of AWS is the cost savings. Rather than paying upfront capital expenses for large on-premises systems, customers can instead pay variables expenses for on-demand services. To further reduce expenses AWS users can reserve resources for specific periods of time, and automatically scale resources as needed.

The AWS Cost Explorer is great for aggregated reporting. However, conducting analysis on the raw data using the flexibility and power of SQL allows for much richer detail and insight, and can be the better choice for the long term. Thankfully, with the introduction of Amazon Athena, monitoring and managing these costs is now easier than ever.

In the post, I walk through setting up the data pipeline for cost and usage reports, Amazon S3, and Athena, and discuss some of the most common levers for cost savings. I surface tables through Looker, which comes with a host of pre-built data models and dashboards to make analysis of your cost and usage data simple and intuitive.

Analysis with Athena

With Athena, there’s no need to create hundreds of Excel reports, move data around, or deploy clusters to house and process data. Athena uses Apache Hive’s DDL to create tables, and the Presto querying engine to process queries. Analysis can be performed directly on raw data in S3. Conveniently, AWS exports raw cost and usage data directly into a user-specified S3 bucket, making it simple to start querying with Athena quickly. This makes continuous monitoring of costs virtually seamless, since there is no infrastructure to manage. Instead, users can leverage the power of the Athena SQL engine to easily perform ad-hoc analysis and data discovery without needing to set up a data warehouse.

After the data pipeline is established, cost and usage data (the recommended billing data, per AWS documentation) provides a plethora of comprehensive information around usage of AWS services and the associated costs. Whether you need the report segmented by product type, user identity, or region, this report can be cut-and-sliced any number of ways to properly allocate costs for any of your business needs. You can then drill into any specific line item to see even further detail, such as the selected operating system, tenancy, purchase option (on-demand, spot, or reserved), and so on.

Walkthrough

By default, the Cost and Usage report exports CSV files, which you can compress using gzip (recommended for performance). There are some additional configuration options for tuning performance further, which are discussed below.

Prerequisites

If you want to follow along, you need the following resources:

Enable the cost and usage reports

First, enable the Cost and Usage report. For Time unit, select Hourly. For Include, select Resource IDs. All options are prompted in the report-creation window.

The Cost and Usage report dumps CSV files into the specified S3 bucket. Please note that it can take up to 24 hours for the first file to be delivered after enabling the report.

Configure the S3 bucket and files for Athena querying

In addition to the CSV file, AWS also creates a JSON manifest file for each cost and usage report. Athena requires that all of the files in the S3 bucket are in the same format, so we need to get rid of all these manifest files. If you’re looking to get started with Athena quickly, you can simply go into your S3 bucket and delete the manifest file manually, skip the automation described below, and move on to the next section.

To automate the process of removing the manifest file each time a new report is dumped into S3, which I recommend as you scale, there are a few additional steps. The folks at Concurrency labs wrote a great overview and set of scripts for this, which you can find in their GitHub repo.

These scripts take the data from an input bucket, remove anything unnecessary, and dump it into a new output bucket. We can utilize AWS Lambda to trigger this process whenever new data is dropped into S3, or on a nightly basis, or whatever makes most sense for your use-case, depending on how often you’re querying the data. Please note that enabling the “hourly” report means that data is reported at the hour-level of granularity, not that a new file is generated every hour.

Following these scripts, you’ll notice that we’re adding a date partition field, which isn’t necessary but improves query performance. In addition, converting data from CSV to a columnar format like ORC or Parquet also improves performance. We can automate this process using Lambda whenever new data is dropped in our S3 bucket. Amazon Web Services discusses columnar conversion at length, and provides walkthrough examples, in their documentation.

As a long-term solution, best practice is to use compression, partitioning, and conversion. However, for purposes of this walkthrough, we’re not going to worry about them so we can get up-and-running quicker.

Set up the Athena query engine

In your AWS console, navigate to the Athena service, and click “Get Started”. Follow the tutorial and set up a new database (we’ve called ours “AWS Optimizer” in this example). Don’t worry about configuring your initial table, per the tutorial instructions. We’ll be creating a new table for cost and usage analysis. Once you walked through the tutorial steps, you’ll be able to access the Athena interface, and can begin running Hive DDL statements to create new tables.

One thing that’s important to note, is that the Cost and Usage CSVs also contain the column headers in their first row, meaning that the column headers would be included in the dataset and any queries. For testing and quick set-up, you can remove this line manually from your first few CSV files. Long-term, you’ll want to use a script to programmatically remove this row each time a new file is dropped in S3 (every few hours typically). We’ve drafted up a sample script for ease of reference, which we run on Lambda. We utilize Lambda’s native ability to invoke the script whenever a new object is dropped in S3.

For cost and usage, we recommend using the DDL statement below. Since our data is in CSV format, we don’t need to use a SerDe, we can simply specify the “separatorChar, quoteChar, and escapeChar”, and the structure of the files (“TEXTFILE”). Note that AWS does have an OpenCSV SerDe as well, if you prefer to use that.

 

CREATE EXTERNAL TABLE IF NOT EXISTS cost_and_usage	 (
identity_LineItemId String,
identity_TimeInterval String,
bill_InvoiceId String,
bill_BillingEntity String,
bill_BillType String,
bill_PayerAccountId String,
bill_BillingPeriodStartDate String,
bill_BillingPeriodEndDate String,
lineItem_UsageAccountId String,
lineItem_LineItemType String,
lineItem_UsageStartDate String,
lineItem_UsageEndDate String,
lineItem_ProductCode String,
lineItem_UsageType String,
lineItem_Operation String,
lineItem_AvailabilityZone String,
lineItem_ResourceId String,
lineItem_UsageAmount String,
lineItem_NormalizationFactor String,
lineItem_NormalizedUsageAmount String,
lineItem_CurrencyCode String,
lineItem_UnblendedRate String,
lineItem_UnblendedCost String,
lineItem_BlendedRate String,
lineItem_BlendedCost String,
lineItem_LineItemDescription String,
lineItem_TaxType String,
product_ProductName String,
product_accountAssistance String,
product_architecturalReview String,
product_architectureSupport String,
product_availability String,
product_bestPractices String,
product_cacheEngine String,
product_caseSeverityresponseTimes String,
product_clockSpeed String,
product_currentGeneration String,
product_customerServiceAndCommunities String,
product_databaseEdition String,
product_databaseEngine String,
product_dedicatedEbsThroughput String,
product_deploymentOption String,
product_description String,
product_durability String,
product_ebsOptimized String,
product_ecu String,
product_endpointType String,
product_engineCode String,
product_enhancedNetworkingSupported String,
product_executionFrequency String,
product_executionLocation String,
product_feeCode String,
product_feeDescription String,
product_freeQueryTypes String,
product_freeTrial String,
product_frequencyMode String,
product_fromLocation String,
product_fromLocationType String,
product_group String,
product_groupDescription String,
product_includedServices String,
product_instanceFamily String,
product_instanceType String,
product_io String,
product_launchSupport String,
product_licenseModel String,
product_location String,
product_locationType String,
product_maxIopsBurstPerformance String,
product_maxIopsvolume String,
product_maxThroughputvolume String,
product_maxVolumeSize String,
product_maximumStorageVolume String,
product_memory String,
product_messageDeliveryFrequency String,
product_messageDeliveryOrder String,
product_minVolumeSize String,
product_minimumStorageVolume String,
product_networkPerformance String,
product_operatingSystem String,
product_operation String,
product_operationsSupport String,
product_physicalProcessor String,
product_preInstalledSw String,
product_proactiveGuidance String,
product_processorArchitecture String,
product_processorFeatures String,
product_productFamily String,
product_programmaticCaseManagement String,
product_provisioned String,
product_queueType String,
product_requestDescription String,
product_requestType String,
product_routingTarget String,
product_routingType String,
product_servicecode String,
product_sku String,
product_softwareType String,
product_storage String,
product_storageClass String,
product_storageMedia String,
product_technicalSupport String,
product_tenancy String,
product_thirdpartySoftwareSupport String,
product_toLocation String,
product_toLocationType String,
product_training String,
product_transferType String,
product_usageFamily String,
product_usagetype String,
product_vcpu String,
product_version String,
product_volumeType String,
product_whoCanOpenCases String,
pricing_LeaseContractLength String,
pricing_OfferingClass String,
pricing_PurchaseOption String,
pricing_publicOnDemandCost String,
pricing_publicOnDemandRate String,
pricing_term String,
pricing_unit String,
reservation_AvailabilityZone String,
reservation_NormalizedUnitsPerReservation String,
reservation_NumberOfReservations String,
reservation_ReservationARN String,
reservation_TotalReservedNormalizedUnits String,
reservation_TotalReservedUnits String,
reservation_UnitsPerReservation String,
resourceTags_userName String,
resourceTags_usercostcategory String  


)
    ROW FORMAT DELIMITED
      FIELDS TERMINATED BY ','
      ESCAPED BY '\\'
      LINES TERMINATED BY '\n'

STORED AS TEXTFILE
    LOCATION 's3://<<your bucket name>>';

Once you’ve successfully executed the command, you should see a new table named “cost_and_usage” with the below properties. Now we’re ready to start executing queries and running analysis!

Start with Looker and connect to Athena

Setting up Looker is a quick process, and you can try it out for free here (or download from Amazon Marketplace). It takes just a few seconds to connect Looker to your Athena database, and Looker comes with a host of pre-built data models and dashboards to make analysis of your cost and usage data simple and intuitive. After you’re connected, you can use the Looker UI to run whatever analysis you’d like. Looker translates this UI to optimized SQL, so any user can execute and visualize queries for true self-service analytics.

Major cost saving levers

Now that the data pipeline is configured, you can dive into the most popular use cases for cost savings. In this post, I focus on:

  • Purchasing Reserved Instances vs. On-Demand Instances
  • Data transfer costs
  • Allocating costs over users or other Attributes (denoted with resource tags)

On-Demand, Spot, and Reserved Instances

Purchasing Reserved Instances vs On-Demand Instances is arguably going to be the biggest cost lever for heavy AWS users (Reserved Instances run up to 75% cheaper!). AWS offers three options for purchasing instances:

  • On-Demand—Pay as you use.
  • Spot (variable cost)—Bid on spare Amazon EC2 computing capacity.
  • Reserved Instances—Pay for an instance for a specific, allotted period of time.

When purchasing a Reserved Instance, you can also choose to pay all-upfront, partial-upfront, or monthly. The more you pay upfront, the greater the discount.

If your company has been using AWS for some time now, you should have a good sense of your overall instance usage on a per-month or per-day basis. Rather than paying for these instances On-Demand, you should try to forecast the number of instances you’ll need, and reserve them with upfront payments.

The total amount of usage with Reserved Instances versus overall usage with all instances is called your coverage ratio. It’s important not to confuse your coverage ratio with your Reserved Instance utilization. Utilization represents the amount of reserved hours that were actually used. Don’t worry about exceeding capacity, you can still set up Auto Scaling preferences so that more instances get added whenever your coverage or utilization crosses a certain threshold (we often see a target of 80% for both coverage and utilization among savvy customers).

Calculating the reserved costs and coverage can be a bit tricky with the level of granularity provided by the cost and usage report. The following query shows your total cost over the last 6 months, broken out by Reserved Instance vs other instance usage. You can substitute the cost field for usage if you’d prefer. Please note that you should only have data for the time period after the cost and usage report has been enabled (though you can opt for up to 3 months of historical data by contacting your AWS Account Executive). If you’re just getting started, this query will only show a few days.

 

SELECT 
	DATE_FORMAT(from_iso8601_timestamp(cost_and_usage.lineitem_usagestartdate),'%Y-%m') AS "cost_and_usage.usage_start_month",
	COALESCE(SUM(cost_and_usage.lineitem_unblendedcost ), 0) AS "cost_and_usage.total_unblended_cost",
	COALESCE(SUM(CASE WHEN (CASE
         WHEN cost_and_usage.lineitem_lineitemtype = 'DiscountedUsage' THEN 'RI Line Item'
         WHEN cost_and_usage.lineitem_lineitemtype = 'RIFee' THEN 'RI Line Item'
         WHEN cost_and_usage.lineitem_lineitemtype = 'Fee' THEN 'RI Line Item'
         ELSE 'Non RI Line Item'
        END = 'RI Line Item') THEN cost_and_usage.lineitem_unblendedcost  ELSE NULL END), 0) AS "cost_and_usage.total_reserved_unblended_cost",
	1.0 * (COALESCE(SUM(CASE WHEN (CASE
         WHEN cost_and_usage.lineitem_lineitemtype = 'DiscountedUsage' THEN 'RI Line Item'
         WHEN cost_and_usage.lineitem_lineitemtype = 'RIFee' THEN 'RI Line Item'
         WHEN cost_and_usage.lineitem_lineitemtype = 'Fee' THEN 'RI Line Item'
         ELSE 'Non RI Line Item'
        END = 'RI Line Item') THEN cost_and_usage.lineitem_unblendedcost  ELSE NULL END), 0)) / NULLIF((COALESCE(SUM(cost_and_usage.lineitem_unblendedcost ), 0)),0)  AS "cost_and_usage.percent_spend_on_ris",
	COALESCE(SUM(CASE WHEN (CASE
         WHEN cost_and_usage.lineitem_lineitemtype = 'DiscountedUsage' THEN 'RI Line Item'
         WHEN cost_and_usage.lineitem_lineitemtype = 'RIFee' THEN 'RI Line Item'
         WHEN cost_and_usage.lineitem_lineitemtype = 'Fee' THEN 'RI Line Item'
         ELSE 'Non RI Line Item'
        END = 'Non RI Line Item') THEN cost_and_usage.lineitem_unblendedcost  ELSE NULL END), 0) AS "cost_and_usage.total_non_reserved_unblended_cost",
	1.0 * (COALESCE(SUM(CASE WHEN (CASE
         WHEN cost_and_usage.lineitem_lineitemtype = 'DiscountedUsage' THEN 'RI Line Item'
         WHEN cost_and_usage.lineitem_lineitemtype = 'RIFee' THEN 'RI Line Item'
         WHEN cost_and_usage.lineitem_lineitemtype = 'Fee' THEN 'RI Line Item'
         ELSE 'Non RI Line Item'
        END = 'Non RI Line Item') THEN cost_and_usage.lineitem_unblendedcost  ELSE NULL END), 0)) / NULLIF((COALESCE(SUM(cost_and_usage.lineitem_unblendedcost ), 0)),0)  AS "cost_and_usage.percent_spend_on_non_ris"
FROM aws_optimizer.cost_and_usage  AS cost_and_usage

WHERE 
	(((from_iso8601_timestamp(cost_and_usage.lineitem_usagestartdate)) >= ((DATE_ADD('month', -5, DATE_TRUNC('MONTH', CAST(NOW() AS DATE))))) AND (from_iso8601_timestamp(cost_and_usage.lineitem_usagestartdate)) < ((DATE_ADD('month', 6, DATE_ADD('month', -5, DATE_TRUNC('MONTH', CAST(NOW() AS DATE))))))))
GROUP BY 1
ORDER BY 2 DESC
LIMIT 500

The resulting table should look something like the image below (I’m surfacing tables through Looker, though the same table would result from querying via command line or any other interface).

With a BI tool, you can create dashboards for easy reference and monitoring. New data is dumped into S3 every few hours, so your dashboards can update several times per day.

It’s an iterative process to understand the appropriate number of Reserved Instances needed to meet your business needs. After you’ve properly integrated Reserved Instances into your purchasing patterns, the savings can be significant. If your coverage is consistently below 70%, you should seriously consider adjusting your purchase types and opting for more Reserved instances.

Data transfer costs

One of the great things about AWS data storage is that it’s incredibly cheap. Most charges often come from moving and processing that data. There are several different prices for transferring data, broken out largely by transfers between regions and availability zones. Transfers between regions are the most costly, followed by transfers between Availability Zones. Transfers within the same region and same availability zone are free unless using elastic or public IP addresses, in which case there is a cost. You can find more detailed information in the AWS Pricing Docs. With this in mind, there are several simple strategies for helping reduce costs.

First, since costs increase when transferring data between regions, it’s wise to ensure that as many services as possible reside within the same region. The more you can localize services to one specific region, the lower your costs will be.

Second, you should maximize the data you’re routing directly within AWS services and IP addresses. Transfers out to the open internet are the most costly and least performant mechanisms of data transfers, so it’s best to keep transfers within AWS services.

Lastly, data transfers between private IP addresses are cheaper than between elastic or public IP addresses, so utilizing private IP addresses as much as possible is the most cost-effective strategy.

The following query provides a table depicting the total costs for each AWS product, broken out transfer cost type. Substitute the “lineitem_productcode” field in the query to segment the costs by any other attribute. If you notice any unusually high spikes in cost, you’ll need to dig deeper to understand what’s driving that spike: location, volume, and so on. Drill down into specific costs by including “product_usagetype” and “product_transfertype” in your query to identify the types of transfer costs that are driving up your bill.

SELECT 
	cost_and_usage.lineitem_productcode  AS "cost_and_usage.product_code",
	COALESCE(SUM(cost_and_usage.lineitem_unblendedcost), 0) AS "cost_and_usage.total_unblended_cost",
	COALESCE(SUM(CASE WHEN REGEXP_LIKE(cost_and_usage.product_usagetype, 'DataTransfer')    THEN cost_and_usage.lineitem_unblendedcost  ELSE NULL END), 0) AS "cost_and_usage.total_data_transfer_cost",
	COALESCE(SUM(CASE WHEN REGEXP_LIKE(cost_and_usage.product_usagetype, 'DataTransfer-In')    THEN cost_and_usage.lineitem_unblendedcost  ELSE NULL END), 0) AS "cost_and_usage.total_inbound_data_transfer_cost",
	COALESCE(SUM(CASE WHEN REGEXP_LIKE(cost_and_usage.product_usagetype, 'DataTransfer-Out')    THEN cost_and_usage.lineitem_unblendedcost  ELSE NULL END), 0) AS "cost_and_usage.total_outbound_data_transfer_cost"
FROM aws_optimizer.cost_and_usage  AS cost_and_usage

WHERE 
	(((from_iso8601_timestamp(cost_and_usage.lineitem_usagestartdate)) >= ((DATE_ADD('month', -5, DATE_TRUNC('MONTH', CAST(NOW() AS DATE))))) AND (from_iso8601_timestamp(cost_and_usage.lineitem_usagestartdate)) < ((DATE_ADD('month', 6, DATE_ADD('month', -5, DATE_TRUNC('MONTH', CAST(NOW() AS DATE))))))))
GROUP BY 1
ORDER BY 2 DESC
LIMIT 500

When moving between regions or over the open web, many data transfer costs also include the origin and destination location of the data movement. Using a BI tool with mapping capabilities, you can get a nice visual of data flows. The point at the center of the map is used to represent external data flows over the open internet.

Analysis by tags

AWS provides the option to apply custom tags to individual resources, so you can allocate costs over whatever customized segment makes the most sense for your business. For a SaaS company that hosts software for customers on AWS, maybe you’d want to tag the size of each customer. The following query uses custom tags to display the reserved, data transfer, and total cost for each AWS service, broken out by tag categories, over the last 6 months. You’ll want to substitute the cost_and_usage.resourcetags_customersegment and cost_and_usage.customer_segment with the name of your customer field.

 

SELECT * FROM (
SELECT *, DENSE_RANK() OVER (ORDER BY z___min_rank) as z___pivot_row_rank, RANK() OVER (PARTITION BY z__pivot_col_rank ORDER BY z___min_rank) as z__pivot_col_ordering FROM (
SELECT *, MIN(z___rank) OVER (PARTITION BY "cost_and_usage.product_code") as z___min_rank FROM (
SELECT *, RANK() OVER (ORDER BY CASE WHEN z__pivot_col_rank=1 THEN (CASE WHEN "cost_and_usage.total_unblended_cost" IS NOT NULL THEN 0 ELSE 1 END) ELSE 2 END, CASE WHEN z__pivot_col_rank=1 THEN "cost_and_usage.total_unblended_cost" ELSE NULL END DESC, "cost_and_usage.total_unblended_cost" DESC, z__pivot_col_rank, "cost_and_usage.product_code") AS z___rank FROM (
SELECT *, DENSE_RANK() OVER (ORDER BY CASE WHEN "cost_and_usage.customer_segment" IS NULL THEN 1 ELSE 0 END, "cost_and_usage.customer_segment") AS z__pivot_col_rank FROM (
SELECT 
	cost_and_usage.lineitem_productcode  AS "cost_and_usage.product_code",
	cost_and_usage.resourcetags_customersegment  AS "cost_and_usage.customer_segment",
	COALESCE(SUM(cost_and_usage.lineitem_unblendedcost ), 0) AS "cost_and_usage.total_unblended_cost",
	1.0 * (COALESCE(SUM(CASE WHEN REGEXP_LIKE(cost_and_usage.product_usagetype, 'DataTransfer')    THEN cost_and_usage.lineitem_unblendedcost  ELSE NULL END), 0)) / NULLIF((COALESCE(SUM(cost_and_usage.lineitem_unblendedcost ), 0)),0)  AS "cost_and_usage.percent_spend_data_transfers_unblended",
	1.0 * (COALESCE(SUM(CASE WHEN (CASE
         WHEN cost_and_usage.lineitem_lineitemtype = 'DiscountedUsage' THEN 'RI Line Item'
         WHEN cost_and_usage.lineitem_lineitemtype = 'RIFee' THEN 'RI Line Item'
         WHEN cost_and_usage.lineitem_lineitemtype = 'Fee' THEN 'RI Line Item'
         ELSE 'Non RI Line Item'
        END = 'Non RI Line Item') THEN cost_and_usage.lineitem_unblendedcost  ELSE NULL END), 0)) / NULLIF((COALESCE(SUM(cost_and_usage.lineitem_unblendedcost ), 0)),0)  AS "cost_and_usage.unblended_percent_spend_on_ris"
FROM aws_optimizer.cost_and_usage_raw  AS cost_and_usage

WHERE 
	(((from_iso8601_timestamp(cost_and_usage.lineitem_usagestartdate)) >= ((DATE_ADD('month', -5, DATE_TRUNC('MONTH', CAST(NOW() AS DATE))))) AND (from_iso8601_timestamp(cost_and_usage.lineitem_usagestartdate)) < ((DATE_ADD('month', 6, DATE_ADD('month', -5, DATE_TRUNC('MONTH', CAST(NOW() AS DATE))))))))
GROUP BY 1,2) ww
) bb WHERE z__pivot_col_rank <= 16384
) aa
) xx
) zz
 WHERE z___pivot_row_rank <= 500 OR z__pivot_col_ordering = 1 ORDER BY z___pivot_row_rank

The resulting table in this example looks like the results below. In this example, you can tell that we’re making poor use of Reserved Instances because they represent such a small portion of our overall costs.

Again, using a BI tool to visualize these costs and trends over time makes the analysis much easier to consume and take action on.

Summary

Saving costs on your AWS spend is always an iterative, ongoing process. Hopefully with these queries alone, you can start to understand your spending patterns and identify opportunities for savings. However, this is just a peek into the many opportunities available through analysis of the Cost and Usage report. Each company is different, with unique needs and usage patterns. To achieve maximum cost savings, we encourage you to set up an analytics environment that enables your team to explore all potential cuts and slices of your usage data, whenever it’s necessary. Exploring different trends and spikes across regions, services, user types, etc. helps you gain comprehensive understanding of your major cost levers and consistently implement new cost reduction strategies.

Note that all of the queries and analysis provided in this post were generated using the Looker data platform. If you’re already a Looker customer, you can get all of this analysis, additional pre-configured dashboards, and much more using Looker Blocks for AWS.


About the Author

Dillon Morrison leads the Platform Ecosystem at Looker. He enjoys exploring new technologies and architecting the most efficient data solutions for the business needs of his company and their customers. In his spare time, you’ll find Dillon rock climbing in the Bay Area or nose deep in the docs of the latest AWS product release at his favorite cafe (“Arlequin in SF is unbeatable!”).

 

 

 

Curb Your Enthusiasm on Those HBO Leaks

Post Syndicated from Ernesto original https://torrentfreak.com/curb-your-enthusiasm-on-those-hbo-leaks-170814/

Late July, news broke that a hacker, or hackers, had compromised the network of the American cable and television network HBO.

Those responsible contacted reporters, informing them about the prominent breach, and leaked files surfaced on the dedicated website Winter-leak.com.

The website wasn’t around for long, but last week the hackers reached out to the press again with a curated batch of new leaks shared through Mega.nz. Among other things, it contained more Game of Thrones spoilers, marketing plans, and other confidential HBO files.

Fast forward another week and there’s yet another freshly curated batch of leaks. This time it includes episodes of the highly anticipated return of ‘Curb Your Enthusiasm,’ which officially airs in October, as well as episodes from “Barry,” “Insecure” and “The Deuce,” AP reports.

These shows are part of the treasure trove of 1.5 terabytes that was taken from HBO. These and several other titles were already teased last week in a screenshot the hackers released to the press.

There’s no reason to doubt that the leaks are real, but thus far they haven’t been widely distributed. It appears that the various journalists who received the latest batch of Mega.nz links are not very eager to post them in public.

TorrentFreak scoured popular torrent sites and streaming portals for public copies of the new Curb Your Enthusiasm episodes and came up empty-handed. And we’re certainly not the only ones having trouble spotting the leaks in public.

“I searched around a lot a few hours ago and couldn’t find anything,” one Curb Your Enthusiasm watcher commented on Reddit. “Why can’t these hackers be courteous and place links?” another added.

This is quite different from the leaked episode of Game of Thrones that came out before its official release two weeks ago. That leak was not related to the HBO hack, but before the news broke in the mainstream press, thousands of copies were already available on pirate sites.

HBO, meanwhile, appears to have had enough of the continued enthusiasm the hacker is managing to generate in the press.

“We are not in communication with the hacker and we’re not going to comment every time a new piece of information is released,” a company spokesperson said.

“It has been widely reported that there was a cyber incident at HBO. The hacker may continue to drop bits and pieces of stolen information in an attempt to generate media attention. That’s a game we’re not going to participate in.”

As for the Curb Your Enthusiasm fans who were hoping for an early preview of the new season. They may have to, well… you know. For now at least.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

New Premier League Blocking Disrupts Pirate IPTV Providers

Post Syndicated from Andy original https://torrentfreak.com/new-premier-league-blocking-disrupts-pirate-iptv-providers-170814/

Top tier football in the UK is handled by the English Premier League (EPL) and broadcasting partners Sky and BT Sport. All face considerable problems with Internet piracy, through free web or Kodi-based streaming and premium IPTV feeds.

To mitigate the threat, earlier this year the Premier League obtained a unique High Court injunction which required ISPs such as Sky, BT, and Virgin to block ‘pirate’ football streams in real-time.

Although the success of the program was initially up for debate, the EPL reported it was able to block 5,000 server IP addresses that were streaming its content. When that temporary injunction ran out, the EPL went back to court for a new one, valid for the season that began this past weekend. There are signs the EPL may have upped its game.

As soon as the matches began on Saturday, issues were reported at several of the more prominent IPTV providers. Within minutes of the match streams going live, subscribers to affected services were met with black screens, causing anger and frustration. While some clearly knew that action was on the cards, relatively few had an effective plan in place.

One provider, which targets subscribers in the UK, scrambled to obtain new domain names, thinking that the existing domains had been placed on some kind of Premier League blacklist. While that may have indeed been the case, making a service more obscure in that sense was never going to outwit the systems deployed by the anti-piracy outfits involved.

Indeed, the provider in question was subjected to much chaos over both Saturday and Sunday, since it’s clear that large numbers of subscribers had absolutely no idea what was going on. Even if they understood that the EPL was blocking, the change of domain flat-footed the rest. The subsequent customer service chaos was not a pretty sight but would’ve been a pleasure for the EPL to behold.

An interesting side effect of this EPL action is that even if IPTV subscribers don’t care about football, many were affected this past weekend anyway.

TF is aware of at least three services (there are probably many more) that couldn’t service their UK customers with any other channels whatsoever while the Premier League games were being aired. This suggests that the IP addresses hit by the EPL and blocked by local ISPs belonged to the same servers carrying the rest of the content offered by the IPTV providers.

When the High Court handed down its original injunction it accepted that some non-Premier League content could be blocked at the same time but since that “consists almost exclusively of [infringing] commercial broadcast content such as other sports, films, and television programs,” there was little concern over collateral damage.

So the big question now is what can IPTV providers and/or subscribers do to tackle the threat?

The first interesting thing to note is not all of the big providers were affected this past weekend, so for many customers the matches passed by as normal. It isn’t clear whether EPL simply didn’t have all of the providers on the list or whether steps were taken to mitigate the threat, but that was certainly the case in a handful of cases.

Information passed to TF shows that at least a small number of providers were not only waiting for the EPL action but actually had a backup plan in place. This appears to have resulted in a minimum of disruption for their customers, something that will prove of interest to the many frustrated subscribers looking for a new service this morning.

While the past few days have been somewhat chaotic, other issues have been muddying the waters somewhat.

TF has learned that at least two, maybe three suppliers, were subjected to DDoS attacks around the time the matches were due to air. It seems unlikely that the EPL has been given permission to carry out such an attack but since the High Court injunction is secret in every way that describes its anti-piracy methods, that will remain a suspicion. In the meantime, rival IPTV services remain possible suspects.

Also, a major IPTV stream ‘wholesaler’ is reported to have had technical issues on Saturday, which affected its ability to serve lower-tier providers. Whether that was also linked to the Premier League action is unknown and TF couldn’t find any source willing to talk about the provider in any detail.

So, sports fans who rely on IPTV for their fix are wondering how things will pan out later this week. If this last weekend is anything to go by, disruption is guaranteed, but it will be less of a surprise given the problems of the last few days. While some don’t foresee huge problems, several providers are already advising customers that VPNs will be necessary.

An IPTV provider suggesting the use of VPNs

While a VPN will indeed solve the problem in most cases, for many subscribers that will amount to an additional expense, not to mention more time spent learning about VPNs, what they can do, and how they can be setup on the hardware they’re using for IPTV.

For users on Android devices running IPTV apps or Kodi-type setups, VPNs are both easy to install and use. However, Mag Box STB users cannot run a VPN directly on the device, meaning that they’ll need either a home router that can run a VPN or a smaller ‘travel’ type router with OpenVPN capabilities to use as a go-between.

Either way, costs are beginning to creep up, if IPTV providers can’t deal with the EPL’s blocking efforts. That makes the new cheaper football packages offered by various providers that little bit more attractive. But that was probably the plan all along.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

Piracy Narrative Isn’t About Ethics Anymore, It’s About “Danger”

Post Syndicated from Andy original https://torrentfreak.com/piracy-narrative-isnt-about-ethics-anymore-its-about-danger-170812/

Over the years there have been almost endless attempts to stop people from accessing copyright-infringing content online. Campaigns have come and gone and almost two decades later the battle is still ongoing.

Early on, when panic enveloped the music industry, the campaigns centered around people getting sued. Grabbing music online for free could be costly, the industry warned, while parading the heads of a few victims on pikes for the world to see.

Periodically, however, the aim has been to appeal to the public’s better nature. The idea is that people essentially want to do the ‘right thing’, so once they understand that largely hard-working Americans are losing their livelihoods, people will stop downloading from The Pirate Bay. For some, this probably had the desired effect but millions of people are still getting their fixes for free, so the job isn’t finished yet.

In more recent years, notably since the MPAA and RIAA had their eyes blacked in the wake of SOPA, the tone has shifted. In addition to educating the public, torrent and streaming sites are increasingly being painted as enemies of the public they claim to serve.

Several studies, largely carried out on behalf of the Digital Citizens Alliance (DCA), have claimed that pirate sites are hotbeds of malware, baiting consumers in with tasty pirate booty only to offload trojans, viruses, and God-knows-what. These reports have been ostensibly published as independent public interest documents but this week an advisor to the DCA suggested a deeper interest for the industry.

Hemanshu Nigam is a former federal prosecutor, ex-Chief Security Officer for News Corp and Fox Interactive Media, and former VP Worldwide Internet Enforcement at the MPAA. In an interview with Deadline this week, he spoke about alleged links between pirate sites and malware distributors. He also indicated that warning people about the dangers of pirate sites has become Hollywood’s latest anti-piracy strategy.

“The industry narrative has changed. When I was at the MPAA, we would tell people that stealing content is wrong and young people would say, yeah, whatever, you guys make a lot of money, too bad,” he told the publication.

“It has gone from an ethical discussion to a dangerous one. Now, your parents’ bank account can be raided, your teenage daughter can be spied on in her bedroom and extorted with the footage, or your computer can be locked up along with everything in it and held for ransom.”

Nigam’s stance isn’t really a surprise since he’s currently working for the Digital Citizens Alliance as an advisor. In turn, the Alliance is at least partly financed by the MPAA. There’s no suggestion whatsoever that Nigam is involved in any propaganda effort, but recent signs suggest that the DCA’s work in malware awareness is more about directing people away from pirate sites than protecting them from the alleged dangers within.

That being said and despite the bias, it’s still worth giving experts like Nigam an opportunity to speak. Largely thanks to industry efforts with brands, pirate sites are increasingly being forced to display lower-tier ads, which can be problematic. On top, some sites’ policies mean they don’t deserve any visitors at all.

In the Deadline piece, however, Nigam alleges that hackers have previously reached out to pirate websites offering $200 to $5000 per day “depending on the size of the pirate website” to have the site infect users with malware. If true, that’s a serious situation and people who would ordinarily use ‘pirate’ sites would definitely appreciate the details.

For example, to which sites did hackers make this offer and, crucially, which sites turned down the offer and which ones accepted?

It’s important to remember that pirates are just another type of consumer and they would boycott sites in a heartbeat if they discovered they’d been paid to infect them with malware. But, as usual, the claims are extremely light in detail. Instead, there’s simply a blanket warning to stay away from all unauthorized sites, which isn’t particularly helpful.

In some cases, of course, operational security will prevent some details coming to light but without these, people who don’t get infected on a ‘pirate’ site (the vast majority) simply won’t believe the allegations. As the author of the Deadline piece pointed out, it’s a bit like Reefer Madness all over again.

The point here is that without hard independent evidence to back up these claims, with reports listing sites alongside the malware they’ve supposed to have spread and when, few people will respond to perceived scaremongering. Free content trumps a few distant worries almost every time, whether that involves malware or the threat of a lawsuit.

It’ll be up to the DCA and their MPAA paymasters to consider whether the approach is working but thus far, not even having government heavyweights on board has helped.

Earlier this year the DCA launched a video campaign, enrolling 15 attorney generals to publish their own anti-piracy PSAs on YouTube. Thus far, interest has been minimal, to say the least.

At the time of writing the 15 PSAs have 3,986 views in total, with 2,441 of those contributed by a single video contributed by Wisconsin Attorney General Brad Schimel. Despite the relative success, even that got slammed with 2 upvotes and 127 downvotes.

A few of the other videos have a couple of hundred views each but more than half have less than 70. Perhaps most worryingly for the DCA, apart from the Schimel PSA, none have any upvotes at all, only down. It’s unclear who the viewers were but it seems reasonable to conclude they weren’t entertained.

The bottom line is nobody likes malware or having their banking details stolen but yet again, people who claim to have the public interest at heart aren’t actually making a difference on the ground. It could be argued that groups advocating online safety should be publishing guides on how to stay protected on the Internet period, not merely advising people to stay away from certain sites.

But of course, that wouldn’t achieve the goals of the MPAA Digital Citizens Alliance.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

Internet Archive Blocked in 2,650 Site Anti-Piracy Sweep

Post Syndicated from Andy original https://torrentfreak.com/internet-archive-blocked-in-2650-site-anti-piracy-sweep-170810/

Reports of sites becoming mysteriously inaccessible in India have been a regular occurance over the past several years. In many cases, sites simply stop functioning, leaving users wondering whether sites are actually down or whether there’s a technical issue.

Due to their increasing prevalence, fingers are often pointed at so-called ‘John Doe’ orders, which are handed down by the court to prevent Internet piracy. Often sweeping in nature (and in some cases pre-emptive rather than preventative), these injunctions have been known to block access to both file-sharing platforms and innocent bystanders.

Earlier this week (and again for no apparent reason), the world renowned Internet Archive was rendered inaccessible to millions of users in India. The platform, which is considered by many to be one of the Internet’s most valued resources, hosts more than 15 petabytes of data, a figure which grows on a daily basis. Yet despite numerous requests for information, none was forthcoming from authorities.

The ‘blocked’ message seen by users accessing Archive.org

Quoted by local news outlet Medianama, Chris Butler, Office Manager at the Internet Archive, said that their attempts to contact the Indian Department of Telecom (DoT) and the Ministry of Electronics and Information Technology (Meity) had proven fruitless.

Noting that site had previously been blocked in India, Butler said they were no clearer on the reasons why the same kind of action had seemingly been taken this week.

“We have no information about why a block would have been implemented,” he said. “Obviously, we are disappointed and concerned by this situation and are very eager to understand why it’s happening and see full access restored to archive.org.”

Now, however, the mystery has been solved. The BBC says a local government agency provided a copy of a court order obtained by two Bollywood production companies who are attempting to slow down piracy of their films in India.

Issued by a local judge, the sweeping order compels local ISPs to block access to 2,650 mainly file-sharing websites, including The Pirate Bay, RARBG, the revived KickassTorrents, and hundreds of other ‘usual suspects’. However, it also includes the URL for the Internet Archive, hence the problems with accessibility this week.

The injunction, which appears to be another John Doe order as previously suspected, was granted by the High Court of the Judicature at Madras on August 2, 2017. Two film productions companies – Prakash Jah Productions and Red Chillies Entertainment – obtained the order to protect their films Lipstick Under My Burkha and Jab Harry Met Sejal.

While India-based visitors to blocked resources are often greeted with a message saying that domains have been blocked at the orders of the Department of Telecommunications, these pages never give a reason why.

This always leads to confusion, with news outlets having to pressure local government agencies to discover the reason behind the blockades. In the interests of transparency, providing a link to a copy of a relevant court order would probably benefit all involved.

A few hours ago, the Internet Archive published a statement questioning the process undertaken before the court order was handed down.

“Is the Court aware of and did it consider the fact that the Internet Archive has a well-established and standard procedure for rights holders to submit take down requests and processes them expeditiously?” the platform said.

“We find several instances of take down requests submitted for one of the plaintiffs, Red Chillies Entertainments, throughout the past year, each of which were processed and responded to promptly.

“After a preliminary review, we find no instance of our having been contacted by anyone at all about these films. Is there a specific claim that someone posted these films to archive.org? If so, we’d be eager to address it directly with the claimant.”

But while the Internet Archive appears to be the highest profile collateral damage following the ISP blocks, it isn’t the only victim. Now that the court orders have become available (1,2), it’s clear that other non-pirate entities have also been affected including news site WN.com, website hosting service Weebly, and French ISP Free.fr.

Also, in a sign that sites aren’t being checked to see if they host the movies in question, one of the orders demands that former torrent index BitSnoop is blocked. The site shut down earlier this year. The same is true for Shaanig.org.

This is not the first time that the Internet Archive has been blocked in India. In 2014/2015, Archive.org was rendered inaccessible after it was accused of hosting extremist material. In common with Google, the site copies and stores huge amounts of data, much of it in automated processes. This can leave it exposed to these kinds of accusations.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

The CNC Wood Burner turning heads (and wood, obviously)

Post Syndicated from Alex Bate original https://www.raspberrypi.org/blog/cnc-wood-burner/

Why stick to conventional laser cutters or CNC machines for creating images on wood, when you can build a device to do the job that is a beautiful piece of art in itself? Mechanical and Computer Science student and Imgur user Tucker Shannon has created a wonderful-looking CNC Wood Burner using a Raspberry Pi and stepper motors. His project has a great vinyl-turntable-like design.

Raspberry Pi CNC Wood Burner

Tucker’s somewhat hypnotic build burns images into wood using a Raspberry Pi and stepper motors
GIF c/o Tucker Shannon

A CNC Wood Burner?

Sure! Why not? Tucker had already put the knowledge he acquired while studying at Oregon State University to good use by catching a bike thief in action with the help of a Raspberry Pi. Thus it’s obvious he has the skills he needed to incorporate our little computer into a project. Moreover, his Skittles portrait of Bill Nye is evidence of his artistic flare, so it’s not surprising that he wanted to make something a little different, and pretty, using code.

Tucker Shannon

“Bill Nye, the Skittles Guy”
Image c/o Tucker Shannon

With an idea in mind and sketches drawn, Tucker first considered using an old record player as the base of his build. Having a rotating deck and arm already in place would have made building his project easier. However, he reports on Imgur:

I thought about that! I couldn’t find any at local thrift shops though. Apparently, they’ve become pretty popular…

We can’t disagree with him. Since his search was unsuccessful, Tucker ended up creating the CNC Wood Burner from scratch.

Raspberry Pi CNC Wood Burner

Concept designs
Image c/o Tucker Shannon

Taking into consideration the lumps and bumps of the wood he would be using as a ‘canvas’, Tucker decided to incorporate a pivot to allow the arm to move smoothly over the rough surface.

The code for the make is currently in ‘spaghetti form’, though Tucker is set to release it, as well as full instructions for the build, in the near future.

The build

Tucker laser-cut the pieces for the wood burner’s box and gear out of birch and pine wood. As the motors require 12v power, the standard Raspberry Pi supply wasn’t going to be enough. Therefore, Tucker scavenged for old computer parts , and ended up rescuing a PSU (power supply unit). He then fitted the PSU and the Raspberry Pi within the box.

Raspberry Pi CNC Wood Burner

The cannibalised PSU, stepper motor controller, and Raspberry Pi fit nicely into Tucker’s handmade pine box.
Image c/o Tucker Shannon

Next, he got to work building runners for the stepper motor controlling the position of the ‘pen thing’ that would scorch the image into the wood.

Raspberry Pi CNC Wood Burner

Initial tests on paper help to align the pen
Image c/o Tucker Shannon

After a few test runs using paper, the CNC Wood Burner was good to go!

The results

Tucker has used his CNC Wood Burner to create some wonderful pieces of art. The few examples he’s shared on Imgur have impressed us with their precision. We’re looking forward to seeing what else he is going to make with it!

Raspberry Pi CNC Wood Burner

The build burns wonderfully clean-lined images into wood
Image c/o Tucker Shannon

Your turn

Image replication using Raspberry Pis and stepper motors isn’t a new thing – though doing it using a wood-burning device may be! We’ve seen some great builds in which makers set up motors and a marker pen to create massive works of art. Are you one of those makers? Or have you been planning a build similar to Tucker’s project, possibly with a new twist?

Share your project with us below, whether it is complete or still merely sketches in a notebook. We’d love to see what you’re getting up to!

The post The CNC Wood Burner turning heads (and wood, obviously) appeared first on Raspberry Pi.

TV Box Seller Emails Sky TV Bosses With ‘Pirate’ Offer, Gets Sued for $1m

Post Syndicated from Andy original https://torrentfreak.com/tv-box-seller-emails-sky-tv-bosses-with-pirate-offer-gets-sued-for-1m-170804/

After relatively quiet treatment in the media, last year press in New Zealand began reporting on the booming ‘pirate’ set-top box business sweeping the world.

Often based around legal Kodi software boosted with third-party addons, the devices are known for providing free movies, TV shows, and sports.

Last November, ‘My Box NZ’ owner Krish Reddy, who said he would take on Sky in its own backyard with his custom streaming boxes, hit the headlines. The 27-year-old told NZHerald that “it seemed like a great idea so we decided to do it ourselves.”

The boxes offered some local free-to-air channels but also the all-important premium offerings from Sky, including Sky Movies and Sky Sports, an expensive proposition for an official subscriber.

“Why pay $80 minimum per month for Sky when for one payment you can have it free for good?” Reddy’s advertising said.

Reddy was confident in the abilities of his product but was also confident he wasn’t breaking the law.

“I don’t see why [Sky] would contact me but if they do contact me and … if there’s something of theirs that they feel I’ve unlawfully taken then yeah … but as it stands I don’t [have any concerns],” he told the Herald.

As things moved on, Reddy’s business really took off. He admitted to having sold 8,000 of the devices and then April this year, Sky appeared to ruh out of patience. In a letter from its lawyers, the pay TV company said Reddy’s devices breached copyright law and the Fair Trading Act. Reddy responded by calling the TV giant “a playground bully” and denied again that he was breaking the law.

“From a legal perspective, what we do is completely within the law. We advertise Sky television channels being available through our website and social media platforms as these are available via streams which you can find through My Box,” he said.

“The content is already available, I’m not going out there and bringing the content so how am I infringing the copyright… the content is already there, if someone uses the box to search for the content, that’s what it is.”

Stuff reports that the initial compensation demand from Sky against Reddy’s company My Box runs to NZD$1.4m (US$1m), an amount that could “rise by millions” by the time a judgment is reached.

“They have given us until September 24 to respond. We are not going to sit and take it,” Reddy told the publication. “How many people can say they went up against a multimillion dollar giant like Sky?”

And it seems that Reddy is absolutely determined to fight back. Earlier this year he said that his father always encouraged him as a child to seek out the big guy for a fight, something that is now playing out with one of the world’s biggest broadcasters.

“[Sky’s] point of view is they own copyright and I’m destroying the market by giving people content for free. To me it is business; I have got something that is new … that’s competition,” he said.

In Europe, where these kinds of cases have already been tested at the highest level, comments like these would be extremely ill-advised and enough to give any defending lawyer a high temperature, but Reddy really doesn’t seem to care.

In fact, a bulk email he sent out to 50,000 people advertising his product as “being better than Sky”, actually found the inboxes of 50 Sky TV staff and directors. He believes this triggered the legal action from the company.

While Reddy was on Sky’s radar long before the mailshot, the blatancy of his advertising and its targets won’t have helped his case one bit. Sky, for its part, is determined to get a ruling against a large player and Reddy seems the perfect catch.

“Anyone selling these boxes are within our sights. You have got to go after the big fish first,” said Sky spokeswoman Kirsty Way.

No case like this has ever gone to court in New Zealand so it could be important for setting the ground rules on several aspects of copyright law, including the making available right.

In addition to prosecutions, Way told Stuff that it could also be possible to introduce site-blocking laws such as those already in place in Australia and the UK. These would aim to render Kodi-powered devices less effective at providing copyrighted content from unauthorized sources.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

Linux kernel hardeners Grsecurity sue Bruce Perens (Register)

Post Syndicated from corbet original https://lwn.net/Articles/729805/rss

The Register reports
that the developers of the grsecurity patch set have filed a defamation
suit against Bruce Perens. “A legal complaint filed on behalf of
Grsecurity in San Francisco, California, insists the company’s software
complies with the GPLv2. Grsecurity’s agreement, the lawsuit states, only
applies to future patches, which have yet to be developed. ‘There is no
explicit or implicit term, section, or clause in the GPLv2 that is
applicable over future versions or updates of the Patches that have not yet
been developed, created, or released by [Grsecurity],’ the complaint
contends.

EFF: Bassel Khartabil, In Memoriam

Post Syndicated from ris original https://lwn.net/Articles/729644/rss

The Electronic Frontier Foundation reports
that Bassel Khartabil, Syrian open source developer, blogger,
entrepreneur, hackerspace founder, and free culture advocate, was executed
by the Syrian authorities. “Bassel was a central figure in the
global free culture movement, connecting it and promoting it to Syria’s
emerging tech community as it existed before the country was ransacked by
civil war. He co-founded Aiki Lab, Syria’s first hackerspace, in Damascus
in 2010. He was a contributor to Mozilla’s Firefox browser and the Syrian
lead for Creative Commons. His influence went beyond Syria, however: he was
a key attendee at the Middle East’s bloggers’ conferences, and played a
vital role in the negotiations in Doha in 2010 that led to a common
language for discussing fair use and copyright across the Arab-speaking
world.
” (Thanks to Paul Wise)

NSA Collects MS Windows Error Information

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2017/08/nsa_collects_ms.html

Back in 2013, Der Spiegel reported that the NSA intercepts and collects Windows bug reports:

One example of the sheer creativity with which the TAO spies approach their work can be seen in a hacking method they use that exploits the error-proneness of Microsoft’s Windows. Every user of the operating system is familiar with the annoying window that occasionally pops up on screen when an internal problem is detected, an automatic message that prompts the user to report the bug to the manufacturer and to restart the program. These crash reports offer TAO specialists a welcome opportunity to spy on computers.

When TAO selects a computer somewhere in the world as a target and enters its unique identifiers (an IP address, for example) into the corresponding database, intelligence agents are then automatically notified any time the operating system of that computer crashes and its user receives the prompt to report the problem to Microsoft. An internal presentation suggests it is NSA’s powerful XKeyscore spying tool that is used to fish these crash reports out of the massive sea of Internet traffic.

The automated crash reports are a “neat way” to gain “passive access” to a machine, the presentation continues. Passive access means that, initially, only data the computer sends out into the Internet is captured and saved, but the computer itself is not yet manipulated. Still, even this passive access to error messages provides valuable insights into problems with a targeted person’s computer and, thus, information on security holes that might be exploitable for planting malware or spyware on the unwitting victim’s computer.

Although the method appears to have little importance in practical terms, the NSA’s agents still seem to enjoy it because it allows them to have a bit of a laugh at the expense of the Seattle-based software giant. In one internal graphic, they replaced the text of Microsoft’s original error message with one of their own reading, “This information may be intercepted by a foreign sigint system to gather detailed information and better exploit your machine.” (“Sigint” stands for “signals intelligence.”)

The article talks about the (limited) value of this information with regard to specific target computers, but I have another question: how valuable would this database be for finding new zero-day Windows vulnerabilities to exploit? Microsoft won’t have the incentive to examine and fix problems until they happen broadly among its user base. The NSA has a completely different incentive structure.

I don’t remember this being discussed back in 2013.

EDITED TO ADD (8/6): Slashdot thread.

HBO Got Hacked, Game of Thrones Spoilers Surface Online

Post Syndicated from Ernesto original https://torrentfreak.com/hbo-got-hacked-game-of-thrones-spoilers-surface-online-170801/

It appears that yet another large media outlet has fallen victim to a high-profile hack.

After Sony and, indirectly, Netflix, hackers have now compromised the network of the American cable and television network HBO.

Sunday evening a mysterious email was sent to reporters, announcing the prominent breach.

“Hi to all mankind. The greatest leak of cyber space era is happening. What’s its name? Oh I forget to tell. Its HBO and Game of Thrones……!!!!!!” the email read.

While several reports were published, the first by Entertainment Weekly, the actual leaked files were not widely available on the usual pirate sites. However, a few hours ago a website appeared online that claims to hold the ‘treasure trove.’

Winter-leak.com, a reference to the famous Game of Thrones “Winter is Coming” phrase, does indeed list several files that appear to come from HBO.

“In a complicate operation, we successfully penetrated in to the HBO Internal Network, Emails, technical platforms, and database and got precious and confidential stuff that blaze your eyes,” the hacker, or hackers write on their website.

The hackers claim to have 1.5 terabytes of data from the company. So far, previously unreleased episodes of Ballers, Barry, Insecure and Room 104 are featured on the site. However, there are also three separate archives listed, with over a terabyte of data.

Most prominent, perhaps, is a preliminary outline of the fourth episode of the current Game of Thrones season, which will air this coming Sunday.

At TorrentFreak, we always strive to find proof for reported leaks, and from what we’ve seen and gathered, it does indeed appear to be the real deal. The Game of Thrones information, for example, lists a preliminary outline of the fourth episode of season 7, including many spoilers.

As can be seen below, the outline itself is watermarked by the hackers, with the tagline “HBO is falling.”

Perhaps even more unusual, the leak also includes a video, featuring Game of Thrones images, the leaders, and a textual outline of the episode. As with the outline, the videos are available for the third and fourth episode of season 7.

HBO’s chairman and CEO, Richard Plepler, has confirmed that the company’s infrastructure was breached, but didn’t mention what information was accessed. He sent an email to employees a few hours ago, informing them about the “cyber incident.”

“As most of you have probably heard by now, there has been a cyber incident directed at the company which has resulted in some stolen proprietary information, including some of our programming,” he wrote.

“Any intrusion of this nature is obviously disruptive, unsettling, and disturbing for all of us. I can assure you that senior leadership and our extraordinary technology team, along with outside experts, are working round the clock to protect our collective interests.”

The full contents of the leaks have yet to be analyzed. It’s doubtful that any Game of Thrones episodes will leak, but there’s likely to be a lot of confidential information in the copied data, which HBO would otherwise prefer to keep to itself.

HBO has already mentioned that it’s doing everything in its power to prevent the leaks from spreading any further. In addition, they are also working with law enforcement to track down the people responsible.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

Italian ISPs Say New Copyright Amendment Infringes Human Rights

Post Syndicated from Andy original https://torrentfreak.com/italian-isps-say-new-copyright-amendment-infringes-human-rights-170728/

After being spoken of in unfavorable terms by the United States Trade Representative in its Special 301 Reports, Italy achieved a sudden breakthrough in 2014.

“Italy’s removal from the Special 301 List reflects the significant steps the Government of Italy has taken to address the problem of online piracy, and the continued U.S. commitment to meaningful and sustained engagement with our critical partner Italy,” the USTR said in a special announcement.

This praise was in part due to the way Italy promised to deal with online piracy. Instead of legislating to make a piracy crackdown easier, the government handed AGCOM, the Italian Communications Regulatory Authority, the power to deal with infringement based on complaints filed by rightsholders.

Without any need for legal cases or court injunctions, at the end of March 2014, AGCOM was granted the power to have allegedly infringing content removed from sites and to have domains blocked at the ISP level.

Now, just over three years later, AGCOM has been granted even more power. Passed last week, Amendment 1.022 effectively gives AGCOM the power to order sites to not only take allegedly infringing content down but to keep it down permanently, all without intervention from the judiciary.

The decision has provoked a furious response from a body representing the country’s ISPs, which describes the “unconstitutional rules” as a way to protect the economic interests of right holders behind various creative works and live sporting events.

“This measure abolishes procedural safeguards for citizens, imposes interception obligations to Internet providers, and damages consumers by imposing technical measures that will result in increased costs,” the Italian Association of Internet Providers (AIIP) said in a statement.

According to AIIP, it is the judiciary that should have sole power over copyright infringement disputes in Italy. When other bodies such as AGCOM are given control over criminal issues, it represents a violation of both constitutional principles and EU law.

“Any rule that would require Internet Providers to filter and carry out preventive checks – as well as to remove content generated by users without a court order – is in violation of the European Convention on Human Rights, Community legislation on electronic communications services, and case law of the European Court of Justice,” AIIP says.

The ISP body says that AGCOM now possesses discretionary powers that even magistrates do not have, which from a technical perspective includes monitoring, interception, and blocking of user activity, a position that amounts to “gigantic state censorship.”

Only time will tell how the situation pans out but it’s crystal clear that ISPs feel that unlike the views of the copyright industry, their concerns have not been taken into consideration.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

Hackers Use Pirate Sites to Ruin Your Life, State Attorneys General Warn

Post Syndicated from Ernesto original https://torrentfreak.com/hackers-use-pirate-sites-to-ruin-your-life-state-attorneys-general-warn-170727/

In recent years copyright holders have tried many things to dissuade the public from visiting pirate websites.

They often claim that piracy costs the entertainment industry thousands of jobs, for example. Another strategy to is to scare the public at large directly, by pointing out all the ills people may encounter on pirate sites.

The Digital Citizens Alliance (DCA), which has deep ties to the content industries, is a proponent of the latter strategy. The group has released a variety of reports pointing out that pirate sites are a hotbed for malware, identity theft, hacking and other evils.

To add some political weight to this message, the DCA recently helped to launch a new series of public service announcements where a group of 15 State Attorneys General warn the public about these threats.

The participating Attorneys General include Arizona’s Mark Brnovich, Kentucky’s Andy Bashear, Washington DC’s Karl Racine, and Wisconsin’s Brad Schimel, who all repeat the exact same words in their PSAs.

“Nowadays we all have to worry about cybersecurity. Hackers are always looking for new ways to break into our computers. Something as simple as visiting pirate websites can put your computer at risk.”

“Hackers use pirate websites to infect your computer and steal your ID and financial information, or even take over your computer’s camera without you knowing it,” the Attorneys General add.

Organized by the Digital Citizens Alliance, the campaign in question runs on TV and radio in several states and also appears on social media during the summer.

The warnings, while over dramatized, do raise a real concern. There are a lot of pirate sites that have lower-tier advertising, where malware regularly slips through. And some ads lead users to fake websites where people should probably not leave their credit card information.

Variety points out that the Attorneys General are tasked with the goal to keep their citizens safe, so the PSA’s message is certainly fitting.

Still, one has to wonder whether the main driver of these ads is online safety. Could perhaps the interests of the entertainment industry play a role too? It certainly won’t be the first time that State Attorneys General have helped out Hollywood.

Just a few years ago the MPAA secretly pushed Mississippi State Attorney General Jim Hood to revive SOPA-like anti-piracy efforts in the United States. That was part of the MPAA’s “Project Goliath,” which was aimed at “convincing state prosecutors to take up the fight” against Google, under an anti-piracy umbrella.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

Premier League Wins New Stream Blocking Injunction to Fight Piracy

Post Syndicated from Andy original https://torrentfreak.com/premier-league-wins-new-stream-blocking-injunction-to-fight-piracy-170727/

Earlier this year the Premier League obtained a rather special High Court injunction to assist in its fight against illegal football match streaming.

Similar in its aims to earlier blocking orders that targeted torrent sites including The Pirate Bay, the injunction enabled the Premier League to act quickly, forcing local ISPs such as Sky, BT, and Virgin to block football streams in real-time.

Although public results varied, the English Premier League (EPL) reports that under the injunction it was able to block 5,000 server IP addresses that were streaming its content. That appears to have encouraged the organization to apply for another injunction for the upcoming 2017-18 season.

According to a statement published on the EPL site, that has now been granted.

“This blocking order is a game-changer in our efforts to tackle the supply and use of illicit streams of our content,” said Premier League Director of Legal Services, Kevin Plumb.

“It will allow us to quickly and effectively block and disrupt the illegal broadcast of Premier League football via any means, including so called ‘pre-loaded Kodi boxes’.”

Although the details of the new injunction are yet to be published by the High Court, the EPL indicates that the injunction is very similar to the one obtained previously, which targets overseas servers streaming Premier League matches into the UK.

Upon notice from the Premier League, ISPs including Sky, BT, Virgin Media, Plusnet, EE and TalkTalk are required to block IP addresses quickly as matches are being streamed, all without any direct intervention from the court.

“The protection of our copyright, and the investment made by our broadcast partners, is hugely important to the Premier League and the future health of English football,” the Premier League said.

The injunction itself lists the Internet service providers as defendants but it’s important to note that most have a vested interest in the injunction being put in place. Sky, BT and Virgin Media all screen Premier League matches in some way so there’s no surprise that none put up a fight when confronted by the football organization.

Indeed, several of the ISPs appeared to have assisted the EPL in some pretty intimate ways, even going as far as sharing a certain level of customer traffic data with the organization.

It will be interesting to see what effect the new blocking efforts will have on stream availability when the new season begins. Saturday afternoons, when matches take place around the country but are prohibited from being screened due to the blackout, should be the main focal point. As previously suggested, the EPL will probably enjoy more success than last season with experience under their belts.

Finally, tabloids in the UK have been giving the injunction their usual dramatic coverage but a special mention must go out to The Sun. In an article titled “Closing the Net“, the paper said that under the injunction, “BRITS who illegally stream Premier League football matches could have their internet connection shut off.”

The way things are worded it suggests that people who watch streams could be disconnected by their ISP. That is not the case.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

Surge of Threatening Piracy Letters Concerns Finnish Authorities

Post Syndicated from Ernesto original https://torrentfreak.com/massive-surge-in-threatening-piracy-letters-concerns-finnish-authorities-170726/

finlandStarting three years ago, copyright holders began sending out thousands of settlement letters to alleged pirates in Finland, a practice often described as copyright trolling.

In a country with a population of just over five million, copyright holders have cast their net wide. According to local reports, Internet providers handed over details of one hundred thousand customers last year alone.

This practice has not been without controversy. As the settlement letters were sent out, recipients – including some pensioners – started to complain. Many of the accused denied downloading any pirated material but felt threatened by the letters.

Thus far, complaints have been filed with the Market Court, the Finnish Communications Regulatory Authority, the Consumer Authority, and the Ministry of Education and Culture.

In May, the Ministry of Education set up a working group to create a set of ‘best practices’ for copyright enforcement. The working group includes, among others, Internet providers, and outfits that are involved in sending the influx of settlement letters.

Anna Vuopala, a Government’s counselor at the Ministry of Education and Culture, told Kauppaleht that rightsholders should act within the boundaries of the law.

“We strive to create good practices [for copyright enforcement] and eliminate practices that are contrary to law,” says Vuopala, who’s leading the working group.

If the parties involved can’t reach an agreement on how to proceed, the Government considers changing existing copyright law to defuse the situation. What these changes could be is unclear at this point.

Earlier this year the Finnish market court already dealt a blow to local copyright trolls. In a unanimous ruling, seven judges ruled that the privacy of alleged BitTorrent pirates outweighs the evidence provided by the rightsholders.

While it was clear that copyright infringement was taking place, the rightsholders failed to show that it was significant enough to hand over the requested personal details.

Although this decision supports the rights of those who are falsely accused, the Government believes that a set of good practices is still needed to prevent future excesses and controversy.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

Intel kills Curie module and Arduino 101 SBC (LinuxGizmos.com)

Post Syndicated from ris original https://lwn.net/Articles/728801/rss

LinuxGizmos reports
that Intel is discontinuing its Curie wearables module and its
Curie-enabled Arduino 101 SBC. “Intel will no longer update the Curie’s Open Developer Kit, and will continue forum support only through Sep. 15. After that, “Intel will make its online resources available for review only and maintain availability to the Intel Curie community until June 15, 2020,” according to the July 18 Intel forum post.

Intel says it is “actively working with alternative manufacturers to continue to make the Arduino 101 development board available to the market.” The chipmaker will support orders of the Arduino 101 through Sep. 17, and will fulfill those orders through Dec. 17. Arduino.cc will continue to offer Arduino IDE support for the 101.”

Kodi Security Risk Emerges After TVAddons Shutdown

Post Syndicated from Andy original https://torrentfreak.com/kodi-security-risk-emerges-after-tvaddons-shutdown-170723/

Formerly known as XBMC, the popularity of the entirely legal Kodi media player has soared in recent years.

Controversial third-party addons that provide access to infringing content have thrust Kodi into the mainstream and the product is now a household name.

Until recently, TVAddons.ag was the leading repository for these addons. During March, the platform had 40 million unique users connected to the site’s servers, together transferring an astounding petabyte of addons and updates.

Everything was going well until news broke last month that the people behind TVAddons were being sued in a federal court in Texas. Shortly after the site went dark and hasn’t been back since.

This was initially a nuisance to the millions of Kodi devices that relied on TVAddons for their addons and updates. With the site gone, none were forthcoming. However, the scene recovered relatively quickly and for users who know what they’re doing, addons are now available from elsewhere.

That being said, something very unusual happened this week. Out of the blue, several key TVAddons domains were transferred to a Canadian law firm. TVAddons, who have effectively disappeared, made no comment. The lawyer involved, Daniel Drapeau, ignored requests for an explanation.

While that’s unusual enough, there’s a bigger issue at play here for millions of former TVAddons users who haven’t yet wiped their devices or upgraded them to work with other repositories.

Without going into huge technical detail, any user of an augmented Kodi device that relied on TVAddons domains (TVAddons.ag, Offshoregit.com) for updates can be reasonably confident that the domains their device is now accessing are not controlled by TVAddons anymore. That is not good news.

When a user installs a Kodi addon or obtains an update, the whole system is based on human trust. People are told about a trustworthy source (repository or ‘repo’) and they feel happy getting their addons and updates from it.

However, any person in control of a repo can make a Kodi addon available that can do pretty much anything. When that’s getting free movies, people tend to be happy, but when that’s making a botnet out of set-top boxes, enthusiasm tends to wane a bit.

If the penny hasn’t yet dropped, consider this.

TVAddons’ domains are now being run by a law firm which refuses to answer questions but has the power to do whatever it likes with them, within the law of course. Currently, the domains are lying dormant and aren’t doing anything nefarious, but if that position changes, millions of people will have absolutely no idea anything is wrong.

TorrentFreak spoke to Kodi Project Manager Nathan Betzen who agrees that the current security situation probably isn’t what former TVAddons users had in mind.

“These are unsandboxed Python addons. The person [in control of] the repo could do whatever they wanted. You guys wrote about the addon that created a DDoS event,” Betzen says.

“If some malware author wanted, he could easily install a watcher that reports back the user’s IP address and everything they were doing in Kodi. If the law firm is actually an anti-piracy group, that seems like the likeliest thing I can think of,” he adds.

While nothing can be ruled out, it seems more likely that the law firm in question has taken control of TVAddons’ domains in order to put them out of action, potentially as part of a settlement in the Dish Network lawsuit. However, since it refuses to answer any questions, everything is open to speculation.

Another possibility is that the domains are being held pending sale, which then raises questions over who the buyer might be and what their intentions are. The bottom line is we simply do not know and since nobody is talking, it might be prudent to consider the worst case scenario.

“If it’s just a holding group, then people [in control of the domain/repo] could do whatever they can think of. Want a few million incredibly inefficient bit mining boxes?” Betzen speculates.

While this scenario is certainly a possibility, one would at least like to think of it as unlikely. That being said, plenty of Internet security fails can be attributed to people simply hoping for the best when things go bad. That rarely works.

On the plus side, Betzen says that since Python code is usually pretty easy to read, any nefarious action could be spotted by vigilant members of the community fairly quickly. However, Martijn Kaijser from Team Kodi warns that it’s possible to ship precompiled Python code instead of the readable versions.

“You can’t even see what’s in the Python files and what they do,” he notes.

Finally, there’s a possibility that TVAddons may be considering some kind of comeback. Earlier this week a new domain – TVAddons.co – was freshly registered, just after the old domains shifted to the law firm. At this stage, however, nothing is known about the site’s plans.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

BREIN Takes Down 231 Pirate Sites in Six Months, But That’s Not All

Post Syndicated from Andy original https://torrentfreak.com/brein-takes-down-231-pirate-sites-in-six-months-but-thats-not-all-170722/

Over the years, the MPAA and RIAA have grabbed hundreds of headlines for their anti-piracy activities but recently their work has been more subtle. The same cannot be said of Dutch anti-piracy group BREIN.

BREIN is the most prominent outfit of its type in the Netherlands but it’s not uncommon for its work to be felt way beyond its geographical borders. The group’s report for the first six months of 2017 illustrates that in very clear terms.

In its ongoing efforts to reduce piracy on movies, music, TV shows, books and games, BREIN says it carried out 268 investigations during the first two quarters of 2017. That resulted in the takedown of 231 piracy-focused sites and services.

They included 45 cyberlocker linking sites, 30 streaming sites and 9 torrent platforms. The last eDonkey site in the Netherlands was among the haul after its operators reached a settlement with BREIN. The anti-piracy outfit reports that nearly all of the sites were operated anonymously so in many instances hosting providers were the ones to pull the plug, at BREIN’s request.

BREIN has also been actively tracking down people who make content available on file-sharing networks. These initial uploaders are considered to be a major part of the problem, so taking them out of the equation is another of BREIN’s goals.

In total, 14 major uploaders to torrent, streaming, and Usenet platforms were targeted by BREIN in the first six months of this year, with each given the opportunity to settle out of court or face legal action. Settlements typically involved a cash payment of between 250 and 7,500 euros but in several instances, uploaders were also required to take down the content they had uploaded.

In one interesting case, BREIN obtained an ex parte court order against a person running a “live cinema” on Facebook. He later settled with the anti-piracy group for 7,500 euros.

BREIN has also been active in a number of other areas. The group says it had almost 693,000 infringing results removed from Google search, pushing its total takedowns to more than 15.8 million. In addition, more than 2,170 listings for infringing content and devices were removed from online marketplaces and seven piracy-focused Facebook groups were taken down.

But while all of these actions have an effect locally, it is BREIN’s persistence in important legal cases that have influenced the copyright landscape across Europe.

Perhaps the most important case so far is BREIN v Filmspeler, which saw the anti-piracy group go all the way to the European Court of Justice for clarification on the law surrounding so-called “fully loaded” set-top boxes.

In a ruling earlier this year, the ECJ not only determined that selling such devices is a breach of copyright law, but also that people streaming content from an illicit source are committing an offense. Although the case began in the Netherlands, its effects will now be felt right across Europe, and that is almost completely down to BREIN.

But despite the reach of the ruling, BREIN has already been making good use of the decision locally. Not only has the operator of the Filmspeler site settled with BREIN “for a substantial amount”, but more than 200 sellers of piracy-configured set-top boxes have ceased trading since the ECJ decision. Some of the providers are the subject of further legal action.

Finally, a notable mention must go to BREIN’s determination to have The Pirate Bay blocked in the Netherlands. The battle against ISPs Ziggo and XS4ALL has been ongoing for seven years and like the Filmspeler case, required the attention of the European Court of Justice. While it’s still not over yet, it seems likely that the Supreme Court will eventually rule in BREIN’s favor.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

‘Game of Thrones Season 7 Premiere Pirated 90 Million Times’

Post Syndicated from Ernesto original https://torrentfreak.com/game-of-thrones-season-7-premiere-pirated-90-million-times-170721/

Last Sunday, the long-awaited seventh season of the hit series Game of Thrones aired in dozens of countries worldwide.

The show has broken several piracy records over the years and, thus far, there has been plenty of interest in the latest season as well.

Like every year, the torrent download figures quickly ran into the millions. However, little is known about the traffic that goes to streaming portals, which have outgrown traditional file-sharing sites in recent years.

One of the main problems is that it’s impossible for outsiders to know exactly how many visitors pirate streaming services get. Traffic data for these sites are not public, which makes it difficult to put an exact figure on the number of views one particular video has.

Piracy monitoring firm MUSO hasn’t shied away from this unexplored territory though and has now released some hard numbers.

According to MUSO, the premiere episode of the seventh season of Game of Thrones has been pirated more than 90 million times in only three days. A massive number, which is largely driven by streaming traffic.

Exactly 77,913,032 pirate views came from streaming portals, while public torrent traffic sits in second place with 8,356,382 downloads. Another 4,949,298 downloads are linked to direct download sites, while the remaining 523,109 come from private torrents.

Why other platforms such as Usenet are not covered remains unexplained in the press release, but without these the total is already quite substantial, to say the least.

MUSO reports that most pirate traffic comes from the United States, with 15.1 million unauthorized downloads and streams. The United Kingdom follows in second place with 6.2 million, before Germany, India, and Indonesia, with between 4 and 5 million each.

Andy Chatterley, MUSO’s CEO and Co-Founder, notes that the results may come as a surprise to some industry insiders, describing them as “huge.”

“There is no denying that these figures are huge, so they’re likely to raise more than a few eyebrows in the mainstream industry, but it’s in line with the sort of scale we see across piracy sites and should be looked at objectively.

“What we’re seeing here isn’t just P2P torrent downloads but unauthorized streams and every type of piracy around the premiere. This is the total audience picture, which is usually unreported,” Chatterley adds.

While there is no denying that the numbers are indeed huge, it would probably be better to view them as estimates. MUSO generally sources its data from SimilarWeb, which uses a sample of 200 million ‘devices’ to estimate website traffic. Website visits are then seen as “downloads,” and the sample data is extrapolated into the totals.

This also explains why other types of download traffic, such as Usenet, are not included at all. These are not web-based. Similarly, the data doesn’t appear to cover all countries. Game of Thrones piracy is very substantial in China, for example, but in its previous reports, MUSO didn’t exclude Chinese traffic.

Taking the caveats above into account, MUSO’s data could be a good estimate of the total (web) pirate traffic for the Game of Thrones premiere. This would suggest some pretty high piracy rates in some countries, but we’ve seen stranger things.

Note: TorrentFreak reached out to MUSO for further details on its methodology. The company confirmed that its data is based on traffic to 23,000 of the most-used piracy sites. The data is collected from over 200 million devices, located in over 200 countries. This appears to confirm that it is indeed SimilarWeb data.

Countries with the highest GoT piracy activity, according to MUSO:

United States of America: 15,075,951
United Kingdom: 6,252,903
Germany: 4,897,280
India: 4,335,331
Indonesia: 4,286,927
Philippines: 4,189,030
Canada: 3,182,851
France: 2,881,467
Turkey: 2,802,458
Vietnam: 2,436,149
Australia: 2,241,463
Russian Federation: 2,196,799
Netherlands: 1,881,718
Brazil: 1,796,759
Malaysia: 1,737,005

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.