Tag Archives: sms

Google Signs Agreement to Tackle YouTube Piracy

Post Syndicated from Andy original https://torrentfreak.com/google-signs-unprecedented-agreement-to-tackle-youtube-piracy-170921/

Once upon a time, people complaining about piracy would point to the hundreds of piracy sites around the Internet. These days, criticism is just as likely to be leveled at Google-owned services.

YouTube, in particular, has come in for intense criticism, with the music industry complaining of exploitation of the DMCA in order to obtain unfair streaming rates from record labels. Along with streaming-ripping, this so-called Value Gap is one of the industry’s hottest topics.

With rightsholders seemingly at war with Google to varying degrees, news from France suggests that progress can be made if people sit down and negotiate.

According to local reports, Google and local anti-piracy outfit ALPA (l’Association de Lutte Contre la Piraterie Audiovisuelle) under the auspices of the CNC have signed an agreement to grant rightsholders direct access to content takedown mechanisms on YouTube.

YouTube has granted access to its Content ID systems to companies elsewhere for years but the new deal will see the system utilized by French content owners for the first time. It’s hoped that the access will result in infringing content being taken down or monetized more quickly than before.

“We do not want fraudsters to use our platforms to the detriment of creators,” said Carlo D’Asaro Biondo, Google’s President of Strategic Relationships in Europe, the Middle East and Africa.

The agreement, overseen by the Ministry of Culture, will see Google provide ALPA with financial support and rightsholders with essential training.

ALPA president Nicolas Seydoux welcomed the deal, noting that it symbolizes the “collapse of the wall of incomprehension” that previously existed between France’s rightsholders and the Internet search giant.

The deal forms part of the French government’s “Plan of Action Against Piracy”, in which it hopes to crack down on infringement in various ways, including tackling the threat of pirate sites, better promotion of services offering legitimate content, and educating children “from an early age” on the need to respect copyright.

“The fight against piracy is the great challenge of the new century in the cultural sphere,” said France’s Minister of Culture, Françoise Nyssen.

“I hope this is just the beginning of a process. It will require other agreements with rights holders and other platforms, as well as at the European level.”

According to NextInpact, the Google agreement will eventually encompass the downgrading of infringing content in search results as part of the Trusted Copyright Removal Program. A similar system is already in place in the UK.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

Ukraine Faces Call for US Trade Sanctions over Online Piracy

Post Syndicated from Ernesto original https://torrentfreak.com/ukraine-faces-call-us-trade-sanctions-over-online-piracy-170918/

The International Intellectual Property Alliance (IIPA) is recommending that the U.S. Government should suspend Ukraine’s GSP trade benefits, claiming that the country doesn’t do enough to protect the interests of copyright holders.

Last year Ukraine enjoyed $53.7 million in unilateral duty-free benefits in the US, while US companies suffering millions of dollars in losses in Ukraine due to online piracy, they argue.

The IIPA, which includes a wide range of copyright groups including the MPAA, RIAA, BSA and ESA, characterizes the country as a safe harbor for pirate sites. While physical piracy was properly addressed ten years ago after a previous sanction, digital piracy remains rampant.

One of the main problems is that local hosting companies are offering their services to a wide variety of copyright-infringing websites. Without proper enforcement, more and more websites have moved their services there.

“By allowing these problems to fester for years, weak digital enforcement has resulted in an exponential increase in the number of illegal peer-to-peer (‘P2P’) hosting and website-based Internet piracy sites, including some of the world’s largest BitTorrent sites located in Ukraine,” IIPA writes.

“Some Internet pirates have purposefully moved their servers and operations to Ukraine in the past few years to take advantage of the current lawless situation. Many of these illegal services and sites target audiences throughout Europe and the United States.”

The copyright holders highlight the defunct ExtraTorrent site as an example but note that there are also many other torrent sites, pirate streaming sites, cyberlockers, and linking sites in Ukraine.

While pirate sites are hosted all over the world, the problem is particularly persistent in Ukraine because many local hosting companies fail to process takedown requests. This, despite repeated calls from copyright holders to work with them.

“Many of the websites offering pirated copyright materials are thriving in part because of the support of local ISPs,” IIPA writes.

“The copyright industries have, for years, sought private agreements with ISPs to establish effective mechanisms to take down illegal websites and slow illegal P2P traffic. In the absence of legislation, however, these voluntary efforts have generally not succeeded, although, some ISPs will delete links upon request.”

In order to make real progress, the copyright holders call for new legislation to hold Internet services accountable and to make it easier to come after pirate sites that are hosted in Ukraine.

“Legislation is needed to institute proper notice and takedown provisions, including a requirement that service providers terminate access to individuals (or entities) that have repeatedly engaged in infringement, and the retention of information for law enforcement, as well as to provide clear third party liability regarding ISPs.”

In addition to addressing online piracy, IIPA further points out that the collecting societies in Ukraine are not functioning properly. At the moment there are 18 active and competing organizations, creating a chaotic situation where rightsholders are not properly rewarded, they suggest.

IIPA recommends that the U.S. Government accepts its petition and suspends or withdraws Ukraine’s benefits until the country takes proper action.

Ukraine’s Government, for its part, informs the US Government that progress is being made. There are already several new laws in the works to improve intellectual property protection. The issue is one of the Government’s “key priorities,” they state, hoping to avert any sanctions.

—

IIPA’s full submission to the US Trade Representative is available here (pdf).

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

Self-Driving Cars Should Be Open Source

Post Syndicated from Bozho original https://techblog.bozho.net/self-driving-cars-open-source/

Self-driving cars are (will be) the pinnacle of consumer products automation – robot vacuum cleaners, smart fridges and TVs are just toys compared to self-driving cars. Both in terms of technology and in terms of impact. We aren’t yet on level 5 self driving cars , but they are behind the corner.

But as software engineers we know how fragile software is. And self-driving cars are basically software, so we can see all the risks involved with putting our lives in the hands anonymous (from our point of view) developers and unknown (to us) processes and quality standards. One may argue that this has been the case for every consumer product ever, but with software is different – software is way more complex than anything else.

So I have an outrageous proposal – self-driving cars should be open source. We have to be able to verify and trust the code that’s navigating our helpless bodies around the highways. Not only that, but we have to be able to verify if it is indeed that code that is currently running in our car, and not something else.

In fact, let me extend that – all cars should be open source. Before you say “but that will ruin the competitive advantage of manufacturers and will be deadly for business”, I don’t actually care how they trained their neural networks, or what their datasets are. That’s actually the secret sauce of the self-driving car and in my view it can remain proprietary and closed. What I’d like to see open-sourced is everything else. (Under what license – I’d be fine to even have it copyrighted and so not “real” open source, but that’s a separate discussion).

Why? This story about remote carjacking using the entertainment system of a Jeep is a scary example. Attackers that reverse engineer the car software can remotely control everything in the car. Why did that happen? Well, I guess it’s complicated and we have to watch the DEFCON talk.

And also read the paper, but a paragraph in wikipedia about the CAN bus used in most cars gives us a hint:

CAN is a low-level protocol and does not support any security features intrinsically. There is also no encryption in standard CAN implementations, which leaves these networks open to man-in-the-middle packet interception. In most implementations, applications are expected to deploy their own security mechanisms; e.g., to authenticate incoming commands or the presence of certain devices on the network. Failure to implement adequate security measures may result in various sorts of attacks if the opponent manages to insert messages on the bus. While passwords exist for some safety-critical functions, such as modifying firmware, programming keys, or controlling antilock brake actuators, these systems are not implemented universally and have a limited number of seed/key pair

I don’t know in what world it makes sense to even have a link between the entertainment system and the low-level network that operates the physical controls. As apparent from the talk, the two systems are supposed to be air-gapped, but in reality they aren’t.

Rookie mistakes were abound – unauthenticated “execute” method, running as root, firmware is not signed, hard-coded passwords, etc. How do we know that there aren’t tons of those in all cars out there right now, and in the self-driving cars of the future (which will likely use the same legacy technologies of the current cars)? Recently I heard a negative comment about the source code of one of the self-driving cars “players”, and I’m pretty sure there are many of those rookie mistakes.

Why this is this even more risky for self-driving cars? I’m not an expert in car programming, but it seems like the attack surface is bigger. I might be completely off target here, but on a typical car you’d have to “just” properly isolate the CAN bus. With self-driving cars the autonomous system that watches the surrounding and makes decisions on what to do next has to be connected to the CAN bus. With Tesla being able to send updates over the wire, the attack surface is even bigger (although that’s actually a good feature – to be able to patch all cars immediately once a vulnerability is discovered).

Of course, one approach would be to introduce legislation that regulates car software. It might work, but it would rely on governments to to proper testing, which won’t always be the case.

The alternative is to open-source it and let all the white-hats find your issues, so that you can close them before the car hits the road. Not only that, but consumers like me will feel safer, and geeks would be able to verify whether the car is really running the software it claims to run by verifying the fingerprints.

Richard Stallman might be seen as a fanatic when he advocates against closed source software, but in cases like … cars, his concerns seem less extreme.

“But the Jeep vulnerability was fixed”, you may say. And that might be seen as being the way things are – vulnerabilities appear, they get fixed, life goes on. No person was injured because of the bug, right? Well, not yet. And “gaining control” is the extreme scenario – there are still pretty bad scenarios, like being able to track a car through its GPS, or cause panic by controlling the entertainment system. It might be over wifi, or over GPRS, or even by physically messing with the car by inserting a flash drive. Is open source immune to those issues? No, but it has proven to be more resilient.

One industry where the problem of proprietary software on a product that the customer bought is … tractors. It turns out farmers are hacking their tractors, because of multiple issues and the inability of the vendor to resolve them in a timely manner. This is likely to happen to cars soon, when only authorized repair shops are allowed to touch anything on the car. And with unauthorized repair shops the attack surface becomes even bigger.

In fact, I’d prefer open source not just for cars, but for all consumer products. The source code of a smart fridge or a security camera is trivial, it would rarely mean sacrificing competitive advantage. But refrigerators get hacked, security cameras are active part of botnets, the “internet of shit” is getting ubiquitous. A huge amount of these issues are dumb, beginner mistakes. We have the right to know what shit we are running – in our frdges, DVRs and ultimatey – cars.

Your fridge may soon by spying on you, your vacuum cleaner may threaten your pet in demand of “ransom”. The terrorists of the future may crash planes without being armed, can crash vans into crowds without being in the van, and can “explode” home equipment without being in the particular home. And that’s not just a hypothetical.

Will open source magically solve the issue? No. But it will definitely make things better and safer, as it has done with operating systems and web servers.

The post Self-Driving Cars Should Be Open Source appeared first on Bozho's tech blog.

Unite Real-Time and Batch Analytics Using the Big Data Lambda Architecture, Without Servers!

Post Syndicated from Laith Al-Saadoon original https://aws.amazon.com/blogs/big-data/unite-real-time-and-batch-analytics-using-the-big-data-lambda-architecture-without-servers/

The Big Data Lambda Architecture seeks to provide data engineers and architects with a scalable, fault-tolerant data processing architecture and framework using loosely coupled, distributed systems. At a high level, the Lambda Architecture is designed to handle both real-time and historically aggregated batched data in an integrated fashion. It separates the duties of real-time and batch processing so purpose-built engines, processes, and storage can be used for each, while serving and query layers present a unified view of all of the data.

Historically, the Lambda Architecture demanded the use of various complex systems to achieve the outcomes of uniting batch and real-time views. Data platform engineers and architects were required to implement services running on Amazon EC2 for data collection and ingestion, batch processing, stream processing, serving layers, and dashboards/reporting. As time has gone on, AWS customers have continued to ask for managed solutions that scale seamlessly and put less focus on infrastructure, allowing teams to focus on what really matters: the data and the resulting insights.

In this post, I show you how you can use AWS services like AWS Glue to build a Lambda Architecture completely without servers. I use a practical demonstration to examine the tight integration between serverless services on AWS and create a robust data processing Lambda Architecture system.

New:  AWS Glue

With the launch of AWS Glue, AWS provides a portfolio of services to architect a Big Data platform without managing any servers or clusters. AWS Glue is a fully managed extract, transform, and load (ETL) service that makes it easy for customers to prepare and load their data for analytics. You can create and run an ETL job with a few clicks in the AWS Management Console. You simply point AWS Glue to your data stored on AWS, and AWS Glue discovers your data and stores the associated metadata (for example, the table definition and schema) in the AWS Glue Data Catalog. After it’s cataloged, your data is immediately searchable, queryable, and available for ETL.

AWS Glue generates the code to execute your data transformations and data loading processes. Furthermore, AWS Glue provides a managed Spark execution environment to run ETL jobs against a data lake in Amazon S3. In short, you can now run a Lambda Architecture in AWS in a completely 100% serverless fashion!

“Serverless” applications allow you to build and run applications without thinking about servers. What this means is that you can now stream data in real-time, process huge volumes of data in S3, and run SQL queries and visualizations against that data without managing server provisioning, installation, patching, or capacity scaling. This frees up your users to spend more time interpreting the data and deriving business value for your organization.

Solution overview

The AWS services involved in this solution include:

The following diagram explains how the services work together:

None of the included services require the creation, configuration, or installation of servers, clusters, and databases.

In this example, you use these services to send and process simulated streaming data of sensor devices to Kinesis Firehose and store the raw data in S3. Using AWS Glue, you analyze the raw data from S3 in batch-oriented fashion to look at the thermostat efficiency over time against the historical data, and store results back in S3.

Using Amazon Kinesis Analytics, you analyze and filter the data to detect inefficient sensors in real time. In this example, you detect inefficiencies in thermostat devices by comparing their temperature settings against the temperature they are reading (where thermostats are typically set between 70-72º F).

Finally, you use Athena and Amazon QuickSight to query and visualize the data and build a dashboard that can be shared with other users of Amazon QuickSight in your organization.

Walkthrough

You use the AWS CLI and AWS CloudFormation throughout this tutorial, therefore a basic understanding of these services is assumed.

At the time of writing this post, AWS Glue is only available in the US East (N. Virginia) AWS Region. Complete all the steps in that region.

Step 1: Set up baseline AWS resources

I’ve set up a CloudFormation template to set up the Kinesis streaming resources for you.

Launch Stack

Before you continue with the rest of the walkthrough, make sure that the Status value is CREATE_COMPLETE. This can take anywhere between 3 and 5 minutes, so grab coffee or take a moment to prepare and review the next steps.

Step 2: Set up the Amazon Kinesis Data Generator

You will be leveraging the Amazon Kinesis Data Generator (KDG) to simulate devices pushing data into Kinesis Firehose.

Kinesis Data Generator

Make sure that you can log in to the KDG producer page before you continue.

Step 3: Send data to Kinesis Firehose using the KDG

In this step, you configure the KDG to send simulated records to simulate devices streaming data into Kinesis Firehose. In the data generator template below, notice the complex weighting. This demonstrates outliers in device temperature readings later on. Kinesis Firehose sends all the raw data to S3 in addition to Kinesis Analytics for real-time processing.

On the KDG producer page, for Region, select “us-east-1”. For Stream/delivery stream, select AWSBigDataBlog-RawDeliveryStream. For Records per second, type 100.

Paste the template shown below into KDG template window:

{
  "uuid": "{{random.uuid}}",
  "devicets": "{{date.utc("YYYY-MM-DD HH:mm:ss.SSS")}}",
  "deviceid": {{random.number(1000)}},
  "temp": {{random.weightedArrayElement(
    {"weights":[0.33, 0.33, 0.33, 0.01],"data":[70, 71, 72, 78]}
  )}}
}

Choose Send data.

Without leaving the KDG page, navigate to the Kinesis Analytics console to view the status of the application processing the data in real time. When you are happy with the amount of data sent, choose Stop Sending Data to Kinesis. I recommend waiting until at least 20,000 records are sent.

Step 4: Submit AWS Glue crawlers to interpret the table definition for Kinesis Firehose outputs in S3

In this step, you use AWS Glue crawlers to crawl and generate table definitions against the produced data in S3. This allows you to analyze data in aggregate over a historical context instead of just using the latest data.

Use CLI commands to define and run AWS Glue crawlers:

  • Replace the <bucketName> text in the command with the bucket name that you specified in the CloudFormation template in step 1.
  • Replace the <RoleArn> text in the command with an IAM role ARN with permissions to use AWS Glue. For more information, see Setting up IAM Permissions for AWS Glue. 
    aws --region us-east-1 glue create-crawler \
--name 'thermostat-data-crawler' \
--database-name 'awsblogsgluedemo' \
--role '<RoleArn>' \
--targets '{"S3Targets":[{"Path": "s3://<bucketName>", "Exclusions": ["etl_code", "temporary_dir"]}]}'

To run the crawler, navigate to the AWS Glue console.

Select the crawler that you created earlier, “thermostat-data-crawler”, and choose Run crawler.

After just a few seconds, the crawler shows as “Ready”, and will have automatically inferred the schema of three tables:

  • “raw”—Data being sent from KDG to Kinesis Firehose, delivered to S3.
  • “results”—Kinesis Analytics results data on real-time processing for percent inefficiency of devices, stored in S3.
  • “sampledata”—Simulated user device settings table for 1,000 thermostats in S3. This data is joined to the raw data to calculate the “percent inefficiency” (absolute value of temperature reading minus temperature setting, divided by the temperature setting).

Zooming in on the architecture diagram, I’ve included the names of these tables. Take a moment to see how the tables fit in the overall Lambda Architecture:

Step 5: Submit an AWS Glue job to pre-process daily thermostat efficiency

In this step, you use an AWS Glue job to join data and pre-process calculated views to assess the efficiency of the thermostat devices. The results are stored in S3 to be queried using Athena.

  • Replace the <bucketName> text in the command with the bucket name that you specified in the CloudFormation template in step 1.
  • Replace the <RoleName> text in the command with an IAM role with permissions to use AWS Glue. For more information, see Setting up IAM Permissions for AWS Glue. 
    aws --region us-east-1 glue create-job \
--name 'daily_avg' \
--role '<RoleName>' \
--execution-property 'MaxConcurrentRuns=1' \
--command '{"Name": "glueetl", "ScriptLocation": "s3://<bucketName>/etl_code/daily_avg.py"}' \
--default-arguments '{"--TempDir":"s3://<bucketName>/temporary_dir","--job-bookmark-option":"job-bookmark-disable"}'

To view and run the job, navigate to the AWS Glue console.

Select the job named “daily_avg”, and choose Action, Run job.

In the Parameters (optional) dialog box, choose Run job.

This job can take between 10–14 minutes to run. Take a moment to view the script definition in the console. Take care not to save any changes to the script at this time:

https://console.aws.amazon.com/glue/home?region=us-east-1#editJob:jobName=daily_avg

You should see a page similar to the following:

Step 6: Query data in S3 directly with Athena

You can now navigate to the Athena console, where Athena already has the tables present and ready to query. No loading or clusters necessary!

This walkthrough assumes that you have already integrated AWS Glue with Athena. If not, see Upgrading to the AWS Glue Data Catalog Step-by-Step.

Try out each of the individual queries below to analyze the data in Athena.

Query and analyze the raw time-series data in the “master dataset” Lambda Architecture layer
--"Events by Device ID"
SELECT uuid,
         devicets,
         deviceid,
         temp
FROM awsblogsgluedemo."raw"
WHERE deviceid = 1
ORDER BY  devicets DESC;

--"Top 20 most active thermostats"
SELECT deviceid,
         COUNT(*) AS num_events
FROM awsblogsgluedemo."raw"
GROUP BY  deviceid
ORDER BY  num_events DESC LIMIT 20;
Query and analyze the data processed by AWS Glue at the “Serving” layer:
--"Top 10 Most inefficient devices"
SELECT deviceid,
        daily_avg_inefficiency
FROM awsblogsgluedemo.daily_avg_inefficiency
ORDER BY  daily_avg_inefficiency DESC LIMIT 10;

--"KPI - Overall device daily inefficiency"
SELECT ( SUM(daily_avg_inefficiency)/COUNT(*) ) AS all_device_avg_inefficiency,
         date
FROM awsblogsgluedemo.daily_avg_inefficiency
GROUP BY  date;
Query and analyze the data filtered by Kinesis Analytics, the outputs of the “speed” layer:
--"Top 10 most inefficient devices - event-level granularity"
SELECT col0 AS "uuid",
   col1 AS "deviceid",
   col2 AS "devicets",
   col3 AS "temp",
   col4 AS "settemp",
   col5 AS "pct_inefficiency"
FROM awsblogsgluedemo.results
ORDER BY pct_inefficiency DESC limit 10;

Step 7: Visualize with Amazon QuickSight

Now you have the data in S3 and table definitions in the AWS Glue Data Catalog, and you can query the data directly in S3 with Athena. You are now ready to begin creating visualizations with Amazon QuickSight.

In these steps, I demonstrate how you can visualize your data in S3, again without provisioning any infrastructure for databases, clusters, or BI tools.

Tie in all the data sources into Amazon QuickSight

  1. Open Amazon QuickSight.
  2. Choose Connect to another data source or upload a file, Athena.
  3. Give the data source a name, like awsblogsgluedemo
  4. Choose daily_avg_inefficiency, Select.
  5. Choose Directly query your data, Edit/Preview data. On the next screen, leave the settings unchanged and choose Save.
  6. Choose New data set in the top right corner.
  7. Under From existing data sources, choose awsblogsgluedemo.
  8. Choose Create data set, raw, and Select.
  9. Choose Directly query your data, Edit/Preview data. On the next screen, leave the settings unchanged and choose Save.

Repeat steps 6–9 to add the “results” dataset. At step 9, rename the column “col1” to “deviceid” by choosing the pencil icon.

Create an analysis that includes all data sources (raw, processed, real-time results)

  1. Return to the Amazon QuickSight main page by choosing QuickSight.
  2. Choose New analysis, daily_avg_inefficiency, and Create analysis.
  3. Choose Fields, Edit analysis data sets.
  4. Choose Add data set, results, and Select.

Repeat steps 3 and 4 to add the “raw” dataset.

Create a KPI chart on AWS Glue processed data

  1. Choose Fields, daily_avg_inefficiency.
  2. In the Visual types pane, choose the Key Performance Indicator (KPI) icon, shown below.
  1. Drag daily_avg_inefficiency from Fields into the Value
  2. Drag date into the Trend group well.
  3. For daily_avg_inefficiency (Sum), choose the down arrow, pause on over Aggregate: Sum, and choose Average. Choose the down arrow again, pause on over Show as: Number and choose Percent.
  4. Click daily_avg_inefficiency (Average) in the Value well to modify the presentation to fewer decimal points.

You should have a visualization similar to the following:

To create a scatter plot of all devices:

  1. At the top right of the screen, choose Add, Add visual*.
  2. For Visual type, choose Scatter plot.
  1. Drag deviceid into the X axis well and the Group/Color
  2. Drag daily_avg_inefficiency into the Y axis well and the Size

You should now have a scatter plot visualization similar to the following figure. See if you can find a poorly functioning device with a high percentage inefficiency!

To create a line chart against the raw time-series dataset

  1. Choose Fields, raw.
  2. Choose Add, Add visual*.
  3. For Visual type, choose Line chart.
  4. Drag devicets into the X axis Drag temp into the Value well.
  5. For temp (Sum), choose the down arrow, pause on over Aggregate: Sum, and then choose Average.
  6. Choose the down-arrow at the top right, and choose Format visual.
  7. Expand the Y-Axis menu on the left, and choose Range, Auto (based on data range).
  8. Drag the slider to expand the time-series line characters.

You should now have a line chart similar to the following image. The spikes represent an increase in average temperature readings across the devices.

To create a heat map of the Kinesis Analytics results dataset

  1. Choose Fields, results.
  2. Choose Add, Add visual*.
  3. For Visual type, choose Heap map.
  4. Drag deviceid into the Columns and Values For Aggregate, choose Count.
  5. Choose deviceid (Count). For Sort, choose the descending icon.

You should have a visualization like the following graph, which shows deviceid values on the far top left that have the most occurrences of an inefficient temperature reading. Notice that the heat map also colors them darker shades for easy visualization. Look for the devices at the top right that have a high frequency of inefficiency readings.

Finally, you can resize the charts and save them as a dashboard to share to other Amazon QuickSight users in your organization:

  1. Resize the charts by choosing the bottom right corner of a visual.
  2. Choose Share, Create dashboard.
  3. For Create new dashboard as, type a dashboard name, such as “Thermostats Performance.” Choose Create dashboard.

You should now have a dashboard containing all your visuals, similar to the following:

Conclusion

In this post, you saw how to process and analyze data from both streaming and batch sources together in a 100% serverless fashion. The Lambda architecture principles guided a system that separates the ingestion and processing mechanisms, but using fully managed, serverless AWS services.

You went from collecting real-time data, processing and joining with a user settings file, and querying and visualizing the data without servers.

I hope you found this post useful. In the comments, please share your thoughts on how you might extend this architecture to suit your needs.

Additional Reading

Learn how SmartNews built a Lambda Architecture on AWS to analyze customer behavior and recommend content!

About the Author

Laith Al-Saadoon is a Solutions Architect with a focus on data analytics at Amazon Web Services. He spends his days obsessing over designing customer architectures to process enormous amounts of data at scale. In his free time, he follows the latest trends in artificial intelligence.

 

 

 

 

Private Torrent Site Legal Battle Heard By Court of Appeal

Post Syndicated from Andy original https://torrentfreak.com/private-torrent-site-legal-battle-heard-by-court-of-appeal-170908/

Founded way back in 2006, SwePiracy grew to become one of the most famous private torrent sites on the Swedish scene. Needless to say, it also became a target for anti-piracy outfits.

Six years after its debut and following an investigation by anti-piracy group Antipiratbyrån (now Rights Alliance), during 2012 police in Sweden and the Netherlands cooperated to shut down the site and arrest its operator.

In early 2016, more than four years on, SwePiracy’s then 25-year-old operator appeared in court to answer charges relating to the unlawful distribution of a sample 27 movies between March 2011 and February 2012. The prosecution demanded several years in prison and nearly $3.13 million (25 million kronor) in damages.

SwePiracy defense lawyer Per E. Samuelsson, who previously took part in The Pirate Bay trial, said the claims against his client were the most unreasonable he’d seen in his 35 years as a lawyer.

In October 2016, three weeks after the full trial, the Norrköping District Court handed down its decision. Given some of the big numbers being thrown around, the case seemed to turn out relatively well for the defendant.

While SwePiracy’s former operator was found guilty of copyright infringement, the prosecution’s demands for harsh punishment were largely pushed aside. A jail sentence was switched to probation plus community service, and the millions of dollars demanded in damages were reduced to ‘just’ $148,000, payable to movie outfit Nordisk Film. On top, $45,600 said to have been generated by SwePiracy was confiscated.

Almost immediately both sides announced an appeal, with the defendant demanding a more lenient sentence and the prosecution naturally leaning the other way. This week the case was heard at the Göta Court of Appeal, one of the six appellate courts in the Swedish system.

“We state that the District Court made an inaccurate assessment of the damages. So the damages claim remains at the same level as before,” Rights Alliance lawyer Henrik Pontén told Sweden’s IDG.

“There are two different approaches. We say that you have to pay for the entire license [for content when you infringe]. The District Court looked at how many times the movies were downloaded during the period.”

According to Pontén, the cost of such a license is hypothetical since there are no licenses available for distributing content through entities such as torrent sites, which have no mechanisms for control and no limits on sharing. That appears to have motivated the prosecution to demand a hefty price tag.

In addition to Rights Alliance wanting a better deal for their theoretical license, the official prosecutor also has issues with the amount of money that was confiscated from the platform.

“The operator has received donations to run the site. I have calculated how much money was received and the sum that the District Court awarded was almost half of my calculations,” Henrik Rasmusson told IDG.

Only time will tell how the Court of Appeal will rule but it’s worth noting that the decision could go either way or might even stand as it is now. In any event, this case has dragged on for far too long already and is unlikely to end positively for any of the parties involved.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

Mandatory Piracy Filters Could Breach Human Rights, EU Members Warn

Post Syndicated from Ernesto original https://torrentfreak.com/mandatory-piracy-filters-could-breach-human-rights-eu-members-170906/

Last year, the European Commission published its long-awaited proposal to modernize EU copyright law. Among other things, it will require online services to do more to fight piracy.

Specifically, Article 13 of the proposed Copyright Directive requires online services to monitor and filter pirated content, in collaboration with rightsholders.

This means that online services, which deal with large volumes of user-uploaded content, must use fingerprinting or other detection mechanisms to block copyright infringing files, similar to YouTube’s Content-ID system.

The Commission stressed that the changes are needed to support copyright holders. However, many legal scholars, digital activists, and members of the public worry that they will violate the rights of regular Internet users.

They believe that mandatory filters ignore established case law and human rights. This critique is now, in part, backed up by questions from several EU member states.

Authorities in Belgium, Czech Republic, Finland, Hungary, Ireland and the Netherlands have recently sent a series of questions to the Council Legal Service, requesting clarification on several issues.

The document (pdf), published by Statewatch, asks whether a mandatory piracy filter is proportionate and compatible with existing law.

“Would the standalone measure/ obligation as currently proposed under Article 13 be compatible with the Charter of Human Rights […] in the light of the jurisprudence of the CJEU that aims to secure a fair balance in the application of competing fundamental rights?

“Are the proposed measures justified and proportionate?” the member states add.

Specifically, the member states suggest that the filters may hinder people’s right to freedom of expression and information, the right to protection of personal data, and freedom to conduct a business.

One of the problems is that such filters work by monitoring the communications of all citizens uploading to platforms, which would go against existing EU law. In the Sabam v Netlog case, the European Court of Justice ruled that hosting sites can’t be forced to filter copyrighted content, as this would violate the privacy of users and hinder freedom of information.

The letter, which was sent on July 25, also stresses that important copyright exceptions, such as parody and the right to quote, are not taken into account.

“The [Commission’s] proposal does not provide for appropriate measures that would enable these users to actually benefit from public interest copyright exceptions. It is important to point out that certain exceptions to copyright, such as e.g. parody or the quotation right are the embodiment in copyright of fundamental rights other than the right to property.”

This is not the first time that member states have responded critically to the proposal. Tweakers notes that the Dutch Government previously stressed that there should be a better balance between the rights of consumers and copyright holders.

The recent letter from the six member states backs up many of the questions that have been asked by activists, scholars and members of the public, including the “Save the Meme” campaign. These critics hope that the proposal will be changed substantially, ideally without mandatory piracy filters, when it’s voted on in the EU Parliament.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

New Techniques in Fake Reviews

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2017/09/new_techniques_.html

Research paper: “Automated Crowdturfing Attacks and Defenses in Online Review Systems.”

Abstract: Malicious crowdsourcing forums are gaining traction as sources of spreading misinformation online, but are limited by the costs of hiring and managing human workers. In this paper, we identify a new class of attacks that leverage deep learning language models (Recurrent Neural Networks or RNNs) to automate the generation of fake online reviews for products and services. Not only are these attacks cheap and therefore more scalable, but they can control rate of content output to eliminate the signature burstiness that makes crowdsourced campaigns easy to detect.

Using Yelp reviews as an example platform, we show how a two phased review generation and customization attack can produce reviews that are indistinguishable by state-of-the-art statistical detectors. We conduct a survey-based user study to show these reviews not only evade human detection, but also score high on “usefulness” metrics by users. Finally, we develop novel automated defenses against these attacks, by leveraging the lossy transformation introduced by the RNN training and generation cycle. We consider countermeasures against our mechanisms, show that they produce unattractive cost-benefit tradeoffs for attackers, and that they can be further curtailed by simple constraints imposed by online service providers.

Search Engines Will Open Systems to Prove Piracy & VPN Blocking

Post Syndicated from Andy original https://torrentfreak.com/search-engines-will-open-systems-to-prove-piracy-vpn-blocking-170901/

Over the past several years, Russia has become something of a world leader when it comes to website blocking. Tens of thousands of websites are now blocked in the country on copyright infringement and a wide range of other grounds.

With circumvention technologies such as VPNs, however, Russian citizens are able to access blocked sites, a position that has irritated Russian authorities who are determined to control what information citizens are allowed to access.

After working on new legislation for some time, late July President Vladimir Putin signed a new law which requires local telecoms watchdog Rozcomnadzor to maintain a list of banned domains while identifying sites, services, and software that provide access to them.

Rozcomnadzor is required to contact the operators of such services with a request for them to block banned resources. If they do not, then they themselves will become blocked. In addition, search engines are also required to remove blocked resources from their search results, in order to discourage people from accessing them.

With compliance now a matter of law, attention has turned to how search engines can implement the required mechanisms. This week Roskomnadzor hosted a meeting with representatives of the largest Russian search engines including Yandex, Sputnik, Search Mail.ru, where this topic was top of the agenda.

Since failure to comply can result in a fine of around $12,000 per breach, search companies have a vested interest in the systems working well against not only pirate sites, but also mirrors and anonymization tools that provide access to them.

“During the meeting, a consolidated position on the implementation of new legislative requirements was developed,” Rozcomnadzor reports.

“It was determined that the list of blocked resources to be removed from search results will be transferred to the operators of search engines in an automated process.”

While sending over lists of domains directly to search engines probably isn’t that groundbreaking, Rozcomnadzor wants to ensure that companies like Yandex are also responding to the removal requests properly.

So, instead of simply carrying out test searches itself, it’s been agreed that the watchdog will gain direct access to the search engines’ systems, so that direct verification can take place.

“In addition, preliminary agreements have been reached that the verification of the enforcement of the law by the search engines will be carried out through the interaction of the information systems of Roskomnadzor and the operators of search engines,” Rozcomnadzor reports.

Time for search engines to come into full compliance is ticking away. The law requiring them to remove listings for ‘pirate’ mirror sites comes into effect October 1. Exactly a month later on November 1, VPNs and anonymization tools will have to be removed too, if they fail to meet the standards required under state regulation.

Part of that regulation requires anonymization services to disclose the identities of their owners to the government.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

An Invitation for CrashPlan Customers: Try Backblaze

Post Syndicated from Gleb Budman original https://www.backblaze.com/blog/crashplan-alternative-backup-solution/

Welcome CrashPlan Users
With news coming out this morning of CrashPlan exiting the consumer market, we know some of you may be considering which backup provider to call home. We welcome you to try us.

For over a decade, Backblaze has provided unlimited cloud backup for Windows and Macintosh computers at $5 per month (or $50 per year).

“Backblaze is excellent if you’re looking for the cheapest online backup option that still offers serious file protection.” — Dann Berg, Tom’s Guide.

That’s it. Ready to make sure your data is safe? Try Backblaze for free — it’ll take you less than a minute and you don’t need a credit card to start protecting your data.

Our customers don’t have to choose between competing feature sets or hard to understand fine print. There are no extra charges and no limits on the size of your files — no matter how many videos you want to back up. And when we say unlimited, we mean unlimited; there are no restrictions on files, gigabytes, or restores. Customers also love the choices they have for getting their data back — web, mobile apps, and our free Restore by Mail option. We’re also the fastest to back up your data. While other services throttle your upload speeds, we want to get you protected as quickly as possible.

Backblaze vs. Carbonite

We know that CrashPlan is encouraging customers to look at Carbonite as an alternative. We would like to offer you another option: Backblaze. We cost less, we offer more, we store over 350 Petabytes of data, we have restored over 20 billion files, and customers in over 120 countries around the world trust us with their data.

Backblaze Carbonite Basic Carbonite Prime
Price per Computer $50/year $59.99/year $149.99/year
Back Up All User Data By Default – No Picking And Choosing Yes No No
Automatically Back Up Files Of Any Size, Including Videos Yes No Yes1
Back Up Multiple USB External Hard Drives Yes No No
Restore by Mail for Free Yes No No
Locate Computer Yes No No
Manage Families & Teams Yes No No
Protect Accounts Via Two Factor VerificationSMS & Authenticator Apps Yes No No
Protect Data Via Private Encryption Key Yes No No2
(1) All videos and files over 4GB require manual selection.  (2) Available on Windows Only

To get just some of the features offered by Backblaze for $50/year, you would need to purchase Carbonite Prime at $149.99/year.

Reminder: Sync is Not Backup

“Backblaze is my favorite online backup service, mostly because everything about it is so simple, especially its pricing and software.“ Tim Fisher — Lifewire: 22 Online Backup Services Reviewed

Of course, there are plenty of options in the marketplace. We encourage you to choose one to make sure you stay backed up. One thing we tell our own friends and family: sync is not backup.

If you’re considering using a sync service — Dropbox, Google Drive, OneDrive, iCloud, etc. — you should know that these services are not designed to back up all your data. Typically, they only sync data from a specific directory or folder. If the service detects a file was deleted from your sync folder, it also will delete it from their server, and you’re out of luck. In addition, most don’t support external drives and have tiered pricing that gets quite expensive.

Backblaze is the Simple, Reliable, and Affordable Choice for Unlimited Backup of All Your Data
People have trusted Backblaze to protect their digital photos, music, movies, and documents for the past 10 years. We look forward to doing the same for your valuable data.

Your CrashPlan service may not be getting shut off today. But there’s no reason to wait until your data is at risk. Try Backblaze for FREE today — all you need to do is pick an email/password and click download.

The post An Invitation for CrashPlan Customers: Try Backblaze appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Hunting for life on Mars assisted by high-altitude balloons

Post Syndicated from Alex Bate original https://www.raspberrypi.org/blog/eclipse-high-altitude-balloons/

Will bacteria-laden high-altitude balloons help us find life on Mars? Today’s eclipse should bring us closer to an answer.

NASA Bacteria Balloons Raspberry Pi HAB Life on Mars

image c/o NASA / Ames Research Center / Tristan Caro

The Eclipse Ballooning Project

Having learned of the Eclipse Ballooning Project set to take place today across the USA, a team at NASA couldn’t miss the opportunity to harness the high-flying project for their own experiments.

NASA Bacteria Balloons Raspberry Pi HAB Life on Mars

The Eclipse Ballooning Project invited students across the USA to aid in the launch of 50+ high-altitude balloons during today’s eclipse. Each balloon is equipped with its own Raspberry Pi and camera for data collection and live video-streaming.

High-altitude ballooning, or HAB as it’s often referred to, has become a popular activity within the Raspberry Pi community. The lightweight nature of the device allows for high ascent, and its Camera Module enables instant visual content collection.

Life on Mars

image c/o Montana State University

The Eclipse Ballooning Project team, headed by Angela Des Jardins of Montana State University, was contacted by Jim Green, Director of Planetary Science at NASA, who hoped to piggyback on the project to run tests on bacteria in the Mars-like conditions the balloons would encounter near space.

Into the stratosphere

At around -35 degrees Fahrenheit, with thinner air and harsher ultraviolet radiation, the conditions in the upper part of the earth’s stratosphere are comparable to those on the surface of Mars. And during the eclipse, the moon will block some UV rays, making the environment in our stratosphere even more similar to the martian one—ideal for NASA’s experiment.

So the students taking part in the Eclipse Ballooning Project could help the scientists out, NASA sent them some small metal tags.

NASA Bacteria Balloons Raspberry Pi HAB Life on Mars

These tags contain samples of a kind of bacterium known as Paenibacillus xerothermodurans. Upon their return to ground, the bacteria will be tested to see whether and how the high-altitude conditions affected them.

Life on Mars

Paenibacillus xerothermodurans is one of the most resilient bacterial species we know. The team at NASA wants to discover how the bacteria react to their flight in order to learn more about whether life on Mars could possibly exist. If the low temperature, UV rays, and air conditions cause the bacteria to mutate or indeed die, we can be pretty sure that the existence of living organisms on the surface of Mars is very unlikely.

Life on Mars

What happens to the bacteria on the spacecraft and rovers we send to space? This experiment should provide some answers.

The eclipse

If you’re in the US, you might have a chance to witness the full solar eclipse today. And if you’re planning to watch, please make sure to take all precautionary measures. In a nutshell, don’t look directly at the sun. Not today, not ever.

If you’re in the UK, you can observe a partial eclipse, if the clouds decide to vanish. And again, take note of safety measures so you don’t damage your eyes.

Life on Mars

You can also watch a live-stream of the eclipse via the NASA website.

If you’ve created an eclipse-viewing Raspberry Pi project, make sure to share it with us. And while we’re talking about eclipses and balloons, check here for our coverage of the 2015 balloon launches coinciding with the UK’s partial eclipse.

The post Hunting for life on Mars assisted by high-altitude balloons appeared first on Raspberry Pi.

Analyzing AWS Cost and Usage Reports with Looker and Amazon Athena

Post Syndicated from Dillon Morrison original https://aws.amazon.com/blogs/big-data/analyzing-aws-cost-and-usage-reports-with-looker-and-amazon-athena/

This is a guest post by Dillon Morrison at Looker. Looker is, in their own words, “a new kind of analytics platform–letting everyone in your business make better decisions by getting reliable answers from a tool they can use.” 

As the breadth of AWS products and services continues to grow, customers are able to more easily move their technology stack and core infrastructure to AWS. One of the attractive benefits of AWS is the cost savings. Rather than paying upfront capital expenses for large on-premises systems, customers can instead pay variables expenses for on-demand services. To further reduce expenses AWS users can reserve resources for specific periods of time, and automatically scale resources as needed.

The AWS Cost Explorer is great for aggregated reporting. However, conducting analysis on the raw data using the flexibility and power of SQL allows for much richer detail and insight, and can be the better choice for the long term. Thankfully, with the introduction of Amazon Athena, monitoring and managing these costs is now easier than ever.

In the post, I walk through setting up the data pipeline for cost and usage reports, Amazon S3, and Athena, and discuss some of the most common levers for cost savings. I surface tables through Looker, which comes with a host of pre-built data models and dashboards to make analysis of your cost and usage data simple and intuitive.

Analysis with Athena

With Athena, there’s no need to create hundreds of Excel reports, move data around, or deploy clusters to house and process data. Athena uses Apache Hive’s DDL to create tables, and the Presto querying engine to process queries. Analysis can be performed directly on raw data in S3. Conveniently, AWS exports raw cost and usage data directly into a user-specified S3 bucket, making it simple to start querying with Athena quickly. This makes continuous monitoring of costs virtually seamless, since there is no infrastructure to manage. Instead, users can leverage the power of the Athena SQL engine to easily perform ad-hoc analysis and data discovery without needing to set up a data warehouse.

After the data pipeline is established, cost and usage data (the recommended billing data, per AWS documentation) provides a plethora of comprehensive information around usage of AWS services and the associated costs. Whether you need the report segmented by product type, user identity, or region, this report can be cut-and-sliced any number of ways to properly allocate costs for any of your business needs. You can then drill into any specific line item to see even further detail, such as the selected operating system, tenancy, purchase option (on-demand, spot, or reserved), and so on.

Walkthrough

By default, the Cost and Usage report exports CSV files, which you can compress using gzip (recommended for performance). There are some additional configuration options for tuning performance further, which are discussed below.

Prerequisites

If you want to follow along, you need the following resources:

Enable the cost and usage reports

First, enable the Cost and Usage report. For Time unit, select Hourly. For Include, select Resource IDs. All options are prompted in the report-creation window.

The Cost and Usage report dumps CSV files into the specified S3 bucket. Please note that it can take up to 24 hours for the first file to be delivered after enabling the report.

Configure the S3 bucket and files for Athena querying

In addition to the CSV file, AWS also creates a JSON manifest file for each cost and usage report. Athena requires that all of the files in the S3 bucket are in the same format, so we need to get rid of all these manifest files. If you’re looking to get started with Athena quickly, you can simply go into your S3 bucket and delete the manifest file manually, skip the automation described below, and move on to the next section.

To automate the process of removing the manifest file each time a new report is dumped into S3, which I recommend as you scale, there are a few additional steps. The folks at Concurrency labs wrote a great overview and set of scripts for this, which you can find in their GitHub repo.

These scripts take the data from an input bucket, remove anything unnecessary, and dump it into a new output bucket. We can utilize AWS Lambda to trigger this process whenever new data is dropped into S3, or on a nightly basis, or whatever makes most sense for your use-case, depending on how often you’re querying the data. Please note that enabling the “hourly” report means that data is reported at the hour-level of granularity, not that a new file is generated every hour.

Following these scripts, you’ll notice that we’re adding a date partition field, which isn’t necessary but improves query performance. In addition, converting data from CSV to a columnar format like ORC or Parquet also improves performance. We can automate this process using Lambda whenever new data is dropped in our S3 bucket. Amazon Web Services discusses columnar conversion at length, and provides walkthrough examples, in their documentation.

As a long-term solution, best practice is to use compression, partitioning, and conversion. However, for purposes of this walkthrough, we’re not going to worry about them so we can get up-and-running quicker.

Set up the Athena query engine

In your AWS console, navigate to the Athena service, and click “Get Started”. Follow the tutorial and set up a new database (we’ve called ours “AWS Optimizer” in this example). Don’t worry about configuring your initial table, per the tutorial instructions. We’ll be creating a new table for cost and usage analysis. Once you walked through the tutorial steps, you’ll be able to access the Athena interface, and can begin running Hive DDL statements to create new tables.

One thing that’s important to note, is that the Cost and Usage CSVs also contain the column headers in their first row, meaning that the column headers would be included in the dataset and any queries. For testing and quick set-up, you can remove this line manually from your first few CSV files. Long-term, you’ll want to use a script to programmatically remove this row each time a new file is dropped in S3 (every few hours typically). We’ve drafted up a sample script for ease of reference, which we run on Lambda. We utilize Lambda’s native ability to invoke the script whenever a new object is dropped in S3.

For cost and usage, we recommend using the DDL statement below. Since our data is in CSV format, we don’t need to use a SerDe, we can simply specify the “separatorChar, quoteChar, and escapeChar”, and the structure of the files (“TEXTFILE”). Note that AWS does have an OpenCSV SerDe as well, if you prefer to use that.

 

CREATE EXTERNAL TABLE IF NOT EXISTS cost_and_usage	 (
identity_LineItemId String,
identity_TimeInterval String,
bill_InvoiceId String,
bill_BillingEntity String,
bill_BillType String,
bill_PayerAccountId String,
bill_BillingPeriodStartDate String,
bill_BillingPeriodEndDate String,
lineItem_UsageAccountId String,
lineItem_LineItemType String,
lineItem_UsageStartDate String,
lineItem_UsageEndDate String,
lineItem_ProductCode String,
lineItem_UsageType String,
lineItem_Operation String,
lineItem_AvailabilityZone String,
lineItem_ResourceId String,
lineItem_UsageAmount String,
lineItem_NormalizationFactor String,
lineItem_NormalizedUsageAmount String,
lineItem_CurrencyCode String,
lineItem_UnblendedRate String,
lineItem_UnblendedCost String,
lineItem_BlendedRate String,
lineItem_BlendedCost String,
lineItem_LineItemDescription String,
lineItem_TaxType String,
product_ProductName String,
product_accountAssistance String,
product_architecturalReview String,
product_architectureSupport String,
product_availability String,
product_bestPractices String,
product_cacheEngine String,
product_caseSeverityresponseTimes String,
product_clockSpeed String,
product_currentGeneration String,
product_customerServiceAndCommunities String,
product_databaseEdition String,
product_databaseEngine String,
product_dedicatedEbsThroughput String,
product_deploymentOption String,
product_description String,
product_durability String,
product_ebsOptimized String,
product_ecu String,
product_endpointType String,
product_engineCode String,
product_enhancedNetworkingSupported String,
product_executionFrequency String,
product_executionLocation String,
product_feeCode String,
product_feeDescription String,
product_freeQueryTypes String,
product_freeTrial String,
product_frequencyMode String,
product_fromLocation String,
product_fromLocationType String,
product_group String,
product_groupDescription String,
product_includedServices String,
product_instanceFamily String,
product_instanceType String,
product_io String,
product_launchSupport String,
product_licenseModel String,
product_location String,
product_locationType String,
product_maxIopsBurstPerformance String,
product_maxIopsvolume String,
product_maxThroughputvolume String,
product_maxVolumeSize String,
product_maximumStorageVolume String,
product_memory String,
product_messageDeliveryFrequency String,
product_messageDeliveryOrder String,
product_minVolumeSize String,
product_minimumStorageVolume String,
product_networkPerformance String,
product_operatingSystem String,
product_operation String,
product_operationsSupport String,
product_physicalProcessor String,
product_preInstalledSw String,
product_proactiveGuidance String,
product_processorArchitecture String,
product_processorFeatures String,
product_productFamily String,
product_programmaticCaseManagement String,
product_provisioned String,
product_queueType String,
product_requestDescription String,
product_requestType String,
product_routingTarget String,
product_routingType String,
product_servicecode String,
product_sku String,
product_softwareType String,
product_storage String,
product_storageClass String,
product_storageMedia String,
product_technicalSupport String,
product_tenancy String,
product_thirdpartySoftwareSupport String,
product_toLocation String,
product_toLocationType String,
product_training String,
product_transferType String,
product_usageFamily String,
product_usagetype String,
product_vcpu String,
product_version String,
product_volumeType String,
product_whoCanOpenCases String,
pricing_LeaseContractLength String,
pricing_OfferingClass String,
pricing_PurchaseOption String,
pricing_publicOnDemandCost String,
pricing_publicOnDemandRate String,
pricing_term String,
pricing_unit String,
reservation_AvailabilityZone String,
reservation_NormalizedUnitsPerReservation String,
reservation_NumberOfReservations String,
reservation_ReservationARN String,
reservation_TotalReservedNormalizedUnits String,
reservation_TotalReservedUnits String,
reservation_UnitsPerReservation String,
resourceTags_userName String,
resourceTags_usercostcategory String  


)
    ROW FORMAT DELIMITED
      FIELDS TERMINATED BY ','
      ESCAPED BY '\\'
      LINES TERMINATED BY '\n'

STORED AS TEXTFILE
    LOCATION 's3://<<your bucket name>>';

Once you’ve successfully executed the command, you should see a new table named “cost_and_usage” with the below properties. Now we’re ready to start executing queries and running analysis!

Start with Looker and connect to Athena

Setting up Looker is a quick process, and you can try it out for free here (or download from Amazon Marketplace). It takes just a few seconds to connect Looker to your Athena database, and Looker comes with a host of pre-built data models and dashboards to make analysis of your cost and usage data simple and intuitive. After you’re connected, you can use the Looker UI to run whatever analysis you’d like. Looker translates this UI to optimized SQL, so any user can execute and visualize queries for true self-service analytics.

Major cost saving levers

Now that the data pipeline is configured, you can dive into the most popular use cases for cost savings. In this post, I focus on:

  • Purchasing Reserved Instances vs. On-Demand Instances
  • Data transfer costs
  • Allocating costs over users or other Attributes (denoted with resource tags)

On-Demand, Spot, and Reserved Instances

Purchasing Reserved Instances vs On-Demand Instances is arguably going to be the biggest cost lever for heavy AWS users (Reserved Instances run up to 75% cheaper!). AWS offers three options for purchasing instances:

  • On-Demand—Pay as you use.
  • Spot (variable cost)—Bid on spare Amazon EC2 computing capacity.
  • Reserved Instances—Pay for an instance for a specific, allotted period of time.

When purchasing a Reserved Instance, you can also choose to pay all-upfront, partial-upfront, or monthly. The more you pay upfront, the greater the discount.

If your company has been using AWS for some time now, you should have a good sense of your overall instance usage on a per-month or per-day basis. Rather than paying for these instances On-Demand, you should try to forecast the number of instances you’ll need, and reserve them with upfront payments.

The total amount of usage with Reserved Instances versus overall usage with all instances is called your coverage ratio. It’s important not to confuse your coverage ratio with your Reserved Instance utilization. Utilization represents the amount of reserved hours that were actually used. Don’t worry about exceeding capacity, you can still set up Auto Scaling preferences so that more instances get added whenever your coverage or utilization crosses a certain threshold (we often see a target of 80% for both coverage and utilization among savvy customers).

Calculating the reserved costs and coverage can be a bit tricky with the level of granularity provided by the cost and usage report. The following query shows your total cost over the last 6 months, broken out by Reserved Instance vs other instance usage. You can substitute the cost field for usage if you’d prefer. Please note that you should only have data for the time period after the cost and usage report has been enabled (though you can opt for up to 3 months of historical data by contacting your AWS Account Executive). If you’re just getting started, this query will only show a few days.

 

SELECT 
	DATE_FORMAT(from_iso8601_timestamp(cost_and_usage.lineitem_usagestartdate),'%Y-%m') AS "cost_and_usage.usage_start_month",
	COALESCE(SUM(cost_and_usage.lineitem_unblendedcost ), 0) AS "cost_and_usage.total_unblended_cost",
	COALESCE(SUM(CASE WHEN (CASE
         WHEN cost_and_usage.lineitem_lineitemtype = 'DiscountedUsage' THEN 'RI Line Item'
         WHEN cost_and_usage.lineitem_lineitemtype = 'RIFee' THEN 'RI Line Item'
         WHEN cost_and_usage.lineitem_lineitemtype = 'Fee' THEN 'RI Line Item'
         ELSE 'Non RI Line Item'
        END = 'RI Line Item') THEN cost_and_usage.lineitem_unblendedcost  ELSE NULL END), 0) AS "cost_and_usage.total_reserved_unblended_cost",
	1.0 * (COALESCE(SUM(CASE WHEN (CASE
         WHEN cost_and_usage.lineitem_lineitemtype = 'DiscountedUsage' THEN 'RI Line Item'
         WHEN cost_and_usage.lineitem_lineitemtype = 'RIFee' THEN 'RI Line Item'
         WHEN cost_and_usage.lineitem_lineitemtype = 'Fee' THEN 'RI Line Item'
         ELSE 'Non RI Line Item'
        END = 'RI Line Item') THEN cost_and_usage.lineitem_unblendedcost  ELSE NULL END), 0)) / NULLIF((COALESCE(SUM(cost_and_usage.lineitem_unblendedcost ), 0)),0)  AS "cost_and_usage.percent_spend_on_ris",
	COALESCE(SUM(CASE WHEN (CASE
         WHEN cost_and_usage.lineitem_lineitemtype = 'DiscountedUsage' THEN 'RI Line Item'
         WHEN cost_and_usage.lineitem_lineitemtype = 'RIFee' THEN 'RI Line Item'
         WHEN cost_and_usage.lineitem_lineitemtype = 'Fee' THEN 'RI Line Item'
         ELSE 'Non RI Line Item'
        END = 'Non RI Line Item') THEN cost_and_usage.lineitem_unblendedcost  ELSE NULL END), 0) AS "cost_and_usage.total_non_reserved_unblended_cost",
	1.0 * (COALESCE(SUM(CASE WHEN (CASE
         WHEN cost_and_usage.lineitem_lineitemtype = 'DiscountedUsage' THEN 'RI Line Item'
         WHEN cost_and_usage.lineitem_lineitemtype = 'RIFee' THEN 'RI Line Item'
         WHEN cost_and_usage.lineitem_lineitemtype = 'Fee' THEN 'RI Line Item'
         ELSE 'Non RI Line Item'
        END = 'Non RI Line Item') THEN cost_and_usage.lineitem_unblendedcost  ELSE NULL END), 0)) / NULLIF((COALESCE(SUM(cost_and_usage.lineitem_unblendedcost ), 0)),0)  AS "cost_and_usage.percent_spend_on_non_ris"
FROM aws_optimizer.cost_and_usage  AS cost_and_usage

WHERE 
	(((from_iso8601_timestamp(cost_and_usage.lineitem_usagestartdate)) >= ((DATE_ADD('month', -5, DATE_TRUNC('MONTH', CAST(NOW() AS DATE))))) AND (from_iso8601_timestamp(cost_and_usage.lineitem_usagestartdate)) < ((DATE_ADD('month', 6, DATE_ADD('month', -5, DATE_TRUNC('MONTH', CAST(NOW() AS DATE))))))))
GROUP BY 1
ORDER BY 2 DESC
LIMIT 500

The resulting table should look something like the image below (I’m surfacing tables through Looker, though the same table would result from querying via command line or any other interface).

With a BI tool, you can create dashboards for easy reference and monitoring. New data is dumped into S3 every few hours, so your dashboards can update several times per day.

It’s an iterative process to understand the appropriate number of Reserved Instances needed to meet your business needs. After you’ve properly integrated Reserved Instances into your purchasing patterns, the savings can be significant. If your coverage is consistently below 70%, you should seriously consider adjusting your purchase types and opting for more Reserved instances.

Data transfer costs

One of the great things about AWS data storage is that it’s incredibly cheap. Most charges often come from moving and processing that data. There are several different prices for transferring data, broken out largely by transfers between regions and availability zones. Transfers between regions are the most costly, followed by transfers between Availability Zones. Transfers within the same region and same availability zone are free unless using elastic or public IP addresses, in which case there is a cost. You can find more detailed information in the AWS Pricing Docs. With this in mind, there are several simple strategies for helping reduce costs.

First, since costs increase when transferring data between regions, it’s wise to ensure that as many services as possible reside within the same region. The more you can localize services to one specific region, the lower your costs will be.

Second, you should maximize the data you’re routing directly within AWS services and IP addresses. Transfers out to the open internet are the most costly and least performant mechanisms of data transfers, so it’s best to keep transfers within AWS services.

Lastly, data transfers between private IP addresses are cheaper than between elastic or public IP addresses, so utilizing private IP addresses as much as possible is the most cost-effective strategy.

The following query provides a table depicting the total costs for each AWS product, broken out transfer cost type. Substitute the “lineitem_productcode” field in the query to segment the costs by any other attribute. If you notice any unusually high spikes in cost, you’ll need to dig deeper to understand what’s driving that spike: location, volume, and so on. Drill down into specific costs by including “product_usagetype” and “product_transfertype” in your query to identify the types of transfer costs that are driving up your bill.

SELECT 
	cost_and_usage.lineitem_productcode  AS "cost_and_usage.product_code",
	COALESCE(SUM(cost_and_usage.lineitem_unblendedcost), 0) AS "cost_and_usage.total_unblended_cost",
	COALESCE(SUM(CASE WHEN REGEXP_LIKE(cost_and_usage.product_usagetype, 'DataTransfer')    THEN cost_and_usage.lineitem_unblendedcost  ELSE NULL END), 0) AS "cost_and_usage.total_data_transfer_cost",
	COALESCE(SUM(CASE WHEN REGEXP_LIKE(cost_and_usage.product_usagetype, 'DataTransfer-In')    THEN cost_and_usage.lineitem_unblendedcost  ELSE NULL END), 0) AS "cost_and_usage.total_inbound_data_transfer_cost",
	COALESCE(SUM(CASE WHEN REGEXP_LIKE(cost_and_usage.product_usagetype, 'DataTransfer-Out')    THEN cost_and_usage.lineitem_unblendedcost  ELSE NULL END), 0) AS "cost_and_usage.total_outbound_data_transfer_cost"
FROM aws_optimizer.cost_and_usage  AS cost_and_usage

WHERE 
	(((from_iso8601_timestamp(cost_and_usage.lineitem_usagestartdate)) >= ((DATE_ADD('month', -5, DATE_TRUNC('MONTH', CAST(NOW() AS DATE))))) AND (from_iso8601_timestamp(cost_and_usage.lineitem_usagestartdate)) < ((DATE_ADD('month', 6, DATE_ADD('month', -5, DATE_TRUNC('MONTH', CAST(NOW() AS DATE))))))))
GROUP BY 1
ORDER BY 2 DESC
LIMIT 500

When moving between regions or over the open web, many data transfer costs also include the origin and destination location of the data movement. Using a BI tool with mapping capabilities, you can get a nice visual of data flows. The point at the center of the map is used to represent external data flows over the open internet.

Analysis by tags

AWS provides the option to apply custom tags to individual resources, so you can allocate costs over whatever customized segment makes the most sense for your business. For a SaaS company that hosts software for customers on AWS, maybe you’d want to tag the size of each customer. The following query uses custom tags to display the reserved, data transfer, and total cost for each AWS service, broken out by tag categories, over the last 6 months. You’ll want to substitute the cost_and_usage.resourcetags_customersegment and cost_and_usage.customer_segment with the name of your customer field.

 

SELECT * FROM (
SELECT *, DENSE_RANK() OVER (ORDER BY z___min_rank) as z___pivot_row_rank, RANK() OVER (PARTITION BY z__pivot_col_rank ORDER BY z___min_rank) as z__pivot_col_ordering FROM (
SELECT *, MIN(z___rank) OVER (PARTITION BY "cost_and_usage.product_code") as z___min_rank FROM (
SELECT *, RANK() OVER (ORDER BY CASE WHEN z__pivot_col_rank=1 THEN (CASE WHEN "cost_and_usage.total_unblended_cost" IS NOT NULL THEN 0 ELSE 1 END) ELSE 2 END, CASE WHEN z__pivot_col_rank=1 THEN "cost_and_usage.total_unblended_cost" ELSE NULL END DESC, "cost_and_usage.total_unblended_cost" DESC, z__pivot_col_rank, "cost_and_usage.product_code") AS z___rank FROM (
SELECT *, DENSE_RANK() OVER (ORDER BY CASE WHEN "cost_and_usage.customer_segment" IS NULL THEN 1 ELSE 0 END, "cost_and_usage.customer_segment") AS z__pivot_col_rank FROM (
SELECT 
	cost_and_usage.lineitem_productcode  AS "cost_and_usage.product_code",
	cost_and_usage.resourcetags_customersegment  AS "cost_and_usage.customer_segment",
	COALESCE(SUM(cost_and_usage.lineitem_unblendedcost ), 0) AS "cost_and_usage.total_unblended_cost",
	1.0 * (COALESCE(SUM(CASE WHEN REGEXP_LIKE(cost_and_usage.product_usagetype, 'DataTransfer')    THEN cost_and_usage.lineitem_unblendedcost  ELSE NULL END), 0)) / NULLIF((COALESCE(SUM(cost_and_usage.lineitem_unblendedcost ), 0)),0)  AS "cost_and_usage.percent_spend_data_transfers_unblended",
	1.0 * (COALESCE(SUM(CASE WHEN (CASE
         WHEN cost_and_usage.lineitem_lineitemtype = 'DiscountedUsage' THEN 'RI Line Item'
         WHEN cost_and_usage.lineitem_lineitemtype = 'RIFee' THEN 'RI Line Item'
         WHEN cost_and_usage.lineitem_lineitemtype = 'Fee' THEN 'RI Line Item'
         ELSE 'Non RI Line Item'
        END = 'Non RI Line Item') THEN cost_and_usage.lineitem_unblendedcost  ELSE NULL END), 0)) / NULLIF((COALESCE(SUM(cost_and_usage.lineitem_unblendedcost ), 0)),0)  AS "cost_and_usage.unblended_percent_spend_on_ris"
FROM aws_optimizer.cost_and_usage_raw  AS cost_and_usage

WHERE 
	(((from_iso8601_timestamp(cost_and_usage.lineitem_usagestartdate)) >= ((DATE_ADD('month', -5, DATE_TRUNC('MONTH', CAST(NOW() AS DATE))))) AND (from_iso8601_timestamp(cost_and_usage.lineitem_usagestartdate)) < ((DATE_ADD('month', 6, DATE_ADD('month', -5, DATE_TRUNC('MONTH', CAST(NOW() AS DATE))))))))
GROUP BY 1,2) ww
) bb WHERE z__pivot_col_rank <= 16384
) aa
) xx
) zz
 WHERE z___pivot_row_rank <= 500 OR z__pivot_col_ordering = 1 ORDER BY z___pivot_row_rank

The resulting table in this example looks like the results below. In this example, you can tell that we’re making poor use of Reserved Instances because they represent such a small portion of our overall costs.

Again, using a BI tool to visualize these costs and trends over time makes the analysis much easier to consume and take action on.

Summary

Saving costs on your AWS spend is always an iterative, ongoing process. Hopefully with these queries alone, you can start to understand your spending patterns and identify opportunities for savings. However, this is just a peek into the many opportunities available through analysis of the Cost and Usage report. Each company is different, with unique needs and usage patterns. To achieve maximum cost savings, we encourage you to set up an analytics environment that enables your team to explore all potential cuts and slices of your usage data, whenever it’s necessary. Exploring different trends and spikes across regions, services, user types, etc. helps you gain comprehensive understanding of your major cost levers and consistently implement new cost reduction strategies.

Note that all of the queries and analysis provided in this post were generated using the Looker data platform. If you’re already a Looker customer, you can get all of this analysis, additional pre-configured dashboards, and much more using Looker Blocks for AWS.

About the Author

Dillon Morrison leads the Platform Ecosystem at Looker. He enjoys exploring new technologies and architecting the most efficient data solutions for the business needs of his company and their customers. In his spare time, you’ll find Dillon rock climbing in the Bay Area or nose deep in the docs of the latest AWS product release at his favorite cafe (“Arlequin in SF is unbeatable!”).

 

 

 

AWS CloudHSM Update – Cost Effective Hardware Key Management at Cloud Scale for Sensitive & Regulated Workloads

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/aws-cloudhsm-update-cost-effective-hardware-key-management/

Our customers run an incredible variety of mission-critical workloads on AWS, many of which process and store sensitive data. As detailed in our Overview of Security Processes document, AWS customers have access to an ever-growing set of options for encrypting and protecting this data. For example, Amazon Relational Database Service (RDS) supports encryption of data at rest and in transit, with options tailored for each supported database engine (MySQL, SQL Server, Oracle, MariaDB, PostgreSQL, and Aurora).

Many customers use AWS Key Management Service (KMS) to centralize their key management, with others taking advantage of the hardware-based key management, encryption, and decryption provided by AWS CloudHSM to meet stringent security and compliance requirements for their most sensitive data and regulated workloads (you can read my post, AWS CloudHSM – Secure Key Storage and Cryptographic Operations, to learn more about Hardware Security Modules, also known as HSMs).

Major CloudHSM Update
Today, building on what we have learned from our first-generation product, we are making a major update to CloudHSM, with a set of improvements designed to make the benefits of hardware-based key management available to a much wider audience while reducing the need for specialized operating expertise. Here’s a summary of the improvements:

Pay As You Go – CloudHSM is now offered under a pay-as-you-go model that is simpler and more cost-effective, with no up-front fees.

Fully Managed – CloudHSM is now a scalable managed service; provisioning, patching, high availability, and backups are all built-in and taken care of for you. Scheduled backups extract an encrypted image of your HSM from the hardware (using keys that only the HSM hardware itself knows) that can be restored only to identical HSM hardware owned by AWS. For durability, those backups are stored in Amazon Simple Storage Service (S3), and for an additional layer of security, encrypted again with server-side S3 encryption using an AWS KMS master key.

Open & Compatible  – CloudHSM is open and standards-compliant, with support for multiple APIs, programming languages, and cryptography extensions such as PKCS #11, Java Cryptography Extension (JCE), and Microsoft CryptoNG (CNG). The open nature of CloudHSM gives you more control and simplifies the process of moving keys (in encrypted form) from one CloudHSM to another, and also allows migration to and from other commercially available HSMs.

More Secure – CloudHSM Classic (the original model) supports the generation and use of keys that comply with FIPS 140-2 Level 2. We’re stepping that up a notch today with support for FIPS 140-2 Level 3, with security mechanisms that are designed to detect and respond to physical attempts to access or modify the HSM. Your keys are protected with exclusive, single-tenant access to tamper-resistant HSMs that appear within your Virtual Private Clouds (VPCs). CloudHSM supports quorum authentication for critical administrative and key management functions. This feature allows you to define a list of N possible identities that can access the functions, and then require at least M of them to authorize the action. It also supports multi-factor authentication using tokens that you provide.

AWS-Native – The updated CloudHSM is an integral part of AWS and plays well with other tools and services. You can create and manage a cluster of HSMs using the AWS Management Console, AWS Command Line Interface (CLI), or API calls.

Diving In
You can create CloudHSM clusters that contain 1 to 32 HSMs, each in a separate Availability Zone in a particular AWS Region. Spreading HSMs across AZs gives you high availability (including built-in load balancing); adding more HSMs gives you additional throughput. The HSMs within a cluster are kept in sync: performing a task or operation on one HSM in a cluster automatically updates the others. Each HSM in a cluster has its own Elastic Network Interface (ENI).

All interaction with an HSM takes place via the AWS CloudHSM client. It runs on an EC2 instance and uses certificate-based mutual authentication to create secure (TLS) connections to the HSMs.

At the hardware level, each HSM includes hardware-enforced isolation of crypto operations and key storage. Each customer HSM runs on dedicated processor cores.

Setting Up a Cluster
Let’s set up a cluster using the CloudHSM Console:

I click on Create cluster to get started, select my desired VPC and the subnets within it (I can also create a new VPC and/or subnets if needed):

Then I review my settings and click on Create:

After a few minutes, my cluster exists, but is uninitialized:

Initialization simply means retrieving a certificate signing request (the Cluster CSR):

And then creating a private key and using it to sign the request (these commands were copied from the Initialize Cluster docs and I have omitted the output. Note that ID identifies the cluster):

$ openssl genrsa -out CustomerRoot.key 2048
$ openssl req -new -x509 -days 365 -key CustomerRoot.key -out CustomerRoot.crt
$ openssl x509 -req -days 365 -in ID_ClusterCsr.csr   \
                              -CA CustomerRoot.crt    \
                              -CAkey CustomerRoot.key \
                              -CAcreateserial         \
                              -out ID_CustomerHsmCertificate.crt

The next step is to apply the signed certificate to the cluster using the console or the CLI. After this has been done, the cluster can be activated by changing the password for the HSM’s administrative user, otherwise known as the Crypto Officer (CO).

Once the cluster has been created, initialized and activated, it can be used to protect data. Applications can use the APIs in AWS CloudHSM SDKs to manage keys, encrypt & decrypt objects, and more. The SDKs provide access to the CloudHSM client (running on the same instance as the application). The client, in turn, connects to the cluster across an encrypted connection.

Available Today
The new HSM is available today in the US East (Northern Virginia), US West (Oregon), US East (Ohio), and EU (Ireland) Regions, with more in the works. Pricing starts at $1.45 per HSM per hour.

— Jeff;

[$] Scaling the kernel’s MAINTAINERS file

Post Syndicated from corbet original https://lwn.net/Articles/730509/rss

The kernel’s development community is large, to the point that it is often
far from obvious who a given patch should be sent to. As the community has
grown, it has developed mechanisms for tracking that information centered
on a text file called MAINTAINERS. But now it would appear that
this scalability mechanism has scalability problems of its own.

Hotspot Shield VPN Reported to FTC For Alleged Privacy Breaches

Post Syndicated from Andy original https://torrentfreak.com/hotspot-shield-vpn-reported-to-ftc-for-alleged-privacy-breaches-170807/

With online privacy becoming an increasingly hot topic, large numbers of companies are offering products which claim to stop third-parties from snooping on users’ Internet activities.

At the forefront are Virtual Private Networks (VPN), which push consumer traffic through encrypted tunnels and remote servers to hide activity from ISPs while offering varying levels of anonymity.

Claims made by VPN companies are often scrutinized by privacy advocates but if a complaint filed this morning by the Center for Democracy and Technology
(CDT) gains momentum, there could be a government investigation into one of the most popular.

Developed by AnchorFree, Inc. and initially released more than nine years ago, the Hotspot Shield application allows users to connect to a VPN service. According to its makers, it’s been downloaded 75 million times and provides “anonymous web surfing with complete privacy.” That claim, however, is now under the spotlight.

In a complaint filed this morning with the Federal Trade Commission, CDT notes that Hotspot Shield makes “strong claims” about the privacy and security of its data collection and sharing practices, including that it “never logs or stores user data.” Crucially, the company also claims never to track or sell its customers’ information, adding that security and privacy are “guaranteed.”

Countering, CDT says that Hotspot Shield engages in logging practices that contradict its claims, noting that it collects information to “identify [a user’s] general location, improve the Service, or optimize advertisements displayed through the Service.”

The complaint says that IP addresses and unique device identifiers are regularly
collected by Hotspot Shield but the service gets around this issue by classing neither sets of data as personal information.

CDT says it used Carnegie Mellon University’s Mobile App Compliance System to gain insight into Hotspot Shield’s functionality and found problems with privacy.

“CMU’s analysis of Hotspot Shield’s Android application permissions found undisclosed data sharing practices with third party advertising networks,” the group notes.

“While an ad-supported VPN may be beneficial in certain instances, it should not be paired with a product or service that tells users that it ensures anonymity, privacy, and security.”

CDT also says that Hotspot Shield tries to cover its back with a disclaimer that the company “may not provide a virtual IP Address for every web site you may visit and third-party web sites may receive your original IP Address when you are visiting those web sites.” But this runs counter to the stated aim of the service, CDT writes.

Accusing Hotspot Shield of unfair and deceptive trade practices, CDT calls on the Commission to conduct an investigation into its data collection and sharing practices.

Hotspot Shield is yet to respond to the complaint or accusations but in a 2014 blog post, welcomed the FTC’s involvement in online security issues.

Full complaint here, courtesy Ars

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

Ethereum, Proof-of-Stake… and the consequences

Post Syndicated from Григор original http://www.gatchev.info/blog/?p=2070

For those who have been living the last few years in a cave without Internet: Ethereum is a cryptocurrency project, based around the coin Ether. It has the support of many big banks, big hedge funds and some states (Russia, China etc). Among the cryptocurrencies, it is second only to Bitcoin – and might even overtake it with the time. (Especially if Bitcoin doesn’t finally move and fix some of its problems.)

Ethereum offers some abilities that few other cryptocurrencies do. The most important one is the support for “smart projects” – kind of electronic contracts that can easily be executed and enforced with little to no human participation. This post however is dedicated to another of its traits – the Proof of Stake.

To work and exist, every cryptocurrency depends on some proof. Most of them use Proof-of-Work scheme. In it, one has to put some work – eg. calculating checksums – behind its participation in the network and its decision, and receive newly generated coins for it. This however results in huge amount of work done only to prove that, well, you can do it and deserve to be in and receive some of the newly squeezed juice.

As of August 2017, Ethereum uses this scheme too. However, they plan to switch to a Proof-of-Stake algorithm named Casper. In it, you prove yourself not by doing work, but by proving to own Ether. As this requires practically no work, it is much more technically effective than the Proof-of-Work schemes.

Technically, Caspar is an amazing design. I congratulate the Ethereum team for it. However, economically its usage appears to have an important weakness. It is described below.

—-

A polarized system

With Casper, the Ether generated by the Ethereum network and the decision power in it are distributed to these who already own Ether. As a consequence, most of both go to those who own most Ether. (There might be attempts to limit that, but these are easily defeatable. For example, limiting the amount distributed to an address can be circumvented by a Sybil attack.)

Such a distribution will create with the time a financial ecosystem where most money and vote are held by a small minority of the participants. The big majority will have little to no of both – it will summarily hold less money and vote than the minority of “haves”. Giving the speed with which the cryptocurrency systems evolve, it is realistic to expect this development in ten, maybe even in five or less years after introducing Casper.

The “middle class”

Economists love to repeat how important is to have a strong middle class. Why, and how that translates to the situation in a cryptocurrency-based financial system?

In systemic terms, “middle class” denotes in a financial system the set of entities that control each a noticeable but not very big amount of resources.

Game theory shows that in a financial system, entities with different clout usually have different interests. These interests usually reflect the amount of resources they control. Entities with little to no resources tend to have interests opposing to these with biggest resources – especially in systems where the total amount of resources changes slowly and the economics is close to a zero-sum game. (For example, in most cryptocurrency systems.) The “middle class” entities interests in most aspects are in the middle.

For an economics to work, there must be a balance of interests that creates incentive for all of its members to participate. In financial systems, where “haves” interests are mostly opposing to “have-nots” interests, creating such a balance depends on the presence and influence of a “middle class”. Its interests are usually the closest to a compromise that satisfies all, and its influence is the key to achieving that compromise within the system.

If the system state is not acceptable for all entities, these who do not accept it eventually leave. (Usually their participation is required for the system survival, so this brings the system down.) If these entities cannot leave the system, they ultimately reject its rules and try to change it by force. If that is impossible too, they usually resort to denying the system what makes them useful for it, thus decreasing its competitiveness to other systems.

The most reliable way to have acceptable compromise enforced in a system is to have in it a “middle class” that summarily controls more resources than any other segment of entities, preferably at least 51% of the system resources. (This assumes that the “middle class” is able and willing to protect their interests. If some of these entities are controlled into defending someone else’s interests – eg. botnets in computer networks, manipulated voters during elections, etc – these numbers apply to the non-controlled among them.)

A system that doesn’t have a non-controlled “middle class” that controls a decisive amount of resources, usually does not have an influential set of interests that are an acceptable compromise between the interests poles. For this reason, it can be called a polarized system.

The limitation on development

In a polarized system, the incentive for development is minimized. (Development is potentially disruptive, and the majority of the financial abilities and the decision power there has only to lose from a disruption. When factoring in the expected profits from development, the situation always becomes a zero-sum game.) The system becomes static (thus cementing the zero-sum game situation in it) and is under threat of being overtaken by a competing financial system. When that happens, it is usually destroyed with all stakes in it.

Also, almost any initiative in such a financial system is bound to turn into a cartel, oligopoly or monopoly, due to the small number of participants with resources to start and support an initiative. That effectively destroys its markets, contributing to the weakness of the system and limiting further its ability to develop.

Another problem that stems from this is that the incentive during an interaction to violate the rules and to push the contragent into a loss is greater than the incentive to compete by giving a better offer. This in turn removes the incentive to increase productivity, which is a key incentive for development.)

Yet another problem of the concentration of most resources into few entities is the increased gain from attacking one of them and appropriating their resources, and thus the incentive to do it. Since good defensive capabilities are usually an excellent offense base, this pulls the “haves” into an “arms race”, redirecting more and more of their resources into defense. This also leaves the development outside the arms race increasingly resource-strapped. (The “arms race” itself generates development, but the race situation prevents that into trickling into “non-military” applications.)

These are only a part of the constraints on development in a polarized system. Listing all of them will make a long read.

Trickle-up and trickle-down

In theory, every economical system involves two processes: trickle-down and trickle-up. So, any concentration of resources on the top should be decreased by an automatically increased trickle-down. However, a better understanding how these processes work shows that this logic is faulty.

Any financial exchange in a system consists of two parts. One of them covers the actual production cost of whatever resource is being exchanged against the finances. The other part is the profit of the entity that obtains the finances. From the viewpoint of that entity, the first part vs. the resource given is zero-sum – its incentive to participate in this exchange is the second part, the profit. That second part is effectively the trickle in the system, as it is the only resource really gained.

The direction and the size of the trickle ultimately depends on the balance of many factors, some of them random, others constant. On the long run, it is the constant factors that determine the size and the direction of the trickle sum.

The most important constant factor is the benefit of scale (BOS). It dictates that the bigger entities are able to pull the balance to their side more strongly than the smaller ones. Some miss that chance, but others use it. It makes the trickle-up stronger than the trickle-down. In a system where the transaction outcome is close to a zero-sum game, this concentrates all resources at the top with a speed depending on the financial interactions volume per an unit of time.

(Actually the formula is a bit more complex. All dynamic entities – eg. living organisms, active companies etc – have an “existence maintenance” expense, which they cannot avoid. However, the amount of resources in a system above the summary existence maintenance follows the simple rule above. And these are the only resources that are available for investing in anything, eg. development.)

In the real-life systems the BOS power is limited. There are many different random factors that compete with and influence one another, some of them outweighing BOS. Also, in every moment some factors lose importance and / or cease to exist, while others appear and / or gain importance. The complexity of this system makes any attempt by an entity or entities pool to take control over it hard and slow. This gives the other entities time and ways to react and try to block the takeover attempt. Also, the real-life systems have many built-in constraints against scale-based takeovers – anti-trust laws, separation of the government powers, enforced financial trickle-down through taxes on the rich and benefits for the poor, etc. All these together manage to prevent most takeover attempts, or to limit them into only a segment of the system.

How a Proof-of-Stake based cryptocurrency fares at these?

A POS-based cryptocurrency financial system has no constraints against scale-based takeovers. It has only one kind of clout – the amount of resources controlled by an entity. This kind of clout is built in it, has all the importance in it and cannot lose that or disappear. It has no other types of resources, and has no slowing due to complexity. It is not segmented – who has these resources has it all. There are no built-in constraints against scale-based takeovers, or mechanisms to strengthen resource trickle-down. In short, it is the ideal ground for creating a polarized financial system.

So, it would be only logical to expect that a Proof-of-Stake based Ether financial system will suffer by the problems a polarized system presents. Despite all of its technical ingenuity, its longer-term financial usability is limited, and the participation in it may be dangerous to any entity smaller than eg. a big bank, a big hedge fund or a big authoritarian state.

All fixes for this problem I could think of by now would be easily beaten by simple attacks. I am not sure if it is possible to have a reliable solution to it at all.

Do smart contracts and secondary tokens change this?

Unhappily, no. Smart contracts are based on having Ether, and need Ether to exist and act. Thus, they are bound to the financial situation of the Ether financial system, and are influenced by it. The bigger is the scope of the smart contract, the bigger is its dependence on the Ether situation.

Due to this, smart contracts of meaningful size will find themselves hampered and maybe even endangered by a polarization in the financial system powered by POS-based Ethereum. It is technically possible to migrate these contracts to a competing underlying system, but it won’t be easy – probably even when the competing system is technically a clone of Ethereum, like Ethereum Classic. The migration cost might exceed the migration benefits at any given stage of the contract project development, even if the total migration benefits are far larger than this cost.

Eventually this problem might become public knowledge and most projects in need of a smart contract might start avoiding Ethereum. This will lead to decreased interest in participation in the Ethereum ecosystem, to a loss of market cap, and eventually maybe even to the demise of this technically great project.

Other dangers

There is a danger that the “haves” minority in a polarized system might start actively investing resources in creating other systems that suffer from the same problem (as they benefit from it), or in modifying existing systems in this direction. This might decrease the potential for development globally. As some of the backers of Ethereum are entities with enormous clout worldwide, that negative influence on the global system might be significant.

Landmine-clearing Pi-powered C-Turtle

Post Syndicated from Janina Ander original https://www.raspberrypi.org/blog/landmine-c-turtle/

In an effort to create a robot that can teach itself to navigate different terrains, scientists at Arizona State University have built C-Turtle, a Raspberry Pi-powered autonomous cardboard robot with turtle flippers. This is excellent news for people who live in areas with landmines: C-Turtle is a great alternative to current landmine-clearing robots, since it is much cheaper, and much easier to assemble.

C-Turtle ASU

Photo by Charlie Leight/ASU Now

Why turtle flippers?

As any user of Python will tell you*, turtles are amazing. Moreover, as the evolutionary biologist of the C-Turtle team, Andrew Jansen, will tell you, considering their bulk** turtles move very well on land with the help of their flippers. Consequently, the team tried out prototypes with cardboard flippers imitating the shape of turtle flippers. Then they compared their performance to that of prototypes with rectangular or oval ‘flippers’. And 157 million years of evolution*** won out: the robots with turtle flippers were best at moving forward.

C-Turtle ASU

Field testing with Assistant Professor Heni Ben Amor, one of the C-Turtle team’s leaders (Photo by Charlie Leight/ASU Now)

If it walks like a C-Turtle…

But the scientists didn’t just slap turtle flippers on their robot and then tell it to move like a turtle! Instead, they implemented machine learning algorithms on the Pi Zero that serves as C-Turtle’s brain, and then simply let the robot do its thing. Left to its own devices, it used the reward and punishment mechanisms of its algorithms to learn the most optimal way of propelling itself forward. And lo and behold, C-Turtle taught itself to move just like a live turtle does!

Robotic C-Turtle

This is “Robotic C-Turtle” by ASU Now on Vimeo, the home for high quality videos and the people who love them.

Landmine clearance with C-Turtle

Robots currently used to clear landmines are very expensive, since they are built to withstand multiple mine explosions. Conversely, the total cost of C-Turtle comes to about $70 (~£50) – that’s cheap enough to make it disposable. It is also more easily assembled, it doesn’t need to be remotely controlled, and it can learn to navigate new terrains. All this makes it perfect for clearing minefields.

BBC Click on Twitter

Meet C-Turtle, the landmine detecting robot. VIDEO https://t.co/Kjc6WxRC8I

C-Turtles in space?****

The researchers hope that robots similar to C-Turtle can used for space exploration. They found that the C-Turtle prototypes that had performed very well in the sandpits in their lab didn’t really do as well when they were released in actual desert conditions. By analogy, robots optimized for simulated planetary conditions might not actually perform well on-site. The ASU scientists imagine that C-Turtle materials and a laser cutter for the cardboard body could be carried on board a Mars mission. Then Martian C-Turtle design could be optimized after landing, and the robot could teach itself how best to navigate real Martian terrain.

There are already Raspberry Pis in space – imagine if they actually made it to Mars! Dave would never recover…

Congrats to Assistant Professors Heni Ben Amor and Daniel Aukes, and to the rest of the C-Turtle team, on their achievement! We at Pi Towers are proud that our little computer is part of this amazing project.

C-Turtle ASU

Photo by Charlie Leight/ASU Now

* Check out our Turtley amazing resource to find out why!

** At a length of 7ft, leatherback sea turtles can weigh 1,500lb!

*** That’s right: turtles survived the extinction of the dinosaurs!

**** Is anyone else thinking of Great A’Tuin right now? Anyone? Just me? Oh well.

The post Landmine-clearing Pi-powered C-Turtle appeared first on Raspberry Pi.

Hacking a Segway

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2017/07/hacking_a_segwa.html

The Segway has a mobile app. It is hackable:

While analyzing the communication between the app and the Segway scooter itself, Kilbride noticed that a user PIN number meant to protect the Bluetooth communication from unauthorized access wasn’t being used for authentication at every level of the system. As a result, Kilbride could send arbitrary commands to the scooter without needing the user-chosen PIN.

He also discovered that the hoverboard’s software update platform didn’t have a mechanism in place to confirm that firmware updates sent to the device were really from Segway (often called an “integrity check”). This meant that in addition to sending the scooter commands, an attacker could easily trick the device into installing a malicious firmware update that could override its fundamental programming. In this way an attacker would be able to nullify built-in safety mechanisms that prevented the app from remote-controlling or shutting off the vehicle while someone was on it.

“The app allows you to do things like change LED colors, it allows you to remote-control the hoverboard and also apply firmware updates, which is the interesting part,” Kilbride says. “Under the right circumstances, if somebody applies a malicious firmware update, any attacker who knows the right assembly language could then leverage this to basically do as they wish with the hoverboard.”

Running an elastic HiveMQ cluster with auto discovery on AWS

Post Syndicated from The HiveMQ Team original http://www.hivemq.com/blog/running-hivemq-cluster-aws-auto-discovery

hivemq-aws

HiveMQ is a cloud-first MQTT broker with elastic clustering capabilities and a resilient software design which is a perfect fit for common cloud infrastructures. This blogpost discussed what benefits a MQTT broker cluster offers. Today’s post aims to be more practical and talk about how to set up a HiveMQ on one of the most popular cloud computing platform: Amazon Webservices.

Running HiveMQ on cloud infrastructure

Running a HiveMQ cluster on cloud infrastructure like AWS not only offers the advantage the possibility of elastically scaling the infrastructure, it also assures that state of the art security standards are in place on the infrastructure side. These platforms are typically highly available and new virtual machines can be spawned in a snap if they are needed. HiveMQ’s unique ability to add (and remove) cluster nodes at runtime without any manual reconfiguration of the cluster allow to scale linearly on IaaS providers. New cluster nodes can be started (manually or automatically) and the cluster sizes adapts automatically. For more detailed information about HiveMQ clustering and how to achieve true high availability and linear scalability with HiveMQ, we recommend reading the HiveMQ Clustering Paper.

As Amazon Webservice is amongst the best known and most used cloud platforms, we want to illustrate the setup of a HiveMQ cluster on AWS in this post. Note that similar concepts as displayed in this step by step guide for Running an elastic HiveMQ cluster on AWS apply to other cloud platforms such as Microsoft Azure or Google Cloud Platform.

Setup and Configuration

Amazon Webservices prohibits the use of UDP multicast, which is the default HiveMQ cluster discovery mode. The use of Amazon Simple Storage Service (S3) buckets for auto-discovery is a perfect alternative if the brokers are running on AWS EC2 instances anyway. HiveMQ has a free off-the-shelf plugin available for AWS S3 Cluster Discovery.

The following provides a step-by-step guide how to setup the brokers on AWS EC2 with automatic cluster member discovery via S3.

Setup Security Group

The first step is creating a security group that allows inbound traffic to the listeners we are going to configure for MQTT communication. It is also vital to have SSH access on the instances. After you created the security group you need to edit the group and add an additional rule for internal communication between the cluster nodes (meaning the source is the security group itself) on all TCP ports.

To create and edit security groups go to the EC2 console – NETWORK & SECURITY – Security Groups

Inbound traffic

Inbound traffic

Outbound traffic

Outbound traffic

The next step is to create an s3-bucket in the s3 console. Make sure to choose a region, close to the region you want to run your HiveMQ instances on.

Option A: Create IAM role and assign to EC2 instance

Our recommendation is to configure your EC2 instances in a way, allowing them to have access to the s3 bucket. This way you don’t need to create a specific user and don’t need to use the user’s credentials in the 

s3discovery.properties

file.

Create IAM Role

Create IAM Role

EC2 Instance Role Type

EC2 Instance Role Type

Select S3 Full Access

Select S3 Full Access

Assign new Role to Instance

Assign new Role to Instance

Option B: Create user and assign IAM policy

The next step is creating a user in the IAM console.

Choose name and set programmatic access

Choose name and set programmatic access

Assign s3 full access role

Assign s3 full access role

Review and create

Review and create

Download credentials

Download credentials

It is important you store these credentials, as they will be needed later for configuring the S3 Cluster Discovery Plugin.

Start EC2 instances with HiveMQ

The next step is spawning 2 or more EC-2 instances with HiveMQ. Follow the steps in the HiveMQ User Guide.

Install s3 discovery plugin

The final step is downloading, installing and configuring the S3 Cluster Discovery Plugin.
After you downloaded the plugin you need to configure the s3 access in the 

s3discovery.properties

file according to which s3 access option you chose.

Option A:

# AWS Credentials                                          #
############################################################

#
# Use environment variables to specify your AWS credentials
# the following variables need to be set:
# AWS_ACCESS_KEY_ID
# AWS_SECRET_ACCESS_KEY
#
#credentials-type:environment_variables

#
# Use Java system properties to specify your AWS credentials
# the following variables need to be set:
# aws.accessKeyId
# aws.secretKey
#
#credentials-type:java_system_properties

#
# Uses the credentials file wich ############################################################
# can be created by calling 'aws configure' (AWS CLI)
# usually this file is located at ~/.aws/credentials (platform dependent)
# The location of the file can be configured by setting the environment variable
# AWS_CREDENTIAL_PROFILE_FILE to the location of your file
#
#credentials-type:user_credentials_file

#
# Uses the IAM Profile assigned to the EC2 instance running HiveMQ to access S3
# Notice: This only works if HiveMQ is running on an EC2 instance !
#
credentials-type:instance_profile_credentials

#
# Tries to access S3 via the default mechanisms in the following order
# 1) Environment variables
# 2) Java system properties
# 3) User credentials file
# 4) IAM profiles assigned to EC2 instance
#
#credentials-type:default

#
# Uses the credentials specified in this file.
# The variables you must provide are:
# credentials-access-key-id
# credentials-secret-access-key
#
#credentials-type:access_key
#credentials-access-key-id:
#credentials-secret-access-key:

#
# Uses the credentials specified in this file to authenticate with a temporary session
# The variables you must provide are:
# credentials-access-key-id
# credentials-secret-access-key
# credentials-session-token
#
#credentials-type:temporary_session
#credentials-access-key-id:{access_key_id}
#credentials-secret-access-key:{secret_access_key}
#credentials-session-token:{session_token}


############################################################
# S3 Bucket                                                #
############################################################

#
# Region for the S3 bucket used by hivemq
# see http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region for a list of regions for S3
# example: us-west-2
#
s3-bucket-region:<your region here>

#
# Name of the bucket used by HiveMQ
#
s3-bucket-name:<your s3 bucket name here>

#
# Prefix for the filename of every node's file (optional)
#
file-prefix:hivemq/cluster/nodes/

#
# Expiration timeout (in minutes).
# Files with a timestamp older than (timestamp + expiration) will be automatically deleted
# Set to 0 if you do not want the plugin to handle expiration.
#
file-expiration:360

#
# Interval (in minutes) in which the own information in S3 is updated.
# Set to 0 if you do not want the plugin to update its own information.
# If you disable this you also might want to disable expiration.
#
update-interval:180

Option B:

# AWS Credentials                                          #
############################################################

#
# Use environment variables to specify your AWS credentials
# the following variables need to be set:
# AWS_ACCESS_KEY_ID
# AWS_SECRET_ACCESS_KEY
#
#credentials-type:environment_variables

#
# Use Java system properties to specify your AWS credentials
# the following variables need to be set:
# aws.accessKeyId
# aws.secretKey
#
#credentials-type:java_system_properties

#
# Uses the credentials file wich ############################################################
# can be created by calling 'aws configure' (AWS CLI)
# usually this file is located at ~/.aws/credentials (platform dependent)
# The location of the file can be configured by setting the environment variable
# AWS_CREDENTIAL_PROFILE_FILE to the location of your file
#
#credentials-type:user_credentials_file

#
# Uses the IAM Profile assigned to the EC2 instance running HiveMQ to access S3
# Notice: This only works if HiveMQ is running on an EC2 instance !
#
#credentials-type:instance_profile_credentials

#
# Tries to access S3 via the default mechanisms in the following order
# 1) Environment variables
# 2) Java system properties
# 3) User credentials file
# 4) IAM profiles assigned to EC2 instance
#
#credentials-type:default

#
# Uses the credentials specified in this file.
# The variables you must provide are:
# credentials-access-key-id
# credentials-secret-access-key
#
credentials-type:access_key
credentials-access-key-id:<your access key id here>
credentials-secret-access-key:<your secret access key here>

#
# Uses the credentials specified in this file to authenticate with a temporary session
# The variables you must provide are:
# credentials-access-key-id
# credentials-secret-access-key
# credentials-session-token
#
#credentials-type:temporary_session
#credentials-access-key-id:{access_key_id}
#credentials-secret-access-key:{secret_access_key}
#credentials-session-token:{session_token}


############################################################
# S3 Bucket                                                #
############################################################

#
# Region for the S3 bucket used by hivemq
# see http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region for a list of regions for S3
# example: us-west-2
#
s3-bucket-region:<your region here>

#
# Name of the bucket used by HiveMQ
#
s3-bucket-name:<your s3 bucket name here>

#
# Prefix for the filename of every node's file (optional)
#
file-prefix:hivemq/cluster/nodes/

#
# Expiration timeout (in minutes).
# Files with a timestamp older than (timestamp + expiration) will be automatically deleted
# Set to 0 if you do not want the plugin to handle expiration.
#
file-expiration:360

#
# Interval (in minutes) in which the own information in S3 is updated.
# Set to 0 if you do not want the plugin to update its own information.
# If you disable this you also might want to disable expiration.
#
update-interval:180

This file has to be identical on all your cluster nodes.

That’s it. Starting HiveMQ on multiple EC2 instances will now result in them forming a cluster, taking advantage of the S3 bucket for discovery.
You know that your setup was successful when HiveMQ logs something similar to this.

Cluster size = 2, members : [0QMpE, jw8wu].

Enjoy an elastic MQTT broker cluster

We are now able to take advantage of rapid elasticity. Scaling the HiveMQ cluster up or down by adding or removing EC2 instances without the need of administrative intervention is now possible.

For production environments it’s recommended to use automatic provisioning of the EC2 instances (e.g. by using Chef, Puppet, Ansible or similar tools) so you don’t need to configure each EC2 instance manually. Of course HiveMQ can also be used with Docker, which can also ease the provisioning of HiveMQ nodes.

Защо четем за епидемия от аутизъм, но не за експлозия от новосъздадени планети?

Post Syndicated from Боян Юруков original https://yurukov.net/blog/2017/autism/

Замислете се над следното изказване. Какво не е наред в него?

„Към днешна дата знаем за повече от 1000 планети извън слънчевата ни система. В списъка има все повече с размерите на Земята и с възможност на живот. Това е истинска експлозия. Като си помисли човек, че до съвсем скоро тези планети не са съществували и се появяват точно в последните години.“

Нещо не е наред, нали? Дразни с наивността си.

Аналогични са изказванията за „епидемия“ от аутизъм. Все едно е болест или киста. Това, че виждаме съобщения за повече деца със състояние някъде в спектъра не означава, че има „епидемия“, а че не сме гледали до сега, не са получавали нужните грижи и напътствия. Това, че сега имаме знанията, разбирането и инструментите да диагностицираме аутизъм в повече деца и много по-рано не означава, че преди не са били също толкова. Просто сега не ги анатемосваме като овладяни от дявола, не ги пишем за шизофреници, олигофрени и идиотчета и не ги оставяме да умрат в гората или по домовете.

Тук не говоря за медицинската страна на нещата, а просто от гледна точка на процедурата и еволюцията на дефинициите. От първата дефиниция на понятието аутизъм са изминали повече от 70 години като в началото е била изключително тясна. Едва в последните 40 години се говори за спектър – за нещо, което обхваща множество състояния. От тогава спектъра се разширява все повече и обхваща неща, които смятаме дори за нормални в ежедневието си. В последните 20 години има широка методология, обучени специалисти и въобще възможност за масово тестване, от където излизат повече случаи и започва този мит.

Канър и Аспергер са започнали работа по аутизма с хуманна цел – да се спре насилието и унизителното отношение както към децата, така и срещу майките им. И двамата говорят за вроденост на тези състояния, за нуждата от търсене на генетични фактори и биомаркери за диагностика, а не наложеното тогава мнение, че липсата на адекватни грижи и любов от майката водят до това, което тогава се е казвало „бавно развиващо се“, а днес вече наричаме аутизъм.

В последно време се връщаме сякаш назад в това отношение. Налага се мнението, че аутизмът е нещо, което може да се предотврати, да не се „отключи“, дори да се излекува. Винят се родителите, че не харчат десетки хиляди за скъпи хотели брандирани като клиники. Виждалите sms кампаниите. Обещанията са грандиозни, а надеждата умира последна. В същото време все още има недостиг от специалисти, образованост и адекватна помощ за родители с деца в спектъра. Стигмата все още е там.

Върнете се към примера ми в началото и се замислете колко абсурдно е всичко това.

Препоръчвам това видео по темата:

Всъщност, мислех да започна с аналогия с рака и как диагностиката в последните десетилетия се е подобрила значително, но се отказах по няколко сериозни причини. Общото с аутизма се изчерпва с подобрената диагностика и разбиране и с това, че няма едно състояние „аутизъм“, както няма един „рак“. Работата обаче е там, че ракът е болест, а аутизмът не е и трябва да спрем да мислим за него като такова. Също така, макар да има силни генетични фактори при рака, немалка роля имат факторите на околната среда. При аутизма няма доказателства, че средата има каквато и да е роля.

Не на последно място, сетих се, че също толкова хора вярват, че рака е „модерна болест“ и пра-бабите ни не са страдали от него. Да, по-малко са умирали точно от рак, защото той най-често започва да става проблем в напреднала възраст, която малко са имали щастието да достигнат. Когато 40% от децата умират до 5 годишна възраст, а почти всички останали умират от насилие, инфекциозни болести, недохранване, по време на раждане или в окопите, някак дори да има начални статии на рак никой не е гледал. Но митовете продължават.