OpenSSL development policy changes

Post Syndicated from corbet original https://lwn.net/Articles/744825/rss

The OpenSSL project has announced
a number of changes to how the project is developed. These include
shutting down the openssl-dev mailing list in favor of discussing all
patches on GitHub and the addition of a new, read-only (for the world)
openssl-project list. “We are changing our release schedule so that
unless there are extenuating circumstances, security releases will go out
on a Tuesday, with the pre-notification being the previous Tuesday. We
don’t see a need to have people ready to sacrifice their weekend every time
a new CVE comes out.

Recent EC2 Goodies – Launch Templates and Spread Placement

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/recent-ec2-goodies-launch-templates-and-spread-placement/

We launched some important new EC2 instance types and features at AWS re:Invent. I’ve already told you about the M5, H1, T2 Unlimited and Bare Metal instances, and about Spot features such as Hibernation and the New Pricing Model. Randall told you about the Amazon Time Sync Service. Today I would like to tell you about two of the features that we launched: Spread placement groups and Launch Templates. Both features are available in the EC2 Console and from the EC2 APIs, and can be used in all of the AWS Regions in the “aws” partition:

Launch Templates
You can use launch templates to store the instance, network, security, storage, and advanced parameters that you use to launch EC2 instances, and can also include any desired tags. Each template can include any desired subset of the full collection of parameters. You can, for example, define common configuration parameters such as tags or network configurations in a template, and allow the other parameters to be specified as part of the actual launch.

Templates give you the power to set up a consistent launch environment that spans instances launched in On-Demand and Spot form, as well as through EC2 Auto Scaling and as part of a Spot Fleet. You can use them to implement organization-wide standards and to enforce best practices, and you can give your IAM users the ability to launch instances via templates while withholding the ability to do so via the underlying APIs.

Templates are versioned and you can use any desired version when you launch an instance. You can create templates from scratch, base them on the previous version, or copy the parameters from a running instance.

Here’s how you create a launch template in the Console:

Here’s how to include network interfaces, storage volumes, tags, and security groups:

And here’s how to specify advanced and specialized parameters:

You don’t have to specify values for all of these parameters in your templates; enter the values that are common to multiple instances or launches and specify the rest at launch time.

When you click Create launch template, the template is created and can be used to launch On-Demand instances, create Auto Scaling Groups, and create Spot Fleets:

The Launch Instance button now gives you the option to launch from a template:

Simply choose the template and the version, and finalize all of the launch parameters:

You can also manage your templates and template versions from the Console:

To learn more about this feature, read Launching an Instance from a Launch Template.

Spread Placement Groups
Spread placement groups indicate that you do not want the instances in the group to share the same underlying hardware. Applications that rely on a small number of critical instances can launch them in a spread placement group to reduce the odds that one hardware failure will impact more than one instance. Here are a couple of things to keep in mind when you use spread placement groups:

  • Availability Zones – A single spread placement group can span multiple Availability Zones. You can have a maximum of seven running instances per Availability Zone per group.
  • Unique Hardware – Launch requests can fail if there is insufficient unique hardware available. The situation changes over time as overall usage changes and as we add additional hardware; you can retry failed requests at a later time.
  • Instance Types – You can launch a wide variety of M4, M5, C3, R3, R4, X1, X1e, D2, H1, I2, I3, HS1, F1, G2, G3, P2, and P3 instances types in spread placement groups.
  • Reserved Instances – Instances launched into a spread placement group can make use of reserved capacity. However, you cannot currently reserve capacity for a placement group and could receive an ICE (Insufficient Capacity Error) even if you have some RI’s available.
  • Applicability – You cannot use spread placement groups in conjunction with Dedicated Instances or Dedicated Hosts.

You can create and use spread placement groups from the AWS Management Console, the AWS Command Line Interface (CLI), the AWS Tools for Windows PowerShell, and the AWS SDKs. The console has a new feature that will help you to learn how to use the command line:

You can specify an existing placement group or create a new one when you launch an EC2 instance:

To learn more, read about Placement Groups.

Jeff;

Friday Squid Blogging: Te Papa Colossal Squid Exhibition Is Being Renovated

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2018/01/friday_squid_bl_610.html

The New Zealand home of the colossal squid exhibit is behind renovated.

As usual, you can also use this squid post to talk about the security stories in the news that I haven’t covered.

Read my blog posting guidelines here.

Kroah-Hartman: Meltdown and Spectre Linux Kernel Status – Update

Post Syndicated from corbet original https://lwn.net/Articles/744803/rss

Here’s a
brief update from Greg Kroah-Hartman
on the kernel’s handling of the
Meltdown and Spectre vulnerabilities. “This shows that my kernel is
properly mitigating the Meltdown problem by implementing PTI (Page Table
Isolation), and that my system is still vulnerable to the Spectre variant
1, but is trying really hard to resolve the variant 2, but is not quite
there (because I did not build my kernel with a compiler to properly
support the retpoline feature).

Linking Is Not Copyright Infringement, Boing Boing Tells Court

Post Syndicated from Ernesto original https://torrentfreak.com/linking-is-not-copyright-infringement-boing-boing-tells-court-180119/

Late last year Playboy sued the popular blog Boing Boing for publishing an article that linked to an archive of every playmate centerfold till then.

“Kind of amazing to see how our standards of hotness, and the art of commercial erotic photography, have changed over time,” Boing Boing’s Xena Jardin commented.

Playboy, instead, was amazed that infringing copies of their work were being shared in public. While Boing Boing didn’t upload or store the images in question, the publisher took the case to court.

The blog’s parent company Happy Mutants was accused of various counts of copyright infringement, with Playboy claiming that it exploited their playmates’ images for commercial purposes.

Boing Boing sees things differently. With help from the Electronic Frontier Foundation, it has filed a motion to dismiss the case, arguing that hyperlinking is not copyright infringement.

“This lawsuit is frankly mystifying. Playboy’s theory of liability seems to be that it is illegal to link to material posted by others on the web — an act performed daily by hundreds of millions of users of Facebook and Twitter, and by journalists like the ones in Playboy’s crosshairs here,” they write.

The article in question

The defense points out that Playboy’s complaint fails to state a claim for direct or contributory copyright infringement. In addition, it argues that this type of reporting should be seen as fair use.

“Boing Boing’s reporting and commenting on the Playboy photos is protected by copyright’s fair use doctrine,” EFF Senior Staff Attorney Daniel Nazer says, commenting on the case.

“We’re asking the court to dismiss this deeply flawed lawsuit. Journalists, scientists, researchers, and everyday people on the web have the right to link to material, even copyrighted material, without having to worry about getting sued.”

The lawsuit shares a lot of similarities with the case between Dutch blog GeenStijl and local Playboy publisher Sanoma. That high-profile case went all the way to the European Court of Justice.

The highest European court eventually decided that hyperlinks to infringing works are to be considered a ‘communication to the public,’ and that a commercial publication can indeed be held liable for copyright infringement.

Boing Boing hopes that US Courts will see things differently, or it might be “the end of the web as we know it.”

“The world can’t afford a judgment against us in this case — it would end the web as we know it, threatening everyone who publishes online, from us five weirdos in our basements to multimillion-dollar, globe-spanning publishing empires like Playboy,” Boing Boing writes.

A copy of Boing Boing’s memorandum in support of the motion to dismiss is available here (pdf). The original Playboy complaint can be found here (pdf).

Source: TF, for the latest info on copyright, file-sharing, torrent sites and more. We also have VPN discounts, offers and coupons

Security updates for Friday

Post Syndicated from ris original https://lwn.net/Articles/744791/rss

Security updates have been issued by Arch Linux (bind, irssi, nrpe, perl-xml-libxml, and transmission-cli), CentOS (java-1.8.0-openjdk), Debian (awstats, libgd2, mysql-5.5, rsync, smarty3, and transmission), Fedora (keycloak-httpd-client-install and rootsh), and Red Hat (java-1.7.0-oracle and java-1.8.0-oracle).

Hollywood Asks New UK Culture Secretary To Fight Online Piracy

Post Syndicated from Andy original https://torrentfreak.com/hollywood-asks-new-uk-culture-secretary-to-fight-online-piracy-180119/

Following Prime Minister Theresa May’s cabinet reshuffle earlier this month, Matt Hancock replaced Karen Bradley as Secretary of State for Digital, Culture, Media and Sport.

Hancock, the 39-year-old MP for West Suffolk, was promoted from his role as Minister for Digital and Culture, a position he’d held since July 2016.

“Thrilled to become DCMS Secretary. Such an exciting agenda, so much to do, and great people. Can’t wait to get stuck in,” he tweeted.

Of course, the influence held by the Culture Secretary means that the entertainment industries will soon come calling, seeking help and support in a number of vital areas. No surprise then that Stan McCoy, president and managing director at the ‎Motion Picture Association’s EMEA division, has just jumped in with some advice for Hancock.

In an open letter published on Screen Daily, McCoy begins by reminding Hancock that the movie industry contributes considerable sums to the UK economy.

“We are one of the country’s most valuable economic and cultural assets – worth almost £92bn, growing at twice the rate of the economy, and making a positive contribution to the UK’s balance of payments,” McCoy writes.

“Britain’s status as a center of excellence for the audiovisual sector in particular is no accident: It results from the hard work and genius of our creative workforce, complemented by the support of governments that have guided their policies toward enabling continued excellence and growth.”

McCoy goes on to put anti-piracy initiatives at the very top of his wishlist – and Hancock’s to-do list.

“A joined-up strategy to curb proliferation of illegal, often age-inappropriate and malware-laden content online must include addressing the websites, environments and apps that host and facilitate piracy,” McCoy says.

“In addition to hurting one of Britain’s most important industries, they are overwhelmingly likely to harm children and adult consumers through nasty ads, links to adult content with no age verification, scams, fraud and other unpleasantness.”

That McCoy begins with the “piracy is dangerous” approach is definitely not a surprise. This Hollywood and wider video industry strategy is now an open secret. However, it feels a little off that the UK is being asked to further tackle pirate sites.

Through earlier actions, facilitated by the UK legal system and largely sympathetic judges, many thousands of URLs and domains linking to pirate sites, mirrors and proxies, are impossible to access directly through the UK’s major ISPs. Although a few slip through the net, directly accessing the majority of pirate sites in the UK is now impossible.

That’s already a considerable overseas anti-piracy position for the MPA who, as the “international voice” of the Motion Picture Association of America (MPAA), represents American corporations including Disney, Paramount, Sony Pictures, 20th Century Fox, Universal, and Warner Bros.

There’s no comparable blocking system for these companies to use in the United States and rightsholders in the UK can even have extra sites blocked without going back to court for permission. In summary, these US companies arguably get a better anti-piracy deal in the UK than they do at home in the United States.

In his next point, McCoy references last year’s deal – which was reached following considerable pressure from the UK government – between rightsholders and search engines including Google and Bing to demote ‘pirate’ results.

“Building on last year’s voluntary deal with search engines, the Government should stay at the cutting edge of ensuring that everyone in the ecosystem – including search engines, platforms and social media companies – takes a fair share of responsibility,” McCoy says.

While this progress is clearly appreciated by the MPA/MPAA, it’s difficult to ignore that the voluntary arrangement to demote infringing content is somewhat special if not entirely unique. There is definitely nothing comparable in the United States so keeping up the pressure on the UK Government feels a little like getting the good kid in class to behave, while his rowdy peers nearer the chalkboard get ignored.

The same is true for McCoy’s call for the UK to “banish dodgy streaming devices”.

“Illegal streaming devices loaded with piracy apps and malware – not to mention the occasional electrical failure – are proliferating across the UK, to the detriment of consumers and industry,” he writes.

“The sector is still waiting for the Intellectual Property Office to publish the report on its Call for Views on this subject. This will be one of several opportunities, along with the promised Digital Charter, to make clear that these devices and the apps and content they supply are unacceptable, dangerous to consumers, and harmful to the creative industry.”

Again, prompting the UK to stay on top of this game doesn’t feel entirely warranted.

With dozens of actions over the past few years, the Police Intellectual Property Crime Unit and the Federation Against Copyright Theft (which Hollywood ironically dumped in 2016) have done more to tackle the pirate set-top box problem than any group on the other side of the Atlantic.

Admittedly the MPAA is now trying to catch up, with recent prosecutions of two ‘pirate’ box vendors (1,2), but largely the work by the studios on their home turf has been outpaced by that of their counterparts in the UK.

Maybe Hancock will mention that to Hollywood at some point in the future.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and more. We also have VPN discounts, offers and coupons

e-paper pocket money tracker using Monzo pots

Post Syndicated from Alex Bate original https://www.raspberrypi.org/blog/monzo-money-tracker/

Jason Barnett used the pots feature of the Monzo banking API to create a simple e-paper display so that his kids can keep track of their pocket money.

Monzo ePaper Pot Jason Barnett Raspberry Pi

Monzo

For those outside the UK: Monzo is a smartphone-based bank that allows costumers to manage their money and payment cards via an app, removing the bank clerk middleman.

In the Monzo banking app, users can set up pots, which allow them to organise their money into various, you guessed it, pots. You want to put aside holiday funds, budget your food shopping, or, like Jason, manage your kids’ pocket money? Using pots is an easy way to do it.

Jason’s Monzo Pot ePaper tracker

After failed attempts at keeping track of his sons’ pocket money via a scrap of paper stuck to the fridge, Jason decided to try a new approach.

He started his build by installing Stretch Lite to the SD card of his Raspberry Pi Zero W. “The Pi will be running headless (without screen, mouse or keyboard)”, he explains on his blog, “so there is no need for a full-fat Raspbian image.” While Stretch Lite was downloading, he set up the Waveshare ePaper HAT on his Zero W. He notes that Pimoroni’s “Inky pHAT would be easiest,” but his tutorial is specific to the Waveshare device.

Monzo ePaper Pot Jason Barnett Raspberry Pi

Before ejecting the SD card, Jason updated the boot partition to allow him to access the Pi via SSH. He talks makers through that process here.

Among the libraries he installed for the project is pyMonzo, a Python wrapper for the Monzo API created by Paweł Adamczak. Monzo is still in its infancy, and the API is partly under construction. Until it’s completed, Paweł’s wrapper offers a more stable way to use it.

After installing the software, it was time to set up the e-paper screen for the tracker. Jason adjusted the code for the API so that the screen reloads information every 15 minutes, displaying the up-to-date amount of pocket money in both kids’ pots.

Here is how Jason describes going to the supermarket with his sons, now that he has completed the tracker:

“Daddy, I want (insert first thing picked up here), I’ve always wanted one of these my whole life!” […] Even though you have never seen that (insert thing here) before, I can quickly open my Monzo app, flick to Account, and say “You have £3.50 in your money box”. If my boy wants it, a 2-second withdrawal is made whilst queueing, and done — he walks away with a new (again, insert whatever he wanted his whole life here) and is happy!

Jason’s blog offers a full breakdown of his project, including all necessary code and the specs for the physical build. Be sure to head over and check it out.

Have you used an API in your projects? What would you build with one?

The post e-paper pocket money tracker using Monzo pots appeared first on Raspberry Pi.

Security Breaches Don’t Affect Stock Price

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2018/01/security_breach.html

Interesting research: “Long-term market implications of data breaches, not,” by Russell Lange and Eric W. Burger.

Abstract: This report assesses the impact disclosure of data breaches has on the total returns and volatility of the affected companies’ stock, with a focus on the results relative to the performance of the firms’ peer industries, as represented through selected indices rather than the market as a whole. Financial performance is considered over a range of dates from 3 days post-breach through 6 months post-breach, in order to provide a longer-term perspective on the impact of the breach announcement.

Key findings:

  • While the difference in stock price between the sampled breached companies and their peers was negative (1.13%) in the first 3 days following announcement of a breach, by the 14th day the return difference had rebounded to + 0.05%, and on average remained positive through the period assessed.
  • For the differences in the breached companies’ betas and the beta of their peer sets, the differences in the means of 8 months pre-breach versus post-breach was not meaningful at 90, 180, and 360 day post-breach periods.

  • For the differences in the breached companies’ beta correlations against the peer indices pre- and post-breach, the difference in the means of the rolling 60 day correlation 8 months pre- breach versus post-breach was not meaningful at 90, 180, and 360 day post-breach periods.

  • In regression analysis, use of the number of accessed records, date, data sensitivity, and malicious versus accidental leak as variables failed to yield an R2 greater than 16.15% for response variables of 3, 14, 60, and 90 day return differential, excess beta differential, and rolling beta correlation differential, indicating that the financial impact on breached companies was highly idiosyncratic.

  • Based on returns, the most impacted industries at the 3 day post-breach date were U.S. Financial Services, Transportation, and Global Telecom. At the 90 day post-breach date, the three most impacted industries were U.S. Financial Services, U.S. Healthcare, and Global Telecom.

The market isn’t going to fix this. If we want better security, we need to regulate the market.

Note: The article is behind a paywall. An older version is here. A similar article is here.

MPAA Wins $19.8 Million From Pirate Site Pubfilm

Post Syndicated from Ernesto original https://torrentfreak.com/mpaa-wins-19-8-million-from-pirate-site-pubfilm-180119/

In recent years the MPAA has pursued legal action against several pirate sites and the streaming service Pubfilm is one of their latest targets.

Hollywood’s industry group initially kept the lawsuit secret. This was done to prevent Pubfilm’s operator from moving to a new domain preemptively. While this strategy worked, Pubfilm didn’t throw in the towel.

Soon after the pubfilm.com domain name was suspended, the site moved to pubfilm.ac. And that wasn’t all. Pubfilm also started to actively advertise its new domain through Google Adsense to regain its lost traffic.

Today, close to a year has passed and Pubfilm is still around. The site moved from domain to domain and currently resides at Pubfilm.is and a few other domains that are advertised on the site.

All this time the company failed to responded in court, so the case saw little movement. This week, however, the MPAA made its demands clear and soon after the court issued a default against the site and its unknown operators.

“Defendants are Internet pirates who own, operate, and promote a ring of interconnected websites under the name “PubFilm” and variants thereof whose purpose is to profit from the infringement of copyrighted works,” the group wrote in its request.

Because of this continued infringement, the MPAA demanded the maximum amount of statutory copyright infringement damages. With 132 titles listed in the complaint, this totals nearly $20 million.

“Given the egregious circumstances of this case, Plaintiffs should be awarded the full amount of statutory damages of $150,000 for each of the 132 Works identified in the Complaint, for a total of $19,800,000,” the MPAA writes in its memorandum.

In addition, the Hollywood studios requested a permanent injunction that will require domain registries to put associated domain names on hold and sign them over to the MPAA.

Both requests were granted by the court on Thursday.

Pubfilm domain hopping

Previously, several domain names were aready seized through a preliminary injunction that resulted in Pubfilm moving from domain to domain in recent months. While these seizures can be effective, not all domain registries will comply with a US court order.

One of Pubfilm’s main domain names at the moment uses the Icelandic .is cTLD. In the past, Iceland’s domain registry ISNIC told TorrentFreak that it would only take action when an Icelandic Court tells it to.

This means that the MPAA’s win might be one without teeth.

Getting millions of dollars from an anonymous site operator, presumably outside the United States, is not easy. And since the site still has several hard-to-shutdown domains, taking it offline isn’t that straightforward either.

The streaming site operators didn’t appear to be impressed by the legal battle either. For the time being, they seem more concerned with fighting fake versions, judging from their most recent Facebook update.

Pubfilm’s latest Facebook post

A copy of the MPAA’s Memorandum in support of the default judgment and permanent injunction is available here (pdf). A copy of court’s order can be found here (pdf).

Source: TF, for the latest info on copyright, file-sharing, torrent sites and more. We also have VPN discounts, offers and coupons

Wine 3.0 released

Post Syndicated from corbet original https://lwn.net/Articles/744741/rss

Version 3.0 of the
Wine Windows emulation layer has been released. “This release
represents a year of development effort and over 6,000 individual
changes.
” Most of the improvements seem to be around Direct3D
graphics, but it also now possible to package up Wine as an Android app;
see the release notes for
details.

Migrating .NET Classic Applications to Amazon ECS Using Windows Containers

Post Syndicated from Sundar Narasiman original https://aws.amazon.com/blogs/compute/migrating-net-classic-applications-to-amazon-ecs-using-windows-containers/

This post contributed by Sundar Narasiman, Arun Kannan, and Thomas Fuller.

AWS recently announced the general availability of Windows container management for Amazon Elastic Container Service (Amazon ECS). Docker containers and Amazon ECS make it easy to run and scale applications on a virtual machine by abstracting the complex cluster management and setup needed.

Classic .NET applications are developed with .NET Framework 4.7.1 or older and can run only on a Windows platform. These include Windows Communication Foundation (WCF), ASP.NET Web Forms, and an ASP.NET MVC web app or web API.

Why classic ASP.NET?

ASP.NET MVC 4.6 and older versions of ASP.NET occupy a significant footprint in the enterprise web application space. As enterprises move towards microservices for new or existing applications, containers are one of the stepping stones for migrating from monolithic to microservices architectures. Additionally, the support for Windows containers in Windows 10, Windows Server 2016, and Visual Studio Tooling support for Docker simplifies the containerization of ASP.NET MVC apps.

Getting started

In this post, you pick an ASP.NET 4.6.2 MVC application and get step-by-step instructions for migrating to ECS using Windows containers. The detailed steps, AWS CloudFormation template, Microsoft Visual Studio solution, ECS service definition, and ECS task definition are available in the aws-ecs-windows-aspnet GitHub repository.

To help you getting started running Windows containers, here is the reference architecture for Windows containers on GitHub: ecs-refarch-cloudformation-windows. This reference architecture is the layered CloudFormation stack, in that it calls the other stacks to create the environment. The CloudFormation YAML template in this reference architecture is referenced to create a single JSON CloudFormation stack, which is used in the steps for the migration.

Steps for Migration

The code and templates to implement this migration can be found on GitHub: https://github.com/aws-samples/aws-ecs-windows-aspnet.

  1. Your development environment needs to have the latest version and updates for Visual Studio 2017, Windows 10, and Docker for Windows Stable.
  2. Next, containerize the ASP.NET application and test it locally. The size of Windows container application images is generally larger compared to Linux containers. This is because the base image of the Windows container itself is large in size, typically greater than 9 GB.
  3. After the application is containerized, the container image needs to be pushed to Amazon Elastic Container Registry (Amazon ECR). Images stored in ECR are compressed to improve pull times and reduce storage costs. In this case, you can see that ECR compresses the image to around 1 GB, for an optimization factor of 90%.
  4. Create a CloudFormation stack using the template in the ‘CloudFormation template’ folder. This creates an ECS service, task definition (referring the containerized ASP.NET application), and other related components mentioned in the ECS reference architecture for Windows containers.
  5. After the stack is created, verify the successful creation of the ECS service, ECS instances, running tasks (with the threshold mentioned in the task definition), and the Application Load Balancer’s successful health check against running containers.
  6. Navigate to the Application Load Balancer URL and see the successful rendering of the containerized ASP.NET MVC app in the browser.

Key Notes

  • Generally, Windows container images occupy large amount of space (in the order of few GBs).
  • All the task definition parameters for Linux containers are not available for Windows containers. For more information, see Windows Task Definitions.
  • An Application Load Balancer can be configured to route requests to one or more ports on each container instance in a cluster. The dynamic port mapping allows you to have multiple tasks from a single service on the same container instance.
  • IAM roles for Windows tasks require extra configuration. For more information, see Windows IAM Roles for Tasks. For this post, configuration was handled by the CloudFormation template.
  • The ECS container agent log file can be accessed for troubleshooting Windows containers: C:\ProgramData\Amazon\ECS\log\ecs-agent.log

Summary

In this post, you migrated an ASP.NET MVC application to ECS using Windows containers.

The logical next step is to automate the activities for migration to ECS and build a fully automated continuous integration/continuous deployment (CI/CD) pipeline for Windows containers. This can be orchestrated by leveraging services such as AWS CodeCommit, AWS CodePipeline, AWS CodeBuild, Amazon ECR, and Amazon ECS. You can learn more about how this is done in the Set Up a Continuous Delivery Pipeline for Containers Using AWS CodePipeline and Amazon ECS post.

If you have questions or suggestions, please comment below.

Optimize Delivery of Trending, Personalized News Using Amazon Kinesis and Related Services

Post Syndicated from Yukinori Koide original https://aws.amazon.com/blogs/big-data/optimize-delivery-of-trending-personalized-news-using-amazon-kinesis-and-related-services/

This is a guest post by Yukinori Koide, an the head of development for the Newspass department at Gunosy.

Gunosy is a news curation application that covers a wide range of topics, such as entertainment, sports, politics, and gourmet news. The application has been installed more than 20 million times.

Gunosy aims to provide people with the content they want without the stress of dealing with a large influx of information. We analyze user attributes, such as gender and age, and past activity logs like click-through rate (CTR). We combine this information with article attributes to provide trending, personalized news articles to users.

In this post, I show you how to process user activity logs in real time using Amazon Kinesis Data Firehose, Amazon Kinesis Data Analytics, and related AWS services.

Why does Gunosy need real-time processing?

Users need fresh and personalized news. There are two constraints to consider when delivering appropriate articles:

  • Time: Articles have freshness—that is, they lose value over time. New articles need to reach users as soon as possible.
  • Frequency (volume): Only a limited number of articles can be shown. It’s unreasonable to display all articles in the application, and users can’t read all of them anyway.

To deliver fresh articles with a high probability that the user is interested in them, it’s necessary to include not only past user activity logs and some feature values of articles, but also the most recent (real-time) user activity logs.

We optimize the delivery of articles with these two steps.

  1. Personalization: Deliver articles based on each user’s attributes, past activity logs, and feature values of each article—to account for each user’s interests.
  2. Trends analysis/identification: Optimize delivering articles using recent (real-time) user activity logs—to incorporate the latest trends from all users.

Optimizing the delivery of articles is always a cold start. Initially, we deliver articles based on past logs. We then use real-time data to optimize as quickly as possible. In addition, news has a short freshness time. Specifically, day-old news is past news, and even the news that is three hours old is past news. Therefore, shortening the time between step 1 and step 2 is important.

To tackle this issue, we chose AWS for processing streaming data because of its fully managed services, cost-effectiveness, and so on.

Solution

The following diagrams depict the architecture for optimizing article delivery by processing real-time user activity logs

There are three processing flows:

  1. Process real-time user activity logs.
  2. Store and process all user-based and article-based logs.
  3. Execute ad hoc or heavy queries.

In this post, I focus on the first processing flow and explain how it works.

Process real-time user activity logs

The following are the steps for processing user activity logs in real time using Kinesis Data Streams and Kinesis Data Analytics.

  1. The Fluentd server sends the following user activity logs to Kinesis Data Streams:
{"article_id": 12345, "user_id": 12345, "action": "click"}
{"article_id": 12345, "user_id": 12345, "action": "impression"}
...
  1. Map rows of logs to columns in Kinesis Data Analytics.

  1. Set the reference data to Kinesis Data Analytics from Amazon S3.

a. Gunosy has user attributes such as gender, age, and segment. Prepare the following CSV file (user_id, gender, segment_id) and put it in Amazon S3:

101,female,1
102,male,2
103,female,3
...

b. Add the application reference data source to Kinesis Data Analytics using the AWS CLI:

$ aws kinesisanalytics add-application-reference-data-source \
  --application-name <my-application-name> \
  --current-application-version-id <version-id> \
  --reference-data-source '{
  "TableName": "REFERENCE_DATA_SOURCE",
  "S3ReferenceDataSource": {
    "BucketARN": "arn:aws:s3:::<my-bucket-name>",
    "FileKey": "mydata.csv",
    "ReferenceRoleARN": "arn:aws:iam::<account-id>:role/..."
  },
  "ReferenceSchema": {
    "RecordFormat": {
      "RecordFormatType": "CSV",
      "MappingParameters": {
        "CSVMappingParameters": {"RecordRowDelimiter": "\n", "RecordColumnDelimiter": ","}
      }
    },
    "RecordEncoding": "UTF-8",
    "RecordColumns": [
      {"Name": "USER_ID", "Mapping": "0", "SqlType": "INTEGER"},
      {"Name": "GENDER",  "Mapping": "1", "SqlType": "VARCHAR(32)"},
      {"Name": "SEGMENT_ID", "Mapping": "2", "SqlType": "INTEGER"}
    ]
  }
}'

This application reference data source can be referred on Kinesis Data Analytics.

  1. Run a query against the source data stream on Kinesis Data Analytics with the application reference data source.

a. Define the temporary stream named TMP_SQL_STREAM.

CREATE OR REPLACE STREAM "TMP_SQL_STREAM" (
  GENDER VARCHAR(32), SEGMENT_ID INTEGER, ARTICLE_ID INTEGER
);

b. Insert the joined source stream and application reference data source into the temporary stream.

CREATE OR REPLACE PUMP "TMP_PUMP" AS
INSERT INTO "TMP_SQL_STREAM"
SELECT STREAM
  R.GENDER, R.SEGMENT_ID, S.ARTICLE_ID, S.ACTION
FROM      "SOURCE_SQL_STREAM_001" S
LEFT JOIN "REFERENCE_DATA_SOURCE" R
  ON S.USER_ID = R.USER_ID;

c. Define the destination stream named DESTINATION_SQL_STREAM.

CREATE OR REPLACE STREAM "DESTINATION_SQL_STREAM" (
  TIME TIMESTAMP, GENDER VARCHAR(32), SEGMENT_ID INTEGER, ARTICLE_ID INTEGER, 
  IMPRESSION INTEGER, CLICK INTEGER
);

d. Insert the processed temporary stream, using a tumbling window, into the destination stream per minute.

CREATE OR REPLACE PUMP "STREAM_PUMP" AS
INSERT INTO "DESTINATION_SQL_STREAM"
SELECT STREAM
  ROW_TIME AS TIME,
  GENDER, SEGMENT_ID, ARTICLE_ID,
  SUM(CASE ACTION WHEN 'impression' THEN 1 ELSE 0 END) AS IMPRESSION,
  SUM(CASE ACTION WHEN 'click' THEN 1 ELSE 0 END) AS CLICK
FROM "TMP_SQL_STREAM"
GROUP BY
  GENDER, SEGMENT_ID, ARTICLE_ID,
  FLOOR("TMP_SQL_STREAM".ROWTIME TO MINUTE);

The results look like the following:

  1. Insert the results into Amazon Elasticsearch Service (Amazon ES).
  2. Batch servers get results from Amazon ES every minute. They then optimize delivering articles with other data sources using a proprietary optimization algorithm.

How to connect a stream to another stream in another AWS Region

When we built the solution, Kinesis Data Analytics was not available in the Asia Pacific (Tokyo) Region, so we used the US West (Oregon) Region. The following shows how we connected a data stream to another data stream in the other Region.

There is no need to continue containing all components in a single AWS Region, unless you have a situation where a response difference at the millisecond level is critical to the service.

Benefits

The solution provides benefits for both our company and for our users. Benefits for the company are cost savings—including development costs, operational costs, and infrastructure costs—and reducing delivery time. Users can now find articles of interest more quickly. The solution can process more than 500,000 records per minute, and it enables fast and personalized news curating for our users.

Conclusion

In this post, I showed you how we optimize trending user activities to personalize news using Amazon Kinesis Data Firehose, Amazon Kinesis Data Analytics, and related AWS services in Gunosy.

AWS gives us a quick and economical solution and a good experience.

If you have questions or suggestions, please comment below.


Additional Reading

If you found this post useful, be sure to check out Implement Serverless Log Analytics Using Amazon Kinesis Analytics and Joining and Enriching Streaming Data on Amazon Kinesis.


About the Authors

Yukinori Koide is the head of development for the Newspass department at Gunosy. He is working on standardization of provisioning and deployment flow, promoting the utilization of serverless and containers for machine learning and AI services. His favorite AWS services are DynamoDB, Lambda, Kinesis, and ECS.

 

 

 

Akihiro Tsukada is a start-up solutions architect with AWS. He supports start-up companies in Japan technically at many levels, ranging from seed to later-stage.

 

 

 

 

Yuta Ishii is a solutions architect with AWS. He works with our customers to provide architectural guidance for building media & entertainment services, helping them improve the value of their services when using AWS.

 

 

 

 

 

Cloud Babble: The Jargon of Cloud Storage

Post Syndicated from Andy Klein original https://www.backblaze.com/blog/what-is-cloud-computing/

Cloud Babble

One of the things we in the technology business are good at is coming up with names, phrases, euphemisms, and acronyms for the stuff that we create. The Cloud Storage market is no different, and we’d like to help by illuminating some of the cloud storage related terms that you might come across. We know this is just a start, so please feel free to add in your favorites in the comments section below and we’ll update this post accordingly.

Clouds

The cloud is really just a collection of purpose built servers. In a public cloud the servers are shared between multiple unrelated tenants. In a private cloud, the servers are dedicated to a single tenant or sometimes a group of related tenants. A public cloud is off-site, while a private cloud can be on-site or off-site – or on-prem or off-prem, if you prefer.

Both Sides Now: Hybrid Clouds

Speaking of on-prem and off-prem, there are Hybrid Clouds or Hybrid Data Clouds depending on what you need. Both are based on the idea that you extend your local resources (typically on-prem) to the cloud (typically off-prem) as needed. This extension is controlled by software that decides, based on rules you define, what needs to be done where.

A Hybrid Data Cloud is specific to data. For example, you can set up a rule that says all accounting files that have not been touched in the last year are automatically moved off-prem to cloud storage. The files are still available; they are just no longer stored on your local systems. The rules can be defined to fit an organization’s workflow and data retention policies.

A Hybrid Cloud is similar to a Hybrid Data Cloud except it also extends compute. For example, at the end of the quarter, you can spin up order processing application instances off-prem as needed to add to your on-prem capacity. Of course, determining where the transactional data used and created by these applications resides can be an interesting systems design challenge.

Clouds in my Coffee: Fog

Typically, public and private clouds live in large buildings called data centers. Full of servers, networking equipment, and clean air, data centers need lots of power, lots of networking bandwidth, and lots of space. This often limits where data centers are located. The further away you are from a data center, the longer it generally takes to get your data to and from there. This is known as latency. That’s where “Fog” comes in.

Fog is often referred to as clouds close to the ground. Fog, in our cloud world, is basically having a “little” data center near you. This can make data storage and even cloud based processing faster for everyone nearby. Data, and less so processing, can be transferred to/from the Fog to the Cloud when time is less a factor. Data could also be aggregated in the Fog and sent to the Cloud. For example, your electric meter could report its minute-by-minute status to the Fog for diagnostic purposes. Then once a day the aggregated data could be send to the power company’s Cloud for billing purposes.

Another term used in place of Fog is Edge, as in computing at the Edge. In either case, a given cloud (data center) usually has multiple Edges (little data centers) connected to it. The connection between the Edge and the Cloud is sometimes known as the middle-mile. The network in the middle-mile can be less robust than that required to support a stand-alone data center. For example, the middle-mile can use 1 Gbps lines, versus a data center, which would require multiple 10 Gbps lines.

Heavy Clouds No Rain: Data

We’re all aware that we are creating, processing, and storing data faster than ever before. All of this data is stored in either a structured or more likely an unstructured way. Databases and data warehouses are structured ways to store data, but a vast amount of data is unstructured – meaning the schema and data access requirements are not known until the data is queried. A large pool of unstructured data in a flat architecture can be referred to as a Data Lake.

A Data Lake is often created so we can perform some type of “big data” analysis. In an over simplified example, let’s extend the lake metaphor a bit and ask the question; “how many fish are in our lake?” To get an answer, we take a sufficient sample of our lake’s water (data), count the number of fish we find, and extrapolate based on the size of the lake to get an answer within a given confidence interval.

A Data Lake is usually found in the cloud, an excellent place to store large amounts of non-transactional data. Watch out as this can lead to our data having too much Data Gravity or being locked in the Hotel California. This could also create a Data Silo, thereby making a potential data Lift-and-Shift impossible. Let me explain:

  • Data Gravity — Generally, the more data you collect in one spot, the harder it is to move. When you store data in a public cloud, you have to pay egress and/or network charges to download the data to another public cloud or even to your own on-premise systems. Some public cloud vendors charge a lot more than others, meaning that depending on your public cloud provider, your data could financially have a lot more gravity than you expected.
  • Hotel California — This is like Data Gravity but to a lesser scale. Your data is in the Hotel California if, to paraphrase, “your data can check out any time you want, but it can never leave.” If the cost of downloading your data is limiting the things you want to do with that data, then your data is in the Hotel California. Data is generally most valuable when used, and with cloud storage that can include archived data. This assumes of course that the archived data is readily available, and affordable, to download. When considering a cloud storage project always figure in the cost of using your own data.
  • Data Silo — Over the years, businesses have suffered from organizational silos as information is not shared between different groups, but instead needs to travel up to the top of the silo before it can be transferred to another silo. If your data is “trapped” in a given cloud by the cost it takes to share such data, then you may have a Data Silo, and that’s exactly opposite of what the cloud should do.
  • Lift-and-Shift — This term is used to define the movement of data or applications from one data center to another or from on-prem to off-prem systems. The move generally occurs all at once and once everything is moved, systems are operational and data is available at the new location with few, if any, changes. If your data has too much gravity or is locked in a hotel, a data lift-and-shift may break the bank.

I Can See Clearly Now

Hopefully, the cloudy terms we’ve covered are well, less cloudy. As we mentioned in the beginning, our compilation is just a start, so please feel free to add in your favorite cloud term in the comments section below and we’ll update this post with your contributions. Keep your entries “clean,” and please no words or phrases that are really adverts for your company. Thanks.

The post Cloud Babble: The Jargon of Cloud Storage appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

[$] Shrinking the kernel with link-time optimization

Post Syndicated from corbet original https://lwn.net/Articles/744507/rss

This is the second article of a series discussing various methods of
reducing the size of the Linux kernel to make it suitable for small
environments.

The first article
provided a short rationale for this topic, and covered the link-time
garbage collection, also called the ld --gc-sections method. We’ve seen
that, though it is pretty straightforward, link-time garbage collection has
issues of its own when applied to the kernel, making achieving optimal
results more
difficult than it is worth. In this article we’ll have a look at what the
compiler itself can do using link-time optimization.

Judge Tells Movie Company That it Can’t Sue Alleged BitTorrent Pirate

Post Syndicated from Andy original https://torrentfreak.com/judge-tells-movie-company-that-it-cant-sue-alleged-bittorrent-pirate-180118/

Despite a considerable migration towards streaming piracy in recent years, copyright trolls are still finding plenty of potential targets around the world. Alleged BitTorrent pirates are target number one since their activities are most easily tracked. However, it isn’t all plain sailing for the pirate hunters.

Last December we reported on the case of Lingfu Zhang, an Oregan resident accused by the makers of the 2015 drama film Fathers & Daughters (F&D) of downloading and sharing their content without permission. While these kinds of cases often disappear, with targets making confidential settlements to make a legal battle go away, Zhang chose to fight back.

Represented by attorney David Madden, Zhang not only denied downloading the movie in question but argued that the filmmakers had signed away their online distribution rights. He noted that (F&D), via an agent, had sold the online distribution rights to a third party not involved in the case.

So, if F&D no longer held the right to distribute the movie online, suing for an infringement of those rights would be impossible. With this in mind, Zhang’s attorney moved for a summary judgment in his client’s favor.

“ZHANG denies downloading the movie but Defendant’s current motion for summary judgment challenges a different portion of F&D’s case,” Madden wrote.

“Defendant argues that F&D has alienated all of the relevant rights necessary to sue for infringement under the Copyright Act.”

In response, F&D argued that they still held some rights, including the right to exploit the movie on “airlines and oceangoing vessels” but since Zhang wasn’t accused of being on either form of transport when the alleged offense occurred, the defense argued that point was moot.

Judge Michael H. Simon handed down his decision yesterday and it heralds bad news for F&D and celebration time for Zhang and his attorney. In a 17-page ruling first spotted by Fight Copyright Trolls, the Judge agrees that F&D has no standing to sue.

Citing the Righthaven LLC v. Hoehn case from 2013, the Judge notes that under the Copyright Act, only the “legal or beneficial owner of an exclusive right under a copyright” has standing to sue for infringement of that right.

Judge Simon notes that while F&D claims it is the ‘legal owner’ of the copyright to the Fathers & Daughters movie, the company “misstates the law”, adding that F&D also failed to present evidence that it is the ‘beneficial owner’ of the relevant exclusive right. On this basis, both claims are rejected.

The Judge noted that the exclusive rights to the movie were granted to a company called Vertical Entertainment which received the exclusive right to “manufacture, reproduce, sell, rent, exhibit, broadcast, transmit, stream, download, license, sub-license, distribute, sub-distribute, advertise, market, promote, publicize and exploit” the movie in the United States.

An exclusive license means that ownership of a copyright is transferred for the term of the license, meaning that Vertical – not F&D – is the legal owner under the Copyright Act. It matters not, the Judge says, that F&D retained the rights to display the movie “on airlines and ships” since only the transferee (Vertical) has standing to sue and those locations are irrelevant to the lawsuit.

“Under the Copyright Act, F&D is not the ‘legal owner’ with standing to sue for infringement relating to the rights that were transferred to Vertical through its exclusive license granted in the distribution agreement,” the Judge writes.

Also at issue was an undated document presented by F&D titled Anti-Piracy and Rights Enforcement Reservation of Rights Addendum. The document, relied upon by F&D, claimed that F&D is authorized to “enforce copyrights against Internet infringers” including those that use peer-to-peer technologies such as BitTorrent.

However, the Judge found that the peer-to-peer rights apparently reserved to F&D were infringing rights, not the display and distribution (exclusive rights) required to sue under the Copyright Act. Furthermore, the Judge determined that there was no evidence that this document existed before the lawsuit was filed. Zhang and his attorney previously asserted the addendum had been created afterwards and the Judge agrees.

“F&D did not dispute that the undated anti-piracy addendum was created after this lawsuit was filed, or otherwise respond to Defendant’s standing argument relating to the untimeliness of this document,” the Judge notes.

“Accordingly, because the only reasonable inference supported by the evidence is that this document was created after the filing of this lawsuit, it is not appropriate to consider for purposes of standing.”

So, with Vertical Entertainment the only company with the right to sue, could they be added to the lawsuit, F&D asked? Citing an earlier case, the Judge said ‘no’, noting that “summary judgment is not a procedural second chance to flesh out inadequate pleadings.”

With that, Judge Simon granted Lingfu Zhang’s request for summary judgment and dismissed F&D’s claims for lack of standing.

As noted by Fight Copyright Trolls, the movie licensing scheme employed by F&D is complex and, given the fact that notorious copyright troll outfit Guardaley is involved (Guardaley filed 24 cases in eight districts on behalf of F&D), it would be interesting if legal professionals could dig deeper, to see how far the rabbit hole goes.

The summary judgment can be found here (pdf)

Source: TF, for the latest info on copyright, file-sharing, torrent sites and more. We also have VPN discounts, offers and coupons

Security updates for Thursday

Post Syndicated from ris original https://lwn.net/Articles/744713/rss

Security updates have been issued by CentOS (linux-firmware and microcode_ctl), Fedora (icecat and transmission), Oracle (java-1.8.0-openjdk and microcode_ctl), Red Hat (java-1.8.0-openjdk), Scientific Linux (java-1.8.0-openjdk), Slackware (bind), SUSE (kernel), and Ubuntu (eglibc).

By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.

Close