Tag Archives: ISTE

Sending Inaudible Commands to Voice Assistants

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2018/05/sending_inaudib.html

Researchers have demonstrated the ability to send inaudible commands to voice assistants like Alexa, Siri, and Google Assistant.

Over the last two years, researchers in China and the United States have begun demonstrating that they can send hidden commands that are undetectable to the human ear to Apple’s Siri, Amazon’s Alexa and Google’s Assistant. Inside university labs, the researchers have been able to secretly activate the artificial intelligence systems on smartphones and smart speakers, making them dial phone numbers or open websites. In the wrong hands, the technology could be used to unlock doors, wire money or buy stuff online ­– simply with music playing over the radio.

A group of students from University of California, Berkeley, and Georgetown University showed in 2016 that they could hide commands in white noise played over loudspeakers and through YouTube videos to get smart devices to turn on airplane mode or open a website.

This month, some of those Berkeley researchers published a research paper that went further, saying they could embed commands directly into recordings of music or spoken text. So while a human listener hears someone talking or an orchestra playing, Amazon’s Echo speaker might hear an instruction to add something to your shopping list.

Pirate IPTV Service Gave Customer Details to Premier League, But What’s the Risk?

Post Syndicated from Andy original https://torrentfreak.com/pirate-iptv-service-gave-customer-details-to-premier-league-but-whats-the-risk-180515/

In a report last weekend, we documented what appear to be the final days of pirate IPTV provider Ace Hosting.

From information provided by several sources including official liquidation documents, it became clear that a previously successful and profitable Ace had succumbed to pressure from the Premier League, which accused the service of copyright infringement.

The company had considerable funds in the bank – £255,472.00 to be exact – but it also had debts of £717,278.84, including £260,000 owed to HMRC and £100,000 to the Premier League as part of a settlement agreement.

Information received by TF late Sunday suggested that £100K was the tip of the iceberg as far as the Premier League was concerned and in a statement yesterday, the football outfit confirmed that was the case.

“A renowned pirate of Premier League content to consumers has been forced to liquidate after agreeing to pay £600,000 for breaching the League’s copyright,” the Premier League announced.

“Ace IPTV, run by Craig Driscoll and Ian Isaac, was selling subscriptions to illegal Premier League streams directly to consumers which allowed viewing on a range of devices, including notorious Kodi-type boxes, as well as to smaller resellers in the UK and abroad.”

Sources familiar with the case suggest that while Ace Hosting Limited didn’t have the funds to pay the Premier League the full £600K, Ace’s operators agreed to pay (and have already paid, to some extent at least) what were essentially their own funds to cover amounts above the final £100K, which is due to be paid next year.

But that’s not the only thing that’s been handed over to the Premier League.

“Ace voluntarily disclosed the personal details of their customers, which the League will now review in compliance with data protection legislation. Further investigations will be conducted, and action taken where appropriate,” the Premier League added.

So, the big question now is how exposed Ace’s former subscribers are.

The truth is that only the Premier League knows for sure but TF has been able to obtain information from several sources which indicate that former subscribers probably aren’t the Premier League’s key interest and even if they were, information obtained on them would be of limited use.

According to a source with knowledge of how a system like Ace’s works, there is a separation of data which appears to help (at least to some degree) with the subscriber’s privacy.

“The system used to manage accounts and take payment is actually completely separate from the software used to manage streams and the lines themselves. They are never usually even on the same server so are two very different databases,” he told TF.

“So at best the only information that has voluntarily been provided to the [Premier League], is just your email, name and address (assuming you even used real details) and what hosting package or credits you bought.”

While this information is bad enough, the action against Ace is targeted, in that it focuses on the Premier League’s content and how Ace (and therefore its users) infringed on the football outfit’s copyrights. So, proving that subscribers actually watched any Premier League content would be an ideal position but it’s not straightforward, despite the potential for detailed logging.

“The management system contains no history of what you watched, when you watched it, when you signed in and so on. That is all contained in a different database on a different server.

“Because every connection is recorded [on the second server], it can create some two million entries a day and as such most providers either turn off this feature or delete the logs daily as having so many entries slows down the system down used for actual streams,” he explains.

Our source says that this data would likely to have been the first to be deleted and is probably “long gone” by now. However, even if the Premier League had obtained it, it’s unlikely they would be able to do much with it due to data protection laws.

“The information was passed to the [Premier League] voluntarily by ACE which means this information has been given from one entity to another without the end users’ consent, not part of the [creditors’ voluntary liquidation] and without a court order to support it. Data Protection right now is taken very seriously in the EU,” he notes.

At this point, it’s probably worth noting that while the word “voluntarily” has been used several times to explain the manner in which Ace handed over its subscribers’ details to the Premier League, the same word can be used to describe the manner in which the £600K settlement amount will be paid.

No one forces someone to pay or hand something over, that’s what the courts are for, and the aim here was to avoid that eventuality.

Other pieces of information culled from various sources suggest that PayPal payment information, limited to amounts only, was also handed over to the Premier League. And, perhaps most importantly (and perhaps predictably) as far as former subscribers are concerned, the football group was more interested in Ace’s upwards supplier chain (the ‘wholesale’ stream suppliers used, for example) than those buying the service.

Finally, while the Premier League is now seeking to send a message to customers that these services are risky to use, it’s difficult to argue with the assertion that it’s unsafe to hand over personal details to an illegal service.

“Ace IPTV’s collapse also highlighted the risk consumers take with their personal data when they sign up to illegal streaming services,” Premier League notes.

TF spoke with three IPTV providers who all confirmed that they don’t care what names and addresses people use to sign up with and that no checks are carried out to make sure they’re correct. However, one concedes that in order to run as a business, this information has to be requested and once a customer types it in, it’s possible that it could be handed over as part of a settlement.

“I’m not going to tell people to put in dummy details, how can I? It’s up to people to use their common sense. If they’re still worried they should give Sky their money because if our backs are against the wall, what do you think is going to happen?” he concludes.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and more. We also have VPN reviews, discounts, offers and coupons.

Police Launch Investigation into Huge Pirate Manga Site Mangamura

Post Syndicated from Andy original https://torrentfreak.com/police-launch-investigation-into-huge-pirate-manga-site-mangamura-180514/

Back in March, Japan’s Chief Cabinet Secretary Yoshihide Suga said that the government was considering measures to prohibit access to pirate sites.

While protecting all content is the overall aim, it became clear that the government was determined to protect Japan’s successful manga and anime industries.

It didn’t take long for a reaction. On Friday April 13, the government introduced emergency website blocking measures, seeking cooperation from the country’s ISPs.

NTT Communications Corp., NTT Docomo Inc. and NTT Plala Inc., quickly announced they would block three leading pirate sites – Mangamura, AniTube! and MioMio which have a huge following in Japan. However, after taking the country by storm during the past two years, Mangamura had already called it quits.

On April 17, in the wake of the government announcement, Mangamura disappeared. It’s unclear whether its vanishing act was directly connected to recent developments but a program on national public broadcasting organization NHK, which claimed to have traced the site’s administrators back to the United States, Ukraine, and other regions, can’t have helped.

Further details released this morning reveal the intense pressure Mangamura was under. With 100 million visits a month it was bound to attract attention and according to Mainichi, several publishing giants ran out of patience last year and reported the platform to the authorities.

Kodansha, Japan’s largest publisher, and three other companies filed criminal complaints with Fukuoka Prefectural Police, Oita Prefectural Police, and other law enforcement departments, claiming the site violated their rights.

“The complaints, which were lodged against an unknown suspect or suspects, were filed on behalf of manga artists who are copyright holders to the pirated works, including Hajime Isayama and Eiichiro Oda, known for their wildly popular ‘Shingeki no Kyojin’ (‘Attack on Titan,’ published by Kodansha) and ‘One Piece’ (Shueisha Inc.), respectively,” the publication reports.

Mangamura launch in January 2016 and became a huge hit in Japan. Anti-piracy group Content Overseas Distribution Association (CODA), which counts publishing giant Kodansha among its members, reports that between September 2017 and February 2018, the site was accessed 620 million times.

Based on a “one visit, one manga title read” formula, CODA estimates that the site caused damages to the manga industry of 319.2 billion yen – around US$2.91 billion.

As a result, police are now stepping up their efforts to identify Mangamura’s operators. Whether that will prove fruitful will remain to be seen but in the meantime, Japan’s site-blocking efforts continue to cause controversy.

As reported last month, lawyer and NTT customer Yuichi Nakazawa launched legal action against NTT, demanding that the corporation immediately end its site-blocking operations.

“NTT’s decision was made arbitrarily on the site without any legal basis. No matter how legitimate the objective of copyright infringement is, it is very dangerous,” Nakazawa told TorrentFreak.

“I felt that ‘freedom,’ which is an important value of the Internet, was threatened. Actually, when the interruption of communications had begun, the company thought it would be impossible to reverse the situation, so I filed a lawsuit at this stage.”

Japan’s Constitution and its Telecommunications Business Act both have “no censorship” clauses, meaning that site-blocking has the potential to be ruled illegal. It’s also illegal in Japan to invade the privacy of Internet users’ communications, which some observers have argued is necessary if users are to be prevented from accessing pirate sites.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and more. We also have VPN reviews, discounts, offers and coupons.

Pirate IPTV Service Goes Bust After Premier League Deal, Exposing Users

Post Syndicated from Andy original https://torrentfreak.com/pirate-iptv-service-goes-bust-after-premier-league-deal-exposing-users-180913/

For those out of the loop, unauthorized IPTV services offering many thousands of unlicensed channels have been gaining in popularity in recent years. They’re relatively cheap, fairly reliable, and offer acceptable levels of service.

They are, however, a huge thorn in the side of rightsholders who are desperate to bring them to their knees. One such organization is the UK’s Premier League, which has been disrupting IPTV services over the past year, hoping they’ll shut down.

Most have simply ridden the wave of blocks but one provider, Ace Hosting in the UK, showed signs of stress last year, revealing that it would no longer sell new subscriptions. There was little doubt in most people’s minds that the Premier League had gotten uncomfortably close to the IPTV provider.

Now, many months later, the amazing story can be told. It’s both incredible and shocking and will leave many shaking their heads in disbelief. First up, some background.

Doing things ‘properly’ – incorporation of a pirate service…

Considering how most operators of questionable services like to stay in the shade, it may come as a surprise to learn that Ace Hosting Limited is a proper company. Incorporated and registered at Companies House on January 3, 2017, Ace has two registered directors – family team Ian and Judith Isaac.

In common with several other IPTV operators in the UK who are also officially registered with the authorities, Ace Hosting has never filed any meaningful accounts. There’s a theory that the corporate structure is basically one of convenience, one that allows for the handling of large volumes of cash while limiting liability. The downside, of course, is that people are often more easily identified, in part due to the comprehensive paper trail.

Thanks to what can only be described as a slow-motion train wreck, the Ace Hosting debacle is revealing a bewildering set of circumstances. Last December, when Ace said it would stop signing up new members due to legal pressure, a serious copyright threat had already been filed against it.

Premier League v Ace Hosting

Documents seen by TorrentFreak reveal that the Premier League sent legal threats to Ace Hosting on December 15, 2017, just days before the subscription closure announcement. Somewhat surprisingly, Ace apparently felt it could pay the Premier League a damages amount and keep on trading.

But early March 2018, with the Premier League threatening Ace with all kinds of bad things, the company made a strange announcement.

“The ISPs in the UK and across Europe have recently become much more aggressive in blocking our service while football games are in progress,” Ace said in a statement.

“In order to get ourselves off of the ISP blacklist we are going to black out the EPL games for all users (including VPN users) starting on Monday. We believe that this will enable us to rebuild the bypass process and successfully provide you with all EPL games.”

It seems doubtful that Ace really intended to thumb its nose at the Premier League but it had continued to sell subscriptions since receiving threats in December, so all things seemed possible. But on March 24 that all changed, when Ace effectively announced its closure.

Premier League 1, Ace Hosting 0

“It is with sorrow that we announce that we are no longer accepting renewals, upgrades to existing subscriptions or the purchase of new credits. We plan to support existing subscriptions until they expire,” the team wrote.

“EPL games including highlights continue to be blocked and are not expected to be reinstated before the end of the season.”

Indeed, just days later the Premier League demanded a six-figure settlement sum from Ace Hosting, presumably to make a lawsuit disappear. It was the straw that broke the camel’s back.

“When the proposed damages amount was received it was clear that the Company would not be able to cover the cost and that there was a very high probability that even with a negotiated settlement that the Company was insolvent,” documents relating to Ace’s liquidation read.

At this point, Ace says it immediately ceased trading but while torrent sites usually shut down and disappear into the night, Ace’s demise is now a matter of record.

Creditors – the good, the bad, and the ugly

On April 11, 2018, Ace’s directors contacted business recovery and insolvency specialists Begbies Traynor (Central) LLP to obtain advice on the company’s financial position. Begbies Traynor was instructed by Ace on April 23 and on May 8, Ace Hosting director Ian Isaac determined that his company could not pay its debts.

First the good news. According to an official report, Ace Hosting has considerable cash in the bank – £255,472.00 to be exact. Now the bad news – Ace has debts of £717,278.84. – the details of which are intriguing to say the least.

First up, Ace has ‘trade creditors’ to whom it owes £104,356. The vast majority of this sum is a settlement Ace agreed to pay to the Premier League.

“The directors entered into a settlement agreement with the Football Association Premier League Limited prior to placing the Company into liquidation as a result of a purported copyright infringement. However, there is a residual claim from the Football Association Premier League Limited which is included within trade creditors totaling £100,000,” Ace’s statement of affairs reads.

Bizarrely (given the nature of the business, at least) Ace also owes £260,000 to Her Majesty’s Revenue and Customs (HMRC) in unpaid VAT and corporation tax, which is effectively the government’s cut of the pirate IPTV business’s labors.

Former Ace Hosting subscriber? Your cash is as good as gone

Finally – and this is where things get a bit sweaty for Joe Public – there are 15,768 “consumer creditors”, split between ‘retail’ and ‘business’ customers of the service. Together they are owed a staggering £353,000.

Although the documentation isn’t explicit, retail customers appear to be people who have purchased an Ace IPTV subscription that still had time to run when the service closed down. Business customers seem likely to be resellers of the service, who purchased ‘credits’ and didn’t get time to sell them before Ace disappeared.

The poison chalice here is that those who are owed money by Ace can actually apply to get some of it back, but that could be extremely risky.

“Creditor claims have not yet been adjudicated but we estimate that the majority of customers who paid for subscription services will receive less than £3 if there is a distribution to unsecured creditors. Furthermore, customer details will be passed to the relevant authorities if there is any suggestion of unlawful conduct,” documentation reads.

We spoke with a former Ace customer who had this to say about the situation.

“It was generally a good service notwithstanding their half-arsed attempts to evade the EPL block. At its heart there were people who seemed to know how to operate a decent service, although the customer-facing side of things was not the greatest,” he said.

“And no, I won’t be claiming a refund. I went into it with my eyes fully open so I don’t hold anyone responsible, except myself. In any case, anyone who wants a refund has to complete a claim form and provide proof of ID (LOL).”

The bad news for former subscribers continues…potentially

While it’s likely that most people will forgo their £3, the bad news isn’t over for subscribers. Begbies Traynor is warning that the liquidators will decide whether to hand over subscribers’ personal details to the Premier League and/or the authorities.

In any event, sometime in the next couple of weeks the names and addresses of all subscribers will be made “available for inspection” at an address in Wiltshire for two days, meaning that any interested parties could potentially gain access to sensitive information.

The bottom line is that Ace Hosting is in the red to the tune of £461,907 and will eventually disappear into the bowels of history. Whether its operators will have to answer for their conduct will remain to be seen but it seems unimaginable at this stage that things will end well.

Subscribers probably won’t get sucked in but in a story as bizarre as this one, anything could yet happen.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and more. We also have VPN reviews, discounts, offers and coupons.

‘Anonymous’ Hackers Deface Russian Govt. Site to Protest Web-Blocking (NSFW)

Post Syndicated from Andy original https://torrentfreak.com/anonymous-hackers-deface-russian-govt-site-to-protest-web-blocking-nsfw-180512/

Last month, Russian authorities demonstrated that when an entity breaks local Internet rules, no stone will be left unturned to make them pay, whatever the cost.

The disaster waiting to happen began when encrypted messaging service Telegram refused to hand over its encryption keys to the state. In response, the Federal Security Service filed a lawsuit, which it won, compelling it Telegram do so. With no response, Roscomnadzor obtained a court order to have Telegram blocked.

In a massive response, Russian ISPs – at Roscomnadzor’s behest – began mass-blocking IP addresses on a massive scale. Millions of IP addresses belong to Amazon, Google and other innocent parties were rendered inaccessible in Russia, causing chaos online.

Even VPN providers were targeted for facilitating access to Telegram but while the service strained under the pressure, it never went down and continues to function today.

In the wake of the operation there has been some attempt at a cleanup job, with Roscomnadzor announcing this week that it had unblocked millions of IP addresses belonging to Google.

“As part of a package of the measures to enforce the court’s decision on Telegram, Roskomnadzor has removed six Google subnets (more than 3.7 million IP-addresses) from the blocklist,” the telecoms watchdog said in a statement.

“In this case, the IP addresses of Telegram, which are part of these subnets, are fully installed and blocked. Subnets are unblocked in order to ensure the correct operation of third-party Internet resources.”

But while Roscomnadzor attempts to calm the seas, those angered by Russia’s carpet-bombing of the Internet were determined to make their voices heard. Hackers attacked the website of the Federal Agency for International Cooperation this week, defacing it with scathing criticism combined with NSFW suggestions and imagery.

“Greetings, Roskomnadzor,” the message began.

“Your recent destructive actions towards the Russian internet sector have led us to believe that you are nothing but a bunch of incompetent mindless worms. You shall not be able to continue this pointless vandalism any further.”

Signing off with advice to consider the defacement as a “final warning”, the hackers disappeared into the night after leaving a simple signature.

“Yours, Anonymous,” they wrote.

But the hackers weren’t done yet. In a NSFW cartoon strip that probably explains itself, ‘Anonymous’ suggested that Roscomnadzor should perhaps consider blocking itself, with the implement depicted in the final frame.

“Anus, block yourself Roscomnadzor”

But while Russia’s attack on Telegram raises eyebrows worldwide, the actions of those in authority continue to baffle.

Last week, Prime Minister Dmitry Medvedev’s press secretary, Natalia Timakova, publicly advised a colleague to circumvent the Telegram blockade using a VPN, effectively undermining the massive efforts of the authorities. This week the head of Roscomnadzor only added to the confusion.

Effectively quashing rumors that he’d resigned due to the Telegram fiasco, Alexander Zharov had a conversation with the editor-in-chief of radio station ‘Says Moscow’.

During the liason, which took place during the Victory Parade in Red Square, Zharov was asked how he could be contacted. When Telegram was presented as a potential method, Zharov confirmed that he could be reached via the platform.

Finally, in a move that’s hoped could bring an end to the attack on the platform and others like it, Telegram filed an appeal this week challenging a decision by the Supreme Court of Russia which allows the Federal Security Service to demand access to encryption keys.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and more. We also have VPN reviews, discounts, offers and coupons.

Analyze Apache Parquet optimized data using Amazon Kinesis Data Firehose, Amazon Athena, and Amazon Redshift

Post Syndicated from Roy Hasson original https://aws.amazon.com/blogs/big-data/analyzing-apache-parquet-optimized-data-using-amazon-kinesis-data-firehose-amazon-athena-and-amazon-redshift/

Amazon Kinesis Data Firehose is the easiest way to capture and stream data into a data lake built on Amazon S3. This data can be anything—from AWS service logs like AWS CloudTrail log files, Amazon VPC Flow Logs, Application Load Balancer logs, and others. It can also be IoT events, game events, and much more. To efficiently query this data, a time-consuming ETL (extract, transform, and load) process is required to massage and convert the data to an optimal file format, which increases the time to insight. This situation is less than ideal, especially for real-time data that loses its value over time.

To solve this common challenge, Kinesis Data Firehose can now save data to Amazon S3 in Apache Parquet or Apache ORC format. These are optimized columnar formats that are highly recommended for best performance and cost-savings when querying data in S3. This feature directly benefits you if you use Amazon Athena, Amazon Redshift, AWS Glue, Amazon EMR, or any other big data tools that are available from the AWS Partner Network and through the open-source community.

Amazon Connect is a simple-to-use, cloud-based contact center service that makes it easy for any business to provide a great customer experience at a lower cost than common alternatives. Its open platform design enables easy integration with other systems. One of those systems is Amazon Kinesis—in particular, Kinesis Data Streams and Kinesis Data Firehose.

What’s really exciting is that you can now save events from Amazon Connect to S3 in Apache Parquet format. You can then perform analytics using Amazon Athena and Amazon Redshift Spectrum in real time, taking advantage of this key performance and cost optimization. Of course, Amazon Connect is only one example. This new capability opens the door for a great deal of opportunity, especially as organizations continue to build their data lakes.

Amazon Connect includes an array of analytics views in the Administrator dashboard. But you might want to run other types of analysis. In this post, I describe how to set up a data stream from Amazon Connect through Kinesis Data Streams and Kinesis Data Firehose and out to S3, and then perform analytics using Athena and Amazon Redshift Spectrum. I focus primarily on the Kinesis Data Firehose support for Parquet and its integration with the AWS Glue Data Catalog, Amazon Athena, and Amazon Redshift.

Solution overview

Here is how the solution is laid out:

 

 

The following sections walk you through each of these steps to set up the pipeline.

1. Define the schema

When Kinesis Data Firehose processes incoming events and converts the data to Parquet, it needs to know which schema to apply. The reason is that many times, incoming events contain all or some of the expected fields based on which values the producers are advertising. A typical process is to normalize the schema during a batch ETL job so that you end up with a consistent schema that can easily be understood and queried. Doing this introduces latency due to the nature of the batch process. To overcome this issue, Kinesis Data Firehose requires the schema to be defined in advance.

To see the available columns and structures, see Amazon Connect Agent Event Streams. For the purpose of simplicity, I opted to make all the columns of type String rather than create the nested structures. But you can definitely do that if you want.

The simplest way to define the schema is to create a table in the Amazon Athena console. Open the Athena console, and paste the following create table statement, substituting your own S3 bucket and prefix for where your event data will be stored. A Data Catalog database is a logical container that holds the different tables that you can create. The default database name shown here should already exist. If it doesn’t, you can create it or use another database that you’ve already created.

CREATE EXTERNAL TABLE default.kfhconnectblog (
  awsaccountid string,
  agentarn string,
  currentagentsnapshot string,
  eventid string,
  eventtimestamp string,
  eventtype string,
  instancearn string,
  previousagentsnapshot string,
  version string
)
STORED AS parquet
LOCATION 's3://your_bucket/kfhconnectblog/'
TBLPROPERTIES ("parquet.compression"="SNAPPY")

That’s all you have to do to prepare the schema for Kinesis Data Firehose.

2. Define the data streams

Next, you need to define the Kinesis data streams that will be used to stream the Amazon Connect events.  Open the Kinesis Data Streams console and create two streams.  You can configure them with only one shard each because you don’t have a lot of data right now.

3. Define the Kinesis Data Firehose delivery stream for Parquet

Let’s configure the Data Firehose delivery stream using the data stream as the source and Amazon S3 as the output. Start by opening the Kinesis Data Firehose console and creating a new data delivery stream. Give it a name, and associate it with the Kinesis data stream that you created in Step 2.

As shown in the following screenshot, enable Record format conversion (1) and choose Apache Parquet (2). As you can see, Apache ORC is also supported. Scroll down and provide the AWS Glue Data Catalog database name (3) and table names (4) that you created in Step 1. Choose Next.

To make things easier, the output S3 bucket and prefix fields are automatically populated using the values that you defined in the LOCATION parameter of the create table statement from Step 1. Pretty cool. Additionally, you have the option to save the raw events into another location as defined in the Source record S3 backup section. Don’t forget to add a trailing forward slash “ / “ so that Data Firehose creates the date partitions inside that prefix.

On the next page, in the S3 buffer conditions section, there is a note about configuring a large buffer size. The Parquet file format is highly efficient in how it stores and compresses data. Increasing the buffer size allows you to pack more rows into each output file, which is preferred and gives you the most benefit from Parquet.

Compression using Snappy is automatically enabled for both Parquet and ORC. You can modify the compression algorithm by using the Kinesis Data Firehose API and update the OutputFormatConfiguration.

Be sure to also enable Amazon CloudWatch Logs so that you can debug any issues that you might run into.

Lastly, finalize the creation of the Firehose delivery stream, and continue on to the next section.

4. Set up the Amazon Connect contact center

After setting up the Kinesis pipeline, you now need to set up a simple contact center in Amazon Connect. The Getting Started page provides clear instructions on how to set up your environment, acquire a phone number, and create an agent to accept calls.

After setting up the contact center, in the Amazon Connect console, choose your Instance Alias, and then choose Data Streaming. Under Agent Event, choose the Kinesis data stream that you created in Step 2, and then choose Save.

At this point, your pipeline is complete.  Agent events from Amazon Connect are generated as agents go about their day. Events are sent via Kinesis Data Streams to Kinesis Data Firehose, which converts the event data from JSON to Parquet and stores it in S3. Athena and Amazon Redshift Spectrum can simply query the data without any additional work.

So let’s generate some data. Go back into the Administrator console for your Amazon Connect contact center, and create an agent to handle incoming calls. In this example, I creatively named mine Agent One. After it is created, Agent One can get to work and log into their console and set their availability to Available so that they are ready to receive calls.

To make the data a bit more interesting, I also created a second agent, Agent Two. I then made some incoming and outgoing calls and caused some failures to occur, so I now have enough data available to analyze.

5. Analyze the data with Athena

Let’s open the Athena console and run some queries. One thing you’ll notice is that when we created the schema for the dataset, we defined some of the fields as Strings even though in the documentation they were complex structures.  The reason for doing that was simply to show some of the flexibility of Athena to be able to parse JSON data. However, you can define nested structures in your table schema so that Kinesis Data Firehose applies the appropriate schema to the Parquet file.

Let’s run the first query to see which agents have logged into the system.

The query might look complex, but it’s fairly straightforward:

WITH dataset AS (
  SELECT 
    from_iso8601_timestamp(eventtimestamp) AS event_ts,
    eventtype,
    -- CURRENT STATE
    json_extract_scalar(
      currentagentsnapshot,
      '$.agentstatus.name') AS current_status,
    from_iso8601_timestamp(
      json_extract_scalar(
        currentagentsnapshot,
        '$.agentstatus.starttimestamp')) AS current_starttimestamp,
    json_extract_scalar(
      currentagentsnapshot, 
      '$.configuration.firstname') AS current_firstname,
    json_extract_scalar(
      currentagentsnapshot,
      '$.configuration.lastname') AS current_lastname,
    json_extract_scalar(
      currentagentsnapshot, 
      '$.configuration.username') AS current_username,
    json_extract_scalar(
      currentagentsnapshot, 
      '$.configuration.routingprofile.defaultoutboundqueue.name') AS               current_outboundqueue,
    json_extract_scalar(
      currentagentsnapshot, 
      '$.configuration.routingprofile.inboundqueues[0].name') as current_inboundqueue,
    -- PREVIOUS STATE
    json_extract_scalar(
      previousagentsnapshot, 
      '$.agentstatus.name') as prev_status,
    from_iso8601_timestamp(
      json_extract_scalar(
        previousagentsnapshot, 
       '$.agentstatus.starttimestamp')) as prev_starttimestamp,
    json_extract_scalar(
      previousagentsnapshot, 
      '$.configuration.firstname') as prev_firstname,
    json_extract_scalar(
      previousagentsnapshot, 
      '$.configuration.lastname') as prev_lastname,
    json_extract_scalar(
      previousagentsnapshot, 
      '$.configuration.username') as prev_username,
    json_extract_scalar(
      previousagentsnapshot, 
      '$.configuration.routingprofile.defaultoutboundqueue.name') as current_outboundqueue,
    json_extract_scalar(
      previousagentsnapshot, 
      '$.configuration.routingprofile.inboundqueues[0].name') as prev_inboundqueue
  from kfhconnectblog
  where eventtype <> 'HEART_BEAT'
)
SELECT
  current_status as status,
  current_username as username,
  event_ts
FROM dataset
WHERE eventtype = 'LOGIN' AND current_username <> ''
ORDER BY event_ts DESC

The query output looks something like this:

Here is another query that shows the sessions each of the agents engaged with. It tells us where they were incoming or outgoing, if they were completed, and where there were missed or failed calls.

WITH src AS (
  SELECT
     eventid,
     json_extract_scalar(currentagentsnapshot, '$.configuration.username') as username,
     cast(json_extract(currentagentsnapshot, '$.contacts') AS ARRAY(JSON)) as c,
     cast(json_extract(previousagentsnapshot, '$.contacts') AS ARRAY(JSON)) as p
  from kfhconnectblog
),
src2 AS (
  SELECT *
  FROM src CROSS JOIN UNNEST (c, p) AS contacts(c_item, p_item)
),
dataset AS (
SELECT 
  eventid,
  username,
  json_extract_scalar(c_item, '$.contactid') as c_contactid,
  json_extract_scalar(c_item, '$.channel') as c_channel,
  json_extract_scalar(c_item, '$.initiationmethod') as c_direction,
  json_extract_scalar(c_item, '$.queue.name') as c_queue,
  json_extract_scalar(c_item, '$.state') as c_state,
  from_iso8601_timestamp(json_extract_scalar(c_item, '$.statestarttimestamp')) as c_ts,
  
  json_extract_scalar(p_item, '$.contactid') as p_contactid,
  json_extract_scalar(p_item, '$.channel') as p_channel,
  json_extract_scalar(p_item, '$.initiationmethod') as p_direction,
  json_extract_scalar(p_item, '$.queue.name') as p_queue,
  json_extract_scalar(p_item, '$.state') as p_state,
  from_iso8601_timestamp(json_extract_scalar(p_item, '$.statestarttimestamp')) as p_ts
FROM src2
)
SELECT 
  username,
  c_channel as channel,
  c_direction as direction,
  p_state as prev_state,
  c_state as current_state,
  c_ts as current_ts,
  c_contactid as id
FROM dataset
WHERE c_contactid = p_contactid
ORDER BY id DESC, current_ts ASC

The query output looks similar to the following:

6. Analyze the data with Amazon Redshift Spectrum

With Amazon Redshift Spectrum, you can query data directly in S3 using your existing Amazon Redshift data warehouse cluster. Because the data is already in Parquet format, Redshift Spectrum gets the same great benefits that Athena does.

Here is a simple query to show querying the same data from Amazon Redshift. Note that to do this, you need to first create an external schema in Amazon Redshift that points to the AWS Glue Data Catalog.

SELECT 
  eventtype,
  json_extract_path_text(currentagentsnapshot,'agentstatus','name') AS current_status,
  json_extract_path_text(currentagentsnapshot, 'configuration','firstname') AS current_firstname,
  json_extract_path_text(currentagentsnapshot, 'configuration','lastname') AS current_lastname,
  json_extract_path_text(
    currentagentsnapshot,
    'configuration','routingprofile','defaultoutboundqueue','name') AS current_outboundqueue,
FROM default_schema.kfhconnectblog

The following shows the query output:

Summary

In this post, I showed you how to use Kinesis Data Firehose to ingest and convert data to columnar file format, enabling real-time analysis using Athena and Amazon Redshift. This great feature enables a level of optimization in both cost and performance that you need when storing and analyzing large amounts of data. This feature is equally important if you are investing in building data lakes on AWS.

 


Additional Reading

If you found this post useful, be sure to check out Analyzing VPC Flow Logs with Amazon Kinesis Firehose, Amazon Athena, and Amazon QuickSight and Work with partitioned data in AWS Glue.


About the Author

Roy Hasson is a Global Business Development Manager for AWS Analytics. He works with customers around the globe to design solutions to meet their data processing, analytics and business intelligence needs. Roy is big Manchester United fan cheering his team on and hanging out with his family.

 

 

 

Court Orders Pirate IPTV Linker to Shut Down or Face Penalties Up to €1.25m

Post Syndicated from Andy original https://torrentfreak.com/court-orders-pirate-iptv-linker-to-shut-down-or-face-penalties-up-to-e1-25m-180911/

There are few things guaranteed in life. Death, taxes, and lawsuits filed regularly by Dutch anti-piracy outfit BREIN.

One of its most recent targets was Netherlands-based company Leaper Beheer BV, which also traded under the names Flickstore, Dump Die Deal and Live TV Store. BREIN filed a complaint at the Limburg District Court in Maastricht, claiming that Leaper provides access to unlicensed live TV streams and on-demand movies.

The anti-piracy outfit claimed that around 4,000 live channels were on offer, including Fox Sports, movie channels, commercial and public channels. These could be accessed after the customer made a payment which granted access to a unique activation code which could be entered into a set-top box.

BREIN told the court that the code returned an .M3U playlist, which was effectively a hyperlink to IPTV channels and more than 1,000 movies being made available without permission from their respective copyright holders. As such, this amounted to a communication to the public in contravention of the EU Copyright Directive, BREIN argued.

In its defense, Leaper said that it effectively provided a convenient link-shortening service for content that could already be found online in other ways. The company argued that it is not a distributor of content itself and did not make available anything that wasn’t already public. The company added that it was completely down to the consumer whether illegal content was viewed or not.

The key question for the Court was whether Leaper did indeed make a new “communication to the public” under the EU Copyright Directive, a standard the Court of Justice of the European Union (CJEU) says should be interpreted in a manner that provides a high level of protection for rightsholders.

The Court took a three-point approach in arriving at its decision.

  • Did Leaper act in a deliberate manner when providing access to copyright content, especially when its intervention provided access to consumers who would not ordinarily have access to that content?
  • Did Leaper communicate the works via a new method to a new audience?
  • Did Leaper have a profit motive when it communicated works to the public?
  • The Court found that Leaper did communicate works to the public and intervened “with full knowledge of the consequences of its conduct” when it gave its customers access to protected works.

    “Access to [the content] in a different way would be difficult for those customers, if Leaper were not to provide its services in question,” the Court’s decision reads.

    “Leaper reaches an indeterminate number of potential recipients who can take cognizance of the protected works and form a new audience. The purchasers who register with Leaper are to be regarded as recipients who were not taken into account by the rightful claimants when they gave permission for the original communication of their work to the public.”

    With that, the Court ordered Leaper to cease-and-desist facilitating access to unlicensed streams within 48 hours of the judgment, with non-compliance penalties of 5,000 euros per IPTV subscription sold, link offered, or days exceeded, to a maximum of one million euros.

    But the Court didn’t stop there.

    “Leaper must submit a statement audited by an accountant, supported by (clear, readable copies of) all relevant documents, within 12 days of notification of this judgment of all the relevant (contact) details of the (person or legal persons) with whom the company has had contact regarding the provision of IPTV subscriptions and/or the provision of hyperlinks to sources where films and (live) broadcasts are evidently offered without the permission of the entitled parties,” the Court ruled.

    Failure to comply with this aspect of the ruling will lead to more penalties of 5,000 euros per day up to a maximum of 250,000 euros. Leaper was also ordered to pay BREIN’s costs of 20,700 euros.

    Describing the people behind Leaper as “crooks” who previously sold media boxes with infringing addons (as previously determined to be illegal in the Filmspeler case), BREIN chief Tim Kuik says that a switch of strategy didn’t help them evade the law.

    “[Leaper] sold a link to consumers that gave access to unauthorized content, i.e. pay-TV channels as well as video-on-demand films and series,” BREIN chief Tim Kuik informs TorrentFreak.

    “They did it for profit and should have checked whether the content was authorized. They did not and in fact were aware the content was unauthorized. Which means they are clearly infringing copyright.

    “This is evident from the CJEU case law in GS Media as well as Filmspeler and The Pirate Bay, aka the Dutch trilogy because the three cases came from the Netherlands, but these rulings are applicable throughout the EU.

    “They just keep at it knowing they’re cheating and we’ll take them to the cleaners,” Kuik concludes.

    Source: TF, for the latest info on copyright, file-sharing, torrent sites and more. We also have VPN reviews, discounts, offers and coupons.

    Securing Your Cryptocurrency

    Post Syndicated from Roderick Bauer original https://www.backblaze.com/blog/backing-up-your-cryptocurrency/

    Securing Your Cryptocurrency

    In our blog post on Tuesday, Cryptocurrency Security Challenges, we wrote about the two primary challenges faced by anyone interested in safely and profitably participating in the cryptocurrency economy: 1) make sure you’re dealing with reputable and ethical companies and services, and, 2) keep your cryptocurrency holdings safe and secure.

    In this post, we’re going to focus on how to make sure you don’t lose any of your cryptocurrency holdings through accident, theft, or carelessness. You do that by backing up the keys needed to sell or trade your currencies.

    $34 Billion in Lost Value

    Of the 16.4 million bitcoins said to be in circulation in the middle of 2017, close to 3.8 million may have been lost because their owners no longer are able to claim their holdings. Based on today’s valuation, that could total as much as $34 billion dollars in lost value. And that’s just bitcoins. There are now over 1,500 different cryptocurrencies, and we don’t know how many of those have been misplaced or lost.



    Now that some cryptocurrencies have reached (at least for now) staggering heights in value, it’s likely that owners will be more careful in keeping track of the keys needed to use their cryptocurrencies. For the ones already lost, however, the owners have been separated from their currencies just as surely as if they had thrown Benjamin Franklins and Grover Clevelands over the railing of a ship.

    The Basics of Securing Your Cryptocurrencies

    In our previous post, we reviewed how cryptocurrency keys work, and the common ways owners can keep track of them. A cryptocurrency owner needs two keys to use their currencies: a public key that can be shared with others is used to receive currency, and a private key that must be kept secure is used to spend or trade currency.

    Many wallets and applications allow the user to require extra security to access them, such as a password, or iris, face, or thumb print scan. If one of these options is available in your wallets, take advantage of it. Beyond that, it’s essential to back up your wallet, either using the backup feature built into some applications and wallets, or manually backing up the data used by the wallet. When backing up, it’s a good idea to back up the entire wallet, as some wallets require additional private data to operate that might not be apparent.

    No matter which backup method you use, it is important to back up often and have multiple backups, preferable in different locations. As with any valuable data, a 3-2-1 backup strategy is good to follow, which ensures that you’ll have a good backup copy if anything goes wrong with one or more copies of your data.

    One more caveat, don’t reuse passwords. This applies to all of your accounts, but is especially important for something as critical as your finances. Don’t ever use the same password for more than one account. If security is breached on one of your accounts, someone could connect your name or ID with other accounts, and will attempt to use the password there, as well. Consider using a password manager such as LastPass or 1Password, which make creating and using complex and unique passwords easy no matter where you’re trying to sign in.

    Approaches to Backing Up Your Cryptocurrency Keys

    There are numerous ways to be sure your keys are backed up. Let’s take them one by one.

    1. Automatic backups using a backup program

    If you’re using a wallet program on your computer, for example, Bitcoin Core, it will store your keys, along with other information, in a file. For Bitcoin Core, that file is wallet.dat. Other currencies will use the same or a different file name and some give you the option to select a name for the wallet file.

    To back up the wallet.dat or other wallet file, you might need to tell your backup program to explicitly back up that file. Users of Backblaze Backup don’t have to worry about configuring this, since by default, Backblaze Backup will back up all data files. You should determine where your particular cryptocurrency, wallet, or application stores your keys, and make sure the necessary file(s) are backed up if your backup program requires you to select which files are included in the backup.

    Backblaze B2 is an option for those interested in low-cost and high security cloud storage of their cryptocurrency keys. Backblaze B2 supports 2-factor verification for account access, works with a number of apps that support automatic backups with encryption, error-recovery, and versioning, and offers an API and command-line interface (CLI), as well. The first 10GB of storage is free, which could be all one needs to store encrypted cryptocurrency keys.

    2. Backing up by exporting keys to a file

    Apps and wallets will let you export your keys from your app or wallet to a file. Once exported, your keys can be stored on a local drive, USB thumb drive, DAS, NAS, or in the cloud with any cloud storage or sync service you wish. Encrypting the file is strongly encouraged — more on that later. If you use 1Password or LastPass, or other secure notes program, you also could store your keys there.

    3. Backing up by saving a mnemonic recovery seed

    A mnemonic phrase, mnemonic recovery phrase, or mnemonic seed is a list of words that stores all the information needed to recover a cryptocurrency wallet. Many wallets will have the option to generate a mnemonic backup phrase, which can be written down on paper. If the user’s computer no longer works or their hard drive becomes corrupted, they can download the same wallet software again and use the mnemonic recovery phrase to restore their keys.

    The phrase can be used by anyone to recover the keys, so it must be kept safe. Mnemonic phrases are an excellent way of backing up and storing cryptocurrency and so they are used by almost all wallets.

    A mnemonic recovery seed is represented by a group of easy to remember words. For example:

    eye female unfair moon genius pipe nuclear width dizzy forum cricket know expire purse laptop scale identify cube pause crucial day cigar noise receive

    The above words represent the following seed:

    0a5b25e1dab6039d22cd57469744499863962daba9d2844243fec 9c0313c1448d1a0b2cd9e230a78775556f9b514a8be45802c2808e fd449a20234e9262dfa69

    These words have certain properties:

    • The first four letters are enough to unambiguously identify the word.
    • Similar words are avoided (such as: build and built).

    Bitcoin and most other cryptocurrencies such as Litecoin, Ethereum, and others use mnemonic seeds that are 12 to 24 words long. Other currencies might use different length seeds.

    4. Physical backups — Paper, Metal

    Some cryptocurrency holders believe that their backup, or even all their cryptocurrency account information, should be stored entirely separately from the internet to avoid any risk of their information being compromised through hacks, exploits, or leaks. This type of storage is called “cold storage.” One method of cold storage involves printing out the keys to a piece of paper and then erasing any record of the keys from all computer systems. The keys can be entered into a program from the paper when needed, or scanned from a QR code printed on the paper.

    Printed public and private keys

    Printed public and private keys

    Some who go to extremes suggest separating the mnemonic needed to access an account into individual pieces of paper and storing those pieces in different locations in the home or office, or even different geographical locations. Some say this is a bad idea since it could be possible to reconstruct the mnemonic from one or more pieces. How diligent you wish to be in protecting these codes is up to you.

    Mnemonic recovery phrase booklet

    Mnemonic recovery phrase booklet

    There’s another option that could make you the envy of your friends. That’s the CryptoSteel wallet, which is a stainless steel metal case that comes with more than 250 stainless steel letter tiles engraved on each side. Codes and passwords are assembled manually from the supplied part-randomized set of tiles. Users are able to store up to 96 characters worth of confidential information. Cryptosteel claims to be fireproof, waterproof, and shock-proof.

    image of a Cryptosteel cold storage device

    Cryptosteel cold wallet

    Of course, if you leave your Cryptosteel wallet in the pocket of a pair of ripped jeans that gets thrown out by the housekeeper, as happened to the character Russ Hanneman on the TV show Silicon Valley in last Sunday’s episode, then you’re out of luck. That fictional billionaire investor lost a USB drive with $300 million in cryptocoins. Let’s hope that doesn’t happen to you.

    Encryption & Security

    Whether you store your keys on your computer, an external disk, a USB drive, DAS, NAS, or in the cloud, you want to make sure that no one else can use those keys. The best way to handle that is to encrypt the backup.

    With Backblaze Backup for Windows and Macintosh, your backups are encrypted in transmission to the cloud and on the backup server. Users have the option to add an additional level of security by adding a Personal Encryption Key (PEK), which secures their private key. Your cryptocurrency backup files are secure in the cloud. Using our web or mobile interface, previous versions of files can be accessed, as well.

    Our object storage cloud offering, Backblaze B2, can be used with a variety of applications for Windows, Macintosh, and Linux. With B2, cryptocurrency users can choose whichever method of encryption they wish to use on their local computers and then upload their encrypted currency keys to the cloud. Depending on the client used, versioning and life-cycle rules can be applied to the stored files.

    Other backup programs and systems provide some or all of these capabilities, as well. If you are backing up to a local drive, it is a good idea to encrypt the local backup, which is an option in some backup programs.

    Address Security

    Some experts recommend using a different address for each cryptocurrency transaction. Since the address is not the same as your wallet, this means that you are not creating a new wallet, but simply using a new identifier for people sending you cryptocurrency. Creating a new address is usually as easy as clicking a button in the wallet.

    One of the chief advantages of using a different address for each transaction is anonymity. Each time you use an address, you put more information into the public ledger (blockchain) about where the currency came from or where it went. That means that over time, using the same address repeatedly could mean that someone could map your relationships, transactions, and incoming funds. The more you use that address, the more information someone can learn about you. For more on this topic, refer to Address reuse.

    Note that a downside of using a paper wallet with a single key pair (type-0 non-deterministic wallet) is that it has the vulnerabilities listed above. Each transaction using that paper wallet will add to the public record of transactions associated with that address. Newer wallets, i.e. “deterministic” or those using mnemonic code words support multiple addresses and are now recommended.

    There are other approaches to keeping your cryptocurrency transaction secure. Here are a couple of them.

    Multi-signature

    Multi-signature refers to requiring more than one key to authorize a transaction, much like requiring more than one key to open a safe. It is generally used to divide up responsibility for possession of cryptocurrency. Standard transactions could be called “single-signature transactions” because transfers require only one signature — from the owner of the private key associated with the currency address (public key). Some wallets and apps can be configured to require more than one signature, which means that a group of people, businesses, or other entities all must agree to trade in the cryptocurrencies.

    Deep Cold Storage

    Deep cold storage ensures the entire transaction process happens in an offline environment. There are typically three elements to deep cold storage.

    First, the wallet and private key are generated offline, and the signing of transactions happens on a system not connected to the internet in any manner. This ensures it’s never exposed to a potentially compromised system or connection.

    Second, details are secured with encryption to ensure that even if the wallet file ends up in the wrong hands, the information is protected.

    Third, storage of the encrypted wallet file or paper wallet is generally at a location or facility that has restricted access, such as a safety deposit box at a bank.

    Deep cold storage is used to safeguard a large individual cryptocurrency portfolio held for the long term, or for trustees holding cryptocurrency on behalf of others, and is possibly the safest method to ensure a crypto investment remains secure.

    Keep Your Software Up to Date

    You should always make sure that you are using the latest version of your app or wallet software, which includes important stability and security fixes. Installing updates for all other software on your computer or mobile device is also important to keep your wallet environment safer.

    One Last Thing: Think About Your Testament

    Your cryptocurrency funds can be lost forever if you don’t have a backup plan for your peers and family. If the location of your wallets or your passwords is not known by anyone when you are gone, there is no hope that your funds will ever be recovered. Taking a bit of time on these matters can make a huge difference.

    To the Moon*

    Are you comfortable with how you’re managing and backing up your cryptocurrency wallets and keys? Do you have a suggestion for keeping your cryptocurrencies safe that we missed above? Please let us know in the comments.


    *To the Moon — Crypto slang for a currency that reaches an optimistic price projection.

    The post Securing Your Cryptocurrency appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

    Virginia Beach Police Want Encrypted Radios

    Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2018/05/virginia_beach_.html

    This article says that the Virginia Beach police are looking to buy encrypted radios.

    Virginia Beach police believe encryption will prevent criminals from listening to police communications. They said officer safety would increase and citizens would be better protected.

    Someone should ask them if they want those radios to have a backdoor.

    Creating a 1.3 Million vCPU Grid on AWS using EC2 Spot Instances and TIBCO GridServer

    Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/creating-a-1-3-million-vcpu-grid-on-aws-using-ec2-spot-instances-and-tibco-gridserver/

    Many of my colleagues are fortunate to be able to spend a good part of their day sitting down with and listening to our customers, doing their best to understand ways that we can better meet their business and technology needs. This information is treated with extreme care and is used to drive the roadmap for new services and new features.

    AWS customers in the financial services industry (often abbreviated as FSI) are looking ahead to the Fundamental Review of Trading Book (FRTB) regulations that will come in to effect between 2019 and 2021. Among other things, these regulations mandate a new approach to the “value at risk” calculations that each financial institution must perform in the four hour time window after trading ends in New York and begins in Tokyo. Today, our customers report this mission-critical calculation consumes on the order of 200,000 vCPUs, growing to between 400K and 800K vCPUs in order to meet the FRTB regulations. While there’s still some debate about the magnitude and frequency with which they’ll need to run this expanded calculation, the overall direction is clear.

    Building a Big Grid
    In order to make sure that we are ready to help our FSI customers meet these new regulations, we worked with TIBCO to set up and run a proof of concept grid in the AWS Cloud. The periodic nature of the calculation, along with the amount of processing power and storage needed to run it to completion within four hours, make it a great fit for an environment where a vast amount of cost-effective compute power is available on an on-demand basis.

    Our customers are already using the TIBCO GridServer on-premises and want to use it in the cloud. This product is designed to run grids at enterprise scale. It runs apps in a virtualized fashion, and accepts requests for resources, dynamically provisioning them on an as-needed basis. The cloud version supports Amazon Linux as well as the PostgreSQL-compatible edition of Amazon Aurora.

    Working together with TIBCO, we set out to create a grid that was substantially larger than the current high-end prediction of 800K vCPUs, adding a 50% safety factor and then rounding up to reach 1.3 million vCPUs (5x the size of the largest on-premises grid). With that target in mind, the account limits were raised as follows:

    • Spot Instance Limit – 120,000
    • EBS Volume Limit – 120,000
    • EBS Capacity Limit – 2 PB

    If you plan to create a grid of this size, you should also bring your friendly local AWS Solutions Architect into the loop as early as possible. They will review your plans, provide you with architecture guidance, and help you to schedule your run.

    Running the Grid
    We hit the Go button and launched the grid, watching as it bid for and obtained Spot Instances, each of which booted, initialized, and joined the grid within two minutes. The test workload used the Strata open source analytics & market risk library from OpenGamma and was set up with their assistance.

    The grid grew to 61,299 Spot Instances (1.3 million vCPUs drawn from 34 instance types spanning 3 generations of EC2 hardware) as planned, with just 1,937 instances reclaimed and automatically replaced during the run, and cost $30,000 per hour to run, at an average hourly cost of $0.078 per vCPU. If the same instances had been used in On-Demand form, the hourly cost to run the grid would have been approximately $93,000.

    Despite the scale of the grid, prices for the EC2 instances did not move during the bidding process. This is due to the overall size of the AWS Cloud and the smooth price change model that we launched late last year.

    To give you a sense of the compute power, we computed that this grid would have taken the #1 position on the TOP 500 supercomputer list in November 2007 by a considerable margin, and the #2 position in June 2008. Today, it would occupy position #360 on the list.

    I hope that you enjoyed this AWS success story, and that it gives you an idea of the scale that you can achieve in the cloud!

    Jeff;

    The US Is Unprepared for Election-Related Hacking in 2018

    Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2018/05/the_us_is_unpre.html

    This survey and report is not surprising:

    The survey of nearly forty Republican and Democratic campaign operatives, administered through November and December 2017, revealed that American political campaign staff — primarily working at the state and congressional levels — are not only unprepared for possible cyber attacks, but remain generally unconcerned about the threat. The survey sample was relatively small, but nevertheless the survey provides a first look at how campaign managers and staff are responding to the threat.

    The overwhelming majority of those surveyed do not want to devote campaign resources to cybersecurity or to hire personnel to address cybersecurity issues. Even though campaign managers recognize there is a high probability that campaign and personal emails are at risk of being hacked, they are more concerned about fundraising and press coverage than they are about cybersecurity. Less than half of those surveyed said they had taken steps to make their data secure and most were unsure if they wanted to spend any money on this protection.

    Security is never something we actually want. Security is something we need in order to avoid what we don’t want. It’s also more abstract, concerned with hypothetical future possibilities. Of course it’s lower on the priorities list than fundraising and press coverage. They’re more tangible, and they’re more immediate.

    This is all to the attackers’ advantage.

    ISPs Win Landmark Case to Protect Privacy of Alleged Pirates

    Post Syndicated from Andy original https://torrentfreak.com/isps-win-landmark-case-protect-privacy-alleged-pirates-180508/

    With waves of piracy settlement letters being sent out across the world, the last line of defense for many accused Internet users has been their ISPs.

    In a number of regions, notably the United States, Europe, and the UK, most ISPs have given up the fight, handing subscriber details over to copyright trolls with a minimum of resistance. However, there are companies out there prepared to stand up for their customers’ rights, if eventually.

    Over in Denmark, Telenor grew tired of tens of thousands of requests for subscriber details filed by a local law firm on behalf of international copyright troll groups. It previously complied with demands to hand over the details of individuals behind 22,000 IP addresses, around 11% of the 200,000 total handled by ISPs in Denmark. But with no end in sight, the ISP dug in its heels.

    “We think there is a fundamental legal problem because the courts do not really decide what is most important: the legal security of the public or the law firms’ commercial interests,” Telenor’s Legal Director Mette Eistrøm Krüger said last year.

    Assisted by rival ISP Telia, Telenor subsequently began preparing a case to protect the interests of their customers, refusing in the meantime to comply with disclosure requests in copyright cases. But last October, the District Court ruled against the telecoms companies, ordering them to provide identities to the copyright trolls.

    Undeterred, the companies took their case to the Østre Landsret, one of Denmark’s two High Courts. Yesterday their determination paid off with a resounding victory for the ISPs and security for the individuals behind approximately 4,000 IP addresses targeted by Copyright Collection Ltd via law firm Njord Law.

    “In its order based on telecommunications legislation, the Court has weighed subscribers’ rights to confidentiality of information regarding their use of the Internet against the interests of rightsholders to obtain information for the purpose of prosecuting claims against the subscribers,” the Court said in a statement.

    Noting that the case raised important questions of European Union law and the European Convention on Human Rights, the High Court said that after due consideration it would overrule the decision of the District Court. The rights of the copyright holders do not trump the individuals right to privacy, it said.

    “The telecommunications companies are therefore not required to disclose the names and addresses of their subscribers,” the Court ruled.

    Telenor welcomed the decision, noting that it had received countless requests from law firms to disclose the identities of thousands of subscribers but had declined to hand them over, a decision that has now been endorsed by the High Court.

    “This is an important victory for our right to protect our customers’ data,” said Telenor Denmark’s Legal Director, Mette Eistrøm Krüger.

    “At Telenor we protect our customers’ data and trust – therefore it has been our conviction that we cannot be forced into almost automatically submitting personal data on our customers simply to support some private actors who are driven by commercial interests.”

    Noting that it’s been putting up a fight since 2016 against handing over customers’ data for purposes other than investigating serious crime, Telenor said that the clarity provided by the decision is most welcome.

    “We and other Danish telecom companies are required to log customer data for the police to fight serious crime and terrorism – but the legislation has just been insufficient in relation to the use of logged data,” Krüger said.

    “Therefore I am pleased that with this judgment the High Court has stated that customers’ legal certainty is most important in these cases.”

    The decision was also welcomed by Telia Denmark, with Legal Director Lasse Andersen describing the company as being “really really happy” with “a big win.”

    “It is a victory for our customers and for all telecom companies’ customers,” Andersen said.

    “They can now feel confident that the data that we collect about them cannot be disclosed for purposes other than the terms under which they are collected as determined by the jurisdiction.

    “Therefore, anyone and everybody cannot claim our data. We are pleased that throughout the process we have determined that we will not hand over our data to anyone other than the police with a court order,” Andersen added.

    But as the ISPs celebrate, the opposite is true for Njord Law and its copyright troll partners.

    “It is a sad message to the Danish film and television industry that the possibilities for self-investigating illegal file sharing are complicated and that the work must be left to the police’s scarce resources,” said Jeppe Brogaard Clausen of Njord Law.

    While the ISPs finally stood up for users in these cases, Telenor in particular wishes to emphasize that supporting the activities of pirates is not its aim. The company says it does not support illegal file-sharing “in any way” and is actively working with anti-piracy outfit Rights Alliance to prevent unauthorized downloading of movies and other content.

    The full decision of the Østre Landsret can be found here (Danish, pdf)

    Source: TF, for the latest info on copyright, file-sharing, torrent sites and more. We also have VPN reviews, discounts, offers and coupons.

    Infamous ‘Kodi Box’ Case Sees Man Pay Back Just £1 to the State

    Post Syndicated from Andy original https://torrentfreak.com/infamous-kodi-box-case-sees-man-pay-back-just-1-to-the-state-180507/

    In 2015, Middlesbrough-based shopkeeper Brian ‘Tomo’ Thompson shot into the headlines after being raided by police and Trading Standards in the UK.

    Thompson had been selling “fully-loaded” piracy-configured Kodi boxes from his shop but didn’t think he’d done anything wrong.

    “All I want to know is whether I am doing anything illegal. I know it’s a gray area but I want it in black and white,” he said.

    Thompson started out with a particularly brave tone. He insisted he’d take the case to Crown Court and even to the European Court. His mission was show what was legal and what wasn’t, he said.

    Very quickly, Thompson’s case took on great importance, with observers everywhere reporting on a potential David versus Goliath copyright battle for the ages. But Thompson’s case wasn’t straightforward.

    The shopkeeper wasn’t charged with basic “making available” under the Copyrights, Designs and Patents Acts that would have found him guilty under the earlier BREIN v Filmspeler case. Instead, he stood accused of two offenses under section 296ZB of the Copyright, Designs and Patents Act, which deals with devices and services designed to “circumvent technological measures”.

    In the end it was all moot. After entering his official ‘not guilty’ plea, last year Thompson suddenly changed his tune. He accepted the prosecution’s version of events, throwing himself at the mercy of the court with a guilty plea.

    In October 2017, Teeside Crown Court heard that Thompson cost Sky around £200,000 in lost subscriptions while the shopkeeper made around £38,500 from selling the devices. But despite the fairly big numbers, Judge Peter Armstrong decided to go reasonably light on the 55-year-old, handing him an 18-month prison term, suspended for two years.

    “I’ve come to the conclusion that in all the circumstances an immediate custodial sentence is not called for. But as a warning to others in future, they may not be so lucky,” the Judge said.

    But things wouldn’t end there for Thompson.

    In the UK, people who make money or obtain assets from criminal activity can be forced to pay back their profits, which are then confiscated by the state under the Proceeds of Crime Act (pdf). Almost anything can be taken, from straight cash to cars, jewellery and houses.

    However, it appears that whatever cash Thompson earned from Kodi Box activities has long since gone.

    During a Proceeds of Crime hearing reported on by Gazette Live, the Court heard that Thompson has no assets whatsoever so any confiscation order would have to be a small one.

    In the end, Judge Simon Hickey decided that Thompson should forfeit a single pound, an amount that could increase if the businessman got lucky moving forward.

    “If anything changes in the future, for instance if you win the lottery, it might come back,” the Judge said.

    With that seeming particularly unlikely, perhaps this will be the end for Thompson. Considering the gravity and importance placed on his case, zero jail time and just a £1 to pay back will probably be acceptable to the 55-year-old and also a lesson to the authorities, who have gotten very little out of this expensive case.

    Who knows, perhaps they might sum up the outcome using the same eight-letter word that Thompson can be seen half-covering in this photograph.

    Source: TF, for the latest info on copyright, file-sharing, torrent sites and more. We also have VPN reviews, discounts, offers and coupons.

    Video Deters People From Pirate Sites…Or Encourages Them to Start One?

    Post Syndicated from Andy original https://torrentfreak.com/video-deters-people-from-pirate-sites-or-encourages-them-to-start-one-180505/

    There are almost as many anti-piracy strategies as there are techniques for downloading.

    Litigation and education are probably the two most likely to be seen by the public, who are often directly targeted by the entertainment industries.

    Over the years this has led to many campaigns, one of which famously stated that piracy is a crime while equating it to the physical theft of a car, a handbag, a television, or a regular movie DVD. It’s debatable whether these campaigns have made much difference but they have raised awareness and some of the responses have been hilarious.

    While success remains hard to measure, it hasn’t stopped these PSAs from being made. The latest efforts come out of Sweden, where the country’s Patent and Registration Office (PRV) was commissioned by the government to increase public awareness of copyright and help change attitudes surrounding streaming and illegal downloading.

    “The purpose is, among other things, to reduce the use of illegal streaming sites and make it easier and safer to find and choose legal options,” PRV says.

    “Every year, criminal networks earn millions of dollars from illegal streaming. This money comes from advertising on illegal sites and is used for other criminal activities. The purpose of our film is to inform about this.”

    The series of videos show pirates in their supposed natural habitats of beautiful mansions, packed with luxurious items such as indoor pools, fancy staircases, and stacks of money. For some reason (perhaps to depict anonymity, perhaps to suggest something more sinister) the pirates are all dressed in animal masks, such as this one enjoying his Dodge Viper.

    The clear suggestion here is that people who visit pirate sites and stream unlicensed content are helping to pay for this guy’s bright green car. The same holds true for his indoor swimming pool, jet bike, and gold chains in the next clip.

    While some might have a problem with pirates getting rich from their clicks, it can’t have escaped the targets of these videos that they too are benefiting from the scheme. Granted, hyena-man gets the pool and the Viper, but they get the latest movies. It seems unlikely that pirate streamers refused to watch the copy of Black Panther that leaked onto the web this week (a month before its retail release) on the basis that someone else was getting rich from it.

    That being said, most people will probably balk at elements of the full PSA, which suggests that revenue from illegal streaming goes on to fuel other crimes, such as prescription drug offenses.

    After reporting piracy cases for more than twelve years, no one at TF has ever seen evidence of this happening with any torrent or streaming site operators. Still, it makes good drama for the full video, embedded below.

    “In the film we follow a fictional occupational criminal who gives us a tour of his beautiful villa. He proudly shows up his multi-criminal activity, which was made possible by means of advertising money from his illegal streaming services,” PRV explains.

    The dark tone and creepy masks are bound to put some people off but one has to question the effect this kind of video could have on younger people. Do pirates really make mountains of money so huge that they can only be counted by machine? If they do, then it’s a lot less risky than almost any other crime that yields this claimed level of profit.

    With that in mind, will this video deter the public or simply encourage people to get involved for some of that big money? We sent a link to the operator of a large pirate site for his considered opinion.

    “WTF,” he responded.

    Source: TF, for the latest info on copyright, file-sharing, torrent sites and more. We also have VPN reviews, discounts, offers and coupons.

    Analyze data in Amazon DynamoDB using Amazon SageMaker for real-time prediction

    Post Syndicated from YongSeong Lee original https://aws.amazon.com/blogs/big-data/analyze-data-in-amazon-dynamodb-using-amazon-sagemaker-for-real-time-prediction/

    Many companies across the globe use Amazon DynamoDB to store and query historical user-interaction data. DynamoDB is a fast NoSQL database used by applications that need consistent, single-digit millisecond latency.

    Often, customers want to turn their valuable data in DynamoDB into insights by analyzing a copy of their table stored in Amazon S3. Doing this separates their analytical queries from their low-latency critical paths. This data can be the primary source for understanding customers’ past behavior, predicting future behavior, and generating downstream business value. Customers often turn to DynamoDB because of its great scalability and high availability. After a successful launch, many customers want to use the data in DynamoDB to predict future behaviors or provide personalized recommendations.

    DynamoDB is a good fit for low-latency reads and writes, but it’s not practical to scan all data in a DynamoDB database to train a model. In this post, I demonstrate how you can use DynamoDB table data copied to Amazon S3 by AWS Data Pipeline to predict customer behavior. I also demonstrate how you can use this data to provide personalized recommendations for customers using Amazon SageMaker. You can also run ad hoc queries using Amazon Athena against the data. DynamoDB recently released on-demand backups to create full table backups with no performance impact. However, it’s not suitable for our purposes in this post, so I chose AWS Data Pipeline instead to create managed backups are accessible from other services.

    To do this, I describe how to read the DynamoDB backup file format in Data Pipeline. I also describe how to convert the objects in S3 to a CSV format that Amazon SageMaker can read. In addition, I show how to schedule regular exports and transformations using Data Pipeline. The sample data used in this post is from Bank Marketing Data Set of UCI.

    The solution that I describe provides the following benefits:

    • Separates analytical queries from production traffic on your DynamoDB table, preserving your DynamoDB read capacity units (RCUs) for important production requests
    • Automatically updates your model to get real-time predictions
    • Optimizes for performance (so it doesn’t compete with DynamoDB RCUs after the export) and for cost (using data you already have)
    • Makes it easier for developers of all skill levels to use Amazon SageMaker

    All code and data set in this post are available in this .zip file.

    Solution architecture

    The following diagram shows the overall architecture of the solution.

    The steps that data follows through the architecture are as follows:

    1. Data Pipeline regularly copies the full contents of a DynamoDB table as JSON into an S3
    2. Exported JSON files are converted to comma-separated value (CSV) format to use as a data source for Amazon SageMaker.
    3. Amazon SageMaker renews the model artifact and update the endpoint.
    4. The converted CSV is available for ad hoc queries with Amazon Athena.
    5. Data Pipeline controls this flow and repeats the cycle based on the schedule defined by customer requirements.

    Building the auto-updating model

    This section discusses details about how to read the DynamoDB exported data in Data Pipeline and build automated workflows for real-time prediction with a regularly updated model.

    Download sample scripts and data

    Before you begin, take the following steps:

    1. Download sample scripts in this .zip file.
    2. Unzip the src.zip file.
    3. Find the automation_script.sh file and edit it for your environment. For example, you need to replace 's3://<your bucket>/<datasource path>/' with your own S3 path to the data source for Amazon ML. In the script, the text enclosed by angle brackets—< and >—should be replaced with your own path.
    4. Upload the json-serde-1.3.6-SNAPSHOT-jar-with-dependencies.jar file to your S3 path so that the ADD jar command in Apache Hive can refer to it.

    For this solution, the banking.csv  should be imported into a DynamoDB table.

    Export a DynamoDB table

    To export the DynamoDB table to S3, open the Data Pipeline console and choose the Export DynamoDB table to S3 template. In this template, Data Pipeline creates an Amazon EMR cluster and performs an export in the EMRActivity activity. Set proper intervals for backups according to your business requirements.

    One core node(m3.xlarge) provides the default capacity for the EMR cluster and should be suitable for the solution in this post. Leave the option to resize the cluster before running enabled in the TableBackupActivity activity to let Data Pipeline scale the cluster to match the table size. The process of converting to CSV format and renewing models happens in this EMR cluster.

    For a more in-depth look at how to export data from DynamoDB, see Export Data from DynamoDB in the Data Pipeline documentation.

    Add the script to an existing pipeline

    After you export your DynamoDB table, you add an additional EMR step to EMRActivity by following these steps:

    1. Open the Data Pipeline console and choose the ID for the pipeline that you want to add the script to.
    2. For Actions, choose Edit.
    3. In the editing console, choose the Activities category and add an EMR step using the custom script downloaded in the previous section, as shown below.

    Paste the following command into the new step after the data ­­upload step:

    s3://#{myDDBRegion}.elasticmapreduce/libs/script-runner/script-runner.jar,s3://<your bucket name>/automation_script.sh,#{output.directoryPath},#{myDDBRegion}

    The element #{output.directoryPath} references the S3 path where the data pipeline exports DynamoDB data as JSON. The path should be passed to the script as an argument.

    The bash script has two goals, converting data formats and renewing the Amazon SageMaker model. Subsequent sections discuss the contents of the automation script.

    Automation script: Convert JSON data to CSV with Hive

    We use Apache Hive to transform the data into a new format. The Hive QL script to create an external table and transform the data is included in the custom script that you added to the Data Pipeline definition.

    When you run the Hive scripts, do so with the -e option. Also, define the Hive table with the 'org.openx.data.jsonserde.JsonSerDe' row format to parse and read JSON format. The SQL creates a Hive EXTERNAL table, and it reads the DynamoDB backup data on the S3 path passed to it by Data Pipeline.

    Note: You should create the table with the “EXTERNAL” keyword to avoid the backup data being accidentally deleted from S3 if you drop the table.

    The full automation script for converting follows. Add your own bucket name and data source path in the highlighted areas.

    #!/bin/bash
    hive -e "
    ADD jar s3://<your bucket name>/json-serde-1.3.6-SNAPSHOT-jar-with-dependencies.jar ; 
    DROP TABLE IF EXISTS blog_backup_data ;
    CREATE EXTERNAL TABLE blog_backup_data (
     customer_id map<string,string>,
     age map<string,string>, job map<string,string>, 
     marital map<string,string>,education map<string,string>, 
     default map<string,string>, housing map<string,string>,
     loan map<string,string>, contact map<string,string>, 
     month map<string,string>, day_of_week map<string,string>, 
     duration map<string,string>, campaign map<string,string>,
     pdays map<string,string>, previous map<string,string>, 
     poutcome map<string,string>, emp_var_rate map<string,string>, 
     cons_price_idx map<string,string>, cons_conf_idx map<string,string>,
     euribor3m map<string,string>, nr_employed map<string,string>, 
     y map<string,string> ) 
    ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe' 
    LOCATION '$1/';
    
    INSERT OVERWRITE DIRECTORY 's3://<your bucket name>/<datasource path>/' 
    SELECT concat( customer_id['s'],',', 
     age['n'],',', job['s'],',', 
     marital['s'],',', education['s'],',', default['s'],',', 
     housing['s'],',', loan['s'],',', contact['s'],',', 
     month['s'],',', day_of_week['s'],',', duration['n'],',', 
     campaign['n'],',',pdays['n'],',',previous['n'],',', 
     poutcome['s'],',', emp_var_rate['n'],',', cons_price_idx['n'],',',
     cons_conf_idx['n'],',', euribor3m['n'],',', nr_employed['n'],',', y['n'] ) 
    FROM blog_backup_data
    WHERE customer_id['s'] > 0 ; 
    

    After creating an external table, you need to read data. You then use the INSERT OVERWRITE DIRECTORY ~ SELECT command to write CSV data to the S3 path that you designated as the data source for Amazon SageMaker.

    Depending on your requirements, you can eliminate or process the columns in the SELECT clause in this step to optimize data analysis. For example, you might remove some columns that have unpredictable correlations with the target value because keeping the wrong columns might expose your model to “overfitting” during the training. In this post, customer_id  columns is removed. Overfitting can make your prediction weak. More information about overfitting can be found in the topic Model Fit: Underfitting vs. Overfitting in the Amazon ML documentation.

    Automation script: Renew the Amazon SageMaker model

    After the CSV data is replaced and ready to use, create a new model artifact for Amazon SageMaker with the updated dataset on S3.  For renewing model artifact, you must create a new training job.  Training jobs can be run using the AWS SDK ( for example, Amazon SageMaker boto3 ) or the Amazon SageMaker Python SDK that can be installed with “pip install sagemaker” command as well as the AWS CLI for Amazon SageMaker described in this post.

    In addition, consider how to smoothly renew your existing model without service impact, because your model is called by applications in real time. To do this, you need to create a new endpoint configuration first and update a current endpoint with the endpoint configuration that is just created.

    #!/bin/bash
    ## Define variable 
    REGION=$2
    DTTIME=`date +%Y-%m-%d-%H-%M-%S`
    ROLE="<your AmazonSageMaker-ExecutionRole>" 
    
    
    # Select containers image based on region.  
    case "$REGION" in
    "us-west-2" )
        IMAGE="174872318107.dkr.ecr.us-west-2.amazonaws.com/linear-learner:latest"
        ;;
    "us-east-1" )
        IMAGE="382416733822.dkr.ecr.us-east-1.amazonaws.com/linear-learner:latest" 
        ;;
    "us-east-2" )
        IMAGE="404615174143.dkr.ecr.us-east-2.amazonaws.com/linear-learner:latest" 
        ;;
    "eu-west-1" )
        IMAGE="438346466558.dkr.ecr.eu-west-1.amazonaws.com/linear-learner:latest" 
        ;;
     *)
        echo "Invalid Region Name"
        exit 1 ;  
    esac
    
    # Start training job and creating model artifact 
    TRAINING_JOB_NAME=TRAIN-${DTTIME} 
    S3OUTPUT="s3://<your bucket name>/model/" 
    INSTANCETYPE="ml.m4.xlarge"
    INSTANCECOUNT=1
    VOLUMESIZE=5 
    aws sagemaker create-training-job --training-job-name ${TRAINING_JOB_NAME} --region ${REGION}  --algorithm-specification TrainingImage=${IMAGE},TrainingInputMode=File --role-arn ${ROLE}  --input-data-config '[{ "ChannelName": "train", "DataSource": { "S3DataSource": { "S3DataType": "S3Prefix", "S3Uri": "s3://<your bucket name>/<datasource path>/", "S3DataDistributionType": "FullyReplicated" } }, "ContentType": "text/csv", "CompressionType": "None" , "RecordWrapperType": "None"  }]'  --output-data-config S3OutputPath=${S3OUTPUT} --resource-config  InstanceType=${INSTANCETYPE},InstanceCount=${INSTANCECOUNT},VolumeSizeInGB=${VOLUMESIZE} --stopping-condition MaxRuntimeInSeconds=120 --hyper-parameters feature_dim=20,predictor_type=binary_classifier  
    
    # Wait until job completed 
    aws sagemaker wait training-job-completed-or-stopped --training-job-name ${TRAINING_JOB_NAME}  --region ${REGION}
    
    # Get newly created model artifact and create model
    MODELARTIFACT=`aws sagemaker describe-training-job --training-job-name ${TRAINING_JOB_NAME} --region ${REGION}  --query 'ModelArtifacts.S3ModelArtifacts' --output text `
    MODELNAME=MODEL-${DTTIME}
    aws sagemaker create-model --region ${REGION} --model-name ${MODELNAME}  --primary-container Image=${IMAGE},ModelDataUrl=${MODELARTIFACT}  --execution-role-arn ${ROLE}
    
    # create a new endpoint configuration 
    CONFIGNAME=CONFIG-${DTTIME}
    aws sagemaker  create-endpoint-config --region ${REGION} --endpoint-config-name ${CONFIGNAME}  --production-variants  VariantName=Users,ModelName=${MODELNAME},InitialInstanceCount=1,InstanceType=ml.m4.xlarge
    
    # create or update the endpoint
    STATUS=`aws sagemaker describe-endpoint --endpoint-name  ServiceEndpoint --query 'EndpointStatus' --output text --region ${REGION} `
    if [[ $STATUS -ne "InService" ]] ;
    then
        aws sagemaker  create-endpoint --endpoint-name  ServiceEndpoint  --endpoint-config-name ${CONFIGNAME} --region ${REGION}    
    else
        aws sagemaker  update-endpoint --endpoint-name  ServiceEndpoint  --endpoint-config-name ${CONFIGNAME} --region ${REGION}
    fi

    Grant permission

    Before you execute the script, you must grant proper permission to Data Pipeline. Data Pipeline uses the DataPipelineDefaultResourceRole role by default. I added the following policy to DataPipelineDefaultResourceRole to allow Data Pipeline to create, delete, and update the Amazon SageMaker model and data source in the script.

    {
     "Version": "2012-10-17",
     "Statement": [
     {
     "Effect": "Allow",
     "Action": [
     "sagemaker:CreateTrainingJob",
     "sagemaker:DescribeTrainingJob",
     "sagemaker:CreateModel",
     "sagemaker:CreateEndpointConfig",
     "sagemaker:DescribeEndpoint",
     "sagemaker:CreateEndpoint",
     "sagemaker:UpdateEndpoint",
     "iam:PassRole"
     ],
     "Resource": "*"
     }
     ]
    }

    Use real-time prediction

    After you deploy a model into production using Amazon SageMaker hosting services, your client applications use this API to get inferences from the model hosted at the specified endpoint. This approach is useful for interactive web, mobile, or desktop applications.

    Following, I provide a simple Python code example that queries against Amazon SageMaker endpoint URL with its name (“ServiceEndpoint”) and then uses them for real-time prediction.

    === Python sample for real-time prediction ===

    #!/usr/bin/env python
    import boto3
    import json 
    
    client = boto3.client('sagemaker-runtime', region_name ='<your region>' )
    new_customer_info = '34,10,2,4,1,2,1,1,6,3,190,1,3,4,3,-1.7,94.055,-39.8,0.715,4991.6'
    response = client.invoke_endpoint(
        EndpointName='ServiceEndpoint',
        Body=new_customer_info, 
        ContentType='text/csv'
    )
    result = json.loads(response['Body'].read().decode())
    print(result)
    --- output(response) ---
    {u'predictions': [{u'score': 0.7528127431869507, u'predicted_label': 1.0}]}

    Solution summary

    The solution takes the following steps:

    1. Data Pipeline exports DynamoDB table data into S3. The original JSON data should be kept to recover the table in the rare event that this is needed. Data Pipeline then converts JSON to CSV so that Amazon SageMaker can read the data.Note: You should select only meaningful attributes when you convert CSV. For example, if you judge that the “campaign” attribute is not correlated, you can eliminate this attribute from the CSV.
    2. Train the Amazon SageMaker model with the new data source.
    3. When a new customer comes to your site, you can judge how likely it is for this customer to subscribe to your new product based on “predictedScores” provided by Amazon SageMaker.
    4. If the new user subscribes your new product, your application must update the attribute “y” to the value 1 (for yes). This updated data is provided for the next model renewal as a new data source. It serves to improve the accuracy of your prediction. With each new entry, your application can become smarter and deliver better predictions.

    Running ad hoc queries using Amazon Athena

    Amazon Athena is a serverless query service that makes it easy to analyze large amounts of data stored in Amazon S3 using standard SQL. Athena is useful for examining data and collecting statistics or informative summaries about data. You can also use the powerful analytic functions of Presto, as described in the topic Aggregate Functions of Presto in the Presto documentation.

    With the Data Pipeline scheduled activity, recent CSV data is always located in S3 so that you can run ad hoc queries against the data using Amazon Athena. I show this with example SQL statements following. For an in-depth description of this process, see the post Interactive SQL Queries for Data in Amazon S3 on the AWS News Blog. 

    Creating an Amazon Athena table and running it

    Simply, you can create an EXTERNAL table for the CSV data on S3 in Amazon Athena Management Console.

    === Table Creation ===
    CREATE EXTERNAL TABLE datasource (
     age int, 
     job string, 
     marital string , 
     education string, 
     default string, 
     housing string, 
     loan string, 
     contact string, 
     month string, 
     day_of_week string, 
     duration int, 
     campaign int, 
     pdays int , 
     previous int , 
     poutcome string, 
     emp_var_rate double, 
     cons_price_idx double,
     cons_conf_idx double, 
     euribor3m double, 
     nr_employed double, 
     y int 
    )
    ROW FORMAT DELIMITED 
    FIELDS TERMINATED BY ',' ESCAPED BY '\\' LINES TERMINATED BY '\n' 
    LOCATION 's3://<your bucket name>/<datasource path>/';

    The following query calculates the correlation coefficient between the target attribute and other attributes using Amazon Athena.

    === Sample Query ===

    SELECT corr(age,y) AS correlation_age_and_target, 
     corr(duration,y) AS correlation_duration_and_target, 
     corr(campaign,y) AS correlation_campaign_and_target,
     corr(contact,y) AS correlation_contact_and_target
    FROM ( SELECT age , duration , campaign , y , 
     CASE WHEN contact = 'telephone' THEN 1 ELSE 0 END AS contact 
     FROM datasource 
     ) datasource ;

    Conclusion

    In this post, I introduce an example of how to analyze data in DynamoDB by using table data in Amazon S3 to optimize DynamoDB table read capacity. You can then use the analyzed data as a new data source to train an Amazon SageMaker model for accurate real-time prediction. In addition, you can run ad hoc queries against the data on S3 using Amazon Athena. I also present how to automate these procedures by using Data Pipeline.

    You can adapt this example to your specific use case at hand, and hopefully this post helps you accelerate your development. You can find more examples and use cases for Amazon SageMaker in the video AWS 2017: Introducing Amazon SageMaker on the AWS website.

     


    Additional Reading

    If you found this post useful, be sure to check out Serving Real-Time Machine Learning Predictions on Amazon EMR and Analyzing Data in S3 using Amazon Athena.

     


    About the Author

    Yong Seong Lee is a Cloud Support Engineer for AWS Big Data Services. He is interested in every technology related to data/databases and helping customers who have difficulties in using AWS services. His motto is “Enjoy life, be curious and have maximum experience.”

     

     

    Russia Blocks 50 VPNs & Anonymizers in Telegram Crackdown, Viber Next

    Post Syndicated from Andy original https://torrentfreak.com/russia-blocks-50-vpns-anonymizers-in-telegram-crackdown-viber-next-180504/

    Any entity operating an encrypted messaging service in Russia needs to register with local authorities. They must also hand over their encryption keys when requested to do so, so that users can be monitored.

    Messaging giant Telegram refused to give in to Russian pressure. Founder Pavel Durov said that he would not compromise the privacy of Telegram’s 200m monthly users, despite losing a lawsuit against the Federal Security Service which compelled him to do so. In response, telecoms watchdog Roscomnadzor filed a lawsuit to degrade Telegram via web-blocking.

    After a Moscow court gave the go-ahead for Telegram to be banned in Russia last month, chaos broke out. ISPs around the country tried to block the service, which was using Amazon and Google to provide connectivity. Millions of IP addresses belonging to both companies were blocked and countless other companies and individuals had their services blocked too.

    But despite the Russian carpet-bombing of Telegram, the service steadfastly remained online. People had problems accessing the service at times, of course, but their determination coupled with that of Telegram and other facilitators largely kept communications flowing.

    Part of the huge counter-offensive was mounted by various VPN and anonymizer services that allowed people to bypass ISP blocks. However, they too have found themselves in trouble, with Russian authorities blocking them for facilitating access to Telegram. In an announcement Thursday, the telecoms watchdog revealed the scale of the crackdown.

    Deputy Head of Roskomnadzor told TASS that dozens of VPNs and similar services had been blocked while hinting at yet more to come.

    “Fifty for the time being,” Subbotin said.

    With VPN providers taking a hit on behalf of Telegram, there could be yet more chaos looming on the horizon. It’s feared that other encrypted services, which have also failed to hand over their keys to the FSB, could be targeted next.

    Ministry of Communications chief Nikolai Nikiforov told reporters this week that if Viber doesn’t fall into line, it could suffer the same fate as Telegram.

    “This is a matter for the Federal Security Service, because the authority with regard to such specific issues in the execution of the order for the provision of encryption keys is the authority of the FSB,” Nikiforov said.

    “If they have problems with the provision of encryption keys, they can also apply to the court and obtain a similar court decision,” the minister said, responding to questions about the Japanese-owned, Luxembourg-based communications app.

    With plenty of chaos apparent online, there are also reports of problems from within Roscomnadzor itself. For the past several days, rumors have been circulating in Russian media that Roskomnadzor chief Alexander Zharov has resigned, perhaps in response to the huge over-blocking that took place when Telegram was targeted.

    When questioned by reporters this week, Ministry of Communications chief Nikolai Nikiforov refused to provide any further information, stating that such a matter would be for the prime minister to handle.

    “I would not like to comment on this. If the chairman of the government takes this decision, I recall that the heads of services are appointed by the decision of the prime minister and personnel decisions are never commented on,” he said.

    Whether Prime Minister Dmitry Medvedev will make a statement is yet to be seen, but this week his office has been dealing with a blocking – or rather unblocking – controversy of its own.

    In a public post on Facebook May 1, Duma deputy Natalya Kostenko revealed that she was having problems due to the Telegram blockades.

    “Dear friends, do not write to me on Telegram, I’m not getting your messages. Use other channels to contact me,” Kostenko wrote.

    In response, Dmitry Medvedev’s press secretary, Natalia Timakova, told her colleague to circumvent the blockade so that she could access Telegram once again.

    “Use a VPN! It’s simple. And it works almost all the time,” Timakov wrote.

    Until those get blocked too, of course…..

    Source: TF, for the latest info on copyright, file-sharing, torrent sites and more. We also have VPN reviews, discounts, offers and coupons.

    The Helium Factor and Hard Drive Failure Rates

    Post Syndicated from Andy Klein original https://www.backblaze.com/blog/helium-filled-hard-drive-failure-rates/

    Seagate Enterprise Capacity 3.5 Helium HDD

    In November 2013, the first commercially available helium-filled hard drive was introduced by HGST, a Western Digital subsidiary. The 6 TB drive was not only unique in being helium-filled, it was for the moment, the highest capacity hard drive available. Fast forward a little over 4 years later and 12 TB helium-filled drives are readily available, 14 TB drives can be found, and 16 TB helium-filled drives are arriving soon.

    Backblaze has been purchasing and deploying helium-filled hard drives over the past year and we thought it was time to start looking at their failure rates compared to traditional air-filled drives. This post will provide an overview, then we’ll continue the comparison on a regular basis over the coming months.

    The Promise and Challenge of Helium Filled Drives

    We all know that helium is lighter than air — that’s why helium-filled balloons float. Inside of an air-filled hard drive there are rapidly spinning disk platters that rotate at a given speed, 7200 rpm for example. The air inside adds an appreciable amount of drag on the platters that in turn requires an appreciable amount of additional energy to spin the platters. Replacing the air inside of a hard drive with helium reduces the amount of drag, thereby reducing the amount of energy needed to spin the platters, typically by 20%.

    We also know that after a few days, a helium-filled balloon sinks to the ground. This was one of the key challenges in using helium inside of a hard drive: helium escapes from most containers, even if they are well sealed. It took years for hard drive manufacturers to create containers that could contain helium while still functioning as a hard drive. This container innovation allows helium-filled drives to function at spec over the course of their lifetime.

    Checking for Leaks

    Three years ago, we identified SMART 22 as the attribute assigned to recording the status of helium inside of a hard drive. We have both HGST and Seagate helium-filled hard drives, but only the HGST drives currently report the SMART 22 attribute. It appears the normalized and raw values for SMART 22 currently report the same value, which starts at 100 and goes down.

    To date only one HGST drive has reported a value of less than 100, with multiple readings between 94 and 99. That drive continues to perform fine, with no other errors or any correlating changes in temperature, so we are not sure whether the change in value is trying to tell us something or if it is just a wonky sensor.

    Helium versus Air-Filled Hard Drives

    There are several different ways to compare these two types of drives. Below we decided to use just our 8, 10, and 12 TB drives in the comparison. We did this since we have helium-filled drives in those sizes. We left out of the comparison all of the drives that are 6 TB and smaller as none of the drive models we use are helium-filled. We are open to trying different comparisons. This just seemed to be the best place to start.

    Lifetime Hard Drive Failure Rates: Helium vs. Air-Filled Hard Drives table

    The most obvious observation is that there seems to be little difference in the Annualized Failure Rate (AFR) based on whether they contain helium or air. One conclusion, given this evidence, is that helium doesn’t affect the AFR of hard drives versus air-filled drives. My prediction is that the helium drives will eventually prove to have a lower AFR. Why? Drive Days.

    Let’s go back in time to Q1 2017 when the air-filled drives listed in the table above had a similar number of Drive Days to the current number of Drive Days for the helium drives. We find that the failure rate for the air-filled drives at the time (Q1 2017) was 1.61%. In other words, when the drives were in use a similar number of hours, the helium drives had a failure rate of 1.06% while the failure rate of the air-filled drives was 1.61%.

    Helium or Air?

    My hypothesis is that after normalizing the data so that the helium and air-filled drives have the same (or similar) usage (Drive Days), the helium-filled drives we use will continue to have a lower Annualized Failure Rate versus the air-filled drives we use. I expect this trend to continue for the next year at least. What side do you come down on? Will the Annualized Failure Rate for helium-filled drives be better than air-filled drives or vice-versa? Or do you think the two technologies will be eventually produce the same AFR over time? Pick a side and we’ll document the results over the next year and see where the data takes us.

    The post The Helium Factor and Hard Drive Failure Rates appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

    Sci-Hub ‘Pirate Bay For Science’ Security Certs Revoked by Comodo

    Post Syndicated from Andy original https://torrentfreak.com/sci-hub-pirate-bay-for-science-security-certs-revoked-by-comodo-ca-180503/

    Sci-Hub is often referred to as the “Pirate Bay of Science”. Like its namesake, it offers masses of unlicensed content for free, mostly against the wishes of copyright holders.

    While The Pirate Bay will index almost anything, Sci-Hub is dedicated to distributing tens of millions of academic papers and articles, something which has turned itself into a target for publishing giants like Elsevier.

    Sci-Hub and its Kazakhstan-born founder Alexandra Elbakyan have been under sustained attack for several years but more recently have been fending off an unprecedented barrage of legal action initiated by the American Chemical Society (ACS), a leading source of academic publications in the field of chemistry.

    After winning a default judgment for $4.8 million in copyright infringement damages last year, ACS was further granted a broad injunction.

    It required various third-party services (including domain registries, hosting companies and search engines) to stop facilitating access to the site. This plunged Sci-Hub into a game of domain whac-a-mole, one that continues to this day.

    Determined to head Sci-Hub off at the pass, ACS obtained additional authority to tackle the evasive site and any new domains it may register in the future.

    While Sci-Hub has been hopping around domains for a while, this week a new development appeared on the horizon. Visitors to some of the site’s domains were greeted with errors indicating that the domains’ security certificates had been revoked.

    Tests conducted by TorrentFreak revealed clear revocations on Sci-Hub.hk and Sci-Hub.nz, both of which returned the error ‘NET::ERR_CERT_REVOKED’.

    Certificate revoked

    These certificates were first issued and then revoked by Comodo CA, the world’s largest certification authority. TF contacted the company who confirmed that it had been forced to take action against Sci-Hub.

    “In response to a court order against Sci-Hub, Comodo CA has revoked four certificates for the site,” Jonathan Skinner, Director, Global Channel Programs at Comodo CA informed TorrentFreak.

    “By policy Comodo CA obeys court orders and the law to the full extent of its ability.”

    Comodo refused to confirm any additional details, including whether these revocations were anything to do with the current ACS injunction. However, Susan R. Morrissey, Director of Communications at ACS, told TorrentFreak that the revocations were indeed part of ACS’ legal action against Sci-Hub.

    “[T]he action is related to our continuing efforts to protect ACS’ intellectual property,” Morrissey confirmed.

    Sci-Hub operates multiple domains (an up-to-date list is usually available on Wikipedia) that can be switched at any time. At the time of writing the domain sci-hub.ga currently returns ‘ERR_SSL_VERSION_OR_CIPHER_MISMATCH’ while .CN and .GS variants both have Comodo certificates that expired last year.

    When TF first approached Comodo earlier this week, Sci-Hub’s certificates with the company hadn’t been completely wiped out. For example, the domain https://sci-hub.tw operated perfectly, with an active and non-revoked Comodo certificate.

    Still in the game…but not for long

    By Wednesday, however, the domain was returning the now-familiar “revoked” message.

    These domain issues are the latest technical problems to hit Sci-Hub as a result of the ACS injunction. In February, Cloudflare terminated service to several of the site’s domains.

    “Cloudflare will terminate your service for the following domains sci-hub.la, sci-hub.tv, and sci-hub.tw by disabling our authoritative DNS in 24 hours,” Cloudflare told Sci-Hub.

    While ACS has certainly caused problems for Sci-Hub, the platform is extremely resilient and remains online.

    The domains https://sci-hub.is and https://sci-hub.nu are fully operational with certificates issued by Let’s Encrypt, a free and open certificate authority supported by the likes of Mozilla, EFF, Chrome, Private Internet Access, and other prominent tech companies.

    It’s unclear whether these certificates will be targeted in the future but Sci-Hub doesn’t appear to be in the mood to back down.

    Source: TF, for the latest info on copyright, file-sharing, torrent sites and more. We also have VPN reviews, discounts, offers and coupons.

    CI/CD with Data: Enabling Data Portability in a Software Delivery Pipeline with AWS Developer Tools, Kubernetes, and Portworx

    Post Syndicated from Kausalya Rani Krishna Samy original https://aws.amazon.com/blogs/devops/cicd-with-data-enabling-data-portability-in-a-software-delivery-pipeline-with-aws-developer-tools-kubernetes-and-portworx/

    This post is written by Eric Han – Vice President of Product Management Portworx and Asif Khan – Solutions Architect

    Data is the soul of an application. As containers make it easier to package and deploy applications faster, testing plays an even more important role in the reliable delivery of software. Given that all applications have data, development teams want a way to reliably control, move, and test using real application data or, at times, obfuscated data.

    For many teams, moving application data through a CI/CD pipeline, while honoring compliance and maintaining separation of concerns, has been a manual task that doesn’t scale. At best, it is limited to a few applications, and is not portable across environments. The goal should be to make running and testing stateful containers (think databases and message buses where operations are tracked) as easy as with stateless (such as with web front ends where they are often not).

    Why is state important in testing scenarios? One reason is that many bugs manifest only when code is tested against real data. For example, we might simply want to test a database schema upgrade but a small synthetic dataset does not exercise the critical, finer corner cases in complex business logic. If we want true end-to-end testing, we need to be able to easily manage our data or state.

    In this blog post, we define a CI/CD pipeline reference architecture that can automate data movement between applications. We also provide the steps to follow to configure the CI/CD pipeline.

     

    Stateful Pipelines: Need for Portable Volumes

    As part of continuous integration, testing, and deployment, a team may need to reproduce a bug found in production against a staging setup. Here, the hosting environment is comprised of a cluster with Kubernetes as the scheduler and Portworx for persistent volumes. The testing workflow is then automated by AWS CodeCommit, AWS CodePipeline, and AWS CodeBuild.

    Portworx offers Kubernetes storage that can be used to make persistent volumes portable between AWS environments and pipelines. The addition of Portworx to the AWS Developer Tools continuous deployment for Kubernetes reference architecture adds persistent storage and storage orchestration to a Kubernetes cluster. The example uses MongoDB as the demonstration of a stateful application. In practice, the workflow applies to any containerized application such as Cassandra, MySQL, Kafka, and Elasticsearch.

    Using the reference architecture, a developer calls CodePipeline to trigger a snapshot of the running production MongoDB database. Portworx then creates a block-based, writable snapshot of the MongoDB volume. Meanwhile, the production MongoDB database continues serving end users and is uninterrupted.

    Without the Portworx integrations, a manual process would require an application-level backup of the database instance that is outside of the CI/CD process. For larger databases, this could take hours and impact production. The use of block-based snapshots follows best practices for resilient and non-disruptive backups.

    As part of the workflow, CodePipeline deploys a new MongoDB instance for staging onto the Kubernetes cluster and mounts the second Portworx volume that has the data from production. CodePipeline triggers the snapshot of a Portworx volume through an AWS Lambda function, as shown here

     

     

     

    AWS Developer Tools with Kubernetes: Integrated Workflow with Portworx

    In the following workflow, a developer is testing changes to a containerized application that calls on MongoDB. The tests are performed against a staging instance of MongoDB. The same workflow applies if changes were on the server side. The original production deployment is scheduled as a Kubernetes deployment object and uses Portworx as the storage for the persistent volume.

    The continuous deployment pipeline runs as follows:

    • Developers integrate bug fix changes into a main development branch that gets merged into a CodeCommit master branch.
    • Amazon CloudWatch triggers the pipeline when code is merged into a master branch of an AWS CodeCommit repository.
    • AWS CodePipeline sends the new revision to AWS CodeBuild, which builds a Docker container image with the build ID.
    • AWS CodeBuild pushes the new Docker container image tagged with the build ID to an Amazon ECR registry.
    • Kubernetes downloads the new container (for the database client) from Amazon ECR and deploys the application (as a pod) and staging MongoDB instance (as a deployment object).
    • AWS CodePipeline, through a Lambda function, calls Portworx to snapshot the production MongoDB and deploy a staging instance of MongoDB• Portworx provides a snapshot of the production instance as the persistent storage of the staging MongoDB
      • The MongoDB instance mounts the snapshot.

    At this point, the staging setup mimics a production environment. Teams can run integration and full end-to-end tests, using partner tooling, without impacting production workloads. The full pipeline is shown here.

     

    Summary

    This reference architecture showcases how development teams can easily move data between production and staging for the purposes of testing. Instead of taking application-specific manual steps, all operations in this CodePipeline architecture are automated and tracked as part of the CI/CD process.

    This integrated experience is part of making stateful containers as easy as stateless. With AWS CodePipeline for CI/CD process, developers can easily deploy stateful containers onto a Kubernetes cluster with Portworx storage and automate data movement within their process.

    The reference architecture and code are available on GitHub:

    ● Reference architecture: https://github.com/portworx/aws-kube-codesuite
    ● Lambda function source code for Portworx additions: https://github.com/portworx/aws-kube-codesuite/blob/master/src/kube-lambda.py

    For more information about persistent storage for containers, visit the Portworx website. For more information about Code Pipeline, see the AWS CodePipeline User Guide.