Tag Archives: perl

Ukraine Faces Call for US Trade Sanctions over Online Piracy

Post Syndicated from Ernesto original https://torrentfreak.com/ukraine-faces-call-us-trade-sanctions-over-online-piracy-170918/

The International Intellectual Property Alliance (IIPA) is recommending that the U.S. Government should suspend Ukraine’s GSP trade benefits, claiming that the country doesn’t do enough to protect the interests of copyright holders.

Last year Ukraine enjoyed $53.7 million in unilateral duty-free benefits in the US, while US companies suffering millions of dollars in losses in Ukraine due to online piracy, they argue.

The IIPA, which includes a wide range of copyright groups including the MPAA, RIAA, BSA and ESA, characterizes the country as a safe harbor for pirate sites. While physical piracy was properly addressed ten years ago after a previous sanction, digital piracy remains rampant.

One of the main problems is that local hosting companies are offering their services to a wide variety of copyright-infringing websites. Without proper enforcement, more and more websites have moved their services there.

“By allowing these problems to fester for years, weak digital enforcement has resulted in an exponential increase in the number of illegal peer-to-peer (‘P2P’) hosting and website-based Internet piracy sites, including some of the world’s largest BitTorrent sites located in Ukraine,” IIPA writes.

“Some Internet pirates have purposefully moved their servers and operations to Ukraine in the past few years to take advantage of the current lawless situation. Many of these illegal services and sites target audiences throughout Europe and the United States.”

The copyright holders highlight the defunct ExtraTorrent site as an example but note that there are also many other torrent sites, pirate streaming sites, cyberlockers, and linking sites in Ukraine.

While pirate sites are hosted all over the world, the problem is particularly persistent in Ukraine because many local hosting companies fail to process takedown requests. This, despite repeated calls from copyright holders to work with them.

“Many of the websites offering pirated copyright materials are thriving in part because of the support of local ISPs,” IIPA writes.

“The copyright industries have, for years, sought private agreements with ISPs to establish effective mechanisms to take down illegal websites and slow illegal P2P traffic. In the absence of legislation, however, these voluntary efforts have generally not succeeded, although, some ISPs will delete links upon request.”

In order to make real progress, the copyright holders call for new legislation to hold Internet services accountable and to make it easier to come after pirate sites that are hosted in Ukraine.

“Legislation is needed to institute proper notice and takedown provisions, including a requirement that service providers terminate access to individuals (or entities) that have repeatedly engaged in infringement, and the retention of information for law enforcement, as well as to provide clear third party liability regarding ISPs.”

In addition to addressing online piracy, IIPA further points out that the collecting societies in Ukraine are not functioning properly. At the moment there are 18 active and competing organizations, creating a chaotic situation where rightsholders are not properly rewarded, they suggest.

IIPA recommends that the U.S. Government accepts its petition and suspends or withdraws Ukraine’s benefits until the country takes proper action.

Ukraine’s Government, for its part, informs the US Government that progress is being made. There are already several new laws in the works to improve intellectual property protection. The issue is one of the Government’s “key priorities,” they state, hoping to avert any sanctions.

IIPA’s full submission to the US Trade Representative is available here (pdf).

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

Security updates for Monday

Post Syndicated from ris original https://lwn.net/Articles/734015/rss

Security updates have been issued by Arch Linux (ffmpeg, lib32-libgcrypt, libgcrypt, linux-zen, and newsbeuter), Debian (emacs25, freexl, and tomcat8), Fedora (cyrus-imapd, FlightGear, freexl, gdm, kernel, LibRaw, ruby, and xen), Gentoo (binutils, chkrootkit, curl, gdk-pixbuf, gimps, git, kpathsea, mod_gnutls, perl, squirrelmail, subversion, supervisor, and webkit-gtk), Mageia (389-ds-base, kernel, kernel-linus, kernel-tmb, and mpg123), openSUSE (ffmpeg, ffmpeg2, qemu, and xen), Slackware (kernel), SUSE (xen), and Ubuntu (gdk-pixbuf).

Kodi ‘Trademark Troll’ Has Interesting Views on Co-Opting Other People’s Work

Post Syndicated from Andy original https://torrentfreak.com/kodi-trademark-troll-has-interesting-views-on-co-opting-other-peoples-work-170917/

The Kodi team, operating under the XBMC Foundation, announced last week that a third-party had registered the Kodi trademark in Canada and was using it for their own purposes.

That person was Geoff Gavora, who had previously been in communication with the Kodi team, expressing how important the software was to his sales.

“We had hoped, given the positive nature of his past emails, that perhaps he was doing this for the benefit of the Foundation. We learned, unfortunately, that this was not the case,” XBMC Foundation President Nathan Betzen said.

According to the Kodi team, Gavora began delisting Amazon ads placed by companies selling Kodi-enabled products, based on infringement of Gavora’s trademark rights.

“[O]nly Gavora’s hardware can be sold, unless those companies pay him a fee to stay on the store,” Betzen explained.

Predictably, Gavora’s move is being viewed as highly controversial, not least since he’s effectively claiming licensing rights in Canada over what should be a free and open source piece of software. TF obtained one of the notices Amazon sent to a seller of a Kodi-enabled device in Canada, following a complaint from Gavora.

Take down Kodi from Amazon, or pay Gavora

So who is Geoff Gavora and what makes him tick? Thanks to a 2016 interview with Ali Salman of the Rapid Growth Podcast, we have a lot of information from the horse’s mouth.

It all began in 2011, when Gavora began jailbreaking Apple TVs, loading them with XBMC, and selling them to friends.

“I did it as a joke, for beer money from my friends,” Gavora told Salman.

“I’d do it for $25 to $50 and word of mouth spread that I was doing this so we could load on this media center to watch content and online streams from it.”

Intro to the interview with Ali Salman

Soon, however, word of mouth caused the business to grow wings, Gavora claims.

“So they started telling people and I start telling people it’s $50, and then I got so busy so I start telling people it’s $75. I’m getting too busy with my work and with this. And it got to the point where I was making more jailbreaking these Apple TVs than I was at my career, and I wasn’t very happy at my career at that time.”

Jailbreaking was supposed to be a side thing to tide Gavora over until another job came along, but he had a problem – he didn’t come from a technical background. Nevertheless, what Gavora did have was a background in marketing and with a decent knowledge of how to succeed in customer service, he majored on that front.

Gavora had come to learn that while people wanted his devices, they weren’t very good at operating XBMC (Kodi’s former name) which he’d loaded onto them. With this in mind, he began offering web support and phone support via a toll-free line.

“I started receiving calls from New York, Dallas, and then Australia, Hong Kong. Everyone around the world was calling me and saying ‘we hear there’s some kid in Calgary, some young child, who’s offering tech support for the Apple TV’,” Gavora said.

But with things apparently going well, a wrench was soon thrown into the works when Apple released the third variant of its Apple TV and Gavorra was unable to jailbreak it. This prompted him to market his own Linux-based set-top device and his business, Raw-Media, grew from there.

While it seems likely that so-called ‘Raw Boxes’ were doing reasonably well with consumers, what was the secret of their success? Podcast host Salman asked Gavora for his ‘networking party 10-second pitch’, and the Canadian was happy to oblige.

“I get this all the time actually. I basically tell people that I sell a box that gives them free TV and movies,” he said.

This was met with laughter from the host, to which Gavora added, “That’s sort of the three-second pitch and everyone’s like ‘Oh, tell me more’.”

“Who doesn’t like free TV, come on?” Salman responded. “Yeah exactly,” Gavora said.

The image below, taken from a January 2016 YouTube unboxing video, shows one of the products sold by Gavora’s company.

Raw-Media Kodi Box packaging (note Kodi logo)

Bearing in mind the offer of free movies and TV, the tagline on the box, “Stop paying for things you don’t want to watch, watch more free tv!” initially looks quite provocative. That being said, both the device and Kodi are perfectly capable of playing plenty of legal content from free sources, so there’s no problem there.

What is surprising, however, is that the unboxing video shows the device being booted up, apparently already loaded with infamous third-party Kodi addons including PrimeWire, Genesis, Icefilms, and Navi-X.

The unboxing video showing the Kodi setup

Given that Gavora has registered the Kodi trademark in Canada and prints the official logo on his packaging, this runs counter to the official Kodi team’s aggressive stance towards boxes ready-configured with what they categorize as banned addons. Matters are compounded when one visits the product support site.

As seen in the image below, Raw-Media devices are delivered with a printed card in the packaging informing people where to get the after-sales services Gavora says he built his business upon. The cards advise people to visit No-Issue.ca, a site setup to offer text and video-based support to set-top box buyers.

No-Issue.ca (which is hosted on the same server as raw-media.ca and claimed officially as a sister site here) now redirects to No-Issue.is, as per a 2016 announcement. It has a fairly bland forum but the connected tutorial videos, found on No Issue’s YouTube channel, offer a lot more spice.

Registered under Gavora’s online nickname Gombeek (which is also used on the official Kodi forums), the channel is full of videos detailing how to install and use a wide range of addons.

The No-issue YouTube Channel tutorials

But while supplying tutorial videos is one thing, providing the actual software addons is another. Surprisingly, No-Issue does that too. Filed away under the URL http://solved.no-issue.is/ is a Kodi repository which distributes a wide range of addons, including many that specialize in infringing content, according to the Kodi team.

The No-Issue repository

A source familiar with Raw-Media’s devices informs TF that they’re no longer delivered with addons installed. However, tools hosted on No-Issue.is automate the installation process for the customer, with unlisted YouTube Videos (1,2) providing the instructions.

XBMC Foundation President Nathan Betzen says that situation isn’t ideal.

“If that really is his repo it is disappointing to see that Gavora is charging a fee or outright preventing the sale of boxes with Kodi installed that do not include infringing add-ons, while at the same time he is distributing boxes himself that do include the infringing add-ons like this,” Betzen told TF.

While the legality of this type of service is yet to be properly tested in Canada and may yet emerge as entirely permissible under local law, Gavora himself previously described his business as operating in a gray area.

“If I could go back in time four years, I would’ve been more aggressive in the beginning because there was a lot of uncertainty being in a gray market business about how far I could push it,” he said.

“I really shouldn’t say it’s a gray market because everything I do is completely above board, I just felt it was more gray market so I was a bit scared,” he added.

But, legality aside (which will be determined in due course through various cases 1,2), the situation is still problematic when it comes to the Kodi trademark.

The official Kodi team indicate they don’t want to be associated with any kind of questionable addon or even tutorials for the same. Nevertheless, several of the addons installed by No-Issue (including PrimeWire, cCloud TV, Genesis, Icefilms, MoviesHD, MuchMovies and Navi-X, to name a few), are present on the Kodi team’s official ban list.

The fact remains, however, that Gavora successfully registered the trademark in Canada (one month later it was transferred to a brand new company at the same address), and Kodi now have no control over the situation in the country, short of a settlement or some kind of legal action.

Kodi matters aside, though, we get more insight into Gavora’s attitudes towards intellectual property after learning that he studied gemology and jewelry at school. He’s a long-standing member of jewelry discussion forum Ganoskin.com (his profile links to Gavora.com, a domain Gavora owns, as per information supplied by Amazon).

Things get particularly topical in a 2006 thread titled “When your work gets ripped“. The original poster asked how people feel when their jewelry work gets copied and Gavora made his opinions known.

“I think that what most people forget to remember is that when a piece from Tiffany’s or Cartier is ripped off or copied they don’t usually just copy the work, they will stamp it with their name as well,” Gavora said.

“This is, in fact, fraud and they are deceiving clients into believing they are purchasing genuine Tiffany’s or Cartier pieces. The client is in fact more interested in purchasing from an artist than they are the piece. Laying claim to designs (unless a symbol or name is involved) is outrageous.”

Unless that ‘design’ is called Kodi, of course, then it’s possible to claim it as your own through an administrative process and begin demanding licensing fees from the public. That being said, Gavora does seem to flip back and forth a little, later suggesting that being copied is sometimes ok.

“If someone copies your design and produces it under their own name, I think one should be honored and revel in the fact that your design is successful and has caused others to imitate it and grow from it,” he wrote.

“I look forward to the day I see one of my original designs copied, that is the day I will know my design is a success.”

From their public statements, this opinion isn’t shared by the Kodi team in respect of their product. Despite the Kodi name, software and logo being all their own work, they now find themselves having to claw back rights in Canada, in order to keep the product free in the region. For now, however, that seems like a difficult task.

TorrentFreak wrote to Gavora and asked him why he felt the need to register the Kodi trademark, but we received no response. That means we didn’t get the chance to ask him why he’s taking down Amazon listings for other people’s devices, or about something else that came up in the podcast.

“My biggest weakness, I guess, is that I’m too ethical about how I do my business,” he said, referring to how he deals with customers.

Only time will tell how that philosophy will affect Gavora’s attitudes to trademarks and people’s desire not to be charged for using free, open source software.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

Self-Driving Cars Should Be Open Source

Post Syndicated from Bozho original https://techblog.bozho.net/self-driving-cars-open-source/

Self-driving cars are (will be) the pinnacle of consumer products automation – robot vacuum cleaners, smart fridges and TVs are just toys compared to self-driving cars. Both in terms of technology and in terms of impact. We aren’t yet on level 5 self driving cars , but they are behind the corner.

But as software engineers we know how fragile software is. And self-driving cars are basically software, so we can see all the risks involved with putting our lives in the hands anonymous (from our point of view) developers and unknown (to us) processes and quality standards. One may argue that this has been the case for every consumer product ever, but with software is different – software is way more complex than anything else.

So I have an outrageous proposal – self-driving cars should be open source. We have to be able to verify and trust the code that’s navigating our helpless bodies around the highways. Not only that, but we have to be able to verify if it is indeed that code that is currently running in our car, and not something else.

In fact, let me extend that – all cars should be open source. Before you say “but that will ruin the competitive advantage of manufacturers and will be deadly for business”, I don’t actually care how they trained their neural networks, or what their datasets are. That’s actually the secret sauce of the self-driving car and in my view it can remain proprietary and closed. What I’d like to see open-sourced is everything else. (Under what license – I’d be fine to even have it copyrighted and so not “real” open source, but that’s a separate discussion).

Why? This story about remote carjacking using the entertainment system of a Jeep is a scary example. Attackers that reverse engineer the car software can remotely control everything in the car. Why did that happen? Well, I guess it’s complicated and we have to watch the DEFCON talk.

And also read the paper, but a paragraph in wikipedia about the CAN bus used in most cars gives us a hint:

CAN is a low-level protocol and does not support any security features intrinsically. There is also no encryption in standard CAN implementations, which leaves these networks open to man-in-the-middle packet interception. In most implementations, applications are expected to deploy their own security mechanisms; e.g., to authenticate incoming commands or the presence of certain devices on the network. Failure to implement adequate security measures may result in various sorts of attacks if the opponent manages to insert messages on the bus. While passwords exist for some safety-critical functions, such as modifying firmware, programming keys, or controlling antilock brake actuators, these systems are not implemented universally and have a limited number of seed/key pair

I don’t know in what world it makes sense to even have a link between the entertainment system and the low-level network that operates the physical controls. As apparent from the talk, the two systems are supposed to be air-gapped, but in reality they aren’t.

Rookie mistakes were abound – unauthenticated “execute” method, running as root, firmware is not signed, hard-coded passwords, etc. How do we know that there aren’t tons of those in all cars out there right now, and in the self-driving cars of the future (which will likely use the same legacy technologies of the current cars)? Recently I heard a negative comment about the source code of one of the self-driving cars “players”, and I’m pretty sure there are many of those rookie mistakes.

Why this is this even more risky for self-driving cars? I’m not an expert in car programming, but it seems like the attack surface is bigger. I might be completely off target here, but on a typical car you’d have to “just” properly isolate the CAN bus. With self-driving cars the autonomous system that watches the surrounding and makes decisions on what to do next has to be connected to the CAN bus. With Tesla being able to send updates over the wire, the attack surface is even bigger (although that’s actually a good feature – to be able to patch all cars immediately once a vulnerability is discovered).

Of course, one approach would be to introduce legislation that regulates car software. It might work, but it would rely on governments to to proper testing, which won’t always be the case.

The alternative is to open-source it and let all the white-hats find your issues, so that you can close them before the car hits the road. Not only that, but consumers like me will feel safer, and geeks would be able to verify whether the car is really running the software it claims to run by verifying the fingerprints.

Richard Stallman might be seen as a fanatic when he advocates against closed source software, but in cases like … cars, his concerns seem less extreme.

“But the Jeep vulnerability was fixed”, you may say. And that might be seen as being the way things are – vulnerabilities appear, they get fixed, life goes on. No person was injured because of the bug, right? Well, not yet. And “gaining control” is the extreme scenario – there are still pretty bad scenarios, like being able to track a car through its GPS, or cause panic by controlling the entertainment system. It might be over wifi, or over GPRS, or even by physically messing with the car by inserting a flash drive. Is open source immune to those issues? No, but it has proven to be more resilient.

One industry where the problem of proprietary software on a product that the customer bought is … tractors. It turns out farmers are hacking their tractors, because of multiple issues and the inability of the vendor to resolve them in a timely manner. This is likely to happen to cars soon, when only authorized repair shops are allowed to touch anything on the car. And with unauthorized repair shops the attack surface becomes even bigger.

In fact, I’d prefer open source not just for cars, but for all consumer products. The source code of a smart fridge or a security camera is trivial, it would rarely mean sacrificing competitive advantage. But refrigerators get hacked, security cameras are active part of botnets, the “internet of shit” is getting ubiquitous. A huge amount of these issues are dumb, beginner mistakes. We have the right to know what shit we are running – in our frdges, DVRs and ultimatey – cars.

Your fridge may soon by spying on you, your vacuum cleaner may threaten your pet in demand of “ransom”. The terrorists of the future may crash planes without being armed, can crash vans into crowds without being in the van, and can “explode” home equipment without being in the particular home. And that’s not just a hypothetical.

Will open source magically solve the issue? No. But it will definitely make things better and safer, as it has done with operating systems and web servers.

The post Self-Driving Cars Should Be Open Source appeared first on Bozho's tech blog.

Seth – RDP Man In The Middle Attack Tool

Post Syndicated from Darknet original https://www.darknet.org.uk/2017/09/seth-rdp-man-in-the-middle-attack-tool/?utm_source=rss&utm_medium=social&utm_campaign=darknetfeed

Seth – RDP Man In The Middle Attack Tool

Seth is an RDP Man In The Middle attack tool written in Python to MiTM RDP connections by attempting to downgrade the connection in order to extract clear text credentials.

It was developed to raise awareness and educate about the importance of properly configured RDP connections in the context of pentests, workshops or talks.

Usage of Seth RDP Man In The Middle Attack Tool

Run it like this:

$ ./seth.sh <INTERFACE> <ATTACKER IP> <VICTIM IP> <GATEWAY IP|HOST IP>

Unless the RDP host is on the same subnet as the victim machine, the last IP address must be that of the gateway.

Read the rest of Seth – RDP Man In The Middle Attack Tool now! Only available at Darknet.

WordPress Reports Surge in ‘Piracy’ Takedown Notices, Rejects 78%

Post Syndicated from Ernesto original https://torrentfreak.com/wordpress-reports-surge-in-piracy-takedown-notices-rejects-78-170909/

Automattic, the company behind the popular WordPress.com blogging platform, receives thousands of takedown requests from rightsholders.

A few days ago the company published its latest transparency report, showing that it had processed 9,273 requests during the first half of 2017.

This is more than double the amount it received during the same period last year, which is a significant increase. Looking more closely at the numbers, we see that this jump is solely due to an increase in incomplete and abusive requests.

Of all the DMCA notices received, only 22% resulted in the takedown of allegedly infringing content. This translates to 2,040 legitimate requests, which is less than the 2,342 Automattic received during the same period last year.

This logically means that the number of abusive and incomplete DMCA notices has skyrocketed. And indeed, in its most recent report, 78% of all requests were rejected due to missing information or plain abuse. That’s much more than the year before when 42% were rejected.

Automattic’s transparency report (first half of 2017)

WordPress prides itself on carefully reviewing the content of each and every takedown notice, to protect its users. This means checking whether a takedown request is properly formatted but also reviewing the legitimacy of the claims.

“We also may decline to remove content if a notice is abusive. ‘Abusive’ notices may be formally complete, but are directed at fair use of content, material that isn’t copyrightable, or content the complaining party misrepresents ownership of a copyright,” Automattic notes.

During the first half of 2017, a total of 649 takedown requests were categorized as abuse. Some of the most blatant examples go into the “Hall of Shame,” such as a recent case where the Canadian city of Abbotsford tried to censor a parody of its logo, which replaced a pine tree with a turd.

While some abuse cases sound trivial they can have a real impact on website operators, as examples outside of WordPress show. Most recently the operator of Oro Jackson, a community dedicated to the anime series “One Piece,” was targeted with several dubious DMCA requests.

The takedown notices were sent by the German company Comeso and were forwarded through their hosting company Linode. The notices urged the operator to remove various forum threads because they included words of phrases such as “G’day” and “Reveries of the Moonlight,” not actual infringing content.

G’day

Fearing legal repercussions, the operator saw no other option than to censor these seemingly harmless discussions (starting a thread with “G’day”!!), until there’s a final decision on the counter-notice. They remain offline today.

It’s understandable that hosting companies have to be strict sometimes, as reviewing copyright claims is not their core business. However, incidents like these show how valuable the skeptical review process of Automattic is.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

Make your own game with CoderDojo’s new book

Post Syndicated from Nuala McHale original https://www.raspberrypi.org/blog/coderdojo-nano/

The first official CoderDojo book, CoderDojo Nano: Build Your Own Website, was a resounding success: thousands of copies have been bought by aspiring CoderDojo Ninjas, and it‘s available in ten languages, including Bulgarian, Czech, Dutch, Lithuanian, Latvian, Portuguese, Spanish, and Slovakian. Now we are delighted to announce the release of the second book in our Create with Code trilogy, titled CoderDojo Nano: Make Your Own Game.

Cover of CoderDojo Nano Make your own game

The paperback book will be available in English from Thursday 7 September (with English flexibound and Dutch versions scheduled to follow in the coming months), enabling young people and adults to learn creative and fun coding skills!

What will you learn?

The new book explains the fundamentals of the JavaScript language in a clear, logical way while supporting you to create your very own computer game.

Pixel image of laptop displaying a jump-and-run game

You will learn how to animate characters, create a world for your game, and use the physics of movement within it. The book is full of clear step-by-step instructions and illustrated screenshots to make reviewing your code easy. Additionally, challenges and open-ended prompts at the end of each section will encourage you to get creative while making your game.

This book is the perfect first step towards understanding game development, particularly for those of you who do not (yet) have a local Dojo. Regardless of where you live, using our books you too can learn to ‘Create with Code’!

Tried and tested

As always, CoderDojo Ninjas from all around the world tested our book, and their reactions have been hugely positive. Here is a selection of their thoughts:

“The book is brilliant. The [game] is simple yet innovative. I personally love it, and want to get stuck in making it right away!”

“What I really like is that, unlike most books on coding, this one properly explains what’s happening, and what each piece of code does and where it comes from.”

“I found the book most enjoyable. The layout is great, with lots of colour, and I found the information very easy to follow. The Ninja Tips are a great help in case you get a bit stuck. I liked that the book represents a mix of boy and girl Ninjas — it really makes coding fun for all.”

“The book is a great guide for both beginners and people who want to do something creative with their knowledge of code. Even people who cannot go to a CoderDojo can learn code using this book!”

Writer Jurie Horneman

Author of CoderDojo Nano: Make Your Own Game Jurie Horneman has been working in the game development industry for more than 15 years.

stuffed toy rabbit wearing glasses

Jurie would get on well with Babbage, I think.

He shares how he got into coding, and what he has learnt while creating this awesome book:

“I’ve been designing and programming games since 1991, starting with ancient home computers, and now I’m working with PCs and consoles. As a game designer, it’s my job to teach players the rules of the game in a fun and playful manner — that gave me some useful experience for writing the book.

I believe that, if you want to understand something properly, you have to teach it to others. Therefore, writing this book was very educational for me, as I hope reading it will be for learners.”

Asked what his favorite thing about the book is, Jurie said he loves the incredible pixel art design: “The artist (Gary J Lucken, Army of Trolls) did a great job to help explain some of the abstract concepts in the book.”

Pixel image of a landscape with an East Asian temple on a lonely mountain

Gary’s art is also just gorgeous.

How can you get your copy?

You can pre-order CoderDojo Nano: Make Your Own Game here. Its initial pricing is £9.99 (around €11), and discounted copies with free international delivery are available here.

The post Make your own game with CoderDojo’s new book appeared first on Raspberry Pi.

Time Warner Hacked – AWS Config Exposes 4M Subscribers

Post Syndicated from Darknet original https://www.darknet.org.uk/2017/09/time-warner-hacked-aws-config-exposes-4m-subscribers/?utm_source=rss&utm_medium=social&utm_campaign=darknetfeed

Time Warner Hacked – AWS Config Exposes 4M Subscribers

What’s the latest on the web, Time Warner Hacked is what it’s about now as a bad AWS S3 config (once again) exposes the details of approximately 4 Million subscribers.

This follows not long after the Instagram API leaking user contact information and a few other recent leaks involving poorly secured Amazon AWS S3 buckets and I’d hazard a guess that it won’t be the last.

Records of roughly four million Time Warner Cable customers in the US were exposed to the public internet after a contractor failed to properly secure an Amazon cloud database.

Read the rest of Time Warner Hacked – AWS Config Exposes 4M Subscribers now! Only available at Darknet.

Security updates for Monday

Post Syndicated from jake original https://lwn.net/Articles/732865/rss

Security updates have been issued by Debian (enigmail, gnupg, libgd2, libidn, libidn2-0, mercurial, and strongswan), Fedora (gd, libidn2, mbedtls, mingw-openjpeg2, openjpeg2, and xen), Mageia (apache-commons-email, botan, iceape, poppler, rt/perl-Encode, samba, and wireshark), and openSUSE (expat, freerdp, git, libzypp, and php7).

Search Engines Will Open Systems to Prove Piracy & VPN Blocking

Post Syndicated from Andy original https://torrentfreak.com/search-engines-will-open-systems-to-prove-piracy-vpn-blocking-170901/

Over the past several years, Russia has become something of a world leader when it comes to website blocking. Tens of thousands of websites are now blocked in the country on copyright infringement and a wide range of other grounds.

With circumvention technologies such as VPNs, however, Russian citizens are able to access blocked sites, a position that has irritated Russian authorities who are determined to control what information citizens are allowed to access.

After working on new legislation for some time, late July President Vladimir Putin signed a new law which requires local telecoms watchdog Rozcomnadzor to maintain a list of banned domains while identifying sites, services, and software that provide access to them.

Rozcomnadzor is required to contact the operators of such services with a request for them to block banned resources. If they do not, then they themselves will become blocked. In addition, search engines are also required to remove blocked resources from their search results, in order to discourage people from accessing them.

With compliance now a matter of law, attention has turned to how search engines can implement the required mechanisms. This week Roskomnadzor hosted a meeting with representatives of the largest Russian search engines including Yandex, Sputnik, Search Mail.ru, where this topic was top of the agenda.

Since failure to comply can result in a fine of around $12,000 per breach, search companies have a vested interest in the systems working well against not only pirate sites, but also mirrors and anonymization tools that provide access to them.

“During the meeting, a consolidated position on the implementation of new legislative requirements was developed,” Rozcomnadzor reports.

“It was determined that the list of blocked resources to be removed from search results will be transferred to the operators of search engines in an automated process.”

While sending over lists of domains directly to search engines probably isn’t that groundbreaking, Rozcomnadzor wants to ensure that companies like Yandex are also responding to the removal requests properly.

So, instead of simply carrying out test searches itself, it’s been agreed that the watchdog will gain direct access to the search engines’ systems, so that direct verification can take place.

“In addition, preliminary agreements have been reached that the verification of the enforcement of the law by the search engines will be carried out through the interaction of the information systems of Roskomnadzor and the operators of search engines,” Rozcomnadzor reports.

Time for search engines to come into full compliance is ticking away. The law requiring them to remove listings for ‘pirate’ mirror sites comes into effect October 1. Exactly a month later on November 1, VPNs and anonymization tools will have to be removed too, if they fail to meet the standards required under state regulation.

Part of that regulation requires anonymization services to disclose the identities of their owners to the government.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

Security updates for Thursday

Post Syndicated from jake original https://lwn.net/Articles/732530/rss

Security updates have been issued by Debian (connman, faad2, gnupg, imagemagick, libdbd-mysql-perl, mercurial, and php5), openSUSE (postgresql93 and samba and resource-agents), Oracle (poppler), Scientific Linux (poppler), SUSE (firefox and php7), and Ubuntu (pyjwt).

HiveMQ 3.2.7 released

Post Syndicated from The HiveMQ Team original http://www.hivemq.com/blog/hivemq-3-2-7-released/

The HiveMQ team is pleased to announce the availability of HiveMQ 3.2.7. This is a maintenance release for the 3.2 series and brings the following improvements:

  • Added logging of the replication state when a node joins a cluster
  • Fixed a reference counting issue
  • Fixed an issue that could cause a client to not receive queued messages until it reconnects again
  • Fixed an issue that could cause other subscriptions to be removed for the same client if an unsubscribe message is sent to the wildcard topic
  • Fixed an issue that caused clients to not be able to change the QoS for their subscriptions
  • Fixed an rare issue that could cause a client to not receive any messages after a successful connection
  • Fixed an cosmetic issue that caused IOExceptions to be wrongfully logged when a client disconnected
  • Fixed an issue that could cause HiveMQ to not shut down properly when a plugin triggers a shutdown via UnrecoverableExceptions
  • Fixed an issue that could cause cluster nodes to not be operational on startup
  • Fixed verbose RESTService logging
  • Fixed an issue that could cause messages to be queued although there is no need to queue them
  • Fixed an issue that could lead to incorrect subscription count metrics
  • Fixed an issue that could cause the loss of retained messages when using the stateful-cluster.sh script

You can download the new HiveMQ version here.

We strongly recommend to upgrade if you are an HiveMQ 3.2.x user.

Have a great day,
The HiveMQ Team

Philips Wants to Stop Piracy With ‘Ambilight’ Technology

Post Syndicated from Ernesto original https://torrentfreak.com/philips-wants-to-stop-piracy-with-ambilight-technology-170729/

The movie industry sees illegally recorded movies as one of the biggest piracy threats and goes to extremes to stop it.

Over the years, movie theaters have used audio and video watermarking tools to detect pirates, for example. And during prominent events, night-vision goggles and other spy tech is used to monitor movie goers.

Most anti-piracy efforts are not particularly pleasing for the public, but a new technology from Philips hopes to change this. In a recent patent application the technology company proposes an ambient lighting system for movie theaters that will hinder pirate recordings.

“The major motion picture studios are losing large amounts of money due to piracy. The worldwide motion picture industry, distributors, theaters, video stores and pay-per-view operators, lose each year billions of dollars as a result of piracy,” the company begins.

Philips notes that many of the current security measures, such as watermarking, make piracy traceable. However, they don’t prevent actual recording and distribution. Watermarks can be simply rendered useless if pirates remove them afterwards, for example.

“The problem with current security measures in digital cinema is that they do not prevent pirates from filming the content from the cinema screen. Watermarks can help to identify the source of pirate content, but it does not prevent the actual copying and spreading of the content.”

The ambient lighting solution is different and Philips believes that it could reduce piracy substantially.

The concept is quite easy. By shining light on the visitors from the side, at a certain frequency, the pirate recordings will be severely degraded up to a point where they’ll become completely unwatchable.

“The solution proposed here is to shine visible light from an (additional) light source with a pre-determined modulation frequency at the cinema screen while the movie plays,” the patent application reads.

“When the frequency is chosen well, a camera will not be able to properly film the content, because the light source runs out of sync with the camera resulting in stripes running through the movie content.”

The idea is partly inspired by Philips existing Ambilight technology for TVs. This generates lighting effects around the TV that correspond to the video content. The cinema setup will do the same while hindering pirates in the process. A win-win situation, according to Philips.

“It is yet a further aspect to provide a more immersive experience when watching video content on a display device in combination with anti-piracy measures,” the patent application reads.

It’s unclear whether there are concrete plans to roll out the technology at movie theaters. Philips, however, is no stranger to these type of anti-piracy solutions. The company previously rolled out CineFence, a forensic marking technology for both image and sound, targeted at digital cinema content.

In any case, it’s a good marketing message to get movie theaters to try ambient lighting. Whether it’s really effective has yet to be seen though.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

Stubbing Key-Value Stores

Post Syndicated from Bozho original https://techblog.bozho.net/stubbing-key-value-stores/

Every project that has a database has dilemma: how to test database-dependent code. There are several options (not mutually exclusive):

  • Use mocks – use only unit tests and mock the data-access layer, assuming the DAO-to-database communication works
  • Use an embedded database that each test starts and shuts down. This can also be viewed as unit-testing
  • Use a real database deployed somewhere (either locally or on a test environment). The hard part is making sure it’s always in a clean state.
  • Use end-to-end/functional tests/bdd/UI tests after deploying the application on a test server (which has a proper database).

None of the above is without problems. Unit tests with mocked DAOs can’t really test more complex interactions that rely on a database state. Embedded databases are not always available (e.g. if you are using a non-relational database, or if you rely on RDBMS-specific functionality, HSQLDB won’t do), or they can be slow to start and this your tests may take too long supporting. A real database installation complicates setup and keeping it clean is not always easy. The coverage of end-to-end tests can’t be easily measured and they don’t necessarily cover all the edge cases, as they are harder to maintain than unit and integration tests.

I’ve recently tried a strange approach that is working pretty well so far – stubbing the database. It is applicable more to key-value stores and less to relational databases.

In my case, even though there is embedded cassandra, it was slow to start, wasn’t easy to setup and had subtle issues. That’s why I replaced the whole thing with an in-memory ConcurrentHashMap.

Since I’m using spring-data-cassandra, I just extended the CassandraTemplate class and implemented all the method in the new StubCassandraTemplate, and used it instead of the regular one in the test spring context. The stub can support all the key/value operations pretty easily and you can have a bit more complicated integration tests (it’s not a good idea to have very complicated tests, of course, but unit tests can either be too simple or too reliant on a lot of mocks). Here’s an excerpt from the code:

@Component("cassandraTemplate")
public class StubCassandraTemplate extends CassandraTemplate {
    
    private Map<Class<?>, Map<Object, Object>> data = new ConcurrentHashMap<>();
    
    @Override
    public void afterPropertiesSet() {
        // no validation
    }
    
    @SuppressWarnings("unchecked")
    @Override
    public <T> T insert(T entity) {
        List<Field> pk = FieldUtils.getFieldsListWithAnnotation(entity.getClass(), PrimaryKey.class);
        initializeClass(entity.getClass());
        try {
            pk.get(0).setAccessible(true);
            return (T) data.get(entity.getClass()).put(pk.get(0).get(entity), entity);
        } catch (IllegalAccessException e) {
            throw new IllegalArgumentException(e);
        }
    }

    private <T> void initializeClass(Class<?> clazz) {
        if (data.get(clazz) == null) {
            data.put(clazz, new ConcurrentHashMap<>());
        }
    }
....
}

Cassandra supports some advanced features like CQL (query language), which isn’t as easy to stub as key-value operations like get and put, but in fact it is not that hard. Especially if you do not rely on complicated where clauses (and this is a bad practice in Cassandra anyway), it’s easy to parse the query with regex and find the appropriate entries in the ConcurrentHashMap.

Key-value stores are a good candidate for this approach, as their main advantage – being easy to scale horizontally – is not needed in an integration test scenario. You simply need to verify that your code correctly handles interactions with the database in terms of what it puts there and what it gets back. The exact implementation of that interaction – whether it’s in-memory or using a binary protocol, may be viewed as out of scope.

Note that these tests do not guarantee that the application will work with a real database. They only guarantee that it will behave properly if the database behaves the same way as an in-memory key-value data structure. Which is normally the assumption, but isn’t always true – e.g. the database can impose additional constraints that your stub implementation doesn’t have. Cassandra, for example, doesn’t allow WHERE queries for non-indexed columns. If you don’t take that into account, obviously, your test will pass, but your application will break.

That’s why you’d still need end-to-end tests and possibly some real integration tests, but you can cover most of the code with a simple in-memory stub and only do some “sanity” full integration tests.

This doesn’t mean you should always stub your database, but it’s a good option in your testing toolbox to consider.

The post Stubbing Key-Value Stores appeared first on Bozho's tech blog.

Analyzing AWS Cost and Usage Reports with Looker and Amazon Athena

Post Syndicated from Dillon Morrison original https://aws.amazon.com/blogs/big-data/analyzing-aws-cost-and-usage-reports-with-looker-and-amazon-athena/

This is a guest post by Dillon Morrison at Looker. Looker is, in their own words, “a new kind of analytics platform–letting everyone in your business make better decisions by getting reliable answers from a tool they can use.” 

As the breadth of AWS products and services continues to grow, customers are able to more easily move their technology stack and core infrastructure to AWS. One of the attractive benefits of AWS is the cost savings. Rather than paying upfront capital expenses for large on-premises systems, customers can instead pay variables expenses for on-demand services. To further reduce expenses AWS users can reserve resources for specific periods of time, and automatically scale resources as needed.

The AWS Cost Explorer is great for aggregated reporting. However, conducting analysis on the raw data using the flexibility and power of SQL allows for much richer detail and insight, and can be the better choice for the long term. Thankfully, with the introduction of Amazon Athena, monitoring and managing these costs is now easier than ever.

In the post, I walk through setting up the data pipeline for cost and usage reports, Amazon S3, and Athena, and discuss some of the most common levers for cost savings. I surface tables through Looker, which comes with a host of pre-built data models and dashboards to make analysis of your cost and usage data simple and intuitive.

Analysis with Athena

With Athena, there’s no need to create hundreds of Excel reports, move data around, or deploy clusters to house and process data. Athena uses Apache Hive’s DDL to create tables, and the Presto querying engine to process queries. Analysis can be performed directly on raw data in S3. Conveniently, AWS exports raw cost and usage data directly into a user-specified S3 bucket, making it simple to start querying with Athena quickly. This makes continuous monitoring of costs virtually seamless, since there is no infrastructure to manage. Instead, users can leverage the power of the Athena SQL engine to easily perform ad-hoc analysis and data discovery without needing to set up a data warehouse.

After the data pipeline is established, cost and usage data (the recommended billing data, per AWS documentation) provides a plethora of comprehensive information around usage of AWS services and the associated costs. Whether you need the report segmented by product type, user identity, or region, this report can be cut-and-sliced any number of ways to properly allocate costs for any of your business needs. You can then drill into any specific line item to see even further detail, such as the selected operating system, tenancy, purchase option (on-demand, spot, or reserved), and so on.

Walkthrough

By default, the Cost and Usage report exports CSV files, which you can compress using gzip (recommended for performance). There are some additional configuration options for tuning performance further, which are discussed below.

Prerequisites

If you want to follow along, you need the following resources:

Enable the cost and usage reports

First, enable the Cost and Usage report. For Time unit, select Hourly. For Include, select Resource IDs. All options are prompted in the report-creation window.

The Cost and Usage report dumps CSV files into the specified S3 bucket. Please note that it can take up to 24 hours for the first file to be delivered after enabling the report.

Configure the S3 bucket and files for Athena querying

In addition to the CSV file, AWS also creates a JSON manifest file for each cost and usage report. Athena requires that all of the files in the S3 bucket are in the same format, so we need to get rid of all these manifest files. If you’re looking to get started with Athena quickly, you can simply go into your S3 bucket and delete the manifest file manually, skip the automation described below, and move on to the next section.

To automate the process of removing the manifest file each time a new report is dumped into S3, which I recommend as you scale, there are a few additional steps. The folks at Concurrency labs wrote a great overview and set of scripts for this, which you can find in their GitHub repo.

These scripts take the data from an input bucket, remove anything unnecessary, and dump it into a new output bucket. We can utilize AWS Lambda to trigger this process whenever new data is dropped into S3, or on a nightly basis, or whatever makes most sense for your use-case, depending on how often you’re querying the data. Please note that enabling the “hourly” report means that data is reported at the hour-level of granularity, not that a new file is generated every hour.

Following these scripts, you’ll notice that we’re adding a date partition field, which isn’t necessary but improves query performance. In addition, converting data from CSV to a columnar format like ORC or Parquet also improves performance. We can automate this process using Lambda whenever new data is dropped in our S3 bucket. Amazon Web Services discusses columnar conversion at length, and provides walkthrough examples, in their documentation.

As a long-term solution, best practice is to use compression, partitioning, and conversion. However, for purposes of this walkthrough, we’re not going to worry about them so we can get up-and-running quicker.

Set up the Athena query engine

In your AWS console, navigate to the Athena service, and click “Get Started”. Follow the tutorial and set up a new database (we’ve called ours “AWS Optimizer” in this example). Don’t worry about configuring your initial table, per the tutorial instructions. We’ll be creating a new table for cost and usage analysis. Once you walked through the tutorial steps, you’ll be able to access the Athena interface, and can begin running Hive DDL statements to create new tables.

One thing that’s important to note, is that the Cost and Usage CSVs also contain the column headers in their first row, meaning that the column headers would be included in the dataset and any queries. For testing and quick set-up, you can remove this line manually from your first few CSV files. Long-term, you’ll want to use a script to programmatically remove this row each time a new file is dropped in S3 (every few hours typically). We’ve drafted up a sample script for ease of reference, which we run on Lambda. We utilize Lambda’s native ability to invoke the script whenever a new object is dropped in S3.

For cost and usage, we recommend using the DDL statement below. Since our data is in CSV format, we don’t need to use a SerDe, we can simply specify the “separatorChar, quoteChar, and escapeChar”, and the structure of the files (“TEXTFILE”). Note that AWS does have an OpenCSV SerDe as well, if you prefer to use that.

 

CREATE EXTERNAL TABLE IF NOT EXISTS cost_and_usage	 (
identity_LineItemId String,
identity_TimeInterval String,
bill_InvoiceId String,
bill_BillingEntity String,
bill_BillType String,
bill_PayerAccountId String,
bill_BillingPeriodStartDate String,
bill_BillingPeriodEndDate String,
lineItem_UsageAccountId String,
lineItem_LineItemType String,
lineItem_UsageStartDate String,
lineItem_UsageEndDate String,
lineItem_ProductCode String,
lineItem_UsageType String,
lineItem_Operation String,
lineItem_AvailabilityZone String,
lineItem_ResourceId String,
lineItem_UsageAmount String,
lineItem_NormalizationFactor String,
lineItem_NormalizedUsageAmount String,
lineItem_CurrencyCode String,
lineItem_UnblendedRate String,
lineItem_UnblendedCost String,
lineItem_BlendedRate String,
lineItem_BlendedCost String,
lineItem_LineItemDescription String,
lineItem_TaxType String,
product_ProductName String,
product_accountAssistance String,
product_architecturalReview String,
product_architectureSupport String,
product_availability String,
product_bestPractices String,
product_cacheEngine String,
product_caseSeverityresponseTimes String,
product_clockSpeed String,
product_currentGeneration String,
product_customerServiceAndCommunities String,
product_databaseEdition String,
product_databaseEngine String,
product_dedicatedEbsThroughput String,
product_deploymentOption String,
product_description String,
product_durability String,
product_ebsOptimized String,
product_ecu String,
product_endpointType String,
product_engineCode String,
product_enhancedNetworkingSupported String,
product_executionFrequency String,
product_executionLocation String,
product_feeCode String,
product_feeDescription String,
product_freeQueryTypes String,
product_freeTrial String,
product_frequencyMode String,
product_fromLocation String,
product_fromLocationType String,
product_group String,
product_groupDescription String,
product_includedServices String,
product_instanceFamily String,
product_instanceType String,
product_io String,
product_launchSupport String,
product_licenseModel String,
product_location String,
product_locationType String,
product_maxIopsBurstPerformance String,
product_maxIopsvolume String,
product_maxThroughputvolume String,
product_maxVolumeSize String,
product_maximumStorageVolume String,
product_memory String,
product_messageDeliveryFrequency String,
product_messageDeliveryOrder String,
product_minVolumeSize String,
product_minimumStorageVolume String,
product_networkPerformance String,
product_operatingSystem String,
product_operation String,
product_operationsSupport String,
product_physicalProcessor String,
product_preInstalledSw String,
product_proactiveGuidance String,
product_processorArchitecture String,
product_processorFeatures String,
product_productFamily String,
product_programmaticCaseManagement String,
product_provisioned String,
product_queueType String,
product_requestDescription String,
product_requestType String,
product_routingTarget String,
product_routingType String,
product_servicecode String,
product_sku String,
product_softwareType String,
product_storage String,
product_storageClass String,
product_storageMedia String,
product_technicalSupport String,
product_tenancy String,
product_thirdpartySoftwareSupport String,
product_toLocation String,
product_toLocationType String,
product_training String,
product_transferType String,
product_usageFamily String,
product_usagetype String,
product_vcpu String,
product_version String,
product_volumeType String,
product_whoCanOpenCases String,
pricing_LeaseContractLength String,
pricing_OfferingClass String,
pricing_PurchaseOption String,
pricing_publicOnDemandCost String,
pricing_publicOnDemandRate String,
pricing_term String,
pricing_unit String,
reservation_AvailabilityZone String,
reservation_NormalizedUnitsPerReservation String,
reservation_NumberOfReservations String,
reservation_ReservationARN String,
reservation_TotalReservedNormalizedUnits String,
reservation_TotalReservedUnits String,
reservation_UnitsPerReservation String,
resourceTags_userName String,
resourceTags_usercostcategory String  


)
    ROW FORMAT DELIMITED
      FIELDS TERMINATED BY ','
      ESCAPED BY '\\'
      LINES TERMINATED BY '\n'

STORED AS TEXTFILE
    LOCATION 's3://<<your bucket name>>';

Once you’ve successfully executed the command, you should see a new table named “cost_and_usage” with the below properties. Now we’re ready to start executing queries and running analysis!

Start with Looker and connect to Athena

Setting up Looker is a quick process, and you can try it out for free here (or download from Amazon Marketplace). It takes just a few seconds to connect Looker to your Athena database, and Looker comes with a host of pre-built data models and dashboards to make analysis of your cost and usage data simple and intuitive. After you’re connected, you can use the Looker UI to run whatever analysis you’d like. Looker translates this UI to optimized SQL, so any user can execute and visualize queries for true self-service analytics.

Major cost saving levers

Now that the data pipeline is configured, you can dive into the most popular use cases for cost savings. In this post, I focus on:

  • Purchasing Reserved Instances vs. On-Demand Instances
  • Data transfer costs
  • Allocating costs over users or other Attributes (denoted with resource tags)

On-Demand, Spot, and Reserved Instances

Purchasing Reserved Instances vs On-Demand Instances is arguably going to be the biggest cost lever for heavy AWS users (Reserved Instances run up to 75% cheaper!). AWS offers three options for purchasing instances:

  • On-Demand—Pay as you use.
  • Spot (variable cost)—Bid on spare Amazon EC2 computing capacity.
  • Reserved Instances—Pay for an instance for a specific, allotted period of time.

When purchasing a Reserved Instance, you can also choose to pay all-upfront, partial-upfront, or monthly. The more you pay upfront, the greater the discount.

If your company has been using AWS for some time now, you should have a good sense of your overall instance usage on a per-month or per-day basis. Rather than paying for these instances On-Demand, you should try to forecast the number of instances you’ll need, and reserve them with upfront payments.

The total amount of usage with Reserved Instances versus overall usage with all instances is called your coverage ratio. It’s important not to confuse your coverage ratio with your Reserved Instance utilization. Utilization represents the amount of reserved hours that were actually used. Don’t worry about exceeding capacity, you can still set up Auto Scaling preferences so that more instances get added whenever your coverage or utilization crosses a certain threshold (we often see a target of 80% for both coverage and utilization among savvy customers).

Calculating the reserved costs and coverage can be a bit tricky with the level of granularity provided by the cost and usage report. The following query shows your total cost over the last 6 months, broken out by Reserved Instance vs other instance usage. You can substitute the cost field for usage if you’d prefer. Please note that you should only have data for the time period after the cost and usage report has been enabled (though you can opt for up to 3 months of historical data by contacting your AWS Account Executive). If you’re just getting started, this query will only show a few days.

 

SELECT 
	DATE_FORMAT(from_iso8601_timestamp(cost_and_usage.lineitem_usagestartdate),'%Y-%m') AS "cost_and_usage.usage_start_month",
	COALESCE(SUM(cost_and_usage.lineitem_unblendedcost ), 0) AS "cost_and_usage.total_unblended_cost",
	COALESCE(SUM(CASE WHEN (CASE
         WHEN cost_and_usage.lineitem_lineitemtype = 'DiscountedUsage' THEN 'RI Line Item'
         WHEN cost_and_usage.lineitem_lineitemtype = 'RIFee' THEN 'RI Line Item'
         WHEN cost_and_usage.lineitem_lineitemtype = 'Fee' THEN 'RI Line Item'
         ELSE 'Non RI Line Item'
        END = 'RI Line Item') THEN cost_and_usage.lineitem_unblendedcost  ELSE NULL END), 0) AS "cost_and_usage.total_reserved_unblended_cost",
	1.0 * (COALESCE(SUM(CASE WHEN (CASE
         WHEN cost_and_usage.lineitem_lineitemtype = 'DiscountedUsage' THEN 'RI Line Item'
         WHEN cost_and_usage.lineitem_lineitemtype = 'RIFee' THEN 'RI Line Item'
         WHEN cost_and_usage.lineitem_lineitemtype = 'Fee' THEN 'RI Line Item'
         ELSE 'Non RI Line Item'
        END = 'RI Line Item') THEN cost_and_usage.lineitem_unblendedcost  ELSE NULL END), 0)) / NULLIF((COALESCE(SUM(cost_and_usage.lineitem_unblendedcost ), 0)),0)  AS "cost_and_usage.percent_spend_on_ris",
	COALESCE(SUM(CASE WHEN (CASE
         WHEN cost_and_usage.lineitem_lineitemtype = 'DiscountedUsage' THEN 'RI Line Item'
         WHEN cost_and_usage.lineitem_lineitemtype = 'RIFee' THEN 'RI Line Item'
         WHEN cost_and_usage.lineitem_lineitemtype = 'Fee' THEN 'RI Line Item'
         ELSE 'Non RI Line Item'
        END = 'Non RI Line Item') THEN cost_and_usage.lineitem_unblendedcost  ELSE NULL END), 0) AS "cost_and_usage.total_non_reserved_unblended_cost",
	1.0 * (COALESCE(SUM(CASE WHEN (CASE
         WHEN cost_and_usage.lineitem_lineitemtype = 'DiscountedUsage' THEN 'RI Line Item'
         WHEN cost_and_usage.lineitem_lineitemtype = 'RIFee' THEN 'RI Line Item'
         WHEN cost_and_usage.lineitem_lineitemtype = 'Fee' THEN 'RI Line Item'
         ELSE 'Non RI Line Item'
        END = 'Non RI Line Item') THEN cost_and_usage.lineitem_unblendedcost  ELSE NULL END), 0)) / NULLIF((COALESCE(SUM(cost_and_usage.lineitem_unblendedcost ), 0)),0)  AS "cost_and_usage.percent_spend_on_non_ris"
FROM aws_optimizer.cost_and_usage  AS cost_and_usage

WHERE 
	(((from_iso8601_timestamp(cost_and_usage.lineitem_usagestartdate)) >= ((DATE_ADD('month', -5, DATE_TRUNC('MONTH', CAST(NOW() AS DATE))))) AND (from_iso8601_timestamp(cost_and_usage.lineitem_usagestartdate)) < ((DATE_ADD('month', 6, DATE_ADD('month', -5, DATE_TRUNC('MONTH', CAST(NOW() AS DATE))))))))
GROUP BY 1
ORDER BY 2 DESC
LIMIT 500

The resulting table should look something like the image below (I’m surfacing tables through Looker, though the same table would result from querying via command line or any other interface).

With a BI tool, you can create dashboards for easy reference and monitoring. New data is dumped into S3 every few hours, so your dashboards can update several times per day.

It’s an iterative process to understand the appropriate number of Reserved Instances needed to meet your business needs. After you’ve properly integrated Reserved Instances into your purchasing patterns, the savings can be significant. If your coverage is consistently below 70%, you should seriously consider adjusting your purchase types and opting for more Reserved instances.

Data transfer costs

One of the great things about AWS data storage is that it’s incredibly cheap. Most charges often come from moving and processing that data. There are several different prices for transferring data, broken out largely by transfers between regions and availability zones. Transfers between regions are the most costly, followed by transfers between Availability Zones. Transfers within the same region and same availability zone are free unless using elastic or public IP addresses, in which case there is a cost. You can find more detailed information in the AWS Pricing Docs. With this in mind, there are several simple strategies for helping reduce costs.

First, since costs increase when transferring data between regions, it’s wise to ensure that as many services as possible reside within the same region. The more you can localize services to one specific region, the lower your costs will be.

Second, you should maximize the data you’re routing directly within AWS services and IP addresses. Transfers out to the open internet are the most costly and least performant mechanisms of data transfers, so it’s best to keep transfers within AWS services.

Lastly, data transfers between private IP addresses are cheaper than between elastic or public IP addresses, so utilizing private IP addresses as much as possible is the most cost-effective strategy.

The following query provides a table depicting the total costs for each AWS product, broken out transfer cost type. Substitute the “lineitem_productcode” field in the query to segment the costs by any other attribute. If you notice any unusually high spikes in cost, you’ll need to dig deeper to understand what’s driving that spike: location, volume, and so on. Drill down into specific costs by including “product_usagetype” and “product_transfertype” in your query to identify the types of transfer costs that are driving up your bill.

SELECT 
	cost_and_usage.lineitem_productcode  AS "cost_and_usage.product_code",
	COALESCE(SUM(cost_and_usage.lineitem_unblendedcost), 0) AS "cost_and_usage.total_unblended_cost",
	COALESCE(SUM(CASE WHEN REGEXP_LIKE(cost_and_usage.product_usagetype, 'DataTransfer')    THEN cost_and_usage.lineitem_unblendedcost  ELSE NULL END), 0) AS "cost_and_usage.total_data_transfer_cost",
	COALESCE(SUM(CASE WHEN REGEXP_LIKE(cost_and_usage.product_usagetype, 'DataTransfer-In')    THEN cost_and_usage.lineitem_unblendedcost  ELSE NULL END), 0) AS "cost_and_usage.total_inbound_data_transfer_cost",
	COALESCE(SUM(CASE WHEN REGEXP_LIKE(cost_and_usage.product_usagetype, 'DataTransfer-Out')    THEN cost_and_usage.lineitem_unblendedcost  ELSE NULL END), 0) AS "cost_and_usage.total_outbound_data_transfer_cost"
FROM aws_optimizer.cost_and_usage  AS cost_and_usage

WHERE 
	(((from_iso8601_timestamp(cost_and_usage.lineitem_usagestartdate)) >= ((DATE_ADD('month', -5, DATE_TRUNC('MONTH', CAST(NOW() AS DATE))))) AND (from_iso8601_timestamp(cost_and_usage.lineitem_usagestartdate)) < ((DATE_ADD('month', 6, DATE_ADD('month', -5, DATE_TRUNC('MONTH', CAST(NOW() AS DATE))))))))
GROUP BY 1
ORDER BY 2 DESC
LIMIT 500

When moving between regions or over the open web, many data transfer costs also include the origin and destination location of the data movement. Using a BI tool with mapping capabilities, you can get a nice visual of data flows. The point at the center of the map is used to represent external data flows over the open internet.

Analysis by tags

AWS provides the option to apply custom tags to individual resources, so you can allocate costs over whatever customized segment makes the most sense for your business. For a SaaS company that hosts software for customers on AWS, maybe you’d want to tag the size of each customer. The following query uses custom tags to display the reserved, data transfer, and total cost for each AWS service, broken out by tag categories, over the last 6 months. You’ll want to substitute the cost_and_usage.resourcetags_customersegment and cost_and_usage.customer_segment with the name of your customer field.

 

SELECT * FROM (
SELECT *, DENSE_RANK() OVER (ORDER BY z___min_rank) as z___pivot_row_rank, RANK() OVER (PARTITION BY z__pivot_col_rank ORDER BY z___min_rank) as z__pivot_col_ordering FROM (
SELECT *, MIN(z___rank) OVER (PARTITION BY "cost_and_usage.product_code") as z___min_rank FROM (
SELECT *, RANK() OVER (ORDER BY CASE WHEN z__pivot_col_rank=1 THEN (CASE WHEN "cost_and_usage.total_unblended_cost" IS NOT NULL THEN 0 ELSE 1 END) ELSE 2 END, CASE WHEN z__pivot_col_rank=1 THEN "cost_and_usage.total_unblended_cost" ELSE NULL END DESC, "cost_and_usage.total_unblended_cost" DESC, z__pivot_col_rank, "cost_and_usage.product_code") AS z___rank FROM (
SELECT *, DENSE_RANK() OVER (ORDER BY CASE WHEN "cost_and_usage.customer_segment" IS NULL THEN 1 ELSE 0 END, "cost_and_usage.customer_segment") AS z__pivot_col_rank FROM (
SELECT 
	cost_and_usage.lineitem_productcode  AS "cost_and_usage.product_code",
	cost_and_usage.resourcetags_customersegment  AS "cost_and_usage.customer_segment",
	COALESCE(SUM(cost_and_usage.lineitem_unblendedcost ), 0) AS "cost_and_usage.total_unblended_cost",
	1.0 * (COALESCE(SUM(CASE WHEN REGEXP_LIKE(cost_and_usage.product_usagetype, 'DataTransfer')    THEN cost_and_usage.lineitem_unblendedcost  ELSE NULL END), 0)) / NULLIF((COALESCE(SUM(cost_and_usage.lineitem_unblendedcost ), 0)),0)  AS "cost_and_usage.percent_spend_data_transfers_unblended",
	1.0 * (COALESCE(SUM(CASE WHEN (CASE
         WHEN cost_and_usage.lineitem_lineitemtype = 'DiscountedUsage' THEN 'RI Line Item'
         WHEN cost_and_usage.lineitem_lineitemtype = 'RIFee' THEN 'RI Line Item'
         WHEN cost_and_usage.lineitem_lineitemtype = 'Fee' THEN 'RI Line Item'
         ELSE 'Non RI Line Item'
        END = 'Non RI Line Item') THEN cost_and_usage.lineitem_unblendedcost  ELSE NULL END), 0)) / NULLIF((COALESCE(SUM(cost_and_usage.lineitem_unblendedcost ), 0)),0)  AS "cost_and_usage.unblended_percent_spend_on_ris"
FROM aws_optimizer.cost_and_usage_raw  AS cost_and_usage

WHERE 
	(((from_iso8601_timestamp(cost_and_usage.lineitem_usagestartdate)) >= ((DATE_ADD('month', -5, DATE_TRUNC('MONTH', CAST(NOW() AS DATE))))) AND (from_iso8601_timestamp(cost_and_usage.lineitem_usagestartdate)) < ((DATE_ADD('month', 6, DATE_ADD('month', -5, DATE_TRUNC('MONTH', CAST(NOW() AS DATE))))))))
GROUP BY 1,2) ww
) bb WHERE z__pivot_col_rank <= 16384
) aa
) xx
) zz
 WHERE z___pivot_row_rank <= 500 OR z__pivot_col_ordering = 1 ORDER BY z___pivot_row_rank

The resulting table in this example looks like the results below. In this example, you can tell that we’re making poor use of Reserved Instances because they represent such a small portion of our overall costs.

Again, using a BI tool to visualize these costs and trends over time makes the analysis much easier to consume and take action on.

Summary

Saving costs on your AWS spend is always an iterative, ongoing process. Hopefully with these queries alone, you can start to understand your spending patterns and identify opportunities for savings. However, this is just a peek into the many opportunities available through analysis of the Cost and Usage report. Each company is different, with unique needs and usage patterns. To achieve maximum cost savings, we encourage you to set up an analytics environment that enables your team to explore all potential cuts and slices of your usage data, whenever it’s necessary. Exploring different trends and spikes across regions, services, user types, etc. helps you gain comprehensive understanding of your major cost levers and consistently implement new cost reduction strategies.

Note that all of the queries and analysis provided in this post were generated using the Looker data platform. If you’re already a Looker customer, you can get all of this analysis, additional pre-configured dashboards, and much more using Looker Blocks for AWS.


About the Author

Dillon Morrison leads the Platform Ecosystem at Looker. He enjoys exploring new technologies and architecting the most efficient data solutions for the business needs of his company and their customers. In his spare time, you’ll find Dillon rock climbing in the Bay Area or nose deep in the docs of the latest AWS product release at his favorite cafe (“Arlequin in SF is unbeatable!”).

 

 

 

AWS Config Update – New Managed Rules to Secure S3 Buckets

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/aws-config-update-new-managed-rules-to-secure-s3-buckets/

AWS Config captures the state of your AWS resources and the relationships between them. Among other features, it allows you to select a resource and then view a timeline of configuration changes that affect the resource (read Track AWS Resource Relationships With AWS Config to learn more).

AWS Config rules extends Config with a powerful rule system, with support for a “managed” collection of AWS rules as well as custom rules that you write yourself (my blog post, AWS Config Rules – Dynamic Compliance Checking for Cloud Resources, contains more info). The rules (AWS Lambda functions) represent the ideal (properly configured and compliant) state of your AWS resources. The appropriate functions are invoked when a configuration change is detected and check to ensure compliance.

You already have access to about three dozen managed rules. For example, here are some of the rules that check your EC2 instances and related resources:

Two New Rules
Today we are adding two new managed rules that will help you to secure your S3 buckets. You can enable these rules with a single click. The new rules are:

s3-bucket-public-write-prohibited – Automatically identifies buckets that allow global write access. There’s rarely a reason to create this configuration intentionally since it allows
unauthorized users to add malicious content to buckets and to delete (by overwriting) existing content. The rule checks all of the buckets in the account.

s3-bucket-public-read-prohibited – Automatically identifies buckets that allow global read access. This will flag content that is publicly available, including web sites and documentation. This rule also checks all buckets in the account.

Like the existing rules, the new rules can be run on a schedule or in response to changes detected by Config. You can see the compliance status of all of your rules at a glance:

Each evaluation runs in a matter of milliseconds; scanning an account with 100 buckets will take less than a minute. Behind the scenes, the rules are evaluated by a reasoning engine that uses some leading-edge constraint solving techniques that can, in many cases, address NP-complete problems in polynomial time (we did not resolve P versus NP; that would be far bigger news). This work is part of a larger effort within AWS, some of which is described in a AWS re:Invent presentation: Automated Formal Reasoning About AWS Systems:

Now Available
The new rules are available now and you can start using them today. Like the other rules, they are priced at $2 per rule per month.

Jeff;

Bank Robbery Tactic

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2017/08/bank_robbery_ta.html

This video purports to be a bank robbery in Kiev. He first threatens a teller, who basically ignores him because she’s behind bullet-proof glass. But then the robber threatens one of her co-workers, who is on his side of the glass. Interesting example of a security system failing for an unexpected reason.

The video is weird, though. The robber seems very unsure of himself, and never really points the gun at anyone or even holds it properly.

Security updates for Friday

Post Syndicated from ris original https://lwn.net/Articles/730629/rss

Security updates have been issued by Arch Linux (firefox, flashplugin, lib32-flashplugin, libsoup, and varnish), Debian (freeradius, git, libsoup2.4, pjproject, postgresql-9.1, postgresql-9.4, postgresql-9.6, subversion, and xchat), Fedora (gsoap, irssi, knot-resolver, php-horde-horde, php-horde-Horde-Core, php-horde-Horde-Form, php-horde-Horde-Url, php-horde-kronolith, php-horde-nag, and php-horde-turba), Mageia (perl-XML-LibXML), Oracle (libsoup), Red Hat (firefox and libsoup), SUSE (kernel and libsoup), and Ubuntu (git, kernel, libsoup2.4, linux, linux-aws, linux-gke, linux-raspi2, linux-snapdragon, linux, linux-raspi2, linux-hwe, linux-lts-trusty, linux-lts-xenial, php5, php7.0, and subversion).