Tag Archives: spot

N O D E’s Handheld Linux Terminal

Post Syndicated from Alex Bate original https://www.raspberrypi.org/blog/n-o-d-es-handheld-linux-terminal/

Fit an entire Raspberry Pi-based laptop into your pocket with N O D E’s latest Handheld Linux Terminal build.

The Handheld Linux Terminal Version 3 (Portable Pi 3)

Hey everyone. Today I want to show you the new version 3 of the Handheld Linux Terminal. It’s taken a long time, but I’m finally finished. This one takes all the things I’ve learned so far, and improves on many of the features from the previous iterations.

N O D E

With interests in modding tech, exploring the boundaries of the digital world, and open source, YouTuber N O D E has become one to watch within the digital maker world. He maintains a channel focused on “the transformative power of technology.”

“Understanding that electronics isn’t voodoo is really powerful”, he explains in his Patreon video. “And learning how to build your own stuff opens up so many possibilities.”

NODE Youtube channel logo - Handheld Linux Terminal v3

The topics of his videos range from stripped-down devices, upgraded tech, and security upgrades, to the philosophy behind technology. He also provides weekly roundups of, and discussions about, new releases.

Essentially, if you like technology, you’ll like N O D E.

Handheld Linux Terminal v3

Subscribers to N O D E’s YouTube channel, of whom there are currently over 44000, will have seen him documenting variations of this handheld build throughout the last year. By stripping down a Raspberry Pi 3, and incorporating a Zero W, he’s been able to create interesting projects while always putting functionality first.

Handheld Linux Terminal v3

With the third version of his terminal, N O D E has taken experiences gained from previous builds to create something of which he’s obviously extremely proud. And so he should be. The v3 handheld is impressively small considering he managed to incorporate a fully functional keyboard with mouse, a 3.5″ screen, and a fan within the 3D-printed body.

Handheld Linux Terminal v3

“The software side of things is where it really shines though, and the Pi 3 is more than capable of performing most non-intensive tasks,” N O D E goes on to explain. He demonstrates various applications running on Raspbian, plus other operating systems he has pre-loaded onto additional SD cards:

“I have also installed Exagear Desktop, which allows it to run x86 apps too, and this works great. I have x86 apps such as Sublime Text and Spotify running without any problems, and it’s technically possible to use Wine to also run Windows apps on the device.”

We think this is an incredibly neat build, and we can’t wait to see where N O D E takes it next!

The post N O D E’s Handheld Linux Terminal appeared first on Raspberry Pi.

"Responsible encryption" fallacies

Post Syndicated from Robert Graham original http://blog.erratasec.com/2017/10/responsible-encryption-fallacies.html

Deputy Attorney General Rod Rosenstein gave a speech recently calling for “Responsible Encryption” (aka. “Crypto Backdoors”). It’s full of dangerous ideas that need to be debunked.

The importance of law enforcement

The first third of the speech talks about the importance of law enforcement, as if it’s the only thing standing between us and chaos. It cites the 2016 Mirai attacks as an example of the chaos that will only get worse without stricter law enforcement.

But the Mira case demonstrated the opposite, how law enforcement is not needed. They made no arrests in the case. A year later, they still haven’t a clue who did it.

Conversely, we technologists have fixed the major infrastructure issues. Specifically, those affected by the DNS outage have moved to multiple DNS providers, including a high-capacity DNS provider like Google and Amazon who can handle such large attacks easily.

In other words, we the people fixed the major Mirai problem, and law-enforcement didn’t.

Moreover, instead being a solution to cyber threats, law enforcement has become a threat itself. The DNC didn’t have the FBI investigate the attacks from Russia likely because they didn’t want the FBI reading all their files, finding wrongdoing by the DNC. It’s not that they did anything actually wrong, but it’s more like that famous quote from Richelieu “Give me six words written by the most honest of men and I’ll find something to hang him by”. Give all your internal emails over to the FBI and I’m certain they’ll find something to hang you by, if they want.
Or consider the case of Andrew Auernheimer. He found AT&T’s website made public user accounts of the first iPad, so he copied some down and posted them to a news site. AT&T had denied the problem, so making the problem public was the only way to force them to fix it. Such access to the website was legal, because AT&T had made the data public. However, prosecutors disagreed. In order to protect the powerful, they twisted and perverted the law to put Auernheimer in jail.

It’s not that law enforcement is bad, it’s that it’s not the unalloyed good Rosenstein imagines. When law enforcement becomes the thing Rosenstein describes, it means we live in a police state.

Where law enforcement can’t go

Rosenstein repeats the frequent claim in the encryption debate:

Our society has never had a system where evidence of criminal wrongdoing was totally impervious to detection

Of course our society has places “impervious to detection”, protected by both legal and natural barriers.

An example of a legal barrier is how spouses can’t be forced to testify against each other. This barrier is impervious.

A better example, though, is how so much of government, intelligence, the military, and law enforcement itself is impervious. If prosecutors could gather evidence everywhere, then why isn’t Rosenstein prosecuting those guilty of CIA torture?

Oh, you say, government is a special exception. If that were the case, then why did Rosenstein dedicate a precious third of his speech discussing the “rule of law” and how it applies to everyone, “protecting people from abuse by the government”. It obviously doesn’t, there’s one rule of government and a different rule for the people, and the rule for government means there’s lots of places law enforcement can’t go to gather evidence.

Likewise, the crypto backdoor Rosenstein is demanding for citizens doesn’t apply to the President, Congress, the NSA, the Army, or Rosenstein himself.

Then there are the natural barriers. The police can’t read your mind. They can only get the evidence that is there, like partial fingerprints, which are far less reliable than full fingerprints. They can’t go backwards in time.

I mention this because encryption is a natural barrier. It’s their job to overcome this barrier if they can, to crack crypto and so forth. It’s not our job to do it for them.

It’s like the camera that increasingly comes with TVs for video conferencing, or the microphone on Alexa-style devices that are always recording. This suddenly creates evidence that the police want our help in gathering, such as having the camera turned on all the time, recording to disk, in case the police later gets a warrant, to peer backward in time what happened in our living rooms. The “nothing is impervious” argument applies here as well. And it’s equally bogus here. By not helping police by not recording our activities, we aren’t somehow breaking some long standing tradit

And this is the scary part. It’s not that we are breaking some ancient tradition that there’s no place the police can’t go (with a warrant). Instead, crypto backdoors breaking the tradition that never before have I been forced to help them eavesdrop on me, even before I’m a suspect, even before any crime has been committed. Sure, laws like CALEA force the phone companies to help the police against wrongdoers — but here Rosenstein is insisting I help the police against myself.

Balance between privacy and public safety

Rosenstein repeats the frequent claim that encryption upsets the balance between privacy/safety:

Warrant-proof encryption defeats the constitutional balance by elevating privacy above public safety.

This is laughable, because technology has swung the balance alarmingly in favor of law enforcement. Far from “Going Dark” as his side claims, the problem we are confronted with is “Going Light”, where the police state monitors our every action.

You are surrounded by recording devices. If you walk down the street in town, outdoor surveillance cameras feed police facial recognition systems. If you drive, automated license plate readers can track your route. If you make a phone call or use a credit card, the police get a record of the transaction. If you stay in a hotel, they demand your ID, for law enforcement purposes.

And that’s their stuff, which is nothing compared to your stuff. You are never far from a recording device you own, such as your mobile phone, TV, Alexa/Siri/OkGoogle device, laptop. Modern cars from the last few years increasingly have always-on cell connections and data recorders that record your every action (and location).

Even if you hike out into the country, when you get back, the FBI can subpoena your GPS device to track down your hidden weapon’s cache, or grab the photos from your camera.

And this is all offline. So much of what we do is now online. Of the photographs you own, fewer than 1% are printed out, the rest are on your computer or backed up to the cloud.

Your phone is also a GPS recorder of your exact position all the time, which if the government wins the Carpenter case, they police can grab without a warrant. Tagging all citizens with a recording device of their position is not “balance” but the premise for a novel more dystopic than 1984.

If suspected of a crime, which would you rather the police searched? Your person, houses, papers, and physical effects? Or your mobile phone, computer, email, and online/cloud accounts?

The balance of privacy and safety has swung so far in favor of law enforcement that rather than debating whether they should have crypto backdoors, we should be debating how to add more privacy protections.

“But it’s not conclusive”

Rosenstein defends the “going light” (“Golden Age of Surveillance”) by pointing out it’s not always enough for conviction. Nothing gives a conviction better than a person’s own words admitting to the crime that were captured by surveillance. This other data, while copious, often fails to convince a jury beyond a reasonable doubt.
This is nonsense. Police got along well enough before the digital age, before such widespread messaging. They solved terrorist and child abduction cases just fine in the 1980s. Sure, somebody’s GPS location isn’t by itself enough — until you go there and find all the buried bodies, which leads to a conviction. “Going dark” imagines that somehow, the evidence they’ve been gathering for centuries is going away. It isn’t. It’s still here, and matches up with even more digital evidence.
Conversely, a person’s own words are not as conclusive as you think. There’s always missing context. We quickly get back to the Richelieu “six words” problem, where captured communications are twisted to convict people, with defense lawyers trying to untwist them.

Rosenstein’s claim may be true, that a lot of criminals will go free because the other electronic data isn’t convincing enough. But I’d need to see that claim backed up with hard studies, not thrown out for emotional impact.

Terrorists and child molesters

You can always tell the lack of seriousness of law enforcement when they bring up terrorists and child molesters.
To be fair, sometimes we do need to talk about terrorists. There are things unique to terrorism where me may need to give government explicit powers to address those unique concerns. For example, the NSA buys mobile phone 0day exploits in order to hack terrorist leaders in tribal areas. This is a good thing.
But when terrorists use encryption the same way everyone else does, then it’s not a unique reason to sacrifice our freedoms to give the police extra powers. Either it’s a good idea for all crimes or no crimes — there’s nothing particular about terrorism that makes it an exceptional crime. Dead people are dead. Any rational view of the problem relegates terrorism to be a minor problem. More citizens have died since September 8, 2001 from their own furniture than from terrorism. According to studies, the hot water from the tap is more of a threat to you than terrorists.
Yes, government should do what they can to protect us from terrorists, but no, it’s not so bad of a threat that requires the imposition of a military/police state. When people use terrorism to justify their actions, it’s because they trying to form a military/police state.
A similar argument works with child porn. Here’s the thing: the pervs aren’t exchanging child porn using the services Rosenstein wants to backdoor, like Apple’s Facetime or Facebook’s WhatsApp. Instead, they are exchanging child porn using custom services they build themselves.
Again, I’m (mostly) on the side of the FBI. I support their idea of buying 0day exploits in order to hack the web browsers of visitors to the secret “PlayPen” site. This is something that’s narrow to this problem and doesn’t endanger the innocent. On the other hand, their calls for crypto backdoors endangers the innocent while doing effectively nothing to address child porn.
Terrorists and child molesters are a clichéd, non-serious excuse to appeal to our emotions to give up our rights. We should not give in to such emotions.

Definition of “backdoor”

Rosenstein claims that we shouldn’t call backdoors “backdoors”:

No one calls any of those functions [like key recovery] a “back door.”  In fact, those capabilities are marketed and sought out by many users.

He’s partly right in that we rarely refer to PGP’s key escrow feature as a “backdoor”.

But that’s because the term “backdoor” refers less to how it’s done and more to who is doing it. If I set up a recovery password with Apple, I’m the one doing it to myself, so we don’t call it a backdoor. If it’s the police, spies, hackers, or criminals, then we call it a “backdoor” — even it’s identical technology.

Wikipedia uses the key escrow feature of the 1990s Clipper Chip as a prime example of what everyone means by “backdoor“. By “no one”, Rosenstein is including Wikipedia, which is obviously incorrect.

Though in truth, it’s not going to be the same technology. The needs of law enforcement are different than my personal key escrow/backup needs. In particular, there are unsolvable problems, such as a backdoor that works for the “legitimate” law enforcement in the United States but not for the “illegitimate” police states like Russia and China.

I feel for Rosenstein, because the term “backdoor” does have a pejorative connotation, which can be considered unfair. But that’s like saying the word “murder” is a pejorative term for killing people, or “torture” is a pejorative term for torture. The bad connotation exists because we don’t like government surveillance. I mean, honestly calling this feature “government surveillance feature” is likewise pejorative, and likewise exactly what it is that we are talking about.

Providers

Rosenstein focuses his arguments on “providers”, like Snapchat or Apple. But this isn’t the question.

The question is whether a “provider” like Telegram, a Russian company beyond US law, provides this feature. Or, by extension, whether individuals should be free to install whatever software they want, regardless of provider.

Telegram is a Russian company that provides end-to-end encryption. Anybody can download their software in order to communicate so that American law enforcement can’t eavesdrop. They aren’t going to put in a backdoor for the U.S. If we succeed in putting backdoors in Apple and WhatsApp, all this means is that criminals are going to install Telegram.

If the, for some reason, the US is able to convince all such providers (including Telegram) to install a backdoor, then it still doesn’t solve the problem, as uses can just build their own end-to-end encryption app that has no provider. It’s like email: some use the major providers like GMail, others setup their own email server.

Ultimately, this means that any law mandating “crypto backdoors” is going to target users not providers. Rosenstein tries to make a comparison with what plain-old telephone companies have to do under old laws like CALEA, but that’s not what’s happening here. Instead, for such rules to have any effect, they have to punish users for what they install, not providers.

This continues the argument I made above. Government backdoors is not something that forces Internet services to eavesdrop on us — it forces us to help the government spy on ourselves.
Rosenstein tries to address this by pointing out that it’s still a win if major providers like Apple and Facetime are forced to add backdoors, because they are the most popular, and some terrorists/criminals won’t move to alternate platforms. This is false. People with good intentions, who are unfairly targeted by a police state, the ones where police abuse is rampant, are the ones who use the backdoored products. Those with bad intentions, who know they are guilty, will move to the safe products. Indeed, Telegram is already popular among terrorists because they believe American services are already all backdoored. 
Rosenstein is essentially demanding the innocent get backdoored while the guilty don’t. This seems backwards. This is backwards.

Apple is morally weak

The reason I’m writing this post is because Rosenstein makes a few claims that cannot be ignored. One of them is how he describes Apple’s response to government insistence on weakening encryption doing the opposite, strengthening encryption. He reasons this happens because:

Of course they [Apple] do. They are in the business of selling products and making money. 

We [the DoJ] use a different measure of success. We are in the business of preventing crime and saving lives. 

He swells in importance. His condescending tone ennobles himself while debasing others. But this isn’t how things work. He’s not some white knight above the peasantry, protecting us. He’s a beat cop, a civil servant, who serves us.

A better phrasing would have been:

They are in the business of giving customers what they want.

We are in the business of giving voters what they want.

Both sides are doing the same, giving people what they want. Yes, voters want safety, but they also want privacy. Rosenstein imagines that he’s free to ignore our demands for privacy as long has he’s fulfilling his duty to protect us. He has explicitly rejected what people want, “we use a different measure of success”. He imagines it’s his job to tell us where the balance between privacy and safety lies. That’s not his job, that’s our job. We, the people (and our representatives), make that decision, and it’s his job is to do what he’s told. His measure of success is how well he fulfills our wishes, not how well he satisfies his imagined criteria.

That’s why those of us on this side of the debate doubt the good intentions of those like Rosenstein. He criticizes Apple for wanting to protect our rights/freedoms, and declare they measure success differently.

They are willing to be vile

Rosenstein makes this argument:

Companies are willing to make accommodations when required by the government. Recent media reports suggest that a major American technology company developed a tool to suppress online posts in certain geographic areas in order to embrace a foreign government’s censorship policies. 

Let me translate this for you:

Companies are willing to acquiesce to vile requests made by police-states. Therefore, they should acquiesce to our vile police-state requests.

It’s Rosenstein who is admitting here is that his requests are those of a police-state.

Constitutional Rights

Rosenstein says:

There is no constitutional right to sell warrant-proof encryption.

Maybe. It’s something the courts will have to decide. There are many 1st, 2nd, 3rd, 4th, and 5th Amendment issues here.
The reason we have the Bill of Rights is because of the abuses of the British Government. For example, they quartered troops in our homes, as a way of punishing us, and as a way of forcing us to help in our own oppression. The troops weren’t there to defend us against the French, but to defend us against ourselves, to shoot us if we got out of line.

And that’s what crypto backdoors do. We are forced to be agents of our own oppression. The principles enumerated by Rosenstein apply to a wide range of even additional surveillance. With little change to his speech, it can equally argue why the constant TV video surveillance from 1984 should be made law.

Let’s go back and look at Apple. It is not some base company exploiting consumers for profit. Apple doesn’t have guns, they cannot make people buy their product. If Apple doesn’t provide customers what they want, then customers vote with their feet, and go buy an Android phone. Apple isn’t providing encryption/security in order to make a profit — it’s giving customers what they want in order to stay in business.
Conversely, if we citizens don’t like what the government does, tough luck, they’ve got the guns to enforce their edicts. We can’t easily vote with our feet and walk to another country. A “democracy” is far less democratic than capitalism. Apple is a minority, selling phones to 45% of the population, and that’s fine, the minority get the phones they want. In a Democracy, where citizens vote on the issue, those 45% are screwed, as the 55% impose their will unwanted onto the remainder.

That’s why we have the Bill of Rights, to protect the 49% against abuse by the 51%. Regardless whether the Supreme Court agrees the current Constitution, it is the sort right that might exist regardless of what the Constitution says. 

Obliged to speak the truth

Here is the another part of his speech that I feel cannot be ignored. We have to discuss this:

Those of us who swear to protect the rule of law have a different motivation.  We are obliged to speak the truth.

The truth is that “going dark” threatens to disable law enforcement and enable criminals and terrorists to operate with impunity.

This is not true. Sure, he’s obliged to say the absolute truth, in court. He’s also obliged to be truthful in general about facts in his personal life, such as not lying on his tax return (the sort of thing that can get lawyers disbarred).

But he’s not obliged to tell his spouse his honest opinion whether that new outfit makes them look fat. Likewise, Rosenstein knows his opinion on public policy doesn’t fall into this category. He can say with impunity that either global warming doesn’t exist, or that it’ll cause a biblical deluge within 5 years. Both are factually untrue, but it’s not going to get him fired.

And this particular claim is also exaggerated bunk. While everyone agrees encryption makes law enforcement’s job harder than with backdoors, nobody honestly believes it can “disable” law enforcement. While everyone agrees that encryption helps terrorists, nobody believes it can enable them to act with “impunity”.

I feel bad here. It’s a terrible thing to question your opponent’s character this way. But Rosenstein made this unavoidable when he clearly, with no ambiguity, put his integrity as Deputy Attorney General on the line behind the statement that “going dark threatens to disable law enforcement and enable criminals and terrorists to operate with impunity”. I feel it’s a bald face lie, but you don’t need to take my word for it. Read his own words yourself and judge his integrity.

Conclusion

Rosenstein’s speech includes repeated references to ideas like “oath”, “honor”, and “duty”. It reminds me of Col. Jessup’s speech in the movie “A Few Good Men”.

If you’ll recall, it was rousing speech, “you want me on that wall” and “you use words like honor as a punchline”. Of course, since he was violating his oath and sending two privates to death row in order to avoid being held accountable, it was Jessup himself who was crapping on the concepts of “honor”, “oath”, and “duty”.

And so is Rosenstein. He imagines himself on that wall, doing albeit terrible things, justified by his duty to protect citizens. He imagines that it’s he who is honorable, while the rest of us not, even has he utters bald faced lies to further his own power and authority.

We activists oppose crypto backdoors not because we lack honor, or because we are criminals, or because we support terrorists and child molesters. It’s because we value privacy and government officials who get corrupted by power. It’s not that we fear Trump becoming a dictator, it’s that we fear bureaucrats at Rosenstein’s level becoming drunk on authority — which Rosenstein demonstrably has. His speech is a long train of corrupt ideas pursuing the same object of despotism — a despotism we oppose.

In other words, we oppose crypto backdoors because it’s not a tool of law enforcement, but a tool of despotism.

The CoderDojo Girls Initiative

Post Syndicated from Nuala McHale original https://www.raspberrypi.org/blog/coderdojo-girls-initiative/

In March, the CoderDojo Foundation launched their Girls Initiative, which aims to increase the average proportion of girls attending CoderDojo clubs from 29% to at least 40% over the next three years.

The CoderDojo Girls Initiative

Six months on, we wanted to highlight what we’ve done so far and what’s next for our initiative.

What we’ve done so far

To date, we have focussed our efforts on four key areas:

  • Developing and improving content
  • Conducting and learning from research
  • Highlighting role models
  • Developing a guide of tried and tested best practices for encouraging and sustaining girls in a Dojo setting (Empowering the Future)

Content

We’ve taken measures to ensure our resources are as friendly to girls as well as boys, and we are improving them based on feedback from girls. For example, we have developed beginner-level content (Sushi Cards) for working with wearables and for building apps using App Inventor. In response to girls’ feedback, we are exploring more creative goal-orientated content.

The CoderDojo Girls Initiative

Moreover, as part of our Empowering the Future guide, we have developed three short ‘Mini-Sushi’ projects which provide a taster of different programming languages, such as Scratch, HTML, and App Inventor.

What’s next?

We are currently finalising our intermediate-level wearables Sushi Cards. These are resources for learners to further explore wearables and integrate them with other coding skills they are developing. The Cards will enable young people to program LEDs which can be sewn into clothing with conductive thread. We are also planning another series of Sushi Cards focused on using coding skills to solve problems Ninjas have reported as important to them.

Research

In June 2017 we conducted the first Ninja survey. It was sent to all young people registered on the CoderDojo community platform, Zen. Hundreds of young people involved in Dojos around the world responded and shared their experiences.

The CoderDojo Girls Initiative

We are currently examining these results to identify areas in which girls feel most or least confident, as well as the motivations and influencing factors that cause them to continue with coding.

What’s next?

Over the coming months we will delve deeper into the findings of this research, and decide how we can improve our content and Dojo support to adapt accordingly. Additionally, as part of sending out our Empowering the Future guide, we’re asking Dojos to provide insights into their current proportions of girls and female Mentors.

The CoderDojo Girls Initiative

We will follow up with recipients of the guide to document the impact of the recommended approaches they try at their Dojo. Thus, we will find out which approaches are most effective in different regional contexts, which will help us improve our support for Dojos wanting to increase their proportion of attending girls.

Role models

Many Dojos, Champions, and Mentors are doing amazing work to support and encourage girls at their Dojos. Female Mentors not only help by supporting attending girls, but they also act as vital role models in an environment which is often male-dominated. Blogs by female Mentors and Ninjas which have already featured on our website include:

What’s next?

We recognise the importance of female role models, and over the coming months we will continue to encourage community members to share their stories so that we bring them to the wider CoderDojo community. Do you know a female Mentor or Ninja you would like to shine a spotline on? Get in touch with us at [email protected] You can also use #CoderDojoGirls on social media.

The CoderDojo Girls Initiative

Empowering the Future guide

Ahead of Ada Lovelace Day and International Day of the Girl Child, the CoderDojo Foundation has released Empowering the Future, a comprehensive guide of practical approaches which Dojos have tested to engage and sustain girls.

Some topics covered in the guide are:

  • Approaches to improve the Dojo environment and layout
  • Language and images used to describe and promote Dojos
  • Content considerations, and suggested resources
  • The importance of female Mentors, and ways to increase access to role models

For the next month, Dojos that want to improve their proportion of girls can still sign up to have the guide book sent to them for free! From today, Dojos and anyone else can also download a PDF file of the guide.

The CoderDojo Girls Initiative

We would like to say a massive thank you to all community members who have shared their insights with us to make our Empowering the Future guide as comprehensive and beneficial as possible for other Dojos.

Tell us what you think

Have you found an approach, or used content, which girls find particularly engaging? Do you have questions about our Girls Initiative? We would love to hear your ideas, insights, and experiences in relation to supporting CoderDojo girls! Feel free to use our forums to share with the global CoderDojo community, and email us at [email protected]

The post The CoderDojo Girls Initiative appeared first on Raspberry Pi.

Raspberry Coulis’s night vision camera

Post Syndicated from Rachel Churcher original https://www.raspberrypi.org/blog/night-vision-camera/

We’ve all been there: zombies at the door, Daleks on the driveway, creatures from the Upside Down in the walls. You want to be able to monitor their movements, but how do you do that without attracting their attention? Wesley Archer (AKA Raspberry Coulis) has the answer: a Pi-powered Night Vision Camera, perfect for catching unearthly creatures on the prowl — and for wildlife spotting, birdwatching, and home security too, I guess…

Wesley's Pi-powered Night Vision Camera

Wesley’s Pi-powered Night Vision Camera

Black box

To build his Night Vision Camera, Wesley ordered an infrared Lisiparoi LED Light Ring, a Cyntech Raspberry Pi case, a Pi NoIR Camera Module, and USB WiFi adapter. He based this project around a Raspberry Pi Model B that was in need of a good home, but there’s no reason why you couldn’t use a more up-to-date model with built-in wireless networking.

Wesley set about adapting the case to hold the camera and the infrared light, cutting a hole for the camera lens and attaching the LED Light Ring to the front of the box. The light ring acts as an infrared floodlight, invisible to the zombies, but when combined with the NoIR Camera Module, capable of capturing useful images in the dark. His blog includes helpful step-by-step instructions for this process — unlike Wesley, we recommend using a metal file from your toolbox, and not a nail file belonging to another member of your family. If you’re surrounded by the undead hordes, domestic harmony is essential.

Cyntech Raspberry Pi case with a hole for the Camera Module - Night Vision Camera

Cutting holes in the case. Please don’t use your spouse’s nail file for this job …

Monitoring

When your hardware is complete, it’s time to install the software. Wesley chose MotionEyeOS to run his camera, and his blog explains the process of downloading and installing the software on your Pi. When everything is set up, and the Pi is connected to your WiFi network, all you need is the Pi’s IP address to view the feed from the camera. Type the IP address into a browser on the same WiFi network, log in, and you’ll soon be spotting intruders (supernatural or otherwise), or possibly watching the fluffy residents of your bird box. Whatever makes you happy.

Visibility

While a camera with night vision is obviously useful, both the Lisiparoi Light Ring and the Camera Module are available for use with the visible spectrum. You can order the Light Ring with infrared or standard white LEDs, and the standard Camera Module works with visible light. If you don’t mind attracting attention, both options could be used to monitor your perimeter for threats.

Saving the world

We think this project would be an amazing inspiration for the current Pioneers challenge, Only you can save us! Set up a camera to control entry to your secret bunker, even when the lights fail. Fend off attacks from zombies, Daleks, or giant spiders, and help save humanity from catastrophe!

Cute knitted zombies dancing - Night Vision Camera

Save yourselves!

Have you built a security system to keep your property safe from marauding zombies? Or even from regular burglars? Has your Pioneers team used infrared monitoring in your build? Tell us about it in the comments!

The post Raspberry Coulis’s night vision camera appeared first on Raspberry Pi.

Spotify Threatened Researchers Who Revealed ‘Pirate’ History

Post Syndicated from Andy original https://torrentfreak.com/spotify-threatened-researchers-who-revealed-pirate-history-171006/

As one of the members of Sweden’s infamous Piratbyrån (Piracy Bureau), Rasmus Fleischer was also one of early key figures at The Pirate Bay. Over the years he’s been a writer, researcher, debater, and musician, and in 2012 he finished his PhD thesis on “music’s political economy.”

As part of a five-person research team (Pelle Snickars, Patrick Vonderau, Anna Johansson, Rasmus Fleischer, Maria Eriksson) funded by the Swedish Research Council, Fleischer has co-written a book about the history of Spotify.

Titled ‘Spotify Teardown – Inside the Black Box of Streaming Music’, the publication is set to shine light on the history of the now famous music service while revealing quite a few past secrets.

With its release scheduled for 2018, Fleischer has already teased a few interesting nuggets, not least that Spotify’s early beta version used ‘pirate’ MP3 files, some of them sourced from The Pirate Bay.

Fleischer says that following an interview earlier this year with DI.se, in which he revealed that Spotify distributed unlicensed music between May 2007 to October 2008, Spotify looked at ways to try and stop his team’s research. However, the ‘pirate’ angle wasn’t the clear target, another facet of the team’s research was.

“Building on the tradition of ‘breaching experiments’ in ethnomethodology, the research group sought to break into the hidden infrastructures of digital music distribution in order to study its underlying norms and structures,” project leader Pelle Snickars previously revealed.

With this goal, the team conducted experiments to see if the system was open to abuse or could be manipulated, as Fleischer now explains.

“For example, some hundreds of robot users were created to study whether the same listening behavior results in different recommendations depending on whether the user was registered as male or female,” he says.

“We have also investigated on a small scale the possibilities of manipulating the system. However, we have not collected any data about real users. Our proposed methods appeared several years ago in our research funding application, which was approved by the Swedish Research Council, which was already noted in 2013.”

Fleischer says that Spotify had been aware of the project for several years but it wasn’t until this year, after he spoke of Spotify’s past as a ‘pirate’ service, that pressure began to mount.

“On May 19, our project manager received a letter from Benjamin Helldén-Hegelund, a lawyer at Spotify. The timing was hardly a coincidence. Spotify demanded that we ‘confirm in writing’ that we had ‘ceased activities contrary to their Terms of Use’,” Fleischer reveals.

A corresponding letter to the Swedish Research Council detailed Spotify’s problems with the project.

“Spotify is particularly concerned about the information that has emerged regarding the research group’s methods in the project. The data indicate that the research team has deliberately taken action that is explicitly in violation of Spotify’s Terms of Use and by means of technical methods they sought to conceal these breaches of conditions,” the letter read.

“The research group has worked, among other things, to artificially increase the number of plays and manipulate Spotify’s services using scripts or other automated processes.

“Spotify assumes that the systematic breach of its conditions has not been known to the Swedish Research Council and is convinced that the Swedish Research Council is convinced that the research undertaken with the support of the Swedish Research Council in all respects meets ethical guidelines and is carried out reasonably and in accordance with applicable law.”

Fleischer admits that part of the research was concerned with the possibility of artificially increasing the number of plays, but he says that was carried out on a small scale without any commercial gain.

“The purpose was simply to test if it is true that Spotify could be manipulated on a larger scale, as claimed by journalists who did similar experiments. It is also true that we ‘sought to hide these crimes’ by using a VPN connection,” he says.

Fleischer says that Spotify’s lawyer blended complaints together, such as correlating terms of service violations with violation of research ethics, while presenting the same as grounds for legal action.

“The argument was quite ridiculous. Nevertheless, the letter could not be interpreted as anything other than an attempt by Spotify to prevent us from pursuing the research project,” he notes.

This week, however, it appears the dispute has reached some kind of conclusion. In a posting on his Copyriot blog (Swedish), Fleischer reveals that Spotify has informed the Swedish Research Council that the case has been closed, meaning that the research into the streaming service can continue.

“It must be acknowledged that Spotify’s threats have taken both time and power from the project. This seems to be the purpose when big companies go after researchers who they perceive as uncomfortable. It may not be possible to stop the research but it can be delayed,” Fleischer says.

“Sure [Spotify] dislikes people being reminded of how the service started as a pirate service. But instead of inviting an open dialogue, lawyers are sent out for the purpose of slowing down researchers.”

Spotify Teardown. Inside the Black Box of Streaming Music is to be published by MIT Press in 2018.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

Microcell through a mobile hotspot

Post Syndicated from Robert Graham original http://blog.erratasec.com/2017/10/microcell-through-mobile-hotspot.html

I accidentally acquired a tree farm 20 minutes outside of town. For utilities, it gets electricity and basic phone. It doesn’t get water, sewer, cable, or DSL (i.e. no Internet). Also, it doesn’t really get cell phone service. While you can get SMS messages up there, you usually can’t get a call connected, or hold a conversation if it does.

We have found a solution — an evil solution. We connect an AT&T “Microcell“, which provides home cell phone service through your Internet connection, to an AT&T Mobile Hotspot, which provides an Internet connection through your cell phone service.

Now, you may be laughing at this, because it’s a circular connection. It’s like trying to make a sailboat go by blowing on the sails, or lifting up a barrel to lighten the load in the boat.

But it actually works.

Since we get some, but not enough, cellular signal, we setup a mast 20 feet high with a directional antenna pointed to the cell tower 7.5 miles to the southwest, connected to a signal amplifier. It’s still an imperfect solution, as we are still getting terrain distortions in the signal, but it provides a good enough signal-to-noise ratio to get a solid connection.

We then connect that directional antenna directly to a high-end Mobile Hotspot. This gives us a solid 2mbps connection with a latency under 30milliseconds. This is far lower than the 50mbps you can get right next to a 4G/LTE tower, but it’s still pretty good for our purposes.

We then connect the AT&T Microcell to the Mobile Hotspot, via WiFi.

To avoid the circular connection, we lock the frequencies for the Mobile Hotspot to 4G/LTE, and to 3G for the Microcell. This prevents the Mobile Hotspot locking onto the strong 3G signal from the Microcell. It also prevents the two from causing noise to the other.

This works really great. We now get a strong cell signal on our phones even 400 feet from the house through some trees. We can be all over the property, out in the lake, down by the garden, and so on, and have our phones work as normal. It’s only AT&T, but that’s what the whole family uses.

You might be asking why we didn’t just use a normal signal amplifier, like they use on corporate campus. It boosts all the analog frequencies, making any cell phone service works.

We’ve tried this, and it works a bit, allowing cell phones to work inside the house pretty well. But they don’t work outside the house, which is where we spend a lot of time. In addition, while our newer phones work, my sister’s iPhone 5 doesn’t. We have no idea what’s going on. Presumably, we could hire professional installers and stuff to get everything working, but nobody would quote us a price lower than $25,000 to even come look at the property.

Another possible solution is satellite Internet. There are two satellites in orbit that cover the United States with small “spot beams” delivering high-speed service (25mbps downloads). However, the latency is 500milliseconds, which makes it impractical for low-latency applications like phone calls.

While I know a lot about the technology in theory, I find myself hopelessly clueless in practice. I’ve been playing with SDR (“software defined radio”) to try to figure out exactly where to locate and point the directional antenna, but I’m not sure I’ve come up with anything useful. In casual tests, it seems rotating the antenna from vertical to horizontal increases the signal-to-noise ratio a bit, which seems counter intuitive, and should not happen. So I’m completely lost.

Anyway, I thought I’d write this up as a blogpost, in case anybody has better suggestion. Or, instead of signals, suggestions to get wired connectivity. Properties a half mile away get DSL, I wish I knew who to talk to at the local phone company to pay them money to extend Internet to our property.

Phone works in all this area now

20th Century Fox is Looking for Anti-Piracy Interns

Post Syndicated from Ernesto original https://torrentfreak.com/20th-century-fox-is-looking-for-anti-piracy-interns-170930/

Piracy remains one of the key threats for most Hollywood movie studios.

Most companies have entire departments dedicated to spotting infringing content, understanding the changing landscape, and figuring out how to respond.

20th Century Fox, for example, has its own Content Protection group, headed by Ron Wheeler. The group keeps an eye on emerging piracy threats and is currently looking for fresh blood.

The company has listed two new internships. The first is for a Graduate JD Law Student, who will be tasked with analyzing fair use cases and finding new targets for lawsuits, among other things.

“Interns will participate in the monitoring of and enforcement against such piracy, including conducting detailed copyright infringement and fair use analyses; identifying and researching litigation targets, and searching the internet for infringing copies of Fox content.”

Fox notes that basic knowledge of the principles of Copyright Law is a plus, but apparently not required. Students who take this internship will learn how film and television piracy affects the media industry and consumers, preparing them for future work in this field.

“This is a great opportunity for students interested in pursuing practice in the fields of Intellectual Property, Entertainment, or Media Law,” the job application explains.

A second anti-piracy internship that was posted recently is a search and analytics position. This includes organizing online copyright infringement intelligence and compiling this in analytical piracy reports for Fox executives.

Undergraduate – Research & Analytics

The research job posting shows that Fox keeps an eye on a wide range of piracy avenues including search engines, forums, eBay and pirate sites.

“Anti-Piracy Internet Investigations and Analysis including, but not limited to, internet research, forum site investigation, eBay searches, video forensics analysis review, database entry, general internet searches for Fox video content, review and summarize pirate websites, piracy trend analysis, and more.”

Those who complete the internship will have a thorough understanding of how widespread piracy issues are. It will provide insight into how this affects the movie industry and consumers alike, Fox explains.

While the average torrenter and streaming pirate might not be very eager to work for ‘the other side,’ these internships are ideal positions for students who have aspirations of working in the anti-piracy field. If any TorrentFreak readers plan to apply and get the job, we’ll be eager to hear what you’ve learned in a few months.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

Creating a Cost-Efficient Amazon ECS Cluster for Scheduled Tasks

Post Syndicated from Nathan Taber original https://aws.amazon.com/blogs/compute/creating-a-cost-efficient-amazon-ecs-cluster-for-scheduled-tasks/

Madhuri Peri
Sr. DevOps Consultant

When you use Amazon Relational Database Service (Amazon RDS), depending on the logging levels on the RDS instances and the volume of transactions, you could generate a lot of log data. To ensure that everything is running smoothly, many customers search for log error patterns using different log aggregation and visualization systems, such as Amazon Elasticsearch Service, Splunk, or other tool of their choice. A module needs to periodically retrieve the RDS logs using the SDK, and then send them to Amazon S3. From there, you can stream them to your log aggregation tool.

One option is writing an AWS Lambda function to retrieve the log files. However, because of the time that this function needs to execute, depending on the volume of log files retrieved and transferred, it is possible that Lambda could time out on many instances.  Another approach is launching an Amazon EC2 instance that runs this job periodically. However, this would require you to run an EC2 instance continuously, not an optimal use of time or money.

Using the new Amazon CloudWatch integration with Amazon EC2 Container Service, you can trigger this job to run in a container on an existing Amazon ECS cluster. Additionally, this would allow you to improve costs by running containers on a fleet of Spot Instances.

In this post, I will show you how to use the new scheduled tasks (cron) feature in Amazon ECS and launch tasks using CloudWatch events, while leveraging Spot Fleet to maximize availability and cost optimization for containerized workloads.

Architecture

The following diagram shows how the various components described schedule a task that retrieves log files from Amazon RDS database instances, and deposits the logs into an S3 bucket.

Amazon ECS cluster container instances are using Spot Fleet, which is a perfect match for the workload that needs to run when it can. This improves cluster costs.

The task definition defines which Docker image to retrieve from the Amazon EC2 Container Registry (Amazon ECR) repository and run on the Amazon ECS cluster.

The container image has Python code functions to make AWS API calls using boto3. It iterates over the RDS database instances, retrieves the logs, and deposits them in the S3 bucket. Many customers choose these logs to be delivered to their centralized log-store. CloudWatch Events defines the schedule for when the container task has to be launched.

Walkthrough

To provide the basic framework, we have built an AWS CloudFormation template that creates the following resources:

  • Amazon ECR repository for storing the Docker image to be used in the task definition
  • S3 bucket that holds the transferred logs
  • Task definition, with image name and S3 bucket as environment variables provided via input parameter
  • CloudWatch Events rule
  • Amazon ECS cluster
  • Amazon ECS container instances using Spot Fleet
  • IAM roles required for the container instance profiles

Before you begin

Ensure that Git, Docker, and the AWS CLI are installed on your computer.

In your AWS account, instantiate one Amazon Aurora instance using the console. For more information, see Creating an Amazon Aurora DB Cluster.

Implementation Steps

  1. Clone the code from GitHub that performs RDS API calls to retrieve the log files.
    git clone https://github.com/awslabs/aws-ecs-scheduled-tasks.git
  2. Build and tag the image.
    cd aws-ecs-scheduled-tasks/container-code/src && ls

    Dockerfile		rdslogsshipper.py	requirements.txt

    docker build -t rdslogsshipper .

    Sending build context to Docker daemon 9.728 kB
Step 1 : FROM python:3
 ---> 41397f4f2887
Step 2 : WORKDIR /usr/src/app
 ---> Using cache
 ---> 59299c020e7e
Step 3 : COPY requirements.txt ./
 ---> 8c017e931c3b
Removing intermediate container df09e1bed9f2
Step 4 : COPY rdslogsshipper.py /usr/src/app
 ---> 099a49ca4325
Removing intermediate container 1b1da24a6699
Step 5 : RUN pip install --no-cache-dir -r requirements.txt
 ---> Running in 3ed98b30901d
Collecting boto3 (from -r requirements.txt (line 1))
  Downloading boto3-1.4.6-py2.py3-none-any.whl (128kB)
Collecting botocore (from -r requirements.txt (line 2))
  Downloading botocore-1.6.7-py2.py3-none-any.whl (3.6MB)
Collecting s3transfer<0.2.0,>=0.1.10 (from boto3->-r requirements.txt (line 1))
  Downloading s3transfer-0.1.10-py2.py3-none-any.whl (54kB)
Collecting jmespath<1.0.0,>=0.7.1 (from boto3->-r requirements.txt (line 1))
  Downloading jmespath-0.9.3-py2.py3-none-any.whl
Collecting python-dateutil<3.0.0,>=2.1 (from botocore->-r requirements.txt (line 2))
  Downloading python_dateutil-2.6.1-py2.py3-none-any.whl (194kB)
Collecting docutils>=0.10 (from botocore->-r requirements.txt (line 2))
  Downloading docutils-0.14-py3-none-any.whl (543kB)
Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1->botocore->-r requirements.txt (line 2))
  Downloading six-1.10.0-py2.py3-none-any.whl
Installing collected packages: six, python-dateutil, docutils, jmespath, botocore, s3transfer, boto3
Successfully installed boto3-1.4.6 botocore-1.6.7 docutils-0.14 jmespath-0.9.3 python-dateutil-2.6.1 s3transfer-0.1.10 six-1.10.0
 ---> f892d3cb7383
Removing intermediate container 3ed98b30901d
Step 6 : COPY . .
 ---> ea7550c04fea
Removing intermediate container b558b3ebd406
Successfully built ea7550c04fea
  3. Run the CloudFormation stack and get the names for the Amazon ECR repo and S3 bucket. In the stack, choose Outputs.
  4. Open the ECS console and choose Repositories. The rdslogs repo has been created. Choose View Push Commands and follow the instructions to connect to the repository and push the image for the code that you built in Step 2. The screenshot shows the final result:
  5. Associate the CloudWatch scheduled task with the created Amazon ECS Task Definition, using a new CloudWatch event rule that is scheduled to run at intervals. The following rule is scheduled to run every 15 minutes:
    aws --profile default --region us-west-2 events put-rule --name demo-ecs-task-rule  --schedule-expression "rate(15 minutes)"

    {
    "RuleArn": "arn:aws:events:us-west-2:12345678901:rule/demo-ecs-task-rule"
}
  6. CloudWatch requires IAM permissions to place a task on the Amazon ECS cluster when the CloudWatch event rule is executed, in addition to an IAM role that can be assumed by CloudWatch Events. This is done in three steps:
    1. Create the IAM role to be assumed by CloudWatch.
      aws --profile default --region us-west-2 iam create-role --role-name Test-Role --assume-role-policy-document file://event-role.json

      {
    "Role": {
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17", 
            "Statement": [
                {
                    "Action": "sts:AssumeRole", 
                    "Effect": "Allow", 
                    "Principal": {
                        "Service": "events.amazonaws.com"
                    }
                }
            ]
        }, 
        "RoleId": "AROAIRYYLDCVZCUACT7FS", 
        "CreateDate": "2017-07-14T22:44:52.627Z", 
        "RoleName": "Test-Role", 
        "Path": "/", 
        "Arn": "arn:aws:iam::12345678901:role/Test-Role"
    }
}

      The following is an example of the event-role.json file used earlier:

      {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
              "Service": "events.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}
    2. Create the IAM policy defining the ECS cluster and task definition. You need to get these values from the CloudFormation outputs and resources.
      aws --profile default --region us-west-2 iam create-policy --policy-name test-policy --policy-document file://event-policy.json

      {
    "Policy": {
        "PolicyName": "test-policy", 
        "CreateDate": "2017-07-14T22:51:20.293Z", 
        "AttachmentCount": 0, 
        "IsAttachable": true, 
        "PolicyId": "ANPAI7XDIQOLTBUMDWGJW", 
        "DefaultVersionId": "v1", 
        "Path": "/", 
        "Arn": "arn:aws:iam::123455678901:policy/test-policy", 
        "UpdateDate": "2017-07-14T22:51:20.293Z"
    }
}

      The following is an example of the event-policy.json file used earlier:

      {
    "Version": "2012-10-17",
    "Statement": [
      {
          "Effect": "Allow",
          "Action": [
              "ecs:RunTask"
          ],
          "Resource": [
              "arn:aws:ecs:*::task-definition/"
          ],
          "Condition": {
              "ArnLike": {
                  "ecs:cluster": "arn:aws:ecs:*::cluster/"
              }
          }
      }
    ]
}
    3. Attach the IAM policy to the role.
      aws --profile default --region us-west-2 iam attach-role-policy --role-name Test-Role --policy-arn arn:aws:iam::1234567890:policy/test-policy
  7. Associate the CloudWatch rule created earlier to place the task on the ECS cluster. The following command shows an example. Replace the AWS account ID and region with your settings.
    aws events put-targets --rule demo-ecs-task-rule --targets "Id"="1","Arn"="arn:aws:ecs:us-west-2:12345678901:cluster/test-cwe-blog-ecsCluster-15HJFWCH4SP67","EcsParameters"={"TaskDefinitionArn"="arn:aws:ecs:us-west-2:12345678901:task-definition/test-cwe-blog-taskdef:8"},"RoleArn"="arn:aws:iam::12345678901:role/Test-Role"

    {
    "FailedEntries": [], 
    "FailedEntryCount": 0
}

That’s it. The logs now run based on the defined schedule.

To test this, open the Amazon ECS console, select the Amazon ECS cluster that you created, and then choose Tasks, Run New Task. Select the task definition created by the CloudFormation template, and the cluster should be selected automatically. As this runs, the S3 bucket should be populated with the RDS logs for the instance.

Conclusion

In this post, you’ve seen that the choices for workloads that need to run at a scheduled time include Lambda with CloudWatch events or EC2 with cron. However, sometimes the job could run outside of Lambda execution time limits or be not cost-effective for an EC2 instance.

In such cases, you can schedule the tasks on an ECS cluster using CloudWatch rules. In addition, you can use a Spot Fleet cluster with Amazon ECS for cost-conscious workloads that do not have hard requirements on execution time or instance availability in the Spot Fleet. For more information, see Powering your Amazon ECS Cluster with Amazon EC2 Spot Instances and Scheduled Events.

If you have questions or suggestions, please comment below.

Natural Language Processing at Clemson University – 1.1 Million vCPUs & EC2 Spot Instances

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/natural-language-processing-at-clemson-university-1-1-million-vcpus-ec2-spot-instances/

My colleague Sanjay Padhi shared the guest post below in order to recognize an important milestone in the use of EC2 Spot Instances.

— Jeff;

A group of researchers from Clemson University achieved a remarkable milestone while studying topic modeling, an important component of machine learning associated with natural language processing, breaking the record for creating the largest high-performance cluster by using more than 1,100,000 vCPUs on Amazon EC2 Spot Instances running in a single AWS region. The researchers conducted nearly half a million topic modeling experiments to study how human language is processed by computers. Topic modeling helps in discovering the underlying themes that are present across a collection of documents. Topic models are important because they are used to forecast business trends and help in making policy or funding decisions. These topic models can be run with many different parameters and the goal of the experiments is to explore how these parameters affect the model outputs.

The Experiment
Professor Amy Apon, Co-Director of the Complex Systems, Analytics and Visualization Institute at Clemson University with Professor Alexander Herzog and graduate students Brandon Posey and Christopher Gropp in collaboration with members of the AWS team as well as AWS Partner Omnibond performed the experiments.  They used software infrastructure based on CloudyCluster that provisions high performance computing clusters on dynamically allocated AWS resources using Amazon EC2 Spot Fleet. Spot Fleet is a collection of biddable spot instances in EC2 responsible for maintaining a target capacity specified during the request. The SLURM scheduler was used as an overlay virtual workload manager for the data analytics workflows. The team developed additional provisioning and workflow automation software as shown below for the design and orchestration of the experiments. This setup allowed them to evaluate various topic models on different data sets with massively parallel parameter sweeps on dynamically allocated AWS resources. This framework can easily be used beyond the current study for other scientific applications that use parallel computing.

Ramping to 1.1 Million vCPUs
The figure below shows elastic, automatic expansion of resources as a function of time, in the US East (Northern Virginia) Region. At just after 21:40 (GMT-1) on Aug. 26, 2017, the number of vCPUs utilized was 1,119,196. Clemson researchers also took advantage of the new per-second billing for the EC2 instances that they launched. The vCPU count usage is comparable to the core count on the largest supercomputers in the world.

Here’s the breakdown of the EC2 instance types that they used:

Campus resources at Clemson funded by the National Science Foundation were used to determine an effective configuration for the AWS experiments as compared to campus resources, and the AWS cloud resources complement the campus resources for large-scale experiments.

Meet the Team
Here’s the team that ran the experiment (Professor Alexander Herzog, graduate students Christopher Gropp and Brandon Posey, and Professor Amy Apon):

Professor Apon said about the experiment:

I am absolutely thrilled with the outcome of this experiment. The graduate students on the project are amazing. They used resources from AWS and Omnibond and developed a new software infrastructure to perform research at a scale and time-to-completion not possible with only campus resources. Per-second billing was a key enabler of these experiments.

Boyd Wilson (CEO, Omnibond, member of the AWS Partner Network) told me:

Participating in this project was exciting, seeing how the Clemson team developed a provisioning and workflow automation tool that tied into CloudyCluster to build a huge Spot Fleet supercomputer in a single region in AWS was outstanding.

About the Experiment
The experiments test parameter combinations on a range of topics and other parameters used in the topic model. The topic model outputs are stored in Amazon S3 and are currently being analyzed. The models have been applied to 17 years of computer science journal abstracts (533,560 documents and 32,551,540 words) and full text papers from the NIPS (Neural Information Processing Systems) Conference (2,484 documents and 3,280,697 words). This study allows the research team to systematically measure and analyze the impact of parameters and model selection on model convergence, topic composition and quality.

Looking Forward
This study constitutes an interaction between computer science, artificial intelligence, and high performance computing. Papers describing the full study are being submitted for peer-reviewed publication. I hope that you enjoyed this brief insight into the ways in which AWS is helping to break the boundaries in the frontiers of natural language processing!

— Sanjay Padhi, Ph.D, AWS Research and Technical Computing

 

Belgium Wants to Blacklist Pirate Sites & Hijack Their Traffic

Post Syndicated from Andy original https://torrentfreak.com/belgium-wants-to-blacklist-pirate-sites-hijack-their-traffic-170924/

The thorny issue of how to deal with the online piracy phenomenon used to be focused on punishing site users. Over time, enforcement action progressed to the services themselves, until they became both too resilient and prevalent to tackle effectively.

In Europe in particular, there’s now a trend of isolating torrent, streaming, and hosting platforms from their users. This is mainly achieved by website blocking carried out by local ISPs following an appropriate court order.

While the UK is perhaps best known for this kind of action, Belgium was one of the early pioneers of the practice.

After filing a lawsuit in 2010, the Belgian Anti-Piracy Foundation (BAF) weathered an early defeat at the Antwerp Commercial Court to achieve success at the Court of Appeal. Since then, local ISPs have been forced to block The Pirate Bay.

Since then there have been several efforts (1,2) to block more sites but rightsholders have complained that the process is too costly, lengthy, and cumbersome. Now the government is stepping in to do something about it.

Local media reports that Deputy Prime Minister Kris Peeters has drafted new proposals to tackle online piracy. In his role as Minister of Economy and Employment, Peeters sees authorities urgently tackling pirate sites with a range of new measures.

For starters, he wants to create a new department, formed within the FPS Economy, to oversee the fight against online infringement. The department would be tasked with detecting pirate sites more quickly and rendering them inaccessible in Belgium, along with any associated mirror sites or proxies.

Peeters wants the new department to add all blocked sites to a national ‘pirate blacklist. Interestingly, when Internet users try to access any of these sites, he wants them to be automatically diverted to legal sites where a fee will have to be paid for content.

While it’s not unusual to try and direct users away from pirate sites, for the most part Internet service providers have been somewhat reluctant to divert subscribers to commercial sites. Their assistance would be needed in this respect, so it will be interesting to see how negotiations pan out.

The Belgian Entertainment Association (BEA), which was formed nine years ago to represent the music, video, software and videogame industries, welcomed Peeters’ plans.

“It’s so important to close the doors to illegal download sites and to actively lead people to legal alternatives,” said chairman Olivier Maeterlinck.

“Surfers should not forget that the motives of illegal download sites are not always obvious. These sites also regularly try to exploit personal data.”

The current narrative that pirate sites are evil places is clearly gaining momentum among anti-piracy bodies, but there’s little sign that the public intends to boycott sites as a result. With that in mind, alternative legal action will still be required.

With that in mind, Peeters wants to streamline the system so that all piracy cases go through a single court, the Commercial Court of Brussels. This should reduce costs versus the existing model and there’s also the potential for more consistent rulings.

“It’s a good idea to have a clearer legal framework on this,” says Maeterlinck from BEA.

“There are plenty of legal platforms, streaming services like Spotify, for example, which are constantly developing and reaching an ever-increasing audience. Those businesses have a business model that ensure that the creators of certain media content are properly compensated. The rotten apples must be tackled, and those procedures should be less time-consuming.”

There’s little doubt that BEA could benefit from a little government assistance. Back in February, the group filed a lawsuit at the French commercial court in Brussels, asking ISPs to block subscriber access to several ‘pirate’ sites.

“Our action aims to block nine of the most popular streaming sites which offer copyright-protected content on a massive scale and without authorization,” Maeterlinck told TF at the time.

“In accordance with the principles established by the CJEU (UPC Telekabel and GS Media), BEA seeks a court order confirming the infringement and imposing site blocking measures on the ISPs, who are content providers as well.”

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

Query and Visualize AWS Cost and Usage Data Using Amazon Athena and Amazon QuickSight

Post Syndicated from Androski Spicer original https://aws.amazon.com/blogs/big-data/query-and-visualize-aws-cost-and-usage-data-using-amazon-athena-and-amazon-quicksight/

If you’ve ever wondered if a serverless alternative existed for consuming and querying your AWS Cost and Usage report data, then wonder no more. The answer is yes, and this post both introduces you to that solution and illustrates the simplicity and effortlessness of deploying it.

This solution leverages AWS serverless technologies Amazon Athena, AWS Lambda, AWS CloudFormation, and Amazon S3. But it doesn’t stop there, as you can also use Amazon QuickSight, a Serverless cloud-powered business analytics service, to build visualizations and perform ad-hoc analysis of your sanitized AWS Cost and Usage report data.

Amazon Athena, what’s that?

Amazon Athena is an interactive query service that makes it easy to analyze data directly in Amazon S3 using standard SQL. It’s a Serverless platform in which there is no need to set up or manage infrastructure. Athena scales automatically—executing queries in parallel—so results are fast, even with large datasets and complex queries.

Athena exposes several API operations that allow developers to automate running queries or using services like Lambda to trigger queries in response to events in other services like S3. This solution takes advantage of these abilities and allows you to focus on running the SQL queries that yield the results you are looking for. This solution builds the components in Athena that are needed for you to run these queries, for example, building and maintaining a database and corresponding table.

So, what’s the solution?

Today, the Billing and Cost Management service writes your AWS Cost and Usage report to an S3 bucket that you designated during the time of creation. You have the option to have AWS write these files as a GZIP or ZIP file on a schedule basis. This schedule can either be hourly or daily.

The CloudFormation template that accompanies this blog post builds a Serverless environment that contains a Lambda function that takes a CUR file, unzips it in memory, removes the header row and writes the modified report to an S3 bucket.  The Lambda function writes this file into an S3 Bucket with a directory structure of “year=current-year” and “month=current-month”. For example, if a file is written for June 1st, 2017 then the Lambda function writes the file in the folder path “bucket-name/year=2017/month=06/file_name”. The S3 bucket in which the Lambda function creates the aforementioned directory structure is constructed at the time the environment is built by the CloudFormation template.

The following diagram provides an example of what you should see in the AWS Management Console after your Lambda function runs.

You might be wondering why the Lambda function writes the files in a directory structure like the following.

The Lambda function writes data in this folder structure to allow you to optimize for query performance and cost when executing queries in Athena against your transformed AWS Cost and Usage report billing data stored in S3. For example, if you wanted to run a query to retrieve all billing data for the month of June, you could run the following query:

SELECT * FROM aws_billing_report.my_cur_report where month=’06’;

This query would only scan the folder path month=06 and return the files within that folder. This means that you only scan the data that you need, which is a smaller subset of your overall stored billing data/reports. Given that Athena only charges you for the amount data scanned (which is $5 per TB), partitioning your data allows you to reduce the cost of querying data using Athena.

The following diagram shows what that query looks like in Athena.

What happens after the Lambda function extracts, transforms, and re-writes this file? After writing the newly transformed file, the Lambda function checks Athena to see if the database “aws_billing_report” exists. If not, then the function creates it. After it’s created, the function then creates the table “my_cur_report” that is partitioned by “year” and “month”. After this table is created, the Lambda function takes the folder structure that you created in your S3 bucket and adds that folder structure to your Athena metadata store as database partitions.

Note:  Athena is not a database; it simply projects the schema that you define for your table on top of the data stored in S3. When you delete a table or database, you are deleting the metadata for the table or database in Athena. You can’t use Athena to delete your data stored in S3. For more information, see the Amazon Athena User Guide.

Things to note

This solution only processes the current and previous month’s bills. This release does not process bills written before that. Today, this solution only supports Gzipped files that are 80 MB and under 1.5 GB uncompressed. Future releases of the Lambda functions for this solution will process zipped files and Cost and Usage reports from six months prior to the current date.

Athena queries

The following table provides a step-by-step guide as to what happens in your serverless environment from start to finish.

Setting up the AWS Cost and Usage report

There are two things you must know and two things you must do in order to launch this solution successfully. First, you must know both the name of your AWS Cost and Usage report, and the name of the S3 bucket in which the reports are currently stored.

If you already have your report set up, ensure that there is a manifest file written to the /date-path folder in your reports folder. The manifest file is important because it’s a map to where the latest reports are stored. For example, if your report name is “andro-cost-n-usage-report”, the name of your bucket is “my-report-bucket” and the current month is August (in the year 2017) then the path you should check is “my-report-bucket/ andro-cost-n-usage-report/20170801-20170901”.

If you don’t have a report set up, then follow the steps in Turning on the AWS Cost and Usage Report. For Include, choose Redshift Manifest. For Enable support for, choose Redshift.

Launching the CloudFormation template

Now for the things that you must do. First, given the name of your AWS Cost and Usage report, launch the CloudFormation template that builds all the serverless components to make running queries against your billing data effortless.

Choose the same AWS Region in which your S3 bucket is located, to which AWS writes your bills. Also, I have only included options for the regions in which Athena, Lambda, and Amazon QuickSight are currently available.

Northern Virginia   Ohio
   
Ireland   Oregon  

 

If your billing reports are stored in a different region, then create a new report and specify a bucket that is in one of the four regions listed.

Follow the instructions in the CloudFormation wizard, using the following options, and then choose Create.

  • For CostnUsageReport, type the name of your AWS Cost and Usage report.
  • For S3BucketName, type a unique name to be given to a new S3 bucket.
  • For s3CURBucket, type the name of the bucket in which your current reports are written.

While your stack is building, a page similar to the following is displayed.

When the Status column shows CREATE_COMPLETE, you have successfully created four new Lambda functions and a S3 bucket in which your transformed bills will be stored. The first time your Lambda functions run, two folders appear in the S3 bucket:

  • year=current_year
    Stores the transformed report.
  • aws-athena-query-results
    Stores the results of the SQL queries that you run in Athena.

Adding a Lambda trigger

After you have successfully built your CloudFormation stack, you create a Lambda trigger that points to the new S3 bucket. I recommend keeping this bucket dedicated to storing AWS Cost and Usage reports.

This trigger can be created by following the steps below.

  1. Open the Lambda console.
  2. Choose Functions, and select the “aws-cost-n-usage-main-lambda-fn-A” Lambda function (don’t choose the check box beside it).
  3. Choose Trigger, Add trigger. There should not be any existing triggers.
  4. For Trigger type (the box with dotted lines), choose S3.

  1. Select the S3 bucket that you just created. For Event type, choose Object Created (All) and check the Enable trigger Choose Submit.

Resources built, trigger created… now what?

Before moving on, I recommend checking to see that all the solution components were created.

In the Lambda console, check that the following five functions were created:

  • aws-cost-n-usage-S3-lambda-fn-B
  • aws-cost-n-usage-main-lambda-fn-A
  • aws-cost-n-usage-S3-lambda-fn-B-2
  • aws-cost-n-usage-Athena-lambda-fn-C
  • aws-cost-n-usage-Athena-lambda-fn-C-2

In the S3 console, check that the bucket was created.

Your database and table are not created until your function runs for the first time. Afterward, Athena holds your database and table.

After your Athena database and table are created, you can begin running your SQL queries. Athena uses Presto with ANSI SQL support and works with a variety of standard data formats. Start by running queries against your sanitized AWS Cost and Usage report dataset.

If you choose the eye icon beside any table name in the left navigation pane, Athena executes a SELECT * FROM [table_name] limit 10; query against your entire dataset.  Or you can run the following query:

SELECT from_iso8601_timestamp(lineitem_usagestartdate) AS lineitem_usagestartdate,
        from_iso8601_timestamp(lineitem_usageenddate) AS lineitem_usageenddate,
        product_instancetype,
        count(*) AS count
FROM aws_billing_report.my_cur_report
WHERE lineitem_productcode='AmazonEC2'and (lineitem_operation LIKE '%RunInstances%'
        OR lineitem_usagetype LIKE '%BoxUsage%')
        AND lineitem_usagetype NOT LIKE 'SpotUsage%'
        AND lineitem_usagetype NOT LIKE '%Out-Bytes%'
        AND lineitem_usagetype NOT LIKE '%In-Bytes%'
        AND lineitem_usagetype NOT LIKE '%DataTransfer%'
        AND pricing_term='OnDemand'
GROUP BY  lineitem_usagestartdate,lineitem_usageenddate,product_instancetype,lineitem_usagetype
ORDER BY  lineitem_usagestartdate, product_instancetype;

Executing this query converts the LineItem_UsageStartDate and LineItem_UsageEndDate values to UTC date format. It also returns a list of On-Demand Instances with a LineItem_Operation value of RunInstances or BoxUsage. The count column provides the results of a sum of RunInstances or BoxUsage requests on the specific instance type to the left.

The results are similar to the following screenshot:

You can run far more complex queries than those listed earlier. Another thing to know is, Athena stores query results in S3 automatically. Each query that you run has a results file in CSV format (*.csv) and a metadata file (*.csv.metadata) that includes header information such as column type, etc. If necessary, you can access the result files to work with them.

This solution creates a path in the S3 bucket, with a prefix path of “aws-athena-query-results”.

Testing

If you have successfully built this solution and added the trigger to the S3 bucket in which the Billing and Cost Management service writes your AWS Cost and Usage reports, then it might be time to conduct a simple test.

Remember, this solution processes the current and previous month billing data and it does it by using the master metadata file. This file acts as a map that tells you where the latest reports are stored.

  1. In the S3 path to which AWS writes your AWS Cost and Usage Billing reports, open the folder with your billing reports. There will be either a set of folders or a single folder with a date range naming format.

  1. Open the folder with the data range for the current month. In this folder, there is a metadata file that can be found at the bottom of the folder. It has a JSON extension and holds the S3 key for the latest report.
  2. Download the metadata file. Ensure that the name of the file on your machine is the same as the version stored on your S3 bucket.
  3. Upload the metadata file to the same S3 path from which you downloaded it. This triggers the Lambda function “aws-cost-n-usage-main-lmbda-fn-A”.
  4. In the S3 bucket that you created to hold your transform files, choose the “year=” folder and then the “month=” folder that corresponds to the current month. You should see the transformed file there, with the time stamp that indicated that it was just written.

How does the solution actually work?

The Billing and Cost Management service writes your latest report to a hashed folder inside your date range folder, as shown earlier. Each time the billing system writes your latest report, it writes it to a new hashed folder, leaving the previous report behind. The result is several hashed folders, each holding a historical view of your AWS Cost and Usage data. This is all noise, redundant data.

So that you can know where the latest data lives, AWS writes a master metadata file that contains the S3 keys and hashed folder for the latest report. This solution uses that master metadata file to ensure that the latest report is being processed, which helps sift through the multiple historical versions.

This is how the solution works. When a new report is written to the S3 bucket that is designated to store your report, S3 does a pull request for your main Lambda function. This function checks the key to ensure that it is indeed the master metadata file. If it is, then the main Lambda function decides whether the report being processed is for the current or previous month.

If it matches either, the main Lambda function calls functions B or B-2. Lambda function B processes the current month’s report, while B2 processes the previous month’s report. Separating the operation into multiple functions allows the solution to process a virtually unlimited number of reports at the same time. This is especially important when dealing with multiple reports that are gigabytes in size. The parallelization of this operation helps to overcome the time constraint of Lambda.

Lambda functions B and B2 stream your report, GUnzip each chunk of data, and remove unwanted rows that may cause an exception to be thrown when you execute a SQL query in Athena against this data. They then leverage the multipart upload capability of S3 to write the unzipped version of your file to the new S3 bucket that stores your transformed report.

During this event, Lambda function C creates your Athena database (IF NOT EXISTS) and uses the column title row in your report to create an Athena table.

Amazon QuickSight visualizations

After you have successfully built your Athena database, you have the option to integrate it with Amazon QuickSight, a BI tool/ fast business analytics service that allows you to build visualizations, perform ad hoc analysis, and quickly get business insights from your data. It seamlessly discovers AWS data sources, including but not limited to S3, Athena, Amazon Redshift, and Amazon RDS. Amazon QuickSight provides tools and filters that gives you the ability to dive deep into your dataset and pull out the data that satisfies your use case.

In this section, I walk through how to connect Amazon QuickSight to your Athena database and table to re-create the example visualization for your very own. If you are new to Amazon QuickSight, sign up for free at https://quicksight.aws.

In the following screenshot, Amazon QuickSight has been used to create several visualizations that provide the monthly total spend on AWS services, the daily spend, a monthly spend broken down by AWS service, and monthly spend by EC2 instance type. You have the ability to create visualizations like these and more.

To create reports like these, first specify how Amazon QuickSight should access your data by creating a dataset.

To get started, point Amazon QuickSight to your Athena database. Follow the steps in Creating a Data Set Using Amazon Athena Data. Type the name of your Athena database (aws_billing_report) and then the table (my_cur_report). To preview the data and apply additional transformations to your billing dataset, choose Edit/Preview data. Under tables, choose Switch to custom SQL tool, Query.

The AWS Cost and Usage report contains many columns, not all of which are necessary for your analysis and dashboard. Using a custom SQL query, you can choose which columns to use from within Amazon QuickSight. This is very useful, especially when working with Athena.

Athena uses Hive DDL to create tables, using a specified SerDe. This solution uses the OpenCSV SerDe, which means that all data returned by an Athena query is returned as “string”. The ability to run SQL queries in Amazon QuickSight before you create your visualizations provides the added advantage of converting/casting certain fields into the data type. The following SELECT statement is a perfect example, by casting bill_billingperiodstartdate from string to UTC timestamp, so that you can create a visualization that specifies a time period.

SELECT from_iso8601_timestamp(bill_billingperiodstartdate) AS bill_billingperiodstartdate,
         from_iso8601_timestamp(bill_billingperiodenddate) AS bill_billingperiodenddate,
         from_iso8601_timestamp(lineItem_UsageStartDate) AS lineItem_UsageStartDate,
         from_iso8601_timestamp(lineItem_UsageEndDate) AS lineItem_UsageEndDate,
         identity_LineItemId,
         identity_TimeInterval,
         bill_InvoiceId,
         bill_BillingEntity,
         bill_BillType,
         bill_PayerAccountId,
         lineItem_UsageAccountId,
         lineItem_ProductCode,
         lineItem_UsageType,
         lineItem_Operation,
         lineItem_AvailabilityZone,
         lineItem_UsageAmount,
         lineItem_NormalizationFactor,
         lineItem_NormalizedUsageAmount,
         lineItem_CurrencyCode,
         lineItem_UnblendedRate,
         lineItem_BlendedRate,
         lineItem_BlendedCost,
         lineItem_TaxType,
         product_ProductName,
         product_databaseEdition,
         product_databaseEngine,
         product_dedicatedEbsThroughput,
         product_deploymentOption,
         product_instanceFamily,
         product_instanceType,
         product_productFamily,
         product_storage,
         product_storageClass,
         product_usagetype,
         product_usageFamily,
         product_volumeType
FROM aws_billing_report.my_cur_report

To start visualizing your billing data, choose Finish, Save, and Visualize.

Amazon QuickSight makes it easy for you to construct different views of your AWS Cost and Usage report. For example, you can create a KPI visualization that tracks the month-to-date spending, a time series chart of daily spend for the last two weeks, and a usage breakdown by service or features, all using the point-and-click UI. Within a few minutes, you can create a comprehensive dashboard of your AWS bill that you can use to keep track of your usage on a daily basis. You can also easily share the dashboard with others in your organization.

Conclusion

In this walkthrough, you successfully created a new S3 bucket and built a Lambda function (written in Node.js) to extract, transform, and write your billing report to an S3 folder structure that looks like a database partition to Athena. You also created a database and table in Athena. You are now ready to run standard ANSI SQL queries against your AWS Cost and Usage report billing data. You won’t see any data in your newly created S3 bucket until your Lambda function is triggered.

If you have questions or suggestions, please comment below.

About the Author

Androski Spicer has been a Solution Architect with Amazon Web Services the past two and half years. He works with customers to ensure that their environments are architected for success and according to AWS best practices. In his free time he can be found cheering for Chelsea FC, Seattle Sounders, Portland Timbers or the Vancouver Whitecaps.

 

 

 

Catching Up on Some Recent AWS Launches and Publications

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/catching-up-on-some-recent-aws-launches-and-publications/

As I have noted in the past, the AWS Blog Team is working hard to make sure that you know about as many AWS launches and publications as possible, without totally burying you in content! As part of our balancing act, we will occasionally publish catch-up posts to clear our queues and to bring more information to your attention. Here’s what I have in store for you today:

  • Monitoring for Cross-Region Replication of S3 Objects
  • Tags for Spot Fleet Instances
  • PCI DSS Compliance for 12 More Services
  • HIPAA Eligibility for WorkDocs
  • VPC Resizing
  • AppStream 2.0 Graphics Design Instances
  • AMS Connector App for ServiceNow
  • Regtech in the Cloud
  • New & Revised Quick Starts

Let’s jump right in!

Monitoring for Cross-Region Replication of S3 Objects
I told you about cross-region replication for S3 a couple of years ago. As I showed you at the time, you simply enable versioning for the source bucket and then choose a destination region and bucket. You can check the replication status manually, or you can create an inventory (daily or weekly) of the source and destination buckets.

The Cross-Region Replication Monitor (CRR Monitor for short) solution checks the replication status of objects across regions and gives you metrics and failure notifications in near real-time.

To learn more, read the CRR Monitor Implementation Guide and then use the AWS CloudFormation template to Deploy the CRR Monitor.

Tags for Spot Instances
Spot Instances and Spot Fleets (collections of Spot Instances) give you access to spare compute capacity. We recently gave you the ability to enter tags (key/value pairs) as part of your spot requests and to have those tags applied to the EC2 instances launched to fulfill the request:

To learn more, read Tag Your Spot Fleet EC2 Instances.

PCI DSS Compliance for 12 More Services
As first announced on the AWS Security Blog, we recently added 12 more services to our PCI DSS compliance program, raising the total number of in-scope services to 42. To learn more, check out our Compliance Resources.

HIPAA Eligibility for WorkDocs
In other compliance news, we announced that Amazon WorkDocs has achieved HIPAA eligibility and PCI DSS compliance in all AWS Regions where WorkDocs is available.

VPC Resizing
This feature allows you to extend an existing Virtual Private Cloud (VPC) by adding additional blocks of addresses. This gives you more flexibility and should help you to deal with growth. You can add up to four secondary /16 CIDRs per VPC. You can also edit the secondary CIDRs by deleting them and adding new ones. Simply select the VPC and choose Edit CIDRs from the menu:

Then add or remove CIDR blocks as desired:

To learn more, read about VPCs and Subnets.

AppStream 2.0 Graphics Design Instances
Powered by AMD FirePro S7150x2 Server GPUs and equipped with AMD Multiuser GPU technology, the new Graphics Design instances for Amazon AppStream 2.0 will let you run and stream graphics applications more cost-effectively than ever. The instances are available in four sizes, with 2-16 vCPUs and 7.5 GB to 61 GB of memory.

To learn more, read Introducing Amazon AppStream 2.0 Graphics Design, a New Lower Costs Instance Type for Streaming Graphics Applications.

AMS Connector App for ServiceNow
AWS Managed Services (AMS) provides Infrastructure Operations Management for the Enterprise. Designed to accelerate cloud adoption, it automates common operations such as change requests, patch management, security and backup.

The new AMS integration App for ServiceNow lets you interact with AMS from within ServiceNow, with no need for any custom development or API integration.

To learn more, read Cloud Management Made Easier: AWS Managed Services Now Integrates with ServiceNow.

Regtech in the Cloud
Regtech (as I learned while writing this), is short for regulatory technology, and is all about using innovative technology such as cloud computing, analytics, and machine learning to address regulatory challenges.

Working together with APN Consulting Partner Cognizant, TABB Group recently published a thought leadership paper that explains why regulations and compliance pose huge challenges for our customers in the financial services, and shows how AWS can help!

New & Revised Quick Starts
Our Quick Starts team has been cranking out new solutions and making significant updates to the existing ones. Here’s a roster:

Alfresco Content Services (v2) Atlassian Confluence Confluent Platform Data Lake
Datastax Enterprise GitHub Enterprise Hashicorp Nomad HIPAA
Hybrid Data Lake with Wandisco Fusion IBM MQ IBM Spectrum Scale Informatica EIC
Magento (v2) Linux Bastion (v2) Modern Data Warehouse with Tableau MongoDB (v2)
NetApp ONTAP NGINX (v2) RD Gateway Red Hat Openshift
SAS Grid SIOS Datakeeper StorReduce SQL Server (v2)

And that’s all I have for today!

— Jeff;

New – Stop & Resume Workloads on EC2 Spot Instances

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/new-stop-resume-workloads-on-ec2-spot-instances/

EC2 Spot Instances give you access to spare EC2 compute capacity at up to 90% off of the On-Demand rates. Starting with the ability to request a specific number of instances of a particular size, we made Spot Instances even more useful and flexible with support for Spot Fleets and Auto Scaling Spot Fleets, allowing you to maintain any desired level of compute capacity.

EC2 users have long had the ability to stop running instances while leaving EBS volumes attached, opening the door to applications that automatically pick up where they left off when the instance starts running again.

Stop and Resume Spot Workloads
Today we are blending these two important features, allowing you to set up Spot bids and Spot Fleets that respond by stopping (rather than terminating) instances when capacity is no longer available at or below your bid price. EBS volumes attached to stopped instances remain intact, as does the EBS-backed root volume. When capacity becomes available, the instances are started and can keep on going without having to spend time provisioning applications, setting up EBS volumes, downloading data, joining network domains, and so forth.

Many AWS customers have enhanced their applications to create and make use of checkpoints, adding some resilience and gaining the ability to take advantage of EC2’s start/stop feature in the process. These customers will now be able to run these applications on Spot Instances, with savings that average 70-90%.

While the instances are stopped, you can modify the EBS Optimization, User data, Ramdisk ID, and Delete on Termination attributes. Stopped Spot Instances do not incur any charges for compute time; space for attached EBS volumes is charged at the usual rates.

Here’s how you create a Spot bid or Spot Fleet and specify the use of stop/start:

Things to Know
This feature is available now and you can start using it today in all AWS Regions where Spot Instances are available. It is designed to work well in conjunction with the new per-second billing for EC2 instances and EBS volumes, with the potential for another dimension of cost savings over and above that provided by Spot Instances.

EBS volumes always exist within a particular Availability Zone (AZ). As a result, Spot and Spot Fleet requests that specify a particular AZ will always restart in that AZ.

Take care when using this feature in conjunction with Spot Fleets that have the potential to span a wide variety of instance types. Because the composition of the fleet can change over time, you need to pay attention to your account’s limits for IP addresses and EBS volumes.

I’m looking forward to hearing about the new and creative uses that you’ll come up with for this feature. If you thought that your application was not a good fit for Spot Instances, or if the overhead needed to handle interruptions was too high, it is time to take another look!

— Jeff;

 

New – Per-Second Billing for EC2 Instances and EBS Volumes

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/new-per-second-billing-for-ec2-instances-and-ebs-volumes/

Back in the old days, you needed to buy or lease a server if you needed access to compute power. When we launched EC2 back in 2006, the ability to use an instance for an hour, and to pay only for that hour, was big news. The pay-as-you-go model inspired our customers to think about new ways to develop, test, and run applications of all types.

Today, services like AWS Lambda prove that we can do a lot of useful work in a short time. Many of our customers are dreaming up applications for EC2 that can make good use of a large number of instances for shorter amounts of time, sometimes just a few minutes.

Per-Second Billing for EC2 and EBS
Effective October 2nd, usage of Linux instances that are launched in On-Demand, Reserved, and Spot form will be billed in one-second increments. Similarly, provisioned storage for EBS volumes will be billed in one-second increments.

Per-second billing also applies to Amazon EMR and AWS Batch:

Amazon EMR – Our customers add capacity to their EMR clusters in order to get their results more quickly. With per-second billing for the EC2 instances in the clusters, adding nodes is more cost-effective than ever.

AWS Batch – Many of the batch jobs that our customers run complete in less than an hour. AWS Batch already launches and terminates Spot Instances; with per-second billing batch processing will become even more economical.

Some of our more sophisticated customers have built systems to get the most value from EC2 by strategically choosing the most advantageous target instances when managing their gaming, ad tech, or 3D rendering fleets. Per-second billing obviates the need for this extra layer of instance management, and brings the costs savings to all customers and all workloads.

While this will result in a price reduction for many workloads (and you know we love price reductions), I don’t think that’s the most important aspect of this change. I believe that this change will inspire you to innovate and to think about your compute-bound problems in new ways. How can you use it to improve your support for continuous integration? Can it change the way that you provision transient environments for your dev and test workloads? What about your analytics, batch processing, and 3D rendering?

One of the many advantages of cloud computing is the elastic nature of provisioning or deprovisioning resources as you need them. By billing usage down to the second we will enable customers to level up their elasticity, save money, and customers will be positioned to take advantage of continuing advances in computing.

Things to Know
This change is effective in all AWS Regions and will be effective October 2, for all Linux instances that are newly launched or already running. Per-second billing is not currently applicable to instances running Microsoft Windows or Linux distributions that have a separate hourly charge. There is a 1 minute minimum charge per-instance.

List prices and Spot Market prices are still listed on a per-hour basis, but bills are calculated down to the second, as is Reserved Instance usage (you can launch, use, and terminate multiple instances within an hour and get the Reserved Instance Benefit for all of the instances). Also, bills will show times in decimal form, like this:

The Dedicated Per Region Fee, EBS Snapshots, and products in AWS Marketplace are still billed on an hourly basis.

— Jeff;

 

Can an Army of Bitcoin “Bounty Hunters” Deter Pirates?

Post Syndicated from Ernesto original https://torrentfreak.com/can-an-army-of-bitcoin-bounty-hunters-deter-pirates-170917/

When we first heard of the idea to use Bitcoin bounties to track down pirated content online, we scratched our heads.

Snitching on copyright infringers is not a new concept, but the idea of instant cash rewards though cryptocurrency was quite novel.

In theory, it’s pretty straightforward. Content producers can add a unique identifying watermark into movies, eBooks, or other digital files before they’re circulated. When these somehow leak to the public, the bounty hunters use the watermark to claim their Bitcoin, alerting the owner in the process.

This helps to spot leaks early on, even on networks where automated tools don’t have access, and identify the source at the same time.

Two years have passed and it looks like the idea was no fluke. Custos, the South African company that owns the technology, has various copyright holders on board and recently announced a new partnership with book publisher Erudition Digital.

With help from anti-piracy outfit Digimarc, the companies will add identifying watermarks to eBook releases, counting on the bounty hunters to keep an eye out for leaks. These bounty hunters don’t have to be anti-piracy experts. On the contrary, pirates are more than welcome to help out.

“The Custos approach is revolutionary in that it attacks the economy of piracy by targeting uploaders rather than downloaders, turning downloaders into an early detection network,” the companies announced a few days ago.

“The result is pirates turn on one another, sowing seeds of distrust amongst their communities. As a result, the Custos system is capable of penetrating hard-to-reach places such as the dark web, peer-to-peer networks, and even email.”



Devon Weston, Director of Market Development for Digimarc Guardian, believes that this approach is the next level in anti-piracy efforts. It complements the automated detection tools that have been available in the past by providing access to hard-to-reach places.

“Together, this suite of products represents the next generation in technical measures against eBook piracy,” Weston commented on the partnership.

TorrentFreak reached out to Custos COO Fred Lutz to find out what progress the company has made in recent years. We were informed that they have been protecting thousands of copies every month, ranging from pre-release movie content to eBooks.

At the moment the company works with a selected group of “bounty hunters,” but they plan to open the extraction tool to the public in the near future, so everyone can join in.

“So far we have carefully seeded the free bounty extractor tool in relevant communities with great success. However, in the next phase, we will open the bounty hunting to the general public. We are just careful not to grow the bounty hunting community faster than the number of bounties in the wild require,” Lutz tells us.

The Bitcoin bounties themselves vary in size based on the specific use case. For a movie screener, they are typically anything between $10 and $50. However, for the most sensitive content, they can be $100 or more.

“We can also adjust the bounty over time based on the customer’s needs. A low-quality screener that was very sensitive prior to cinematic release does not require as large a bounty after cam-rips becomes available,” Lutz notes.

Thus far, roughly 50 Bitcoin bounties have been claimed. Some of these were planted by Custos themselves, as an incentive for the bounty hunters. Not a very high number, but that doesn’t mean that it’s not working.

“While this number might seem a bit small compared to the number of copies we protect, our aim is first and foremost not to detect leaks, but to pose a credible threat of quick detection and being caught.”

People who receive content protected by Custos are made aware of the watermarks, which may make them think twice about sharing it. If that’s the case, then it’s having effect without any bounties being claimed.

The question remains how many people will actively help to spot bounties. The success of the system largely depends on volunteers, and not all pirates are eager to rat on the people that provide free content.

On the other hand, there’s also room to abuse the system. In theory, people could claim the bounties on their own eBooks and claim that they’ve lost their e-reader. That would be fraud, of course, but since the bounties are in Bitcoin this isn’t easy to prove.

That brings us to the final question. What happens of a claimed bounty identifies a leaker? Custos admits that this alone isn’t enough evidence to pursue a legal case, but the measures that are taken in response are up to the copyright holders.

“A claim of a bounty is never a sufficient legal proof of piracy, instead, it is an invaluable first piece of evidence on which a legal case could be built if the client so requires. Legal prosecution is definitely not always the best approach to dealing with leaks,” Lutz says.

Time will tell if the Bitcoin bounty approach works…

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

MPAA: Net Neutrality Rules Should Not Hinder Anti-Piracy Efforts

Post Syndicated from Ernesto original https://torrentfreak.com/mpaa-net-neutrality-rules-should-not-hinder-anti-piracy-efforts-170907/

This summer, millions of people protested the FCC’s plan to repeal the net neutrality rules that were put in place by the former Obama administration.

Well over 22 million comments are listed on the FCC site already and among those we spotted a response from the main movie industry lobby group, the MPAA.

Acting on behalf of six major Hollywood studios, the MPAA is not getting involved in the repeal debate. It instead highlights that, if the FCC maintains any type of network neutrality rules, these shouldn’t get in the way of its anti-piracy efforts.

The Hollywood group stresses that despite an increase in legal services, online piracy remains a problem. Through various anti-piracy measures, rightsholders are working hard to combat this threat, which is their right by law.

“Copyright owners and content providers have a right under the Copyright and Communications acts to combat theft of their content, and the law encourages internet intermediaries to collaborate with content creators to do so,” the MPAA writes.

Now that the net neutrality rules are facing a possible revision or repeal, the MPAA wants to make it very clear that any future regulation should not get in the way of these anti-piracy efforts.

“The MPAA therefore asks that any network neutrality rules the FCC maintains or adopts make explicit that such rules do not limit the ability of copyright owners and their licensees to combat copyright infringement,” the group writes to the FCC.

This means that measures such as website blocking, which could be considered to violate net neutrality as it discriminates against specific traffic, should be allowed. The same is true for other filtering and blocking efforts.

The MPAA’s position doesn’t come as a surprise and given the FCC’s actions in the past, Hollywood has little to worry about. The current net neutrality rules, which were put in place by the Obama administration, specifically exclude pirate traffic.

“Nothing in this part prohibits reasonable efforts by a provider of broadband Internet access service to address copyright infringement or other unlawful activity,” the current net neutrality order reads.

“We reiterate that our rules do not alter the copyright laws and are not intended to prohibit or discourage voluntary practices undertaken to address or mitigate the occurrence of copyright infringement,” the FCC previously clarified.

Still, the MPAA is better safe than sorry.

This is not the first time that the MPAA has got involved in net neutrality debates. Behind the scenes the group has been lobbying US lawmakers on this issue for several years, previously arguing for similar net neutrality exceptions in Brazil and India.

—

The MPAA’s full comments can be found here (pdf).

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

Datavalet Wi-Fi Blocks TorrentFreak Over ‘Criminal Hacking Skills’

Post Syndicated from Ernesto original https://torrentfreak.com/datavalet-wi-fi-blocks-torrentfreak-over-criminal-hacking-skills-170903/

At TorrentFreak we regularly write about website blocking efforts around the globe, usually related to well-known pirate sites.

Unfortunately, our own news site is not immune to access restrictions either. While no court has ordered ISPs to block access to our articles, some are doing this voluntarily.

This is especially true for companies that provide Wi-Fi hotspots, such as Datavalet. This wireless network provider works with various large organizations, including McDonald’s, Starbucks, and airports, to offer customers free Internet access.

Or rather to a part of the public Internet, we should say.

Over the past several months, we have had several reports from people who are unable to access TorrentFreak on Datavalet’s network. Users who load our website get an ominous warning instead, suggesting that we run some kind of a criminal hacking operation.

“Access to TORRENTFREAK.COM is not permitted as it is classified as: CRIMINAL SKILLS / HACKING.”

Criminal Skills?

Although we see ourselves as skilled writing news in our small niche, which incidentally covers crime and hacking, our own hacking skills are below par. Admittedly, mistakes are easily made but Datavalet’s blocking efforts are rather persistent.

The same issue was brought to our attention several years ago. At the time, we reached out to Datavalet and a friendly senior network analyst promised that they would look into it.

“We have forwarded your concerns to the proper resources and as soon as we have an update we will let you know,” the response was. But a few years later the block is still active, or active again.

Datavalet is just one one the many networks where TorrentFreak is blocked. Often, we are categorized as a file-sharing site, probably due to the word “torrent” in our name. This recently happened at the NYC Brooklyn library, for example.

After a reader kindly informed the library that we’re a news site, we were suddenly transferred from the “Peer-to-Peer File Sharing” to the “Proxy Avoidance” category.

“It appears that the website you want to access falls under the category ‘Proxy Avoidance’. These are sites that provide information about how to bypass proxy server features or to gain access to URLs in any way that bypass the proxy server,” the library explained.

Still blocked of course.

At least we’re not the only site facing this censorship battle. Datavelet and others regularly engage in overblocking to keep their network and customers safe. For example, Reddit was recently banned because it offered “nudity,” which is another no-go area.

Living up to our “proxy avoidance” reputation, we have to mention that people who regularly face these type of restrictions may want to invest in a VPN. These are generally quite good at bypassing these type of blockades. If they are not blocked themselves, that is.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

Plex Gets Tough Over Unofficial Kodi Addon But Crisis Quickly Averted

Post Syndicated from Andy original https://torrentfreak.com/plex-gets-tough-over-unofficial-kodi-addon-but-crisis-quickly-averted-170831/

Earlier this week, TF reported on the surprise retirement of established and respected Kodi addon developer, MetalKettle.

After several years of producing and distributing a wide range of addons for the popular media center, ‘MK’ threw in the towel.

“Over the past year or so Kodi has become more mainstream and public we’ve all seen the actions of others become highlighted legally, with authorities determined to target 3rd party addons making traction,” MK announced.

“This has eventually caused me to consider ‘what if?’ – the result of which never ends well in my mind.”

The “what if?” here is whether MK himself would ever find himself targeted by legal action. As the recent case filed against Kodi addons site TVAddons by Bell, Videotron, Groupe TVA, and Rogers shows, copyright holders are prepared to act aggressively to protect their rights.

While we don’t doubt MK’s reasons for retirement, it now transpires that MK was already in the spotlight of Plex Inc., the company behind Plex. They seem particularly unimpressed with some of his recent work.

Plex is a system which allows users to manage and organize their entire media library, import artwork and all kinds of metadata, and then stream it to any device, such as TVs, phones, and set-top boxes.

For people with content already around it’s a beautiful Netflix-beating system that’s free to use up to a point, but it also comes with premium features as part of a $5pm Plex Pass subscription. Those extras include a Kodi addon.

“Currently available as a preview to our premium Plex Pass subscribers, the Plex for Kodi add-on gives users the best of both worlds; the advanced customization of Kodi home theater and the beauty and simplicity of Plex for access on any device, anywhere, anytime,” the Plex site reads.

MK Plex, the addon produced by MetalKettle, unlocks the restrictions. Installable in Kodi with just a few clicks, the addon allows users of Plex to achieve the above without the need for a Plex Pass subscription. The addon was submitted by MK to the TVAddons repository last week but it took only days for Plex’s legal department to swing into action.

TorrentFreak obtained a copy of the company’s letter to TVAddons from a source close to the platform.

“We represent Plex, Inc. of Los Gatos, California, USA (‘Plex’) in trademark and other intellectual property legal matters. Plex recently learned that you offer a software product called MK Plex that is promoted on [your] website,” it begins.

“The designation MK Plex infringes on Plex’s intellectual property rights and therefore Plex hereby demands that you remove all web pages that use the designation MK Plex, delete all software branded MK Plex and disable all downloading, and cease all other use of Plex’s trademarks immediately.”

While this request would’ve been enough for TVAddons to respond, the lawyers went on in some detail about trademark issues, reiterating that “the designation ‘unofficial plugin’ indicates that your use of MK Plex is intended to trade on the reputation and goodwill of the Plex brand.”

In summary, Plex said that by presenting itself as an unofficial plug-in, MK Plex infringes trademark and unfair competition law in the United States. There was no mention at all of the features provided by the software, the entire issue was framed as a trademark dispute.

Plex’s lawyers continued by threatening formal legal action against TVAddons, including a claim for damages and a court order to shut down the entire site. That’s pretty aggressive for a letter that references an addon that was completely produced by a third-party. Nevertheless, TVAddons informs TF that it had no problem complying with Plex’s takedown request as a matter of urgency.

“We love Plex and would never do anything to cause them grief. The user generated addon in question should never have made its way into our repository in the first place and we apologize for that,” a site representative said.

“We’d like to stress, however, that we took swift action as soon as Plex made us aware of the issue. This is why it’s so important that all web sites with user generated institute a legitimate takedown notice system, so that issues like this can be amicably resolved much faster than they could be in court, without any harm done to either party. TV ADDONS has always and will continue to respect the rights of others.”

TVAddons doesn’t know if MK himself is fully aware of the problem Plex has with his addon or whether he’s going to put it into retirement along with himself. However, the undertone in our conversation indicated that might be a prudent course of action.

TVAddons said they responded immediately to all of Plex’s demands and were thanked for dealing with the matter comprehensively, so it’s at least possible that MK has similar options available.

While a trademark dispute for a Kodi addon is interesting enough, this case offers a perfect illustration of how problems with intellectual property can be worked through, if the parties are prepared to work together. In the ongoing case of the Canadian broadcasters against TVAddons, not a single takedown notice or piece of correspondence was sent to TVAddons in advance of super-aggressive legal action.

Had a simple email been sent, the problem could’ve been solved in a day, rather than the years currently expected.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

Hardening the Kernel in Android Oreo (Android Developers Blog)

Post Syndicated from corbet original https://lwn.net/Articles/732475/rss

The Android Developers Blog has an
overview of the security features added to the kernel in the Android
“Oreo” release. “Usercopy functions are used by the kernel to
transfer data from user space to kernel space memory and back again. Since
2014, missing or invalid bounds checking has caused about 45% of Android’s
kernel vulnerabilities. Hardened usercopy adds bounds checking to usercopy
functions, which helps developers spot misuse and fix bugs in their
code. Also, if obscure driver bugs slip through, hardening these functions
prevents the exploitation of such bugs.”

Deadline 10 – Launch a Rendering Fleet in AWS

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/deadline-10-launch-a-rendering-fleet-in-aws/

Graphical rendering is a compute-intensive task that is, as they say, embarrassingly parallel. Looked at another way, this means that there’s a more or less linear relationship between the number of processors that are working on the problem and the overall wall-clock time that it takes to complete the task. In a creative endeavor such as movie-making, getting the results faster spurs creativity, improves the feedback loop, gives you time to make more iterations and trials, and leads to a better result. Even if you have a render farm in-house, you may still want to turn to the cloud in order to gain access to more compute power at peak times. Once you do this, the next challenge is to manage the combination of in-house resources, cloud resources, and the digital assets in a unified fashion.

Deadline 10
Earlier this week we launched Deadline 10, a powerful render management system. Building on technology that we brought on board with the acquisition of Thinkbox Software, Deadline 10 is designed to extend existing on-premises rendering into the AWS Cloud, giving you elasticity and flexibility while remaining simple and easy to use. You can set up and manage large-scale distributed jobs that span multiple AWS regions and benefit from elastic, usage-based AWS licensing for popular applications like Deadline for Autodesk 3ds Max, Maya, Arnold, and dozens more, all available from the Thinkbox Marketplace. You can purchase software licenses from the marketplace, use your existing licenses, or use them together.

Deadline 10 obtains cloud-based compute resources by managing bids for EC2 Spot Instances, providing you with access to enough low-cost compute capacity to let your imagination run wild! It uses your existing AWS account, tags EC2 instances for tracking, and synchronizes your local assets to the cloud before rendering begins.

A Quick Tour
Let’s take a quick tour of Deadline 10 and see how it makes use of AWS. The AWS Portal is available from the View menu:

The first step is to log in to my AWS account:

Then I configure the connection server, license server, and the S3 bucket that will be used to store rendering assets:

Next, I set up my Spot fleet, establishing a maximum price per hour for each EC2 instance, setting target capacity, and choosing the desired rendering application:

I can also choose any desired combination of EC2 instance types:

When I am ready to render I click on Start Spot Fleet:

This will initiate the process of bidding for and managing Spot Instances. The running instances are visible from the Portal:

I can monitor the progress of my rendering pipeline:

I can stop my Spot fleet when I no longer need it:

Deadline 10 is now available for usage based license customers; a new license is needed for traditional floating license users. Pricing for yearly Deadline licenses has been reduced to $48 annually. If you are already using an earlier version of Deadline, feel free to contact us to learn more about licensing options.

— Jeff;