This is well-worth reading (non-paywalled version). Here’s the opening:
Cryptocurrencies, although a seemingly interesting idea, are simply not fit for purpose. They do not work as currencies, they are grossly inefficient, and they are not meaningfully distributed in terms of trust. Risks involving cryptocurrencies occur in four major areas: technical risks to participants, economic risks to participants, systemic risks to the cryptocurrency ecosystem, and societal risks.
I haven’t written much about cryptocurrencies, but I share Weaver’s skepticism.
We have seen a lot of discussion this past week about the role of Amazon Rekognition in facial recognition, surveillance, and civil liberties, and we wanted to share some thoughts.
Amazon Rekognition is a service we announced in 2016. It makes use of new technologies – such as deep learning – and puts them in the hands of developers in an easy-to-use, low-cost way. Since then, we have seen customers use the image and video analysis capabilities of Amazon Rekognition in ways that materially benefit both society (e.g. preventing human trafficking, inhibiting child exploitation, reuniting missing children with their families, and building educational apps for children), and organizations (enhancing security through multi-factor authentication, finding images more easily, or preventing package theft). Amazon Web Services (AWS) is not the only provider of services like these, and we remain excited about how image and video analysis can be a driver for good in the world, including in the public sector and law enforcement.
There have always been and will always be risks with new technology capabilities. Each organization choosing to employ technology must act responsibly or risk legal penalties and public condemnation. AWS takes its responsibilities seriously. But we believe it is the wrong approach to impose a ban on promising new technologies because they might be used by bad actors for nefarious purposes in the future. The world would be a very different place if we had restricted people from buying computers because it was possible to use that computer to do harm. The same can be said of thousands of technologies upon which we all rely each day. Through responsible use, the benefits have far outweighed the risks.
Customers are off to a great start with Amazon Rekognition; the evidence of the positive impact this new technology can provide is strong (and growing by the week), and we’re excited to continue to support our customers in its responsible use.
-Dr. Matt Wood, general manager of artificial intelligence at AWS
Over the past several years, anyone looking for a piracy-configured set-top box could do worse than search for one on Amazon or eBay.
Historically, people deploying search terms including “Kodi” or “fully-loaded” were greeted by page after page of Android-type boxes, each ready for illicit plug-and-play entertainment consumption following delivery.
Although the problem persists on both platforms, people are now much less likely to find infringing devices than they were 12 to 24 months ago. Under pressure from entertainment industry groups, both Amazon and eBay have tightened the screws on sellers of such devices. Now, however, both companies have received requests to stem sales from a completetey different direction.
In a letter to eBay CEO Devin Wenig and Amazon CEO Jeff Bezos first spotted by Ars, FCC Commissioner Michael O’Rielly calls on the platforms to take action against piracy-configured boxes that fail to comply with FCC equipment authorization requirements or falsely display FCC logos, contrary to United States law.
“Disturbingly, some rogue set-top box manufacturers and distributors are exploiting the FCC’s trusted logo by fraudulently placing it on devices that have not been approved via the Commission’s equipment authorization process,” O’Rielly’s letter reads.
“Specifically, nine set-top box distributors were referred to the FCC in October for enabling the unlawful streaming of copyrighted material, seven of which displayed the FCC logo, although there was no record of such compliance.”
While O’Rielly admits that the copyright infringement aspects fall outside the jurisdiction of the FCC, he says it’s troubling that many of these devices are used to stream infringing content, “exacerbating the theft of billions of dollars in American innovation and creativity.”
As noted above, both Amazon and eBay have taken steps to reduce sales of pirate boxes on their respective platforms on copyright infringement grounds, something which is duly noted by O’Rielly. However, he points out that devices continue to be sold to members of the public who may believe that the devices are legal since they’re available for sale from legitimate companies.
“For these reasons, I am seeking your further cooperation in assisting the FCC in taking steps to eliminate the non-FCC compliant devices or devices that fraudulently bear the FCC logo,” the Commissioner writes (pdf).
“Moreover, if your company is made aware by the Commission, with supporting evidence, that a particular device is using a fraudulent FCC label or has not been appropriately certified and labeled with a valid FCC logo, I respectfully request that you commit to swiftly removing these products from your sites.”
In the event that Amazon and eBay take action under this request, O’Rielly asks both platforms to hand over information they hold on offending manufacturers, distributors, and suppliers.
Amazon was quick to respond to the FCC. In a letter published by Ars, Amazon’s Public Policy Vice President Brian Huseman assured O’Rielly that the company is not only dedicated to tackling rogue devices on copyright-infringement grounds but also when there is fraudulent use of the FCC’s logos.
Noting that Amazon is a key member of the Alliance for Creativity and Entertainment (ACE) – a group that has been taking legal action against sellers of infringing streaming devices (ISDs) and those who make infringing addons for Kodi-type systems – Huseman says that dealing with the problem is a top priority.
“Our goal is to prevent the sale of ISDs anywhere, as we seek to protect our customers from the risks posed by these devices, in addition to our interest in protecting Amazon Studios content,” Huseman writes.
“In 2017, Amazon became the first online marketplace to prohibit the sale of streaming media players that promote or facilitate piracy. To prevent the sale of these devices, we proactively scan product listings for signs of potentially infringing products, and we also invest heavily in sophisticated, automated real-time tools to review a variety of data sources and signals to identify inauthentic goods.
“These automated tools are supplemented by human reviewers that conduct manual investigations. When we suspect infringement, we take immediate action to remove suspected listings, and we also take enforcement action against sellers’ entire accounts when appropriate.”
Huseman also reveals that since implementing a proactive policy against such devices, “tens of thousands” of listings have been blocked from Amazon. In addition, the platform has been making criminal referrals to law enforcement as well as taking civil action (1,2,3) as part of ACE.
“As noted in your letter, we would also appreciate the opportunity to collaborate further with the FCC to remove non-compliant devices that improperly use the FCC logo or falsely claim FCC certification. If any FCC non-compliant devices are identified, we seek to work with you to ensure they are not offered for sale,” Huseman concludes.
As Internet-capable set-top boxes pour into homes across all populated continents, authorities seem almost powerless to come up with a significant response to the growing threat.
In standard form these devices, which are often Android-based, are entirely legal. However, when configured with specialist software they become piracy powerhouses providing access to all content imaginable, often at copyright holders’ expense.
A large proportion of these devices come from Asia, China in particular, but it’s relatively rare to hear of enforcement action in that part of the world. That changed this week with an announcement from Hong Kong customs detailing a series of raids in the areas of Sham Shui Po and Wan Chai.
After conducting an in-depth investigation with the assistance of copyright holders, on May 25 and 26 Customs and Excise officers launched Operation Trojan Horse, carrying out a series of raids on four premises selling suspected piracy-configured set-top boxes.
During the operation, officers arrested seven men and one woman aged between 18 and 45. Four of them were shop owners and the other four were salespeople. Around 354 suspected ‘pirate’ boxes were seized with an estimated market value of HK$320,000 (US$40,700).
“In the past few months, the department has stepped up inspections of hotspots for TV set-top boxes,” a statement from authorities reads.
“We have discovered that some shops have sold suspected illegal set-top boxes that bypass the copyright protection measures imposed by copyright holders of pay television programs allowing people to watch pay television programs for free.”
Some of the devices seized by Hong Kong Customs
During a press conference yesterday, a representative from the Customs Copyright and Trademark Investigations (Action) Division said that in the run up to the World Cup in 2018, measures against copyright infringement will be strengthened both on and online.
The announcement was welcomed by the Cable and Satellite Broadcasting Association of Asia’s (CASBAA) Coalition Against Piracy, which is back by industry heavyweights including Disney, Fox, HBO Asia, NBCUniversal, Premier League, Turner Asia-Pacific, A&E Networks, Astro, BBC Worldwide, National Basketball Association, TV5MONDE, Viacom International, and others.
“We commend the great work of Hong Kong Customs in clamping down on syndicates who profit from the sale of Illicit Streaming Devices,” said General Manager Neil Gane.
“The prevalence of ISDs in Hong Kong and across South East Asia is staggering. The criminals who sell ISDs, as well as those who operate the ISD networks and pirate websites, are profiting from the hard work of talented creators, seriously damaging the legitimate content ecosystem as well as exposing consumers to dangerous malware.”
Malware warnings are very prevalent these days but it’s not something the majority of set-top box owners have a problem with. Indeed, a study carried by Sycamore Research found that pirates aren’t easily deterred by such warnings.
Nevertheless, there are definite risks for individuals selling devices when they’re configured for piracy.
Recent cases, particularly in the UK, have shown that hefty jail sentences can hit offenders while over in the United States (1,2,3), lawsuits filed by the Alliance for Creativity and Entertainment (ACE) have the potential to end in unfavorable rulings for multiple defendants.
Although rarely reported, offenders in Hong Kong also face stiff sentences for this kind of infringement including large fines and custodial sentences of up to four years.
On a large number of occasions over the past decade, Ukraine has played host to some of the world’s largest pirate sites.
At various points over the years, The Pirate Bay, KickassTorrents, ExtraTorrent, Demonoid and raft of streaming portals could be found housed in the country’s data centers, reportedly taking advantage of laws more favorable than those in the US and EU.
As a result, Ukraine has been regularly criticized for not doing enough to combat piracy but when placed under pressure, it does take action. In 2010, for example, the local government expressed concerns about the hosting of KickassTorrents in the country and in August the same year, the site was kicked out by its host.
“Kickasstorrents.com main web server was shut down by the hosting provider after it was contacted by local authorities. One way or another I’m afraid we must say goodbye to Ukraine and move the servers to other countries,” the site’s founder told TF at the time.
In the years since, Ukraine has launched sporadic action against pirate sites and has taken steps to tighten up copyright law. The Law on State Support of Cinematography came into force during April 2017 and gave copyright owners new tools to combat infringement by forcing (in theory, at least) site operators and web hosts to respond to takedown requests.
But according to the United States and Europe, not enough is being done. After the EU Commission warned that Ukraine risked damaging relations with the EU, last September US companies followed up with another scathing attack.
In a recommendation to the U.S. Government, the IIPA, which counts the MPAA, RIAA, and ESA among its members, asked U.S. authorities to suspend or withdraw Ukraine’s trade benefits until the online piracy situation improves.
“Legislation is needed to institute proper notice and takedown provisions, including a requirement that service providers terminate access to individuals (or entities) that have repeatedly engaged in infringement, and the retention of information for law enforcement, as well as to provide clear third party liability regarding ISPs,” the IIPA wrote.
But amid all the criticism, Ukraine cyber police chief Sergey Demedyuk says that while his department is committed to tackling piracy, it can only do so when complaints are filed with him.
“Yes, we are engaged in piracy very closely. The problem is that piracy is a crime of private accusation. So here we deal with them only in cases where we are contacted,” Demedyuk said in an Interfax interview published yesterday.
Surprisingly, given the number of dissenting voices, it appears that complaints about these matters aren’t exactly prevalent. So are there many at all?
“Unfortunately, no. In the media, many companies claim that their rights are being violated by pirates. But if you count the applications that come to us, they are one,” Demedyuk reveals.
“In general, we are handling Ukrainian media companies, who produce their own product and are worried about its fate. Also on foreign films, the ‘Anti-Piracy Agency’ refers to us, but not as intensively as before.”
Why complaints are going down, Demedyuk does not know, but when his unit is asked to take action it does so, he claims. Indeed, Demedyuk cites two particularly significant historical operations against a pair of large ‘pirate’ sites.
In 2012, Ukraine shut down EX.ua, a massive cyberlocker site following a six-month investigation initiated by international tech companies including Microsoft, Graphisoft and Adobe. Around 200 servers were seized, together hosting around 6,000 terabytes of data.
Then in November 2016, following a complaint from the MPAA, police raided FS.to, one of Ukraine’s most popular pirate sites. Initial reports indicated that 60 servers were seized and 19 people were arrested.
“To see the effect of combating piracy, this should not be done at the level of cyberpolicy, but at the state level,” Demedyuk advises.
“This requires constant close interaction between law enforcement agencies and rights holders. Only by using all these tools will we be able to effectively counteract copyright infringements.”
Meanwhile, the Office of the United States Trade Representative has maintained Ukraine’s position on the Priority Watchlist of its latest Special 301 Report and there a no signs it will be leaving anytime soon.
Exactly one year ago, millions of users in the BitTorrent community went into mourning with the shock depature of one of its major players.
ExtraTorrent was founded in back in November 2006, at a time when classic platforms such as TorrentSpy and Mininova were dominating the torrent site landscape. But with dedication and determination, the site amassed millions of daily visitors, outperforming every other torrent site apart from the mighty Pirate Bay.
Then, on May 17, 2017, everything came crashing down.
“ExtraTorrent has shut down permanently,” a note in the site read. “ExtraTorrent with all mirrors goes offline. We permanently erase all data. Stay away from fake ExtraTorrent websites and clones. Thx to all ET supporters and torrent community. ET was a place to be….”
While ExtraTorrent staff couldn’t be more clear in advising people to stay away from clones, few people listened to their warnings. Within hours, new sites appeared claiming to be official replacements for the much-loved torrent site and people flocked to them in their millions.
One of those was ExtraTorrent.ag, a torrent site connected to the operators of EZTV.ag, which appeared as a replacement in the wake of the official EZTV’s demise. Graphically very similar to the original ExtraTorrent, the .ag ‘replacement’ had none of its namesake’s community or unique content. But that didn’t dent its popularity.
At the start of this week, ExtraTorrent.ag was one of the most popular torrent sites on the Internet. With an Alexa rank of around 2,200, it would’ve clinched ninth position in our Top 10 Torrent Sites report earlier this year. However, after registering the site’s domain a year ago, something seems to have gone wrong.
Yesterday, on the anniversary of ExtraTorrent’s shutdown and exactly a year after the ExtraTorrent.ag domain was registered, ExtraTorrent.ag disappeared only to be replaced by a generic landing page, as shown below.
ExtraTorrent.ag landing page
This morning, however, there appear to be additional complications. Accessing with Firefox produces the page above but attempting to do so with Chrome produces an ominous security warning.
Indeed, those protected by MalwareBytes won’t be able to access the page at all, since ExtraTorrent.ag redirects to the domain FindBetterResults.com, which the anti-malware app flags as malicious.
The change was reported to TF by the operator of domain unblocking site Unblocked.lol, which offers torrent site proxies as well as access to live TV and sports.
“I noticed when I started receiving emails saying ExtraTorrent was redirecting to some parked domain. When I jumped on the PC and checked myself it was just redirecting to a blank page,” he informs us.
“First I thought they’d blocked our IP address so I used some different ones. But I soon discovered the domain was in fact parked.”
So what has happened to this previously-functioning domain?
Whois records show that ExtraTorrent.ag was created on May 17, 2017 and appears to have been registered for a year. Yesterday, on May 17, 2018, the domain was updated to list what could potentially be a new owner, with an expiry date of May 17, 2019.
Once domains have expired, they usually enter an ‘Auto-Renew Grace Period’ for up to 45 days. This is followed by a 30-day ‘Redemption Grace Period’. At the end of this second period, domains cannot be renewed and are released for third-parties to register. That doesn’t appear to have been the case here.
So, to find out more about the sudden changes we reached out to the email address listed in the WHOIS report but received no response. Should we hear more we’ll update this report but in the meantime the Internet has lost one of its largest torrent sites and gained a rather pointless landing page with potential security risks.
Spencer Ackerman has this interesting story about a guy assigned to crack down on unauthorized White House leaks. It’s necessarily light on technical details, so I thought I’d write up some guesses, either as a guide for future reporters asking questions, or for people who want to better know the risks when leak information.
It should come as no surprise that your work email and phone are already monitored. They can get every email you’ve sent or received, even if you’ve deleted it. They can get every text message you’ve sent or received, the metadata of every phone call sent or received, and so forth.
To a lesser extent, this also applies to your well-known personal phone and email accounts. Law enforcement can get the metadata (which includes text messages) for these things without a warrant. In the above story, the person doing the investigation wasn’t law enforcement, but I’m not sure that’s a significant barrier if they can pass things onto the Secret Service or something.
The danger here isn’t that you used these things to leak, it’s that you’ve used these things to converse with the reporter before you made the decision to leak. That’s what happened in the Reality Winner case: she communicated with The Intercept before she allegedly leaked a printed document to them via postal mail. While it wasn’t conclusive enough to convict her, the innocent emails certainly put the investigators on her trail.
The path to leaking often starts this way: innocent actions before the decision to leak was made that will come back to haunt the person afterwards. That includes emails. That also includes Google searches. That includes websites you visit (like this one). I’m not sure how to solve this, except that if you’ve been in contact with The Intercept, and then you decide to leak, send it to anybody but The Intercept.
By the way, the other thing that caught Reality Winner is the records they had of her accessing files and printing them on a printer. Depending where you work, they may have a record of every file you’ve accessed, every intranet page you visited. Because of the way printers put secret dots on documents, investigators know precisely which printer and time the document leaked to The Intercept was printed.
Photographs suffer the same problem: your camera and phone tag the photographs with GPS coordinates and time the photograph was taken, as well as information about the camera. This accidentally exposed John McAfee’s hiding location when Vice took pictures of him a few years ago. Some people leak by taking pictures of the screen — use a camera without GPS for this (meaning, a really old camera you bought from a pawnshop).
These examples should impress upon you the dangers of not understanding technology. As soon as you do something to evade surveillance you know about, you may get caught by surveillance you don’t know about.
If you nonetheless want to continue forward, the next step may be to get a “burner phone”. You can get an adequate Android “prepaid” phone for cash at the local Walmart, electronics store, or phone store.
There’s some problems with such phones, though. They can often be tracked back to the store that sold them, and the store will have security cameras that record you making the purchase. License plate readers and GPS tracking on your existing phone may also place you at that Walmart.
I don’t know how to resolve these problems. Perhaps the best is grow a beard and on the last day of your vacation, color your hair, take a long bike/metro ride (without your existing phone) to a store many miles away and pick up a phone, then shave and change your color back again. I don’t know — there’s a good chance any lame attempt you or I might think of has already been experienced by law enforcement, so they are likely ahead of you. Maybe ask your local drug dealer where they get their burner phones, and if they can sell you one. Of course, that just means when they get caught for drug dealing, they can reduce their sentence by giving up the middle class person who bought a phone from them.
Lastly, they may age out old security videos, so simply waiting six months before using the phone might work. That means prepaying for an entire year.
Note that I’m not going to link to examples of cheap burner phones on this page. Web browsers will sometimes prefetch some information from links in a webpage, so simply including links in this page can condemn you as having interest in burner phones. You are already in enough trouble for having visited this web page.
Burner phones have GPS. Newer the technology, like the latest Android LTE phones, have pretty accurate GPS that the police can query (without a warrant). If you take the phone home and turn it on, they’ll then be able to trace back the phone to your home. Carrying the phone around with you has the same problem, with the phone’s location correlating with your existing phone (which presumably you also carry) or credit card receipts. Rumors are that Petraeus was partly brought down by tracking locations where he used his credit card, namely, matching the hotel he was in with Internet address information.
Older phones that support 3G or even 2G have poorer GPS capabilities. They’ll still located you to the nearest cell tower, but not as accurately to your exact location.
A better strategy than a burner phone would be a burner laptop computer used with WiFi. You can get a cheap one for $200 at Amazon.com. My favorite are the 11 inch ones with a full sized keyboard and Windows 10. Better yet, get an older laptop for cash from a pawn shop.
You can install chat apps on this like “Signal Desktop”, “Wire Desktop”, or “WhatsApp” that will allow you to securely communicate. Or use “Discord”, which isn’t really encrypted, but it’s popular among gamers so therefore less likely to stand out. You can sit in a bar with free WiFi and a USB headset and talk to reporters without having a phone. If the reporter you want to leak to doesn’t have those apps (either on their own laptop or phone) then you don’t want to talk to them.
Needless to say, don’t cross the streams. Don’t log onto your normal accounts like Facebook. If you create fake Facebook accounts, don’t follow the same things. Better yet, configure your browser to discard all information (especially “cookies”) every time you log off, so you can’t be tracked. Install ad blockers, or use the “Brave” web browser, to remove even more trackers. A common trick among hackers is to change the “theme” to a red background, as a constant subliminal reminder that you using your dangerous computer, and never to do anything that identifies the real you.
Put tape over the camera. I’m not sure it’s a really big danger, but put tape over the camera. If they infect you enough to get your picture, they’ve also infected you enough to record any audio on your computer. Remember that proper encryption is end-to-end (they can’t eavesdrop in transit), but if they hack the ends (your laptop, or the reporter’s) they can still record the audio.
Note that when your burner laptop is in “sleep” mode, it can still be talking to the local wifi. Before taking it home, make sure it’s off. Go into the settings and configure it so that when the lid is closed, the computer is turned completely off.
It goes without saying: don’t use that burner laptop from home. Luckily, free wifi is everyone, so the local cafe, bar, or library can be used.
The next step is to also use a VPN or Tor to mask your Internet address. If there’s an active investigation into the reporter, they’ll get the metadata, the Internet address of the bar/cafe you are coming from. A good VPN provider or especially Tor will stop this. Remember that these providers increase latency, making phone calls a bit harder, but they are a lot safer.
Remember that Ross Ulbricht (owner of dark website market Silk Road) was caught in a library. They’d traced back his Internet address and grabbed his laptop out of his hands. Having it turn off (off off, not sleep off) when the lid is closed is one way to reduce this risk. Configuring your web browser to flush all cookies and passwords on restart is another. If they catch you in mid conversation with your secret contact, though, they’ll at least be able to hear your side of the conversation, and know who you are talking to.
The best measure, though it takes some learning, is “Tails live”. It’s a Linux distribution preconfigured with Tor and various secure chat apps that’ll boot from the USB or SD card. When you turn off the computer, nothing will be saved, so there will be no evidence saved to the disk for investigators to retrieve later.
While we are talking about Tor, it should be noted that many news organizations (NYTimes, Washington Post, The Intercept, etc.) support “SecureDrop” accessed only through Tor for receiving anonymous tips. Burner laptops you use from bars from Tails is the likely your most secure way of doing things.
The point of this post was not to provide a howto guide, but to discuss many of the technological issues involved. In a story about White House people investigating leaks, I’d like to see something in this technological direction. I’d like to know exactly how they were investigating leaks. Certainly, they were investigating all work computers, accounts, and phones. Where they also able to get to non-work computers, accounts, phones? Did they have law enforcement powers? What could they do about burner phones and laptops?
In any case, if you do want a howto guide, the discussion above should put some fear into you how easily you can inadvertently make a mistake.
Most likely you’ve read the tantalizing stories of big gains from investing in cryptocurrencies. Someone who invested $1,000 into bitcoins five years ago would have over $85,000 in value now. Alternatively, someone who invested in bitcoins three months ago would have seen their investment lose 20% in value. Beyond the big price fluctuations, currency holders are possibly exposed to fraud, bad business practices, and even risk losing their holdings altogether if they are careless in keeping track of the all-important currency keys.
It’s certain that beyond the rewards and risks, cryptocurrencies are here to stay. We can’t ignore how they are changing the game for how money is handled between people and businesses.
Some Advantages of Cryptocurrency
Cryptocurrency is accessible to anyone.
Decentralization means the network operates on a user-to-user (or peer-to-peer) basis.
Transactions can completed for a fraction of the expense and time required to complete traditional asset transfers.
Transactions are digital and cannot be counterfeited or reversed arbitrarily by the sender, as with credit card charge-backs.
There aren’t usually transaction fees for cryptocurrency exchanges.
Cryptocurrency allows the cryptocurrency holder to send exactly what information is needed and no more to the merchant or recipient, even permitting anonymous transactions (for good or bad).
Cryptocurrency operates at the universal level and hence makes transactions easier internationally.
There is no other electronic cash system in which your account isn’t owned by someone else.
On top of all that, blockchain, the underlying technology behind cryptocurrencies, is already being applied to a variety of business needs and itself becoming a hot sector of the tech economy. Blockchain is bringing traceability and cost-effectiveness to supply-chain management — which also improves quality assurance in areas such as food, reducing errors and improving accounting accuracy, smart contracts that can be automatically validated, signed and enforced through a blockchain construct, the possibility of secure, online voting, and many others.
Like any new, booming marketing there are risks involved in these new currencies. Anyone venturing into this domain needs to have their eyes wide open. While the opportunities for making money are real, there are even more ways to lose money.
We’re going to cover two primary approaches to staying safe and avoiding fraud and loss when dealing with cryptocurrencies. The first is to thoroughly vet any person or company you’re dealing with to judge whether they are ethical and likely to succeed in their business segment. The second is keeping your critical cryptocurrency keys safe, which we’ll deal with in this and a subsequent post.
Caveat Emptor — Buyer Beware
The short history of cryptocurrency has already seen the demise of a number of companies that claimed to manage, mine, trade, or otherwise help their customers profit from cryptocurrency. Mt. Gox, GAW Miners, and OneCoin are just three of the many companies that disappeared with their users’ money. This is the traditional equivalent of your bank going out of business and zeroing out your checking account in the process.
That doesn’t happen with banks because of regulatory oversight. But with cryptocurrency, you need to take the time to investigate any company you use to manage or trade your currencies. How long have they been around? Who are their investors? Are they affiliated with any reputable financial institutions? What is the record of their founders and executive management? These are all important questions to consider when evaluating a company in this new space.
Would you give the keys to your house to a service or person you didn’t thoroughly know and trust? Some companies that enable you to buy and sell currencies online will routinely hold your currency keys, which gives them the ability to do anything they want with your holdings, including selling them and pocketing the proceeds if they wish.
That doesn’t mean you shouldn’t ever allow a company to keep your currency keys in escrow. It simply means that you better know with whom you’re doing business and if they’re trustworthy enough to be given that responsibility.
Keys To the Cryptocurrency Kingdom — Public and Private
If you’re an owner of cryptocurrency, you know how this all works. If you’re not, bear with me for a minute while I bring everyone up to speed.
Cryptocurrency has no physical manifestation, such as bills or coins. It exists purely as a computer record. And unlike currencies maintained by governments, such as the U.S. dollar, there is no central authority regulating its distribution and value. Cryptocurrencies use a technology called blockchain, which is a decentralized way of keeping track of transactions. There are many copies of a given blockchain, so no single central authority is needed to validate its authenticity or accuracy.
The validity of each cryptocurrency is determined by a blockchain. A blockchain is a continuously growing list of records, called “blocks”, which are linked and secured using cryptography. Blockchains by design are inherently resistant to modification of the data. They perform as an open, distributed ledger that can record transactions between two parties efficiently and in a verifiable, permanent way. A blockchain is typically managed by a peer-to-peer network collectively adhering to a protocol for validating new blocks. Once recorded, the data in any given block cannot be altered retroactively without the alteration of all subsequent blocks, which requires collusion of the network majority. On a scaled network, this level of collusion is impossible — making blockchain networks effectively immutable and trustworthy.
The other element common to all cryptocurrencies is their use of public and private keys, which are stored in the currency’s wallet. A cryptocurrency wallet stores the public and private “keys” or “addresses” that can be used to receive or spend the cryptocurrency. With the private key, it is possible to write in the public ledger (blockchain), effectively spending the associated cryptocurrency. With the public key, it is possible for others to send currency to the wallet.
Cryptocurrency “coins” can be lost if the owner loses the private keys needed to spend the currency they own. It’s as if the owner had lost a bank account number and had no way to verify their identity to the bank, or if they lost the U.S. dollars they had in their wallet. The assets are gone and unusable.
The Cryptocurrency Wallet
Given the importance of these keys, and lack of recourse if they are lost, it’s obviously very important to keep track of your keys.
If you’re being careful in choosing reputable exchanges, app developers, and other services with whom to trust your cryptocurrency, you’ve made a good start in keeping your investment secure. But if you’re careless in managing the keys to your bitcoins, ether, Litecoin, or other cryptocurrency, you might as well leave your money on a cafe tabletop and walk away.
What Are the Differences Between Hot and Cold Wallets?
Just like other numbers you might wish to keep track of — credit cards, account numbers, phone numbers, passphrases — cryptocurrency keys can be stored in a variety of ways. Those who use their currencies for day-to-day purchases most likely will want them handy in a smartphone app, hardware key, or debit card that can be used for purchases. These are called “hot” wallets. Some experts advise keeping the balances in these devices and apps to a minimal amount to avoid hacking or data loss. We typically don’t walk around with thousands of dollars in U.S. currency in our old-style wallets, so this is really a continuation of the same approach to managing spending money.
A “hot” wallet, the Bread mobile app
Some investors with large balances keep their keys in “cold” wallets, or “cold storage,” i.e. a device or location that is not connected online. If funds are needed for purchases, they can be transferred to a more easily used payment medium. Cold wallets can be hardware devices, USB drives, or even paper copies of your keys.
A “cold” wallet, the Trezor hardware wallet
A “cold” wallet, the Ledger Nano S
A “cold” Bitcoin paper wallet
Wallets are suited to holding one or more specific cryptocurrencies, and some people have multiple wallets for different currencies and different purposes.
A paper wallet is nothing other than a printed record of your public and private keys. Some prefer their records to be completely disconnected from the internet, and a piece of paper serves that need. Just like writing down an account password on paper, however, it’s essential to keep the paper secure to avoid giving someone the ability to freely access your funds.
How to Keep your Keys, and Cryptocurrency Secure
In a post this coming Thursday, Securing Your Cryptocurrency, we’ll discuss the best strategies for backing up your cryptocurrency so that your currencies don’t become part of the millions that have been lost. We’ll cover the common (and uncommon) approaches to backing up hot wallets, cold wallets, and using paper and metal solutions to keeping your keys safe.
In the meantime, please tell us of your experiences with cryptocurrencies — good and bad — and how you’ve dealt with the issue of cryptocurrency security.
Bad software is everywhere. One can even claim that every software is bad. Cool companies, tech giants, established companies, all produce bad software. And no, yours is not an exception.
Who’s to blame for bad software? It’s all complicated and many factors are intertwined – there’s business requirements, there’s organizational context, there’s lack of sufficient skilled developers, there’s the inherent complexity of software development, there’s leaky abstractions, reliance on 3rd party software, consequences of wrong business and purchase decisions, time limitations, flawed business analysis, etc. So yes, despite the catchy title, I’m aware it’s actually complicated.
But in every “it’s complicated” scenario, there’s always one or two factors that are decisive. All of them contribute somehow, but the major drivers are usually a handful of things. And in the case of base software, I think it’s the fault of technical people. Developers, architects, ops.
We don’t seem to care about best practices. And I’ll do some nasty generalizations here, but bear with me. We can spend hours arguing about tabs vs spaces, curly bracket on new line, git merge vs rebase, which IDE is better, which framework is better and other largely irrelevant stuff. But we tend to ignore the important aspects that span beyond the code itself. The context in which the code lives, the non-functional requirements – robustness, security, resilience, etc.
We don’t seem to get security. Even trivial stuff such as user authentication is almost always implemented wrong. These days Twitter and GitHub realized they have been logging plain-text passwords, for example, but that’s just the tip of the iceberg. Too often we ignore the security implications.
“But the business didn’t request the security features”, one may say. The business never requested 2-factor authentication, encryption at rest, PKI, secure (or any) audit trail, log masking, crypto shredding, etc., etc. Because the business doesn’t know these things – we do and we have to put them on the backlog and fight for them to be implemented. Each organization has its specifics and tech people can influence the backlog in different ways, but almost everywhere we can put things there and prioritize them.
The other aspect is testing. We should all be well aware by now that automated testing is mandatory. We have all the tools in the world for unit, functional, integration, performance and whatnot testing, and yet many software projects lack the necessary test coverage to be able to change stuff without accidentally breaking things. “But testing takes time, we don’t have it”. We are perfectly aware that testing saves time, as we’ve all had those “not again!” recurring bugs. And yet we think of all sorts of excuses – “let the QAs test it”, we have to ship that now, we’ll test it later”, “this is too trivial to be tested”, etc.
And you may say it’s not our job. We don’t define what has do be done, we just do it. We don’t define the budget, the scope, the features. We just write whatever has been decided. And that’s plain wrong. It’s not our job to make money out of our code, and it’s not our job to define what customers need, but apart from that everything is our job. The way the software is structured, the security aspects and security features, the stability of the code base, the way the software behaves in different environments. The non-functional requirements are our job, and putting them on the backlog is our job.
You’ve probably heard that every software becomes “legacy” after 6 months. And that’s because of us, our sloppiness, our inability to mitigate external factors and constraints. Too often we create a mess through “just doing our job”.
And of course that’s a generalization. I happen to know a lot of great professionals who don’t make these mistakes, who strive for excellence and implement things the right way. But our industry as a whole doesn’t. Our industry as a whole produces bad software. And it’s our fault, as developers – as the only people who know why a certain piece of software is bad.
In a talk of his, Bob Martin warns us of the risks of our sloppiness. We have been building websites so far, but we are more and more building stuff that interacts with the real world, directly and indirectly. Ultimately, lives may depend on our software (like the recent unfortunate death caused by a self-driving car). And I’ll agree with Uncle Bob that it’s high time we self-regulate as an industry, before some technically incompetent politician decides to do that.
How, I don’t know. We’ll have to think more about it. But I’m pretty sure it’s our fault that software is bad, and no amount of blaming the management, the budget, the timing, the tools or the process can eliminate our responsibility.
Why do I insist on bashing my fellow software engineers? Because if we start looking at software development with more responsibility; with the fact that if it fails, it’s our fault, then we’re more likely to get out of our current bug-ridden, security-flawed, fragile software hole and really become the experts of the future.
To help protect their assets, many security-conscious enterprises require their system administrators to go through a “bastion” (or “jump”) host to gain administrative access to backend systems in protected or sensitive network segments.
A bastion host is a special-purpose instance that hosts a minimal number of administrative applications, such as RDP for Windows or Putty for Linux-based distributions. All other unnecessary services are removed. The host is typically placed in a segregated network (or “DMZ”), and is often protected with multi-factor authentication (MFA) and monitored with auditing tools. And most enterprises require that the access trail to the bastion host be auditable.
In this post, I demonstrate the use of Amazon AppStream 2.0 as a hardened and auto-scaled bastion host solution, and show how it could reduce the attack surface by stripping away the underlying OS and exposing only the necessary tools to system administrators that need access to protected network segments.
A Virtual Private Cloud (VPC) with a dedicated subnet for AppStream 2.0.
An existing Active Directory (AD) domain. This may be on premises, on AWS EC2 for Windows, or AWS Directory Service for Microsoft Active Directory used as a user directory.
Active Directory Federation Services (ADFS).
A Linux or Windows instance for which AppStream 2.0 will be acting as a bastion host.
Amazon AppStream 2.0 is a fully managed application streaming service that provides users instant access to their desktop applications from anywhere by using an HTML5-compatible desktop browser. When a user requests access to an application, AppStream 2.0 uses a base image to deploy a streaming instance and destroys the instance after the user closes their session. This ensures the same consistent experience during each logon.
You can use AppStream 2.0 as a bastion solution to enable your system administrators to manage their environment without giving them a full bastion host. Because AppStream 2.0 freshly builds instances each time a user requests access, a compromised instance will only last for the duration of a user session. As soon as the user closes their session and the Disconnect Timeout period is reached, AppStream 2.0 terminates the instance and, with it, you’ve reduced your risks of compromised instances.
You will also potentially reduce your costs because AppStream 2.0 has built-in auto-scaling to increase and decrease capacity based on user demand. It allows you to take advantage of the pay-as-you-go model, where you only pay for what you use.
High-level AppStream 2.0 architecture
The diagram below depicts a high-level AppStream 2.0 architecture used as a bastion host for servers in another VPC.
There are three VPCs shown: AppStream 2.0 VPC, Bastion host VPC, and application VPC. The AppStream 2.0 VPC is an AWS-owned VPC where the AppStream 2.0 maintains its infrastructure. Customers are not responsible for this VPC and have no access to it. AppStream 2.0 builds each streaming instance with two Elastic Network Interfaces (ENI); one in the AppStream 2.0 VPC and one in the VPC where you choose to deploy your AppStream 2.0 instances. The third VPC is the application VPC where you would typically keep your backend servers.
The diagram also depicts the end-user process to access the AppStream 2.0 environment, which works as follow:
Using an HTML5 desktop browser the user logs on to a Single Sign-On URL. This authenticates the user against the corporate directory using SAML 2.0 federation and with optional MFA.
After successful authentication, the user will see a list of provisioned applications.
The user can launch applications, such as RDP and Putty, which are only visible within the browser and with its underlying OS hidden. The user is then able to connect to the backend systems over the ports that were opened through security groups. The user logs off and AppStream 2.0 destroys the instance used for the session.
Figure 1: Architecture diagram
This walk-through assumes you have created the following resources as prerequisites.
A single VPC with a /23 CIDR range and two private subnets in two AZs.
Note: “private” subnet refers to a subnet that has no internet gateway (IGW) attached.
Bastion Subnets — used for the AppStream 2.0 instances that will be hosting the bastion applications.
Apps Subnets — used for the servers for which the AppStream 2.0 instances will be acting as a bastion host.
Figure 2: Screen shot of bastion and apps subnets
A peering connection to a VPC where the corporate Active Directory resides and with updated routing tables. This is only necessary if your AD resides in a different VPC.
Two EC2 instances with private IP addresses in the app subnet.
Phase 1: Create the DHCP Options Set
For the AppStream 2.0 instances to be able to join the corporate domain, they need to have their DNS entries point to the corporate domain controller(s). To accomplish this, you need to create a DHCP Options Set and assign it to the VPC:
Sign in to the AWS console, and then select VPC Dashboard > DHCP Option Sets > Create DHCP options set.
Give the DHCP Options Set a name, enter the domain name and DNS server(s) of your corporate domain controller(s), and then select Yes, Create.
Select your VPC Dashboard > your VPC > Actions > Edit DHCP Options Set.
Select the DHCP Options Set created in the previous step, and then select Save.
Figure 3: The “Edit DHCP Options Set” dialog
Phase 2: Create the AppStream 2.0 Stack
An AppStream 2.0 stack consists of a fleet, user access policies, and storage configuration. To create a stack, follow these steps:
Sign in to the AWS console and select AppStream 2.0 > Stack > Create Stack.
Give the stack a name, and then select Next.
Enable Home Folders, if you want persistent storage, and then select Review.
Figure 4: The “Enable Home Folders” dialog
Phase 3: Create the AppsStream 2.0 Directory Configuration
First create a directory configuration so you can join the AppStream 2.0 instances to an Organizational Unit (OU) in your corporate directory.
Note: AppStream 2.0 instances must be placed in an OU and can’t reside in the Computer Container.
To create a directory configuration, follow these steps:
Sign in to the AWS console and select AppStream 2.0 > Directory Configs > Create Directory Config.
Enter the following Directory Config information:
Directory name: The FQDN of your corporate domain.
Service Account Name: The account AppStream 2.0 uses to join the instances to the corporate domain. The required service account privileges are documented here.
Organizational Unit (OU): The OUs where AppStream 2.0 will create your instances. You can add additional OUs by clicking the plus (+) sign.
Select Next, and then select Create.
Create Security Groups
Now, create AWS security groups for your AppStream 2.0 instances and backend servers.
For your AppStream 2.0 instances, you must attach a “BastionHostSecurityGroup” in order to communicate to the backend servers. This security group is only used as a “source” by the security groups the backend servers are attached to and, therefore, they don’t require any inbound ports to be opened.
To create a security group, follow these steps:
Sign in to the AWS console and select VPC > Security Groups > Create Security Group.
Give your “BastionHostSecurityGroup” Security Group a name, select the VPC where you will place the AppStream 2.0 instances, and then select Yes, Create.
For your backend servers, you must attach a “BastionHostAccessSecurityGroup” that allows incoming traffic from the AppStream 2.0 instance. Unlike the “BastionHostSecurityGroup”, this one requires open inbound ports.
Sign in to the AWS console and select VPC > Security Groups > Create Security Group.
Give your “BastionHostAccessSecurityGroup” security group a name, select the correct VPC, and then select Yes, Create.
In the Security Group console, select the newly created security group, select the Inbound Rule tab, and then select Edit.
Add rules to open port 3389 and 22, use the previously-created security group as the source, and then select Save.
Figure 5: Opening ports 3389 and 22
Note: In addition to security groups, you can place Network ACLs (NACLs) around the subnet you use for AppStream 2.0 as an additional layer of security. The main differences between security groups and NACLs are that security groups are mandatory and you apply them to the instance level, while you apply NACLs to the subnet level and are optional. Another difference worth pointing out is that NACLs are “stateless” while security groups are “stateful.” This means that any port allowed inbound via NACLs will need a corresponding outbound rule. For more information on NACLs, refer to this documentation.
Phase 4: Build the AppStream 2.0 Image
An AppStream 2.0 image contains applications that you can stream to users. AppStream 2.0 uses the image to launch streaming instances that are part of an AppStream 2.0 fleet.
Once you have created the stack, create a custom image to make custom applications available to the users:
Sign in to the AWS console and select AppStream 2.0 > Images > Image Builder > Launch Image Builder.
Choose the image you want to use as a starting point, and then select Next. For this example, I chose a generic image from the General Purpose stock.
Figure 6: Choosing an image
Give your image a name, choose the instance family, and then select Next.
Choose the VPC and subnet you want to deploy the AppStream 2.0 instances in.
Select the security group you created for the AppStream 2.0 instances.
Select the directory configuration you created, the OU you want your AppStream 2.0 instances to reside in, and then select Review.
Once the image is built and in a running state, select the image, and then select Connect. This will open a new browser tab where you’ll be able to connect to and manage the image.
Select Administrator and log in.
Figure 7: Log in as a local administrator
Once logged in as administrator, select the Image Assistant shortcut on the desktop.
Figure 8: The Image Assistant shortcut on the desktop
Add all the applications you want to make available to your users for streaming, and then select Next.
Note: If you need to upload installation or configuration files, you can use the My Files option in the Control menu. Any files uploaded through this method will show up under the X: drive on the Image Builder.
Figure 9: The “Control” menu
If you want to test the applications as a non-privileged user, follow the on-screen instructions to switch the user. Otherwise, select Next.
Figure 10: “Switch User” on-screen instructions
Select Launch to have the Image Assistant optimize the applications.
Give the image a name, and then select Next.
Select Disconnect and Create Image.
Go back to the AppStream 2.0 console and wait for the “snapshotting” to complete and for the image to be in an available state before continuing to the next step.
Phase 5: Create the AppStream 2.0 Fleet
Once you create your Stack and image, you need to create a Fleet and associate it with your Stack.
AppStream 2.0 fleets consist of streaming instances that run the image that you specify. The fleet type determines when your instances run and how you pay for them. You can specify a fleet type when you create a fleet, and you can’t change them once they’ve been created.
To create a fleet, follow these steps:
Sign in to the AWS console and select AppStream 2.0 > Fleets > Create Fleet.
Give your fleet a name, and then select Next.
Select the newly created image, and then select Next.
Choose your preferred settings, and then select Next.
Important: Pay special attention to the Fleet capacity value. Fleet capacity determines the number of running instances you have at any given time, and it affects your costs.
Figure 11: The “Fleet capacity” dialog
Select your VPC, subnet(s), security Group(s), Active Directory settings, and then select Next.
Review the information, and then select Create.
Figure 12: Review your settings
Associate the fleet with the stack
Follow these steps:
Sign in to the AWS console and select AppStream 2.0 > Stacks.
Select the stack, select Actions, and then select Associate Fleet.
Select the fleet, and then select Associate.
Phase 6: Configure ADFS for AppStream 2.0
To have users authenticate against the corporate directory prior to accessing AppStream 2.0, use a Single Sign-On solution. For this demo, I use ADFS. If you choose another solution, follow the instructions that come with the solution. For help with setting up ADFS with AppStream 2.0, review Enabling Identify Federation with ADSF and Amazon Appstream 2.0.
This section shows you what the AppStream 2.0 end user experience is like when connecting to backend Windows and Linux instances.
Note: Make sure you have backend servers to connect to, as indicated in the prerequisites.
Access the ADFS URL that you created as part of the ADFS setup.
Sign in using your corporate credentials.
Select Remote Desktop from the list of applications.
Enter your corporate credentials.
Enter the private IP address of the backend windows instance you want to remote in to.
Figure 13: Enter the private IP address
You’re now logged on to the backend Windows instance through AppStream 2.0.
To test in Linux, open putty. Select the Launch app icon in the Control menu, and then select putty.
Figure 14: The “Control” menu showing putty
Provide the private IP address of a backend Linux host you want to connect to, and then select Open.
Note: For putty to connect to a Linux instance on AWS, you will need to provide a KeyPair. For information on how to configure putty and KeyPairs, refer to this documentation.
You’re now logged on to a backend Linux host through AppStream 2.0.
You can monitor AppStream 2.0 use by default with the following AWS monitoring services.
Amazon CloudWatch is a monitoring service for AWS cloud resources. You can use CloudWatch to collect and track metrics, collect and monitor log files, set alarms, and automatically react to changes in your AWS resources. For more information, refer to this documentation. Here’s a sample CloudWatch metric showing in-use capacity was 100% at 14:30, which indicates the Fleet capacity may need to be adjusted.
Figure 15: An example CloudWatch metric
AWS CloudTrail is a service that enables governance, compliance, operational auditing, and risk auditing of your AWS account. With CloudTrail, you can log, continuously monitor, and retain account activity related to actions across your AWS infrastructure. For more information, refer to this documentation. Here’s a sample CloudTrail event. For example, from this event you can see that user Bob logged on to AppStream 2.0 on March 4, 2018, and you can see his source IP.
Figure 16: An example CloudTrail event
Amazon AppStream 2.0 is a cost-effective way to provide administrators with a secure and auditable method to access their backend environments.
The AppStream 2.0 built-in auto-scaling feature offers a pay-as-you-go model, where the number of instances running is based on user demand. This allows you to keep costs down without compromising availability. Another cost-saving benefit of AppStream 2.0 is its underlying infrastructure being managed and maintained by AWS, so you can deploy AppStream 2.0 with minimal effort.
AppStream 2.0 helps with reducing the attack surface by hiding the shell of the streaming OS. This prevents administrators from interacting with executables that haven’t been made available to them through AppStream 2.0.
Another security benefit of AppStream 2.0 is that it destroys streaming instances after each use, reducing risks. This is a good mitigation strategy against compromised instances, as the lifespan of an instance is limited to the length of a user’s session.
AppStream 2.0 support for SAML provides yet another layer of security, allowing you to restrict access to SAML-federated URLs from corporate networks only, as well as the ability to enforce multi-factor authentication (MFA).
You can monitor the AppStream 2.0 environment through the use of AWS CloudTrail and Amazon CloudWatch, allowing you to monitor and trace the usage of AppStream 2.0.
For all of these reasons, AppStream 2.0 makes for a uniquely attractive bastion host solution.
For more information on the technologies mentioned in this blog, see the links below:
If you have comments about this post, submit them in the Comments section below. If you have questions about anything in this post, start a new thread on the Amazon AppStream 2.0 forum or contact AWS Support.
Want more AWS Security news? Follow us on Twitter.
We launched AWS Support a full decade ago, with Gold and Silver plans focused on Amazon EC2, Amazon S3, and Amazon SQS. Starting from that initial offering, backed by a small team in Seattle, AWS Support now encompasses thousands of people working from more than 60 locations.
A Quick Look Back Over the years, that offering has matured and evolved in order to meet the needs of an increasingly diverse base of AWS customers. We aim to support you at every step of your cloud adoption journey, from your initial experiments to the time you deploy mission-critical workloads and applications.
We have worked hard to make our support model helpful and proactive. We do our best to provide you with the tools, alerts, and knowledge that will help you to build systems that are secure, robust, and dependable. Here are some of our most recent efforts toward that goal:
Trusted Advisor S3 Bucket Policy Check – AWS Trusted Advisor provides you with five categories of checks and makes recommendations that are designed to improve security and performance. Earlier this year we announced that the S3 Bucket Permissions Check is now free, and available to all AWS users. If you are signed up for the Business or Professional level of AWS Support, you can also monitor this check (and many others) using Amazon CloudWatch Events. You can use this to monitor and secure your buckets without human intervention.
Personal Health Dashboard – This tool provides you with alerts and guidance when AWS is experiencing events that may affect you. You get a personalized view into the performance and availability of the AWS services that underlie your AWS resources. It also generates Amazon CloudWatch Events so that you can initiate automated failover and remediation if necessary.
Well Architected / Cloud Ops Review – We’ve learned a lot about how to architect AWS-powered systems over the years and we want to share everything we know with you! The AWS Well-Architected Framework provide proven, detailed guidance in critical areas including operational excellence, security, reliability, performance efficiency, and cost optimization. You can read the materials online and you can also sign up for the online training course. If you are signed up for Enterprise support, you can also benefit from our Cloud Ops review.
Infrastructure Event Management – If you are launching a new app, kicking off a big migration, or hosting a large-scale event similar to Prime Day, we are ready with guidance and real-time support. Our Infrastructure Event Management team will help you to assess the readiness of your environment and work with you to identify and mitigate risks ahead of time.
The Amazon retail site makes heavy use of AWS. You can read my post, Prime Day 2017 – Powered by AWS, to learn more about the process of preparing to sustain a record-setting amount of traffic and to accept a like number of orders.
Come and Join Us The AWS Support Team is in continuous hiring mode and we have openings all over the world! Here are a couple of highlights:
Near the top of most security recommendations is to use “strong passwords”. We need to stop doing this.
Yes, weak passwords can be a problem. If a website gets hacked, weak passwords are easier to crack. It’s not that this is wrong advice.
On the other hand, it’s not particularly good advice, either. It’s far down the list of important advice that people need to remember. “Weak passwords” are nowhere near the risk of “password reuse”. When your Facebook or email account gets hacked, it’s because you used the same password across many websites, not because you used a weak password.
Important websites, where the strength of your password matters, already take care of the problem. They use strong, salted hashes on the backend to protect the password. On the frontend, they force passwords to be a certain length and a certain complexity. Maybe the better advice is to not trust any website that doesn’t enforce stronger passwords (minimum of 8 characters consisting of both letters and non-letters).
To some extent, this “strong password” advice has become obsolete. A decade ago, websites had poor protection (MD5 hashes) and no enforcement of complexity, so it was up to the user to choose strong passwords. Now that important websites have changed their behavior, such as using bcrypt, there is less onus on the user.
But the real issue here is that “strong password” advice reflects the evil, authoritarian impulses of the infosec community. Instead of measuring insecurity in terms of costs vs. benefits, risks vs. rewards, we insist that it’s an issue of moral weakness. We pretend that flaws happen because people are greedy, lazy, and ignorant. We pretend that security is its own goal, a benefit we should achieve, rather than a cost we must endure.
We like giving moral advice because it’s easy: just be “stronger”. Discussing “password reuse” is more complicated, forcing us discuss password managers, writing down passwords on paper, that it’s okay to reuse passwords for crappy websites you don’t care about, and so on.
What I’m trying to say is that the moral weakness here is us. Rather then give pertinent advice we give lazy advice. We give the advice that victim shames them for being weak while pretending that we are strong.
So stop telling people to use strong passwords. It’s crass advice on your part and largely unhelpful for your audience, distracting them from the more important things.
User authentication is the functionality that every web application shared. We should have perfected that a long time ago, having implemented it so many times. And yet there are so many mistakes made all the time.
Part of the reason for that is that the list of things that can go wrong is long. You can store passwords incorrectly, you can have a vulnerably password reset functionality, you can expose your session to a CSRF attack, your session can be hijacked, etc. So I’ll try to compile a list of best practices regarding user authentication. OWASP top 10 is always something you should read, every year. But that might not be enough.
So, let’s start. I’ll try to be concise, but I’ll include as much of the related pitfalls as I can cover – e.g. what could go wrong with the user session after they login:
Store passwords with bcrypt/scrypt/PBKDF2. No MD5 or SHA, as they are not good for password storing. Long salt (per user) is mandatory (the aforementioned algorithms have it built in). If you don’t and someone gets hold of your database, they’ll be able to extract the passwords of all your users. And then try these passwords on other websites.
Use HTTPS. Period. (Otherwise user credentials can leak through unprotected networks). Force HTTPS if user opens a plain-text version.
Mark cookies as secure. Makes cookie theft harder.
Use CSRF protection (e.g. CSRF one-time tokens that are verified with each request). Frameworks have such functionality built-in.
Logout – let your users logout by deleting all cookies and invalidating the session. This makes usage of shared computers safer (yes, users should ideally use private browsing sessions, but not all of them are that savvy)
Session expiry – don’t have forever-lasting sessions. If the user closes your website, their session should expire after a while. “A while” may still be a big number depending on the service provided. For ajax-heavy website you can have regular ajax-polling that keeps the session alive while the page stays open.
Remember me – implementing “remember me” (on this machine) functionality is actually hard due to the risks of a stolen persistent cookie. Spring-security uses this approach, which I think should be followed if you wish to implement more persistent logins.
Forgotten password flow – the forgotten password flow should rely on sending a one-time (or expiring) link to the user and asking for a new password when it’s opened. 0Auth explain it in this post and Postmark gives some best pracitces. How the link is formed is a separate discussion and there are several approaches. Store a password-reset token in the user profile table and then send it as parameter in the link. Or do not store anything in the database, but send a few params: userId:expiresTimestamp:hmac(userId+expiresTimestamp). That way you have expiring links (rather than one-time links). The HMAC relies on a secret key, so the links can’t be spoofed. It seems there’s no consensus, as the OWASP guide has a bit different approach
One-time login links – this is an option used by Slack, which sends one-time login links instead of asking users for passwords. It relies on the fact that your email is well guarded and you have access to it all the time. If your service is not accessed to often, you can have that approach instead of (rather than in addition to) passwords.
Limit login attempts – brute-force through a web UI should not be possible; therefore you should block login attempts if they become too many. One approach is to just block them based on IP. The other one is to block them based on account attempted. (Spring example here). Which one is better – I don’t know. Both can actually be combined. Instead of fully blocking the attempts, you may add a captcha after, say, the 5th attempt. But don’t add the captcha for the first attempt – it is bad user experience.
Don’t leak information through error messages – you shouldn’t allow attackers to figure out if an email is registered or not. If an email is not found, upon login report just “Incorrect credentials”. On passwords reset, it may be something like “If your email is registered, you should have received a password reset email”. This is often at odds with usability – people don’t often remember the email they used to register, and the ability to check a number of them before getting in might be important. So this rule is not absolute, though it’s desirable, especially for more critical systems.
Consider using a 3rd party authentication – OpenID Connect, OAuth by Google/Facebook/Twitter (but be careful with OAuth flaws as well). There’s an associated risk with relying on a 3rd party identity provider, and you still have to manage cookies, logout, etc., but some of the authentication aspects are simplified.
For high-risk or sensitive applications use 2-factor authentication. There’s a caveat with Google Authenticator though – if you lose your phone, you lose your accounts (unless there’s a manual process to restore it). That’s why Authy seems like a good solution for storing 2FA keys.
I’m sure I’m missing something. And you see it’s complicated. Sadly we’re still at the point where the most common functionality – authenticating users – is so tricky and cumbersome, that you almost always get at least some of it wrong.
The U.S. Consumer Product Safety Commission (CPSC, Commission, or we) will conduct a public hearing to receive information from all interested parties about potential safety issues and hazards associated with internet-connected consumer products. The information received from the public hearing will be used to inform future Commission risk management work. The Commission also requests written comments.
Over the past several years, the last couple in particular, piracy has stormed millions of homes around the world.
From being a widespread but still fairly geeky occupation among torrenters, movie and TV show piracy can now be achieved by anyone with the ability to click a mouse or push a button on a remote control. Much of this mainstream interest can be placed at the feet of the Kodi media player.
An entirely legal platform in its own right, Kodi can be augmented with third-party add-ons that enable users to access an endless supply of streaming media. As such, piracy-configured Kodi installations are operated by an estimated 26 million people, according to the MPAA.
This popularity has led to much interest from tabloid newspapers in the UK which, for reasons best known to them, choose to both promote and demonize Kodi almost every week. While writing about news events is clearly par for the course, when one considers some of the reports, their content, and what inspired them, something doesn’t seem right.
This week The Express, which has published many overly sensational stories about Kodi in recent times, published another. The title – as always – promised something special.
Sounds like big news….
Reading the text, however, reveals nothing new whatsoever. The piece simply rehashes some of the historic claims that have been leveled at Kodi that can easily apply to any Internet-enabled software or system. But beyond that, some of its content is pretty weird.
The piece is centered on comments from two McAfee security experts – Chief Scientist Raj Samani and Chief Consumer Security Evangelist Gary Davis. It’s unclear whether The Express approached them for comment (if they did, there is no actual story for McAfee to comment on) or whether McAfee offered the comments and The Express built a story around them. Either way, here’s a taster.
“Kodi has been pretty open about the fact that it’s a streaming site but my view has always been if I use Netflix I know that I’m not going to get any issues, if I use Amazon I’m not going to get any issues,” Samani told the publication.
Ok, stop right there. Kodi admits that it’s a streaming site? Really? Kodi is a piece of software. It’s a media player. It can do many things but Kodi is not a streaming site and no one at Kodi has ever labeled it otherwise. To think that neither McAfee nor the publication caught that one is a bit embarrassing.
The argument that Samani was trying to make is that services like Netflix and Amazon are generally more reliable than third-party sources and there are few people out there who would argue with that.
“Look, ultimately you’ve got to do the research and you’ve got to decide if it’s right for you but personally I don’t use [Kodi] and I know full well that by not using [Kodi] I’m not going to get any issues. If I pay for the service I know exactly what I’m going to get,” he said.
But unlike his colleague who doesn’t use Kodi, Gary Davis has more experience.
McAfee’s Chief Consumer Security Evangelist admits to having used Kodi in the past but more recently decided not to use it when the security issues apparently got too much for him.
“I did use [Kodi] but turned it off as I started getting worried about some of the risks,” he told The Express.
“You may search for something and you may get what you are looking for but you may get something that you are not looking for and that’s where the problem lies with Kodi.”
This idea, that people search for a movie or TV show yet get something else, is bewildering to most experienced Kodi users. If this was indeed the case, on any large scale, people wouldn’t want to use it anymore. That’s clearly not the case.
Also, incorrect content appearing is not the kind of security threat that the likes of McAfee tend to be worried about. However, Davis suggests things can get worse.
“I’m not saying they’ve done anything wrong but if somebody is able to embed code to turn on a microphone or other things or start sending data to a place it shouldn’t go,” he said.
The sentence appears to have some words missing and struggles to make sense but the suggestion is that someone’s Kodi installation could be corrupted to the point that someone people could hijack the user’s microphone.
We are not aware of anything like that happening, ever, via Kodi. There are instances where that has happened completely without it in a completely different context, but that seems here nor there. By the same count, everyone should stop using Windows perhaps?
The big question is why these ‘scary’ Kodi non-stories keep getting published and why experts are prepared to weigh-in on them?
It would be too easy to quickly put it down to some anti-piracy agenda, even though there are plenty of signs that anti-piracy groups have been habitually feeding UK tabloids with information on that front. Indeed, a source at a UK news outlet (that no longer publishes such stories) told TF that they were often prompted to write stories about Kodi and streaming in general, none with a positive spin.
If Kodi is so bad as to warrant an article telling people to avoid it FOREVER on one day, why is it good enough to be promoted on another? It can only come down to the number of clicks – but the clickbait headline should’ve given that away at the start.
For the past several years most of the world has been waking up to the streaming piracy phenomenon, with pre-configured set-top boxes making inroads into millions of homes.
While other countries, notably the UK, arrested many individuals while warning of a grave and looming danger, complaints from the United States remained relatively low-key. It was almost as if the stampede towards convenient yet illegal streaming had caught the MPAA and friends by surprise.
In October 2017, things quickly began to change. The Alliance for Creativity and Entertainment sued Georgia-based Tickbox TV, a company selling “fully-loaded” Kodi boxes. In January 2018, the same anti-piracy group targeted Dragon Media, a company in the same line of business.
With this growing type of piracy now firmly on the radar, momentum seems to be building. Yesterday, a panel discussion on the challenges associated with piracy from streaming media boxes took place on Capitol Hill.
Hosted by the Information Technology and Innovation Foundation (ITIF), ‘Unboxing the Piracy Threat of Streaming Media Boxes’ went ahead with some big name speakers in attendance, not least Neil Fried, Senior Vice President, Federal Advocacy and Regulatory Affairs at the MPAA.
ITIF and various industry groups tweeted many interesting comments throughout the event. Kevin Madigan from Center for the Protection of Intellectual Property told the panel that torrent-based content “is becoming obsolete” in an on-demand digital environment that’s switching to streaming-based piracy.
While there’s certainly a transition taking place, 150 million worldwide torrent users would probably argue against the term “obsolete”. Nevertheless, the same terms used to describe torrent sites are now being used to describe players in the streaming field.
“There’s a criminal enterprise going on here that’s stealing content and making a profit,” Fried told those in attendance.
“The piracy activity out there is bad, it’s hurting a lot of economic activity & creators aren’t being compensated for their work,” he added.
Tom Galvin, Executive Director at the Digital Citizens Alliance, was also on the panel. Unsurprisingly, given the organization’s focus on the supposed dangers of piracy, Galvin took the opportunity to underline that position.
“If you go down the piracy road, those boxes aren’t following proper security protocols, there are many malware risks,” he said. It’s a position shared by Fried, who told the panel that “video piracy is the leading source of malware.”
Similar claims were made recently on Safer Internet Day but the facts don’t seem to back up the scare stories. Still, with the “Piracy is Dangerous” strategy already out in the open, the claims aren’t really unexpected.
What might also not come as a surprise is that ACE’s lawsuits against Tickbox and Dragon Media could be just a warm-up for bigger things to come. In the tweet embedded below, Fried can be seen holding a hexagonal-shaped streaming box, warning that the Department of Justice is now looking for candidates for criminal action.
Neil Fried of @MPAA with one of the streaming Kodi boxes leading to big piracy problems during Capitol Hill panel talk. Says DOJ looking at ‘variety of candidates’ for criminal action. @Comm_Dailypic.twitter.com/aYIRA4wgTC
What form this action will take when it arrives isn’t clear but when the DoJ hits targets on home soil, it tends to cherry-pick the most blatant of infringers in order to set an example with reasonably cut-and-dried cases.
Of course, every case can be argued but with hundreds of so-called “Kodi box” sellers active all over the United States, many of them clearly breaking the law as they, in turn, invite their customers to break the law, picking a sitting duck shouldn’t be too difficult.
And then, of course, we come to President Trump. Not usually that vocal on matters of intellectual property and piracy, yesterday – perhaps coincidentally, perhaps not – he suddenly delivered one of his “something is coming” tweets.
The U.S. is acting swiftly on Intellectual Property theft. We cannot allow this to happen as it has for many years!
Since you don’t have enough to worry about, here’s a paper postulating that space aliens could send us malware capable of destroying humanity.
Abstract: A complex message from space may require the use of computers to display, analyze and understand. Such a message cannot be decontaminated with certainty, and technical risks remain which can pose an existential threat. Complex messages would need to be destroyed in the risk averse case.
I think we’re more likely to be enslaved by malicious AIs.
Product security is an interesting animal: it is a uniquely cross-disciplinary endeavor that spans policy, consulting,
process automation, in-depth software engineering, and cutting-edge vulnerability research. And in contrast to many
other specializations in our field of expertise – say, incident response or network security – we have virtually no
time-tested and coherent frameworks for setting it up within a company of any size.
In my previous post, I shared some thoughts
on nurturing technical organizations and cultivating the right kind of leadership within. Today, I figured it would
be fitting to follow up with several notes on what I learned about structuring product security work – and about actually
making the effort count.
The “comfort zone” trap
For security engineers, knowing your limits is a sought-after quality: there is nothing more dangerous than a security
expert who goes off script and starts dispensing authoritatively-sounding but bogus advice on a topic they know very
little about. But that same quality can be destructive when it prevents us from growing beyond our most familiar role: that of
a critic who pokes holes in other people’s designs.
The role of a resident security critic lends itself all too easily to a sense of supremacy: the mistaken
belief that our cognitive skills exceed the capabilities of the engineers and product managers who come to us for help
– and that the cool bugs we file are the ultimate proof of our special gift. We start taking pride in the mere act
of breaking somebody else’s software – and then write scathing but ineffectual critiques addressed to executives,
demanding that they either put a stop to a project or sign off on a risk. And hey, in the latter case, they better
brace for our triumphant “I told you so” at some later date.
Of course, escalations of this type have their place, but they need to be a very rare sight; when practiced routinely, they are a telltale
sign of a dysfunctional team. We might be failing to think up viable alternatives that are in tune with business or engineering needs; we might
be very unpersuasive, failing to communicate with other rational people in a language they understand; or it might be that our tolerance for risk
is badly out of whack with the rest of the company. Whatever the cause, I’ve seen high-level escalations where the security team
spoke of valiant efforts to resist inexplicably awful design decisions or data sharing setups; and where product leads in turn talked about
pressing business needs randomly blocked by obstinate security folks. Sometimes, simply having them compare their notes would be enough to arrive
at a technical solution – such as sharing a less sensitive subset of the data at hand.
To be effective, any product security program must be rooted in a partnership with the rest of the company, focused on helping them get stuff done
while eliminating or reducing security risks. To combat the toxic us-versus-them mentality, I found it helpful to have some team members with
software engineering backgrounds, even if it’s the ownership of a small open-source project or so. This can broaden our horizons, helping us see
that we all make the same mistakes – and that not every solution that sounds good on paper is usable once we code it up.
Getting off the treadmill
All security programs involve a good chunk of operational work. For product security, this can be a combination of product launch reviews, design consulting requests, incoming bug reports, or compliance-driven assessments of some sort. And curiously, such reactive work also has the property of gradually expanding to consume all the available resources on a team: next year is bound to bring even more review requests, even more regulatory hurdles, and even more incoming bugs to triage and fix.
Being more tractable, such routine tasks are also more readily enshrined in SDLs, SLAs, and all kinds of other official documents that are often mistaken for a mission statement that justifies the existence of our teams. Soon, instead of explaining to a developer why they should fix a particular problem right away, we end up pointing them to page 17 in our severity classification guideline, which defines that “severity 2” vulnerabilities need to be resolved within a month. Meanwhile, another policy may be telling them that they need to run a fuzzer or a web application scanner for a particular number of CPU-hours – no matter whether it makes sense or whether the job is set up right.
To run a product security program that scales sublinearly, stays abreast of future threats, and doesn’t erect bureaucratic speed bumps just for the sake of it, we need to recognize this inherent tendency for operational work to take over – and we need to reign it in. No matter what the last year’s policy says, we usually don’t need to be doing security reviews with a particular cadence or to a particular depth; if we need to scale them back 10% to staff a two-quarter project that fixes an important API and squashes an entire class of bugs, it’s a short-term risk we should feel empowered to take.
As noted in my earlier post, I find contingency planning to be a valuable tool in this regard: why not ask ourselves how the team would cope if the workload went up another 30%, but bad financial results precluded any team growth? It’s actually fun to think about such hypotheticals ahead of the time – and hey, if the ideas sound good, why not try them out today?
Living for a cause
It can be difficult to understand if our security efforts are structured and prioritized right; when faced with such uncertainty, it is natural to stick to the safe fundamentals – investing most of our resources into the very same things that everybody else in our industry appears to be focusing on today.
I think it’s important to combat this mindset – and if so, we might as well tackle it head on. Rather than focusing on tactical objectives and policy documents, try to write down a concise mission statement explaining why you are a team in the first place, what specific business outcomes you are aiming for, how do you prioritize it, and how you want it all to change in a year or two. It should be a fluid narrative that reads right and that everybody on your team can take pride in; my favorite way of starting the conversation is telling folks that we could always have a new VP tomorrow – and that the VP’s first order of business could be asking, “why do you have so many people here and how do I know they are doing the right thing?”. It’s a playful but realistic framing device that motivates people to get it done.
In general, a comprehensive product security program should probably start with the assumption that no matter how many resources we have at our disposal, we will never be able to stay in the loop on everything that’s happening across the company – and even if we did, we’re not going to be able to catch every single bug. It follows that one of our top priorities for the team should be making sure that bugs don’t happen very often; a scalable way of getting there is equipping engineers with intuitive and usable tools that make it easy to perform common tasks without having to worry about security at all. Examples include standardized, managed containers for production jobs; safe-by-default APIs, such as strict contextual autoescaping for XSS or type safety for SQL; security-conscious style guidelines; or plug-and-play libraries that take care of common crypto or ACL enforcement tasks.
Of course, not all problems can be addressed on framework level, and not every engineer will always reach for the right tools. Because of this, the next principle that I found to be worth focusing on is containment and mitigation: making sure that bugs are difficult to exploit when they happen, or that the damage is kept in check. The solutions in this space can range from low-level enhancements (say, hardened allocators or seccomp-bpf sandboxes) to client-facing features such as browser origin isolation or Content Security Policy.
The usual consulting, review, and outreach tasks are an important facet of a product security program, but probably shouldn’t be the sole focus of your team. It’s also best to avoid undue emphasis on vulnerability showmanship: while valuable in some contexts, it creates a hypercompetitive environment that may be hostile to less experienced team members – not to mention, squashing individual bugs offers very limited value if the same issue is likely to be reintroduced into the codebase the next day. I like to think of security reviews as a teaching opportunity instead: it’s a way to raise awareness, form partnerships with engineers, and help them develop lasting habits that reduce the incidence of bugs. Metrics to understand the impact of your work are important, too; if your engagements are seen mostly as a yet another layer of red tape, product teams will stop reaching out to you for advice.
The other tenet of a healthy product security effort requires us to recognize at a scale and given enough time, every defense mechanism is bound to fail – and so, we need ways to prevent bugs from turning into incidents. The efforts in this space may range from developing product-specific signals for the incident response and monitoring teams; to offering meaningful vulnerability reward programs and nourishing a healthy and respectful relationship with the research community; to organizing regular offensive exercises in hopes of spotting bugs before anybody else does.
Oh, one final note: an important feature of a healthy security program is the existence of multiple feedback loops that help you spot problems without the need to micromanage the organization and without being deathly afraid of taking chances. For example, the data coming from bug bounty programs, if analyzed correctly, offers a wonderful way to alert you to systemic problems in your codebase – and later on, to measure the impact of any remediation and hardening work.
This is part one of a series. The second part will be posted later this week. Use the Join button above to receive notification of future posts in this series.
Though most of us have never set foot inside of a data center, as citizens of a data-driven world we nonetheless depend on the services that data centers provide almost as much as we depend on a reliable water supply, the electrical grid, and the highway system. Every time we send a tweet, post to Facebook, check our bank balance or credit score, watch a YouTube video, or back up a computer to the cloud we are interacting with a data center.
In this series, The Challenges of Opening a Data Center, we’ll talk in general terms about the factors that an organization needs to consider when opening a data center and the challenges that must be met in the process. Many of the factors to consider will be similar for opening a private data center or seeking space in a public data center, but we’ll assume for the sake of this discussion that our needs are more modest than requiring a data center dedicated solely to our own use (i.e. we’re not Google, Facebook, or China Telecom).
Data center technology and management are changing rapidly, with new approaches to design and operation appearing every year. This means we won’t be able to cover everything happening in the world of data centers in our series, however, we hope our brief overview proves useful.
What is a Data Center?
A data center is the structure that houses a large group of networked computer servers typically used by businesses, governments, and organizations for the remote storage, processing, or distribution of large amounts of data.
While many organizations will have computing services in the same location as their offices that support their day-to-day operations, a data center is a structure dedicated to 24/7 large-scale data processing and handling.
Depending on how you define the term, there are anywhere from a half million data centers in the world to many millions. While it’s possible to say that an organization’s on-site servers and data storage can be called a data center, in this discussion we are using the term data center to refer to facilities that are expressly dedicated to housing computer systems and associated components, such as telecommunications and storage systems. The facility might be a private center, which is owned or leased by one tenant only, or a shared data center that offers what are called “colocation services,” and rents space, services, and equipment to multiple tenants in the center.
A large, modern data center operates around the clock, placing a priority on providing secure and uninterrrupted service, and generally includes redundant or backup power systems or supplies, redundant data communication connections, environmental controls, fire suppression systems, and numerous security devices. Such a center is an industrial-scale operation often using as much electricity as a small town.
Types of Data Centers
There are a number of ways to classify data centers according to how they will be used, whether they are owned or used by one or multiple organizations, whether and how they fit into a topology of other data centers; which technologies and management approaches they use for computing, storage, cooling, power, and operations; and increasingly visible these days: how green they are.
Data centers can be loosely classified into three types according to who owns them and who uses them.
Exclusive Data Centers are facilities wholly built, maintained, operated and managed by the business for the optimal operation of its IT equipment. Some of these centers are well-known companies such as Facebook, Google, or Microsoft, while others are less public-facing big telecoms, insurance companies, or other service providers.
Managed Hosting Providers are data centers managed by a third party on behalf of a business. The business does not own data center or space within it. Rather, the business rents IT equipment and infrastructure it needs instead of investing in the outright purchase of what it needs.
Colocation Data Centers are usually large facilities built to accommodate multiple businesses within the center. The business rents its own space within the data center and subsequently fills the space with its IT equipment, or possibly uses equipment provided by the data center operator.
Backblaze, for example, doesn’t own its own data centers but colocates in data centers owned by others. As Backblaze’s storage needs grow, Backblaze increases the space it uses within a given data center and/or expands to other data centers in the same or different geographic areas.
Availability is Key
When designing or selecting a data center, an organization needs to decide what level of availability is required for its services. The type of business or service it provides likely will dictate this. Any organization that provides real-time and/or critical data services will need the highest level of availability and redundancy, as well as the ability to rapidly failover (transfer operation to another center) when and if required. Some organizations require multiple data centers not just to handle the computer or storage capacity they use, but to provide alternate locations for operation if something should happen temporarily or permanently to one or more of their centers.
Organizations operating data centers that can’t afford any downtime at all will typically operate data centers that have a mirrored site that can take over if something happens to the first site, or they operate a second site in parallel to the first one. These data center topologies are called Active/Passive, and Active/Active, respectively. Should disaster or an outage occur, disaster mode would dictate immediately moving all of the primary data center’s processing to the second data center.
While some data center topologies are spread throughout a single country or continent, others extend around the world. Practically, data transmission speeds put a cap on centers that can be operated in parallel with the appearance of simultaneous operation. Linking two data centers located apart from each other — say no more than 60 miles to limit data latency issues — together with dark fiber (leased fiber optic cable) could enable both data centers to be operated as if they were in the same location, reducing staffing requirements yet providing immediate failover to the secondary data center if needed.
This redundancy of facilities and ensured availability is of paramount importance to those needing uninterrupted data center services.
Leadership in Energy and Environmental Design (LEED) is a rating system devised by the United States Green Building Council (USGBC) for the design, construction, and operation of green buildings. Facilities can achieve ratings of certified, silver, gold, or platinum based on criteria within six categories: sustainable sites, water efficiency, energy and atmosphere, materials and resources, indoor environmental quality, and innovation and design.
Green certification has become increasingly important in data center design and operation as data centers require great amounts of electricity and often cooling water to operate. Green technologies can reduce costs for data center operation, as well as make the arrival of data centers more amenable to environmentally-conscious communities.
The ACT, Inc. data center in Iowa City, Iowa was the first data center in the U.S. to receive LEED-Platinum certification, the highest level available.
ACT Data Center exterior
ACT Data Center interior
Factors to Consider When Selecting a Data Center
There are numerous factors to consider when deciding to build or to occupy space in a data center. Aspects such as proximity to available power grids, telecommunications infrastructure, networking services, transportation lines, and emergency services can affect costs, risk, security and other factors that need to be taken into consideration.
The size of the data center will be dictated by the business requirements of the owner or tenant. A data center can occupy one room of a building, one or more floors, or an entire building. Most of the equipment is often in the form of servers mounted in 19 inch rack cabinets, which are usually placed in single rows forming corridors (so-called aisles) between them. This allows staff access to the front and rear of each cabinet. Servers differ greatly in size from 1U servers (i.e. one “U” or “RU” rack unit measuring 44.50 millimeters or 1.75 inches), to Backblaze’s Storage Pod design that fits a 4U chassis, to large freestanding storage silos that occupy many square feet of floor space.
Location will be one of the biggest factors to consider when selecting a data center and encompasses many other factors that should be taken into account, such as geological risks, neighboring uses, and even local flight paths. Access to suitable available power at a suitable price point is often the most critical factor and the longest lead time item, followed by broadband service availability.
With more and more data centers available providing varied levels of service and cost, the choices increase each year. Data center brokers can be employed to find a data center, just as one might use a broker for home or other commercial real estate.
Websites listing available colocation space, such as upstack.io, or entire data centers for sale or lease, are widely used. A common practice is for a customer to publish its data center requirements, and the vendors compete to provide the most attractive bid in a reverse auction.
Business and Customer Proximity
The center’s closeness to a business or organization may or may not be a factor in the site selection. The organization might wish to be close enough to manage the center or supervise the on-site staff from a nearby business location. The location of customers might be a factor, especially if data transmission speeds and latency are important, or the business or customers have regulatory, political, tax, or other considerations that dictate areas suitable or not suitable for the storage and processing of data.
Local climate is a major factor in data center design because the climatic conditions dictate what cooling technologies should be deployed. In turn this impacts uptime and the costs associated with cooling, which can total as much as 50% or more of a center’s power costs. The topology and the cost of managing a data center in a warm, humid climate will vary greatly from managing one in a cool, dry climate. Nevertheless, data centers are located in both extremely cold regions and extremely hot ones, with innovative approaches used in both extremes to maintain desired temperatures within the center.
Geographic Stability and Extreme Weather Events
A major obvious factor in locating a data center is the stability of the actual site as regards weather, seismic activity, and the likelihood of weather events such as hurricanes, as well as fire or flooding.
Backblaze’s Sacramento data center describes its location as one of the most stable geographic locations in California, outside fault zones and floodplains.
Sometimes the location of the center comes first and the facility is hardened to withstand anticipated threats, such as Equinix’s NAP of the Americas data center in Miami, one of the largest single-building data centers on the planet (six stories and 750,000 square feet), which is built 32 feet above sea level and designed to withstand category 5 hurricane winds.
Equinix “NAP of the Americas” Data Center in Miami
Most data centers don’t have the extreme protection or history of the Bahnhof data center, which is located inside the ultra-secure former nuclear bunker Pionen, in Stockholm, Sweden. It is buried 100 feet below ground inside the White Mountains and secured behind 15.7 in. thick metal doors. It prides itself on its self-described “Bond villain” ambiance.
Bahnhof Data Center under White Mountain in Stockholm
Usually, the data center owner or tenant will want to take into account the balance between cost and risk in the selection of a location. The Ideal quadrant below is obviously favored when making this compromise.
Risk mitigation also plays a strong role in pricing. The extent to which providers must implement special building techniques and operating technologies to protect the facility will affect price. When selecting a data center, organizations must make note of the data center’s certification level on the basis of regulatory requirements in the industry. These certifications can ensure that an organization is meeting necessary compliance requirements.
Electrical power usually represents the largest cost in a data center. The cost a service provider pays for power will be affected by the source of the power, the regulatory environment, the facility size and the rate concessions, if any, offered by the utility. At higher level tiers, battery, generator, and redundant power grids are a required part of the picture.
Fault tolerance and power redundancy are absolutely necessary to maintain uninterrupted data center operation. Parallel redundancy is a safeguard to ensure that an uninterruptible power supply (UPS) system is in place to provide electrical power if necessary. The UPS system can be based on batteries, saved kinetic energy, or some type of generator using diesel or another fuel. The center will operate on the UPS system with another UPS system acting as a backup power generator. If a power outage occurs, the additional UPS system power generator is available.
Many data centers require the use of independent power grids, with service provided by different utility companies or services, to prevent against loss of electrical service no matter what the cause. Some data centers have intentionally located themselves near national borders so that they can obtain redundant power from not just separate grids, but from separate geopolitical sources.
Higher redundancy levels required by a company will of invariably lead to higher prices. If one requires high availability backed by a service-level agreement (SLA), one can expect to pay more than another company with less demanding redundancy requirements.
Stay Tuned for Part 2 of The Challenges of Opening a Data Center
That’s it for part 1 of this post. In subsequent posts, we’ll take a look at some other factors to consider when moving into a data center such as network bandwidth, cooling, and security. We’ll take a look at what is involved in moving into a new data center (including stories from Backblaze’s experiences). We’ll also investigate what it takes to keep a data center running, and some of the new technologies and trends affecting data center design and use. You can discover all posts on our blog tagged with “Data Center” by following the link https://www.backblaze.com/blog/tag/data-center/.
The second part of this series on The Challenges of Opening a Data Center will be posted later this week. Use the Join button above to receive notification of future posts in this series.
The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.