Realtime Robotics’ motion-planning processor helps autonomous cars make better decisions
About two years ago, we covered a research project from Duke University that sped up motion planning for a tabletop robot arm by several orders of magnitude. The robot relied on a custom processor to do in milliseconds what normally takes seconds. The Duke researchers formed a company based on this tech called Realtime Robotics, and recently they’ve been focused on applying it to autonomous vehicles.
The reason that you should care about fast motion planning for autonomous vehicles is because motion planning encompasses the process by which the vehicle decides what it’s going to do next. Making this process faster doesn’t just mean that the vehicle can make decisions more quickly, but that it can make much better decisions as well—keeping you, and everyone around you, as safe as possible.
Lincoln and Amazon are the latest companies to harness smartphone apps for easy vehicle entry
Roll-up windows. Cigarette lighters. Physical ignition keys. All of these features have gone virtually extinct in modern automobiles. The quaint metal key gave way to transponder fobs, which led to “proximity keys” that don’t leave your pocket at all. Now, smartphones are becoming the new gatekeepers, as car companies roll out features that let drivers unlock and start their cars through an app.
Volvo began offering its subscription-based On Call service in 2016; it allows owners to use the company’s smartphone app to lock, unlock, and start their cars. With it, users can also remotely check vehicle fuel levels, receive service alerts, or send destinations to the onboard navigation system.
Tesla, which has never been shy about beta testing on its customers, then attempted to sell a car with no fob at all: The Model 3 sedan was initially available with only smartphone-based entry and ignition, and a backup RFID card. Beset by customer complaints of spotty operation, Tesla began offering a familiar fob last fall for an extra US $150. Yet that fob didn’t allow the “passive entry” of the smartphone system, requiring owners to push a button to enter or lock the car.
Now it’s Lincoln’s turn. Ford’s luxury division will bring two SUVs to market this year, the midsize 2020 Aviator and compact 2020 Corsair. Traditionalists will still receive a standard fob, but adventurous types can pay extra for the Lincoln’s app-based, optional “Phone as a Key” system.
Post Syndicated from Lawrence Ulrich original https://spectrum.ieee.org/transportation/advanced-cars/2019s-top-10-tech-cars-ram-1500
Frugal with fuel in stop-and-go traffic
The 2019 Ram has been garnering early praise for several technologies unheard of in full-size pickup trucks: a Tesla-like touch screen, a coil-spring rear suspension and self-leveling air suspension. But its best tech trick is under the hood: mild hybrid power. It’s called eTorque, and it’s standard on every V-6 Ram and an option on Hemi V-8 models.
Mild hybrids can’t propel themselves on electricity alone, but they can supplement gasoline power and trim fuel consumption. On the Ram, a liquid-cooled motor/generator connects to the Pentastar V-6’s crankshaft to deliver an electric boost of 8.9 kilowatts (12 horsepower) and as many as 122 newton meters (90 pound-feet) of torque. It’s powered by a 48-volt electrical system, the new wave in automotive electricals, with a DC/DC converter and a compact, 0.4-kilowatt-hour lithium-ion battery.
That 48-V system permits the use of engine stop/start tech that cycles so seamlessly that it’s nearly undetectable: The Ram rolls from stoplights under electric power before it cranks the gasoline engine to whispery life, without the shuddering or noise that make typical stop/start systems so annoying.
Throw in an incredibly creamy ride, and a back seat (in Crew Cab models) with more legroom than any full-size luxury sedan, and you realize how far we’ve come from the days when the General Motors GMT 400 was hailed for having independent front suspension with torsion bars.
Ram says the eTorque system saves 5 centiliters (1.7 ounces) of fuel for every 90-second stop. Do that just 10 times a day and you’re conserving 190 liters (50 gallons) of fuel a year. It also saves energy through regenerative hybrid brakes. The latest, 227-kW (305-hp) Pentastar V-6 adds variable intake-valve lift and cam phasing that can run the efficient Atkinson combustion cycle, familiar from hybrids like the Toyota Prius. The 295-kW (395-hp) Hemi V-8 adds its own goodies, including fuel-saving cylinder deactivation, electronic mass dampers on frame rails and active cabin-noise cancellation, the latter two techs designed to erase telltale vibrations when the Ram runs on just four cylinders.
The upshot is the kind of fuel economy once associated with family cars. The V-6 Ram has an EPA fuel economy of 12.4 liters/100 kilometers (19 miles per gallon) on local roads and 9.8 L/100 km (24 mpg) on the highway, and an unmatched driving range of 1,000 km (624 miles) on a tank of gasoline. Even the burly V-8 eTorque model manages up to 17/23 mpg, in a truck that can tow a whopping 5,783 kilograms, or approximately one African bull elephant.
Post Syndicated from Lawrence Ulrich original https://spectrum.ieee.org/transportation/advanced-cars/2019s-top-10-tech-cars-ferrari-488-pista
It’s a turbo with the satisfying sound of an air-breathing monster
When I get my first knee-wobbling glimpse of the 488 Pista and its zesty racing stripes, I’m not thinking about technology, I have to confess. But when I strap aboard the Pista at Ferrari’s fabled Fiorano test circuit in Maranello, Italy, I’m soon saying grazie for the sheer technical prowess of the fastest V-8 Ferrari ever produced.
A midmounted, dry-sump, twin-turbo V-8 spools up 529 kilowatts (710 horsepower) from just 3.9 liters of displacement, in a Ferrari that weighs just 1,382 kilograms (3,047 pounds). That’s 6 percent lighter than a 488 GTB, the standard version of Ferrari’s midengine marvel. The diet that slimmed down the Pista included carbon-fiber wheels that weigh 40 percent less than standard rims.
The result is a new idea of insanity in a street-legal Ferrari: The 0-to-100-kilometer-per-hour run (62 miles per hour) takes 2.85 seconds. You get to 200 km/h (124 mph) in 7.6 seconds, which is faster than many cars take to reach 100 km/h.
Engineers trimmed 23 kg (50 lbs) from the engine alone, using carbon-fiber intake plenums and titanium connecting rods, just like in Ferrari’s F1 racers. The engine’s total rotational inertia—created by its moving parts and by friction—is reduced by 17 percent for faster, more-joyous revving. The Inconel-alloy exhaust manifold is just 1 millimeter wide at its thinnest sections, and it saves nearly 9 kg (20 lbs). The design minimizes energy losses incurred when the engine pumps out exhaust. It also helps deliver the fortissimo sound that went missing in the GTB, a major challenge as supercars switch en masse to more-efficient turbocharged power plants.
Engineers also added more “color” to the sound of the engine by augmenting the richer, more-pleasing frequencies. Turbocharger speed sensors on each cylinder bank measure how well it’s working in real time to enable engine controllers to maximize power, regardless of altitude or ambient temperature.
The Ferrari takes aerodynamic and handling cues from Ferrari Challenge racers, along with the 488 GTBs that have dominated FIA Endurance Racing. Compared with a standard GTB, the Pista enjoys a huge 20 percent gain in aero efficiency, including up to 240 kg (529 lbs) of downforce at 200 km/h (124 mph). Giant carbon-ceramic brakes feel strong enough to halt Earth’s rotation. The S-Duct, a Ferrari showroom first, channels air through the front fascia and over the hood to clamp front tires to the road surface. Front radiators are inverted and canted rearward to direct hot air along the underbody but well away from side intercooler intakes.
As in Challenge cars, the engine is actually fed from the rear, where air intakes mounted just below the rear spoiler take advantage of the high-pressure atmosphere there; the 488’s signature cleavages in rear fenders are now put to use feeding air into turbo intercoolers and cooling the engine bay. The rear diffuser incorporates three active flaps that can rotate up to 14 degrees to minimize drag, hastening runs to the car’s top speed of 340 km/h (211 mph). The result is a track-day carnival.
The car’s coolest hand-me-down from racing tech may be the new “wall effect” rev limiter. Traditional engine-speed limiters, Ferrari says, cut off the fuel well before the engine gets to its redline. In the Pista, there’s no sudden slump in power, the dispiriting thrustus interruptus that you feel when a car’s engine bangs off the rev limiter. Instead, the Ferrari continues to accelerate right up to the engine’s peak, and holds it there. All 710 of these prancing ponies are on tap, anywhere from 6,750 rpm to the 8,000-rpm redline.
Ferrari will build just 500 Pistas for the world’s consumption. If only technology could make the Pista multiply while sharply reducing the price.
Post Syndicated from Lawrence Ulrich original https://spectrum.ieee.org/transportation/advanced-cars/2019s-top-10-tech-cars-jaguar-ipace-etrophy
A production EV, refitted for the racetrack
Making an electric car go fast in a straight line on the street is one thing; making one that can withstand the rigors of racing is another. Jaguar’s I-Pace eTrophy showed the difference in December by kicking off the world’s first all-electric, production-based championship with a historic run in Saudi Arabia. That eTrophy series, a global undercard for the FIA’s Formula E, included pro women—such as Katherine Legge with the Rahal Letterman Lanigan Racing team—competing against men in the kingdom for the first time.
I did my laps in that same car at the Silverstone circuit in the United Kingdom. And if the I-Pace isn’t the fastest race car I’ve tested—give this technology time, please—it was definitely the quietest. As I shot around Silverstone’s Stowe Circuit, I could barely hear the motor’s gentle whir above the groans of the suspension and the rumbling of the 56-centimeter (22-inch) Michelin Pilot Sport tires.
Jaguar’s Special Vehicle Operations has built 20 of these groundbreaking racers, which use as a starting point the same electric I-Pace you see in showrooms. The racers have the same 90-kilowatt-hour battery, for example, and also the same dual synchronous motors with 294 kilowatts (394 horsepower) and 700 newton meters of torque. Despite a 610-kilogram battery, the racers trim 225 kilos from the road-going version’s hefty 1,965 kg.
In this newfangled series, durability and safety matter as much as performance: Motors are designed to last for the series’ entire three years, or 30 races, with only the battery getting replaced after each season. The Terra 51 Charge Station, a DC fast charger with roughly 50 kW of power, is custom-designed to be compact and easily portable, for transport and racing. Those races last for 25 minutes plus one lap, or roughly 100 kilometers (62 miles), depending on the course.
I’m strapped into the car, sharing its safety cage with the battery, which is cradled inboard for safety. Gasoline fire has always been a prime danger in racing, but the Jaguar’s electrical system, with 389 volts and up to 550 amperes, brings its own challenges: If I crash, I’ll need to toggle separate switches that trigger two levels of electrical isolation and minimize the chance I’ll be electrocuted. Lights on the Jag’s exterior and dash glow green when there’s no live electricity but glow blue or red for potential threats. In that case, emergency workers are equipped with rubber gloves, mats, and specialized gear to aid the driver and car.
To maximize cooling, the race version has hood and fascia openings larger than those in the showroom car, as well as double the A/C system capacity, the better to cool the battery pack. Regenerative brakes create up to 0.4 g’s of deceleration. When the battery is fully charged, no energy recovery is possible, but as the race proceeds, drivers can adjust the regenerative brake over 11 settings. The results are pretty impressive for a two-metric-ton crossover: The racing I-Pace scoots to 97 km/h (60 mph) in 4.5 seconds, with a top speed twice as high, at 121 mph.
But this competition is also about putting Jaguar on the EV map, according to James Barclay, director for the Panasonic Jaguar Racing Team. “We have 9,500 engineers in [England] developing this technology,” Barclay says. “As a British company, we’re the first to have gone racing with our electric car, and we’re very proud of that.”
Post Syndicated from Lawrence Ulrich original https://spectrum.ieee.org/transportation/advanced-cars/2019s-top-10-tech-cars-lexus-es-350
Who needs mirrors when you’ve got cameras?
For years, automakers have teased us with sleek concept cars that ditch conventional side mirrors in favor of digital cameras. Now, Lexus becomes the first automaker to bring the technology to showrooms, albeit only in Japan for now: The all-new ES 350 sedan will beat the Audi E-tron’s “virtual mirrors” to market by mere months.
Lexus calls theirs the Digital Side-View Monitor: a pair of slender exterior stalks housing digital cameras that beam a high-definition view to 5.0-inch LCD screens mounted on either side of the cabin. The main benefit is to save fuel by reducing the aerodynamic drag of chunky conventional mirrors and damping the wind noise that’s been a longtime challenge for car designers. Lexus says the cameras and interior monitors deliver better visibility in foul weather, with the lenses heated and tucked into housings to ward off raindrops, ice, or snow. Drivers can adjust screen perspectives, or the camera can automatically adjust— zooming in when drivers hit their turn signals, or highlighting and alerting to cars that loom in blind spots.
Now, if only American motorists could get a peek. Though regulators in Japan and Europe have already approved the digital cameras, U.S. regulations prohibit them.
Post Syndicated from Lawrence Ulrich original https://spectrum.ieee.org/transportation/advanced-cars/2019s-top-10-tech-cars-bmw-3series
Reverse assistant gets you right back where you started from
BMW’s franchise model, its 3-Series sport sedan, has been slipping in sales and reputation. It’s been hammered lately by the Alfa Romeo Giulia, among other sporty chariots. But an all-new 3-Series has armored itself for the battle with loads of new tech. First things first: The new 3-Series is fun to drive, in the way BMW fans demand. I learn this firsthand in Portugal while romping an M340i on the devilish Portimao circuit.
The car’s 285 kilowatts (382 horsepower) are a nearly 20 percent jump from last year, even though the car has a similar 3.0-liter in-line 6 at its core. The upgrades focus on the turbocharger: Fuel-injection pressure is nearly doubled, and the twin-scroll unit is lighter and more efficient, inhaling exhaust gas at a toasty 1,000 °C.
When drivers aren’t pushing the BMW’s limits, they can relax and enjoy the tech treats. Semiautonomous functions, including hands-off driving on highways, are managed through radar and three bundled cameras from MobilEye, an Israel-based Intel subsidiary that produces imagers and software for driver-assistance systems. Another feature, called Narrow Passage Support, keeps the BMW centered in white-knuckle maneuvers, such as driving between giant trucks. The built-in Intelligent Personal Assistant follows orders like an onboard Siri. Blue-laser-powered high beams can illuminate the road ahead for a remarkable 600 meters. And a plug-in hybrid version, designated 330e, heads to showrooms in 2020.
Now, let’s back up and talk about Reversing Assistant: At speeds below 35 kilometers per hour (22 miles per hour), the BMW records the car’s path in continuous 50-meter increments. Go ahead, drive along that dead-end forest path, a steep uphill driveway, the trickiest courtyard or parking garage. You can even park the BMW and get a good night’s sleep. When you return, the BMW will automatically steer itself to mirror the inbound path at up to 5.5 km/h, with the driver touching only the brake and throttle. The system worked like a charm, precisely reversing and scanning for pedestrians, cars, or obstacles, while I monitored its progress on a display screen; all without me having to crane my neck and worry about crunching into something. If every car had the BMW’s system, insurance claims due to reverse maneuvers gone wrong would drop faster than a shorn-off side mirror.
The automaker’s new wheels-eye-view feature gets rid of the ultimate blind spot: the point where the rubber meets the road
Venturing off-road in a four-wheel-drive SUV can whiten the knuckles of even experienced drivers. So, even as engineers race to bring self-driving cars to paved roads, Land Rover is harnessing similar technology to help vehicles navigate the roughest terrain. The British automaker’s latest advance, which it calls Clear Sight Ground View, provided a literal eye-opener in Greece during my test of the sumptuous new Range Rover Evoque.
Like every new bit of semiautonomous tech, the groundbreaking system expands on an existing suite of hardware, sensors, and software. The latest Rovers could already robotically manage their acceleration, braking, suspensions, and four-wheel-drive traction well enough to tackle daunting climbs and descents and traverse terrain that would rattle the nerves of off-road novices. All that was left for the human driver was to steer, which reduced the mental and physical workload. Now, Clear Sight Ground View removes even more stress and uncertainty from the equation. At speeds up to 30 kilometers per hour (18 miles per hour), forward-facing cameras mounted on the side mirrors and front grille deliver a 180-degree view of the front wheels and the ground beneath and ahead of the vehicle—a perspective that’s typically obscured by the hood of a high-riding SUV.
This real-time view is fed to the Evoque’s center touch screen, along with a graphical, transparent overlay of the vehicle’s sides and corners. Computer controls slightly delay the images from the front camera and stitch them together with side-mirror views to create a seamless real-time feed. Rover engineers say the system makes the Evoque’s hood—or, these being Brits, its “bonnet”—virtually see-through. It’s definitely like having another set of eyes outside the vehicle. And that’s precisely the point.
Proponents of autonomous cars talk about replacing fallible drivers, but the Rover system could help replace another occupation: the spotter that traditionally guides an off-road adventurer through particularly tricky off-road obstacles. When I aimed the Evoque up a steep, narrow goat trail overlooking the Aegean Sea, the virtual view showed precisely where my front wheels were pointed, as well as the rocks and ruts I needed to clamber over or steer around. This eliminated any need to have a passenger climb out and hand-signal me as I picked my way up the slope. The system again came in handy on a vertigo-inducing crossing of a rusting, out-of-service railway bridge that spans the Corinth Canal, the spectacular shipping channel that separates the Greek mainland from the Peloponnese peninsula. The screen view offered precise, reassuring confirmation that my wheels were straddling the railroad tracks, and that I wasn’t at risk of making a roughly 90-meter plunge to the azure waters below.
Speaking of water, the all-new Evoque can ford bodies of water up to 60 centimeters (23.6 inches) deep—or 9.9 cm deeper than earlier models could withstand. To be on the safe side, a new ultrasonic sensor measures and displays the water’s depth as you proceed, again bypassing the need for guesswork, a pair of waders, or a sharp stick to poke into a rushing stream.
It’s all very clever. But for this off-roading fan, foolproofing the experience with electronic guardian angels raises some important questions: Isn’t doing it yourself half (or even more than half) the fun of trekking an SUV through challenging terrain? Isn’t experiencing the attendant sense of adventure and overcoming obstacles skillfully the point of the exercise? Would Lewis and Clark have preferred a mountain-lion sensor and a set of Google Maps? (OK, it’s a definite “yes” to the third question.)
Nathan Hoyt, U.S. communications chief at Jaguar Land Rover, the marque’s parent company, notes that engaging the various helpers is optional. Owners are welcome to do it the old-fashioned way, relying on their own skills and senses when traveling off-road. But expensive as Range Rovers are, Clear Sight Ground View will likely find its primary use in keeping them pristine when negotiating urban and suburban obstacles such as narrow passages or tall curbs. The viewpoint it provides lowers the odds of scraping a pricey alloy wheel on a curb or dinging a body panel on the base of a light post in a shopping mall parking lot.
Furthermore, says Hoyt, “the [new suite of driver assistance] systems will help you get up a steep driveway in winter, or across a muddy soccer pitch without getting stuck.” In other words, even if your posh Rover spends more time in valet lines than the rugged wilderness, these technologies may still come to the rescue.
Post Syndicated from Karthik Kumar Odapally original https://aws.amazon.com/blogs/big-data/10-visualizations-to-try-in-amazon-quicksight-with-sample-data/
If you’re not already familiar with building visualizations for quick access to business insights using Amazon QuickSight, consider this your introduction. In this post, we’ll walk through some common scenarios with sample datasets to provide an overview of how you can connect yuor data, perform advanced analysis and access the results from any web browser or mobile device.
The following visualizations are built from the public datasets available in the links below. Before we jump into that, let’s take a look at the supported data sources, file formats and a typical QuickSight workflow to build any visualization.
Which data sources does Amazon QuickSight support?
At the time of publication, you can use the following data methods:
- Connect to AWS data sources, including:
- Amazon RDS
- Amazon Aurora
- Amazon Redshift
- Amazon Athena
- Amazon S3
- Upload Excel spreadsheets or flat files (CSV, TSV, CLF, and ELF)
- Connect to on-premises databases like Teradata, SQL Server, MySQL, and PostgreSQL
- Import data from SaaS applications like Salesforce and Snowflake
- Use big data processing engines like Spark and Presto
This list is constantly growing. For more information, see Supported Data Sources.
Answers in instants
SPICE is the Amazon QuickSight super-fast, parallel, in-memory calculation engine, designed specifically for ad hoc data visualization. SPICE stores your data in a system architected for high availability, where it is saved until you choose to delete it. Improve the performance of database datasets by importing the data into SPICE instead of using a direct database query. To calculate how much SPICE capacity your dataset needs, see Managing SPICE Capacity.
Typical Amazon QuickSight workflow
When you create an analysis, the typical workflow is as follows:
- Connect to a data source, and then create a new dataset or choose an existing dataset.
- (Optional) If you created a new dataset, prepare the data (for example, by changing field names or data types).
- Create a new analysis.
- Add a visual to the analysis by choosing the fields to visualize. Choose a specific visual type, or use AutoGraph and let Amazon QuickSight choose the most appropriate visual type, based on the number and data types of the fields that you select.
- (Optional) Modify the visual to meet your requirements (for example, by adding a filter or changing the visual type).
- (Optional) Add more visuals to the analysis.
- (Optional) Add scenes to the default story to provide a narrative about some aspect of the analysis data.
- (Optional) Publish the analysis as a dashboard to share insights with other users.
The following graphic illustrates a typical Amazon QuickSight workflow.
Visualizations created in Amazon QuickSight with sample datasets
Visualizations for a data analyst
Download and Resources: https://datacatalog.worldbank.org/dataset/world-development-indicators
Data catalog: The World Bank invests into multiple development projects at the national, regional, and global levels. It’s a great source of information for data analysts.
The following graph shows the percentage of the population that has access to electricity (rural and urban) during 2000 in Asia, Africa, the Middle East, and Latin America.
The following graph shows the share of healthcare costs that are paid out-of-pocket (private vs. public). Also, you can maneuver over the graph to get detailed statistics at a glance.
Visualizations for a trading analyst
Source: Deutsche Börse Public Dataset (DBG PDS)
Download and resources: https://aws.amazon.com/public-datasets/deutsche-boerse-pds/
Data catalog: The DBG PDS project makes real-time data derived from Deutsche Börse’s trading market systems available to the public for free. This is the first time that such detailed financial market data has been shared freely and continually from the source provider.
The following graph shows the market trend of max trade volume for different EU banks. It builds on the data available on XETRA engines, which is made up of a variety of equities, funds, and derivative securities. This graph can be scrolled to visualize trade for a period of an hour or more.
The following graph shows the common stock beating the rest of the maximum trade volume over a period of time, grouped by security type.
Visualizations for a data scientist
Download and resources: https://catalog.data.gov/dataset/road-weather-information-stations-788f8
Data catalog: Data derived from different sensor stations placed on the city bridges and surface streets are a core information source. The road weather information station has a temperature sensor that measures the temperature of the street surface. It also has a sensor that measures the ambient air temperature at the station each second.
The following graph shows the present max air temperature in Seattle from different RWI station sensors.
The following graph shows the minimum temperature of the road surface at different times, which helps predicts road conditions at a particular time of the year.
Visualizations for a data engineer
Download and resources: https://www.kaggle.com/datasnaek/youtube-new/data
Data catalog: Kaggle has come up with a platform where people can donate open datasets. Data engineers and other community members can have open access to these datasets and can contribute to the open data movement. They have more than 350 datasets in total, with more than 200 as featured datasets. It has a few interesting datasets on the platform that are not present at other places, and it’s a platform to connect with other data enthusiasts.
The following graph shows the trending YouTube videos and presents the max likes for the top 20 channels. This is one of the most popular datasets for data engineers.
The following graph shows the YouTube daily statistics for the max views of video titles published during a specific time period.
Visualizations for a business user
Source: New York Taxi Data
Download and resources: https://data.cityofnewyork.us/Transportation/2016-Green-Taxi-Trip-Data/hvrh-b6nb
Data catalog: NYC Open data hosts some very popular open data sets for all New Yorkers. This platform allows you to get involved in dive deep into the data set to pull some useful visualizations. 2016 Green taxi trip dataset includes trip records from all trips completed in green taxis in NYC in 2016. Records include fields capturing pick-up and drop-off dates/times, pick-up and drop-off locations, trip distances, itemized fares, rate types, payment types, and driver-reported passenger counts.
The following graph presents maximum fare amount grouped by the passenger count during a period of time during a day. This can be further expanded to follow through different day of the month based on the business need.
The following graph shows the NewYork taxi data from January 2016, showing the dip in the number of taxis ridden on January 23, 2016 across all types of taxis.
A quick search for that date and location shows you the following news report:
Using Amazon QuickSight, you can see patterns across a time-series data by building visualizations, performing ad hoc analysis, and quickly generating insights. We hope you’ll give it a try today!
If you found this post useful, be sure to check out Amazon QuickSight Adds Support for Combo Charts and Row-Level Security and Visualize AWS Cloudtrail Logs Using AWS Glue and Amazon QuickSight.
Karthik Odapally is a Sr. Solutions Architect in AWS. His passion is to build cost effective and highly scalable solutions on the cloud. In his spare time, he bakes cookies and cupcakes for family and friends here in the PNW. He loves vintage racing cars.
Pranabesh Mandal is a Solutions Architect in AWS. He has over a decade of IT experience. He is passionate about cloud technology and focuses on Analytics. In his spare time, he likes to hike and explore the beautiful nature and wild life of most divine national parks around the United States alongside his wife.
Post Syndicated from Roderick Bauer original https://www.backblaze.com/blog/ransomware-update-viruses-targeting-business-it-servers/
As ransomware attacks have grown in number in recent months, the tactics and attack vectors also have evolved. While the primary method of attack used to be to target individual computer users within organizations with phishing emails and infected attachments, we’re increasingly seeing attacks that target weaknesses in businesses’ IT infrastructure.
How Ransomware Attacks Typically Work
In our previous posts on ransomware, we described the common vehicles used by hackers to infect organizations with ransomware viruses. Most often, downloaders distribute trojan horses through malicious downloads and spam emails. The emails contain a variety of file attachments, which if opened, will download and run one of the many ransomware variants. Once a user’s computer is infected with a malicious downloader, it will retrieve additional malware, which frequently includes crypto-ransomware. After the files have been encrypted, a ransom payment is demanded of the victim in order to decrypt the files.
What’s Changed With the Latest Ransomware Attacks?
In 2016, a customized ransomware strain called SamSam began attacking the servers in primarily health care institutions. SamSam, unlike more conventional ransomware, is not delivered through downloads or phishing emails. Instead, the attackers behind SamSam use tools to identify unpatched servers running Red Hat’s JBoss enterprise products. Once the attackers have successfully gained entry into one of these servers by exploiting vulnerabilities in JBoss, they use other freely available tools and scripts to collect credentials and gather information on networked computers. Then they deploy their ransomware to encrypt files on these systems before demanding a ransom. Gaining entry to an organization through its IT center rather than its endpoints makes this approach scalable and especially unsettling.
SamSam’s methodology is to scour the Internet searching for accessible and vulnerable JBoss application servers, especially ones used by hospitals. It’s not unlike a burglar rattling doorknobs in a neighborhood to find unlocked homes. When SamSam finds an unlocked home (unpatched server), the software infiltrates the system. It is then free to spread across the company’s network by stealing passwords. As it transverses the network and systems, it encrypts files, preventing access until the victims pay the hackers a ransom, typically between $10,000 and $15,000. The low ransom amount has encouraged some victimized organizations to pay the ransom rather than incur the downtime required to wipe and reinitialize their IT systems.
The success of SamSam is due to its effectiveness rather than its sophistication. SamSam can enter and transverse a network without human intervention. Some organizations are learning too late that securing internet-facing services in their data center from attack is just as important as securing endpoints.
The typical steps in a SamSam ransomware attack are:
Attackers gain access to vulnerable server
|Attackers exploit vulnerable software or weak/stolen credentials.|
Attack spreads via remote access tools
|Attackers harvest credentials, create SOCKS proxies to tunnel traffic, and abuse RDP to install SamSam on more computers in the network.|
Ransomware payload deployed
|Attackers run batch scripts to execute ransomware on compromised machines.|
Ransomware demand delivered requiring payment to decrypt files
|Demand amounts vary from victim to victim. Relatively low ransom amounts appear to be designed to encourage quick payment decisions.|
What all the organizations successfully exploited by SamSam have in common is that they were running unpatched servers that made them vulnerable to SamSam. Some organizations had their endpoints and servers backed up, while others did not. Some of those without backups they could use to recover their systems chose to pay the ransom money.
Timeline of SamSam History and Exploits
Since its appearance in 2016, SamSam has been in the news with many successful incursions into healthcare, business, and government institutions.
SamSam campaign targets vulnerable JBoss servers
Attackers hone in on healthcare organizations specifically, as they’re more likely to have unpatched JBoss machines.
SamSam finds new targets
SamSam begins targeting schools and government.
After initial success targeting healthcare, attackers branch out to other sectors.
SamSam shuts down Atlanta
• Second attack on Colorado Department of Transportation.
• City of Atlanta suffers a devastating attack by SamSam.
The attack has far-reaching impacts — crippling the court system, keeping residents from paying their water bills, limiting vital communications like sewer infrastructure requests, and pushing the Atlanta Police Department to file paper reports.
• SamSam campaign nets $325,000 in 4 weeks.
Infections spike as attackers launch new campaigns. Healthcare and government organizations are once again the primary targets.
How to Defend Against SamSam and Other Ransomware Attacks
The best way to respond to a ransomware attack is to avoid having one in the first place. If you are attacked, making sure your valuable data is backed up and unreachable by ransomware infection will ensure that your downtime and data loss will be minimal or none if you ever suffer an attack.
In our previous post, How to Recover From Ransomware, we listed the ten ways to protect your organization from ransomware.
- Use anti-virus and anti-malware software or other security policies to block known payloads from launching.
- Make frequent, comprehensive backups of all important files and isolate them from local and open networks. Cybersecurity professionals view data backup and recovery (74% in a recent survey) by far as the most effective solution to respond to a successful ransomware attack.
- Keep offline backups of data stored in locations inaccessible from any potentially infected computer, such as disconnected external storage drives or the cloud, which prevents them from being accessed by the ransomware.
- Install the latest security updates issued by software vendors of your OS and applications. Remember to patch early and patch often to close known vulnerabilities in operating systems, server software, browsers, and web plugins.
- Consider deploying security software to protect endpoints, email servers, and network systems from infection.
- Exercise cyber hygiene, such as using caution when opening email attachments and links.
- Segment your networks to keep critical computers isolated and to prevent the spread of malware in case of attack. Turn off unneeded network shares.
- Turn off admin rights for users who don’t require them. Give users the lowest system permissions they need to do their work.
- Restrict write permissions on file servers as much as possible.
- Educate yourself, your employees, and your family in best practices to keep malware out of your systems. Update everyone on the latest email phishing scams and human engineering aimed at turning victims into abettors.
Please Tell Us About Your Experiences with Ransomware
Have you endured a ransomware attack or have a strategy to avoid becoming a victim? Please tell us of your experiences in the comments.
The post Ransomware Update: Viruses Targeting Business IT Servers appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.
Post Syndicated from Caleb Phillips original https://aws.amazon.com/blogs/big-data/power-from-wind-open-data-on-aws/
Data that describe processes in a spatial context are everywhere in our day-to-day lives and they dominate big data problems. Map data, for instance, whether describing networks of roads or remote sensing data from satellites, get us where we need to go. Atmospheric data from simulations and sensors underlie our weather forecasts and climate models. Devices and sensors with GPS can provide a spatial context to nearly all mobile data.
In this post, we introduce the WIND toolkit, a huge (500 TB), open weather model dataset that’s available to the world on Amazon’s cloud services. We walk through how to access this data and some of the open-source software developed to make it easily accessible. Our solution considers a subset of geospatial data that exist on a grid (raster) and explores ways to provide access to large-scale raster data from weather models. The solution uses foundational AWS services and the Hierarchical Data Format (HDF), a well adopted format for scientific data.
The approach developed here can be extended to any data that fit in an HDF5 file, which can describe sparse and dense vectors and matrices of arbitrary dimensions. This format is already popular within the physical sciences for both experimental and simulation data. We discuss solutions to gridded data storage for a massive dataset of public weather model outputs called the Wind Integration National Dataset (WIND) toolkit. We also highlight strategies that are general to other large geospatial data management problems.
Wind Integration National Dataset
As variable renewable power penetration levels increase in power systems worldwide, the importance of renewable integration studies to ensure continued economic and reliable operation of the power grid is also increasing. The WIND toolkit is the largest freely available grid integration dataset to date.
The WIND toolkit was developed by 3TIER by Vaisala. They were under a subcontract to the National Renewable Energy Laboratory (NREL) to support studies on integration of wind energy into the existing US grid. NREL is a part of a network of national laboratories for the US Department of Energy and has a mission to advance the science and engineering of energy efficiency, sustainable transportation, and renewable power technologies.
The toolkit has been used by consultants, research groups, and universities worldwide to support grid integration studies. Less traditional uses also include resource assessments for wind plants (such as those powering Amazon data centers), and studying the effects of weather on California condor migrations in the Baja peninsula.
The diversity of applications highlights the value of accessible, open public data. Yet, there’s a catch: the dataset is huge. The WIND toolkit provides simulated atmospheric (weather) data at a two-km spatial resolution and five-minute temporal resolution at multiple heights for seven years. The entire dataset is half a petabyte (500 TB) in size and is stored in the NREL High Performance Computing data center in Golden, Colorado. Making this dataset publicly available easily and in a cost-effective manner is a major challenge.
As other laboratories and public institutions work to release their data to the world, they may face similar challenges to those that we experienced. Some prior, well-intentioned efforts to release huge datasets as-is have resulted in data resources that are technically available but fundamentally unusable. They may be stored in an unintuitive format or indexed and organized to support only a subset of potential uses. Downloading hundreds of terabytes of data is often impractical. Most users don’t have access to a big data cluster (or super computer) to slice and dice the data as they need after it’s downloaded.
We aim to provide a large amount of data (50 terabytes) to the public in a way that is efficient, scalable, and easy to use. In many cases, researchers can access these huge cloud-located datasets using the same software and algorithms they have developed for smaller datasets stored locally. Only the pieces of data they need for their individual analysis must be downloaded. To make this work in practice, we worked with the HDF Group and have built upon their forthcoming Highly Scalable Data Service.
In the rest of this post, we discuss how the HSDS software was developed to use Amazon EC2 and Amazon S3 resources to provide convenient and scalable access to these huge geospatial datasets. We describe how the HSDS service has been put to work for the WIND Toolkit dataset and demonstrate how to access it using the h5pyd Python library and the REST API. We conclude with information about our ongoing work to release more ‘open’ datasets to the public using AWS services, and ways to improve and extend the HSDS with newer Amazon services like Amazon ECS and AWS Lambda.
Developing a scalable service for big geospatial data
The HDF5 file format and API have been used for many years and is an effective means of storing large scientific datasets. For example, NASA’s Earth Observing System (EOS) satellites collect more than 16 TBs of data per day using HDF5.
With the rise of the cloud, there are new challenges and opportunities to rethink how HDF5 can be enhanced to work effectively as a component in a cloud-native architecture. For the HDF Group, working with NREL has been a great opportunity to put ideas into practice with a production-size dataset.
An HDF5 file consists of a directed graph of group and dataset objects. Datasets can be thought of as a multidimensional array with support for user-defined metadata tags and compression. Typical operations on datasets would be reading or writing data to a regular subregion (a hyperslab) or reading and writing individual elements (a point selection). Also, group and dataset objects may each contain an arbitrary number of the user-defined metadata elements known as attributes.
Many people have used the HDF library in applications developed or ported to run on EC2 instances, but there are a number of constraints that often prove problematic:
- The HDF5 library can’t read directly from HDF5 files stored as S3 objects. The entire file (often many GB in size) would need to be copied to local storage before the first byte can be read. Also, the instance must be configured with the appropriately sized EBS volume)
- The HDF library only has access to the computational resources of the instance itself (as opposed to a cluster of instances), so many operations are bottlenecked by the library.
- Any modifications to the HDF5 file would somehow have to be synchronized with changes that other instances have made to same file before writing back to S3.
Using a pattern common to many offerings from AWS, the solution to these constraints is to develop a service framework around the HDF data model. Using this model, the HDF Group has created the Highly Scalable Data Service (HSDS) that provides all the functionality that traditionally was provided by the HDF5 library. By using the service, you don’t need to manage your own file volumes, but can just read and write whatever data that you need.
Because the service manages the actual data persistence to a durable medium (S3, in this case), you don’t need to worry about disk management. Simply stream the data you need from the service as you need it. Secondly, putting the functionality behind a service allows some tricks to increase performance (described in more detail later). And lastly, HSDS allows any number of clients to access the data at the same time, enabling HDF5 to be used as a coordination mechanism for multiple readers and writers.
In designing the HSDS architecture, we gave much thought to how to achieve scalability of the HSDS service. For accessing HDF5 data, there are two different types of scaling to consider:
- Multiple clients making many requests to the service
- Single requests that require a significant amount of data processing
To deal with the first scaling challenge, as with most services, we considered how the service responds as the request rate increases. AWS provides some great tools that help in this regard:
- Auto Scaling groups
- Elastic Load Balancing load balancers
- The ability of S3 to handle large aggregate throughput rates
By using a cluster of EC2 instances behind a load balancer, you can handle different client loads in a cost-effective manner.
The second scaling challenge concerns single requests that would take significant processing time with just one compute node. One example of this from the WIND toolkit would be extracting all the values in the seven-year time span for a given geographic point and dataset.
In HDF5, large datasets are typically stored as “chunks”; that is, a regular partition of the array. In HSDS, each chunk is stored as a binary object in S3. The sequential approach to retrieving the time series values would be for the service to read each chunk needed from S3, extract the needed elements, and go on to the next chunk. In this case, that would involve processing 2557 chunks, and would be quite slow.
Fortunately, with HSDS, you can speed this up quite a bit by exploiting the compute and I/O capabilities of the cluster. Upon receiving the request, the receiving node can use other nodes in the cluster to read different portions of the selection. With multiple nodes reading from S3 in parallel, performance improves as the cluster size increases.
The diagram below illustrates how this works in simplified case of four chunks and four nodes.
This architecture has worked in well in practice. In testing with the WIND toolkit and time series extraction, we observed a request latency of ~60 seconds using four nodes vs. ~5 seconds with 40 nodes. Performance roughly scales with the size of the cluster.
A planned enhancement to this is to use AWS Lambda for the worker processing. This enables 1000-way parallel reads at a reasonable cost, as you only pay for the milliseconds of CPU time used with AWS Lambda.
Public access to atmospheric data using HSDS and AWS
An early challenge in releasing the WIND toolkit data was in deciding how to subset the data for different use cases. In general, few researchers need access to the entire 0.5 PB of data and a great deal of efficiency and cost reduction can be gained by making directed constituent datasets.
NREL grid integration researchers initially extracted a 2-TB subset by selecting 120,000 points where the wind resource seemed appropriate for development. They also chose only those data important for wind applications (100-m wind speed, converted to power), the most interesting locations for those performing grid studies. To support the remaining users who needed more data resolution, we down-sampled the data to a 60-minute temporal resolution, keeping all the other variables and spatial resolution intact. This reduced dataset is 50 TB of data describing 30+ atmospheric variables of data for 7 years at a 60-minute temporal resolution.
Programmatic access is possible using the h5pyd Python library, a distributed analog to the widely used h5py library. Users interact with the datasets (variables) and slice the data from its (time x longitude x latitude) cube form as they see fit.
Examples and use cases are described in a set of Jupyter notebooks and available on GitHub:
To run these notebooks on an EC2 instance in the Oregon Region, run the following commands:
Now you have a Jupyter notebook server running on your EC2 server.
From your laptop, create an SSH tunnel:
Now, you can browse to localhost:8888 using the correct token, and interact with the notebooks as if they were local. Within the directory, there are examples for accessing the HSDS API and plotting wind and weather data using matplotlib.
Controlling access and defraying costs
A final concern is rate limiting and access control. Although the HSDS service is scalable and relatively robust, we had a few practical concerns:
- How can we protect from malicious or accidental use that may lead to high egress fees (for example, someone who attempts to repeatedly download the entire dataset from S3)?
- How can we keep track of who is using the data both to document the value of the data resource and to justify the costs?
- If costs become too high, can we charge for some or all API use to help cover the costs?
To approach these problems, we investigated using Amazon API Gateway and its simplified integration with the AWS Marketplace for SaaS monetization as well as third-party API proxies.
In the end, we chose to use API Umbrella due to its close involvement with http://data.gov. While AWS Marketplace is a compelling option for future datasets, the decision was made to keep this dataset entirely open, at least for now. As community use and associated costs grow, we’ll likely revisit Marketplace. Meanwhile, API Umbrella provides controls for rate limiting and API key registration out of the box and was simple to implement as a front-end proxy to HSDS. Those applications that may want to charge for API use can accomplish a similar strategy using Amazon API Gateway and AWS Marketplace.
Ongoing work and other resources
As NREL and other government research labs, municipalities, and organizations try to share data with the public, we expect many of you will face similar challenges to those we have tried to approach with the architecture described in this post. Providing large datasets is one challenge. Doing so in a way that is affordable and convenient for users is an entirely more difficult goal. Using AWS cloud-native services and the existing foundation of the HDF file format has allowed us to tackle that challenge in a meaningful way.
If you found this post useful, be sure to check out Perform Near Real-time Analytics on Streaming Data with Amazon Kinesis and Amazon Elasticsearch Service, Analyze OpenFDA Data in R with Amazon S3 and Amazon Athena and Querying OpenStreetMap with Amazon Athena.
About the Authors
Dr. Caleb Phillips is a senior scientist with the Data Analysis and Visualization Group within the Computational Sciences Center at the National Renewable Energy Laboratory. Caleb comes from a background in computer science systems, applied statistics, computational modeling, and optimization. His work at NREL spans the breadth of renewable energy technologies and focuses on applying modern data science techniques to data problems at scale.
Dr. Caroline Draxl is a senior scientist at NREL. She supports the research and modeling activities of the US Department of Energy from mesoscale to wind plant scale. Caroline uses mesoscale models to research wind resources in various countries, and participates in on- and offshore boundary layer research and in the coupling of the mesoscale flow features (kilometer scale) to the microscale (tens of meters). She holds a M.S. degree in Meteorology and Geophysics from the University of Innsbruck, Austria, and a PhD in Meteorology from the Technical University of Denmark.
John Readey has been a Senior Architect at The HDF Group since he joined in June 2014. His interests include web services related to HDF, applications that support the use of HDF and data visualization.Before joining The HDF Group, John worked at Amazon.com from 2006–2014 where he developed service-based systems for eCommerce and AWS.
Jordan Perr-Sauer is an RPP intern with the Data Analysis and Visualization Group within the Computational Sciences Center at the National Renewable Energy Laboratory. Jordan hopes to use his professional background in software engineering and his academic training in applied mathematics to solve the challenging problems facing America and the world.
Post Syndicated from Roderick Bauer original https://www.backblaze.com/blog/choosing-data-center/
Though most of us have never set foot inside of a data center, as citizens of a data-driven world we nonetheless depend on the services that data centers provide almost as much as we depend on a reliable water supply, the electrical grid, and the highway system. Every time we send a tweet, post to Facebook, check our bank balance or credit score, watch a YouTube video, or back up a computer to the cloud we are interacting with a data center.
In this series, The Challenges of Opening a Data Center, we’ll talk in general terms about the factors that an organization needs to consider when opening a data center and the challenges that must be met in the process. Many of the factors to consider will be similar for opening a private data center or seeking space in a public data center, but we’ll assume for the sake of this discussion that our needs are more modest than requiring a data center dedicated solely to our own use (i.e. we’re not Google, Facebook, or China Telecom).
Data center technology and management are changing rapidly, with new approaches to design and operation appearing every year. This means we won’t be able to cover everything happening in the world of data centers in our series, however, we hope our brief overview proves useful.
What is a Data Center?
A data center is the structure that houses a large group of networked computer servers typically used by businesses, governments, and organizations for the remote storage, processing, or distribution of large amounts of data.
While many organizations will have computing services in the same location as their offices that support their day-to-day operations, a data center is a structure dedicated to 24/7 large-scale data processing and handling.
Depending on how you define the term, there are anywhere from a half million data centers in the world to many millions. While it’s possible to say that an organization’s on-site servers and data storage can be called a data center, in this discussion we are using the term data center to refer to facilities that are expressly dedicated to housing computer systems and associated components, such as telecommunications and storage systems. The facility might be a private center, which is owned or leased by one tenant only, or a shared data center that offers what are called “colocation services,” and rents space, services, and equipment to multiple tenants in the center.
A large, modern data center operates around the clock, placing a priority on providing secure and uninterrrupted service, and generally includes redundant or backup power systems or supplies, redundant data communication connections, environmental controls, fire suppression systems, and numerous security devices. Such a center is an industrial-scale operation often using as much electricity as a small town.
Types of Data Centers
There are a number of ways to classify data centers according to how they will be used, whether they are owned or used by one or multiple organizations, whether and how they fit into a topology of other data centers; which technologies and management approaches they use for computing, storage, cooling, power, and operations; and increasingly visible these days: how green they are.
Data centers can be loosely classified into three types according to who owns them and who uses them.
Exclusive Data Centers are facilities wholly built, maintained, operated and managed by the business for the optimal operation of its IT equipment. Some of these centers are well-known companies such as Facebook, Google, or Microsoft, while others are less public-facing big telecoms, insurance companies, or other service providers.
Managed Hosting Providers are data centers managed by a third party on behalf of a business. The business does not own data center or space within it. Rather, the business rents IT equipment and infrastructure it needs instead of investing in the outright purchase of what it needs.
Colocation Data Centers are usually large facilities built to accommodate multiple businesses within the center. The business rents its own space within the data center and subsequently fills the space with its IT equipment, or possibly uses equipment provided by the data center operator.
Backblaze, for example, doesn’t own its own data centers but colocates in data centers owned by others. As Backblaze’s storage needs grow, Backblaze increases the space it uses within a given data center and/or expands to other data centers in the same or different geographic areas.
Availability is Key
When designing or selecting a data center, an organization needs to decide what level of availability is required for its services. The type of business or service it provides likely will dictate this. Any organization that provides real-time and/or critical data services will need the highest level of availability and redundancy, as well as the ability to rapidly failover (transfer operation to another center) when and if required. Some organizations require multiple data centers not just to handle the computer or storage capacity they use, but to provide alternate locations for operation if something should happen temporarily or permanently to one or more of their centers.
Organizations operating data centers that can’t afford any downtime at all will typically operate data centers that have a mirrored site that can take over if something happens to the first site, or they operate a second site in parallel to the first one. These data center topologies are called Active/Passive, and Active/Active, respectively. Should disaster or an outage occur, disaster mode would dictate immediately moving all of the primary data center’s processing to the second data center.
While some data center topologies are spread throughout a single country or continent, others extend around the world. Practically, data transmission speeds put a cap on centers that can be operated in parallel with the appearance of simultaneous operation. Linking two data centers located apart from each other — say no more than 60 miles to limit data latency issues — together with dark fiber (leased fiber optic cable) could enable both data centers to be operated as if they were in the same location, reducing staffing requirements yet providing immediate failover to the secondary data center if needed.
This redundancy of facilities and ensured availability is of paramount importance to those needing uninterrupted data center services.
Leadership in Energy and Environmental Design (LEED) is a rating system devised by the United States Green Building Council (USGBC) for the design, construction, and operation of green buildings. Facilities can achieve ratings of certified, silver, gold, or platinum based on criteria within six categories: sustainable sites, water efficiency, energy and atmosphere, materials and resources, indoor environmental quality, and innovation and design.
Green certification has become increasingly important in data center design and operation as data centers require great amounts of electricity and often cooling water to operate. Green technologies can reduce costs for data center operation, as well as make the arrival of data centers more amenable to environmentally-conscious communities.
The ACT, Inc. data center in Iowa City, Iowa was the first data center in the U.S. to receive LEED-Platinum certification, the highest level available.
ACT Data Center exterior
ACT Data Center interior
Factors to Consider When Selecting a Data Center
There are numerous factors to consider when deciding to build or to occupy space in a data center. Aspects such as proximity to available power grids, telecommunications infrastructure, networking services, transportation lines, and emergency services can affect costs, risk, security and other factors that need to be taken into consideration.
The size of the data center will be dictated by the business requirements of the owner or tenant. A data center can occupy one room of a building, one or more floors, or an entire building. Most of the equipment is often in the form of servers mounted in 19 inch rack cabinets, which are usually placed in single rows forming corridors (so-called aisles) between them. This allows staff access to the front and rear of each cabinet. Servers differ greatly in size from 1U servers (i.e. one “U” or “RU” rack unit measuring 44.50 millimeters or 1.75 inches), to Backblaze’s Storage Pod design that fits a 4U chassis, to large freestanding storage silos that occupy many square feet of floor space.
Location will be one of the biggest factors to consider when selecting a data center and encompasses many other factors that should be taken into account, such as geological risks, neighboring uses, and even local flight paths. Access to suitable available power at a suitable price point is often the most critical factor and the longest lead time item, followed by broadband service availability.
With more and more data centers available providing varied levels of service and cost, the choices increase each year. Data center brokers can be employed to find a data center, just as one might use a broker for home or other commercial real estate.
Websites listing available colocation space, such as upstack.io, or entire data centers for sale or lease, are widely used. A common practice is for a customer to publish its data center requirements, and the vendors compete to provide the most attractive bid in a reverse auction.
Business and Customer Proximity
The center’s closeness to a business or organization may or may not be a factor in the site selection. The organization might wish to be close enough to manage the center or supervise the on-site staff from a nearby business location. The location of customers might be a factor, especially if data transmission speeds and latency are important, or the business or customers have regulatory, political, tax, or other considerations that dictate areas suitable or not suitable for the storage and processing of data.
Local climate is a major factor in data center design because the climatic conditions dictate what cooling technologies should be deployed. In turn this impacts uptime and the costs associated with cooling, which can total as much as 50% or more of a center’s power costs. The topology and the cost of managing a data center in a warm, humid climate will vary greatly from managing one in a cool, dry climate. Nevertheless, data centers are located in both extremely cold regions and extremely hot ones, with innovative approaches used in both extremes to maintain desired temperatures within the center.
Geographic Stability and Extreme Weather Events
A major obvious factor in locating a data center is the stability of the actual site as regards weather, seismic activity, and the likelihood of weather events such as hurricanes, as well as fire or flooding.
Backblaze’s Sacramento data center describes its location as one of the most stable geographic locations in California, outside fault zones and floodplains.
Sometimes the location of the center comes first and the facility is hardened to withstand anticipated threats, such as Equinix’s NAP of the Americas data center in Miami, one of the largest single-building data centers on the planet (six stories and 750,000 square feet), which is built 32 feet above sea level and designed to withstand category 5 hurricane winds.
Equinix “NAP of the Americas” Data Center in Miami
Most data centers don’t have the extreme protection or history of the Bahnhof data center, which is located inside the ultra-secure former nuclear bunker Pionen, in Stockholm, Sweden. It is buried 100 feet below ground inside the White Mountains and secured behind 15.7 in. thick metal doors. It prides itself on its self-described “Bond villain” ambiance.
Bahnhof Data Center under White Mountain in Stockholm
Usually, the data center owner or tenant will want to take into account the balance between cost and risk in the selection of a location. The Ideal quadrant below is obviously favored when making this compromise.
Cost = Construction/lease, power, bandwidth, cooling, labor, taxes
Risk = Environmental (seismic, weather, water, fire), political, economic
Risk mitigation also plays a strong role in pricing. The extent to which providers must implement special building techniques and operating technologies to protect the facility will affect price. When selecting a data center, organizations must make note of the data center’s certification level on the basis of regulatory requirements in the industry. These certifications can ensure that an organization is meeting necessary compliance requirements.
Electrical power usually represents the largest cost in a data center. The cost a service provider pays for power will be affected by the source of the power, the regulatory environment, the facility size and the rate concessions, if any, offered by the utility. At higher level tiers, battery, generator, and redundant power grids are a required part of the picture.
Fault tolerance and power redundancy are absolutely necessary to maintain uninterrupted data center operation. Parallel redundancy is a safeguard to ensure that an uninterruptible power supply (UPS) system is in place to provide electrical power if necessary. The UPS system can be based on batteries, saved kinetic energy, or some type of generator using diesel or another fuel. The center will operate on the UPS system with another UPS system acting as a backup power generator. If a power outage occurs, the additional UPS system power generator is available.
Many data centers require the use of independent power grids, with service provided by different utility companies or services, to prevent against loss of electrical service no matter what the cause. Some data centers have intentionally located themselves near national borders so that they can obtain redundant power from not just separate grids, but from separate geopolitical sources.
Higher redundancy levels required by a company will of invariably lead to higher prices. If one requires high availability backed by a service-level agreement (SLA), one can expect to pay more than another company with less demanding redundancy requirements.
Stay Tuned for Part 2 of The Challenges of Opening a Data Center
That’s it for part 1 of this post. In subsequent posts, we’ll take a look at some other factors to consider when moving into a data center such as network bandwidth, cooling, and security. We’ll take a look at what is involved in moving into a new data center (including stories from Backblaze’s experiences). We’ll also investigate what it takes to keep a data center running, and some of the new technologies and trends affecting data center design and use. You can discover all posts on our blog tagged with “Data Center” by following the link https://www.backblaze.com/blog/tag/data-center/.
A scientist with a rather unusual name, Meow-Ludo Meow-Meow, gave a talk at
about the current trends in “do it yourself” (DIY) biology or
“biohacking”. He is perhaps most famous for being
prosecuted for implanting an Opal card RFID chip into his hand; the
Opal card is used for public transportation fares in Sydney. He gave more
details about his implant as well as describing some other biohacking
projects in an engaging presentation.
Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2018/01/subway_elevator.html
Local residents are opposing adding an elevator to a subway station because terrorists might use it to detonate a bomb. No, really. There’s no actual threat analysis, only fear:
“The idea that people can then ride in on the subway with a bomb or whatever and come straight up in an elevator is awful to me,” said Claudia Ward, who lives in 15 Broad Street and was among a group of neighbors who denounced the plan at a recent meeting of the local community board. “It’s too easy for someone to slip through. And I just don’t want my family and my neighbors to be the collateral on that.”
Local residents plan to continue to fight, said Ms. Gerstman, noting that her building’s board decided against putting decorative planters at the building’s entrance over fears that shards could injure people in the event of a blast.
“Knowing that, and then seeing the proposal for giant glass structures in front of my building - ding ding ding! — what does a giant glass structure become in the event of an explosion?” she said.
In 2005, I coined the term “movie-plot threat” to denote a threat scenario that caused undue fear solely because of its specificity. Longtime readers of this blog will remember my annual Movie-Plot Threat Contests. I ended the contest in 2015 because I thought the meme had played itself out. Clearly there’s more work to be done.
Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2018/01/security_breach.html
Interesting research: “Long-term market implications of data breaches, not,” by Russell Lange and Eric W. Burger.
Abstract: This report assesses the impact disclosure of data breaches has on the total returns and volatility of the affected companies’ stock, with a focus on the results relative to the performance of the firms’ peer industries, as represented through selected indices rather than the market as a whole. Financial performance is considered over a range of dates from 3 days post-breach through 6 months post-breach, in order to provide a longer-term perspective on the impact of the breach announcement.
- While the difference in stock price between the sampled breached companies and their peers was negative (1.13%) in the first 3 days following announcement of a breach, by the 14th day the return difference had rebounded to + 0.05%, and on average remained positive through the period assessed.
- For the differences in the breached companies’ betas and the beta of their peer sets, the differences in the means of 8 months pre-breach versus post-breach was not meaningful at 90, 180, and 360 day post-breach periods.
- For the differences in the breached companies’ beta correlations against the peer indices pre- and post-breach, the difference in the means of the rolling 60 day correlation 8 months pre- breach versus post-breach was not meaningful at 90, 180, and 360 day post-breach periods.
- In regression analysis, use of the number of accessed records, date, data sensitivity, and malicious versus accidental leak as variables failed to yield an R2 greater than 16.15% for response variables of 3, 14, 60, and 90 day return differential, excess beta differential, and rolling beta correlation differential, indicating that the financial impact on breached companies was highly idiosyncratic.
- Based on returns, the most impacted industries at the 3 day post-breach date were U.S. Financial Services, Transportation, and Global Telecom. At the 90 day post-breach date, the three most impacted industries were U.S. Financial Services, U.S. Healthcare, and Global Telecom.
The market isn’t going to fix this. If we want better security, we need to regulate the market.
Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/aws-iot-greengrass-and-machine-learning-for-connected-vehicles-at-ces/
Last week I attended a talk given by Bryan Mistele, president of Seattle-based INRIX. Bryan’s talk provided a glimpse into the future of transportation, centering around four principle attributes, often abbreviated as ACES:
Autonomous – Cars and trucks are gaining the ability to scan and to make sense of their environments and to navigate without human input.
Connected – Vehicles of all types have the ability to take advantage of bidirectional connections (either full-time or intermittent) to other cars and to cloud-based resources. They can upload road and performance data, communicate with each other to run in packs, and take advantage of traffic and weather data.
Electric – Continued development of battery and motor technology, will make electrics vehicles more convenient, cost-effective, and environmentally friendly.
Shared – Ride-sharing services will change usage from an ownership model to an as-a-service model (sound familiar?).
Individually and in combination, these emerging attributes mean that the cars and trucks we will see and use in the decade to come will be markedly different than those of the past.
On the Road with AWS
AWS customers are already using our AWS IoT, edge computing, Amazon Machine Learning, and Alexa products to bring this future to life – vehicle manufacturers, their tier 1 suppliers, and AutoTech startups all use AWS for their ACES initiatives. AWS Greengrass is playing an important role here, attracting design wins and helping our customers to add processing power and machine learning inferencing at the edge.
AWS customer Aptiv (formerly Delphi) talked about their Automated Mobility on Demand (AMoD) smart vehicle architecture in a AWS re:Invent session. Aptiv’s AMoD platform will use Greengrass and microservices to drive the onboard user experience, along with edge processing, monitoring, and control. Here’s an overview:
Another customer, Denso of Japan (one of the world’s largest suppliers of auto components and software) is using Greengrass and AWS IoT to support their vision of Mobility as a Service (MaaS). Here’s a video:
AWS at CES
The AWS team will be out in force at CES in Las Vegas and would love to talk to you. They’ll be running demos that show how AWS can help to bring innovation and personalization to connected and autonomous vehicles.
Personalized In-Vehicle Experience – This demo shows how AWS AI and Machine Learning can be used to create a highly personalized and branded in-vehicle experience. It makes use of Amazon Lex, Polly, and Amazon Rekognition, but the design is flexible and can be used with other services as well. The demo encompasses driver registration, login and startup (including facial recognition), voice assistance for contextual guidance, personalized e-commerce, and vehicle control. Here’s the architecture for the voice assistance:
Connected Vehicle Solution – This demo shows how a connected vehicle can combine local and cloud intelligence, using edge computing and machine learning at the edge. It handles intermittent connections and uses AWS DeepLens to train a model that responds to distracted drivers. Here’s the overall architecture, as described in our Connected Vehicle Solution:
Digital Content Delivery – This demo will show how a customer uses a web-based 3D configurator to build and personalize their vehicle. It will also show high resolution (4K) 3D image and an optional immersive AR/VR experience, both designed for use within a dealership.
Autonomous Driving – This demo will showcase the AWS services that can be used to build autonomous vehicles. There’s a 1/16th scale model vehicle powered and driven by Greengrass and an overview of a new AWS Autonomous Toolkit. As part of the demo, attendees drive the car, training a model via Amazon SageMaker for subsequent on-board inferencing, powered by Greengrass ML Inferencing.
To speak to one of my colleagues or to set up a time to see the demos, check out the Visit AWS at CES 2018 page.
If you are interested in this topic and want to learn more, the AWS for Automotive page is a great starting point, with discussions on connected vehicles & mobility, autonomous vehicle development, and digital customer engagement.
When you are ready to start building a connected vehicle, the AWS Connected Vehicle Solution contains a reference architecture that combines local computing, sophisticated event rules, and cloud-based data processing and storage. You can use this solution to accelerate your own connected vehicle projects.
Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2017/12/tracking_people_5.html
The trick in accurately tracking a person with this method is finding out what kind of activity they’re performing. Whether they’re walking, driving a car, or riding in a train or airplane, it’s pretty easy to figure out when you know what you’re looking for.
The sensors can determine how fast a person is traveling and what kind of movements they make. Moving at a slow pace in one direction indicates walking. Going a little bit quicker but turning at 90-degree angles means driving. Faster yet, we’re in train or airplane territory. Those are easy to figure out based on speed and air pressure.
After the app determines what you’re doing, it uses the information it collects from the sensors. The accelerometer relays your speed, the magnetometer tells your relation to true north, and the barometer offers up the air pressure around you and compares it to publicly available information. It checks in with The Weather Channel to compare air pressure data from the barometer to determine how far above sea level you are. Google Maps and data offered by the US Geological Survey Maps provide incredibly detailed elevation readings.
Once it has gathered all of this information and determined the mode of transportation you’re currently taking, it can then begin to narrow down where you are. For flights, four algorithms begin to estimate the target’s location and narrows down the possibilities until its error rate hits zero.
If you’re driving, it can be even easier. The app knows the time zone you’re in based on the information your phone has provided to it. It then accesses information from your barometer and magnetometer and compares it to information from publicly available maps and weather reports. After that, it keeps track of the turns you make. With each turn, the possible locations whittle down until it pinpoints exactly where you are.
To demonstrate how accurate it is, researchers did a test run in Philadelphia. It only took 12 turns before the app knew exactly where the car was.
This is a good example of how powerful synthesizing information from disparate data sources can be. We spend too much time worried about individual data collection systems, and not enough about analysis techniques of those systems.
During Q4, Backblaze deployed 100 petabytes worth of Seagate hard drives to our data centers. The newly deployed Seagate 10 and 12 TB drives are doing well and will help us meet our near term storage needs, but we know we’re going to need more drives — with higher capacities. That’s why the success of new hard drive technologies like Heat-Assisted Magnetic Recording (HAMR) from Seagate are very relevant to us here at Backblaze and to the storage industry in general. In today’s guest post we are pleased to have Mark Re, CTO at Seagate, give us an insider’s look behind the hard drive curtain to tell us how Seagate engineers are developing the HAMR technology and making it market ready starting in late 2018.
What is HAMR and How Does It Enable the High-Capacity Needs of the Future?
Guest Blog Post by Mark Re, Seagate Senior Vice President and Chief Technology Officer
Earlier this year Seagate announced plans to make the first hard drives using Heat-Assisted Magnetic Recording, or HAMR, available by the end of 2018 in pilot volumes. Even as today’s market has embraced 10TB+ drives, the need for 20TB+ drives remains imperative in the relative near term. HAMR is the Seagate research team’s next major advance in hard drive technology.
HAMR is a technology that over time will enable a big increase in the amount of data that can be stored on a disk. A small laser is attached to a recording head, designed to heat a tiny spot on the disk where the data will be written. This allows a smaller bit cell to be written as either a 0 or a 1. The smaller bit cell size enables more bits to be crammed into a given surface area — increasing the areal density of data, and increasing drive capacity.
It sounds almost simple, but the science and engineering expertise required, the research, experimentation, lab development and product development to perfect this technology has been enormous. Below is an overview of the HAMR technology and you can dig into the details in our technical brief that provides a point-by-point rundown describing several key advances enabling the HAMR design.
As much time and resources as have been committed to developing HAMR, the need for its increased data density is indisputable. Demand for data storage keeps increasing. Businesses’ ability to manage and leverage more capacity is a competitive necessity, and IT spending on capacity continues to increase.
History of Increasing Storage Capacity
For the last 50 years areal density in the hard disk drive has been growing faster than Moore’s law, which is a very good thing. After all, customers from data centers and cloud service providers to creative professionals and game enthusiasts rarely go shopping looking for a hard drive just like the one they bought two years ago. The demands of increasing data on storage capacities inevitably increase, thus the technology constantly evolves.
According to the Advanced Storage Technology Consortium, HAMR will be the next significant storage technology innovation to increase the amount of storage in the area available to store data, also called the disk’s “areal density.” We believe this boost in areal density will help fuel hard drive product development and growth through the next decade.
Why do we Need to Develop Higher-Capacity Hard Drives? Can’t Current Technologies do the Job?
Why is HAMR’s increased data density so important?
Data has become critical to all aspects of human life, changing how we’re educated and entertained. It affects and informs the ways we experience each other and interact with businesses and the wider world. IDC research shows the datasphere — all the data generated by the world’s businesses and billions of consumer endpoints — will continue to double in size every two years. IDC forecasts that by 2025 the global datasphere will grow to 163 zettabytes (that is a trillion gigabytes). That’s ten times the 16.1 ZB of data generated in 2016. IDC cites five key trends intensifying the role of data in changing our world: embedded systems and the Internet of Things (IoT), instantly available mobile and real-time data, cognitive artificial intelligence (AI) systems, increased security data requirements, and critically, the evolution of data from playing a business background to playing a life-critical role.
Consumers use the cloud to manage everything from family photos and videos to data about their health and exercise routines. Real-time data created by connected devices — everything from Fitbit, Alexa and smart phones to home security systems, solar systems and autonomous cars — are fueling the emerging Data Age. On top of the obvious business and consumer data growth, our critical infrastructure like power grids, water systems, hospitals, road infrastructure and public transportation all demand and add to the growth of real-time data. Data is now a vital element in the smooth operation of all aspects of daily life.
All of this entails a significant infrastructure cost behind the scenes with the insatiable, global appetite for data storage. While a variety of storage technologies will continue to advance in data density (Seagate announced the first 60TB 3.5-inch SSD unit for example), high-capacity hard drives serve as the primary foundational core of our interconnected, cloud and IoT-based dependence on data.
HAMR Hard Drive Technology
Seagate has been working on heat assisted magnetic recording (HAMR) in one form or another since the late 1990s. During this time we’ve made many breakthroughs in making reliable near field transducers, special high capacity HAMR media, and figuring out a way to put a laser on each and every head that is no larger than a grain of salt.
The development of HAMR has required Seagate to consider and overcome a myriad of scientific and technical challenges including new kinds of magnetic media, nano-plasmonic device design and fabrication, laser integration, high-temperature head-disk interactions, and thermal regulation.
A typical hard drive inside any computer or server contains one or more rigid disks coated with a magnetically sensitive film consisting of tiny magnetic grains. Data is recorded when a magnetic write-head flies just above the spinning disk; the write head rapidly flips the magnetization of one magnetic region of grains so that its magnetic pole points up or down, to encode a 1 or a 0 in binary code.
Increasing the amount of data you can store on a disk requires cramming magnetic regions closer together, which means the grains need to be smaller so they won’t interfere with each other.
Heat Assisted Magnetic Recording (HAMR) is the next step to enable us to increase the density of grains — or bit density. Current projections are that HAMR can achieve 5 Tbpsi (Terabits per square inch) on conventional HAMR media, and in the future will be able to achieve 10 Tbpsi or higher with bit patterned media (in which discrete dots are predefined on the media in regular, efficient, very dense patterns). These technologies will enable hard drives with capacities higher than 100 TB before 2030.
The major problem with packing bits so closely together is that if you do that on conventional magnetic media, the bits (and the data they represent) become thermally unstable, and may flip. So, to make the grains maintain their stability — their ability to store bits over a long period of time — we need to develop a recording media that has higher coercivity. That means it’s magnetically more stable during storage, but it is more difficult to change the magnetic characteristics of the media when writing (harder to flip a grain from a 0 to a 1 or vice versa).
That’s why HAMR’s first key hardware advance required developing a new recording media that keeps bits stable — using high anisotropy (or “hard”) magnetic materials such as iron-platinum alloy (FePt), which resist magnetic change at normal temperatures. Over years of HAMR development, Seagate researchers have tested and proven out a variety of FePt granular media films, with varying alloy composition and chemical ordering.
In fact the new media is so “hard” that conventional recording heads won’t be able to flip the bits, or write new data, under normal temperatures. If you add heat to the tiny spot on which you want to write data, you can make the media’s coercive field lower than the magnetic field provided by the recording head — in other words, enable the write head to flip that bit.
So, a challenge with HAMR has been to replace conventional perpendicular magnetic recording (PMR), in which the write head operates at room temperature, with a write technology that heats the thin film recording medium on the disk platter to temperatures above 400 °C. The basic principle is to heat a tiny region of several magnetic grains for a very short time (~1 nanoseconds) to a temperature high enough to make the media’s coercive field lower than the write head’s magnetic field. Immediately after the heat pulse, the region quickly cools down and the bit’s magnetic orientation is frozen in place.
Applying this dynamic nano-heating is where HAMR’s famous “laser” comes in. A plasmonic near-field transducer (NFT) has been integrated into the recording head, to heat the media and enable magnetic change at a specific point. Plasmonic NFTs are used to focus and confine light energy to regions smaller than the wavelength of light. This enables us to heat an extremely small region, measured in nanometers, on the disk media to reduce its magnetic coercivity,
Moving HAMR Forward
As always in advanced engineering, the devil — or many devils — is in the details. As noted earlier, our technical brief provides a point-by-point short illustrated summary of HAMR’s key changes.
Although hard work remains, we believe this technology is nearly ready for commercialization. Seagate has the best engineers in the world working towards a goal of a 20 Terabyte drive by 2019. We hope we’ve given you a glimpse into the amount of engineering that goes into a hard drive. Keeping up with the world’s insatiable appetite to create, capture, store, secure, manage, analyze, rapidly access and share data is a challenge we work on every day.
With thousands of HAMR drives already being made in our manufacturing facilities, our internal and external supply chain is solidly in place, and volume manufacturing tools are online. This year we began shipping initial units for customer tests, and production units will ship to key customers by the end of 2018. Prepare for breakthrough capacities.
Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2017/12/remote_hack_of_.html
Last month, the DHS announced that it was able to remotely hack a Boeing 757:
“We got the airplane on Sept. 19, 2016. Two days later, I was successful in accomplishing a remote, non-cooperative, penetration,” said Robert Hickey, aviation program manager within the Cyber Security Division of the DHS Science and Technology (S&T) Directorate.
“[Which] means I didn’t have anybody touching the airplane, I didn’t have an insider threat. I stood off using typical stuff that could get through security and we were able to establish a presence on the systems of the aircraft.” Hickey said the details of the hack and the work his team are doing are classified, but said they accessed the aircraft’s systems through radio frequency communications, adding that, based on the RF configuration of most aircraft, “you can come to grips pretty quickly where we went” on the aircraft.
Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/amazon-quicksight-update-geospatial-visualization-private-vpc-access-and-more/
We don’t often recognize or celebrate anniversaries at AWS. With nearly 100 services on our list, we’d be eating cake and drinking champagne several times a week. While that might sound like fun, we’d rather spend our working hours listening to customers and innovating. With that said, Amazon QuickSight has now been generally available for a little over a year and I would like to give you a quick update!
QuickSight in Action
Today, tens of thousands of customers (from startups to enterprises, in industries as varied as transportation, legal, mining, and healthcare) are using QuickSight to analyze and report on their business data.
Here are a couple of examples:
Gemini provides legal evidence procurement for California attorneys who represent injured workers. They have gone from creating custom reports and running one-off queries to creating and sharing dynamic QuickSight dashboards with drill-downs and filtering. QuickSight is used to track sales pipeline, measure order throughput, and to locate bottlenecks in the order processing pipeline.
Jivochat provides a real-time messaging platform to connect visitors to website owners. QuickSight lets them create and share interactive dashboards while also providing access to the underlying datasets. This has allowed them to move beyond the sharing of static spreadsheets, ensuring that everyone is looking at the same and is empowered to make timely decisions based on current data.
Transfix is a tech-powered freight marketplace that matches loads and increases visibility into logistics for Fortune 500 shippers in retail, food and beverage, manufacturing, and other industries. QuickSight has made analytics accessible to both BI engineers and non-technical business users. They scrutinize key business and operational metrics including shipping routes, carrier efficient, and process automation.
Looking Back / Looking Ahead
The feedback on QuickSight has been incredibly helpful. Customers tell us that their employees are using QuickSight to connect to their data, perform analytics, and make high-velocity, data-driven decisions, all without setting up or running their own BI infrastructure. We love all of the feedback that we get, and use it to drive our roadmap, leading to the introduction of over 40 new features in just a year. Here’s a summary:
- December 2016 – QuickSight Enterprise Edition.
- February 2017 – Support for Amazon Athena, Scheduled Refresh of SPICE data.
- April 2017 – KPI Charts, Export to CVS, AD Connector; Availability in US East (Ohio); Audit Logging with AWS CloudTrail.
- May 2017 – Presto and Apache Spark Connectors, Federated Single Sign-On Using SAML 2.0.
- June 2017 – Support for Amazon Redshift Spectrum, 1-Click Visualization of S3 Analytics.
- August 2017 – Availability in Asia Pacific (Singapore) and Asia Pacific (Sydney).
- September 2017 – Search & Filter Groups, Amazon S3 Analytics Connector, Create Analyses From Dashboards, Import Custom Date Formats.
- October 2017 – Combo Charts, Row-Level Security.
Looking forward, we are watching an interesting trend develop within our customer base. As these customers take a close look at how they analyze and report on data, they are realizing that a serverless approach offers some tangible benefits. They use Amazon Simple Storage Service (S3) as a data lake and query it using a combination of QuickSight and Amazon Athena, giving them agility and flexibility without static infrastructure. They also make great use of QuickSight’s dashboards feature, monitoring business results and operational metrics, then sharing their insights with hundreds of users. You can read Building a Serverless Analytics Solution for Cleaner Cities and review Serverless Big Data Analytics using Amazon Athena and Amazon QuickSight if you are interested in this approach.
New Features and Enhancements
We’re still doing our best to listen and to learn, and to make sure that QuickSight continues to meet your needs. I’m happy to announce that we are making seven big additions today:
Geospatial Visualization – You can now create geospatial visuals on geographical data sets.
Private VPC Access – You can now sign up to access a preview of a new feature that allows you to securely connect to data within VPCs or on-premises, without the need for public endpoints.
Flat Table Support – In addition to pivot tables, you can now use flat tables for tabular reporting. To learn more, read about Using Tabular Reports.
Calculated SPICE Fields – You can now perform run-time calculations on SPICE data as part of your analysis. Read Adding a Calculated Field to an Analysis for more information.
Wide Table Support – You can now use tables with up to 1000 columns.
Other Buckets – You can summarize the long tail of high-cardinality data into buckets, as described in Working with Visual Types in Amazon QuickSight.
HIPAA Compliance – You can now run HIPAA-compliant workloads on QuickSight.
Everyone seems to want this feature! You can now take data that contains a geographic identifier (country, city, state, or zip code) and create beautiful visualizations with just a few clicks. QuickSight will geocode the identifier that you supply, and can also accept lat/long map coordinates. You can use this feature to visualize sales by state, map stores to shipping destinations, and so forth. Here’s a sample visualization:
Private VPC Access Preview
If you have data in AWS (perhaps in Amazon Redshift, Amazon Relational Database Service (RDS), or on EC2) or on-premises in Teradata or SQL Server on servers without public connectivity, this feature is for you. Private VPC Access for QuickSight uses an Elastic Network Interface (ENI) for secure, private communication with data sources in a VPC. It also allows you to use AWS Direct Connect to create a secure, private link with your on-premises resources. Here’s what it looks like:
If you are ready to join the preview, you can sign up today.