Tag Archives: ASD

Using AWS Lambda and Amazon DynamoDB in an Automated Approach to Managing AWS CloudFormation Template Parameters and Mappings

Post Syndicated from Anuj Sharma original https://aws.amazon.com/blogs/devops/custom-lookup-using-aws-lambda-and-amazon-dynamodb/

AWS CloudFormation gives you an easy way to codify the creation and management of related AWS resources. The optional Mappings and Parameters sections of CloudFormation templates help you organize and parameterize your templates so you can quickly customize your stack. As organizations adopt Infrastructure as Code best practices, the number of mappings and parameters can quickly grow. In this blog post, we’ll discuss how AWS Lambda and Amazon DynamoDB can be used to simplify updates, reuse, quick lookups, and reporting for these mappings and parameters.

Solution overview

There are three parts to the solution described in this post. You’ll find sample code for this solution in this AWS Labs GitHub repository.

  1. DynamoDB table: Used as a central location to store and update all key-value pairs used in the ‘Mappings’ and ‘Parameters’ sections of the CloudFormation template. This could be a centralized table for the whole organization, with a partition key consisting of the team name and environment (for example, development, test, production) and a sort key for the application name. For more information about the types of keys supported by DynamoDB, see Core Components in the Amazon DynamoDB Developer Guide.

Here is the sample data in the table. “teamname-environment” is the partition key and “appname” is the sort key.

{
        "teamname-environment": "team1-dev",
        "appname": "app1",
        "mappings": {
            "elbsubnet1": "subnet-123456",
            "elbsubnet2": "subnet-234567",
            "appsubnet1": "subnet-345678",
            "appsubnet2": "subnet-456789",
            "vpc": "vpc-123456",
            "appname":"app1",
            "costcenter":"123456",
            "teamname":"team1",
            "environment":"dev",
            "certificate":"arn-123456qwertyasdfgh",
            "compliancetype":"pci",
            "amiid":"ami-123456",
            "region":"us-west-1",
            "publichostedzoneid":"Z234asdf1234asdf",
            "privatehostedzoneid":"Z345SDFGCVHD123",
            "hostedzonename":"demo.internal"
        }
}

  1. Lambda function: Accepts the inputs of the primary keys, looks up the DynamoDB table, and returns all the key-value data.
  2. Custom lookup resource: Makes a call to the Lambda function, with the inputs of primary keys (accepted by the Lambda function) and retrieves the key-value data. All CloudFormation templates can duplicate this generic custom resource.

 

The diagram shows the interaction of services and resources in this solution.

  1. Users create a DynamoDB table and, using the DynamoDB console, AWS SDK, or AWS CLI, insert the mappings, parameters, and key-value data.
  2. CloudFormation templates have a custom resource, which calls a Lambda function. The combination of team name and application environment (“teamname-environment”) is the partition key input. Application Name (“appname”) is the sort key input to the Lambda function.
  3. The Lambda function queries the DynamoDB table based on the inputs.
  4. DynamoDB responds to the Lambda function with the results.
  5. The Lambda function responds to the custom resource in the CloudFormation stack, with the key-value data.
  6. The key-value data retrieved by the custom resource is then used by other resources in the stack using GetAtt Intrinsic function.

 

Using the sample code

To use the sample code for this solution, follow these steps:

  1. Clone the repository.
git clone https://github.com/awslabs/custom-lookup-lambda.git
cd custom-lookup-lambda
  1. The values in the sample-mappings.json file will be inserted into DynamoDB. Each record in sample-mappings.json corresponds to an item in DynamoDB. The mappings object contains the mapping.
  2. This solution uses Python as a programming model for AWS Lambda. If you haven’t set up your local development environment for Python, follow these steps, and then install the awscli python package.
pip install awscli
  1. To prepare your access keys or assume-role to make calls to AWS, configure the AWS Command Line Interface as described here. The IAM user or the assumed role used to make API calls must have, at minimum, this access. You can attach this policy to the IAM user or IAM group if you are using access keys, or to the IAM role if you are assuming a role. For more information, see Attaching Managed Policies in the IAM User Guide.
  2. Run insertrecord.sh to create the DynamoDB table named custom-lookup and insert the items in sample-mappings.json.
./insertrecord.sh

This script does the following:

  • Installs boto3 and requests Python packages through pip.
  • Executes the Python script to create the DynamoDB table (custom-lookup) and puts the data in sample-mappings.json.
  1. Run deployer.sh to package and create the Lambda function.
./deployer.sh

This script does the following:

  1. Using sample-stack.yaml, create a stack that makes a call to the Lambda function created in step 6. The function queries the DynamoDB table and produces as output the values corresponding to team1-dev and app1.
aws cloudformation deploy --template-file sample-stack.yaml --stack-name sample-stack

  1. Examine the output in the AWS CloudFormation console. You should see the values retrieved from the DynamoDB table. The Fn::GetAtt function allows you to use the values retrieved by the custom resource Lambda function.

For example, if the custom resource Lambda function is called using resource name as CUSTOMLOOKUP in the sample-stack, the value of key=amiid will be used in the stack using !GetAtt CUSTOMLOOKUP.amiid. Likewise, the value of key=vpc will be used in the stack using !GetAtt CUSTOMLOOKUP.vpc and so on.

Conclusion

In this blog post, we showed how to use an AWS CloudFormation custom resource backed by an AWS Lambda function to query Amazon DynamoDB to retrieve key-value data, thereby replacing the Mappings and Parameter sections of the CloudFormation template. This solution provides a more automated approach to managing template parameters and mappings. You can use the DynamoDB table to simplify updates, reuse, quick lookups, and reporting for these mappings and parameters.

Incredible Raspberry Pi projects in issue 56 of The MagPi

Post Syndicated from Rob Zwetsloot original https://www.raspberrypi.org/blog/magpi-magazine-56/

Hi, Rob from The MagPi here! It’s the last Thursday of the month and that means there’s a new issue of the official Raspberry Pi magazine: yay!

MagPi Magazine 56 cover image

Grab your copy today!

The MagPi Magazine 56

The MagPi 56 covers some incredible Raspberry Pi projects built by members of our community, from simple things you can make quickly, like an easy robot or LEGO Pi case, to more advanced projects to experiment with, like a set of Pinoculars.

Our news section looks at some great new happenings in the world of Pi, such as the new Pimoroni kits for Zero W, the Cambridge theme for PIXEL, and our fifth birthday celebrations.

Also not to be missed in this issue is our lowdown of every Raspberry Pi operating system: which is your favourite? While you’re weighing up the pros and cons of Raspbian vs. Ubuntu MATE, you can also read about our DJ Hero harmonograph, some hot command line tips, and much more.

The MagPi is the only monthly magazine written by and for the Pi community. Regardless of your experience with the Raspberry Pi, there’s something for everyone.

Get your copy

You can grab the latest issue of The MagPi today from WHSmith, Tesco, Sainsbury’s, and Asda. Alternatively, you can order your copy online, or get it digitally via our app on Android and iOS. There’s even a free PDF of it as well.

We also have a fantastic subscription offer to celebrate the new Raspberry Pi Zero W: grab a twelve-month subscription and you’ll get a Raspberry Pi Zero W absolutely free, along with a free official case and a bundle of adapter cables. Get yours online right now!

MagPi Magazine Free Pi Zero W

Free Creative Commons download

As always, you can download your copy of The MagPi completely free. Grab it straight from the issue page for The MagPi 56.

Don’t forget, though, that, as with sales of the Raspberry Pi itself, all proceeds from the print and digital editions of the magazine go to help the Raspberry Pi Foundation achieve its charitable goals. Help us democratise computing!

We hope you enjoy the issue! That’s it until next month…

The post Incredible Raspberry Pi projects in issue 56 of The MagPi appeared first on Raspberry Pi.

The MagPi 55 is out, with plenty about the Pi Zero W

Post Syndicated from Rob Zwetsloot original https://www.raspberrypi.org/blog/magpi-55-pi-zero-w/

Rob from The MagPi here! We’re still incredibly excited about the brand-new, wireless-enabled Raspberry Pi Zero W, and it’s in our latest issue, out now. Here’s a video of me talking about it.

Introducing Raspberry Pi Zero W

The Raspberry Pi Zero W, the new wireless-enabled Raspberry Pi, is out now! Rob from The MagPi, the official Raspberry Pi magazine, reveals the specifications, price, and more. Get a free Pi Zero W with a twelve-month print sub to The MagPi – http://magpi.cc/SubsNew The subscription offer includes a free Raspberry Pi Zero W, an official case with three covers, and a cable bundle.

We have not just one, but two, big articles about the Raspberry Pi Zero W in issue 55 of The MagPi. Our Big Build feature teaches you how to make a modified PiGRRL handheld retro console, and you’ll also find a full ten-page breakdown of everything that’s cool and new with the Raspberry Pi Zero W.

As usual we have loads of other excellent articles in the magazine, from tutorials on how to create an Amazon Alexa-powered robot to reviews of the brand new version of Kodi.

Pi Zero W, back-ups, advanced GPIO, 3D modelling, and more. We think issue 55 is fabulous!

Get your copy
You can grab a copy of The MagPi 55 in stores now at WHSmith, Tesco, Sainsbury’s, and Asda. Alternatively you can order your copy online, or get it digitally via our app on Android and iOS. There’s even a free PDF of it as well.

We also have a new subscription offer to celebrate the new Raspberry Pi Zero W: grab a twelve-month subscription and you’ll get a Raspberry Pi Zero W absolutely free, along with a free official case and a bundle of adapter cables. Get yours online right now!

New Subs Banner_new

Free Creative Commons download
As always, you can download your copy of The MagPi completely free. Grab it straight from the issue page for The MagPi 55.

Don’t forget, though, that as with sales of the Raspberry Pi itself, all proceeds from the print and digital editions of the magazine go to help the Raspberry Pi Foundation achieve its charitable goals. Help us democratise computing!

Lastly, here’s a full zip of the code from this issue, to help you get off to a flying start with your projects. We hope you enjoy it!

The post The MagPi 55 is out, with plenty about the Pi Zero W appeared first on Raspberry Pi.

Build a magic mirror in issue 54 of The MagPi

Post Syndicated from Rob Zwetsloot original https://www.raspberrypi.org/blog/build-a-magic-mirror-in-issue-54-of-the-magpi/

Hey there folks! It’s Rob from The MagPi again. Did you miss me? I missed you.

Anyway, I’m here today to tell you that we’ve finally gone and done it: we’ve got a build-your-own-magic-mirror feature in the magazine. Not only that, it’s our cover feature. This amazing project won the community vote in our top 50 Raspberry Pi projects poll, so we decided to go all out and worked closely with Michael Teeuw (the creator of the winning project) to put together this definitive guide.

magic mirror

The latest issue is packed with excellent content

We also have a follow-up to our beginner’s guide to coding from last issue, as Lucy Hattersley delves deeper into object-oriented programming by using examples in Scratch and Python. And we continue our popular Learn to code with C series from Simon Long, along with our usual selection of finely crafted tutorials, guides, project focus articles, and reviews.

As well as all that, we have a new regular extra for you: starting from this issue, you can grab all the code from the magazine in one handy zip from our website or from our GitHub page for each issue of the mag. We’ve started doing this after a reader request: if you have any ideas that would make The MagPi even better, drop us an email and we’ll see what we can do.

The MagPi 54 is available in stores now from WHSmith, Tesco, Sainsbury’s, and Asda. Alternatively, you can buy The MagPi online or get it digitally via our app on Android and iOS. There’s even a free PDF of it as well.

Get a free Pi Zero
Want to make sure you never miss an issue? Subscribe today and get a Pi Zero bundle featuring the new, camera-enabled Pi Zero, together with a cable bundle that includes the camera adapter.

Free Pi Zeros: what’s not to love about a MagPi subscription?

Free Creative Commons download
As always, you can download your copy of The MagPi completely free. Grab it straight from the issue page for The MagPi 54.

Don’t forget, though, that as with sales of the Raspberry Pi itself, all proceeds from the print and digital editions of the magazine go to help the Raspberry Pi Foundation achieve its charitable goals. Help us democratise computing!

We hope you enjoy this issue!

The post Build a magic mirror in issue 54 of The MagPi appeared first on Raspberry Pi.

Бабите днес са по-нездрави от техните баби, внучетата – по-здрави от всички поколения до сега

Post Syndicated from Боян Юруков original http://yurukov.net/blog/2017/babite-dnes-sa-po-nezdravi/

Здравето на населението е много трудно за измерване и сравнение. Може да гледаме разходите за здравеопазване, взетите болнични и дори анкетите питащи хората дали имат болежки. Всички те обаче или разчитат на субективно възприятие, или се влияят от моментното икономическо състояние и дори конкретни политически мерки. Единствената обективна мярка, за съжаление, е смъртността. Колкото повече умрели има в даден период, толкова можеда кажем, че е по-зле здравето и здравната система.

Преди година писах за заблудите, които витаят около индекса за смъртността. В последния месец темата отново обиколи медиите покрай обновената класация на ЦРУ, която ни поставя на второ място след Лесото. В същата World Factbook ЦРУ бяха объркали населението на България занижавайки го с почти милион. Преди година и половина се поправиха, но с такава грешка не трябва да ни учудва, че са объркали и тази класация. Именно цитирайки нея толкова издания писаха, че сме най-нездрави и умираме по-често от почти всички страни по света.

WHO, Световната банка и Евростат посочват, че не бива да се използва абсолютната смъртност (или crude mortality) за сравнение между държавите. Проблемът е в различната структура на населението. Смъртността е нищо повече от общ брой умирания спрямо населението. Заедно с още няколко метрики е индикатор единствено колко намалява населението, но не може да се използва за оценка на здравето на хората и дори вероятността да починат.

Обсъдих този проблем подробно в предишната ми статия. Примерът, който пък дадох във вчерашния ми текст за Капитал, беше за два блока с еднакъв брой жители. В единия има най-вече пенсионери, а в другия – предимно млади двойки с деца. Съвсем очаквано в първия ще има повече некролози. Аналогично на другите развити страни, застаряващото население в България е причината смъртността сред по-възрастните да има по-голяма тежест в общата статистика.

Верни данни, грешна употреба

Именно затова България има доста по-висока абсолютна смъртност отколкото Румъния, която пък е същата като Германия. Двете страни са съответно на 26-то и 27-мо място в класацията. Дори само тези позиции говорят, че нещо не е наред в индекса на ЦРУ. Ако вземем смъртността по възрастови групи (колко души на дадена възраст умират спрямо общият брой хора на същата възраст), виждаме съвсем друга картина. Тук съм взел смъртността на Германия за база и съм показал колко процента разлика има по възрасти в България и Румъния. Включил съм и Латвия, която в класацията е на трето място точно след нас.

death1

Вижда се ясно, че България има между 25 и 75% по-голяма смъртност сред непълнолетните от тази в Германия. В Румъния обаче има между 100 и 175% повече смъртност от Германия. С други думи, два пъти по-далеч са от нивото на Германия от нас. Поне по този показател. Във всяка възрастова група немците имат по-голяма смъртност от нас, с изключение на хората около 50 и повечето над 60-те.

Именно тук е разковничето – ние имаме повече хора в тази възрастова група от тях. Аатова сме на второ място, а те по-назад от нас. Аналогично, вижда се колко по-голяма смъртност има Румъния от Германия, а в класацията на ЦРУ са едни до други. Разликата в структурата на населението проличава ясно.

Разликата с Естония е още по-голяма. Във възрастите между 5 и 55 имат в пъти по-голяма смъртност от нас. Малко по-добре са единствено при най-малките и възрастните. въпреки това, са точно зад нас по абсолютна смъртност. Интересното също е, че и трите държави имаме по-малка смъртност от Германия сред онези на 85 и повече години.

Алтернативата

Отчитайки тези проблеми, редица организации са взели мерки. Едно решение е т.н. ASDR – възрастово стандартизирана смъртност. Пресмятат смъртността по възрастови групи и приемат, че структурата на населението е еднаква за целия свят. Така се получава индекс, който позволява сравнение между държавите и показва много по-точно състоянието на здравната система и здравето на населението им.

Методите за изчисление на ASDR са практически еднакви. Разликата е каква стандартизирана структура приемат. WHO, например, взима предвижданията за средните стойности за населението на света между 2000 и 2025 г. Подробности и сравнения с други методи ще намерите тук. Вече 10 години използват този индекс за по-точно сравнение. Следният атлас показва картата на света според групата, в която се намират държавите – критична, висока, средна и ниска смъртност.

dr-ch2

България е посредата на класацията и е в групата по средна смъртност. Данните са от 2012-та и има определено подобрение у нас за последните 4 години. Общото положение в света обаче също се подобрява бързо, защото през 2000 г. също сме били по средата на класацията. При това независимо, че средно намаляваме смъртността си по възрастови групи с 26%.

Отново Румъния е по-напред от нас заради по-доброто здраве на възрастното си население, но само с малко. Германия е много по-напред. Този индекс показва много по-добре разликите между държавите. Сами се сещате, че не може да се сравнява здравеопазването ни с това в централна Африка, където на места дори ваксинациите са под 50%. Всъщност, ако разчитаме на типичния за нас жлъчен глад за негативни новини, може и да повярваме на такова твърдение. Достатъчно е да бъде поднесено шокиращо в жълта медия или от водещ на сутришен блок на национална телевизия. Затова е толкова важно да имаме критично мислене и да разпознаваме фалшивите новини.

Питай баба си

Друг начин да се погледне на този въпрос е с историческа справка. Това направих в предишната си статия, но разглеждах само последните 15 години. Тук сравнявам последните 55. Следната графика взима 1990 г. за база и сравнява колко повече или по-малко е била смъртността през различни години и възрастови групи.

От нея разбираме, например, че през 1960-та година, хората между 35 и 60 г. са умирали с до 25% по-малко от онези на същата възраст през 1990-та. Същото важи и за онези след 75. Това може би говори за сериозно влошаване на здравеопазването през 70-те и 80-те. Тенденцията на влошаване се забелязва и през 2004-та, когато 50-60 годишните имат дори по-голяма смъртност. За последните 10 години обаче има сериосно подобрение по всички показатели, както показах преди.

death2

При младежите и децата ситуацията е обратна. От 1960-та има постоянен спад в детската смъртност. В следващите 30 години намалява 3.5 пъти смъртността сред децата до 5 годинки и още наполовина в последните 25 години. Тенденцията продължава с бързи темпове надолу. Това говори за по-ниска заболевамост, по-добра диагностика и предотвратяване на фатални състояния.

Нещо, което ме учуди беше разминаването между 45 и 60 години. Независимо, че смъртността в тази възрастова група е намаляла от 90-те насам, все още е над нивата от 1960 и 1975. Затова обърнах перспективата и разделих населението на 5 възрастови групи – деца, младежи, възрастни, напреднала възраст и пенсионери. На следната графика взимам смъртността на всяка група през 1960-та за база и показвам как се е променяла сред населението на същата възраст през всяка от следващите 55 години.

death3

Вижда се това, което коментирахме до сега – при децата има непрестанно подобрение, при младежите тенденцията е надолу с леко прекъсване през 90-те. При възрастните (30-45 г.) има подобрение през 60-те, но положенето се влошава силно през 70-те и 80-те. При хората в напреднала възраст (до 60 г.) влошаването е още по-голямо и започва дори по-рано. Пикът и при двете възрастови групи е през кризата през 1997-ма, след което има значително подобрение надолу. При най-възрастните (60+) влошаването продължава до 1983-ра, след което има леко, но непостоянно подобрение.

През последните 20 години сме копенсирали доста от влошаването на здравето и смъртността през 70-те и 80-те. При българите между 30 и 45 вече нивата са под тези през 1960-та. При младите и децата намаляват постоянно и вероятността да загубите дете е в пъти по-малка от баба ви или майка ви, например. При хората в напреднала възраст обаче смъртността все още е висока. Причините за това са доста. Несъмнено може да споменем неща като недобре работещата спешна помощ и замърсяването на въздуха. Също толкова важни са обаче нежеланието да се ходи на контролни прегледи (рак на гърдата, матката, простатата…), инатливото нежелание да се грижим за здравето си (абе какво ще ми каже толкова доктора), липсата на спорт и влошаване на общото здраве на сравнително ранна възраст (изживяла съм си живота аз; на 40 е нормално да имаш болежки). Все пак, не трябва да забравяме, че има подобрение в последните 10 години, така че навярно нещата се променят в правилната посока.

„И все пак намаляваме…“

Няма спор. Причината всичко изброено до тук да влияе толкова на общата смъртност е увеличаването на делът на хората в напреднала възраст от общото население. Причините затова са както бумът в раждаемостта през 50-те комбиниран с ниска детска смъртност (в сравнение с началото на века), така и емиграцията на младите в последните 25 години и намаляването на броят деца на семейство започнало в края на 70-те и стабилизирал се през 90-те.

Всичко това може да се изобрази със следната графика. Тя показва как дялът на българите между 30 и 60 (жълто и зелено) практически не се променя за 55 години. С увеличение на средната продължителност на живота има много повече българи над 60-те. Българите под 30 пък намаляват значително. В това няма никакъв спор. Това обаче не означава, че ще „изчезнем“. Трябва да вземем мерки, но надали някой си представя, че може да накараме българките да раждат по 3-4 деца. Има какво да се направи, но за това ще пиша в по-нататък.

death4

The MagPi 53 out now! Free Debian + PIXEL DVD

Post Syndicated from Lucy Hattersley original https://www.raspberrypi.org/blog/magpi-53-now-free-debian-pixel-dvd/

The MagPi 53 is out now. This month’s community magazine comes with a free Debian + PIXEL DVD.

With the DVD, you can run the Debian + PIXEL desktop on a PC or Mac.

The MagPi 53 Free DVD

Click here to download The MagPi 53.

The MagPi Translated Edition 2

Launching alongside this month’s The MagPi are more international titles. The second edition of our bite-sized version of The MagPi, translated into four languages, is now available.

The MagPi Translated Edition 2 contains the best projects, reviews, and tutorials from The MagPi. These are translated into Italian, French, Spanish, and Hebrew.

Click here to download The MagPi Translated Edition 2.

Inside The MagPi 53: The best projects and guides

The MagPi 53 Tutorial

Here are some of the incredible projects you will find in this month’s The MagPi:

  • Google DeepDream: how to create surreal works of art with a Raspberry Pi and Google’s AI software
  • Master remote access: use a Raspberry Pi and SSH to connect remotely via the command line
  • Make a GPIO Music Box: Program push buttons to make different sounds
  • Create a horse race game: Get yourselves to the derby with Pi Bakery
  • Use BOINC to donate your Raspberry Pi’s resources to science

Inside The MagPi 53: Debian + PIXEL DVD

The MagPi 53 Beginner's Guide to Coding

There’s a huge amount in this month’s magazine. Here are just some of the features in this issue:

  • Beginner’s guide to programming: learn to code with our complete starter guide
  • Use the Debian + PIXEL DVD: try out our new OS on your computer with your free DVD
  • Create a bootable flash drive that can boot a Mac or PC into the PIXEL desktop

There’s also some amazing news this month: the Raspberry Pi has now sold 11 million units, and Raspberry Pi co-founder Eben Upton has been awarded a CBE! We have exclusive interviews with Eben about his CBE and the launch of Debian + PIXEL.

The MagPI 53 News

The best community projects

We also cover some of the most fantastic community projects ever built:

  • Pegasus and the North American Eagle. Inside the land-speed challenge car, with a Raspberry Pi in the driver’s cockpit
  • QBEE social media dress. This wearable tech posts automatically to social media
  • Self-playing pipe organ. This giant musical instrument is played by the Raspberry Pi
  • Water tank level monitor. This PIoT challenge winning project automates water collection in rural America

The MagPi 53 Project Focus

It’s one of the most feature-packed editions of The MagPi we’ve ever made. Don’t miss out on the free DVD that can bring an old computer back to life as a coding powerhouse. It really is something special.

You can grab The MagPi 53 in stores today: it’s in WHSmith, Tesco, Sainsbury’s, and Asda in the UK, and it will be in Micro Center and selected Barnes & Noble stores when it comes to the US. You can also buy the print edition online from our store, and it’s available digitally on our Android and iOS app.

Get a free Pi Zero

Want to make sure you never miss an issue? Subscribe today and get a Pi Zero bundle featuring the new, camera-enabled Pi Zero, and a cable bundle that includes the camera adapter.

The post The MagPi 53 out now! Free Debian + PIXEL DVD appeared first on Raspberry Pi.

Christmas Special: The MagPi 52 is out now!

Post Syndicated from Lucy Hattersley original https://www.raspberrypi.org/blog/magpi-christmas-special/

The MagPi Christmas Special is out now.

For the festive season, the official magazine of the Raspberry Pi community is having a maker special. This edition is packed with fun festive projects!

The MagPi issue 52 cover

The MagPi issue 52

Click here to download the MagPi Christmas Special

Here are just some of the fun projects inside this festive issue:

  • Magazine tree: turn the special cover into a Christmas tree, using LED lights to create a shiny, blinky display
  • DIY decorations: bling out your tree with NeoPixels and code
  • Santa tracker: follow Santa Claus around the world with a Raspberry Pi
  • Christmas crackers: the best low-cost presents for makers and hackers
  • Yuletide game: build Sliders, a fab block-sliding game with a festive feel.

Sliders

A Christmas game from the MagPi No.52

Inside the MagPi Christmas special

If you’re a bit Grinchy when it comes to Christmas, there’s plenty of non-festive fun to be found too:

  • Learn to use VNC Viewer
  • Find out how to build a sunrise alarm clock
  • Read our in-depth guide to Amiga emulation
  • Discover the joys of parallel computing

There’s also a huge amount of community news this month. The MagPi has an exclusive feature on Pioneers, our new programme for 12- to 15-year-olds, and news about Astro Pi winning the Arthur Clarke Award.

The Pioneers

The MagPi outlines our new Pioneers programme in detail

After that, we see some of the most stylish projects ever. Inside is the beautiful Sisyphus table; that’s a moving work of art, a facial recognition door lock, and a working loom controlled by a Raspberry Pi.

The MagPi 52 Sisyphus Project Focus

The MagPi interviews the maker of this amazing Sisyphus table

If that wasn’t enough, we also have a big feature on adding sensors to your robots. These can be used to built a battle-bot ready for the upcoming Pi Wars challenge.

The MagPi team wishes you all a merry Christmas! You can grab The MagPi 52 in stores today: it’s in WHSmith, Tesco, Sainsbury’s, and Asda in the UK, and it will be in Micro Center and selected Barnes & Noble stores when it comes to the US. You can also buy the print edition online from our store, and it’s available digitally on our Android and iOS app.

Get a free Pi Zero
Want to make sure you never miss an issue? Subscribe today and get a Pi Zero bundle featuring the new, camera-enabled Pi Zero, and a cable bundle that includes the camera adapter.

If you subscribe to The MagPi before 7 December 2016, you will get a free Pi Zero in time for Christmas.

The post Christmas Special: The MagPi 52 is out now! appeared first on Raspberry Pi.

Accessible games

Post Syndicated from Eevee original https://eev.ee/blog/2016/10/29/accessible-games/

I’ve now made a few small games. One of the trickiest and most interesting parts of designing them has been making them accessible.

I mean that in a very general and literal sense. I want as many people as possible to experience as much of my games as possible. Finding and clearing out unnecessary hurdles can be hard, but every one I leave risks losing a bunch of players who can’t or won’t clear it.

I’ve noticed three major categories of hurdle, all of them full of tradeoffs. Difficulty is what makes a game challenging, but if a player can’t get past a certain point, they can never see the rest of the game. Depth is great, but not everyone has 80 hours to pour into a game, and it’s tough to spend weeks of dev time on stuff most people won’t see. Distribution is a question of who can even get your game in the first place.

Here are some thoughts.

Mario Maker

Mario Maker is most notable for how accessible it is to budding game designers, which is important but also a completely different sense of accessibility.

The really nice thing about Mario Maker is that its levels are also accessible to players. Virtually everyone who’s heard of video games has heard of Mario. You don’t need to know many rules to be able to play. Move to the right, jump over/on things, and get to the flag.

(The “distribution” model is a bit of a shame, though — you need to own a particular console and a $60 game. If I want people to play a single individual level I made, that’s a lot of upfront investment to ask for. Ultimately Nintendo is in this to sell their own game more than to help people show off their own.)

But the emergent depth of Mario Maker’s myriad objects — the very property that makes the platform more than a toy — also makes it less accessible. Everyone knows you move around and jump, but not everyone knows you can pick up an item with B, or that you can put on a hat you’re carrying by pressing , or that you can spinjump on certain hazards. And these are fairly basic controls — Mario Maker contains plenty of special interactions between more obscure objects, and no manual explaining them all.

I thought it was especially interesting that Nintendo’s own comic series on building Mario Maker levels specifically points out that running jumps don’t come naturally to everyone. It’s hard to imagine too many people playing Mario Maker and not knowing how to jump while running.

And yet.

And yet, imagine being one such person, and encountering a level that requires a running jump early on. You can’t get past it. You might not even understand how to get past it; perhaps you don’t even know Mario can run. Now what? That’s it, you’re stuck. You’ll never see the rest of that level. It’s a hurdle, in a somewhat more literal sense.

Why make the level that way in the first place, then? Does any seasoned Mario player jump over a moderate-width gap and come away feeling proud for having conquered it? Seems unlikely.

I’ve tried playing through 100 Mario Challenge on Expert a number of times (without once managing to complete it), and I’ve noticed three fuzzy categories. Some levels are an arbitrary mess of hazards right from the start, so I don’t expect them to get any easier. Some levels are clearly designed as difficult obstacle courses, so again, I assume they’ll be just as hard all the way through. In both cases, if I give up and skip to the next level, I don’t feel like I’m missing out on anything — I’m not the intended audience.

But there are some Expert-ranked levels that seem pretty reasonable… until this one point where all hell breaks loose. I always wonder how deliberate those parts are, and I vaguely regret skipping them — would the rest of the level have calmed back down and been enjoyable?

That’s the kind of hurdle I think about when I see conspicuous clusters of death markers in my own levels. How many people died there and gave up? I make levels intending for people to play them, to see them through, but how many players have I turned off with some needlessly tricky part?

One of my levels is a Boo house with a few cute tricks in it. Unfortunately, I also put a ring of Boos right at the beginning that’s tricky to jump through, so it’s very easy for a player to die several times right there and never see anything else.

I wanted my Boo house to be interesting rather than difficult, but I let difficulty creep in accidentally, and so I’ve reduced the number of people who can appreciate the interestingness. Every level I’ve made since then, I’ve struggled to keep the difficulty down, and still sometimes failed. It’s easy to make a level that’s very hard; it’s surprisingly hard to make a level that’s fairly easy. All it takes is a single unintended hurdle — a tricky jump, an awkwardly-placed enemy — to start losing players.

This isn’t to say that games should never be difficult, but difficulty needs to be deliberately calibrated, and that’s a hard thing to do. It’s very easy to think only in terms of “can I beat this”, and even that’s not accurate, since you know every nook and cranny of your own level. Can you beat it blind, on the first few tries? Could someone else?

Those questions are especially important in Mario Maker, where the easiest way to encounter an assortment of levels is to play 100 Mario Challenge. You have 100 lives and need to beat 16 randomly-chosen levels. If you run out of lives, you’re done, and you have to start over. If I encounter your level here, I can’t afford to burn more than six or seven lives on it, or I’ll game over and have wasted my time. So if your level looks ridiculously hard (and not even in a fun way), I’ll just skip it and hope I get a better level next time.

I wonder if designers forget to calibrate for this. When you spend a lot of time working on something, it’s easy to imagine it exists in a vacuum, to assume that other people will be as devoted to playing it as you were to making it.

Mario Maker is an extreme case: millions of levels are available, and any player can skip to another one with the push of a button. That might be why I feel like I’ve seen a huge schism in level difficulty: most Expert levels are impossible for me, whereas most Normal levels are fairly doable with one or two rough patches. I haven’t seen much that’s in the middle, that feels like a solid challenge. I suspect that people who are very good at Mario are looking for an extreme challenge, and everyone else just wants to play some Mario, so moderate-difficulty levels just aren’t as common. The former group will be bored by them, and the latter group will skip them.

Or maybe that’s a stretch. It’s hard to generalize about the game’s pool of levels when they number in the millions, and I can’t have played more than a few hundred.

What Mario Maker has really taught me is what a hurdle looks like. The game keeps track of everywhere a player has ever died. I may not be able to watch people play my levels, but looking back at them later and seeing clumps of death markers is very powerful. Those are the places people failed. Did they stop playing after that? Did I intend for those places to be so difficult?

Doom

Doom is an interesting contrast to Mario Maker. A great many Doom maps have been produced over the past two decades, but nowhere near as many levels as Mario Maker has produced in a couple years. On the other hand, many people who still play Doom have been playing Doom this entire time, so a greater chunk of the community is really good at the game and enjoys a serious challenge.

I’ve only released a couple Doom maps of my own: Throughfare (the one I contributed to DUMP 2 earlier this year) and a few one-hour speedmaps I made earlier this week. I like building in Doom, with its interesting balance of restrictions — it’s a fairly accessible way to build an interesting 3D world, and nothing else is quite like it.

I’ve had the privilege of watching a few people play through my maps live, and I have learned some things.

The first is that the community’s love of difficulty is comically misleading. It’s not wrong, but, well, that community isn’t actually my target audience. So far I’ve “published” maps on this blog and Twitter, where my audience hasn’t necessarily even played Doom in twenty years. If at all! Some of my followers are younger than Doom.

Most notably, this creates something of a distribution problem: to play my maps, you need to install a thing (ZDoom) and kinda figure out how to use it and also get a copy of Doom 2 which probably involves spending five bucks. Less of a hurdle than getting Mario Maker, yes, but still some upfront effort.

Also, ZDoom’s default settings are… not optimal. Out of the box, it’s similar to classic Doom: no WASD, no mouselook. I don’t know who this is meant to appeal to. If you’ve never played Doom, the controls are goofy. If you’ve played other shooters, the controls are goofy. If you played Doom when it came out but not since, you probably don’t remember the controls, so they’re still goofy. Oof.

Not having mouselook is more of a problem than you’d think. If you as the designer play with mouselook, it’s really easy to put important things off the top or bottom of the screen and never realize it’ll be a problem. I watched someone play through Throughfare a few days ago and get completely stuck at what seemed to be a dead end — because he needed to drop down a hole in a small platform, and the hole was completely hidden by the status bar.

That’s actually an interesting example for another reason. Here’s the room where he got stuck.

A small room with a raised platform at the end, a metal section in the floor, and a switch on the side wall

When you press the switch, the metal plates on the ground rise up and become stairs, so you can get onto the platform. He did that, saw nowhere obvious to go, and immediately turned around and backtracked quite a ways looking for some other route.

This surprised me! The room makes no sense as a dead end. It’s not an easter egg or interesting feature; it has no obvious reward; it has a button that appears to help you progress. If I were stuck here, I’d investigate the hell out of this room — yet this player gave up almost immediately.

Not to say that the player is wrong and the level is right. This room was supposed to be trivially simple, and I regret that it became a hurdle for someone. It’s just a difference in playstyle I didn’t account for. Besides the mouselook problem, this player tended to move very quickly in general, charging straight ahead in new areas without so much as looking around; I play more slowly, looking around for nooks and crannies. He ended up missing the plasma gun for much the same reason — it was on a ledge slightly below the default view angle, making it hard to see without mouselook.

Speaking of nooks and crannies: watching someone find or miss secrets in a world I built is utterly fascinating. I’ve watched several people play Throughfare now, and the secrets are the part I love watching the most. I’ve seen people charge directly into secrets on accident; I’ve seen people run straight to a very clever secret just because they had the same idea I did; I’ve seen people find a secret switch and then not press it. It’s amazing how different just a handful of players have been.

I think the spread of secrets in Throughfare is pretty good, though I slightly regret using the same trick three times; either you get it right away and try it everywhere, or you don’t get it at all and miss out on a lot of goodies. Of course, the whole point of secrets is that not everyone will find them on the first try (or at all), so it’s probably okay to err on the trickier side.


As for the speedmaps, I’ve only watched one person play them live. The biggest hurdle was a room I made that required jumping.

Jumping wasn’t in the original Doom games. People thus don’t really expect to need to jump in Doom maps. Worse, ZDoom doesn’t even have a key bound to jump out of the box, which I only discovered later.

See, when I made the room (very quickly), I was imagining a ZDoom veteran seeing it and immediately thinking, “oh, this is one of those maps where I need to jump”. I’ve heard people say that about other maps before, so it felt like common knowledge. But it’s only common knowledge if you’re part of the community and have run into a few maps that require jumping.

The situation is made all the more complicated by the way ZDoom handles it. Maps can use a ZDoom-specific settings file to explicitly allow or forbid jumping, but the default is to allow it. The stock maps and most third-party vanilla maps won’t have this ZDoom-specific file, so jumping will be allowed, even though they’re not designed for it. Most mappers only use this file at all if they’re making something specifically for ZDoom, in which case they might as well allow jumping anyway. It’s opt-out, but the maps that don’t want it are the ones least likely to use the opt-out, so in practice everyone has to assume jumping isn’t allowed until they see some strong indication otherwise. It’s a mess. Oh, and ZDoom also supports crouching, which is even more obscure.

I probably should’ve thought of all that at the time. In my defense, you know, speedmap.

One other minor thing was that, of course, ZDoom uses the traditional Doom HUD out of the box, and plenty of people play that way on purpose. I’m used to ZDoom’s “alternative” HUD, which not only expands your field of view slightly, but also shows a permanent count of how many secrets are in the level and how many you’ve found. I love that, because it tells me how much secret-hunting I’ll need to do from the beginning… but if you don’t use that HUD (and don’t look at the count on the automap), you won’t even know whether there are secrets or not.


For a third-party example: a recent (well, late 2014) cool release was Going Down, a set of small and devilish maps presented as the floors of a building you’re traversing from the roof downwards. I don’t actually play a lot of Doom, but I liked this concept enough to actually play it, and I enjoyed the clever traps and interwoven architecture.

Then I reached MAP12, Dead End. An appropriate name, because I got stuck here. Permanently stuck. The climax of the map is too many monsters in not enough space, and it’s cleverly rigged to remove the only remaining cover right when you need it. I couldn’t beat it.

That was a year ago. I haven’t seen any of the other 20 maps beyond this point. I’m sure they’re very cool, but I can’t get to them. This one is too high a hurdle.

Granted, hopping around levels is trivially easy in Doom games, but I don’t want to cheat my way through — and anyway, if I can’t beat MAP12, what hope do I have of beating MAP27?

I feel ambivalent about this. The author describes the gameplay as “chaotic evil”, so it is meant to be very hard, and I appreciate the design of the traps… but I’m unable to appreciate any more of them.

This isn’t the author’s fault, anyway; it’s baked into the design of Doom. If you can’t beat one level, you don’t get to see any future levels. In vanilla Doom it was particularly bad: if you die, you restart the level with no weapons or armor, probably making it even harder than it was before. You can save any time, and some modern source ports like ZDoom will autosave when you start a level, but the original game never saved automatically.

Isaac’s Descent

Isaac’s Descent is the little PICO-8 puzzle platformer I made for Ludum Dare 36 a couple months ago. It worked out surprisingly well; pretty much everyone who played it (and commented on it to me) got it, finished it, and enjoyed it. The PICO-8 exports to an HTML player, too, so anyone with a keyboard can play it with no further effort required.

I was really happy with the puzzle design, especially considering I hadn’t really made a puzzle game before and was rushing to make some rooms in a very short span of time. Only two were perhaps unfair. One was the penultimate room, which involved a tricky timing puzzle, so I’m not too bothered about that. The other was this room:

A cavern with two stone slab doors, one much taller than the other, and a wooden wheel on the wall

Using the wheel raises all stone doors in the room. Stone doors open at a constant rate, wait for a fixed time, and then close again. The tricky part with this puzzle is that by the time the very tall door has opened, the short door has already closed again. The solution is simply to use the wheel again right after the short door has closed, while the tall door is still opening. The short door will reopen, while the tall door won’t be affected since it’s already busy.

This isn’t particularly difficult to figure out, but it did catch a few people, and overall it doesn’t sit particularly well with me. Using the wheel while a door is opening feels like a weird edge case, not something that a game would usually rely on, yet I based an entire puzzle around it. I don’t know. I might be overthinking this. The problem might be that “ignore the message” is a very computery thing to do and doesn’t match with how such a wheel would work in practice; perhaps I’d like the puzzle more if the wheel always interrupted whatever a door was doing and forced it to raise again.

Overall, though, the puzzles worked well.

The biggest snags I saw were control issues with the PICO-8 itself. The PICO-8 is a “fantasy console” — effectively an emulator for a console that never existed. One of the consequences of this is that the controls aren’t defined in terms of keyboard keys, but in terms of the PICO-8’s own “controller”. Unfortunately, that controller is only defined indirectly, and the web player doesn’t indicate in any way how it works.

The controller’s main inputs — the only ones a game can actually read — are a directional pad and two buttons, and , which map to z and x on a keyboard. The PICO-8 font has glyphs for and , so I used those to indicate which button does what. Unfortunately, if you aren’t familiar with the PICO-8, those won’t make a lot of sense to you. It’s nice that looks like the keyboard key it’s bound to, but looks like the wrong keyboard key. This caused a little confusion.

Well,” I hear you say, “why not just refer to the keys directly?” Ah, but there’s a very good reason the PICO-8 is defined in terms of buttons: those aren’t the only keys you can use! n and m also work, as do c and v. The PocketCHIP also allows… 0 and =, I think, which is good because z and x are directly under the arrow keys on the PocketCHIP keyboard. And of course you can play on a USB controller, or rebind the keys.

I could’ve mentioned that z and x are the defaults, but that’s wrong for the PocketCHIP, and now I’m looking at a screenful of text explaining buttons that most people won’t read anyway.

A similar problem is the pause menu, accessible with p or enter. I’d put an option on the pause menu for resetting the room you’re in, just in case, but didn’t bother to explain how to get to the pause menu.Or that a pause menu exists. Also, the ability to put custom things on the pause menu is new, so a lot of people might not even know about it. I’m sure you can see this coming: a few rooms (including the two-door one) had places you could get stuck, and without any obvious way to restart the room, a few people thought they had to start the whole game over. Whoops.

In my defense, the web player is actively working against me here: it has a “pause” link below the console, but all the link does is freeze the player, not bring up the pause menu.

This is a recurring problem, and perhaps a fundamental question of making games accessible: how much do you need to explain to people who aren’t familiar with the platform or paradigm? Should every single game explain itself? Players who don’t need the explanation can easily get irritated by it, and that’s a bad way to start a game. The PICO-8 in particular has the extra wrinkle that its cartridge space is very limited, and any kind of explanation/tutorial costs space you could be using for gameplay. On the other hand, I’ve played more than one popular PICO-8 game that was completely opaque to me because it didn’t explain its controls at all.

I’m reminded of Counterfeit Monkey, a very good interactive fiction game that goes out of its way to implement a hint system and a gentle tutorial. The tutorial knits perfectly with the story, and the hints are trivially turned off, so neither is a bother. The game also has a hard mode, which eliminates some of the more obvious solutions and gives a nod to seasoned IF players as well. The author is very interested in making interactive fiction more accessible in general, and it definitely shows. I think this game alone convinced me it’s worth the effort — I’m putting many of the same touches in my own IF foray.

Under Construction

Under Construction is the PICO-8 game that Mel and I made early this year. It’s a simple, slightly surreal, slightly obtuse platformer.

Traditional wisdom has it that you don’t want games to be obtuse. That acts as a hurdle, and loses you players. Here, though, it’s part of the experience, so the question becomes how to strike a good balance without losing the impact.

A valid complaint we heard was that the use of color is slightly inconsistent in places. For the most part, foreground objects (those you can stand on) are light and background decorations are gray, but a couple tiles break that pattern. A related problem that came up almost immediately in beta testing was that spikes were difficult to pick out. I addressed that — fairly effectively, I think — by adding a single dark red pixel to the tip of the spikes.

But the most common hurdle by far was act 3, which caught us completely by surprise. Spoilers!

From the very beginning, the world contains a lot of pillars containing eyeballs that look at you. They don’t otherwise do anything, beyond act as platforms you can stand on.

In act 2, a number of little radios appear throughout the world. Mr. 5 complains that it’s very noisy, so you need to break all the radios by jumping on them.

In act 3, the world seems largely the same… but the eyes in the pillars now turn to ❌’s when you touch them. If this happens before you make it to the end, Mr. 5 complains that he’s in pain, and the act restarts.

The correct solution is to avoid touching any of the eye pillars. But because this comes immediately after act 2, where we taught the player to jump on things to defeat them — reinforcing a very common platforming mechanic — some players thought you were supposed to jump on all of them.

I don’t know how we could’ve seen that coming. The acts were implemented one at a time and not in the order they appear in the game, so we were both pretty used to every individual mechanic before we started playing through the entire game at once. I suppose when a game is developed and tested in pieces (as most games are), the order and connection between those pieces is a weak point and needs some extra consideration.

We didn’t change the game to address this, but the manual contains a strong hint.

Under Construction also contains a couple of easter eggs and different endings. All are fairly minor changes, but they added a lot of character to the game and gave its fans something else to delve into once they’d beaten it.

Crucially, these things worked as well as they did because they weren’t accessible. Easily-accessed easter eggs aren’t really easter eggs any more, after all. I don’t think the game has any explicit indication that the ending can vary, which meant that players would only find out about it from us or other fans.

I don’t yet know the right answer for balancing these kinds of extras, and perhaps there isn’t one. If you spend a lot of time on easter eggs, multiple endings, or even just multiple paths through the game, you’re putting a lot of effort into stuff that many players will never see. On the other hand, they add an incredible amount of depth and charm to a game and reward those players who do stick around to explore.

This is a lot like the balancing act with software interfaces. You want your thing to be accessible in the sense that a newcomer can sit down and get useful work done, but you also want to reward long-time users with shortcuts and more advanced features. You don’t want to hide advanced features too much, but you also don’t want to have an interface with a thousand buttons.

How larger and better-known games deal with this

I don’t have the patience for Zelda I. I never even tried it until I got it for free on my 3DS, as part of a pack of Virtual Console games given to everyone who bought a 3DS early. I gave it a shot, but I got bored really quickly. The overworld was probably the most frustrating part: the connections between places are weird, everything looks pretty much the same, the map is not very helpful, and very little acts as a landmark. I could’ve drawn my own map, but, well, I usually can’t be bothered to do that for games.

I contrast this with Skyward Sword, which I mostly enjoyed. Ironically, one of my complaints is that it doesn’t quite have an overworld. It almost does, but they stopped most of the way, leaving us with three large chunks of world and a completely-open sky area reminiscent of Wind Waker’s ocean.

Clearly, something about huge open spaces with no barriers whatsoever appeals to the Zelda team. I have to wonder if they’re trying to avoid situations like my experience with Zelda I. If a player gets lost in an expansive overworld, either they’ll figure out where to go eventually, or they’ll give up and never see the rest of the game. Losing players that way, especially in a story-driven game, is a huge shame.

And this is kind of a problem with the medium in general. For all the lip service paid to nonlinearity and sandboxes, the vast majority of games require some core progression that’s purely linear. You may be able to wander around a huge overworld, but you still must complete these dungeons and quests in this specific order. If something prevents you from doing one of them, you won’t be able to experience the others. You have to do all of the first x parts of the game before you can see part x + 1.

This is really weird! No other media is like this. If you watch a movie or read a book or listen to a song and some part of it is inaccessible for whatever reason — the plot is poorly explained, a joke goes over your head, the lyrics are mumbled — you can still keep going and experience the rest. The stuff that comes later might even help you make sense of the part you didn’t get.

In games, these little bumps in the road can become walls.

It’s not even necessarily difficulty, or getting lost, or whatever. A lot of mobile puzzle games use the same kind of artificial progression where you can only do puzzles in sequential batches; solving enough of the available puzzles will unlock the next batch. But in the interest of padding out the length, many of these games will have dozens of trivially easy and nearly identical puzzles in the beginning, which you have to solve to get to the later interesting ones. Sometimes I’ve gotten so bored by this that I’ve given up on a game before reaching the interesting puzzles.

In a way, that’s the same problem as getting lost in an overworld. Getting lost isn’t a hard wall, after all — you can always do an exhaustive search and talk to every NPC twice. But that takes time, and it’s not fun, much like the batches of required baby puzzles. People generally don’t like playing games that waste their time.

I love the Picross “e” series on the 3DS, because over time they’ve largely figured out that this is pointless: in the latest game in the series, everything is available from the beginning. Want to do easy puzzles? Do easy puzzles. Want to skip right to the hard stuff? Sure, do that. Don’t like being told when you made a wrong move? Turn it off.

(It’s kinda funny that the same people then made Pokémon Picross, which has some of the most absurd progression I’ve ever seen. Progressing beyond the first half-dozen puzzles requires spending weeks doing a boring minigame every day to grind enough pseudocurrency to unlock more puzzles. Or you can just pay for pseudocurrency, and you’ll have unlocked pretty much the whole game instantly. It might as well just be a demo; the non-paid progression is useless.)

Chip’s Challenge also handled this pretty well. You couldn’t skip around between levels arbitrarily, which was somewhat justified by the (very light) plot. Instead, if you died or restarted enough times, the game would offer to skip you to the next level, and that would be that. You weren’t denied the rest of the game just because you couldn’t figure out an ice maze or complete some horrible nightmare like Blobnet.

I wish this sort of mechanic were more common. Not so games could be more difficult, but so games wouldn’t have to worry as much about erring on the side of ease. I don’t know how it could work for a story-driven game where much of the story is told via experiencing the game itself, though — skipping parts of Portal would work poorly. On the other hand, Portal took the very clever step of offering “advanced” versions of several levels, which were altered very slightly to break all the obvious easy solutions.

Slapping on difficulty settings is nice for non-puzzle games (and even some puzzle games), but unless your game lets you change the difficulty partway through, someone who hits a wall still has to replay the entire game to change the difficulty. (Props to Doom 4, which looks to have taken difficulty levels very seriously — some have entirely different rules, and you can change whenever you want.)

I have a few wisps of ideas for how to deal with this in Isaac HD, but I can’t really talk about them before the design of the game has solidified a little more. Ultimately, my goal is the same as with everything else I do: to make something that people have a chance to enjoy, even if they don’t otherwise like the genre.

Build a remote control robot with The MagPi 51

Post Syndicated from Rob Zwetsloot original https://www.raspberrypi.org/blog/build-remote-control-robot-magpi-51/

Hi, Rob from The MagPi here! Issue 51 is out and just in case you weren’t sold on it already, here’s a little something to tempt you.

issue51-issue-cover

Brian Corteil, writer of the feature, has christened this robot ‘Tiny’

Over the past few years, Raspberry Pi robotics has really come into its own, taking strides to make building robots just that little bit more fun and accessible. This month in The MagPi, we’ve decided to take all these advances and use them to create an incredible little robot.

We’ll guide you through the process of making your robot while also giving you top advice on other methods for robot construction, in case you feel the spark of inspiration once the build is over. It’s not just robots we’re building this issue though. We have some amazing tutorials on the following:

  • Building an underwater camera
  • Finishing up our RaspCade arcade cabinet build
  • A guide to NOOBS for beginners
  • Using WiFi signals as a people detector

There’s lots more to enjoy, including reviews, columns, and a series of spooky and simple Halloween projects.

You can get hold of the latest issue in stores now from WHSmith, Tesco, Sainsbury’s, and Asda. Alternatively you can grab an issue online or get it digitally via our app on Android and iOS. There’s even a free PDF of it as well! You’re spoilt for choice…

Get a free Pi Zero
Want to make sure you never miss an issue? Subscribe today and get a Pi Zero bundle featuring the new, camera-enabled Pi Zero, together with a cable bundle that includes the camera adapter.

Free Pi Zeros and posters: what’s not to love about a MagPi subscription?

Free Creative Commons download
As always, you can download your copy of The MagPi completely free. Grab it straight from the issue page for The MagPi 51.

Don’t forget, though, that like sales of the Raspberry Pi itself, all proceeds from the print and digital editions of the magazine go to help the Foundation achieve its charitable goals. Help us democratise computing!

We hope you enjoy the issue!

The post Build a remote control robot with The MagPi 51 appeared first on Raspberry Pi.

Fact or Fiction: Google Big Query Outperforms Amazon Redshift as an Enterprise Data Warehouse?

Post Syndicated from Randall Hunt original https://aws.amazon.com/blogs/big-data/fact-or-fiction-google-big-query-outperforms-amazon-redshift-as-an-enterprise-data-warehouse/

Randall Hunt is a Technical Evangelist for Amazon Web Services

A few weeks ago, 2nd Watch (a leading cloud native Systems Integrator) wrote the Benchmarking Amazon Aurora post, analyzing Google’s benchmark of their Cloud SQL database service against AWS’s Amazon Aurora. In that analysis, 2nd Watch found that Aurora outperforms Cloud SQL consistently and that Google’s benchmarks were incorrect and misleading. 2nd Watch also pointed out the peculiar aspects in Google’s approach (i.e., artificially constraining database threads to remove Aurora’s advantage of much better performance at high thread counts—something most Aurora customers take advantage of). 2nd Watch then provided their take on what a fair and reasonable performance benchmark would look like.

On September 29, 2016, Google began presenting a new data warehouse performance benchmark that had many people scratching their heads again. This time, Google compared AWS’s Amazon Redshift with Google BigQuery. Similar to the previous example, the approach taken on the tests and the marketing claims they derived from the results were oddly misleading.

Publishing misleading performance benchmarks is a classic old guard marketing tactic. It’s not surprising to see old guard companies (like Oracle) doing this, but we were kind of surprised to see Google take this approach, too. So, when Google presented their BigQuery vs. Amazon Redshift benchmark results at a private event in San Francisco on September 29, 2016, it piqued our interest and we decided to dig deeper.

For their tests, Google used the TPC-H benchmark, which measures performance against 22 different queries and is typically used to evaluate data warehouses and other decision support systems. Instead of presenting the results from all the queries, as is the standard practice, Google cherry-picked one single query that generated favorable results for BigQuery (Query #22, which happens to be one of the least sophisticated queries—one with simple filters and no joins), and used this data to make the broad claim that BigQuery outperforms Amazon Redshift.

To verify Google’s claim with our own testing, we ran the full TPC-H benchmark, consisting of all 22 queries, using a 10 TB dataset on Amazon Redshift against the latest version of BigQuery. We set up Amazon Redshift with basic configurations that our customers typically put in place, like compression, distribution keys on large tables, and sort keys on commonly filtered columns. We used an 8-node DC1.8XL Amazon Redshift cluster for the tests. Below is a summary of our findings.

Amazon Redshift outperformed BigQuery on 18 of 22 TPC-H benchmark queries by an average of 3.6X

When we ran the entire 22-query benchmark, we found that Amazon Redshift outperformed BigQuery by 3.6X on average on 18 of 22 TPC-H queries. Looking at relative performance for the entire set of queries, Amazon Redshift outperforms BigQuery by 2X. The chart below summarizes the comparison of elapsed times between the two services for the entire TPC-H benchmark (lower is better).

o_Fact_1_2

There’s a reason why TPC-H was carefully designed to evaluate the various aspects of data warehouse performance. You can’t just cherry-pick the one query where a given product is the fastest. The workloads in the database domain have broad requirements, so to get a complete view of how a database will perform for your workloads, you need to show how the benchmarked product does overall—anyone can be fast running one type of query that’s optimized for their service.

TPC-DS is a better data warehousing benchmark in which Amazon Redshift is 6 times more performant

Even though Google used the simpler TPC-H benchmark for their study, TPC-DS is well-known in the industry as the right benchmark for data warehousing workloads because it uses larger tables and more complex queries representative of real analytics use cases. We ran the entire TPC-DS consisting of 99 queries on Amazon Redshift and BigQuery. Amazon Redshift outperformed BigQuery on 94 of the 99 queries. Nine of the 99 queries did not even run successfully on BigQuery because it doesn’t fully support standard SQL and has other limitations. For the rest of the queries that did run on BigQuery, Amazon Redshift was on average 6X faster than BigQuery. The chart below summarizes the comparison of elapsed times between the two services for the TPC-DS benchmark.

o_Fact_2_2

As an aside, we had to edit the SQL for 9 of the 99 TPC-DS queries to get them to run on BigQuery, which is technically not allowed in standard benchmark studies. Given Google’s claims that BigQuery now “supports standard SQL,” we were surprised to see incompatibility with queries that were completely within the standard ANSI SQL syntax. After some extensive testing, we determined that BigQuery doesn’t support at least 9 different pieces of standard ANSI SQL syntax. We believe that most customers are likely to struggle with BigQuery as an enterprise data warehouse if they can’t run their standard SQL queries without modifications. For details, see Appendix 1.

Additional factors to consider when benchmarking Amazon Redshift against BigQuery

Amazon Redshift offers customers far greater flexibility in configuring the service to get the desired level of performance for their workload. Amazon Redshift customers can choose a cluster configuration (1 to 125 nodes, HDD or SSD) that fits their price and performance requirements. Performance typically grows linearly with the number of nodes in a cluster. For example, a 10-node Amazon Redshift cluster can deliver an order of magnitude improvement in performance compared to a single-node cluster. Google’s BigQuery has no such option and distributes the data across a specific number of nodes by default. As a result, we have seen BigQuery’s performance fluctuate substantially as the same query against the same data set will often run twice (or ½) as fast on different days. This means that you have to live with the performance you get on BigQuery. This is also important to take into account when conducting your own benchmarking.

Conclusion

One of the great things about the cloud is the transparency that customers have in testing and debunking overstated performance claims and misleading “benchmark” tests. This transparency encourages the best cloud vendors to publish clear and repeatable performance metrics, making it faster and easier for their customers to select the right cloud service for a given workload. We encourage you to try benchmarking Amazon Redshift yourself to see how it will perform with your workload. Test it at scale, select the right tests for your application, and validate your assumptions. Many AWS customers, like NASDAQ, NTT Docomo, Amgen, Boingo Wireless, Cablevision, Johnson & Johnson and Philips Healthcare, have migrated their data warehouses to Amazon Redshift, increasing their performance while saving costs. We encourage you to review their experiences and apply it to your workloads as well.

 

 


Appendix 1: SQL limitations of BigQuery

While BigQuery claims to have added support for ANSI SQL, it still has several cases where query syntax deviates from ANSI SQL for unknown reasons. This requires users to learn these nuances and affects productivity. Below is a sample of such limitations:

  1. Does not support the ANSI SQL syntax of substring. Need to change to substr.
  2. Does not support Top N. Only supports LIMIT N at the end of the SQL
  3. Does not support NUMERIC. Only supports INT or FLOAT. There is a loss of precision on many queries (data quality). TPC-DS queries set a specific number of positions for intermediate calculations with CAST. It is not supported by BigQuery.
  4. DATE datatype is new and none of BigQuery’s functions support it yet in ANSI. Even if you try to convert to TIMESTAMP first, DATEDIFF and DATE_ADD does not work. BigQuery in “Legacy SQL” mode supports this but then joins don’t really work.
  5. While Group by <expression> allows positioning references, copying an expression (substr for example) from the projection down to the group by is not allowed. You have to use positioning.
  6. Does not support INTERSECT or EXCEPT.
  7. Does not support UNION syntax. Requires you to use UNION DISTINCT or ALL.
  8. Column Alias needs to be a valid identifier. Should only be letters or numbers. No quotes can be used. So “SUM OF SALES” is not a valid alias.
  9. You always have to prefix the dataset name on all SQL. No concept of search path or default schema/database.

The 50 greatest Pi projects ever in The MagPi 50

Post Syndicated from Rob Zwetsloot original https://www.raspberrypi.org/blog/50-greatest-pi-projects-ever-magpi-50/

Rob from The MagPi here! We’re absolutely thrilled finally to be able to share with you The MagPi 50, our landmark issue with a super special feature on the 50 greatest Raspberry Pi projects of all time, the top 20 of which were voted on by you, the Raspberry Pi community.

The MagPi magazine issue 50: silver text on the cover reads "50 greatest Raspberry Pi projects"

The MagPi 50, out right this instant

As well as the thousands who voted, we had a panel of judges choosing the best projects in a few special categories. Eben Upton, the man behind Raspberry Pi, gave us his picks of software projects. Philip Colligan, CEO of the Raspberry Pi Foundation, carefully selected some incredible humanitarian projects. Liz Upton, Director of Communications/my boss, made some tough decisions in the young makers category. Finally, Michael Horne and Tim Richardson of CamJam and Pi Wars fame presided over the Pi robots.

No Title

No Description

Hopefully your favourite project made its way into the top 50! It was a hard task whittling it down to this number, and to be perfectly honest we could probably feature another 50 projects next month that are equally good. The Raspberry Pi community has done some incredible things over the last four years and change, and I’m immensely proud that we can share some of the outstanding work you folk have done in this issue.

No Title

No Description

But wait, there’s more! As well as our big community celebration, we also have our usual selection of excellent tutorials, news, and reviews. If the reveal of USB and Ethernet booting on Pi 3 piqued your interest a few weeks ago, we have a full eight-page guide on how you can do that yourself. We cover the #10MillionPi event at the Houses of Parliament in the news, along with some wonderful Raspberry Pi-powered tech that’s being used in the health industry.

Also, here’s Mike’s dancing skeleton from the Pi Bakery, in plenty of time for you to get your own spooky version ready for Halloween. We love it.

Danse Macabre

Danse Macabre or Skeleton Dance is a project in the MagPi Magazine No.50 October 2016. It uses the spectrum board from The MagPi No. 46 June 2016 ( https://vimeo.com/167914646 ) , to make one to three skeletons dance to music.

You can grab The MagPi 50 in stores today: it’s in WHSmith, Tesco, Sainsbury’s, and Asda in the UK, and it will be in Micro Center and selected Barnes & Noble stores when it comes to America. You can also buy the print edition online from our store, and it’s available digitally on our Android and iOS app.

Get a free Pi Zero
Want to make sure you never miss an issue? Subscribe today and get a Pi Zero bundle featuring the new, camera-enabled Pi Zero, and a cable bundle that includes the camera adapter.

Free Pi Zeros and posters: what’s not to love about a MagPi subscription?

Free Creative Commons download
As always, you can download your copy of The MagPi completely free. Grab it straight from the issue page for The MagPi 50.

Don’t forget, though, that like sales of the Raspberry Pi itself, all proceeds from the print and digital editions of the magazine go to help the Foundation achieve its charitable goals. Help us democratise computing!

We hope you enjoy this issue. We’re off for a cup of tea. See you soon!

The post The 50 greatest Pi projects ever in The MagPi 50 appeared first on Raspberry Pi.

Encrypt Data At-Rest and In-Flight on Amazon EMR with Security Configurations

Post Syndicated from Jonathan Fritz original https://blogs.aws.amazon.com/bigdata/post/Tx31P2UUJKR4ONF/Encrypt-Data-At-Rest-and-In-Flight-on-Amazon-EMR-with-Security-Configurations

Customers running analytics, stream processing, machine learning, and ETL workloads on personally identifiable information, health information, and financial data have strict requirements for encryption of data at-rest and in-transit. The Apache Spark and Hadoop ecosystems lend themselves to these big data use cases, and customers have asked us to provide a quick and easy way to encrypt data at-rest and data in-transit between nodes in each execution framework.

With the release of security configurations for Amazon EMR release 5.0.0 and 4.8.0, customers can now easily enable encryption for data at-rest in Amazon S3, HDFS, and local disk, and enable encryption for data in-flight in the Apache Spark, Apache Tez, and Apache Hadoop MapReduce frameworks.

Security configurations make it easy to specify the encryption keys and certificates to use, ranging from AWS Key Management Service to supplying your own custom encryption materials provider (for an example of custom providers, see the Nasdaq about EMRFS and Amazon S3 client-side encryption post). Additionally, you can apply a security configuration to multiple clusters, making it easy to standardize your security settings. For instance, this makes it easy for customers to encrypt data across their HIPAA-compliant Amazon EMR workloads.

The following is an example security configuration specifying SSE-KMS for Amazon S3 encryption (using EMRFS), AWS KMS key for local disk encryption (which will also encrypt HDFS blocks), and a set of TLS certificates in Amazon S3 for applications that require them for encryption in-transit:

After you create a security configuration, you can specify it when creating a cluster and apply the settings. Security configurations can also be created using the AWS CLI or SDK. For more information, see Encrypting Data with Amazon EMR. If you have any questions or would like to share an interesting use case about encryption on Amazon EMR, please leave a comment below.

 

Encrypt Data At-Rest and In-Flight on Amazon EMR with Security Configurations

Post Syndicated from Jonathan Fritz original https://aws.amazon.com/blogs/big-data/encrypt-data-at-rest-and-in-flight-on-amazon-emr-with-security-configurations/

Customers running analytics, stream processing, machine learning, and ETL workloads on personally identifiable information, health information, and financial data have strict requirements for encryption of data at-rest and in-transit. The Apache Spark and Hadoop ecosystems lend themselves to these big data use cases, and customers have asked us to provide a quick and easy way to encrypt data at-rest and data in-transit between nodes in each execution framework.

With the release of security configurations for Amazon EMR release 5.0.0 and 4.8.0, customers can now easily enable encryption for data at-rest in Amazon S3, HDFS, and local disk, and enable encryption for data in-flight in the Apache Spark, Apache Tez, and Apache Hadoop MapReduce frameworks.

Security configurations make it easy to specify the encryption keys and certificates to use, ranging from AWS Key Management Service to supplying your own custom encryption materials provider (for an example of custom providers, see the Nasdaq about EMRFS and Amazon S3 client-side encryption post). Additionally, you can apply a security configuration to multiple clusters, making it easy to standardize your security settings. For instance, this makes it easy for customers to encrypt data across their HIPAA-compliant Amazon EMR workloads.

The following is an example security configuration specifying SSE-KMS for Amazon S3 encryption (using EMRFS), AWS KMS key for local disk encryption (which will also encrypt HDFS blocks), and a set of TLS certificates in Amazon S3 for applications that require them for encryption in-transit:

After you create a security configuration, you can specify it when creating a cluster and apply the settings. Security configurations can also be created using the AWS CLI or SDK. For more information, see Encrypting Data with Amazon EMR. If you have any questions or would like to share an interesting use case about encryption on Amazon EMR, please leave a comment below.

 

Begin your journey with Raspberry Pi in The MagPi 49

Post Syndicated from Rob Zwetsloot original https://www.raspberrypi.org/blog/begin-your-journey-with-raspberry-pi-in-the-magpi-49/

We’ve all seen the numbers. The Raspberry Pi is selling faster and faster every year, which means there are new people getting Raspberry Pis every day. With this in mind, we decided to make a brand new beginner’s guide in issue 49 of The MagPi, out now.

Get started with Raspberry Pi with The MagPi 49!

Get started with Raspberry Pi with The MagPi 49!

The Raspberry Pi beginner’s guide takes you from selecting your Raspberry Pi all the way through setting it up and getting to know the Raspbian OS that powers it. We’re also using it to jump-start a beginner’s tutorial that will be a monthly feature in The MagPi from now on.

#49 Apollo Pi

Set your Pi up so it can take you to the moon! (Moon rocket not included)

As well as the cover feature, we also have a feature on the recently released Apollo 11 source code and how you can emulate a virtual Apollo computer on your Raspberry Pi, along with some historical factoids about making and programming a computer to take people to the moon. There’s also our usual range of amazing tutorials, projects, and product reviews for you to read about as well, including Mike Cook’s rhythmic gymnastics project in the Pi Bakery.

Rhythmic Gymnastics Ribbons

Inspired by the Rio Olympics Gymnastic display of ribbon twirling. In the MagPi 49 – September 2016, https://www.raspberrypi.org/magpi/ twirl your own virtual ribbons.

You can grab the latest issue of The MagPi in stores today from WH Smith, Tesco, Sainsbury’s, and Asda in the UK, and it will be in Micro Center and selected Barnes & Noble stores when it comes to America. It’s also available in print online from our store, and digitally on our Android and iOS app.

Get a free Pi Zero
Want to make sure you never miss an issue? Subscribe today and start with issue 47 to get not only the Astro Pi poster and mission patch, but also a Pi Zero bundle featuring the new, camera-enabled Pi Zero, and a cable bundle that includes the camera adapter.

Free Pi Zeros and posters: what’s not to love about a MagPi subscription?

Free Creative Commons download
As always, you can download your copy of The MagPi completely free. Grab it straight from the issue page for The MagPi 49.

Don’t forget, though, that like sales of the Raspberry Pi itself, all proceeds from the print and digital editions of the magazine go to help the Foundation achieve its charitable goals. Help us democratise computing!

I also want to remind you that we’re running a poll to find out what you, the community, think are the top 20 Raspberry Pi projects to be included in our 50th issue spectacular. Get voting!

No Title

No Description

The post Begin your journey with Raspberry Pi in The MagPi 49 appeared first on Raspberry Pi.

Learn how to make with Windows 10 IoT Core in The MagPi 48

Post Syndicated from Rob Zwetsloot original https://www.raspberrypi.org/blog/learn-make-windows-10-iot-core-magpi-48/

Rob here from The MagPi. It’s the last Thursday of the month, which can only mean one thing: a new issue is out!

Windows 10 is better than ever on Raspberry Pi

Windows 10 is better than ever on Raspberry Pi

Whenever a new piece of hardware comes out, there are always people trying to port or emulate different operating systems onto them. The Raspberry Pi was no different, with several attempts at porting differing operating systems when it was first launched. For over a year now though, Microsoft has officially supported Windows on the Raspberry Pi through Windows 10 IoT Core.

In The MagPi 48 we cover the latest developments in Windows 10 IoT Core that have come about since the Raspberry Pi 3 was launched, and how to make use of them in your own projects. We’ve also got exclusive news on an upcoming kit specifically for the Raspberry Pi 3 that lets you create amazing projects right out of the box.

Compete in the Scratch Olympics. You don't even have to leave your house

Compete in the Scratch Olympics. You don’t even have to leave your house.

As well as all the Windows talk, we invite you to take part in the Scratch Olympics, continue building the arcade machine of your dreams, learn about Twitch-controlled robots, and read a review of the long-awaited NatureBytes wildlife camera.

You can also learn how to make this swimming game from the legendary Mike Cook, which involves paddling your arms wildly in the air in the general direction of a home-built sensor board to control your character.

Raspberry Pi Olympic swimming

From the The MagPi 48 – August 2016 – an Olympic swimming simulator for the Raspberry Pi.

The MagPi 48 is out today in WH Smith, Tesco, Sainsburys, and Asda in the UK and will be in Micro Center and selected Barnes & Noble when it comes to America. You can also buy a copy online from our store, or get it digitally on our app that’s available for iOS and Android.

Get a free Pi Zero
Want to make sure you never miss an issue? Subscribe today and start with issue 47 to not only get the poster and mission patch, but also a Pi Zero bundle featuring the new, camera-enabled Pi Zero and a cable bundle that includes the camera adapter.

Free Pi Zeros and posters: what’s not to love about a MagPi subscription?

Free Creative Commons download
As always, you can download your copy of The MagPi completely free. Grab it straight from the issue page for The MagPi 48.

Don’t forget, though, that like sales of the Raspberry Pi itself, all proceeds from the print and digital editions of the magazine go to help the Foundation achieve its charitable goals. Help us democratise computing!

We hope you enjoy the issue! We’re off for a haircut.

The post Learn how to make with Windows 10 IoT Core in The MagPi 48 appeared first on Raspberry Pi.

Blast off with The MagPi 47 Astro Pi special!

Post Syndicated from Rob Zwetsloot original https://www.raspberrypi.org/blog/blast-off-magpi-47-astro-pi-special/

Get your free poster and mission patch exclusively in the print edition of The MagPi 47!

Get your free poster and mission patch exclusively in the print edition of The MagPi 47!

We’ve been avidly following Tim Peake’s adventures in space in The MagPi for the last six months, especially all the excellent work he’s been doing with the Astro Pis running code from school students across the UK. Tim returned to Earth a couple of weeks ago, so we thought we’d celebrate in The MagPi 47 with a massive feature about his time in space, along with the results of the Astro Pi experiments and the project’s future…

The space celebration doesn’t stop there: print copies of The MagPi 47 come with an exclusive Astro Pi mission patch and a Tim Peake Astro Pi poster!

The results of what Tim, Ed, and Izzy have been up to for the past six months

The results of what Tim, Ed, and Izzy have been up to for the past six months

The issue also has our usual range of excellent tutorials, from programming dinosaurs to creating motion sensor games and optical illusions. We also have the hottest news on high-altitude balloons and how you can get involved in sending a Pi to the edge of space, as well as the details on the next Pi Wars Pi-powered robot competition.

You can get your latest spaceworthy issue in-store from WH Smith, Tesco, Sainsburys, and Asda. Our American cousins will be able to buy issues from Barnes & Noble and Micro Center when the issue makes its way over there. It’s also available right now in print on our online store, which delivers internationally. If you prefer digital, it’s ready to download on the Android and iOS apps.

Get a free Pi Zero
Want to make sure you never miss an issue? Subscribe today and start with issue 47 to not only get the poster and mission patch, but also a Pi Zero bundle featuring the new, camera-enabled Pi Zero and a cable bundle that includes the camera adapter.

Free Pi Zeros and posters: what’s not to love about a MagPi subscription?

Free Creative Commons download
As always, you can download your copy of The MagPi completely free. Grab it straight from the issue page for The MagPi 47.

Don’t forget, though, that like sales of the Raspberry Pi itself, all proceeds from the print and digital editions of the magazine go to help the Foundation achieve its charitable goals. Help us democratise computing!

This is not the end of Astro Pi. It’s only the beginning.

The post Blast off with The MagPi 47 Astro Pi special! appeared first on Raspberry Pi.

Amazon Redshift – Up to 2X Throughput and 10X Vacuuming Performance Improvements

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/amazon-redshift-up-to-2x-throughput-and-10x-vacuuming-performance-improvements/

My colleague Maor Kleider wrote today’s guest post!


Jeff;


Amazon Redshift, AWS’s fully managed data warehouse service, makes petabyte-scale data analysis fast, cheap, and simple. Since launch, it has been one of AWS’s fastest growing services, with many thousands of customers across many industries. Enterprises such as NTT DOCOMO, NASDAQ, FINRA, Johnson & Johnson, Hearst, Amgen, and web-scale companies such as Yelp, Foursquare and Yahoo! have made Amazon Redshift a key component of their analytics infrastructure.

In this blog post, we look at performance improvements we’ve made over the last several months to Amazon Redshift, improving throughput by more than 2X and vacuuming performance by 10X.

Column Store
Large scale data warehousing is largely an I/O problem, and Amazon Redshift uses a distributed columnar architecture to minimize and parallelize I/O. In a column-store, each column of a table is stored in its own data block. This reduces data size, since we can choose compression algorithms optimized for each type of column. It also reduces I/O time during queries, because only the columns in the table that are being selected need to be retrieved.

However, while a column-store is very efficient at reading data, it is less efficient than a row-store at loading and committing data, particularly for small data sets. In patch 1.0.1012 (December 17, 2015), we released a significant improvement to our I/O and commit logic. This helped with small data loads and queries using temporary tables. While the improvements are workload-dependent, we estimate the typical customer saw a 35% improvement in overall throughput.

Regarding this feature, Naeem Ali, Director of Software Development, Data Science at Cablevision, told us:

Following the release of the I/O and commit logic enhancement, we saw a 2X performance improvement on a wide variety of workloads. The more complex the queries, the higher the performance improvement.

Improved Query Processing
In addition to enhancing the I/O and commit logic for Amazon Redshift, we released an improvement to the memory allocation for query processing in patch 1.0.1056 (May 17, 2016), increasing overall throughput by up to 60% (as measured on standard benchmarks TPC-DS, 3TB), depending on the workload and the number of queries that spill from memory to disk. The query throughput improvement increases with the number of concurrent queries, as less data is spilled from memory to disk, reducing required I/O.

Taken together, these two improvements, should double performance for customer workloads where a portion of the workload contains complex queries that spill to disk or cause temporary tables to be created.

Better Vacuuming
Amazon Redshift uses multi-version concurrency control to reduce contention between readers and writers to a table. Like PostgreSQL, it does this by marking old versions of data as deleted and new versions as inserted, using the transaction ID as a marker. This allows readers to build a snapshot of the data they are allowed to see and traverse the table without locking. One issue with this approach is the system becomes slower over time, requiring a vacuum command to reclaim the space. This command reclaims the space from deleted rows and ensures new data that has been added to the table is placed in the right sorted order.

We are releasing a significant performance improvement to vacuum in patch 1.0.1056, available starting May 17, 2016. Customers previewing the feature have seen dramatic improvements both in vacuum performance and overall system throughput as vacuum requires less resources.

Ari Miller, a Software Development Manager at TripAdvisor, told me:

We estimate that the vacuum operation on a 15TB table went about 10X faster with the recent patch, ultimately improving overall query performance.

 You can query the
VERSION function to verify that you are running at the desired patch level.

Available Now
Unlike on-premise data warehousing solutions, there are no license or maintenance fees for these improvements or work required on your part to obtain them. They simply show up as part of the automated patching process during your maintenance window.

Maor Kleider, Senior Product Manager, Amazon Redshift

 

Learn all about the new Raspberry Pi Camera Module v2 in The MagPi 45

Post Syndicated from Rob Zwetsloot original https://www.raspberrypi.org/blog/learn-new-raspberry-pi-camera-module-v2-magpi-45/

Earlier this week, the brand new Raspberry Pi Camera Module v2 was revealed to the world, its headline feature being an 8-megapixel sensor. It’s been a few years since the original came out and the new camera is an excellent little upgrade to the existing model; you can find out all the details in our complete breakdown in Issue 45 of The MagPi magazine, which is out today.

Picture perfect, the new Pi Camera Module v2

Picture perfect, the new Pi Camera Module v2

As well as covering the camera and giving you some projects to start you off with it, we also have a look at the ten best Pi-powered arcade machines, which should give you some ideas for a retro games system of your own. There are also tutorials on creating lighting effects for costumes with a Pi and some NeoPixels, making an Asteroids clone in Basic, and building an IoT thermometer. We also have Astro Pi news, excellent projects, reviews, and everything else you’d expect from your monthly MagPi.

A model railway, in-part powered by Pi Zero

A model railway, powered in-part by Pi Zero

Highlights from issue 45:

  • Replicate an Astro Pi experiment
    Create a humidity sensor, similar to the Sweaty Astronaut code
  • Hacking with dinosaurs
    The MagPi heads to the Isle of Wight to see how some animatronic dinos are being hacked with Pi
  • Original games on the Pi
    Play three brand-new games on your Pi thanks to YoYo and GameMaker Studio
  • Moon pictures
    Find out how to use the camera board to take amazing photos of the moon
  • And much, much more!

How to buy
As usual, you can get The MagPi in store from WH Smith, Tesco, Sainsbury’s, and Asda as well as buying copies online from our store. It’s also available digitally via our app on Android and iOS. If you fancy subscribing to the magazine to make sure you never miss an issue, you can do that to on our subscription site.

Free Creative Commons download
As always, you can download your copy of The MagPi completely free. Grab it straight from the issue page for The MagPi 45.

Don’t forget, though, that like sales of the Raspberry Pi itself, all proceeds from the print and digital editions of the magazine go to help the Foundation achieve its charitable goals. Help us democratise computing!

We hope you enjoy this month’s issue! Before anyone asks, no, the magazine unfortunately does not come with a free camera. Sorry!

The post Learn all about the new Raspberry Pi Camera Module v2 in The MagPi 45 appeared first on Raspberry Pi.

A media player for Scott

Post Syndicated from Liz Upton original https://www.raspberrypi.org/blog/media-player-scott/

Projects don’t have to be hugely complicated to make a huge difference. In Luxembourg, Alain Wall has used a Raspberry Pi to make a very simple media player for his autistic son, Scott. It’s very easy to use, very robust, and easy to clean; and it offers Scott a limited (so not overwhelming) but meaningful degree of choice. Here’s Scott using his player. Watch to the end for the best smile in the world.

Dem Scott sain neien TV. Scott’s new TV

Hei ass den Scott deen sain neien Mediaplayer test. En kann sech seng Filmer selwer starten an stoppen. A media player nearly indestructible an controllable with 6 Buttons to choose a movie Deutsch: http://awallelectronic.blogspot.lu/2016/04/scott-tv.html English: http://www.instructables.com/id/ScottTV-a-Simple-Media-Player-for-My-Austic-Son/ or https://hackaday.io/project/11000-scotttv-a-simple-mediaplayer-for-my-autistic-son

Alain hooked up six big piezo buttons and some speakers to a 20-in monitor and a Raspberry Pi – this isn’t the most complicated build you’ll see around these parts. (You can see a how-to guide over at Instructables.) But it is one of the most effective: as Alain says, “Scott loves it.”

Here’s another video from Alain demonstrating the setup.

Scott TV Simple MediaPlayer For My Autistic Son Scott

This is a simple media player for my autistic son. It had to be easy to use, nearly indestructible and easy to clean http://www.instructables.com/id/ScottTV-a-Simple-Media-Player-for-My-Austic-Son/ Deutsch: http://awallelectronic.blogspot.lu/2016/04/scott-tv.html

Thanks very much for sharing the project, Alain; all the very best from us at Pi Towers to you and the rest of the family, especially Scott!

 

The post A media player for Scott appeared first on Raspberry Pi.