We see lots of addressable LED projects, but there was something weirdly charming and very pretty about this cylinder of squares. It’d make for a lovely interactive nightlight in a kids’ room, or for a grown-up lighting feature that you could also use as a news ticker or something that monitors your in-home IoT devices. Once you’ve built something like this, you’re only limited by your imagination — and it’s nice enough to display in your home.
This project is from makeTVee on Instructables. The cleverness is in the layout and the really meticulous execution: vertical strips of LEDs form a cylinder in a laser-cut frame, with a very thin layer of wood veneer glued around the whole thing to act as a diffuser. It’s simple, but really rather beautiful and very effective.
In the case to the side is the Raspberry Pi Zero that’s driving the whole thing. Here it is doing its thing:
LED matrix cylinder based on WS2812 LEDs and some laser cutter parts. https://hackaday.io/project/162035-led-matrix-cylinder https://www.instructables.com/id/LED-Matrix-Cylinder/ #WS2812 #LEDcylinder
makeTVee has built a Pygame-based simulator of the whole matrix so you can program it to do exactly what you want: scroll marquee text, make pretty patterns, twinkle at random, display images: the world’s your (pixellated) oyster. The code’s available at GitHub.
Thanks, makeTVee — if you’d like to leave your real name below, we’ll credit you properly here!
Once I had the three separate files, I laser-engraved them onto 4cm-wide pieces of 3mm-thick clear acrylic. As there are four lines of LEDs on the Unicorn pHAT, I cut the fourth piece to illuminate the background.
To keep the engraved acrylic pieces together, I cut out a pair of acrylic brackets (see above) with four 3mm indentations. Then, after a bit of fiddling with the Unicorn pHAT library, I was able to light the pHAT’s rows of LEDs in white, red, green, and white.
The final result looks pretty spectacular, especially in the dark, and you can build on this basic idea to create fun animations — especially if you use a HAT with more rows of LEDs.
Iterations
This is just a prototype. I plan on building a sturdier frame for the pieces that securely fits a Raspberry Pi Zero W and lets users replace layers easily. As with many projects, I’m sure this will grow and grow as each interaction inspires a new add-on.
Create something beautiful with silicon, electricity, your endless imagination, and HackSpace magazine issue 16 — out today!
LEDs are awesome
Basically, LEDs are components that convert electrical power into light. Connect them to a power source (with some form of current limiter) in the right orientation, and they’ll glow.
Each LED has a single colour. Fortunately, manufacturers can pack three LEDs (red, green, and blue) into a single component, and varying the power to each LED-within-an-LED produces a wide range of hues. However, by itself, this type of colourful LED is a little tricky to control: each requires three inputs, so a simple 10×10 matrix would require 300 inputs. But there’s a particular trick electronics manufacturers have that make RGB LEDs easy to use: making the LEDs addressable!
Look: you can clearly see the red, green, and blue elements of this RGB LED
Addressable LEDs
Addressable LEDs have microcontrollers built into them. These aren’t powerful, programmable microcontrollers, they’re just able to handle a simple communications protocol. There are quite a few different types of addressable LEDs, but two are most popular with makers: WS2812 (often called NeoPixels) and APA102 (often called DotStars). Both are widely available from maker stores and direct-from-China websites. NeoPixels use a single data line, while DotStars use a signal and a clock line. Both, however, are chainable. This means that you connect one (for NeoPixels) or two (for DotStars) pins of your microcontroller to the Data In connectors on the first LED, then the output of this LED to the input of the next, and so on.
Exactly how many LEDs you can chain together depends on a few different things, including the power of the microcontroller and the intended refresh rate. Often, though, the limiting factor for most hobbyists is the amount of electricity you need.
Which type to use
The big difference between NeoPixels and DotStars comes down to the speed of them. LEDs are made dimmer by turning them off and on very quickly. The proportion of the time they’re off, the dimmer they are. This is known as pulse-width modulation (PWM). The speed at which this blinking on and off can have implications for some makes, such as when the LEDs are moving quickly.
NeoPixels
Cheap
Slowish refresh rate
Slowish PWM rate
DotStars
More expensive
Faster refresh rate
Fast PWM rate
As a NeoPixel is moved through a long-exposure photograph, you can see it blink on and off. DotStars – which have a faster PWM rate – avoid this.
Safety first!
HackSpace magazine’s LED feature is just a whistle-stop guide to the basics of powering LEDs — it’s not a comprehensive guide to all things power-related. Once you go above a few amperes, you need to think about what you’re doing with power. Once you start to approach double figures, you need to make sure you know what you’re doing and, if you find yourself shopping for an industrial power supply, then you really need to make sure you know how to use it safely.
Read more
Read the rest of the exclusive 14-page LED special in HackSpace magazine issue 16, out today. Buy your copy now from the Raspberry Pi Press store, major newsagents in the UK, or Barnes & Noble, Fry’s, or Micro Center in the US. Or, download your free PDF copy from the HackSpace magazine website.
We’re also shipping to stores in Australia, Hong Kong, Canada, Singapore, Belgium, and Brazil, so be sure to ask your local newsagent whether they’ll be getting HackSpace magazine.
Subscribe now
Subscribe to HackSpace on a monthly, quarterly, or twelve-month basis to save money against newsstand prices.
Twelve-month print subscribers get a free Adafruit Circuit Playground Express, loaded with inputs and sensors and ready for your next project. Tempted?
Python code creates curious, wordless comic strips at random, spewing them from the thermal printer mouth of a laser-cut body reminiscent of Disney Pixar’s WALL-E: meet the Vomit Comic Robot!
The age of the thermal printer!
Thermal printers allow you to instantly print photos, data, and text using a few lines of code, with no need for ink. More and more makers are using this handy, low-maintenance bit of kit for truly creative projects, from Pierre Muth’s tiny PolaPi-Zero camera to the sound-printing Waves project by Eunice Lee, Matthew Zhang, and Bomani McClendon (and our own Secret Santa Babbage).
Vomiting robots
Interaction designer and developer Cadin Batrack, whose background is in game design and interactivity, has built the Vomit Comic Robot, which creates “one-of-a-kind comics on demand by processing hand-drawn images through a custom software algorithm.”
The robot is made up of a Raspberry Pi 3, a USB thermal printer, and a handful of LEDs.
At the press of a button, Processing code selects one of a set of Cadin’s hand-drawn empty comic grids and then randomly picks images from a library to fill in the gaps.
Each image is associated with data that allows the code to fit it correctly into the available panels. Cadin says about the concept behing his build:
Although images are selected and placed randomly, the comic panel format suggests relationships between elements. Our minds create a story where there is none in an attempt to explain visuals created by a non-intelligent machine.
The Raspberry Pi saves the final image as a high-resolution PNG file (so that Cadin can sell prints on thick paper via Etsy), and a Python script sends it to be vomited up by the thermal printer.
For more about the Vomit Comic Robot, check out Cadin’s blog. If you want to recreate it, you can find the info you need in the Imgur album he has put together.
We cute robots
We have a soft spot for cute robots here at Pi Towers, and of course we make no exception for the Vomit Comic Robot. If, like us, you’re a fan of adorable bots, check out Mira, the tiny interactive robot by Alonso Martinez, and Peeqo, the GIF bot by Abhishek Singh.
We’re usually averse to buzzwords at HackSpace magazine, but not this month: in issue 7, we’re taking a deep dive into the Internet of Things.
Internet of Things (IoT)
To many people, IoT is a shady term used by companies to sell you something you already own, but this time with WiFi; to us, it’s a way to make our builds smarter, more useful, and more connected. In HackSpace magazine #7, you can join us on a tour of the boards that power IoT projects, marvel at the ways in which other makers are using IoT, and get started with your first IoT project!
Awesome projects
DIY retro computing: this issue, we’re taking our collective hat off to Spencer Owen. He stuck his home-brew computer on Tindie thinking he might make a bit of beer money — now he’s paying the mortgage with his making skills and inviting others to build modules for his machine. And if that tickles your fancy, why not take a crack at our Z80 tutorial? Get out your breadboard, assemble your jumper wires, and prepare to build a real-life computer!
Shameless patriotism: combine Lego, Arduino, and the car of choice for 1960 gold bullion thieves, and you’ve got yourself a groovy weekend project. We proudly present to you one man’s epic quest to add LED lights (controllable via a smartphone!) to his daughter’s LEGO Mini Cooper.
Makerspaces
Patriotism intensifies: for the last 200-odd years, the Black Country has been a hotbed of making. Urban Hax, based in Walsall, is the latest makerspace to show off its riches in the coveted Space of the Month pages. Every space has its own way of doing things, but not every space has a portrait of Rob Halford on the wall. All hail!
Diversity: advice on diversity often boils down to ‘Be nice to people’, which might feel more vague than actionable. This is where we come in to help: it is truly worth making the effort to give people of all backgrounds access to your makerspace, so we take a look at why it’s nice to be nice, and at the ways in which one makerspace has put niceness into practice — with great results.
And there’s more!
We also show you how to easily calculate the size and radius of laser-cut gears, use a bank of LEDs to etch PCBs in your own mini factory, and use chemistry to mess with your lunch menu.
All this plus much, much more waits for you in HackSpace magazine issue 7!
Get your copy of HackSpace magazine
If you like the sound of that, you can find HackSpace magazine in WHSmith, Tesco, Sainsbury’s, and independent newsagents in the UK. If you live in the US, check out your local Barnes & Noble, Fry’s, or Micro Center next week. We’re also shipping to stores in Australia, Hong Kong, Canada, Singapore, Belgium, and Brazil, so be sure to ask your local newsagent whether they’ll be getting HackSpace magazine.
A simple Raspberry Pi based project using TCS3200 Color Sensor. The project demonstrates how to interface a Color Sensor (like TCS3200) with Raspberry Pi and implement a simple Color Detector using Raspberry Pi.
What is a TCS3200 colour sensor?
Colour sensors sense reflected light from nearby objects. The bright light of the TCS3200’s on-board white LEDs hits an object’s surface and is reflected back. The sensor has an 8×8 array of photodiodes, which are covered by either a red, blue, green, or clear filter. The type of filter determines what colour a diode can detect. Then the overall colour of an object is determined by how much light of each colour it reflects. (For example, a red object reflects mostly red light.)
As Electronics Hub explains:
TCS3200 is one of the easily available colour sensors that students and hobbyists can work on. It is basically a light-to-frequency converter, i.e. based on colour and intensity of the light falling on it, the frequency of its output signal varies.
I’ll save you a physics lesson here, but you can find a detailed explanation of colour sensing and the TCS3200 on the Electronics Hub blog.
Raspberry Pi colour sensor
The TCS3200 colour sensor is connected to several of the onboard General Purpose Input Output (GPIO) pins on the Raspberry Pi.
These connections allow the Raspberry Pi 3 to run one of two Python scripts that Electronics Hub has written for the project. The first displays the RAW RGB values read by the sensor. The second detects the primary colours red, green, and blue, and it can be expanded for more colours with the help of the first script.
Electronic Hub’s complete build uses a breadboard for simply prototyping
Use it in your projects
This colour sensing setup is a simple means of adding a new dimension to your builds. Why not build a candy-sorting robot that organises your favourite sweets by colour? Or add colour sensing to your line-following buggy to allow for multiple path options!
If your Raspberry Pi project uses colour sensing, we’d love to see it, so be sure to share it in the comments!
Today is Easter Monday and as such, the drawbridge is up at Pi Towers. So while we spend time with family…too much chocolate…family and chocolate, here are some great Pi-themed videos from members of our community. Enjoy!
Allie assembles this Google Home kit, that runs on a Raspberry Pi, then uses the Google Home to test her space knowledge with a little trivia game. Stay tuned at the end to see a few printed cases you can use instead of the cardboard.
Mission date : March 26 2018 My raspberry pi project. I use LTE modem to connect internet. python programming. raspberry pi controls pi cam, 2servo motor, 2dc motor. (This video recoded with gopro to upload youtube. Actually I controll this rover by pi cam.
I built my first security camera with motion-control connected to my raspberry pi with MotionEyeOS. What you need: *Raspberry pi 3 (I prefer pi 3) *Any Webcam or raspberry pi cam *Mirco SD card (min 8gb) Useful links : Download the motioneyeOS software here ➜ https://github.com/ccrisan/motioneyeos/releases How to do it: – Download motioneyeOS to your empty SD card (I mounted it via Etcher ) – I always do a sudo apt-upgrade & sudo apt-update on my projects, in the Pi.
Looking to incorporate some digital making into your Easter weekend? You’ve come to the right place! With a Raspberry Pi, a few wires, and some simple code, you can take your festivities to the next level — here’s how!
If you logged in to watch our Instagram live-stream yesterday, you’ll have seen me put together a simple egg carton and some wires to create circuits. These circuits, when closed by way of a foil-wrapped chocolate egg, instruct a Raspberry Pi to reveal the whereabouts of a larger chocolate egg!
Make it
You’ll need an egg carton, two male-to-female jumper wire, and two crocodile leads for each egg you use.
Connect your leads together in pairs: one end of a crocodile lead to the male end of one jumper wire. Attach the free crocodile clips of two leads to each corner of the egg carton (as shown up top). Then hook up the female ends to GPIO pins: one numbered pin and one ground pin per egg. I recommend pins 3, 4, 18 and 24, as they all have adjacent GND pins.
Your foil-wrapped Easter egg will complete the circuit — make sure it’s touching both the GPIO- and GND-connected clips when resting in the carton.
Wrap it
For your convenience (and our sweet tooth), we tested several foil-wrapped eggs (Easter and otherwise) to see which are conductive.
We’re egg-sperimenting with Easter deliciousness to find which treat is the most conductive. Why? All will be revealed in our Instagram Easter live-stream tomorrow.
The result? None of them are! But if you unwrap an egg and rewrap it with the non-decorative foil side outward, this tends to work. You could also use aluminium foil or copper tape to create a conductive layer.
Code it
Next, you’ll need to create the code for your hunt. The script below contains the bare bones needed to make the project work — you can embellish it however you wish using GUIs, flashing LEDs, music, etc.
Open Thonny or IDLE on Raspbian and create a new file called egghunt.py. Then enter the following code:
We’re using ButtonBoard from the gpiozero library. This allows us to link several buttons together as an object and set an action for when any number of the buttons are pressed. Here, the script waits for all four circuits to be completed before printing the location of the prize in the Python shell.
Your turn
And that’s it! Now you just need to hide your small foil eggs around the house and challenge your kids/friends/neighbours to find them. Then, once every circuit is completed with an egg, the great prize will be revealed.
Give it a go this weekend! And if you do, be sure to let us know on social media.
(Thank you to Lauren Hyams for suggesting we “do something for Easter” and Ben ‘gpiozero’ Nuttall for introducing me to ButtonBoard.)
Germany-based Andreas Rottach’s multi-purpose LED table is an impressive build within a gorgeous-looking body. Play games, view (heavily pixelated) images, and become hypnotised by flashy lights, once you’ve built your own using his newly released tutorial.
This is a short presentation of my LED-Matrix Table. The table is controlled by a raspberry pi computer that executes a control engine, written in c++. It supports input from keyboards or custom made game controllers. A full list of all features as well as the source code is available on GitHub (https://github.com/rottaca/LEDTableEngine).
Much excitement
Andreas uploaded a video of his LED Matrix Table to YouTube back in February, with the promise of publishing a complete write-up within the coming weeks. And so the members of Pi Towers sat, eagerly waiting and watching. Now the write-up has arrived, to our cheers of acclaim for this beautful, shiny, flashy, LED-based wonderment.
Andreas created the table’s impressive light matrix using a strip of 300 LEDs, chained together and connected to the Raspberry Pi via an LED controller.
The LEDs are set out in zigzags
For the code, he used several open-source tools, such as SDL for image and audio support, and CMake for building the project software.
Anyone planning to recreate Andreas’ table can compile its engine by downloading the project repository from GitHub. Again, find full instructions for this on his GitHub.
Features
The table boasts multiple cool features, including games and visualisation tools. Using the controllers, you can play simplified versions of Flappy Bird and Minesweeper, or go on a nostalgia trip with Tetris, Pong, and Snake.
There’s also a version of Conway’s Game of Life. Andreas explains: “The lifespan of each cell is color-coded. If the game field gets static, the animation is automatically reset to a new random cell population.”
The table can also display downsampled Bitmap images, or show clear static images such as a chess board, atop of which you can place physical game pieces.
Find all the 3D-printable aspects of the LED table on Thingiverse here and here, and the full GitHub tutorial and repository here. If you build your own, or have already dabbled in LED tables and displays, be sure to share your project with us, either in the comments below or via our social media accounts. What other functions would you integrate into this awesome build?
The Ruiz brothers at Adafruit have used Phillip Burgess’s PixieDust code to turn a 64×64 LED Matrix and a Raspberry Pi Zero into an awesome sand toy that refuses to defy the laws of gravity. Here’s how to make your own.
Simulated LED Sand Physics! These LEDs interact with motion and looks like they’re affect by gravity. An Adafruit LED matrix displays the LEDs as little grains of sand which are driven by sampling an accelerometer with Raspberry Pi Zero!
Obey gravity
As the latest addition to their online learning system, Adafruit have produced the BIG LED Sand Toy, or as I like to call it, Have you seen this awesome thing Adafuit have made?
Alongside the aforementioned ingredients, the project utilises the Adafruit LIS3DH Triple-Axis Accelerometer. This sensor is packed with features, and it allows the Raspberry Pi to control the virtual sand depending on how the toy is moved.
The Ruiz brothers inserted an SD card loaded with Raspbian Lite into the Raspberry Pi Zero, installed the LED Matrix driver, cloned the Adafruit_PixieDust library, and then just executed the code. They created some preset modes, but once you’re comfortable with the project code, you’ll be able to add your own take on the project.
Accelerometers and Raspberry Pi
This isn’t the first time a Raspberry Pi has met an accelerometer: the two Raspberry Pis aboard the International Space Station for the Astro Pi mission both have accelerometers thanks to their Sense HATs.
Comprised of a bundle of sensors, an LED matrix, and a five-point joystick, the Sense HAT is a great tool for exploring your surroundings with the Raspberry Pi, as well as for using your surroundings to control the Pi. You can find a whole variety of Sense HAT–based projects and tutorials on our website.
And if you’d like to try out the Sense HAT, including its onboard accelerometer, without purchasing one, head over to our online emulator, or use the emulator preinstalled on Raspbian.
With the launch of our first new free online course of 2018 — Scratch to Python: Moving from Block- to Text-based Programming — two weeks away, I thought this would be a great opportunity to introduce you to the ins and outs of the course content so you know what to expect.
Learn how to apply the thinking and programming skills you’ve learnt in Scratch to text-based programming languages like Python.
Take the plunge into text-based programming
The idea for this course arose from our conversations with educators who had set up a Code Club in their schools. Most people start a club by teaching Scratch, a block-based programming language, because it allows learners to drag and drop blocks of pre-written code into a window to create a program. The blocks automatically snap together, making it easy to build fun and educational projects that don’t require much troubleshooting. You can do almost anything a beginner could wish for with Scratch, even physical computing to control LEDs, buzzers, buttons, motors, and more!
However, on our face-to-face training programme Picademy, educators told us that they were finding it hard to engage children who had outgrown Scratch and needed a new challenge. It was easy for me to imagine: a young learner, who once felt confident about programming using Scratch, is now confused by the alien, seemingly awkward interface of Python. What used to take them minutes in Scratch now takes them hours to code, and they start to lose interest — not a good result, I’m sure you’ll agree. I wanted to help educators to navigate this period in their learners’ development, and so I’ve written a course that shows you how to take the programming and thinking skills you and your learners have developed in Scratch, and apply them to Python.
Who is the course for?
Educators from all backgrounds who are working with secondary school-aged learners. It will also be interesting to anyone who has spent time working with Scratch and wants to understand how programming concepts translate between different languages.
“It was great fun, and I thought that the ideas and resources would be great to use with Year 7 classes.” Sue Grey, Classroom Teacher
What is covered?
After showing you the similarities and differences of Scratch and Python, and how the skills learned using one can be applied to the other, we will look at turning more complex Scratch scripts into Python programs. Through creating a Mad Libs game and developing a username generator, you will see how programs can be simplified in a text-based language. We will give you our top tips for debugging Python code, and you’ll have the chance to share your ideas for introducing more complex programs to your students.
After that, we will look at different data types in Python and write a script to calculate how old you are in dog years. Finally, you’ll dive deeper into the possibilities of Python by installing and using external Python libraries to perform some amazing tasks.
By the end of the course, you’ll be able to:
Transfer programming and thinking skills from Scratch to Python
Use fundamental Python programming skills
Identify errors in your Python code based on error messages, and debug your scripts
Produce tools to support students’ transition from block-based to text-based programming
Understand the power of text-based programming and what you can create with it
Piano keys are so limiting! Why not swap them out for LEDs and the wealth of instruments in Pygame to build air keys, as demonstrated by Instructables maker 2fishy?
Raspberry Pi LED Light Schroeder Piano – Twinkle Little Star
Keys? Where we’re going you don’t need keys!
This project, created by either Yolanda or Ken Fisher (or both!), uses an array of LEDs and photoresistors to form a MIDI sequencer. Twelve LEDs replace piano keys, and another three change octaves and access the menu.
Each LED is paired with a photoresistor, which detects the emitted light to form a closed circuit. Interrupting the light beam — in this case with a finger — breaks the circuit, telling the Python program to perform an action.
We’re all hoping this is just the scaled-down prototype of a full-sized LED grand piano
Using Pygame, the 2fishy team can access 75 different instruments and 128 notes per instrument, making their wooden piano more than just a one-hit wonder.
Piano building
The duo made the piano’s body out of plywood, hardboard, and dowels, and equipped it with a Raspberry Pi 2, a speaker, and the aforementioned LEDs and photoresistors.
A Raspberry Pi 2 and speaker sit within the wooden body, with LEDs and photoresistors in place of the keys.
A complete how-to for the build, including some rather fancy and informative schematics, is available at Instructables, where 2fishy received a bronze medal for their project. Congratulations!
Learn more
If you’d like to learn more about using Pygame, check out The MagPi’s Make Games with Python Essentials Guide, available both in print and as a free PDF download.
And for more music-based projects using a variety of tech, be sure to browse our free resources.
Lastly, if you’d like to see more piano-themed Raspberry Pi projects, take a look at our Big Minecraft Piano, these brilliant piano stairs, this laser-guided piano teacher, and our video below about the splendid Street Fighter duelling pianos we witnessed at Maker Faire.
Two pianos wired up as Playstation 2 controllers allow users to battle…musically! We caught up with makers Eric Redon and Cyril Chapellier of foobarflies a…
We’re making with a purpose in issue 3 of HackSpace magazine. Not only are we discovering ways in which 3D printing is helping to save resources — and in some case lives — in the developing world, we’re also going all out with recycling. While others might be content with separating their glass and plastic waste, we’re going much, much further by making useful things out of discarded old bits of rubbish you can find at your local scrapyard.
Hackspaces
We’re going to Cheltenham Hackspace to learn how to make a leather belt, to Liverpool to discover the ways in which an open-source design and some bits and bobs from IKEA are protecting our food supply, and we also take a peek through the doors of Nottingham Hackspace.
Tutorials
The new issue also has the most tutorials you’ll have seen anywhere since…well, since HackSpace magazine issue 2! Guides to 3D-printing on fabric, Arduino programming, and ESP8266 hacking are all to be found in issue 3. Plus, we’ve come up with yet another way to pipe numbers from the internet into big, red, glowing boxes — it’s what LEDs were made for.
With the addition of racing drones, an angry reindeer, and an intelligent toaster, we think we’ve definitely put together an issue you’ll enjoy.
Get your copy
The physical copy of HackSpace magazine is available at all good UK newsagents today, and you can order it online from the Raspberry Pi Press store wherever you are based. Moreover, you can download the free PDF version from our website. And if you’ve read our first two issues and enjoyed what you’ve seen, be sure to subscribe!
Write for us
Are you working on a cool project? Do you want to share your skills with the world, inspire others, and maybe show off a little? HackSpace magazine wants your article! Send an outline of your piece to us, and we’ll get back to you about including it in a future issue.
Standard clocks with easily recognisable numbers are so last season. Who wants to save valuable seconds simply telling the time, when a series of LEDs and numerical notation can turn every time query into an adventure in mathematics?
In this video I’ll be showing how I built a binary clock using a Raspberry Pi, NeoPixels and a few lines of Python. I also take a stab at explaining how the binary number system works so that we can decipher what said clock is trying to tell us.
How to read binary
I’ll be honest: I have to think pretty hard to read binary. It stretches my brain quite vigorously. But I am a fan of flashy lights and pretty builds, so YouTube and Instagram rising star Mattias Jähnke, aka engineerish, had my full attention from the off.
“If you have a problem with your friends being able to tell the time way too easily while in your house, this is your answer.”
Mattias offers a beginners’ guide in to binary in his video and then explains how his clock displays values in binary, before moving on to the actual clock build process. So make some tea, pull up a chair, and jump right in.
Binary clock
To build the clock, Mattias used a Raspberry Pi and NeoPixel strips, fitted snugly within a simple 3D-printed case. With a few lines of Python, he coded his clock to display the current time using the binary system, with columns for seconds, minutes, and hours.
418 Likes, 14 Comments – Mattias (@engineerish) on Instagram: “The real kicker with a binary clock is that by the time you’ve deciphered what time it is – you’re…”
The Python code isn’t currently available on Mattias’s GitHub account, but if you’re keen to see how he did it, and you ask politely, and he’s not too busy, you never know.
Make your own
In the meantime, while we batter our eyelashes in the general direction of Stockholm and hope for a response, I challenge any one of you to code a binary display project for the Raspberry Pi. It doesn’t have to be a clock. And it doesn’t have to use NeoPixels. Maybe it could use an LED matrix such as the SenseHat, or a series of independently controlled LEDs on a breadboard. Maybe there’s something to be done with servo motors that flip discs with different-coloured sides to display a binary number.
Whatever you decide to build, the standard reward applies: ten imaginary house points (of absolutely no practical use, but immense emotional value) and a great sense of achievement to all who give it a go.
As Simon mentioned in his recent blog post about Raspbian Stretch, we have developed a new piece of software called PiServer. Use this tool to easily set up a network of client Raspberry Pis connected to a single x86-based server via Ethernet. With PiServer, you don’t need SD cards, you can control all clients via the server, and you can add and configure user accounts — it’s ideal for the classroom, your home, or an industrial setting.
Client? Server?
Before I go into more detail, let me quickly explain some terms.
Server — the server is the computer that provides the file system, boot files, and password authentication to the client(s)
Client — a client is a computer that retrieves boot files from the server over the network, and then uses a file system the server has shared. More than one client can connect to a server, but all clients use the same file system.
User – a user is a user name/password combination that allows someone to log into a client to access the file system on the server. Any user can log into any client with their credentials, and will always see the same server and share the same file system. Users do not have sudo capability on a client, meaning they cannot make significant changes to the file system and software.
I see no SD cards
Last year we described how the Raspberry Pi 3 Model B can be booted without an SD card over an Ethernet network from another computer (the server). This is called network booting or PXE (pronounced ‘pixie’) booting.
Why would you want to do this?
A client computer (the Raspberry Pi) doesn’t need any permanent storage (an SD card) to boot.
You can network a large number of clients to one server, and all clients are exactly the same. If you log into one of the clients, you will see the same file system as if you logged into any other client.
The server can be run on an x86 system, which means you get to take advantage of the performance, network, and disk speed on the server.
Sounds great, right? Of course, for the less technical, creating such a network is very difficult. For example, there’s setting up all the required DHCP and TFTP servers, and making sure they behave nicely with the rest of the network. If you get this wrong, you can break your entire network.
PiServer to the rescue
To make network booting easy, I thought it would be nice to develop an application which did everything for you. Let me introduce: PiServer!
PiServer has the following functionalities:
It automatically detects Raspberry Pis trying to network boot, so you don’t have to work out their Ethernet addresses.
It sets up a DHCP server — the thing inside the router that gives all network devices an IP address — either in proxy mode or in full IP mode. No matter the mode, the DHCP server will only reply to the Raspberry Pis you have specified, which is important for network safety.
It creates user names and passwords for the server. This is great for a classroom full of Pis: just set up all the users beforehand, and everyone gets to log in with their passwords and keep all their work in a central place. Moreover, users cannot change the software, so educators have control over which programs their learners can use.
It uses a slightly altered Raspbian build which allows separation of temporary spaces, doesn’t have the default ‘pi’ user, and has LDAP enabled for log-in.
What can I do with PiServer?
Serve a whole classroom of Pis
In a classroom, PiServer allows all files for lessons or projects to be stored on a central x86-based computer. Each user can have their own account, and any files they create are also stored on the server. Moreover, the networked Pis doesn’t need to be connected to the internet. The teacher has centralised control over all Pis, and all Pis are user-agnostic, meaning there’s no need to match a person with a computer or an SD card.
Build a home server
PiServer could be used in the home to serve file systems for all Raspberry Pis around the house — either a single common Raspbian file system for all Pis or a different operating system for each. Hopefully, our extensive OS suppliers will provide suitable build files in future.
Use it as a controller for networked Pis
In an industrial scenario, it is possible to use PiServer to develop a network of Raspberry Pis (maybe even using Power over Ethernet (PoE)) such that the control software for each Pi is stored remotely on a server. This enables easy remote control and provisioning of the Pis from a central repository.
How to use PiServer
The client machines
So that you can use a Pi as a client, you need to enable network booting on it. Power it up using an SD card with a Raspbian Lite image, and open a terminal window. Type in
echo program_usb_boot_mode=1| sudo tee -a /boot/config.txt
and press Return. This adds the line program_usb_boot_mode=1 to the end of the config.txt file in /boot. Now power the Pi down and remove the SD card. The next time you connect the Pi to a power source, you will be able to network boot it.
The server machine
As a server, you will need an x86 computer on which you can install x86 Debian Stretch. Refer to Simon’s blog post for additional information on this. It is possible to use a Raspberry Pi to serve to the client Pis, but the file system will be slower, especially at boot time.
Make sure your server has a good amount of disk space available for the file system — in general, we recommend at least 16Gb SD cards for Raspberry Pis. The whole client file system is stored locally on the server, so the disk space requirement is fairly significant.
Next, start PiServer by clicking on the start icon and then clicking Preferences > PiServer. This will open a graphical user interface — the wizard — that will walk you through setting up your network. Skip the introduction screen, and you should see a screen looking like this:
If you’ve enabled network booting on the client Pis and they are connected to a power source, their MAC addresses will automatically appear in the table shown above. When you have added all your Pis, click Next.
On the Add users screen, you can set up users on your server. These are pairs of user names and passwords that will be valid for logging into the client Raspberry Pis. Don’t worry, you can add more users at any point. Click Next again when you’re done.
The Add software screen allows you to select the operating system you want to run on the attached Pis. (You’ll have the option to assign an operating system to each client individually in the setting after the wizard has finished its job.) There are some automatically populated operating systems, such as Raspbian and Raspbian Lite. Hopefully, we’ll add more in due course. You can also provide your own operating system from a local file, or install it from a URL. For further information about how these operating system images are created, have a look at the scripts in /var/lib/piserver/scripts.
Once you’re done, click Next again. The wizard will then install the necessary components and the operating systems you’ve chosen. This will take a little time, so grab a coffee (or decaffeinated drink of your choice).
When the installation process is finished, PiServer is up and running — all you need to do is reboot the Pis to get them to run from the server.
Shooting troubles
If you have trouble getting clients connected to your network, there are a fewthings you can do to debug:
If some clients are connecting but others are not, check whether you’ve enabled the network booting mode on the Pis that give you issues. To do that, plug an Ethernet cable into the Pi (with the SD card removed) — the LEDs on the Pi and connector should turn on. If that doesn’t happen, you’ll need to follow the instructions above to boot the Pi and edit its /boot/config.txt file.
If you can’t connect to any clients, check whether your network is suitable: format an SD card, and copy bootcode.bin from /boot on a standard Raspbian image onto it. Plug the card into a client Pi, and check whether it appears as a new MAC address in the PiServer GUI. If it does, then the problem is a known issue, and you can head to our forums to ask for advice about it (the network booting code has a couple of problems which we’re already aware of). For a temporary fix, you can clone the SD card on which bootcode.bin is stored for all your clients.
If neither of these things fix your problem, our forums are the place to find help — there’s a host of people there who’ve got PiServer working. If you’re sure you have identified a problem that hasn’t been addressed on the forums, or if you have a request for a functionality, then please add it to the GitHub issues.
Twinkly lights are to Christmas what pumpkins are to Halloween. And when you add a Raspberry Pi to your light show, the result instantly goes from “Meh, yeah.” to “OMG, wow!”
Here are some cool light-based Christmas projects to inspire you this weekend.
In his Christmas lights project, Caleb Johnson uses an app as a control panel to switch between predefined displays. The full code is available on his GitHub, and it connects a Raspberry Pi A+ to a strip of programmable LEDs that change their pattern at the touch of a phone screen.
What’s great about this project, aside from the simplicity of its design, is the scope for extending it. Why not share the app with friends and family, allowing them to control your lights remotely? Or link the lights to social media so they are triggered by a specific hashtag, like in Alex Ellis’ #cheerlights project below.
Here we have a smart holiday light which will only run when it detects your presence in the room through a passive infrared PIR sensor. I’ve used hot glue for the fixings and an 8-LED NeoPixel strip connected to port 18.
Cheerlights, an online service created by Hans Scharler, allows makers to incorporate hashtag-controlled lighting into the projects. By tweeting the hashtag #cheerlights, followed by a colour, you can control a network of lights so that they are all displaying the same colour.
For his holiday light hack using Cheerlights, Alex incorporated the Pimoroni Blinkt! and a collection of cheap Christmas decorations to create cute light-up ornaments for the festive season.
To make your own, check out Alex’s blog post, and head to your local £1/$1 store for hackable decor. You could even link your Christmas tree and the trees of your family, syncing them all in one glorious, Santa-pleasing spectacular.
With just a few bucks of extra material, I walk you through converting your regular Christmas lights into a whole-house light show. The goal here is to go from scratch. Although this guide is intended for people who don’t know how to use linux at all and those who do alike, the focus is for people for whom linux and the raspberry pi are a complete mystery.
Looking to outdo your neighbours with your Christmas light show this year? YouTuber Makin’Things has created a beginners guide to setting up a Raspberry Pi–based musical light show for your facade, complete with information on soldering, wiring, and coding.
Once you’ve wrapped your house in metres and metres of lights and boosted your speakers so they can be heard for miles around, why not incorporate #cheerlights to make your outdoor decor interactive?
Still not enough? How about controlling your lights using a drum kit? Christian Kratky’s MIDI-Based Christmas Lights Animation system (or as I like to call it, House Rock) does exactly that.
Project documentation and source code: https://www.hackster.io/cyborg-titanium-14/light-pi-1c88b0 The song is taken from: https://www.youtube.com/watch?v=G6r1dAire0Y
Any more?
We know these projects are just the tip of the iceberg when it comes to the Raspberry Pi–powered Christmas projects out there, and as always, we’d love you to share yours with us. So post a link in the comments below, or tag us on social media when posting your build photos, videos, and/or blog links. ‘Tis the season for sharing after all.
Solving problems that are meaningful to us is at the core of our approach to teaching and learning about technology here at the Raspberry Pi Foundation. Over the last eight months, I’ve noticed that the types of digital making projects that motivate and engage me have changed (can’t think why). Always looking for ways to save money and automate my life and the lives of my loved ones, I’ve been thinking a lot about how digital making projects could be the new best friend of any new parent.
I never knew how much equipment one small child needs until very recently. I also had no idea of the range of technology that is on offer to support you as a new parent to ensure the perfect environment outside of the womb. Baby monitors are at the top of this list. There are lots of Raspberry Pi baby monitor projects with a range of sensing functionality already in existence, and we’ve blogged about some of them before. They’re a great example of how an understanding of technology can open up a range of solutions that won’t break the bank. I’m looking forward to using all the capabilities of the Raspberry Pi to keep an eye on baby.
Baby name generator
Another surprising discovery was just how difficult it is to name a human being. Surprising because I can give a name to an inanimate object in less than three seconds, and come up with nicknames for colleagues in less than a day. My own offspring, though, and I draw a blank. The only solution: write a Python program to randomly generate names based on some parameters!
import names
from time import sleep
from guizero import App, ButtonGroup, Text, PushButton, TextBox
def get_name():
boyname = names.get_first_name(gender='male')
girlname = names.get_first_name(gender='female')
othername = names.get_first_name()
if babygender.get() == "male":
name.set(str(boyname)+" "+str(babylastname.get()))
elif babygender.get() == "female":
name.set(str(girlname)+" "+str(babylastname.get()))
else:
name.set(str(othername)+" "+str(babylastname.get()))
app = App("Baby name generator")
surname_label = Text(app, "What is your surname?")
babylastname = TextBox(app, width=50)
babygender = ButtonGroup(app, options=[["boy", "male"], ["girl", "female"], ["all", "all"]], selected="male", horizontal=True)
intro = Text(app, "Your baby name could be")
name = Text(app, "")
button = PushButton(app, get_name, text="Generate me a name")
app.display()
Thanks to the names and GUIZero Python libraries, it is super simple to create, resolving any possible parent-to-be naming disputes in mere minutes.
Food, Poo, or Love?
I love data. Not just in Star Trek, but also more generally. Collecting and analysing data to understand my sleep patterns, my eating habits, how much exercise I do, and how much time I spend watching YouTube videos consumes much of my time. So of course I want to know lots about the little person we’ve made, long before he can use language to tell us himself.
I’m told that most newborns’ needs are quite simple: they want food, they want to be changed, or they just want some cuddles. I’m certain it’s more complicated than this, but it’s a good starting point for a data set, so stick with me here. I also wondered whether there might be a correlation between the amplitude of the cry and the type of need the baby has. A bit of an imprecise indicator, maybe, but fun to start to think about.
This build’s success is mostly thanks to Pimoroni’s Rainbow HAT, which, conveniently, has three capacitive touch buttons to record the newborn’s need, four fourteen-segment displays to display the words “FOOD”, “POO”, and “LOVE” when a button is pressed, and seven multicoloured LEDs to indicate the ferociousness of the baby’s cry in glorious technicolour. With the addition of a microphone, the ‘Food, Poo, Love Machine’ was born. Here it is in action:
Food Poo Love – The Raspberry Pi baby monitor project that allows you to track data on your new born baby.
Automatic Baby mobile
Another project that I’ve not had time to hack on, but that I think would be really awesome, is to automate a baby cot mobile. Imagine this one moving to the Star Trek theme music:
Looking for the perfect Christmas gift for a beloved maker in your life? Maybe you’d like to give a relative or friend a taste of the world of coding and Raspberry Pi? Whatever you’re looking for, the Raspberry Pi Christmas shopping list will point you in the right direction.
For those getting started
Thinking about introducing someone special to the wonders of Raspberry Pi during the holidays? Although you can set up your Pi with peripherals from around your home, such as a mobile phone charger, your PC’s keyboard, and the old mouse dwelling in an office drawer, a starter kit is a nice all-in-one package for the budding coder.
You can also buy the Raspberry Pi Press’s brand-new Raspberry Pi Beginners Book, which includes a Raspberry Pi Zero W, a case, a ready-made SD card, and adapter cables.
Once you’ve presented a lucky person with their first Raspberry Pi, it’s time for them to spread their maker wings and learn some new skills.
If you’re looking for something for a confident digital maker, you can’t go wrong with adding to their arsenal of electric and electronic bits and bobs that are no doubt cluttering drawers and boxes throughout their house.
Components such as servomotors, displays, and sensors are staples of the maker world. And when it comes to jumper wires, buttons, and LEDs, one can never have enough.
You could also consider getting your person a soldering iron, some helpings hands, or small tools such as a Dremel or screwdriver set.
And to make their life a little less messy, pop it all inside a Really Useful Box…because they’re really useful.
For kit makers
While some people like to dive into making head-first and to build whatever comes to mind, others enjoy working with kits.
The Naturebytes kit allows you to record the animal visitors of your garden with the help of a camera and a motion sensor. Footage of your local badgers, birds, deer, and more will be saved to an SD card, or tweeted or emailed to you if it’s in range of WiFi.
Coretec’s Tiny 4WD is a kit for assembling a Pi Zero–powered remote-controlled robot at home. Not only is the robot adorable, building it also a great introduction to motors and wireless control.
Finally, why not help your favourite maker create their own gaming arcade using the Arcade Building Kit from The Pi Hut?
For the reader
For those who like to curl up with a good read, or spend too much of their day on public transport, a book or magazine subscription is the perfect treat.
For makers, hackers, and those interested in new technologies, our brand-new HackSpace magazine and the ever popular community magazine The MagPi are ideal. Both are available via a physical or digital subscription, and new subscribers to The MagPi also receive a free Raspberry Pi Zero W plus case.
Looking for something small to keep your loved ones occupied on Christmas morning? Or do you have to buy a Secret Santa gift for the office tech? Here are some wonderful stocking fillers to fill your boots with this season.
The Pi Hut 3D Xmas Tree: available as both a pre-soldered and a DIY version, this gadget will work with any 40-pin Raspberry Pi and allows you to create your own mini light show.
Google AIY Voice kit: build your own home assistant using a Raspberry Pi, the MagPi Essentials guide, and this brand-new kit. “Google, play Mariah Carey again…”
Pimoroni’s Raspberry Pi Zero W Project Kits offer everything you need, including the Pi, to make your own time-lapse cameras, music players, and more.
The official Raspberry Pi Sense HAT, Camera Module, and cases for the Pi 3 and Pi Zero will complete the collection of any Raspberry Pi owner, while also opening up exciting project opportunities.
LEGO Idea’s bought out this amazing ‘Women of NASA’ set, and I thought it would be fun to build, play and learn from these inspiring women! First up, let’s discover a little more about Sally Ride and Mae Jemison, two AWESOME ASTRONAUTS!
Treat the kids, and big kids, in your life to the newest LEGO Ideas set, the Women of NASA — starring Nancy Grace Roman, Margaret Hamilton, Sally Ride, and Mae Jemison!
Explore the world of wearables with Pimoroni’s sewable, hackable, wearable, adorable Bearables kits.
Add lights and motors to paper creations with the Activating Origami Kit, available from The Pi Hut.
With so many amazing kits, HATs, and books available from members of the Raspberry Pi community, it’s hard to only pick a few. Have you found something splendid for the maker in your life? Maybe you’ve created your own kit that uses the Raspberry Pi? Share your favourites with us in the comments below or via our social media accounts.
Tired of pulling names out of a hat for office Secret Santa? Upgrade your festive tradition with a Raspberry Pi, thermal printer, and everybody’s favourite microcomputer mascot, Babbage Bear.
The name’s Santa. Secret Santa.
It’s that time of year again, when the cosiness gets turned up to 11 and everyone starts thinking about jolly fat men, reindeer, toys, and benevolent home invasion. At Raspberry Pi, we’re running a Secret Santa pool: everyone buys a gift for someone else in the office. Obviously, the person you buy for has to be picked in secret and at random, or the whole thing wouldn’t work. With that in mind, I created Secret Santa Babbage to do the somewhat mundane task of choosing gift recipients. This could’ve just been done with some names in a hat, but we’re Raspberry Pi! If we don’t make a Python-based Babbage robot wearing a jaunty hat and programmed to spread Christmas cheer, who will?
Ho ho ho!
Mecha-Babbage Xmas shenanigans
The script the robot runs is pretty basic: a list of names entered as comma-separated strings is shuffled at the press of a GPIO button, then a name is popped off the end and stored as a variable. The name is matched to a photo of the person stored on the Raspberry Pi, and a thermal printer pinched from Alex’s super awesome PastyCam (blog post forthcoming, maybe) prints out the picture and name of the person you will need to shower with gifts at the Christmas party. (Well, OK — with one gift. No more than five quid’s worth. Nothing untoward.) There’s also a redo function, just in case you pick yourself: press another button and the last picked name — still stored as a variable — is appended to the list again, which is shuffled once more, and a new name is popped off the end.
Prototyping!
As the build was a bit of a rush job undertaken at the request of our ‘Director of Vibe’ Emily, there are a few things I’d like to improve about this functionality that I didn’t get around to — more on that later. To add some extra holiday spirit to the project at the last minute, I used Pygame to play a WAV file of Santa’s jolly laugh while Babbage chooses a name for you. The file is included in the GitHub repo along with everything else, because ‘tis the season, etc., etc.
Editor’s note: Considering these desk adornments, Mark’s Secret Santa gift-giver has a lot to go on.
Writing the code for Xmas Mecha-Babbage was fairly straightforward, though it uses some tricky bits for managing the thermal printer. You’ll need to install the drivers to make it go, as well as the CUPS package for managing the print hosting. You can find instructions for these things here, thanks to the wonderful Adafruit crew. Also, for reasons I couldn’t fathom, this will all only work on a Pi 2 and not a Pi 3, as there are some compatibility issues with the thermal printer otherwise. (I also tested the script on a Pi Zero W…no dice.)
Building a Christmassy throne
The hardest (well, fiddliest) parts of making the whole build were constructing the throne and wiring the bear. Using MakerCase, Inkscape, a bit of ingenuity, and a laser cutter, I was able to rig up a Christmassy plywood throne which has a hole through the seat so I could run the wires down from Babbage and to the Pi inside. I finished the throne by rubbing a couple of fingers of beeswax into it; as well as making the wood shine just a little bit and protecting it against getting wet, this had the added bonus of making it smell awesome.
Next year’s iteration will be mulled wine–scented.
I next soldered two LEDs to some lengths of wire, and then ran the wires through holes at the top of the throne and down the back along a small channel I had carved with a narrow chisel to connect them to the Pi’s GPIO pins. The green LED will remain on as long as Babbage is running his program, and the red one will light up while he is processing your request. Once the red LED goes off again, the next person can have a go. I also laser-cut a final piece of wood to overlay the back of Babbage’s Xmas throne and cover the wiring a bit.
Creating a Xmas cyborg bear
Taking two 6 mm tactile buttons, I clipped the spiky metal legs off one side of each (the buttons were going into a stuffed christmas toy, after all) and soldered a length of wire to each of the remaining legs. Next, I made a small incision into Babbage with my trusty Swiss army knife (in a place that actually made me cringe a little) and fed the buttons up into his paws. At some point in this process I was standing in the office wrestling with the bear and muttering to myself, which elicited some very strange looks from my colleagues.
Poor Babbage…
One thing to note here is to make sure the wires remain attached at the solder points while you push them up into Babbage’s paws. The first time I tried it, I snapped one of my connections and had to start again. It helped to remove some stuffing like a tunnel and then replace it afterward. Moreover, you can use your fingertip to support the joints as you poke the wire in. Finally, a couple of squirts of hot glue to keep Babbage’s furry cheeks firmly on the seat, and done!
Next year: Game of Thrones–inspired candy cane throne
The Secret Santa Babbage masterpiece
The whole build process was the perfect holiday mix of cheerful and macabre, and while getting the thermal printer to work was a little time-consuming, the finished product definitely raised some smiles around the office and added a bit of interesting digital flavour to a staid office tradition. And it also helped people who are new to the office or from other branches of the Foundation to know for whom they will be buying a gift.
Ready to dispense Christmas cheer!
There are a few ways in which I’ll polish this project before next year, such as having the script write the names to external text files to create a record that will persist in case of a reboot, and maybe having Secret Santa Babbage play you a random Christmas carol when you squeeze his paw instead of just laughing merrily every time. (I also thought about adding electric shocks for those people who are on the naughty list, but HR said no. Bah, humbug!)
Make your own
The code and laser cut plans for the whole build are available here. If you plan to make your own, let us know which stuffed toy you will be turning into a Secret Santa cyborg! And if you’ve been working on any other Christmas-themed Raspberry Pi projects, we’d like to see those too, so tag us on social media to share the festive maker cheer.
When we discuss how to build applications with customers, we often align to the Well Architected Framework pillars of security, reliability, performance efficiency, cost optimization, and operational excellence. Designing for failure is an essential component to developing well architected applications that are resilient to spurious errors that may occur.
There are many ways you can use AWS services to achieve high availability and resiliency of your applications. For example, you can couple Elastic Load Balancing with Auto Scaling and Amazon EC2 instances to build highly available applications. Or use Amazon API Gateway and AWS Lambda to rapidly scale out a microservices-based architecture. Many AWS services have built in solutions to help with the appropriate error handling, such as Dead Letter Queues (DLQ) for Amazon SQS or retries in AWS Batch.
AWS Step Functions is an AWS service that makes it easy for you to coordinate the components of distributed applications and microservices. Step Functions allows you to easily design for failure, by incorporating features such as error retries and custom error handling from AWS Lambda exceptions. These features allow you to programmatically handle many common error modes and build robust, reliable applications.
In some rare cases, however, your application may fail in an unexpected manner. In these situations, you might not want to duplicate in a repeat execution those portions of your state machine that have already run. This is especially true when orchestrating long-running jobs or executing a complex state machine as part of a microservice. Here, you need to know the last successful state in your state machine from which to resume, so that you don’t duplicate previous work. In this post, we present a solution to enable you to resume from any given state in your state machine in the case of an unexpected failure.
Resuming from a given state
To resume a failed state machine execution from the state at which it failed, you first run a script that dynamically creates a new state machine. When the new state machine is executed, it resumes the failed execution from the point of failure. The script contains the following two primary steps:
Parse the execution history of the failed execution to find the name of the state at which it failed, as well as the JSON input to that state.
Create a new state machine, which adds an additional state to failed state machine, called "GoToState". "GoToState" is a choice state at the beginning of the state machine that branches execution directly to the failed state, allowing you to skip states that had succeeded in the previous execution.
The full script along with a CloudFormation template that creates a demo of this is available in the aws-sfn-resume-from-any-state GitHub repo.
Diving into the script
In this section, we walk you through the script and highlight the core components of its functionality. The script contains a main function, which adds a command line parameter for the failedExecutionArn so that you can easily call the script from the command line:
First, the script extracts the name of the failed state along with the input to that state. It does so by using the failed state machine execution history, which is identified by the Amazon Resource Name (ARN) of the execution. The failed state is marked in the execution history, along with the input to that state (which is also the output of the preceding successful state). The script is able to parse these values from the log.
The script loops through the execution history of the failed state machine, and traces it backwards until it finds the failed state. If the state machine failed in a parallel state, then it must restart from the beginning of the parallel state. The script is able to capture the name of the parallel state that failed, rather than any substate within the parallel state that may have caused the failure. The following code is the Python function that does this.
def parseFailureHistory(failedExecutionArn):
'''
Parses the execution history of a failed state machine to get the name of failed state and the input to the failed state:
Input failedExecutionArn = A string containing the execution ARN of a failed state machine y
Output = A list with two elements: [name of failed state, input to failed state]
'''
failedAtParallelState = False
try:
#Get the execution history
response = client.get\_execution\_history(
executionArn=failedExecutionArn,
reverseOrder=True
)
failedEvents = response['events']
except Exception as ex:
raise ex
#Confirm that the execution actually failed, raise exception if it didn't fail.
try:
failedEvents[0]['executionFailedEventDetails']
except:
raise('Execution did not fail')
'''
If you have a 'States.Runtime' error (for example, if a task state in your state machine attempts to execute a Lambda function in a different region than the state machine), get the ID of the failed state, and use it to determine the failed state name and input.
'''
if failedEvents[0]['executionFailedEventDetails']['error'] == 'States.Runtime':
failedId = int(filter(str.isdigit, str(failedEvents[0]['executionFailedEventDetails']['cause'].split()[13])))
failedState = failedEvents[-1 \* failedId]['stateEnteredEventDetails']['name']
failedInput = failedEvents[-1 \* failedId]['stateEnteredEventDetails']['input']
return (failedState, failedInput)
'''
You need to loop through the execution history, tracing back the executed steps.
The first state you encounter is the failed state. If you failed on a parallel state, you need the name of the parallel state rather than the name of a state within a parallel state that it failed on. This is because you can only attach goToState to the parallel state, but not a substate within the parallel state.
This loop starts with the ID of the latest event and uses the previous event IDs to trace back the execution to the beginning (id 0). However, it returns as soon it finds the name of the failed state.
'''
currentEventId = failedEvents[0]['id']
while currentEventId != 0:
#multiply event ID by -1 for indexing because you're looking at the reversed history
currentEvent = failedEvents[-1 \* currentEventId]
'''
You can determine if the failed state was a parallel state because it and an event with 'type'='ParallelStateFailed' appears in the execution history before the name of the failed state
'''
if currentEvent['type'] == 'ParallelStateFailed':
failedAtParallelState = True
'''
If the failed state is not a parallel state, then the name of failed state to return is the name of the state in the first 'TaskStateEntered' event type you run into when tracing back the execution history
'''
if currentEvent['type'] == 'TaskStateEntered' and failedAtParallelState == False:
failedState = currentEvent['stateEnteredEventDetails']['name']
failedInput = currentEvent['stateEnteredEventDetails']['input']
return (failedState, failedInput)
'''
If the failed state was a parallel state, then you need to trace execution back to the first event with 'type'='ParallelStateEntered', and return the name of the state
'''
if currentEvent['type'] == 'ParallelStateEntered' and failedAtParallelState:
failedState = failedState = currentEvent['stateEnteredEventDetails']['name']
failedInput = currentEvent['stateEnteredEventDetails']['input']
return (failedState, failedInput)
#Update the ID for the next execution of the loop
currentEventId = currentEvent['previousEventId']
Create the new state machine
The script uses the name of the failed state to create the new state machine, with "GoToState" branching execution directly to the failed state.
To do this, the script requires the Amazon States Language (ASL) definition of the failed state machine. It modifies the definition to append "GoToState", and create a new state machine from it.
The script gets the ARN of the failed state machine from the execution ARN of the failed state machine. This ARN allows it to get the ASL definition of the failed state machine by calling the DesribeStateMachine API action. It creates a new state machine with "GoToState".
When the script creates the new state machine, it also adds an additional input variable called "resuming". When you execute this new state machine, you specify this resuming variable as true in the input JSON. This tells "GoToState" to branch execution to the state that had previously failed. Here’s the function that does this:
def attachGoToState(failedStateName, stateMachineArn):
'''
Given a state machine ARN and the name of a state in that state machine, create a new state machine that starts at a new choice state called 'GoToState'. "GoToState" branches to the named state, and sends the input of the state machine to that state, when a variable called "resuming" is set to True.
Input failedStateName = A string with the name of the failed state
stateMachineArn = A string with the ARN of the state machine
Output response from the create_state_machine call, which is the API call that creates a new state machine
'''
try:
response = client.describe\_state\_machine(
stateMachineArn=stateMachineArn
)
except:
raise('Could not get ASL definition of state machine')
roleArn = response['roleArn']
stateMachine = json.loads(response['definition'])
#Create a name for the new state machine
newName = response['name'] + '-with-GoToState'
#Get the StartAt state for the original state machine, because you point the 'GoToState' to this state
originalStartAt = stateMachine['StartAt']
'''
Create the GoToState with the variable $.resuming.
If new state machine is executed with $.resuming = True, then the state machine skips to the failed state.
Otherwise, it executes the state machine from the original start state.
'''
goToState = {'Type':'Choice', 'Choices':[{'Variable':'$.resuming', 'BooleanEquals':False, 'Next':originalStartAt}], 'Default':failedStateName}
#Add GoToState to the set of states in the new state machine
stateMachine['States']['GoToState'] = goToState
#Add StartAt
stateMachine['StartAt'] = 'GoToState'
#Create new state machine
try:
response = client.create_state_machine(
name=newName,
definition=json.dumps(stateMachine),
roleArn=roleArn
)
except:
raise('Failed to create new state machine with GoToState')
return response
Testing the script
Now that you understand how the script works, you can test it out.
The following screenshot shows an example state machine that has failed, called "TestMachine". This state machine successfully completed "FirstState" and "ChoiceState", but when it branched to "FirstMatchState", it failed.
Use the script to create a new state machine that allows you to rerun this state machine, but skip the "FirstState" and the "ChoiceState" steps that already succeeded. You can do this by calling the script as follows:
This creates a new state machine called "TestMachine-with-GoToState", and returns its ARN, along with the input that had been sent to "FirstMatchState". You can then inspect the input to determine what caused the error. In this case, you notice that the input to "FirstMachState" was the following:
{
"foo": 1,
"Message": true
}
However, this state machine expects the "Message" field of the JSON to be a string rather than a Boolean. Execute the new "TestMachine-with-GoToState" state machine, change the input to be a string, and add the "resuming" variable that "GoToState" requires:
When you execute the new state machine, it skips "FirstState" and "ChoiceState", and goes directly to "FirstMatchState", which was the state that failed:
Look at what happens when you have a state machine with multiple parallel steps. This example is included in the GitHub repository associated with this post. The repo contains a CloudFormation template that sets up this state machine and provides instructions to replicate this solution.
The following state machine, "ParallelStateMachine", takes an input through two subsequent parallel states before doing some final processing and exiting, along with the JSON with the ASL definition of the state machine.
{
"Comment": "An example of the Amazon States Language using a parallel state to execute two branches at the same time.",
"StartAt": "Parallel",
"States": {
"Parallel": {
"Type": "Parallel",
"ResultPath":"$.output",
"Next": "Parallel 2",
"Branches": [
{
"StartAt": "Parallel Step 1, Process 1",
"States": {
"Parallel Step 1, Process 1": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-west-2:XXXXXXXXXXXX:function:LambdaA",
"End": true
}
}
},
{
"StartAt": "Parallel Step 1, Process 2",
"States": {
"Parallel Step 1, Process 2": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-west-2:XXXXXXXXXXXX:function:LambdaA",
"End": true
}
}
}
]
},
"Parallel 2": {
"Type": "Parallel",
"Next": "Final Processing",
"Branches": [
{
"StartAt": "Parallel Step 2, Process 1",
"States": {
"Parallel Step 2, Process 1": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-west-2:XXXXXXXXXXXXX:function:LambdaB",
"End": true
}
}
},
{
"StartAt": "Parallel Step 2, Process 2",
"States": {
"Parallel Step 2, Process 2": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-west-2:XXXXXXXXXXXX:function:LambdaB",
"End": true
}
}
}
]
},
"Final Processing": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-west-2:XXXXXXXXXXXX:function:LambdaC",
"End": true
}
}
}
First, use an input that initially fails:
{
"Message": "Hello!"
}
This fails because the state machine expects you to have a variable in the input JSON called "foo" in the second parallel state to run "Parallel Step 2, Process 1" and "Parallel Step 2, Process 2". Instead, the original input gets processed by the first parallel state and produces the following output to pass to the second parallel state:
Run the script on the failed state machine to create a new state machine that allows it to resume directly at the second parallel state instead of having to redo the first parallel state. This creates a new state machine called "ParallelStateMachine-with-GoToState". The following JSON was created by the script to define the new state machine in ASL. It contains the "GoToState" value that was attached by the script.
{
"Comment":"An example of the Amazon States Language using a parallel state to execute two branches at the same time.",
"States":{
"Final Processing":{
"Resource":"arn:aws:lambda:us-west-2:XXXXXXXXXXXX:function:LambdaC",
"End":true,
"Type":"Task"
},
"GoToState":{
"Default":"Parallel 2",
"Type":"Choice",
"Choices":[
{
"Variable":"$.resuming",
"BooleanEquals":false,
"Next":"Parallel"
}
]
},
"Parallel":{
"Branches":[
{
"States":{
"Parallel Step 1, Process 1":{
"Resource":"arn:aws:lambda:us-west-2:XXXXXXXXXXXX:function:LambdaA",
"End":true,
"Type":"Task"
}
},
"StartAt":"Parallel Step 1, Process 1"
},
{
"States":{
"Parallel Step 1, Process 2":{
"Resource":"arn:aws:lambda:us-west-2:XXXXXXXXXXXX:LambdaA",
"End":true,
"Type":"Task"
}
},
"StartAt":"Parallel Step 1, Process 2"
}
],
"ResultPath":"$.output",
"Type":"Parallel",
"Next":"Parallel 2"
},
"Parallel 2":{
"Branches":[
{
"States":{
"Parallel Step 2, Process 1":{
"Resource":"arn:aws:lambda:us-west-2:XXXXXXXXXXXX:function:LambdaB",
"End":true,
"Type":"Task"
}
},
"StartAt":"Parallel Step 2, Process 1"
},
{
"States":{
"Parallel Step 2, Process 2":{
"Resource":"arn:aws:lambda:us-west-2:XXXXXXXXXXXX:function:LambdaB",
"End":true,
"Type":"Task"
}
},
"StartAt":"Parallel Step 2, Process 2"
}
],
"Type":"Parallel",
"Next":"Final Processing"
}
},
"StartAt":"GoToState"
}
You can then execute this state machine with the correct input by adding the "foo" and "resuming" variables:
This yields the following result. Notice that this time, the state machine executed successfully to completion, and skipped the steps that had previously failed.
Conclusion
When you’re building out complex workflows, it’s important to be prepared for failure. You can do this by taking advantage of features such as automatic error retries in Step Functions and custom error handling of Lambda exceptions.
Nevertheless, state machines still have the possibility of failing. With the methodology and script presented in this post, you can resume a failed state machine from its point of failure. This allows you to skip the execution of steps in the workflow that had already succeeded, and recover the process from the point of failure.
If you have questions or suggestions, please comment below.
The collective thoughts of the interwebz
By continuing to use the site, you agree to the use of cookies. more information
The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.