Tag Archives: developing world

HackSpace magazine 3: Scrap Heap Hacking

Post Syndicated from Andrew Gregory original https://www.raspberrypi.org/blog/hackspace-magazine-3-scrap-heap-hacking/

We’re making with a purpose in issue 3 of HackSpace magazine. Not only are we discovering ways in which 3D printing is helping to save resources — and in some case lives — in the developing world, we’re also going all out with recycling. While others might be content with separating their glass and plastic waste, we’re going much, much further by making useful things out of discarded old bits of rubbish you can find at your local scrapyard.

Hackspaces

We’re going to Cheltenham Hackspace to learn how to make a leather belt, to Liverpool to discover the ways in which an open-source design and some bits and bobs from IKEA are protecting our food supply, and we also take a peek through the doors of Nottingham Hackspace.

Tutorials

The new issue also has the most tutorials you’ll have seen anywhere since…well, since HackSpace magazine issue 2! Guides to 3D-printing on fabric, Arduino programming, and ESP8266 hacking are all to be found in issue 3. Plus, we’ve come up with yet another way to pipe numbers from the internet into big, red, glowing boxes — it’s what LEDs were made for.



With the addition of racing drones, an angry reindeer, and an intelligent toaster, we think we’ve definitely put together an issue you’ll enjoy.

Get your copy

The physical copy of HackSpace magazine is available at all good UK newsagents today, and you can order it online from the Raspberry Pi Press store wherever you are based. Moreover, you can download the free PDF version from our website. And if you’ve read our first two issues and enjoyed what you’ve seen, be sure to subscribe!

Write for us

Are you working on a cool project? Do you want to share your skills with the world, inspire others, and maybe show off a little? HackSpace magazine wants your article! Send an outline of your piece to us, and we’ll get back to you about including it in a future issue.

The post HackSpace magazine 3: Scrap Heap Hacking appeared first on Raspberry Pi.

Querying OpenStreetMap with Amazon Athena

Post Syndicated from Seth Fitzsimmons original https://aws.amazon.com/blogs/big-data/querying-openstreetmap-with-amazon-athena/

This is a guest post by Seth Fitzsimmons, member of the 2017 OpenStreetMap US board of directors. Seth works with clients including the Humanitarian OpenStreetMap Team, Mapzen, the American Red Cross, and World Bank to craft innovative geospatial solutions.

OpenStreetMap (OSM) is a free, editable map of the world, created and maintained by volunteers and available for use under an open license. Companies and non-profits like Mapbox, Foursquare, Mapzen, the World Bank, the American Red Cross and others use OSM to provide maps, directions, and geographic context to users around the world.

In the 12 years of OSM’s existence, editors have created and modified several billion features (physical things on the ground like roads or buildings). The main PostgreSQL database that powers the OSM editing interface is now over 2TB and includes historical data going back to 2007. As new users join the open mapping community, more and more valuable data is being added to OpenStreetMap, requiring increasingly powerful tools, interfaces, and approaches to explore its vastness.

This post explains how anyone can use Amazon Athena to quickly query publicly available OSM data stored in Amazon S3 (updated weekly) as an AWS Public Dataset. Imagine that you work for an NGO interested in improving knowledge of and access to health centers in Africa. You might want to know what’s already been mapped, to facilitate the production of maps of surrounding villages, and to determine where infrastructure investments are likely to be most effective.

Note: If you run all the queries in this post, you will be charged approximately $1 based on the number of bytes scanned. All queries used in this post can be found in this GitHub gist.

What is OpenStreetMap?

As an open content project, regular OSM data archives are made available to the public via planet.openstreetmap.org in a few different formats (XML, PBF). This includes both snapshots of the current state of data in OSM as well as historical archives.

Working with “the planet” (as the data archives are referred to) can be unwieldy. Because it contains data spanning the entire world, the size of a single archive is on the order of 50 GB. The format is bespoke and extremely specific to OSM. The data is incredibly rich, interesting, and useful, but the size, format, and tooling can often make it very difficult to even start the process of asking complex questions.

Heavy users of OSM data typically download the raw data and import it into their own systems, tailored for their individual use cases, such as map rendering, driving directions, or general analysis. Now that OSM data is available in the Apache ORC format on Amazon S3, it’s possible to query the data using Athena without even downloading it.

How does Athena help?

You can use Athena along with data made publicly available via OSM on AWS. You don’t have to learn how to install, configure, and populate your own server instances and go through multiple steps to download and transform the data into a queryable form. Thanks to AWS and partners, a regularly updated copy of the planet file (available within hours of OSM’s weekly publishing schedule) is hosted on S3 and made available in a format that lends itself to efficient querying using Athena.

Asking questions with Athena involves registering the OSM planet file as a table and making SQL queries. That’s it. Nothing to download, nothing to configure, nothing to ingest. Athena distributes your queries and returns answers within seconds, even while querying over 9 years and billions of OSM elements.

You’re in control. S3 provides high availability for the data and Athena charges you per TB of data scanned. Plus, we’ve gone through the trouble of keeping scanning charges as small as possible by transcoding OSM’s bespoke format as ORC. All the hard work of transforming the data into something highly queryable and making it publicly available is done; you just need to bring some questions.

Registering Tables

The OSM Public Datasets consist of three tables:

  • planet
    Contains the current versions of all elements present in OSM.
  • planet_history
    Contains a historical record of all versions of all elements (even those that have been deleted).
  • changesets
    Contains information about changesets in which elements were modified (and which have a foreign key relationship to both the planet and planet_history tables).

To register the OSM Public Datasets within your AWS account so you can query them, open the Athena console (make sure you are using the us-east-1 region) to paste and execute the following table definitions:

planet

CREATE EXTERNAL TABLE planet (
  id BIGINT,
  type STRING,
  tags MAP<STRING,STRING>,
  lat DECIMAL(9,7),
  lon DECIMAL(10,7),
  nds ARRAY<STRUCT<ref: BIGINT>>,
  members ARRAY<STRUCT<type: STRING, ref: BIGINT, role: STRING>>,
  changeset BIGINT,
  timestamp TIMESTAMP,
  uid BIGINT,
  user STRING,
  version BIGINT
)
STORED AS ORCFILE
LOCATION 's3://osm-pds/planet/';

planet_history

CREATE EXTERNAL TABLE planet_history (
    id BIGINT,
    type STRING,
    tags MAP<STRING,STRING>,
    lat DECIMAL(9,7),
    lon DECIMAL(10,7),
    nds ARRAY<STRUCT<ref: BIGINT>>,
    members ARRAY<STRUCT<type: STRING, ref: BIGINT, role: STRING>>,
    changeset BIGINT,
    timestamp TIMESTAMP,
    uid BIGINT,
    user STRING,
    version BIGINT,
    visible BOOLEAN
)
STORED AS ORCFILE
LOCATION 's3://osm-pds/planet-history/';

changesets

CREATE EXTERNAL TABLE changesets (
    id BIGINT,
    tags MAP<STRING,STRING>,
    created_at TIMESTAMP,
    open BOOLEAN,
    closed_at TIMESTAMP,
    comments_count BIGINT,
    min_lat DECIMAL(9,7),
    max_lat DECIMAL(9,7),
    min_lon DECIMAL(10,7),
    max_lon DECIMAL(10,7),
    num_changes BIGINT,
    uid BIGINT,
    user STRING
)
STORED AS ORCFILE
LOCATION 's3://osm-pds/changesets/';

 

Under the Hood: Extract, Transform, Load

So, what happens behind the scenes to make this easier for you? In a nutshell, the data is transcoded from the OSM PBF format into Apache ORC.

There’s an AWS Lambda function (running every 15 minutes, triggered by CloudWatch Events) that watches planet.openstreetmap.org for the presence of weekly updates (using rsync). If that function detects that a new version has become available, it submits a set of AWS Batch jobs to mirror, transcode, and place it as the “latest” version. Code for this is available at osm-pds-pipelines GitHub repo.

To facilitate the transcoding into a format appropriate for Athena, we have produced an open source tool, OSM2ORC. The tool also includes an Osmosis plugin that can be used with complex filtering pipelines and outputs an ORC file that can be uploaded to S3 for use with Athena, or used locally with other tools from the Hadoop ecosystem.

What types of questions can OpenStreetMap answer?

There are many uses for OpenStreetMap data; here are three major ones and how they may be addressed using Athena.

Case Study 1: Finding Local Health Centers in West Africa

When the American Red Cross mapped more than 7,000 communities in West Africa in areas affected by the Ebola epidemic as part of the Missing Maps effort, they found themselves in a position where collecting a wide variety of data was both important and incredibly beneficial for others. Accurate maps play a critical role in understanding human communities, especially for populations at risk. The lack of detailed maps for West Africa posed a problem during the 2014 Ebola crisis, so collecting and producing data around the world has the potential to improve disaster responses in the future.

As part of the data collection, volunteers collected locations and information about local health centers, something that will facilitate treatment in future crises (and, more importantly, on a day-to-day basis). Combined with information about access to markets and clean drinking water and historical experiences with natural disasters, this data was used to create a vulnerability index to select communities for detailed mapping.

For this example, you find all health centers in West Africa (many of which were mapped as part of Missing Maps efforts). This is something that healthsites.io does for the public (worldwide and editable, based on OSM data), but you’re working with the raw data.

Here’s a query to fetch information about all health centers, tagged as nodes (points), in Guinea, Sierra Leone, and Liberia:

SELECT * from planet
WHERE type = 'node'
  AND tags['amenity'] IN ('hospital', 'clinic', 'doctors')
  AND lon BETWEEN -15.0863 AND -7.3651
  AND lat BETWEEN 4.3531 AND 12.6762;

Buildings, as “ways” (polygons, in this case) assembled from constituent nodes (points), can also be tagged as medical facilities. In order to find those, you need to reassemble geometries. Here you’re taking the average of all nodes that make up a building (which will be the approximate center point, which is close enough for this purpose). Here is a query that finds both buildings and points that are tagged as medical facilities:

-- select out nodes and relevant columns
WITH nodes AS (
  SELECT
    type,
    id,
    tags,
    lat,
    lon
  FROM planet
  WHERE type = 'node'
),
-- select out ways and relevant columns
ways AS (
  SELECT
    type,
    id,
    tags,
    nds
  FROM planet
  WHERE type = 'way'
    AND tags['amenity'] IN ('hospital', 'clinic', 'doctors')
),
-- filter nodes to only contain those present within a bounding box
nodes_in_bbox AS (
  SELECT *
  FROM nodes
  WHERE lon BETWEEN -15.0863 AND -7.3651
    AND lat BETWEEN 4.3531 AND 12.6762
)
-- find ways intersecting the bounding box
SELECT
  ways.type,
  ways.id,
  ways.tags,
  AVG(nodes.lat) lat,
  AVG(nodes.lon) lon
FROM ways
CROSS JOIN UNNEST(nds) AS t (nd)
JOIN nodes_in_bbox nodes ON nodes.id = nd.ref
GROUP BY (ways.type, ways.id, ways.tags)
UNION ALL
SELECT
  type,
  id,
  tags,
  lat,
  lon
FROM nodes_in_bbox
WHERE tags['amenity'] IN ('hospital', 'clinic', 'doctors');

You could go a step further and query for additional tags included with these (for example, opening_hours) and use that as a metric for measuring “completeness” of the dataset and to focus on additional data to collect (and locations to fill out).

Case Study 2: Generating statistics about mapathons

OSM has a history of holding mapping parties. Mapping parties are events where interested people get together and wander around outside, gathering and improving information about sites (and sights) that they pass. Another form of mapping party is the mapathon, which brings together armchair and desk mappers to focus on improving data in another part of the world.

Mapathons are a popular way of enlisting volunteers for Missing Maps, a collaboration between many NGOs, education institutions, and civil society groups that aims to map the most vulnerable places in the developing world to support international and local NGOs and individuals. One common way that volunteers participate is to trace buildings and roads from aerial imagery, providing baseline data that can later be verified by Missing Maps staff and volunteers working in the areas being mapped.

(Image and data from the American Red Cross)

Data collected during these events lends itself to a couple different types of questions. People like competition, so Missing Maps has developed a set of leaderboards that allow people to see where they stand relative to other mappers and how different groups compare. To facilitate this, hashtags (such as #missingmaps) are included in OSM changeset comments. To do similar ad hoc analysis, you need to query the list of changesets, filter by the presence of certain hashtags in the comments, and group things by username.

Now, find changes made during Missing Maps mapathons at George Mason University (using the #gmu hashtag):

SELECT *
FROM changesets
WHERE regexp_like(tags['comment'], '(?i)#gmu');

This includes all tags associated with a changeset, which typically include a mapper-provided comment about the changes made (often with additional hashtags corresponding to OSM Tasking Manager projects) as well as information about the editor used, imagery referenced, etc.

If you’re interested in the number of individual users who have mapped as part of the Missing Maps project, you can write a query such as:

SELECT COUNT(DISTINCT uid)
FROM changesets
WHERE regexp_like(tags['comment'], '(?i)#missingmaps');

25,610 people (as of this writing)!

Back at GMU, you’d like to know who the most prolific mappers are:

SELECT user, count(*) AS edits
FROM changesets
WHERE regexp_like(tags['comment'], '(?i)#gmu')
GROUP BY user
ORDER BY count(*) DESC;

Nice job, BrokenString!

It’s also interesting to see what types of features were added or changed. You can do that by using a JOIN between the changesets and planet tables:

SELECT planet.*, changesets.tags
FROM planet
JOIN changesets ON planet.changeset = changesets.id
WHERE regexp_like(changesets.tags['comment'], '(?i)#gmu');


Using this as a starting point, you could break down the types of features, highlight popular parts of the world, or do something entirely different.

Case Study 3: Building Condition

With building outlines having been produced by mappers around (and across) the world, local Missing Maps volunteers (often from local Red Cross / Red Crescent societies) go around with Android phones running OpenDataKit  and OpenMapKit to verify that the buildings in question actually exist and to add additional information about them, such as the number of stories, use (residential, commercial, etc.), material, and condition.

This data can be used in many ways: it can provide local geographic context (by being included in map source data) as well as facilitate investment by development agencies such as the World Bank.

Here are a collection of buildings mapped in Dhaka, Bangladesh:

(Map and data © OpenStreetMap contributors)

For NGO staff to determine resource allocation, it can be helpful to enumerate and show buildings in varying conditions. Building conditions within an area can be a means of understanding where to focus future investments.

Querying for buildings is a bit more complicated than working with points or changesets. Of the three core OSM element types—node, way, and relation, only nodes (points) have geographic information associated with them. Ways (lines or polygons) are composed of nodes and inherit vertices from them. This means that ways must be reconstituted in order to effectively query by bounding box.

This results in a fairly complex query. You’ll notice that this is similar to the query used to find buildings tagged as medical facilities above. Here you’re counting buildings in Dhaka according to building condition:

-- select out nodes and relevant columns
WITH nodes AS (
  SELECT
    id,
    tags,
    lat,
    lon
  FROM planet
  WHERE type = 'node'
),
-- select out ways and relevant columns
ways AS (
  SELECT
    id,
    tags,
    nds
  FROM planet
  WHERE type = 'way'
),
-- filter nodes to only contain those present within a bounding box
nodes_in_bbox AS (
  SELECT *
  FROM nodes
  WHERE lon BETWEEN 90.3907 AND 90.4235
    AND lat BETWEEN 23.6948 AND 23.7248
),
-- fetch and expand referenced ways
referenced_ways AS (
  SELECT
    ways.*,
    t.*
  FROM ways
  CROSS JOIN UNNEST(nds) WITH ORDINALITY AS t (nd, idx)
  JOIN nodes_in_bbox nodes ON nodes.id = nd.ref
),
-- fetch *all* referenced nodes (even those outside the queried bounding box)
exploded_ways AS (
  SELECT
    ways.id,
    ways.tags,
    idx,
    nd.ref,
    nodes.id node_id,
    ARRAY[nodes.lat, nodes.lon] coordinates
  FROM referenced_ways ways
  JOIN nodes ON nodes.id = nd.ref
  ORDER BY ways.id, idx
)
-- query ways matching the bounding box
SELECT
  count(*),
  tags['building:condition']
FROM exploded_ways
GROUP BY tags['building:condition']
ORDER BY count(*) DESC;


Most buildings are unsurveyed (125,000 is a lot!), but of those that have been, most are average (as you’d expect). If you were to further group these buildings geographically, you’d have a starting point to determine which areas of Dhaka might benefit the most.

Conclusion

OSM data, while incredibly rich and valuable, can be difficult to work with, due to both its size and its data model. In addition to the time spent downloading large files to work with locally, time is spent installing and configuring tools and converting the data into more queryable formats. We think Amazon Athena combined with the ORC version of the planet file, updated on a weekly basis, is an extremely powerful and cost-effective combination. This allows anyone to start querying billions of records with simple SQL in no time, giving you the chance to focus on the analysis, not the infrastructure.

To download the data and experiment with it using other tools, the latest OSM ORC-formatted file is available via OSM on AWS at s3://osm-pds/planet/planet-latest.orcs3://osm-pds/planet-history/history-latest.orc, and s3://osm-pds/changesets/changesets-latest.orc.

We look forward to hearing what you find out!

The Raspberry Pi-powered loom

Post Syndicated from Liz Upton original https://www.raspberrypi.org/blog/the-raspberry-pi-powered-loom/

We’re a small organisation full of makers, and I think at least two of us own a hand loom for weaving textiles. (One of the reasons I enjoy the TV show Vikings so much is the casual looming that’s going on as backdrop in many of the indoor scenes – the textile sort, not the impending-doom sort, although there’s plenty of that too.)

siggy laergatha loom

Siggy and Laergatha (personal role model) get down to a spot of light weaving before commencing to crush skulls and pillage.

Here in the 21st century, Lorna and I use hand looms because powered looms are very expensive. They’re also usually pretty enormous, being meant for enterprise rather than home use. This is pesky, because there’s a lot of repetitive action involved, which can be hell on the carpal tunnels; weaving can be slow, tough work.

loom

Suspicious automation

Enter the Raspberry Pi.

Fred Hoefler has taken a desktop loom and added a Raspberry Pi to automate it. (Your computer’s fine: this video has no sound.)

Loom Operation

The general sequence of events for running my Raspberry Pi controlled loom. The project was really a proof of concept idea rather than an actual production model. This video is intended to supplement my blog at www.photographic-perspectives.com Sorry, there is not audio with this.

Fred wrote about the project on his website, explaining that he came up with the idea for very personal reasons. His wife Gina has been a weaver for 30 years, but she began to experience difficulties with the physical aspects of using her loom as she grew older. Conversations with other unwillingly retired weavers told Fred that Gina’s situation was not uncommon, and led him to design something to help. His device is intended to help older weavers who have trouble with the hard work of throwing the shuttle and holding down the pedals. Assistive looms cost upwards of $10,000: Fred’s solution comes in at a tidy $150, factoring in loom, Pi, and some motors from Amazon. So this isn’t for hobbyists like me: this loom can be a way for people whose livelihoods depend on being able to weave to continue working long after they might have had to retire.

One of the most satisfying things about the Raspberry Pi for me is its power to drive cost out of devices like this, and to change the way we work. This is a simple build, but it has so much potential to keep someone’s income flowing: we hope to see more as Fred develops the project.

The post The Raspberry Pi-powered loom appeared first on Raspberry Pi.

[$] An asynchronous Internet in GNOME

Post Syndicated from n8willis original http://lwn.net/Articles/699636/rss

At GUADEC 2016 in
Karlsruhe, Germany, Jonathan Blandford challenged the GNOME project to
rethink how its desktop software uses network access. The GNOME
desktop assumes Internet connectivity is always available, which has
the side effect of making the software stack considerably less useful
and, indeed, usable to people who live in those places regarded as the developing world.

Photocatalysis with a Raspberry Pi

Post Syndicated from Lorna Lynch original https://www.raspberrypi.org/blog/photocatalysis-raspberry-pi/

Access to clean, safe drinking water is a global problem: as water.org notes, 663 million people lack access to water that’s safe to drink. That’s twice the population of the United States, or one person in every ten. Additionally, a recent review of rural water system sustainability in eight countries in Africa, South Asia, and Central America found an average water project failure rate of 20-40 percent. It’s no surprise that the search for a solution to this crisis preoccupies scientists the world over, but what you may not have expected is that, in a lab in Cardiff University, researchers are using Raspberry Pi to help in their efforts to bring safe drinking water to some of the poorest areas of the world.

A tap set into a wall, with sign above reading "SAFE DRINKING WATER"

There are three processes involved in water purification, two of which are reasonably straightforward: filtration can remove particulate matter, while heating water to near 100°C kills bacteria. However, the third process — the removal of highly toxic hydrocarbons, typically from fertiliser and pesticide runoff — is very difficult and, currently, very expensive. The Cardiff group is working on a project to find a cheap, effective method of removing these hydrocarbons from water by means of photocatalysis. Essentially, this means they are finding a way to produce clean water using little more than sunlight, which is really pretty mind-blowing.

Here’s a picture of their experimental setup; you can see the Raspberry Pi in its case on the right-hand side.

A laboratory photocatalysis setup at Cardiff University: on a bench are a beaker of water dosed with methylene blue "pollutant" under UV LED illumination, semi-transparent tubing connecting the contents of the beaker to a flow cell, a Raspberry Pi, and other components.

Raspberry Pi in the lab

A cheap, readily available chemical, titanium dioxide, is spin-coated onto a glass wafer which sits in the bottom of the beaker with a UV LED above it. This wafer coating acts as a semiconductor; when UV photons from the LED strike it, its electrons become mobile, creating locations with positive charge and others with negative charge. As a result, both oxidation reactions and reduction reactions are set off. These reactions break down the hydrocarbons, leaving you with pure water, carbon dioxide, and hydrogen. The solution is pumped through a flow cell (you can see this in the centre of the picture), where an LED light source is shone through the stream and the amount of light passing through is registered by a photodiode. The photodiode turns this output into a voltage, which can be read by the Raspberry Pi with the help of an ADC.

The team are currently using two organic dyes, methyl orange and methylene blue, to simulate pollutants for the purposes of the experiment: it is possible to see the reaction take place with the naked eye, as the colour of the dye becomes progressively less saturated. A colourless solution means the “pollutants” have been entirely broken down. You can see both dyes in situ here:

Laboratory photocatalysis setups at Cardiff University: on a bench are a large LCD display with a desktop showing the Raspberry Pi logo, beakers of water dosed with methyl orange and methylene blue "pollutants", semi-transparent tubing connecting the beakers' contents to flow cells, a Raspberry Pi, and other components.

Experimental setup with methyl orange and methylene blue

In previous versions of the setup, it was necessary to use some very large, expensive pieces of equipment to drive the experiment and assess the rate and efficacy of the reaction (two power sources and a voltmeter, each of which cost several hundred pounds); the Raspberry Pi performs the same function for a fraction of the price, enabling multiple experiments to be run in the lab, and offering the possibility of building a neat, cost-effective unit for use in the real world in the future.

Several of the team have very personal reasons for being involved in the project: Eman Alghamdi is from Saudi Arabia, a country which, despite its wealth, struggles to supply water to its people. Her colleague Jess Mabin was inspired by spending time in Africa working with an anti-poverty charity. They hope to produce a device which will be both cheap to manufacture and rugged enough to be used in rural areas throughout the world.

Jess, a research scientist, smiles as she pipettes methylene blue into a beaker that is part of her group's photocatalysis setup.

Jess demonstrates the experiment: methylene blue going in!

As well as thoroughly testing the reaction rate and the lifespan of the wafer coating, the team are hoping to streamline their equipment by building their own version of a HAT to incorporate the ADC, the photodiode, and other components. Ultimately the Pi and its peripherals could form a small, rugged, cost-effective, essentially self-sustaining device which could be used all over the world to help produce clean, safe drinking water. We are really pleased to see the Raspberry Pi being used in this way, and we wish Jess, Eman, and their colleagues every success!

The post Photocatalysis with a Raspberry Pi appeared first on Raspberry Pi.