Tag Archives: adblock

Неделя, 3 Юни 2018

Post Syndicated from georgi original http://georgi.unixsol.org/diary/archive.php/2018-06-03

Всеки има нужда да бъде спасен от свинщината, наречена “реклама” във
всичките и форми. За хората с компютър и бразуер, това отдавна е решен
проблем благодарение на AdBlock и подобни плъгини (стига да не
използвате браузер като Chrome, но в този случай си заслужавате
всичко дето ви се случва).

По-принцип не оставям компютър без инсталиран AdBlock, това си е направо
обществено полезна дейност. Кофтито е, че на мобилния телефон, дори и да
използвате Firefox и да имате подходящите Addons, програмчетата пак
се изхитряват и ви спамят.

Сега, ако сте root-нали телефона (което никой не прави), можете да
направите нещо по въпроса, но си е разправия, а както всички знаем,
удобството винаги печели пред сигурността.

За щастие има има много лесен начин, да се отървете от долните
спамери в две прости стъпки:

1. Инсталирате си Blokada.

2. Активирате я.

Et voilà – никъде повече няма да ви изкача спам,

Как работи нещото? Прави се на vpn защото това му дава възможност
да филтрира dns заявките и съответно когато някоя програма пита
за pagead.doubleclick.net и подобни – просто му отговаря с

Просто, ефективно, не изисква root и бърка директно в джоба на
всичката интернет паплач, която си въобразява, че може да ви залива
с лайна 24/7.

Detecting Adblocker Blockers

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2018/01/detecting_adblo.html

Interesting research on the prevalence of adblock blockers: “Measuring and Disrupting Anti-Adblockers Using Differential Execution Analysis“:

Abstract: Millions of people use adblockers to remove intrusive and malicious ads as well as protect themselves against tracking and pervasive surveillance. Online publishers consider adblockers a major threat to the ad-powered “free” Web. They have started to retaliate against adblockers by employing anti-adblockers which can detect and stop adblock users. To counter this retaliation, adblockers in turn try to detect and filter anti-adblocking scripts. This back and forth has prompted an escalating arms race between adblockers and anti-adblockers.

We want to develop a comprehensive understanding of anti-adblockers, with the ultimate aim of enabling adblockers to bypass state-of-the-art anti-adblockers. In this paper, we present a differential execution analysis to automatically detect and analyze anti-adblockers. At a high level, we collect execution traces by visiting a website with and without adblockers. Through differential execution analysis, we are able to pinpoint the conditions that lead to the differences caused by anti-adblocking code. Using our system, we detect anti-adblockers on 30.5% of the Alexa top-10K websites which is 5-52 times more than reported in prior literature. Unlike prior work which is limited to detecting visible reactions (e.g., warning messages) by anti-adblockers, our system can discover attempts to detect adblockers even when there is no visible reaction. From manually checking one third of the detected websites, we find that the websites that have no visible reactions constitute over 90% of the cases, completely dominating the ones that have visible warning messages. Finally, based on our findings, we further develop JavaScript rewriting and API hooking based solutions (the latter implemented as a Chrome extension) to help adblockers bypass state-of-the-art anti-adblockers.

News article.

[$] Process tagging with ptags

Post Syndicated from corbet original https://lwn.net/Articles/741261/rss

For various reasons related to accounting and security, there is recurring
interest in having the kernel identify the container that holds any given
process. Attempts to implement that functionality tend to run into the
same roadblock, though: the kernel has no concept of what a “container” is,
and there is seemingly little desire to change that state of affairs. A
solution to this problem may exist in the form of a neglected
patch called “ptags”, which enables the attachment of arbitrary tags to

Pioneers winners: only you can save us

Post Syndicated from Erin Brindley original https://www.raspberrypi.org/blog/pioneers-winners-only-you-can-save-us/

She asked for help, and you came to her aid. Pioneers, the winners of the Only you can save us challenge have been picked!

Can you see me? Only YOU can save us!

I need your help. This is a call out for those between 11- and 16-years-old in the UK and Republic of Ireland. Something has gone very, very wrong and only you can save us. I’ve collected together as much information for you as I can. You’ll find it at http://www.raspberrypi.org/pioneers.

The challenge

In August we intercepted an emergency communication from a lonesome survivor. She seemed to be in quite a bit of trouble, and asked all you young people aged 11 to 16 to come up with something to help tackle the oncoming crisis, using whatever technology you had to hand. You had ten weeks to work in teams of two to five with an adult mentor to fulfil your mission.

The judges

We received your world-saving ideas, and our savvy survivor pulled together a ragtag bunch of apocalyptic experts to help us judge which ones would be the winning entries.

Dr Shini Somara

Dr Shini Somara is an advocate for STEM education and a mechanical engineer. She was host of The Health Show and has appeared in documentaries for the BBC, PBS Digital, and Sky. You can check out her work hosting Crash Course Physics on YouTube.

Prof Lewis Dartnell is an astrobiologist and author of the book The Knowledge: How to Rebuild Our World From Scratch.

Emma Stephenson has a background in aeronautical engineering and currently works in the Shell Foundation’s Access to Energy and Sustainable Mobility portfolio.

Currently sifting through the entries with the other judges of #makeyourideas with @raspberrypifoundation @_raspberrypi_

151 Likes, 3 Comments – Shini Somara (@drshinisomara) on Instagram: “Currently sifting through the entries with the other judges of #makeyourideas with…”

The winners

Our survivor is currently putting your entries to good use repairing, rebuilding, and defending her base. Our judges chose the following projects as outstanding examples of world-saving digital making.

Theme winner: Computatron

Raspberry Pioneers 2017 – Nerfus Dislikus Killer Robot

This is our entry to the pioneers ‘Only you can save us’ competition. Our team name is Computatrum. Hope you enjoy!

Are you facing an unknown enemy whose only weakness is Nerf bullets? Then this is the robot for you! We loved the especially apocalyptic feel of the Computatron’s cleverly hacked and repurposed elements. The team even used an old floppy disc mechanism to help fire their bullets!

Technically brilliant: Robot Apocalypse Committee

Pioneers Apocalypse 2017 – RationalPi

Thousands of lines of code… Many sheets of acrylic… A camera, touchscreen and fingerprint scanner… This is our entry into the Raspberry Pi Pioneers2017 ‘Only YOU can Save Us’ theme. When zombies or other survivors break into your base, you want a secure way of storing your crackers.

The Robot Apocalypse Committee is back, and this time they’ve brought cheese! The crew designed a cheese- and cracker-dispensing machine complete with face and fingerprint recognition to ensure those rations last until the next supply drop.

Best explanation: Pi Chasers

Tala – Raspberry Pi Pioneers Project

Hi! We are PiChasers and we entered the Raspberry Pi Pionners challenge last time when the theme was “Make it Outdoors!” but now we’ve been faced with another theme “Apocolypse”. We spent a while thinking of an original thing that would help in an apocolypse and decided upon a ‘text-only phone’ which uses local radio communication rather than cellular.

This text-based communication device encased in a tupperware container could be a lifesaver in a crisis! And luckily, the Pi Chasers produced an excellent video and amazing GitHub repo, ensuring that any and all survivors will be able to build their own in the safety of their base.

Most inspiring journey: Three Musketeers

Pioneers Entry – The Apocalypse

Pioneers Entry Team Name: The Three Musketeers Team Participants: James, Zach and Tom

We all know that zombies are terrible at geometry, and the Three Musketeers used this fact to their advantage when building their zombie security system. We were impressed to see the team working together to overcome the roadblocks they faced along the way.

We appreciate what you’re trying to do: Zombie Trolls

Zombie In The Middle

Uploaded by CDA Bodgers on 2017-12-01.

Playing piggy in the middle with zombies sure is a unique way of saving humankind from total extinction! We loved this project idea, and although the Zombie Trolls had a little trouble with their motors, we’re sure with a little more tinkering this zombie-fooling contraption could save us all.

Most awesome

Our judges also wanted to give a special commendation to the following teams for their equally awesome apocalypse-averting ideas:

  • PiRates, for their multifaceted zombie-proofing defence system and the high production value of their video
  • Byte them Pis, for their beautiful zombie-detecting doormat
  • Unatecxon, for their impressive bunker security system
  • Team Crompton, for their pressure-activated door system
  • Team Ernest, for their adventures in LEGO

The prizes

All our winning teams have secured exclusive digital maker boxes. These are jam-packed with tantalising tech to satisfy all tinkering needs, including:

Our theme winners have also secured themselves a place at Coolest Projects 2018 in Dublin, Ireland!

Thank you to everyone who got involved in this round of Pioneers. Look out for your awesome submission swag arriving in the mail!

The post Pioneers winners: only you can save us appeared first on Raspberry Pi.

Coaxing 2D platforming out of Unity

Post Syndicated from Eevee original https://eev.ee/blog/2017/10/13/coaxing-2d-platforming-out-of-unity/

An anonymous donor asked a question that I can’t even begin to figure out how to answer, but they also said anything else is fine, so here’s anything else.

I’ve been avoiding writing about game physics, since I want to save it for ✨ the book I’m writing ✨, but that book will almost certainly not touch on Unity. Here, then, is a brief run through some of the brick walls I ran into while trying to convince Unity to do 2D platforming.

This is fairly high-level — there are no blocks of code or helpful diagrams. I’m just getting this out of my head because it’s interesting. If you want more gritty details, I guess you’ll have to wait for ✨ the book ✨.

The setup

I hadn’t used Unity before. I hadn’t even used a “real” physics engine before. My games so far have mostly used LÖVE, a Lua-based engine. LÖVE includes box2d bindings, but for various reasons (not all of them good), I opted to avoid them and instead write my own physics completely from scratch. (How, you ask? ✨ Book ✨!)

I was invited to work on a Unity project, Chaos Composer, that someone else had already started. It had basic movement already implemented; I taught myself Unity’s physics system by hacking on it. It’s entirely possible that none of this is actually the best way to do anything, since I was really trying to reproduce my own homegrown stuff in Unity, but it’s the best I’ve managed to come up with.

Two recurring snags were that you can’t ask Unity to do multiple physics updates in a row, and sometimes getting the information I wanted was difficult. Working with my own code spoiled me a little, since I could invoke it at any time and ask it anything I wanted; Unity, on the other hand, is someone else’s black box with a rigid interface on top.

Also, wow, Googling for a lot of this was not quite as helpful as expected. A lot of what’s out there is just the first thing that works, and often that’s pretty hacky and imposes severe limits on the game design (e.g., “this won’t work with slopes”). Basic movement and collision are the first thing you do, which seems to me like the worst time to be locking yourself out of a lot of design options. I tried very (very, very, very) hard to minimize those kinds of constraints.

Problem 1: Movement

When I showed up, movement was already working. Problem solved!

Like any good programmer, I immediately set out to un-solve it. Given a “real” physics engine like Unity prominently features, you have two options: ⓐ treat the player as a physics object, or ⓑ don’t. The existing code went with option ⓑ, like I’d done myself with LÖVE, and like I’d seen countless people advise. Using a physics sim makes for bad platforming.

But… why? I believed it, but I couldn’t concretely defend it. I had to know for myself. So I started a blank project, drew some physics boxes, and wrote a dozen-line player controller.

Ah! Immediate enlightenment.

If the player was sliding down a wall, and I tried to move them into the wall, they would simply freeze in midair until I let go of the movement key. The trouble is that the physics sim works in terms of forces — moving the player involves giving them a nudge in some direction, like a giant invisible hand pushing them around the level. Surprise! If you press a real object against a real wall with your real hand, you’ll see the same effect — friction will cancel out gravity, and the object will stay in midair..

Platformer movement, as it turns out, doesn’t make any goddamn physical sense. What is air control? What are you pushing against? Nothing, really; we just have it because it’s nice to play with, because not having it is a nightmare.

I looked to see if there were any common solutions to this, and I only really found one: make all your walls frictionless.

Game development is full of hacks like this, and I… don’t like them. I can accept that minor hacks are necessary sometimes, but this one makes an early and widespread change to a fundamental system to “fix” something that was wrong in the first place. It also imposes an “invisible” requirement, something I try to avoid at all costs — if you forget to make a particular wall frictionless, you’ll never know unless you happen to try sliding down it.

And so, I swiftly returned to the existing code. It wasn’t too different from what I’d come up with for LÖVE: it applied gravity by hand, tracked the player’s velocity, computed the intended movement each frame, and moved by that amount. The interesting thing was that it used MovePosition, which schedules a movement for the next physics update and stops the movement if the player hits something solid.

It’s kind of a nice hybrid approach, actually; all the “physics” for conscious actors is done by hand, but the physics engine is still used for collision detection. It’s also used for collision rejection — if the player manages to wedge themselves several pixels into a solid object, for example, the physics engine will try to gently nudge them back out of it with no extra effort required on my part. I still haven’t figured out how to get that to work with my homegrown stuff, which is built to prevent overlap rather than to jiggle things out of it.

But wait, what about…

Our player is a dynamic body with rotation lock and no gravity. Why not just use a kinematic body?

I must be missing something, because I do not understand the point of kinematic bodies. I ran into this with Godot, too, which documented them the same way: as intended for use as players and other manually-moved objects. But by default, they don’t even collide with other kinematic bodies or static geometry. What? There’s a checkbox to turn this on, which I enabled, but then I found out that MovePosition doesn’t stop kinematic bodies when they hit something, so I would’ve had to cast along the intended path of movement to figure out when to stop, thus duplicating the same work the physics engine was about to do.

But that’s impossible anyway! Static geometry generally wants to be made of edge colliders, right? They don’t care about concave/convex. Imagine the player is standing on the ground near a wall and tries to move towards the wall. Both the ground and the wall are different edges from the same edge collider.

If you try to cast the player’s hitbox horizontally, parallel to the ground, you’ll only get one collision: the existing collision with the ground. Casting doesn’t distinguish between touching and hitting. And because Unity only reports one collision per collider, and because the ground will always show up first, you will never find out about the impending wall collision.

So you’re forced to either use raycasts for collision detection or decomposed polygons for world geometry, both of which are slightly worse tools for no real gain.

I ended up sticking with a dynamic body.

Oh, one other thing that doesn’t really fit anywhere else: keep track of units! If you’re adding something called “velocity” directly to something called “position”, something has gone very wrong. Acceleration is distance per time squared; velocity is distance per time; position is distance. You must multiply or divide by time to convert between them.

I never even, say, add a constant directly to position every frame; I always phrase it as velocity and multiply by Δt. It keeps the units consistent: time is always in seconds, not in tics.

Problem 2: Slopes

Ah, now we start to get off in the weeds.

A sort of pre-problem here was detecting whether we’re on a slope, which means detecting the ground. The codebase originally used a manual physics query of the area around the player’s feet to check for the ground, which seems to be somewhat common, but that can’t tell me the angle of the detected ground. (It’s also kind of error-prone, since “around the player’s feet” has to be specified by hand and may not stay correct through animations or changes in the hitbox.)

I replaced that with what I’d eventually settled on in LÖVE: detect the ground by detecting collisions, and looking at the normal of the collision. A normal is a vector that points straight out from a surface, so if you’re standing on the ground, the normal points straight up; if you’re on a 10° incline, the normal points 10° away from straight up.

Not all collisions are with the ground, of course, so I assumed something is ground if the normal pointed away from gravity. (I like this definition more than “points upwards”, because it avoids assuming anything about the direction of gravity, which leaves some interesting doors open for later on.) That’s easily detected by taking the dot product — if it’s negative, the collision was with the ground, and I now have the normal of the ground.

Actually doing this in practice was slightly tricky. With my LÖVE engine, I could cram this right into the middle of collision resolution. With Unity, not quite so much. I went through a couple iterations before I really grasped Unity’s execution order, which I guess I will have to briefly recap for this to make sense.

Unity essentially has two update cycles. It performs physics updates at fixed intervals for consistency, and updates everything else just before rendering. Within a single frame, Unity does as many fixed physics updates as it has spare time for (which might be zero, one, or more), then does a regular update, then renders. User code can implement either or both of Update, which runs during a regular update, and FixedUpdate, which runs just before Unity does a physics pass.

So my solution was:

  • At the very end of FixedUpdate, clear the actor’s “on ground” flag and ground normal.

  • During OnCollisionEnter2D and OnCollisionStay2D (which are called from within a physics pass), if there’s a collision that looks like it’s with the ground, set the “on ground” flag and ground normal. (If there are multiple ground collisions, well, good luck figuring out the best way to resolve that! At the moment I’m just taking the first and hoping for the best.)

That means there’s a brief window between the end of FixedUpdate and Unity’s physics pass during which a grounded actor might mistakenly believe it’s not on the ground, which is a bit of a shame, but there are very few good reasons for anything to be happening in that window.

Okay! Now we can do slopes.

Just kidding! First we have to do sliding.

When I first looked at this code, it didn’t apply gravity while the player was on the ground. I think I may have had some problems with detecting the ground as result, since the player was no longer pushing down against it? Either way, it seemed like a silly special case, so I made gravity always apply.

Lo! I was a fool. The player could no longer move.

Why? Because MovePosition does exactly what it promises. If the player collides with something, they’ll stop moving. Applying gravity means that the player is trying to move diagonally downwards into the ground, and so MovePosition stops them immediately.

Hence, sliding. I don’t want the player to actually try to move into the ground. I want them to move the unblocked part of that movement. For flat ground, that means the horizontal part, which is pretty much the same as discarding gravity. For sloped ground, it’s a bit more complicated!

Okay but actually it’s less complicated than you’d think. It can be done with some cross products fairly easily, but Unity makes it even easier with a couple casts. There’s a Vector3.ProjectOnPlane function that projects an arbitrary vector on a plane given by its normal — exactly the thing I want! So I apply that to the attempted movement before passing it along to MovePosition. I do the same thing with the current velocity, to prevent the player from accelerating infinitely downwards while standing on flat ground.

One other thing: I don’t actually use the detected ground normal for this. The player might be touching two ground surfaces at the same time, and I’d want to project on both of them. Instead, I use the player body’s GetContacts method, which returns contact points (and normals!) for everything the player is currently touching. I believe those contact points are tracked by the physics engine anyway, so asking for them doesn’t require any actual physics work.

(Looking at the code I have, I notice that I still only perform the slide for surfaces facing upwards — but I’d want to slide against sloped ceilings, too. Why did I do this? Maybe I should remove that.)

(Also, I’m pretty sure projecting a vector on a plane is non-commutative, which raises the question of which order the projections should happen in and what difference it makes. I don’t have a good answer.)

(I note that my LÖVE setup does something slightly different: it just tries whatever the movement ought to be, and if there’s a collision, then it projects — and tries again with the remaining movement. But I can’t ask Unity to do multiple moves in one physics update, alas.)

Okay! Now, slopes. But actually, with the above work done, slopes are most of the way there already.

One obvious problem is that the player tries to move horizontally even when on a slope, and the easy fix is to change their movement from speed * Vector2.right to speed * new Vector2(ground.y, -ground.x) while on the ground. That’s the ground normal rotated a quarter-turn clockwise, so for flat ground it still points to the right, and in general it points rightwards along the ground. (Note that it assumes the ground normal is a unit vector, but as far as I’m aware, that’s true for all the normals Unity gives you.)

Another issue is that if the player stands motionless on a slope, gravity will cause them to slowly slide down it — because the movement from gravity will be projected onto the slope, and unlike flat ground, the result is no longer zero. For conscious actors only, I counter this by adding the opposite factor to the player’s velocity as part of adding in their walking speed. This matches how the real world works, to some extent: when you’re standing on a hill, you’re exerting some small amount of effort just to stay in place.

(Note that slope resistance is not the same as friction. Okay, yes, in the real world, virtually all resistance to movement happens as a result of friction, but bracing yourself against the ground isn’t the same as being passively resisted.)

From here there are a lot of things you can do, depending on how you think slopes should be handled. You could make the player unable to walk up slopes that are too steep. You could make walking down a slope faster than walking up it. You could make jumping go along the ground normal, rather than straight up. You could raise the player’s max allowed speed while running downhill. Whatever you want, really. Armed with a normal and awareness of dot products, you can do whatever you want.

But first you might want to fix a few aggravating side effects.

Problem 3: Ground adherence

I don’t know if there’s a better name for this. I rarely even see anyone talk about it, which surprises me; it seems like it should be a very common problem.

The problem is: if the player runs up a slope which then abruptly changes to flat ground, their momentum will carry them into the air. For very fast players going off the top of very steep slopes, this makes sense, but it becomes visible even for relatively gentle slopes. It was a mild nightmare in the original release of our game Lunar Depot 38, which has very “rough” ground made up of lots of shallow slopes — so the player is very frequently slightly off the ground, which meant they couldn’t jump, for seemingly no reason. (I even had code to fix this, but I disabled it because of a silly visual side effect that I never got around to fixing.)

Anyway! The reason this is a problem is that game protagonists are generally not boxes sliding around — they have legs. We don’t go flying off the top of real-world hilltops because we put our foot down until it touches the ground.

Simulating this footfall is surprisingly fiddly to get right, especially with someone else’s physics engine. It’s made somewhat easier by Cast, which casts the entire hitbox — no matter what shape it is — in a particular direction, as if it had moved, and tells you all the hypothetical collisions in order.

So I cast the player in the direction of gravity by some distance. If the cast hits something solid with a ground-like collision normal, then the player must be close to the ground, and I move them down to touch it (and set that ground as the new ground normal).

There are some wrinkles.

Wrinkle 1: I only want to do this if the player is off the ground now, but was on the ground last frame, and is not deliberately moving upwards. That latter condition means I want to skip this logic if the player jumps, for example, but also if the player is thrust upwards by a spring or abducted by a UFO or whatever. As long as external code goes through some interface and doesn’t mess with the player’s velocity directly, that shouldn’t be too hard to track.

Wrinkle 2: When does this logic run? It needs to happen after the player moves, which means after a Unity physics pass… but there’s no callback for that point in time. I ended up running it at the beginning of FixedUpdate and the beginning of Update — since I definitely want to do it before rendering happens! That means it’ll sometimes happen twice between physics updates. (I could carefully juggle a flag to skip the second run, but I… didn’t do that. Yet?)

Wrinkle 3: I can’t move the player with MovePosition! Remember, MovePosition schedules a movement, it doesn’t actually perform one; that means if it’s called twice before the physics pass, the first call is effectively ignored. I can’t easily combine the drop with the player’s regular movement, for various fiddly reasons. I ended up doing it “by hand” using transform.Translate, which I think was the “old way” to do manual movement before MovePosition existed. I’m not totally sure if it activates triggers? For that matter, I’m not sure it even notices collisions — but since I did a full-body Cast, there shouldn’t be any anyway.

Wrinkle 4: What, exactly, is “some distance”? I’ve yet to find a satisfying answer for this. It seems like it ought to be based on the player’s current speed and the slope of the ground they’re moving along, but every time I’ve done that math, I’ve gotten totally ludicrous answers that sometimes exceed the size of a tile. But maybe that’s not wrong? Play around, I guess, and think about when the effect should “break” and the player should go flying off the top of a hill.

Wrinkle 5: It’s possible that the player will launch off a slope, hit something, and then be adhered to the ground where they wouldn’t have hit it. I don’t much like this edge case, but I don’t see a way around it either.

This problem is surprisingly awkward for how simple it sounds, and the solution isn’t entirely satisfying. Oh, well; the results are much nicer than the solution. As an added bonus, this also fixes occasional problems with running down a hill and becoming detached from the ground due to precision issues or whathaveyou.

Problem 4: One-way platforms

Ah, what a nightmare.

It took me ages just to figure out how to define one-way platforms. Only block when the player is moving downwards? Nope. Only block when the player is above the platform? Nuh-uh.

Well, okay, yes, those approaches might work for convex players and flat platforms. But what about… sloped, one-way platforms? There’s no reason you shouldn’t be able to have those. If Super Mario World can do it, surely Unity can do it almost 30 years later.

The trick is, again, to look at the collision normal. If it faces away from gravity, the player is hitting a ground-like surface, so the platform should block them. Otherwise (or if the player overlaps the platform), it shouldn’t.

Here’s the catch: Unity doesn’t have conditional collision. I can’t decide, on the fly, whether a collision should block or not. In fact, I think that by the time I get a callback like OnCollisionEnter2D, the physics pass is already over.

I could go the other way and use triggers (which are non-blocking), but then I have the opposite problem: I can’t stop the player on the fly. I could move them back to where they hit the trigger, but I envision all kinds of problems as a result. What if they were moving fast enough to activate something on the other side of the platform? What if something else moved to where I’m trying to shove them back to in the meantime? How does this interact with ground detection and listing contacts, which would rightly ignore a trigger as non-blocking?

I beat my head against this for a while, but the inability to respond to collision conditionally was a huge roadblock. It’s all the more infuriating a problem, because Unity ships with a one-way platform modifier thing. Unfortunately, it seems to have been implemented by someone who has never played a platformer. It’s literally one-way — the player is only allowed to move straight upwards through it, not in from the sides. It also tries to block the player if they’re moving downwards while inside the platform, which invokes clumsy rejection behavior. And this all seems to be built into the physics engine itself somehow, so I can’t simply copy whatever they did.

Eventually, I settled on the following. After calculating attempted movement (including sliding), just at the end of FixedUpdate, I do a Cast along the movement vector. I’m not thrilled about having to duplicate the physics engine’s own work, but I do filter to only things on a “one-way platform” physics layer, which should at least help. For each object the cast hits, I use Physics2D.IgnoreCollision to either ignore or un-ignore the collision between the player and the platform, depending on whether the collision was ground-like or not.

(A lot of people suggested turning off collision between layers, but that can’t possibly work — the player might be standing on one platform while inside another, and anyway, this should work for all actors!)

Again, wrinkles! But fewer this time. Actually, maybe just one: handling the case where the player already overlaps the platform. I can’t just check for that with e.g. OverlapCollider, because that doesn’t distinguish between overlapping and merely touching.

I came up with a fairly simple fix: if I was going to un-ignore the collision (i.e. make the platform block), and the cast distance is reported as zero (either already touching or overlapping), I simply do nothing instead. If I’m standing on the platform, I must have already set it blocking when I was approaching it from the top anyway; if I’m overlapping it, I must have already set it non-blocking to get here in the first place.

I can imagine a few cases where this might go wrong. Moving platforms, especially, are going to cause some interesting issues. But this is the best I can do with what I know, and it seems to work well enough so far.

Oh, and our player can deliberately drop down through platforms, which was easy enough to implement; I just decide the platform is always passable while some button is held down.

Problem 5: Pushers and carriers

I haven’t gotten to this yet! Oh boy, can’t wait. I implemented it in LÖVE, but my way was hilariously invasive; I’m hoping that having a physics engine that supports a handwaved “this pushes that” will help. Of course, you also have to worry about sticking to platforms, for which the recommended solution is apparently to parent the cargo to the platform, which sounds goofy to me? I guess I’ll find out when I throw myself at it later.

Overall result

I ended up with a fairly pleasant-feeling system that supports slopes and one-way platforms and whatnot, with all the same pieces as I came up with for LÖVE. The code somehow ended up as less of a mess, too, but it probably helps that I’ve been down this rabbit hole once before and kinda knew what I was aiming for this time.

Animation of a character running smoothly along the top of an irregular dinosaur skeleton

Sorry that I don’t have a big block of code for you to copy-paste into your project. I don’t think there are nearly enough narrative discussions of these fundamentals, though, so hopefully this is useful to someone. If not, well, look forward to ✨ my book, that I am writing ✨!

DMCA Used to Remove Ad Server URL From Easylist Ad Blocklist

Post Syndicated from Andy original https://torrentfreak.com/dmca-used-to-remove-ad-server-url-from-easylist-ad-blocklist-170811/

The default business model on the Internet is “free” for consumers. Users largely expect websites to load without paying a dime but of course, there’s no such thing as a free lunch. To this end, millions of websites are funded by advertising revenue.

Sensible sites ensure that any advertising displayed is unobtrusive to the visitor but lots seem to think that bombarding users with endless ads, popups, and other hindrances is the best way to do business. As a result, ad blockers are now deployed by millions of people online.

In order to function, ad-blocking tools – such as uBlock Origin or Adblock – utilize lists of advertising domains compiled by third parties. One of the most popular is Easylist, which is distributed by authors fanboy, MonztA, Famlam, and Khrinunder, under dual Creative Commons Attribution-ShareAlike and GNU General Public Licenses.

With the freedom afforded by those licenses, copyright tends not to figure high on the agenda for Easylist. However, a legal problem that has just raised its head is causing serious concern among those in the ad-blocking community.

Two days ago a somewhat unusual commit appeared in the Easylist repo on Github. As shown in the image below, a domain URL previously added to Easylist had been removed following a DMCA takedown notice filed with Github.

Domain text taken down by DMCA?

The DMCA notice in question has not yet been published but it’s clear that it targets the domain ‘functionalclam.com’. A user called ‘ameshkov’ helpfully points out a post by a new Github user called ‘DMCAHelper’ which coincided with the start of the takedown process more than three weeks ago.

A domain in a list circumvents copyright controls?

Aside from the curious claims of a URL “circumventing copyright access controls” (domains themselves cannot be copyrighted), the big questions are (i) who filed the complaint and (ii) who operates Functionalclam.com? The domain WHOIS is hidden but according to a helpful sleuth on Github, it’s operated by anti ad-blocking company Admiral.

Ad-blocking means money down the drain….

If that is indeed the case, we have the intriguing prospect of a startup attempting to protect its business model by using a novel interpretation of copyright law to have a domain name removed from a list. How this will pan out is unclear but a notice recently published on Functionalclam.com suggests the route the company wishes to take.

“This domain is used by digital publishers to control access to copyrighted content in accordance with the Digital Millenium Copyright Act and understand how visitors are accessing their copyrighted content,” the notice begins.

Combined with the comments by DMCAHelper on Github, this statement suggests that the complainants believe that interference with the ad display process (ads themselves could be the “copyrighted content” in question) represents a breach of section 1201 of the DMCA.

If it does, that could have huge consequences for online advertising but we will need to see the original DMCA notice to have a clearer idea of what this is all about. Thus far, Github hasn’t published it but already interest is growing. A representative from the EFF has already contacted the Easylist team, so this battle could heat up pretty quickly.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

MagPi 60: the ultimate troubleshooting guide

Post Syndicated from Rob Zwetsloot original https://www.raspberrypi.org/blog/magpi-60/

Hey folks, Rob from The MagPi here! It’s the last Thursday of the month, and that can only mean one thing: a brand-new The MagPi issue is out! In The MagPi 60, we’re bringing you the top troubleshooting tips for your Raspberry Pi, sourced directly from our amazing community.

The MagPi 60 cover with DVD slip case shown

The MagPi #60 comes with a huge troubleshooting guide

The MagPi 60

Our feature-length guide covers snags you might encounter while using a Raspberry Pi, and it is written for newcomers and veterans alike! Do you hit a roadblock while booting up your Pi? Are you having trouble connecting it to a network? Don’t worry – in this issue you’ll find troubleshooting advice you can use to solve your problem. And, as always, if you’re still stuck, you can head over to the Raspberry Pi forums for help.

More than troubleshooting

That’s not all though – Issue 60 also includes a disc with Raspbian-x86! This version of Raspbian for PCs contains all the recent updates and additions, such as offline Scratch 2.0 and the new Thonny IDE. And – *drumroll* – the disc version can be installed to your PC or Mac. The last time we had a Raspbian disc on the cover, many of you requested an installable version, so here you are! There is an installation guide inside the mag, so you’ll be all set to get going.

On top of that, you’ll find our usual array of amazing tutorials, projects, and reviews. There’s a giant guitar, Siri voice control, Pi Zeros turned into wireless-connected USB drives, and even a review of a new robot kit. You won’t want to miss it!

A spread from The MagPi 60 showing a giant Raspberry Pi-powered guitar

I wasn’t kidding about the giant guitar

How to get a copy

Grab your copy today in the UK from WHSmith, Sainsbury’s, Asda, and Tesco. Copies will be arriving very soon in US stores, including Barnes & Noble and Micro Center. You can also get the new issue online from our store, or digitally via our Android or iOS app. And don’t forget, there’s always the free PDF as well.

Subscribe for free goodies

Some of you have asked me about the goodies that we give out to subscribers. This is how it works: if you take out a twelve-month print subscription of The MagPi, you’ll get a Pi Zero W, Pi Zero case, and adapter cables absolutely free! This offer does not currently have an end date.

Alright, I think I’ve covered everything! So that’s it. I’ll see you next month.

Jean-Luc Picard sitting at a desk playing with a pen and sighing

The post MagPi 60: the ultimate troubleshooting guide appeared first on Raspberry Pi.

Burner laptops for DEF CON

Post Syndicated from Robert Graham original http://blog.erratasec.com/2017/07/burner-laptops-for-def-con.html

Hacker summer camp (Defcon, Blackhat, BSidesLV) is upon us, so I thought I’d write up some quick notes about bringing a “burner” laptop. Chrome is your best choice in terms of security, but I need Windows/Linux tools, so I got a Windows laptop.

I chose the Asus e200ha for $199 from Amazon with free (and fast) shipping. There are similar notebooks with roughly the same hardware and price from other manufacturers (HP, Dell, etc.), so I’m not sure how this compares against those other ones. However, it fits my needs as a “burner” laptop, namely:

  • cheap
  • lasts 10 hours easily on battery
  • weighs 2.2 pounds (1 kilogram)
  • 11.6 inch and thin

Some other specs are:

  • 4 gigs of RAM
  • 32 gigs of eMMC flash memory
  • quad core 1.44 GHz Intel Atom CPU
  • Windows 10
  • free Microsoft Office 365 for one year
  • good, large keyboard
  • good, large touchpad
  • USB 3.0
  • microSD
  • WiFi ac
  • no fans, completely silent

There are compromises, of course.

  • The Atom CPU is slow, thought it’s only noticeable when churning through heavy webpages. Adblocking addons or Brave are a necessity. Most things are usably fast, such as using Microsoft Word.
  • Crappy sound and video, though VLC does a fine job playing movies with headphones on the airplane. Using in bright sunlight will be difficult.
  • micro-HDMI, keep in mind if intending to do presos from it, you’ll need an HDMI adapter
  • It has limited storage, 32gigs in theory, about half that usable.
  • Does special Windows 10 compressed install that you can’t actually upgrade without a completely new install. It doesn’t have the latest Windows 10 Creators update. I lost a gig thinking I could compress system files.

Copying files across the 802.11ac WiFi to the disk was quite fast, several hundred megabits-per-second. The eMMC isn’t as fast as an SSD, but its a lot faster than typical SD card speeds.

The first thing I did once I got the notebook was to install the free VeraCrypt full disk encryption. The CPU has AES acceleration, so it’s fast. There is a problem with the keyboard driver during boot that makes it really hard to enter long passwords — you have to carefully type one key at a time to prevent extra keystrokes from being entered.

You can’t really install Linux on this computer, but you can use virtual machines. I installed VirtualBox and downloaded the Kali VM. I had some problems attaching USB devices to the VM. First of all, VirtualBox requires a separate downloaded extension to get USB working. Second, it conflicts with USBpcap that I installed for Wireshark.

It comes with one year of free Office 365. Obviously, Microsoft is hoping to hook the user into a longer term commitment, but in practice next year at this time I’d get another burner $200 laptop rather than spend $99 on extending the Office 365 license.

Let’s talk about the CPU. It’s Intel’s “Atom” processor, not their mainstream (Core i3 etc.) processor. Even though it has roughly the same GHz as the processor in a 11inch MacBook Air and twice the cores, it’s noticeably and painfully slower. This is especially noticeable on ad-heavy web pages, while other things seem to work just fine. It has hardware acceleration for most video formats, though I had trouble getting Netflix to work.

The tradeoff for a slow CPU is phenomenal battery life. It seems to last forever on battery. It’s really pretty cool.


A Chromebook is likely more secure, but for my needs, this $200 is perfect.

Assert() in the hands of bad coders

Post Syndicated from Robert Graham original http://blog.erratasec.com/2017/03/assert-in-hands-of-bad-coders.html

Using assert() creates better code, as programmers double-check assumptions. But only if used correctly. Unfortunately, bad programmers tend to use them badly, making code worse than if no asserts were used at all. They are a nuanced concept that most programmers don’t really understand.

We saw this recently with the crash of “Bitcoin Unlimited”, a version of Bitcoin that allows more transactions. They used an assert() to check the validity of input, and when they received bad input, most of the nodes in the network crashed.

The Bitcoin Classic/Unlimited code is full of bad uses of assert. The following examples are all from the file main.cpp.

Example #1this line of code:

  1.     if (nPos >= coins->vout.size() || coins->vout[nPos].IsNull())
  2.         assert(false); 

This use of assert is silly. The code should look like this:

  1.     assert(nPos < coins->vout.size());
  2.     assert(!coins->vout[nPos].IsNull());

This is the least of their problems. It understandable that as code ages, and things are added/changed, that odd looking code like this appears. But still, it’s an example of wrong thinking about asserts. Among the problems this would cause is that if asserts were ever turned off, you’d have to deal with dead code elimination warnings in static analyzers.

Example #2line of code:

  1.     assert(view.Flush());

The code within assert is supposed to only read values, not change values. In this example, the Flush function changes things. Normally, asserts are only compiled into debug versions of the code, and removed for release versions. However, doing so for Bitcoin will cause the program to behave incorrectly, as things like the Flush() function are no longer called. That’s why they put at the top of this code, to inform people that debug must be left on.

  1. #if defined(NDEBUG)
  2. # error “Bitcoin cannot be compiled without assertions.”
  3. #endif

Example #3: line of code:

  1.     CBlockIndex* pindexNew = new CBlockIndex(block);
  2.     assert(pindexNew);

The new operator never returns NULL, but throws its own exception instead. Not only is this a misconception about what new does, it’s also a misconception about assert. The assert is supposed to check for bad code, not check errors.

Example #4: line of code

  1.     BlockMap::iterator mi = mapBlockIndex.find(inv.hash);
  2.     CBlock block;
  3.     const Consensus::Params& consensusParams = Params().GetConsensus();
  4.     if (!ReadBlockFromDisk(block, (*mi).second, consensusParams))
  5.         assert(!“cannot load block from disk”);

This is the feature that crashed Bitcoin Unlimited, and would also crash main Bitcoin nodes that use the “XTHIN” feature. The problem comes from parsing input (inv.hash). If the parsed input is bad, then the block won’t exist on the disk, and the assert will fail, and the program will crash.

Again, assert is for checking for bad code that leads to impossible conditions, not checking errors in input, or checking errors in system functions.


The above examples were taken from only one file in the Bitcoin Classic source code. They demonstrate the typically wrong ways bad programmers use asserts. It’d be a great example to show students of programming how not to write bad code.

More generally, though, it shows why there’s a difference between 1x and 10x programmers. 1x programmers, like those writing Bitcoin code, make the typical mistake of treating assert() as error checking. The nuance of assert is lost on them.

Updated to reflect that I’m refering to the “Bitcoin Classic” source code, which isn’t the “Bitcoin Core” source code. However, all the problems above appear to also be problems in the Bitcoin Core source code.

Create Tables in Amazon Athena from Nested JSON and Mappings Using JSONSerDe

Post Syndicated from Rick Wiggins original https://aws.amazon.com/blogs/big-data/create-tables-in-amazon-athena-from-nested-json-and-mappings-using-jsonserde/

Most systems use Java Script Object Notation (JSON) to log event information. Although it’s efficient and flexible, deriving information from JSON is difficult.

In this post, you will use the tightly coupled integration of Amazon Kinesis Firehose for log delivery, Amazon S3 for log storage, and Amazon Athena with JSONSerDe to run SQL queries against these logs without the need for data transformation or insertion into a database. It’s done in a completely serverless way. There’s no need to provision any compute.

Amazon SES provides highly detailed logs for every message that travels through the service and, with SES event publishing, makes them available through Firehose. However, parsing detailed logs for trends or compliance data would require a significant investment in infrastructure and development time. Athena is a boon to these data seekers because it can query this dataset at rest, in its native format, with zero code or architecture. On top of that, it uses largely native SQL queries and syntax.

Walkthrough: Establishing a dataset

We start with a dataset of an SES send event that looks like this:

	"eventType": "Send",
	"mail": {
		"timestamp": "2017-01-18T18:08:44.830Z",
		"source": "[email protected]",
		"sourceArn": "arn:aws:ses:us-west-2:111222333:identity/[email protected]",
		"sendingAccountId": "111222333",
		"messageId": "01010159b2c4471e-fc6e26e2-af14-4f28-b814-69e488740023-000000",
		"destination": ["[email protected]"],
		"headersTruncated": false,
		"headers": [{
				"name": "From",
				"value": "[email protected]"
			}, {
				"name": "To",
				"value": "[email protected]"
			}, {
				"name": "Subject",
				"value": "Bounced Like a Bad Check"
			}, {
				"name": "MIME-Version",
				"value": "1.0"
			}, {
				"name": "Content-Type",
				"value": "text/plain; charset=UTF-8"
			}, {
				"name": "Content-Transfer-Encoding",
				"value": "7bit"
		"commonHeaders": {
			"from": ["[email protected]"],
			"to": ["[email protected]"],
			"messageId": "01010159b2c4471e-fc6e26e2-af14-4f28-b814-69e488740023-000000",
			"subject": "Test"
		"tags": {
			"ses:configuration-set": ["Firehose"],
			"ses:source-ip": [""],
			"ses:from-domain": ["amazon.com"],
			"ses:caller-identity": ["root"]
	"send": {}

This dataset contains a lot of valuable information about this SES interaction. There are thousands of datasets in the same format to parse for insights. Getting this data is straightforward.

1. Create a configuration set in the SES console or CLI that uses a Firehose delivery stream to send and store logs in S3 in near real-time.

2. Use SES to send a few test emails. Be sure to define your new configuration set during the send.

To do this, when you create your message in the SES console, choose More options. This will display more fields, including one for Configuration Set.
You can also use your SES verified identity and the AWS CLI to send messages to the mailbox simulator addresses.

$ aws ses send-email --to [email protected] --from [email protected] --subject "Bounced Like a Bad Check" --text "This should bounce" --configuration-set-name Firehose

3. Select your S3 bucket to see that logs are being created.

Walkthrough: Querying with Athena

Amazon Athena is an interactive query service that makes it easy to use standard SQL to analyze data resting in Amazon S3. Athena requires no servers, so there is no infrastructure to manage. You pay only for the queries you run. This makes it perfect for a variety of standard data formats, including CSV, JSON, ORC, and Parquet.

You now need to supply Athena with information about your data and define the schema for your logs with a Hive-compliant DDL statement. Athena uses Presto, a distributed SQL engine, to run queries. It also uses Apache Hive DDL syntax to create, drop, and alter tables and partitions. Athena uses an approach known as schema-on-read, which allows you to use this schema at the time you execute the query. Essentially, you are going to be creating a mapping for each field in the log to a corresponding column in your results.

If you are familiar with Apache Hive, you might find creating tables on Athena to be pretty similar. You can create tables by writing the DDL statement in the query editor or by using the wizard or JDBC driver. An important part of this table creation is the SerDe, a short name for “Serializer and Deserializer.” Because your data is in JSON format, you will be using org.openx.data.jsonserde.JsonSerDe, natively supported by Athena, to help you parse the data. Along the way, you will address two common problems with Hive/Presto and JSON datasets:

  • Nested or multi-level JSON.
  • Forbidden characters (handled with mappings).

In the Athena Query Editor, use the following DDL statement to create your first Athena table. For  LOCATION, use the path to the S3 bucket for your logs:

  eventType string,
  mail struct<`timestamp`:string,
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'

In this DDL statement, you are declaring each of the fields in the JSON dataset along with its Presto data type. You are using Hive collection data types like Array and Struct to set up groups of objects.

Walkthrough: Nested JSON

Defining the mail key is interesting because the JSON inside is nested three levels deep. In the example, you are creating a top-level struct called mail which has several other keys nested inside. This includes fields like messageId and destination at the second level. You can also see that the field timestamp is surrounded by the backtick (`) character. timestamp is also a reserved Presto data type so you should use backticks here to allow the creation of a column of the same name without confusing the table creation command. On the third level is the data for headers. It contains a group of entries in name:value pairs. You define this as an array with the structure of <name:string,value:string> defining your schema expectations here. You must enclose `from` in the commonHeaders struct with backticks to allow this reserved word column creation.

Now that you have created your table, you can fire off some queries!

SELECT * FROM sesblog limit 10;

This output shows your two top-level columns (eventType and mail) but this isn’t useful except to tell you there is data being queried. You can use some nested notation to build more relevant queries to target data you care about.

“Which messages did I bounce from Monday’s campaign?”

SELECT eventtype as Event,
       mail.destination as Destination, 
       mail.messageId as MessageID,
       mail.timestamp as Timestamp
FROM sesblog
WHERE eventType = 'Bounce' and mail.timestamp like '2017-01-09%'

“How many messages have I bounced to a specific domain?”

SELECT COUNT(*) as Bounces 
FROM sesblog
WHERE eventType = 'Bounce' and mail.destination like '%amazonses.com%'

“Which messages did I bounce to the domain amazonses.com?”

SELECT eventtype as Event,
       mail.destination as Destination, 
       mail.messageId as MessageID 
FROM sesblog
WHERE eventType = 'Bounce' and mail.destination like '%amazonses.com%'

There are much deeper queries that can be written from this dataset to find the data relevant to your use case. You might have noticed that your table creation did not specify a schema for the tags section of the JSON event. You’ll do that next.

Walkthrough: Handling forbidden characters with mappings

Here is a major roadblock you might encounter during the initial creation of the DDL to handle this dataset: you have little control over the data format provided in the logs and Hive uses the colon (:) character for the very important job of defining data types. You need to give the JSONSerDe a way to parse these key fields in the tags section of your event. This is some of the most crucial data in an auditing and security use case because it can help you determine who was responsible for a message creation.

In the Athena query editor, use the following DDL statement to create your second Athena table. For LOCATION, use the path to the S3 bucket for your logs:

  eventType string,
  mail struct<`timestamp`:string,
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'

In your new table creation, you have added a section for SERDEPROPERTIES. This allows you to give the SerDe some additional information about your dataset. For your dataset, you are using the mapping property to work around your data containing a column name with a colon smack in the middle of it. ses:configuration-set would be interpreted as a column named ses with the datatype of configuration-set. Unlike your earlier implementation, you can’t surround an operator like that with backticks. The JSON SERDEPROPERTIES mapping section allows you to account for any illegal characters in your data by remapping the fields during the table’s creation.

For example, you have simply defined that the column in the ses data known as ses:configuration-set will now be known to Athena and your queries as ses_configurationset. This mapping doesn’t do anything to the source data in S3. This is a Hive concept only. It won’t alter your existing data. You have set up mappings in the Properties section for the four fields in your dataset (changing all instances of colon to the better-supported underscore) and in your table creation you have used those new mapping names in the creation of the tags struct.

Now that you have access to these additional authentication and auditing fields, your queries can answer some more questions.

“Who is creating all of these bounced messages?”

SELECT eventtype as Event,
         mail.timestamp as Timestamp,
         mail.tags.ses_source_ip as SourceIP,
         mail.tags.ses_caller_identity as AuthenticatedBy,
         mail.commonHeaders."from" as FromAddress,
         mail.commonHeaders.to as ToAddress
FROM sesblog2
WHERE eventtype = 'Bounce'

Of special note here is the handling of the column mail.commonHeaders.”from”. Because from is a reserved operational word in Presto, surround it in quotation marks (“) to keep it from being interpreted as an action.

Walkthrough: Querying using SES custom tagging

What makes this mail.tags section so special is that SES will let you add your own custom tags to your outbound messages. Now you can label messages with tags that are important to you, and use Athena to report on those tags. For example, if you wanted to add a Campaign tag to track a marketing campaign, you could use the –tags flag to send a message from the SES CLI:

$ aws ses send-email --to [email protected] --from [email protected] --subject "Perfume Campaign Test" --text "Buy our Smells" --configuration-set-name Firehose --tags Name=Campaign,Value=Perfume

This results in a new entry in your dataset that includes your custom tag.

		"tags": {
			"ses:configuration-set": ["Firehose"],
			"Campaign": ["Perfume"],
			"ses:source-ip": [""],
			"ses:from-domain": ["amazon.com"],
			"ses:caller-identity": ["root"],
			"ses:outgoing-ip": [""]

You can then create a third table to account for the Campaign tagging.

  eventType string,
  mail struct<`timestamp`:string,
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'

Then you can use this custom value to begin to query which you can define on each outbound email.

SELECT eventtype as Event,
       mail.destination as Destination, 
       mail.messageId as MessageID,
       mail.tags.Campaign as Campaign
FROM sesblog3
where mail.tags.Campaign like '%Perfume%'


Walkthrough: Building your own DDL programmatically with hive-json-schema

In all of these examples, your table creation statements were based on a single SES interaction type, send. SES has other interaction types like delivery, complaint, and bounce, all which have some additional fields. I’ll leave you with this, a master DDL that can parse all the different SES eventTypes and can create one table where you can begin querying your data.

Building a properly working JSONSerDe DLL by hand is tedious and a bit error-prone, so this time around you’ll be using an open source tool commonly used by AWS Support. All you have to do manually is set up your mappings for the unsupported SES columns that contain colons.

This sample JSON file contains all possible fields from across the SES eventTypes. It has been run through hive-json-schema, which is a great starting point to build nested JSON DDLs.

Here is the resulting “master” DDL to query all types of SES logs:

  eventType string,
  complaint struct<arrivaldate:string, 
  bounce struct<bouncedrecipients:array<struct<action:string, diagnosticcode:string, emailaddress:string, status:string>>,
  mail struct<`timestamp`:string,
  send string,
  delivery struct<processingtimemillis:int,
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'


In this post, you’ve seen how to use Amazon Athena in real-world use cases to query the JSON used in AWS service logs. Some of these use cases can be operational like bounce and complaint handling. Others report on trends and marketing data like querying deliveries from a campaign. Still others provide audit and security like answering the question, which machine or user is sending all of these messages? You’ve also seen how to handle both nested JSON and SerDe mappings so that you can use your dataset in its native format without making changes to the data to get your queries running.

With the new AWS QuickSight suite of tools, you also now have a data source that that can be used to build dashboards. This makes reporting on this data even easier. For information about using Athena as a QuickSight data source, see this blog post.

There are also optimizations you can make to these tables to increase query performance or to set up partitions to query only the data you need and restrict the amount of data scanned. If you only need to report on data for a finite amount of time, you could optionally set up S3 lifecycle configuration to transition old data to Amazon Glacier or to delete it altogether.

Feel free to leave questions or suggestions in the comments.



About the  Author

rick_wiggins_100Rick Wiggins is a Cloud Support Engineer for AWS Premium Support. He works with our customers to build solutions for Email, Storage and Content Delivery, helping them spend more time on their business and less time on infrastructure. In his spare time, he enjoys traveling the world with his family and volunteering at his children’s school teaching lessons in Computer Science and STEM.




Migrate External Table Definitions from a Hive Metastore to Amazon Athena









Introducing PIXEL

Post Syndicated from Simon Long original https://www.raspberrypi.org/blog/introducing-pixel/

It was just over two years ago when I walked into Pi Towers for the first time. I only had the vaguest idea of what I was going to be doing, but on the first day Eben and I sat down and played with the Raspbian desktop for half an hour, then he asked me “do you think you can make it better?”


Bear in mind that at this point I’d barely ever used Linux or Xwindows, never mind made any changes to them, so when I answered “hmmm – I think so”, it was with rather more confidence than I actually felt. It was obvious that there was a lot that could be done in terms of making it a better experience for the user, and I spent many years working in user interface design in previous jobs. But I had no idea where to start in terms of changing Raspbian. I clearly had a bit of a learning curve in front of me…

Well, that was two years ago, and I’ve learnt an awful lot since then. It’s actually surprisingly easy to hack about with the LXDE desktop once you get your head around what all the bits do, and since then I’ve been slowly chipping away at the bits that I felt would most benefit from tweaking. Stuff has slowly been becoming more and more like my original concept for the desktop; with the latest changes, I think the desktop has reached the point where it’s a complete product in its own right and should have its own name. So today, we’re announcing the release of the PIXEL desktop, which will ship with the Foundation’s Raspbian image from now on.



One of the things I said (at least partly in jest) to my colleagues in those first few weeks was that I’d quite like to rename the desktop environment once it was a bit more Pi-specific, and I had the name “pixel” in my mind about two weeks in. It was a nice reminder of my days learning to program in BASIC on the Sinclair ZX81; nowadays, everything from your TV to your phone has pixels on it, but back then it was a uniquely “computer-y” word and concept. I also like crosswords and word games, and once it occurred to me that “pixel” could be made up from the initials of words like Pi and Xwindows, the name stuck in my head and never quite went away. So PIXEL it is, which now officially stands for “Pi Improved Xwindows Environment, Lightweight”.

What’s new?

The latest set of changes are almost entirely to do with the appearance of the desktop; there are some functional changes and a few new applications, about which more below, but this is mostly about making things look nicer.

The first thing you’ll notice on rebooting is that the trail of cryptic boot messages has (mostly) gone, replaced by a splash screen. One feature which has frequently been requested is an obvious version number for our Raspbian image, and this can now be seen at the bottom-right of the splash image. We’ll update this whenever we release a new version of the image, so it should hopefully be slightly easier to know exactly what version you’re running in future.


I should mention that the code for the splash screen has been carefully written and tested, and should not slow down the Pi’s boot process; the time to go from powering on to the desktop appearing is identical, whether the splash is shown or not.

Desktop pictures

Once the desktop appears, the first thing you’ll notice is the rather stunning background image. We’re very fortunate in that Greg Annandale, one of the Foundation’s developers, is also a very talented (and very well-travelled) photographer, and he has kindly allowed us to use some of his work as desktop pictures for PIXEL. There are 16 images to choose from; you can find them in /usr/share/pixel-wallpaper/, and you can use the Appearance Settings application to choose which one you prefer. Do have a look through them, as Greg’s work is well worth seeing! If you’re curious, the EXIF data in each image will tell you where it was taken.





You’ll also notice that the icons on the taskbar, menu, and file manager have had a makeover. Sam Alder and Alex Carter, the guys responsible for all the cartoons and graphics you see on our website, have been sweating blood over these for the last few months, with Eben providing a watchful eye to make sure every pixel was exactly the right colour! We wanted something that looked businesslike enough to be appropriate for those people who use the Pi desktop for serious work, but with just a touch of playfulness, and Sam and Alex did a great job. (Some of the icons you don’t see immediately are even nicer; it’s almost worth installing some education or engineering applications just so those categories appear in the menu…)


Speaking of icons, the default is now not to show icons in individual application menus. These always made menus look a bit crowded, and didn’t really offer any improvement in usability, not least because it wasn’t always that obvious what the icon was supposed to represent… The menus look cleaner and more readable as a result, since the lack of visual clutter now makes them easier to use.

Finally on the subject of icons, in the past if your Pi was working particularly hard, you might have noticed some yellow and red squares appearing in the top-right corner of the screen, which were indications of overtemperature or undervoltage. These have now been replaced with some new symbols that make it a bit more obvious what’s actually happening; there’s a lightning bolt for undervoltage, and a thermometer for overtemperature.


If you open a window, you’ll see that the window frame design has now changed significantly. The old window design always looked a bit dated compared to what Apple and Microsoft are now shipping, so I was keen to update it. Windows now have a subtle curve on the corners, a cleaner title bar with new close / minimise / maximise icons, and a much thinner frame. One reason the frame was quite thick on the old windows was so that the grab handles for resizing were big enough to find with the mouse. To avoid this problem, the grab handles now extend slightly outside the window; if you hold the mouse pointer just outside the window which has focus, you’ll see the pointer change to show the handle.



Steve Jobs said that one thing he was insistent on about the Macintosh was that its typography was good, and it’s true that using the right fonts makes a big difference. We’ve been using the Roboto font in the desktop for the last couple of years; it’s a nice-looking modern font, and it hasn’t changed for this release. However, we have made it look better in PIXEL by including the Infinality font rendering package. This is a library of tweaks and customisations that optimises how fonts are mapped to pixels on the screen; the effect is quite subtle, but it does give a noticeable improvement in some places.


Most people have their Pi set up to automatically log in when the desktop starts, as this is the default setting for a new install. For those who prefer to log in manually each time, the login screen has been redesigned to visually match the rest of the desktop; you now see the login box (known as the “greeter”) over your chosen desktop design, with a seamless transition from greeter to desktop.


Wireless power switching

One request we have had in the past is to be able to shut off WiFi and/or Bluetooth completely, particularly on Pi 3. There are now options in the WiFi and Bluetooth menus to turn off the relevant devices. These work on the Pi 3’s onboard wireless hardware; they should also work on most external WiFi and Bluetooth dongles.

You can also now disconnect from an associated wireless access point by clicking on its entry in the WiFi menu.

New applications

There are a couple of new applications now included in the image.

RealVNC have ported their VNC server and viewer applications to Pi, and they are now integrated with the system. To enable the server, select the option on the Interfaces tab in Raspberry Pi Configuration; you’ll see the VNC menu appear on the taskbar, and you can then log in to your Pi and control it remotely from a VNC viewer.

The RealVNC viewer is also included – you can find it from the Internet section of the Applications menu – and it allows you to control other RealVNC clients, including other Pis. Have a look here on RealVNC’s site for more information.


Please note that if you already use xrdp to remotely access your Pi, this conflicts with the RealVNC server, so you shouldn’t install both at once. If you’re updating an existing image, don’t run the sudo apt-get install realvnc-vnc-server line in the instructions below. If you want to use xrdp on a clean image, first uninstall the RealVNC server with sudo apt-get purge realvnc-vnc-server before installing xrdp. (If the above paragraph means nothing to you, then you probably aren’t using xrdp, so you don’t have to worry about any of it!)

Also included is the new SenseHAT emulator, which was described in a blog post a couple of weeks ago; have a look here for all the details.



There are updates for a number of the built-in applications; these are mostly tweaks and bug fixes, but there have been improvements made to Scratch and Node-RED.

One more thing…

We’ve been shipping the Epiphany web browser for the last couple of years, but it’s now starting to show its age. So for this release (and with many thanks to Gustav Hansen from the forums for his invaluable help with this), we’re including an initial release of Chromium for the Pi. This uses the Pi’s hardware to accelerate playback of streaming video content.


We’ve preinstalled a couple of extensions; the uBlock Origin adblocker should hopefully keep intrusive adverts from slowing down your browsing experience, and the h264ify extension forces YouTube to serve videos in a format which can be accelerated by the Pi’s hardware.

Chromium is a much more demanding piece of software than Epiphany, but it runs well on Pi 2 and Pi 3; it can struggle slightly on the Pi 1 and Pi Zero, but it’s still usable. (Epiphany is still installed in case you find it useful; launch it from the command line by typing “epiphany-browser”.)

How do I get it?

The Raspbian + PIXEL image is available from the Downloads page on our website now.

To update an existing Jessie image, type the following at the command line:

sudo apt-get update
sudo apt-get dist-upgrade
sudo apt-get install -y rpi-chromium-mods
sudo apt-get install -y python-sense-emu python3-sense-emu
sudo apt-get install -y python-sense-emu-doc realvnc-vnc-viewer

and then reboot.

If you don’t use xrdp and would like to use the RealVNC server to remotely access your Pi, type the following:

sudo apt-get install -y realvnc-vnc-server

As always, your feedback on the new release is very welcome; feel free to let us know what you think in the comments or on the forums.

The post Introducing PIXEL appeared first on Raspberry Pi.

I entered Ludum Dare 36

Post Syndicated from Eevee original https://eev.ee/blog/2016/08/29/i-entered-ludum-dare-36/

Short story: I made a video game again! This time it was for Ludum Dare, a game jam with some tight rules: solo only, 48 hours to make the game and all its (non-code) assets.

(This is called the “Compo”; there’s also a 72-hour “Jam” which is much more chill, but I did hard mode. Usually there’s a ratings round, but not this time, for reasons.)

I used the PICO-8 again, so you can play it on the web as long as you have a keyboard. It’s also on Ludum Dare, and in splore, and here’s the cartridge too.

Isaac's Descent

But wait! Read on a bit first.


I’ve never entered a game jam before, and I slightly regretted that I missed a PICO-8 jam that was happening while I was making Under Construction. I’ve certainly never made a game in 48 hours, so that seemed exciting.

More specifically, I have some trouble with shaking ideas loose. I don’t know a more specific word than “idea” for this, but I mean creative, narrative ideas: worldbuilding, characters, events, gameplay mechanics, and the like. They have a different texture from “how could I solve this technical problem” ideas or “what should I work on today” ideas.

I’ll often have an idea or two, maybe a theme I want to move towards, and then hit a wall. I can’t think of any more concepts; I can’t find any way to connect the handful I have. I end up shelving the idea, sometimes indefinitely. This has been particularly haunting with my interactive fiction game in progress, Runed Awakening, which by its very nature is nothing but narrative ideas.

My true goal for entering Ludum Dare was to jiggle the idea faucet and maybe loosen it a bit. Nothing’s quite as motivating as an extreme time limit. I went in without anything in mind; I didn’t even know it was coming up until two days beforehand. (The start time is softly enforced by the announcement of a theme, anyway.) I knew it would probably resemble a platformer, since I already had the code available to make that work, but that was about it.

I already wrote about the approach to making our last game, so I can’t very well just do that again. Instead, here’s something a little different: I took regular notes on the state of the game (and myself), all weekend. You can see exactly how it came together, almost hour by hour. Is that interesting? I think it’s interesting.

I don’t know if this is a better read if you play the game first or last. Maybe both?

There’s also a surprise at the very end, as a reward for reading through it all! No, wait, stop, you can’t just scroll down, that’s cheating—



09:00 — Already nervous. Registered for the site yesterday; voted on the themes today; jam actually starts tomorrow. I have no idea if I can do this. What a great start.


09:00 — Even more nervous. Last night I started getting drowsy around 5pm, I guess because my sleep is still a bit weird. So not only do I only have 48 hours, but by the looks of things, I’ll be spending half that time asleep.

17:00 — I can’t even sit still and do anything for the next hour; I’m too antsy about getting started.

START!! 18:00 — Theme revealed: “Ancient Technology”. I have no ideas.

Well, no, hang on. Shortly before the theme was announced, I had a brief Twitter conversation that shook something loose. I’d mentioned that I rarely seem to have enough ideas to fill a game. Someone accidentally teased out of me that it’s more specific than that: I have trouble coming up with ideas that appeal to me, that satisfy me in the way I really like in games and stories. In retrospect, I probably have a bad habit of rejecting ideas by reflex before I even have a chance to think about them and turn them into something more inspiring.

The same person also asked how I want games to feel, and of course, that’s what I should be keeping front and center, before even worrying about genre or mechanics or anything. How does this feel, and how does it make me feel? I know that’s important, but I’m not in the habit of thinking about it.

With that in mind, how does “ancient technology” make me feel?

It reminds me immediately of two things: Indiana Jones-esque temples, full of centuries-old mechanisms and unseen triggers that somehow still work perfectly; and also Stargate, where a race literally called “Ancients” made preposterously advanced devices with such a sleek and minimalist design that they might as well have been magic.

The common thread is a sense of, hm, “soft wonder”? You’re never quite sure what’s around the next corner, but it won’t be a huge surprise, just a new curiosity. There’s the impression of a coherent set of rules somewhere behind the scenes, but you never get to see it, and it doesn’t matter that much anyway. You catch a glimpse of what’s left behind, and half your astonishment is that it’s still here at all.

Also, I bet I can make a puzzle-platformer out of this.

18:20 — Okay, well! I have a character Isaac (stolen from Glip, ahem) who exists in Runed Awakening but otherwise has never seen any real use. I might as well use them now, which means this game is also set somewhere in Flora.

I’ve drawn a two-frame walking animation and saved it as isaac.p8 for now. It’s enough to get started. I’m gonna copy/paste all the engine gunk from my unfinished game, rainblob — it’s based on what was in Under Construction, with some minor cleanups and enhancements.

19:00 — I’m struggling a little bit here, because Isaac is two tiles tall, and I never got around to writing real support for actors that are bigger than a single tile. Most of the sprite drawing is now wrapped in a little sprite type, so I don’t think this will be too bad — I almost have it working, except that it doesn’t run yet.

19:07 — Success! Apparently I was closer than I thought. The solution is a bit of a hack: instead of a list of tiles (as animation frames), Isaac has a list of lists of tiles, where each outer list is the animation for one grid space. It required some type-checking to keep the common case working (boo), and it blindly assumes any multi-tile actor is a 1×n rectangle. It’s fine. Whatever. I’ll fix it if I really need to.

19:16 — I drew and placed some cave floor tiles. Isaac can no longer walk left or jump. I am not sure why. I really, really hope it’s not another collision bug. The collision function has been such a nightmare. Is it choking on a moving object that’s more than a tile tall?

19:20 — I have been asked to put a new bag in the trash can. This is wildly unjust. I do not have time for such trivialities. But I have to pee anyway, so it’s okay — I’ll batch these two standing-up activities together to save time. Speed strats.

19:28 — The left/jump thing seems to be a bug with the PICO-8; the button presses don’t register at all. Restarting the “console” fixed it. This is ominous; I hope a mysterious heisenbug doesn’t plague me for the next 46½ hours.

19:51 — Isaac is a wizard. Surely, they should be able to cast spells or whatever. Teeny problem: the PICO-8 only has two buttons, and I need one of them for jumping. (Under Construction uses up for jump, but I’ve seen several impassioned pleas against doing that because it makes using a real d-pad very awkward, and after using the pocketCHIP I’m inclined to agree.)

New plan, then: you have an inventory. Up and down scroll through it, and the spare button means “use the selected item”. Accordingly, I’ve put a little “selected item” indicator in the top left of the screen.

Isaac hasn’t seen too much real character development; it’s hard to develop a character without actually putting them in something. Their backstory thusfar isn’t really important for this game, but I did have the idea that they travel with a staff that can create a reflective bubble. That’s interesting, because it suggests that Isaac prefers to operate defensively. I made a staff sprite and put it in the starting inventory, but I’m not quite sure what to do with it yet; I don’t know how the bubble idea would work in a tiny game.

20:01 — As a proof of concept, I made the staff shoot out particles when you use it. The particle system is from rainblob, and is pretty neat — they’re just dumb actors that draw themselves as a single pixel.

I bound the X button to “use”. Should jumping be X or O? I’m not sure, hm. My Nintendo instincts tell me the right button is for jumping, but on a keyboard, the “d-pad” and buttons are reversed.

20:04 — I realize I added a sound effect for jumping, then accientally overwrote the code that plays it. Oops; fixing that. Good thing I didn’t overwrite the sound! This is what I get for trying to edit the assets in the PICO-8 and the code in vim, when it’s all stored in a single file.

20:37 — I have a printat function (from Under Construction) which prints text to the screen with a given horizontal and vertical alignment. It needs to know the width of text to do this, which is easy enough: the PICO-8 font is fixed-width. Alas! The latest PICO-8 release added characters to represent the controller buttons, and I’d really like to use them, but they’re double-wide. Hacking around this is proving a bit awkward, especially since there’s no ord() function available oh my god.

20:50 — Okay, done. The point of that was: I rigged a little hint that tells you what button to press to jump. When you approach the first ledge, Isaac sprouts a tiny thought bubble with the O button symbol in it. PICO-8 games tend not to explain themselves (something that has frustrated me more than once), so I think that’s nice. It’s the kind of tiny detail I love including in my work.

21:04 — I wrote a tiny fragment of music, but I really don’t know what I’m doing here, so… I don’t know.

I had the idea that there’d be runes carved in the back wall of this cave, so I made a sprite for that, though it’s basically unrecognizable at this size. I don’t know what reading them will do, yet.

I also made the staff draw a bubble (in the form of a circle around you) while you’re holding the “use” button down, via a cheap hack. Kinda just throwing stuff at the wall in the hopes that something will stick.

21:07 — I’ve decided to eat these chips while I ponder where to go from here.

21:22 — So, argh. Isaac’s staff is supposed to create a bubble that reflects magical attacks. The immediate problem there is that my collision assumes everything is a rectangle. I really don’t want to be rewriting collision with only a weekend to spend on this. I could make the bubble rectangular, but who’s ever heard of a rectangular magic bubble?

Maybe I could make this work, but it raises more questions: what magical attacks? What attacks you? Are there monsters? Do I have to write monster AI? Can Isaac die? I need to translate these scraps of thematics into game mechanics, somehow.

I try to remember to think about the feel. I want you to feel like you’re exploring an old cavern/temple/something, laden with traps meant to keep you out. I think that means death, and death means save points, and save points mean saving the game state, which I don’t have extant code for. Oof.

22:00 — Not much has changed; I started doodling sprites as a distraction. Still getting this thing where left and up stop working, what the hell.

22:05 — Actually, I’m getting tired; I should deal with the cat litter before it gets too late. Please hold.

22:59 — I wrote some saving, which doesn’t work yet. Almost, maybe. I do have a pretty cool death animation, though it looks a bit wonky in-game, because animations are on a global timer. Whoops! All of them have been really simple so far, so it hasn’t mattered, but this is something that really needs to start at the beginning and play through exactly once.

23:15 — Okay! I have a save, and I have death, and I even have some sound effects for them. The animation is still off, alas (and loops forever), and there’s no way to load after you die, but the basic cycle of this kind of game is coming together. If I can get a little more engine stuff working tomorrow, I should be able to build a little game. Goodnight for now.


07:48 — I’m. I’m up.

08:28 — Made the animation start when the player dies and stop after it’s played once. Also made the music stop immediately on death and touched up the sprites a bit. Still no loading, so death pretty much ends the game forever; that’s up next and should be easy enough. First, breakfast.

09:09 — The world is now restored after you die, and I fixed a few bugs as well. Cool beans.

09:14 — So, ah. That’s a decent start mechanically, but I need a little more concept, especially as it relates to the theme. I don’t expect this game to be particularly deep, what with its non-plot of “explore these caverns”, but I do want to explore the theme a bit. I want something that’s interesting to play, too, even if for only five minutes.

Isaac is a clever wizard. Canonically, he might be the cleverest wizard. What does his staff do?

What kind of traps would be in a place like this? Spikes, falling floors, puzzles? Monsters? Pressure plates?

What does Isaac’s staff do?

Hang on, let me approach this a much more sensible way: if I were going to explore a cavern like this, what would I want my staff to do?

09:59 — I’m still struggling with this question. I thought perhaps the cavern would only be the introductory part, and then you’d find a cool teleporter to a dusty sleek place that looked a lot more techy. I tried drawing some sleek bricks, but I can’t figure out how to get the aesthetic I want with the PICO-8’s palette. So I distracted myself by drawing some foreground tiles again. Whoops?

10:01 — I’d tweeted two GIFs of Isaac’s death while working on it, complete with joking melodramatic captions like “death has no power here”. I also lamented that I didn’t know yet what the game was about, to which someone jokingly replied that so far it seemed to be “about death”.

Aha. Maybe the power of Isaac’s staff is to create savepoints, and maybe some puzzles or items or whatever transcend death, sticking around after you respawn. I’ll work with that for a bit and see what falls out of it.

11:12 — Wow, I’ve been busy! The staff now creates savepoints, complete with a post-death menu, a sound effect, a flash (bless you, UC’s scenefader), a thought-bubble hint, and everything. It’s pretty great? And it fits perfectly: if you’re exploring a trap-laden cavern then you’d want some flavor of safety equipment with you, right? What’s safer than outright resurrection?

I can see some interesting puzzles coming out of this: you have to pick your savepoint carefully to interact with mechanisms in the right way, or you have to make sure you can kill yourself if you need to, since that’s the only way to hop back to a savepoint. And it’s a purely defensive ability, just as I wanted. And something impossibly cool and powerful but hilariously impractical seems extremely up Isaac’s alley, from what I know about them so far.

11:59 — Still busy, which is a good sign! I’ve been working on making some objects for Isaac to interact with in the world; so far I’ve focused on the runes on the wall, though I’m not quite sold on them yet. The entire game so far is that you have to make a save point, jump down a pit to use a thing that extends a bridge over the pit, then kill yourself to get back to the save point and cross the bridge. It’s very rough, but it’s finally looking like a game, which is really great to see.

12:28 — I finally got sick enough of left/up breaking that I sat down and tried every distinct action I could think of, one at a time, to figure out the cause. Turns out it was my drawing tablet, which I’d used a couple times to draw sprites? If the pen is close enough to even register as a pointer, left and up break. I know I’ve seen the tablet listed as a “joypad” in other SDL applications, so my best guess is that it’s somehow acting as an axis and confusing PICO-8? I can’t imagine why or how. Super, super weird, but at least now I know what the problem is.

14:28 — Uh, whoops. Somehow I spent two hours yelling on Twitter. I don’t know how that happened.

16:42 — Hey, what’s up. I’ve been working on music (with very mixed results) and fixing bugs. I’m still missing a lot of minor functionality — for example, resetting the room doesn’t actually clear the platforms, because resetting the map only asks actors to reset themselves, and the platforms are new actors who don’t know they should vanish. Oops.

Oh, I also have them appearing on a timer, which is cool. I want their appearance to be animated, too, but that’s tricky with the current approach of just drawing tiles directly on the map. I guess I could turn them into real actors that are always present but can appear and vanish, which would also fix the reset thing.

For now, it’s time to eat and swim, so I’ll get back to this later.

18:22 — I’m so fucked. Everything is a mess. The room still doesn’t reset correctly. The time is half up and I have almost one room so far.

I need to shift gears here: fix the bugs as quickly as I can, then focus on rooms.

20:05 — I fixed a bunch of reset bugs, but I’m getting increasingly agitated by how half-assed this engine is. It’s alright for what it is, I guess, but it clearly wasn’t designed for anything in particular, and I feel like I have to bolt features on haphazardly as I need them.

Anyway, I made progression work, kinda: when you touch the right side of the room, you move on to the next one. When you touch the right side of the final room, you win, and the game celebrates by crashing.

I made a little moving laser eye thing that kills you on contact, creating a cute puzzle where you just resurrect yourself as soon as it’s gone past you. Changed death so time keeps passing while the prompt is up, of course.

Now I have a whopping, what, three world objects? And one item you can use, the one you start with? And I’m not sure how to put these together into any more puzzles.

I made Isaac’s cloak flutter a bit while they walk. Cool.

20:31 — For lack of any better ideas, I added something I’d wanted since the beginning: Isaac’s color scheme is now chosen randomly at startup. They are a newt, you see.

21:07 — Did some cleanup and minor polishing, but still feeling blocked. Going to brainstorm with myself a bit.

What are some “ancient” mechanisms? Pressure plates; blowdarts; secret doors; hidden buttons; …?

Does Isaac get an improved resurrection ability later? Resurrect where you died? I don’t know how that would be especially useful unless you died on a moving platform, and I don’t have anything like that.

Other magical objects you find…?

Puzzle ideas? Set up a way to kill yourself so you can use it later? Currently there’s no way to interact with the world other than to add those platforms, so I don’t see how this would work. I also like “conflict” puzzles where two goals seem to depend on each other, but offhand I can’t think of anything along those lines besides the first room.

21:55 — I’ve built a third puzzle, which is just some slightly aggravating platforming, made a little less so by the ability to save your progress.

22:19 — I started on a large room marking the end of the cave sequence and the entrance to the sleek brick area. I made a few tiles and a sound effect for it, but I’m not quite sure how the puzzle will work. I want a bigger and more elaborate setup with some slight backtracking, and I want to give the player a new toy to play with, but I’m not sure what.

I’ll have to figure it out tomorrow.


08:49 — Uggh, I’m awake. Barely. I keep sleeping for only six hours or so, which sucks.

I think I want to start out by making a title screen and some sort of ending. Even if I only have three puzzles, a front and back cover will make it look much more like an actual game.

09:57 — I made a little title screen and wrote a simple ditty for it, which I might even almost like?

11:09 — Made a credits screen as well, which implies that there’s an actual ending. And there is! You get the Flurry, an enchanted rapier I thought of a little while ago. It’s not described in the game or even mentioned outside of the “credits”, in true 8-bit fashion.

Now I have a complete game no matter what, so I can focus on hammering out some levels without worrying too much about time.

I also fixed up the ingame music; it used to have some high notes come in on a separate track, in my clumsy attempts at corralling multiple instruments, but I think they destroyed the mood. Now it’s mostly those low notes and some light “bass”. It works as a loop now, too. Much better in every way.

The awkward-platforming room had a particularly tricky jump that turned out to be trickier than I thought — I suddenly couldn’t do it at all when trying to demo the game for Mel. At their suggestion, I made it a bit less terrible, though hopefully still tricky enough that it might need a second try.

13:05 — Hi! Wow! I’ve been super busy! I came up with a new puzzle involving leaving a save point in midair while dropping down a pit. Then I finally added a new item, mostly inspired by how easy it was to implement: a spellbook that makes you float but doesn’t let you jump, so you can only move back and forth horizontally until you turn it off. I also added a thought bubble for how to cycle through the inventory, some really cute sound effects for when you use the book, and an introductory puzzle for it. It’s coming along pretty nicely!

14:13 — Trying to design a good puzzle for the next area. I made a stone door object which can open and close, though the way it does so is pretty gross, and a wooden wheel that opens it. I really like the wheel; my first thought was to use a different color lever, but I wanted the doors to be reusable whereas the platform lever isn’t, and using the same type of mechanism seemed misleading.

I might be trying to cram too much into the same room at the moment? It introduces the spellbook and the doors/wheel, then makes you solve a puzzle using both. I might split this up and try to introduce both ideas separately.

I think around 16:00, I’m gonna stop making puzzle rooms (unless I still have an amazing idea) and focus on cleaning stuff up, fixing weird bugs, and maybe un-hacking some of these hacks.

15:19 — Someone asked if I streamed my dev process, and I realized that this would’ve been a perfect opportunity to do that, since everything happens within a single small box. Oops. I guess I’ll stream the last few hours, though now no one can watch without getting all he puzzle spoiled.

I made a separate room for getting the spellbook, plus another for introducing the stone doors. The pacing is much much better, and now there are more puzzles overall, which is nice.

15:54 — My puzzles seem to be pretty solid, and I’ve only got space for one more on the map, so I’m thinking about what I’d like it to be.

I want something else that combines mechanics, like, using the platforms to block a door from closing all the way. But a door and a platform can’t coexist on the same tile, so the door has to start out partially open. And… what happens if you summon the platform after closing the door all the way? Hm. I wish my physics were more thorough, but right now none of these objects interact with each other terribly well; the stone door in particular just kinda ignores anything in its way until it hits solid wall.

16:04 — Instead of all that, I fixed the animation on the wheel (it wasn’t playing at all?), gave it a sound effect that I love, and finally added an explicit way to control draw order. The savepoint rune had been drawing over the player since the very beginning, which had been bugging me all weekend. Now the player is always on top. Very glad I had sort lying around.

16::57 — I guess I’m done? I filled that last puzzle room with an interesting timing thing that uses the lever, wheel, runes, and floating, but there are a couple different ways to go about it, and one way is 1-cycle. It bugs me a little that the original setup I wanted (repeat the platforming, then discover it won’t get you all the way to the exit and have to rethink it) doesn’t work, but, there’s no reason you’d think to do it the fastest way the first time, and I think being able to notice that adds an extra “aha”. Gotta resist the urge to railroad!

(Editor’s note: I later fixed a bug that removed the 1-cycle solution.)

I’ll call this done and let people playtest it, once I make it fit within the compressed size limit.

17:08 — God, fuck the compressed size limit. I started at 20538; I deleted all the debug and unused stuff inherited from rainblob and UC, and now I’m at 18491. The limit is 15360. God dammit. I don’t want to have to strip all the comments again.

17:39 — I ended up deleting all the comments again. Oh, well. I ran through it from start to finish once, and all seems good! The game is done and online, and all that’s left is figuring out how to put it on the LD website.

18:46 — Time is up, but this is “submission hour” and the rules allow fixing minor bugs, so I fixed a few things people have pointed out:

  • Two obvious places you could get stuck now have spikes. You can reset the room from the menu, but I’m pretty sure nobody noticed the “enter = menu” on the title screen, and a few people have thought they had to reset the entire game.

  • The last spike pit in the spellbook room required you to walk through spikes, which wasn’t what I intended and looks fatal, even though it’s not. The intention was for it to be an exact replica of the previous pit, except that you have to float across it from a tile higher; this solution now works.

  • One of those half-rock-brick tiles somehow ended up in the first room? Not sure how. It’s gone now.

  • Mel expressed annoyance at having to align a float across the wide penultimate room with no kind of hint, so I added a half-rock-brick tile to the place where you need to stand to use the high-up wheel.

Parting thoughts

I enjoyed making this! It definitely accomplished its ultimate goal of giving me more experience shaking ideas loose. Looking back over those notes, the progression is fascinating: I didn’t even know the core mechanic of resurrecting until 16 hours in (a third of the time), and it was inspired by a joke reply on Twitter. At the 41-hour mark, I still only had three and a half puzzle rooms; the final game has ten. The spellbook seriously only exists because “don’t apply gravity” was so trivial to implement, and the floating effect is something I’d already added for making the Flurry dramatically float above its platform. Half the game only exists because I decided a puzzle was too complicated and tried to split it up.

I almost can’t believe I actually churned all this out in 48 hours. I’ve pretty much never made music before, but I ended up really liking the main theme, and I adore the sound effects. The sprites are great, considering the limitations. I’d never drawn a serious sprite animation before, either, but I love Isaac’s death sequence. The cave texture is great, and a last-minute improvement over my original sprite, which looked more like scratched-up wood. I also drew a scroll sprite that I adored, but I never found an excuse to use it in the game, alas.

Almost everyone who’s played it has made it all the way through without too much trouble, but also seemed to enjoy the puzzles. I take that to mean the game has a good learning curve, which I’m really happy about.

I’m glad I already had a little engine, or I would’ve gotten nowhere.

I have some more ideas that I discarded as impractical due to time or size constraints, so I may port the game to LÖVE and try to expand on it. When I say “may”, I mean I started working on this about two hours after finishing the game.

Oh, and I’m writing a book

Right, yes, about that. I’ve been mumbling about this for ages, but I didn’t want to go on about the idea so much that actually doing it lost its appeal. I think I’ve made enough of a dent now that I’m likely to stick with it.

I’m writing a book about game development — the literal act of game development. I made a list of about a dozen free (well, except PICO-8) and cross-platform game engines spanning a wide range of ease-of-use, creative freedom, and age. I’m going to make a little game in each of them and explain what I’m doing as I do it, give background on totally new things, preserve poor choices and show how I recovered from them, say what inspired an idea or how I got past a creative roadblock, etc. The goal is to write something that someone with no experience in programming or art or storytelling can follow from beginning to end, getting at least an impression of what it looks like to create a game from scratch.

It’s kind of a response to the web’s mountains of tutorials and beginner docs that take you from “here’s what a variable is” all the way to “here’s what a function is”, then abandon you. I hate trying to get into a new thing and only finding slow, dull introductions that don’t tell me how to do anything interesting, or even show what kinds of things are possible. I hope that for anyone who learns the way I do, “here’s how I made a whole game” will be more than enough to hit the ground running.

I have part of an early chapter on MegaZeux written; I wanted to finish it by the end of August, but that’s clearly not happening, oops. I also started on a Godot chapter, which will be a little different since it’s for a game that will hopefully have multiple people working on it.

Isaac’s Descent will be the subject of a PICO-8 chapter — that’s why I took the notes! It’ll expand considerably on what I wrote above, starting with going through all the code I inherited from Under Construction (and recreating how I wrote it in the first place). I also have about 20 snapshots of the game as it developed, which I’m holding onto myself for now.

I want to put rough drafts of each chapter on the $4 Patreon tier as I finish them, so keep an eye out for that, though I don’t have any ETA just yet. I imagine MegaZeux or PICO-8 will be ready within the next couple months.

Weekly roundup: Ludum Dare

Post Syndicated from Eevee original https://eev.ee/dev/2016/08/29/ludum-dare/

August is loosely about video games, but really it’s about three big things in particular.

  • book: Wrangled LaTeX some more. Came up with a new style for admonitions (little set-out boxes) that I really like. Drew some icons for a few of them. Started on another chapter, for reasons; see below.

  • veekun: Regexing machine code for addresses was getting really clumsy, so I went one step further and wrote a disassembling pattern matcher thing. You write some assembly with some variables in it, and it finds occurrences of that code and tells you what the variables are. I can pretty much paste in entire functions, massage them slightly, and find matches. It’s pretty slick.

    The upshot of this is that loading original Japanese Red and Green now works! But Yellow doesn’t. So I fixed that, and now Japanese Blue is broken. Or maybe I fixed it and that broke Yellow again? I’m not sure. There were some tiny changes to core code between some of these games, and the pattern-matcher has no way to express alternatives. I don’t know if I’m better off inventing one or just fudging it.

    Anyway, pretty close to having all of gen 1 dumping Pokémon reliably. Still need to actually dump other stuff — moves, items, encounters, and the like — but that’s much more straightforward.

  • hax: I was still in a mood to dink around with Game Boy stuff, so I added Python 3 support to some relevant tooling and wrote a proof of concept for storing Pokémon maps in Tiled format.

  • blog: I wrote a thing about writing tests.

  • twitter: I taught @perlin_noise a few new tricks.

  • art: I drew a friend’s lizard pal based on a reference photo, which isn’t something I’d seriously tried before. Value-only, only one layer, only one brush. It came out surprisingly well.

  • gamedev: I participated in Ludum Dare 36, a 48-hour game jam. I’d never done LD before, and naturally I picked the only one that has no ratings round (for administrative shuffling reasons). Oh, well.

    The result was Isaac’s Descent, a short puzzle-platformer for the PICO-8. You can play it via the web (source code included), and I also wrote a post about it.

So! There are a few days left, but it’s pretty much the end of August. Let’s see how I did.

  • Draft three chapters of this book, August: one chapter

    Well, I didn’t get a chapter done. I did make huge progress on the chapter I started, though — plus I began a second chapter, and generated enough notes for the entirety of a third. I spent a decent amount of time wrangling Sphinx and LaTeX, too, which I would’ve had to do sooner or later regardless.

    So I didn’t do quite what I wanted, but I did do far more than I’ve put into any previous harebrained book idea, and it was a pretty decent chunk of work. I’m okay with that.

    Just what is this damn book, you ask? Ah, perhaps you should read that Ludum Dare post.

  • Get veekun beta-worthy, August: basics of the new schema committed; basics of gen 1 and gen 6 games dumped; skeleton cli and site

    Haha, no. I got gen 1 almost working for Pokémon only. It turns out that while gen 1 has the simplest data, it probably has the most convoluted storage.

    On the other hand, the detours taught me a lot about Game Boy architecture, which was interesting and helpful for making the dumper fairly robust thusfar. I also made some breakthroughs on architecture that had been haunting me for a while. I’ll have to move my ass in the next week or two to catch up — hopefully finish gen 1 and get a few other generations dumped real soon — but I think this is still doable.

  • Finish Runed Awakening, August: working ending; at least one solution to each puzzle; private beta

    Whoops! I did basically squat on Runed Awakening. I figured out most of the ending, which had been my major roadblock, but I didn’t touch the code or run the game a single time. Dang. It’s not like I was goofing off all month, either; I just didn’t have a big block of time to devote to the weird mishmash of writing and planning and programming that IF requires.

    I really want to finish this game, but end of October is not looking too great. I don’t know why it’s proving so difficult; it’s not that complicated, and I started on it almost two years ago now. I’ve made multiple other games just so far this year! Argh.

    If it’s any consolation (to me): I picked November as a target because Mel wanted to embed Runed Awakening on Floraverse as an update around that time. But Isaac’s Descent takes place in the same universe, so it works just as well. Goal accomplished!

Onwards to September. The only thing on the list with a real solid deadline is veekun, since the new games will be coming out. It’s a bit behind, but I’m pretty sure I can catch up. Gen 2 shouldn’t be too different from gen 1, and I’ve done gen 4 and onwards before.

Weekly roundup: Ludum Dare

Post Syndicated from Eevee original https://eev.ee/dev/2016/08/29/weekly-roundup-ludum-dare/

August is loosely about video games, but really it’s about three big things in particular.

  • book: Wrangled LaTeX some more. Came up with a new style for admonitions (little set-out boxes) that I really like. Drew some icons for a few of them. Started on another chapter, for reasons; see below.

  • veekun: Regexing machine code for addresses was getting really clumsy, so I went one step further and wrote a disassembling pattern matcher thing. You write some assembly with some variables in it, and it finds occurrences of that code and tells you what the variables are. I can pretty much paste in entire functions, massage them slightly, and find matches. It’s pretty slick.

    The upshot of this is that loading original Japanese Red and Green now works! But Yellow doesn’t. So I fixed that, and now Japanese Blue is broken. Or maybe I fixed it and that broke Yellow again? I’m not sure. There were some tiny changes to core code between some of these games, and the pattern-matcher has no way to express alternatives. I don’t know if I’m better off inventing one or just fudging it.

    Anyway, pretty close to having all of gen 1 dumping Pokémon reliably. Still need to actually dump other stuff — moves, items, encounters, and the like — but that’s much more straightforward.

  • hax: I was still in a mood to dink around with Game Boy stuff, so I added Python 3 support to some relevant tooling and wrote a proof of concept for storing Pokémon maps in Tiled format.

  • blog: I wrote a thing about writing tests.

  • twitter: I taught @perlin_noise a few new tricks.

  • art: I drew a friend’s lizard pal based on a reference photo, which isn’t something I’d seriously tried before. Value-only, only one layer, only one brush. It came out surprisingly well.

  • gamedev: I participated in Ludum Dare 36, a 48-hour game jam. I’d never done LD before, and naturally I picked the only one that has no ratings round (for administrative shuffling reasons). Oh, well.

    The result was Isaac’s Descent, a short puzzle-platformer for the PICO-8. You can play it via the web (source code included), and I also wrote a post about it.

So! There are a few days left, but it’s pretty much the end of August. Let’s see how I did.

  • Draft three chapters of this book, August: one chapter

    Well, I didn’t get a chapter done. I did make huge progress on the chapter I started, though — plus I began a second chapter, and generated enough notes for the entirety of a third. I spent a decent amount of time wrangling Sphinx and LaTeX, too, which I would’ve had to do sooner or later regardless.

    So I didn’t do quite what I wanted, but I did do far more than I’ve put into any previous harebrained book idea, and it was a pretty decent chunk of work. I’m okay with that.

    Just what is this damn book, you ask? Ah, perhaps you should read that Ludum Dare post.

  • Get veekun beta-worthy, August: basics of the new schema committed; basics of gen 1 and gen 6 games dumped; skeleton cli and site

    Haha, no. I got gen 1 almost working for Pokémon only. It turns out that while gen 1 has the simplest data, it probably has the most convoluted storage.

    On the other hand, the detours taught me a lot about Game Boy architecture, which was interesting and helpful for making the dumper fairly robust thusfar. I also made some breakthroughs on architecture that had been haunting me for a while. I’ll have to move my ass in the next week or two to catch up — hopefully finish gen 1 and get a few other generations dumped real soon — but I think this is still doable.

  • Finish Runed Awakening, August: working ending; at least one solution to each puzzle; private beta

    Whoops! I did basically squat on Runed Awakening. I figured out most of the ending, which had been my major roadblock, but I didn’t touch the code or run the game a single time. Dang. It’s not like I was goofing off all month, either; I just didn’t have a big block of time to devote to the weird mishmash of writing and planning and programming that IF requires.

    I really want to finish this game, but end of October is not looking too great. I don’t know why it’s proving so difficult; it’s not that complicated, and I started on it almost two years ago now. I’ve made multiple other games just so far this year! Argh.

    If it’s any consolation (to me): I picked November as a target because Mel wanted to embed Runed Awakening on Floraverse as an update around that time. But Isaac’s Descent takes place in the same universe, so it works just as well. Goal accomplished!

Onwards to September. The only thing on the list with a real solid deadline is veekun, since the new games will be coming out. It’s a bit behind, but I’m pretty sure I can catch up. Gen 2 shouldn’t be too different from gen 1, and I’ve done gen 4 and onwards before.

Python FAQ: Why should I use Python 3?

Post Syndicated from Eevee original https://eev.ee/blog/2016/07/31/python-faq-why-should-i-use-python-3/

Part of my Python FAQ, which is doomed to never be finished.

The short answer is: because it’s the actively-developed version of the language, and you should use it for the same reason you’d use 2.7 instead of 2.6.

If you’re here, I’m guessing that’s not enough. You need something to sweeten the deal. Well, friend, I have got a whole mess of sugar cubes just for you.

And once you’re convinced, you may enjoy the companion article, how to port to Python 3! It also has some more details on the diffences between Python 2 and 3, whereas this article doesn’t focus too much on the features removed in Python 3.

Some background

If you aren’t neck-deep in Python, you might be wondering what the fuss is all about, or why people keep telling you that Python 3 will set your computer on fire. (It won’t.)

Python 2 is a good language, but it comes with some considerable baggage. It has two integer types; it may or may not be built in a way that completely mangles 16/17 of the Unicode space; it has a confusing mix of lazy and eager functional tools; it has a standard library that takes “batteries included” to lengths beyond your wildest imagination; it boasts strong typing, then casually insists that None < 3 < "2"; overall, it’s just full of little dark corners containing weird throwbacks to the days of Python 1.

(If you’re really interested, Nick Coghlan has written an exhaustive treatment of the slightly different question of why Python 3 was created. This post is about why Python 3 is great, so let’s focus on that.)

Fixing these things could break existing code, whereas virtually all code written for 2.0 will still work on 2.7. So Python decided to fix them all at once, producing a not-quite-compatible new version of the language, Python 3.

Nothing like this has really happened with a mainstream programming language before, and it’s been a bit of a bumpy ride since then. Python 3 was (seemingly) designed with the assumption that everyone would just port to Python 3, drop Python 2, and that would be that. Instead, it’s turned out that most libraries want to continue to run on both Python 2 and Python 3, which was considerably difficult to make work at first. Python 2.5 was still in common use at the time, too, and it had none of the helpful backports that showed up in Python 2.6 and 2.7; likewise, Python 3.0 didn’t support u'' strings. Writing code that works on both 2.5 and 3.0 was thus a ridiculous headache.

The porting effort also had a dependency problem: if your library or app depends on library A, which depends on library B, which depends on C, which depends on D… then none of those projects can even think about porting until D’s porting effort is finished. Early days were very slow going.

Now, though, things are looking brighter. Most popular libraries work with Python 3, and those that don’t are working on it. Python 3’s Unicode handling, one of its most contentious changes, has had many of its wrinkles ironed out. Python 2.7 consists largely of backported Python 3 features, making it much simpler to target 2 and 3 with the same code — and both 2.5 and 2.6 are no longer supported.

Don’t get me wrong, Python 2 will still be around for a while. A lot of large applications have been written for Python 2 — think websites like Yelp, YouTube, Reddit, Dropbox — and porting them will take some considerable effort. I happen to know that at least one of those websites was still running 2.6 last year, years after 2.6 had been discontinued, if that tells you anything about the speed of upgrades for big lumbering software.

But if you’re just getting started in Python, or looking to start a new project, there aren’t many reasons not to use Python 3. There are still some, yes — but unless you have one specifically in mind, they probably won’t affect you.

I keep having Python beginners tell me that all they know about Python 3 is that some tutorial tried to ward them away from it for vague reasons. (Which is ridiculous, since especially for beginners, Python 2 and 3 are fundamentally not that different.) Even the #python IRC channel has a few people who react, ah, somewhat passive-aggressively towards mentions of Python 3. Most of the technical hurdles have long since been cleared; it seems like one of the biggest roadblocks now standing in the way of Python 3 adoption is the community’s desire to sabotage itself.

I think that’s a huge shame. Not many people seem to want to stand up for Python 3, either.

Well, here I am, standing up for Python 3. I write all my new code in Python 3 now — because Python 3 is great and you should use it. Here’s why.

Hang on, let’s be real for just a moment

None of this is going to 💥blow your mind💥. It’s just a programming language. I mean, the biggest change to Python 2 in the last decade was probably the addition of the with statement, which is nice, but hardly an earth-shattering innovation. The biggest changes in Python 3 are in the same vein: they should smooth out some points of confusion, help avoid common mistakes, and maybe give you a new toy to play with.

Also, if you’re writing a library that needs to stay compatible with Python 2, you won’t actually be able to use any of this stuff. Sorry. In that case, the best reason to port is so application authors can use this stuff, rather than citing your library as the reason they’re trapped on Python 2 forever. (But hey, if you’re starting a brand new library that will blow everyone’s socks off, do feel free to make it Python 3 exclusive.)

Application authors, on the other hand, can go wild.

Unicode by default

Let’s get the obvious thing out of the way.

In Python 2, there are two string types: str is a sequence of bytes (which I would argue makes it not a string), and unicode is a sequence of Unicode codepoints. A literal string in source code is a str, a bytestring. Reading from a file gives you bytestrings. Source code is assumed ASCII by default. It’s an 8-bit world.

If you happen to be an English speaker, it’s very easy to write Python 2 code that seems to work perfectly, but chokes horribly if fed anything outside of ASCII. The right thing involves carefully specifying encodings everywhere and using u'' for virtually all your literal strings, but that’s very tedious and easily forgotten.

Python 3 reshuffles this to put full Unicode support front and center.

Most obviously, the str type is a real text type, similar to Python 2’s unicode. Literal strings are still str, but now that makes them Unicode strings. All of the “structural” strings — names of types, functions, modules, etc. — are likewise Unicode strings. Accordingly, identifiers are allowed to contain any Unicode “letter” characters. repr() no longer escapes printable Unicode characters, though there’s a new ascii() (and corresponding !a format cast and %a placeholder) that does. Unicode completely pervades the language, for better or worse.

And just for the record: this is way better. It is so much better. It is incredibly better. Do you know how much non-ASCII garbage I type? Every single em dash in this damn post was typed by hand, and Python 2 would merrily choke on them.

Source files are now assumed to be UTF-8 by default, so adding an em dash in a comment will no longer break your production website. (I have seen this happen.) You’re still free to specify another encoding explicitly if you want, using a magic comment.

There is no attempted conversion between bytes and text, as in Python 2; b'a' + 'b' is a TypeError. Some modules require you to know what you’re dealing with: zlib.compress only accepts bytes, because zlib is defined in terms of bytes; json.loads only accepts str, because JSON is defined in terms of Unicode codepoints. Calling str() on some bytes will defer to repr, producing something like "b'hello'". (But see -b and -bb below.) Overall it’s pretty obvious when you’ve mixed bytes with text.

Oh, and two huge problem children are fixed: both the csv module and urllib.parse (formerly urlparse) can handle text. If you’ve never tried to make those work, trust me, this is miraculous.

I/O does its best to make everything Unicode. On Unix, this is a little hokey, since the filesystem is explicitly bytes with no defined encoding; Python will trust the various locale environment variables, which on most systems will make everything UTF-8. The default encoding of text-mode file I/O is derived the same way and thus usually UTF-8. (If it’s not what you expect, run locale and see what you get.) Files opened in binary mode, with a 'b', will still read and write bytes.

Python used to come in “narrow” and “wide” builds, where “narrow” builds actually stored Unicode as UTF-16, and this distinction could leak through to user code in subtle ways. On a narrow build, unichr(0x1F4A3) raises ValueError, and the length of u'💣' is 2. Surprise! Maybe your code will work on someone else’s machine, or maybe it won’t. Python 3.3 eliminated narrow builds.

I think those are the major points. For the most part, you should be able to write code as though encodings don’t exist, and the right thing will happen more often. And the wrong thing will immediately explode in your face. It’s good for you.

If you work with binary data a lot, you might be frowning at me at this point; it was a bit of a second-class citizen in Python 3.0. I think things have improved, though: a number of APIs support both bytes and text, the bytes-to-bytes codec issue has largely been resolved, we have bytes.hex() and bytes.fromhex(), bytes and bytearray both support % now, and so on. They’re listening!

Refs: Python 3.0 release notes; myriad mentions all over the documentation

Backported features

Python 3.0 was released shortly after Python 2.6, and a number of features were then backported to Python 2.7. You can use these if you’re only targeting Python 2.7, but if you were stuck with 2.6 for a long time, you might not have noticed them.

  • Set literals:

    {1, 2, 3}
  • Dict and set comprehensions:

    {word.lower() for word in words}
    {value: key for (key, value) in dict_to_invert.items()}
  • Multi-with:

    with open("foo") as f1, open("bar") as f2:
  • print is now a function, with a couple bells and whistles added: you can change the delimiter with the sep argument, you can change the terminator to whatever you want (including nothing) with the end argument, and you can force a flush with the flush argument. In Python 2.6 and 2.7, you still have to opt into this with from __future__ import print_function.

  • The string representation of a float now uses the shortest decimal number that has the same underlying value — for example, repr(1.1) was '1.1000000000000001' in Python 2.6, but is just '1.1' in Python 2.7 and 3.1+, because both are represented the same way in a 64-bit float.

  • collections.OrderedDict is a dict-like type that remembers the order of its keys.

    Note that you cannot do OrderedDict(a=1, b=2), because the constructor still receives its keyword arguments in a regular dict, losing the order. You have to pass in a sequence of 2-tuples or assign keys one at a time.

  • collections.Counter is a dict-like type for counting a set of things. It has some pretty handy operations that allow it to be used like a multiset.

  • The entire argparse module is a backport from 3.2.

  • str.format learned a , formatting specifier for numbers, which always uses commas and groups of three digits. This is wrong for many countries, and the correct solution involves using the locale module, but it’s useful for quick output of large numbers.

  • re.sub, re.subn, and re.split accept a flags argument. Minor, but, thank fucking God.

Ref: Python 2.7 release notes

Iteration improvements

Everything is lazy

Python 2 has a lot of pairs of functions that do the same thing, except one is eager and one is lazy: range and xrange, map and itertools.imap, dict.keys and dict.iterkeys, and so on.

Python 3.0 eliminated all of the lazy variants and instead made the default versions lazy. Iterating over them works exactly the same way, but no longer creates an intermediate list — for example, range(1000000000) won’t eat all your RAM. If you need to index them or store them for later, you can just wrap them in list(...).

Even better, the dict methods are now “views“. You can keep them around, and they’ll reflect any changes to the underlying dict. They also act like sets, so you can do a.keys() & b.keys() to get the set of keys that exist in both dicts.

Refs: dictionary view docs; Python 3.0 release notes


Unpacking got a huge boost. You could always do stuff like this in Python 2:

a, b, c = range(3)  # a = 0, b = 1, c = 2

Python 3.0 introduces:

a, b, *c = range(5)  # a = 0, b = 1, c = [2, 3, 4]
a, *b, c = range(5)  # a = 0, b = [1, 2, 3], c = 4

Python 3.5 additionally allows use of the * and ** unpacking operators in literals, or multiple times in function calls:

print(*range(3), *range(3))  # 0 1 2 0 1 2

x = [*range(3), *range(3)]  # x = [0, 1, 2, 0, 1, 2]
y = {*range(3), *range(3)}  # y = {0, 1, 2}  (it's a set, remember!)
z = {**dict1, **dict2}  # finally, syntax for dict merging!

Refs: Python 3.0 release notes; PEP 3132; Python 3.5 release notes; PEP 448

yield from

yield from is an extension of yield. Where yield produces a single value, yield from yields an entire sequence.

def flatten(*sequences):
    for seq in sequences:
        yield from seq

list(flatten([1, 2], [3, 4]))  # [1, 2, 3, 4]

Of course, for a simple example like that, you could just do some normal yielding in a for loop. The magic of yield from is that it can also take another generator or other lazy iterable, and it’ll effectively pause the current generator until the given one has been exhausted. It also takes care of passing values back into the generator using .send() or .throw().

def foo():
    a = yield 1
    b = yield from bar(a)
    print("foo got back", b)
    yield 4

def bar(a):
    print("in bar", a)
    x = yield 2
    y = yield 3
    print("leaving bar")
    return x + y

gen = foo()
val = None
while True:
        newval = gen.send(val)
    except StopIteration:
    print("yielded", newval)
    val = newval * 10

# yielded 1
# in bar 10
# yielded 2
# yielded 3
# leaving bar
# foo got back 50
# yielded 4

Oh yes, and you can now return a value from a generator. The return value becomes the result of a yield from, or if the caller isn’t using yield from, it’s available as the argument to the StopIteration exception.

A small convenience, perhaps. The real power here isn’t in the use of generators as lazy iterators, but in the use of generators as coroutines.

A coroutine is a function that can “suspend” itself, like yield does, allowing other code to run until the function is resumed. It’s kind of like an alternative to threading, but only one function is actively running at any given time, and that function has to delierately relinquish control (or end) before anything else can run.

Generators could do this already, more or less, but only one stack frame deep. That is, you can yield from a generator to suspend it, but if the generator calls another function, that other function has no way to suspend the generator. This is still useful, but significantly less powerful than the coroutine functionality in e.g. Lua, which lets any function yield anywhere in the call stack.

With yield from, you can create a whole chain of generators that yield from one another, and as soon as the one on the bottom does a regular yield, the entire chain will be suspended.

This laid the groundwork for making the asyncio module possible. I’ll get to that later.

Refs: docs; Python 3.3 release notes; PEP 380

Syntactic sugar

Keyword-only arguments

Python 3.0 introduces “keyword-only” arguments, which must be given by name. As a corollary, you can now accept a list of args and have more arguments afterwards. The full syntax now looks something like this:

def foo(a, b=None, *args, c=None, d, **kwargs):

Here, a and d are required, b and c are optional. c and d must be given by name.

foo(1)                      # TypeError: missing d
foo(1, 2)                   # TypeError: missing d
foo(d=4)                    # TypeError: missing a
foo(1, d=4)                 # a = 1, d = 4
foo(1, 2, d=4)              # a = 1, b = 2, d = 4
foo(1, 2, 3, d=4)           # a = 1, b = 2, args = (3,), d = 4
foo(1, 2, c=3, d=4)         # a = 1, b = 2, c = 3, d = 4
foo(1, b=2, c=3, d=4, e=5)  # a = 1, b = 2, c = 3, d = f, kwargs = {'e': 5}

This is extremely useful for functions with a lot of arguments, functions with boolean arguments, functions that accept *args (or may do so in the future) but also want some options, etc. I use it a lot!

If you want keyword-only arguments, but you don’t want to accept *args, you just leave off the variable name:

def foo(*, arg=None):

Refs: Python 3.0 release notes; PEP 3102

Format strings

Python 3.6 (not yet out) will finally bring us string interpolation, more or less, using the str.format() syntax:

a = 0x133
b = 0x352
print(f"The answer is {a + b:04x}.")

It’s pretty much the same as str.format(), except that instead of a position or name, you can give an entire expression. The formatting suffixes with : still work, the special built-in conversions like !r still work, and __format__ is still invoked.

Refs: docs; Python 3.6 release notes; PEP 498

async and friends

Right, so, about coroutines.

Python 3.4 introduced the asyncio module, which offers building blocks for asynchronous I/O (and bringing together the myriad third-party modules that do it already).

The design is based around coroutines, which are really generators using yield from. The idea, as I mentioned above, is that you can create a stack of generators that all suspend at once:

def foo():
    # do some stuff
    yield from bar()
    # do more stuff

def bar():
    # do some stuff
    response = yield from get_url("https://eev.ee/")
    # do more stuff

When this code calls get_url() (not actually a real function, but see aiohttp), get_url will send a request off into the æther, and then yield. The entire stack of generators — get_url, bar, and foo — will all suspend, and control will return to whatever first called foo, which with asyncio will be an “event loop”.

The event loop’s entire job is to notice that get_url yielded some kind of “I’m doing a network request” thing, remember it, and resume other coroutines in the meantime. (Or just twiddle its thumbs, if there’s nothing else to do.) When a response comes back, the event loop will resume get_url and send it the response. get_url will do some stuff and return it up to bar, who continues on, none the wiser that anything unusual happened.

The magic of this is that you can call get_url several times, and instead of having to wait for each request to completely finish before the next one can even start, you can do other work while you’re waiting. No threads necessary; this is all one thread, with functions cooperatively yielding control when they’re waiting on some external thing to happen.

Now, notice that you do have to use yield from each time you call another coroutine. This is nice in some ways, since it lets you see exactly when and where your function might be suspended out from under you, which can be important in some situations. There are also arguments about why this is bad, and I don’t care about them.

However, yield from is a really weird phrase to be sprinkling all over network-related code. It’s meant for use with iterables, right? Lists and tuples and things. get_url is only one thing. What are we yielding from it? Also, what’s this @coroutine decorator that doesn’t actually do anything?

Python 3.5 smoothed over this nonsense by introducing explicit syntax for these constructs, using new async and await keywords:

async def foo():
    # do some stuff
    await bar()
    # do more stuff

async def bar():
    # do some stuff
    response = await get_url("https://eev.ee/")
    # do more stuff

async def clearly identifies a coroutine, even one that returns immediately. (Before, you’d have a generator with no yield, which isn’t actually a generator, which causes some problems.) await explains what’s actually happening: you’re just waiting for another function to be done.

async for and async with are also available, replacing some particularly clumsy syntax you’d need to use before. And, handily, you can only use any of these things within an async def.

The new syntax comes with corresponding new special methods like __await__, whereas the previous approach required doing weird things with __iter__, which is what yield from ultimately calls.

I could fill a whole post or three with stuff about asyncio, and can’t possibly give it justice in just a few paragraphs. The short version is: there’s built-in syntax for doing network stuff in parallel without threads, and that’s cool.

Refs for asyncio: docs (asyncio); Python 3.4 release notes; PEP 3156

Refs for async and await: docs (await); docs (async); docs (special methods); Python 3.5 release notes; PEP 492

Function annotations

Function arguments and return values can have annotations:

def foo(a: "hey", b: "what's up") -> "whoa":

The annotations are accessible via the function’s __annotations__ attribute. They have no special meaning to Python, so you’re free to experiment with them.


You were free to experiment with them, but the addition of the typing module (mentioned below) has hijacked them for type hints. There’s no clear way to attach a type hint and some other value to the same argument, so you’ll have a tough time making function annotations part of your API.

There’s still no hard requirement that annotations be used exclusively for type hints (and it’s not like Python does anything with type hints, either), but the original PEP suggests it would like that to be the case someday. I guess we’ll see.

If you want to see annotations preserved for other uses as well, it would be a really good idea to do some creative and interesting things with them as soon as possible. Just saying.

Refs: docs; Python 3.0 release notes; PEP 3107

Matrix multiplication

Python 3.5 learned a new infix operator for matrix multiplication, spelled @. It doesn’t do anything for any built-in types, but it’s supported in NumPy. You can implement it yourself with the __matmul__ special method and its r and i variants.

Shh. Don’t tell anyone, but I suspect there are fairly interesting things you could do with an operator called @ — some of which have nothing to do with matrix multiplication at all!

Refs: Python 3.5 release notes; PEP 465


... is now valid syntax everywhere. It evaluates to the Ellipsis singleton, which does nothing. (This exists in Python 2, too, but it’s only allowed when slicing.)

It’s not of much practical use, but you can use it to indicate an unfinished stub, in a way that’s clearly not intended to be final but will still parse and run:

class ReallyComplexFiddlyThing:
    # fuck it, do this later

Refs: docs; Python 3.0 release notes

Enhanced exceptions

A slightly annoying property of Python 2’s exception handling is that if you want to do your own error logging, or otherwise need to get at the traceback, you have to use the slightly funky sys.exc_info() API and carry the traceback around separately. As of Python 3.0, exceptions automatically have a __traceback__ attribute, as well as a .with_traceback() method that sets the traceback and returns the exception itself (so you can use it inline).

This makes some APIs a little silly — __exit__ still accepts the exception type and value and traceback, even though all three are readily available from just the exception object itself.

A much more annoying property of Python 2’s exception handling was that custom exception handling would lose track of where the problem actually occurred. Consider the following call stack.


Now say an exception happens in E, and it’s caught by code like this in C.

except Exception as e:
    raise CustomError("Failed to call D")

Because this creates and raises a new exception, the traceback will start from this point and not even mention E. The best workaround for this involves manually creating a traceback between C and E, formatting it as a string, and then including that in the error message. Preposterous.

Python 3.0 introduced exception chaining, which allows you to do this:

raise CustomError("Failed to call D") from e

Now, if this exception reaches the top level, Python will format it as:

Traceback (most recent call last):
File C, blah blah
File D, blah blah
File E, blah blah

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File A, blah blah
File B, blah blah
File C, blah blah
CustomError: Failed to call D

The best part is that you don’t need to explicitly say from e at all — if you do a plain raise while there’s already an active exception, Python will automatically chain them together. Even internal Python exceptions will have this behavior, so a broken exception handler won’t lose the original exception. (In the implicit case, the intermediate text becomes “During handling of the above exception, another exception occurred:”.)

The chained exception is stored on the new exception as either __cause__ (if from an explicit raise ... from) or __context__ (if automatic).

If you direly need to hide the original exception, Python 3.3 introduced raise ... from None.

Speaking of exceptions, the error messages for missing arguments have been improved. Python 2 does this:

TypeError: foo() takes exactly 1 argument (0 given)

Python 3 does this:

TypeError: foo() missing 1 required positional argument: 'a'


Cooler classes

super() with no arguments

You can call super() with no arguments. It Just Works. Hallelujah.

Also, you can call super() with no arguments. That’s so great that I could probably just fill the rest of this article with it and be satisfied.

Did I mention you can call super() with no arguments?

Refs: docs; Python 3.0 release notes; PEP 3135

New metaclass syntax and kwargs for classes

Compared to that, everything else in this section is going to sound really weird and obscure.

For example, __metaclass__ is gone. It’s now a keyword-only argument to the class statement.

class Foo(metaclass=FooMeta):

That doesn’t sound like much, right? Just some needless syntax change that makes porting harder, right?? Right??? Haha nope watch this because it’s amazing but it barely gets any mention at all.

class Foo(metaclass=FooMeta, a=1, b=2, c=3):

You can include arbitrary keyword arguments in the class statement, and they will be passed along to the metaclass call as keyword arguments. (You have to catch them in both __new__ and __init__, since they always get the same arguments.) (Also, the class statement now has the general syntax of a function call, so you can put *args and **kwargs in it.)

This is pretty slick. Consider SQLAlchemy, which uses a metaclass to let you declare a table with a class.

class SomeTable(TableBase):
    __tablename__ = 'some_table'
    id = Column()

Note that SQLAlchemy has you put the name of the table in the clumsy __tablename__ attribute, which it invented. Why not just name? Well, because then you couldn’t declare a column called name! Any “declarative” metaclass will have the same problem of separating the actual class contents from configuration. Keyword arguments offer an easy way out.

# only hypothetical, alas
class SomeTable(TableBase, name='some_table'):
    id = Column()

Refs: docs; Python 3.0 release notes; PEP 3115


Another new metaclass feature is the introduction of the __prepare__ method.

You may have noticed that the body of a class is just a regular block, which can contain whatever code you want. Before decorators were a thing, you’d actually declare class methods in two stages:

class Foo:
    def do_the_thing(cls):
    do_the_thing = classmethod(do_the_thing)

That’s not magical class-only syntax; that’s just regular code assigning to a variable. You can put ifs and fors and whiles and dels inside a class body, too; you just don’t see it very often because there aren’t very many useful reasons to do it.

A class body is a kind of weird pseudo-scope. It can create locals, and it can read values from outer scopes, but methods don’t see the class body as an outer scope. Once the class body reaches its end, any remaining locals are passed to the type constructor and become the new class’s attributes. (This is why, for example, you can’t refer to a class directly within its own body — the class doesn’t and can’t exist until after the body has executed.)

All of this is to say: __prepare__ is a new hook that returns the dict the class body’s locals go into.

Maybe that doesn’t sound particularly interesting, but consider: the value you return doesn’t have to be an actual dict. It can be anything that understands __setitem__. You could, say, use an OrderedDict, and keep track of the order your attributes were declared. That’s useful for declarative metaclasses, where the order of attributes may be important (consider a C struct).

But you can go further. You might allow more than one attribute of the same name. You might do something special with the attributes as soon as they’re assigned, rather than at the end of the body. You might predeclare some attributes. __prepare__ is passed the class’s kwargs, so you might alter the behavior based on those.

For a nice practical example, consider the new enum module, which I briefly mention later on. One drawback of this module is that you have to specify a value for every variant, since variants are defined as class attributes, which must have a value. There’s an example of automatic numbering, but it still requires assigning a dummy value like (). Clever use of __prepare__ would allow lifting this restriction:

# XXX: Should prefer MutableMapping here, but the ultimate call to type()
# raises a TypeError if you pass a namespace object that doesn't inherit
# from dict!  Boo.
class EnumLocals(dict):
    def __init__(self):
        self.nextval = 1

    def __getitem__(self, key):
        if key not in self and not key.startswith('_') and not key.endswith('_'):
            self[key] = self.nextval
            self.nextval += 1
        return super().__getitem__(key)

class EnumMeta(type):
    def __prepare__(meta, name, bases):
        return EnumLocals()

class Colors(metaclass=EnumMeta):

print(Colors.red, Colors.green, Colors.blue)
# 1 2 3

Deciding whether this is a good idea is left as an exercise.

This is an exceptionally obscure feature that gets very little attention — it’s not even mentioned explicitly in the 3.0 release notes — but there’s nothing else like it in the language. Between __prepare__ and keyword arguments, the class statement has transformed into a much more powerful and general tool for creating all kinds of objects. I almost wish it weren’t still called class.

Refs: docs; Python 3.0 release notes; PEP 3115

Attribute definition order

If that’s still too much work, don’t worry: a proposal was just accepted for Python 3.6 that makes this even easier. Now every class will have a __definition_order__ attribute, a tuple listing the names of all the attributes assigned within the class body, in order. (To make this possible, the default return value of __prepare__ will become an OrderedDict, but the __dict__ attribute will remain a regular dict.)

Now you don’t have to do anything at all: you can always check to see what order any class’s attributes were defined in.

Additionally, descriptors can now implement a __set_name__ method. When a class is created, any descriptor implementing the method will have it called with the containing class and the name of the descriptor.

I’m very excited about this, but let me try to back up. A descriptor is a special Python object that can be used to customize how a particular class attribute works. The built-in property decorator is a descriptor.

class MyClass:
    foo = SomeDescriptor()

c = MyClass()
c.foo = 5  # calls SomeDescriptor.__set__!
print(c.foo)  # calls SomeDescriptor.__get__!

This is super cool and can be used for all sorts of DSL-like shenanigans.

Now, most descriptors ultimately want to store a value somewhere, and the obvious place to do that is in the object’s __dict__. Above, SomeDescriptor might want to store its value in c.__dict__['foo'], which is fine since Python will still consult the descriptor first. If that weren’t fine, it could also use the key '_foo', or whatever. It probably wants to use its own name somehow, because otherwise… what would happen if you had two SomeDescriptors in the same class?

Therein lies the problem, and one of my long-running and extremely minor frustrations with Python. Descriptors have no way to know their own name! There are only really two solutions to this:

  1. Require the user to pass the name in as an argument, too: foo = SomeDescriptor('foo'). Blech!

  2. Also have a metaclass (or decorator, or whatever), which can iterate over all the class’s attributes, look for SomeDescriptor objects, and tell them what their names are. Needing a metaclass means you can’t make general-purpose descriptors meant for use in arbitrary classes; a decorator would work, but boy is that clumsy.

Both of these suck and really detract from what could otherwise be very neat-looking syntax trickery.

But now! Now, when MyClass is created, Python will have a look through its attributes. If it sees that the foo object has a __set_name__ method, it’ll call that method automatically, passing it both the owning class and the name 'foo'! Huzzah!

This is so great I am so happy you have no idea.

Lastly, there’s now an __init_subclass__ class method, which is called when the class is subclassed. A great many metaclasses exist just to do a little bit of work for each new subclass; now, you don’t need a metaclass at all in many simple cases. You want a plugin registry? No problem:

class Plugin:
    _known_plugins = {}

    def __init_subclass__(cls, *, name, **kwargs):
        cls._known_plugins[name] = cls

    def get_plugin(cls, name):
        return cls._known_plugins[name]

    # ...probably some interface stuff...

class FooPlugin(Plugin, name="foo"):

No metaclass needed at all.

Again, none of this stuff is available yet, but it’s all slated for Python 3.6, due out in mid-December. I am super pumped.

Refs: docs (customizing class creation); docs (descriptors); Python 3.6 release notes; PEP 520 (attribute definition order); PEP 487 (__init_subclass__ and __set_name__)

Math stuff

int and long have been merged, and there is no longer any useful distinction between small and very large integers. I’ve actually run into code that breaks if you give it 1 instead of 1L, so, good riddance. (Python 3.0 release notes; PEP 237)

The / operator always does “true” division, i.e., gives you a float. If you want floor division, use //. Accordingly, the __div__ magic method is gone; it’s split into two parts, __truediv__ and __floordiv__. (Python 3.0 release notes; PEP 238)

decimal.Decimal, fractions.Fraction, and floats now interoperate a little more nicely: numbers of different types hash to the same value; all three types can be compared with one another; and most notably, the Decimal and Fraction constructors can accept floats directly. (docs (decimal); docs (fractions); Python 3.2 release notes)

math.gcd returns the greatest common divisor of two integers. This existed before, but was in the fractions module, where nobody knew about it. (docs; Python 3.5 release notes)

math.inf is the floating-point infinity value. Previously, this was only available by writing float('inf'). There’s also a math.nan, but let’s not? (docs; Python 3.5 release notes)

math.isclose (and the corresponding complex version, cmath.isclose) determines whether two values are “close enough”. Intended to do the right thing when comparing floats. (docs; Python 3.5 release notes; PEP 485)

More modules

The standard library has seen quite a few improvements. In fact, Python 3.2 was developed with an explicit syntax freeze, so it consists almost entirely of standard library enhancements. There are far more changes across six and a half versions than I can possibly list here; these are the ones that stood out to me.

The module shuffle

Python 2, rather inexplicably, had a number of top-level modules that were named after the single class they contained, CamelCase and all. StringIO and SimpleHTTPServer are two obvious examples. In Python 3, the StringIO class lives in io (along with BytesIO), and SimpleHTTPServer has been renamed to http.server. If you’re anything like me, you’ll find this deeply satisfying.

Wait, wait, there’s a practical upside here. Python 2 had several pairs of modules that did the same thing with the same API, but one was pure Python and one was much faster C: pickle/cPickle, profile/cProfile, and StringIO/cStringIO. I’ve seen code (cough, older versions of Babel, cough) that spent a considerable amount of its startup time reading pickles with the pure Python version, because it did the obvious thing and used the pickle module. Now, these pairs have been merged: importing pickle gives you the faster C implementation, importing profile gives you the faster C implementation, and BytesIO/StringIO are the fast C implementations in the io module.

Refs: docs (sort of); Python 3.0 release notes; PEP 3108 (exhaustive list of removed and renamed modules)

Additions to existing modules

A number of file format modules, like bz2 and gzip, went through some cleanup and modernization in 3.2 through 3.4: some learned a more straightforward open function, some gained better support for the bytes/text split, and several learned to use their file types as context managers (i.e., with with).

collections.ChainMap is a mapping type that consults some number of underlying mappings in order, allowing for a “dict with defaults” without having to merge them together. (docs; Python 3.3 release notes)

configparser dropped its ridiculous distinction between ConfigParser and SafeConfigParser; there is now only ConfigParser, which is safe. The parsed data now preserves order by default and can be read or written using normal mapping syntax. Also there’s a fancier alternative interpolation parser. (docs; Python 3.2 release notes)

contextlib.ContextDecorator is some sort of devilry that allows writing a context manager which can also be used as a decorator. It’s used to implement the @contextmanager decorator, so those can be used as decorators as well. (docs; Python 3.2 release notes)

contextlib.ExitStack offers cleaner and more fine-grained handling of multiple context managers, as well as resources that don’t have their own context manager support. (docs; Python 3.3 release notes)

contextlib.suppress is a context manager that quietly swallows a given type of exception. (docs; Python 3.4 release notes)

contextlib.redirect_stdout is a context manager that replaces sys.stdout for the duration of a block. (docs; Python 3.4 release notes)

datetime.timedelta already existed, of course, but now it supports being multiplied and divided by numbers or divided by other timedeltas. The upshot of this is that timedelta finally, finally has a .total_seconds() method which does exactly what it says on the tin. (docs; Python 3.2 release notes)

datetime.timezone is a new concrete type that can represent fixed offsets from UTC. There has long been a datetime.tzinfo, but it was a useless interface, and you were left to write your own actual class yourself. datetime.timezone.utc is a pre-existing instance that represents UTC, an offset of zero. (docs; Python 3.2 release notes)

functools.lru_cache is a decorator that caches the results of a function, keyed on the arguments. It also offers cache usage statistics and a method for emptying the cache. (docs; Python 3.2 release notes)

functools.partialmethod is like functools.partial, but the resulting object can be used as a descriptor (read: method). (docs; Python 3.4 release notes)

functools.singledispatch allows function overloading, based on the type of the first argument. (docs; Python 3.4 release notes; PEP 443)

functools.total_ordering is a class decorator that allows you to define only __eq__ and __lt__ (or any other) and defines the other comparison methods in terms of them. Note that since Python 3.0, __ne__ is automatically the inverse of __eq__ and doesn’t need defining. Note also that total_ordering doesn’t correctly support NotImplemented until Python 3.4. For an even easier way to do this, consider my classtools.keyed_ordering decorator. (docs; Python 3.2 release notes)

inspect.getattr_static fetches an attribute like getattr but avoids triggering dynamic lookup like @property. (docs; Python 3.2 release notes)

inspect.signature fetches the signature of a function as the new and more featureful Signature object. It also knows to follow the __wrapped__ attribute set by functools.wraps since Python 3.2, so it can see through well-behaved wrapper functions to the “original” signature. (docs; Python 3.3 release notes; PEP 362)

The logging module can, finally, use str.format-style string formatting by passing style='{' to Formatter. (docs; Python 3.2 release notes)

The logging module spits warnings and higher to stderr if logging hasn’t been otherwise configured. This means that if your app doesn’t use logging, but it uses a library that does, you’ll get actual output rather than the completely useless “No handlers could be found for logger ‘foo’”. (docs; Python 3.2 release notes)

os.scandir lists the contents of a directory while avoiding stat calls as much as possible, making it significantly faster. (docs; Python 3.5 release notes; PEP 471)

re.fullmatch checks for a match against the entire input string, not just a substring. (docs; Python 3.4 release notes)

reprlib.recursive_repr is a decorator for __repr__ implementations that can detect recursive calls to the same object and replace them with ..., just like the built-in structures. Believe it or not, reprlib is an existing module, though in Python 2 it was called repr. (docs; Python 3.2 release notes)

shutil.disk_usage returns disk space statistics for a given path with no fuss. (docs; Python 3.3 release notes)

shutil.get_terminal_size tries very hard to detect the size of the terminal window. (docs; Python 3.3 release notes)

subprocess.run is a new streamlined function that consolidates several other helpers in the subprocess module. It returns an object that describes the final state of the process, and it accepts arguments for a timeout, requiring that the process return success, and passing data as stdin. This is now the recommended way to run a single subprocess. (docs; Python 3.5 release notes)

tempfile.TemporaryDirectory is a context manager that creates a temporary directory, then destroys it and its contents at the end of the block. (docs; Python 3.2 release notes)

textwrap.indent can add an arbitrary prefix to every line in a string. (docs; Python 3.3 release notes)

time.monotonic returns the value of a monotonic clock — i.e., it will never go backwards. You should use this for measuring time durations within your program; using time.time() will produce garbage results if the system clock changes due to DST, a leap second, NTP, manual intervention, etc. (docs; Python 3.3 release notes; PEP 418)

time.perf_counter returns the value of the highest-resolution clock available, but is only suitable for measuring a short duration. (docs; Python 3.3 release notes; PEP 418)

time.process_time returns the total system and user CPU time for the process, excluding sleep. Note that the starting time is undefined, so only durations are meaningful. (docs; Python 3.3 release notes; PEP 418)

traceback.walk_stack and traceback.walk_tb are small helper functions that walk back along a stack or traceback, so you can use simple iteration rather than the slightly clumsier linked-list approach. (docs; Python 3.5 release notes)

types.MappingProxyType offers a read-only proxy to a dict. Since it holds a reference to the dict in C, you can return MappingProxyType(some_dict) to effectively create a read-only dict, as the original dict will be inaccessible from Python code. This is the same type used for the __dict__ of an immutable object. Note that this has existed in various forms for a while, but wasn’t publicly exposed or documented; see my module dictproxyhack for something that does its best to work on every Python version. (docs; Python 3.3 release notes)

types.SimpleNamespace is a blank type for sticking arbitrary unstructed attributes to. Previously, you would have to make a dummy subclass of object to do this. (docs; Python 3.3 release notes)

weakref.finalize allows you to add a finalizer function to an arbitrary (weakrefable) object from the “outside”, without needing to add a __del__. The finalize object will keep itself alive, so there’s no need to hold onto it. (docs; Python 3.4 release notes)

New modules with backports

These are less exciting, since they have backports on PyPI that work in Python 2 just as well. But they came from Python 3 development, so I credit Python 3 for them, just like I credit NASA for inventing the microwave.

asyncio is covered above, but it’s been backported as trollius for 2.6+, with the caveat that Pythons before 3.3 don’t have yield from and you have to use yield From(...) as a workaround. That caveat means that third-party asyncio libraries will almost certainly not work with trollius! For this and other reasons, the maintainer is no longer supporting it. Alas. Guess you’ll have to upgrade to Python 3, then.

enum finally provides an enumeration type, something which has long been desired in Python and solved in myriad ad-hoc ways. The variants become instances of a class, can be compared by identity, can be converted between names and values (but only explicitly), can have custom methods, and can implement special methods as usual. There’s even an IntEnum base class whose values end up as subclasses of int (!), making them perfectly compatible with code expecting integer constants. Enums have a surprising amount of power, far more than any approach I’ve seen before; I heartily recommend that you skim the examples in the documentation. Backported as enum34 for 2.4+. (docs; Python 3.4 release notes; PEP 435)

ipaddress offers types for representing IPv4 and IPv6 addresses and subnets. They can convert between several representations, perform a few set-like operations on subnets, identify special addresses, and so on. Backported as ipaddress for 2.6+. (There’s also a py2-ipaddress, but its handling of bytestrings differs from Python 3’s built-in module, which is likely to cause confusing compatibility problems.) (docs; Python 3.3 release notes; PEP 3144)

pathlib provides the Path type, representing a filesystem path that you can manipulate with methods rather than the mountain of functions in os.path. It also overloads / so you can do path / 'file.txt', which is kind of cool. PEP 519 intends to further improve interoperability of Paths with classic functions for the not-yet-released Python 3.6. Backported as pathlib2 for 2.6+; there’s also a pathlib, but it’s no longer maintained, and I don’t know what happened there. (docs; Python 3.4 release notes; PEP 428)

selectors (created as part of the work on asyncio) attempts to wrap select in a high-level interface that doesn’t make you want to claw your eyes out. A noble pursuit. Backported as selectors34 for 2.6+. (docs; Python 3.4 release notes)

statistics contains a number of high-precision statistical functions. Backported as backports.statistics for 2.6+. (docs; Python 3.4 release notes; PEP 450)

unittest.mock provides multiple ways for creating dummy objects, temporarily (with a context manager or decorator) replacing an object or some of its attributes, and verifying that some sequence of operations was performed on a dummy object. I’m not a huge fan of mocking so much that your tests end up mostly testing that your source code hasn’t changed, but if you have to deal with external resources or global state, some light use of unittest.mock can be very handy — even if you aren’t using the rest of unittest. Backported as mock for 2.6+. (docs; Python 3.3, but no release notes)

New modules without backports

Perhaps more exciting because they’re Python 3 exclusive! Perhaps less exciting because they’re necessarily related to plumbing.


faulthandler is a debugging aid that can dump a Python traceback during a segfault or other fatal signal. It can also be made to hook on an arbitrary signal, and can intervene even when Python code is deadlocked. You can use the default behavior with no effort by passing -X faulthandler on the command line, by setting the PYTHONFAULTHANDLER environment variable, or by using the module API manually.

I think -X itself is new as of Python 3.2, though it’s not mentioned in the release notes. It’s reserved for implementation-specific options; there are a few others defined for CPython, and the options can be retrieved from Python code via sys._xoptions.

Refs: docs; Python 3.3 release notes


importlib is the culmination of a whole lot of work, performed in multiple phases across numerous Python releases, to extend, formalize, and cleanly reimplement the entire import process.

I can’t possibly describe everything the import system can do and what Python versions support what parts of it. Suffice to say, it can do a lot of things: Python has built-in support for importing from zip files, and I’ve seen third-party import hooks that allow transparently importing modules written in another programming language.

If you want to mess around with writing your own custom importer, importlib has a ton of tools for helping you do that. It’s possible in Python 2, too, using the imp module, but that’s a lot rougher around the edges.

If not, the main thing of interest is the import_module function, which imports a module by name without all the really weird semantics of __import__. Seriously, don’t use __import__. It’s so weird. It probably doesn’t do what you think. importlib.import_module even exists in Python 2.7.

Refs: docs; Python 3.3 release notes; PEP 302?


tracemalloc is another debugging aid which tracks Python’s memory allocations. It can also compare two snapshots, showing how much memory has been allocated or released between two points in time, and who was responsible. If you have rampant memory use issues, this is probably more helpful than having Python check its own RSS.

Technically, tracemalloc can be used with Python 2.7… but that involves patching and recompiling Python, so I hesitate to call it a backport. Still, if you really need it, give it a whirl.

Refs: docs; Python 3.4 release notes; PEP 454


typing offers a standard way to declare type hints — the expected types of arguments and return values. Type hints are given using the function annotation syntax.

Python itself doesn’t do anything with the annotations, though they’re accessible and inspectable at runtime. An external tool like mypy can perform static type checking ahead of time, using these standard types. mypy is an existing project that predates typing (and works with Python 2), but the previous syntax relied on magic comments; typing formalizes the constructs and puts them in the standard library.

I haven’t actually used either the type hints or mypy myself, so I can’t comment on how helpful or intrusive they are. Give them a shot if they sound useful to you.

Refs: docs; Python 3.5 release notes; PEP 484

venv and ensurepip

I mean, yes, of course, virtualenv and pip are readily available in Python 2. The whole point of these is that they are bundled with Python, so you always have them at your fingertips and never have to worry about installing them yourself.

Installing Python should now give you pipX and pipX.Y commands automatically, corresponding to the latest stable release of pip when that Python version was first released. You’ll also get pyvenv, which is effectively just virtualenv.

There’s also a module interface: python -m ensurepip will install pip (hopefully not necessary), python -m pip runs pip with a specific Python version (a feature of pip and not new to the bundling), and python -m venv runs the bundled copy of virtualenv with a specific Python version.

There was a time where these were completely broken on Debian, because Debian strongly opposes vendoring (the rationale being that it’s easiest to push out updates if there’s only one copy of a library in the Debian package repository), so they just deleted ensurepip and venv? Which completely defeated the point of having them in the first place? I think this has been fixed by now, but it might still bite you if you’re on the Ubuntu 14.04 LTS.

Refs: ensurepip docs; pyvenv docs; Python 3.4 release notes; PEP 453


zipapp makes it easy to create executable zip applications, which have been a thing since 2.6 but have languished in obscurity. Well, no longer.

This wasn’t particularly difficult before: you just zip up some code, make sure there’s a __main__.py in the root, and pass it to Python. Optionally, you can set it executable and add a shebang line, since the ZIP format ignores any leading junk in the file. That’s basically all zipapp does. (It does not magically infer your dependencies and bundle them as well; you’re on your own there.)

I can’t find a backport, which is a little odd, since I don’t think this module does anything too special.

Refs: docs; Python 3.5 release notes; PEP 441

Miscellaneous nice enhancements

There were a lot of improvements to language semantics that don’t fit anywhere else above, but make me a little happier.

The interactive interpreter does tab-completion by default. I say “by default” because I’ve been told that it was supported before, but you had to do some kind of goat blood sacrifice to get it to work. Also, command history persists between runs. (docs; Python 3.4 release notes)

The -b command-line option produces a warning when calling str() on a bytes or bytearray, or when comparing text to bytes. -bb produces an error. (docs)

The -I command-like option runs Python in “isolated mode”: it ignores all PYTHON* environment variables and leaves the current directory and user site-packages directories off of sys.path. The idea is to use this when running a system script (or in the shebang line of a system script) to insulate it from any weird user-specific stuff. (docs; Python 3.4 release notes)

Functions and classes learned a __qualname__ attribute, which is a dotted name describing (lexically) where they were defined. For example, a method’s __name__ might be foo, but its __qualname__ would be something like SomeClass.foo. Similarly, a class or function defined within another function will list that containing function in its __qualname__. (docs; Python 3.3 release notes; PEP 3155)

Generators signal their end by raising StopIteration internally, but it was also possible to raise StopIteration directly within a generator — most notably, when calling next() on an exhausted iterator. This would cause the generator to end prematurely and silently. Now, raising StopIteration inside a generator will produce a warning, which will become a RuntimeError in Python 3.7. You can opt into the fatal behavior early with from __future__ import generator_stop. (Python 3.5 release notes; PEP 479)

Implicit namespace packages allow a package to span multiple directories. The most common example is a plugin system, foo.plugins.*, where plugins may come from multiple libraries, but all want to share the foo.plugins namespace. Previously, they would collide, and some sys.path tricks were necessary to make it work; now, support is built in. (This feature also allows you to have a regular package without an __init__.py, but I’d strongly recommend still having one.) (Python 3.3 release notes; PEP 420)

Object finalization behaves in less quirky ways when destroying an isolated reference cycle. Also, modules no longer have their contents changed to None during shutdown, which fixes a long-running type of error when a __del__ method tries to call, say, os.path.join() — if you were unlucky, os.path would have already have had its contents replaced with Nones, and you’d get an extremely confusing TypeError from trying to call a standard library function. (Python 3.4 release notes; PEP 442)

str.format_map is like str.format, but it accepts a mapping object directly (instead of having to flatten it with **kwargs). This allows some fancy things that weren’t previously possible, like passing a fake map that creates values on the fly based on the keys looked up in it. (docs; Python 3.2 release notes)

When a blocking system call is interrupted by a signal, it returns EINTR, indicating that the calling code should try the same system call again. In Python, this becomes OSError or InterruptedError. I have never in my life seen any C or Python code that actually deals with this correctly. Now, Python will do it for you: all the built-in and standard library functions that make use of system calls will automatically retry themselves when interrupted. (Python 3.5 release notes; PEP 475)

File descriptors created by Python code are now flagged “non-inheritable”, meaning they’re closed automatically when spawning a child process. (docs; Python 3.4 release notes; PEP 446)

A number of standard library functions now accept file descriptors in addition to paths. (docs; Python 3.3 release notes)

Several different OS and I/O exceptions were merged into a single and more fine-grained hierarchy, rooted at OSError. Code can now catch a specific subclass in most cases, rather than examine .errno. (docs; Python 3.3 release notes; PEP 3151)

ResourceWarning is a new kind of warning for issues with resource cleanup. One is produced if a file object is destroyed, but was never closed, which can cause issues on Windows or with garbage-collected Python implementations like PyPy; one is also produced if uncollectable objects still remain when Python shuts down, indicating some severe finalization problems. The warning is ignored by default, but can be enabled with -W default on the command line. (Python 3.2 release notes)

hasattr() only catches (and returns False for) AttributeErrors. Previously, any exception would be considered a sign that the attribute doesn’t exist, even though an unusual exception like an OSError usually means the attribute is computed dynamically, and that code is broken somehow. Now, exceptions other than AttributeError are allowed to propagate to the caller. (docs; Python 3.2 release notes)

Hash randomization is on by default, meaning that dict and set iteration order is different per Python runs. This protects against some DoS attacks, but more importantly, it spitefully forces you not to rely on incidental ordering. (docs; Python 3.3 release notes)

List comprehensions no longer leak their loop variables into the enclosing scope. (Python 3.0 release notes)

nonlocal allows writing to a variable in an enclosing (but non-global) scope. (docs; Python 3.0 release notes; PEP 3104)

Comparing objects of incompatible types now produces a TypeError, rather than using Python 2’s very silly fallback. (Python 3.0 release notes)

!= defaults to returning the opposite of ==. (Python 3.0 release notes)

Accessing a method as a class attribute now gives you a regular function, not an “unbound method” object. (Python 3.0 release notes)

The input builtin no longer performs an eval (!), removing a huge point of confusion for beginners. This is the behavior of raw_input in Python 2. (docs; Python 3.0 release notes; PEP 3111)

Fast and furious

These aren’t necessarily compelling, and they may not even make any appreciable difference for your code, but I think they’re interesting technically.

Objects’ __dict__s can now share their key storage internally. Instances of the same type generally have the same attribute names, so this provides a modest improvement in speed and memory usage for programs that create a lot of user-defined objects. (Python 3.3 release notes; PEP 412)

OrderedDict is now implemented in C, making it “4 to 100” (!) times faster. Note that the backport in the 2.7 standard library is pure Python. So, there’s a carrot. (Python 3.5 release notes)

The GIL was made more predictable. My understanding is that the old behavior was to yield after some number of Python bytecode operations, which could take wildly varying amounts of time; the new behavior yields after a given duration, by default 5ms. (Python 3.2 release notes)

The io library was rewritten in C, making it more fast. Again, the Python 2.7 implementation is pure Python. (Python 3.1 release notes)

Tuples and dicts containing only immutable objects — i.e., objects that cannot possibly contain circular references — are ignored by the garbage collector. This was backported to Python 2.7, too, but I thought it was super interesting. (Python 3.1 release notes)

That’s all I’ve got

Huff, puff.

I hope something here appeals to you as a reason to at least experiment with Python 3. It’s fun over here. Give it a try.


Post Syndicated from RealEnder original http://alex.stanev.org/blog/?p=335

Скоро не се бях занимавал активно с българските списъци за блокиране на нежелано съдържание. След като хората зад Adblock Plus ореваха, че насоките на страницата са доста out-of-date, преди малко направих доста обновления:

Премахната поддръжката за urlfilter.ini за Opera – това беше доста крив вариант за филтриране, който отдавна е deprecated;
Добавена информация за Adblock Plus за Android;
Коригирани статистиките за операционните системи – имаше проблем с визуализацията на Mac OS;
Обновен списъка с филтрите – след голямото коледно прочистване са добавени доста нови, подадени от потребители филтри.

Ако все още не сте изпробвали технологията, открийте как на Adblock за българи. Промените могат да се следят и в Twitter.

AdBlock за Opera

Post Syndicated from RealEnder original http://alex.stanev.org/blog/?p=281

Потребителите на Opera 11.10 вече могат да използват сравнително читаво разширение за блокиране на реклами – Opera AdBlock. Разширението все още е далече като функционалност от Adblock Plus, като не поддържа атрибутни селектори, опции за филтрите и изключения. Обновяването на филтрите също има проблеми, но авторът Léomike Hébert все още работи усилено по разширението.
Още с публикуването на addons.opera.com, статистиката за използването на българския лист се промени драстично, като Opera се изравни и в момента изпреварва Firefox. Не е ясно дали това е поради кривия update механизъм или реална работа, но това ще си проличи съвсем скоро.
Ако харесвате и използвате Opera, опитайте новото разширение!


Post Syndicated from RealEnder original http://alex.stanev.org/blog/?p=242

След почти три годишно ослушване, най-после днес намерих време да скова (свестен) сайт за българския adblock списък. Като подхванах, направих и интеграция с Twitter, където commit log-овете от кодовото хранилище заминават директно.
Другото ново нещо е филтъра за Opera, който се генерира автоматично от моя списък в комбинация с актуалния на Fanboy.
В процес на разработка са и статистики за използването(10x myst!) на списъка, но за тях – по-нататък.
Приятно сърфиране!