Tag Archives: jquery

Needed: Associate Front End Developer

Post Syndicated from Yev original https://www.backblaze.com/blog/needed-associate-front-end-developer/

Want to work at a company that helps customers in over 150 countries around the world protect the memories they hold dear? Do you want to challenge yourself with a business that serves consumers, SMBs, Enterprise, and developers?

If all that sounds interesting, you might be interested to know that Backblaze is looking for an Associate Front End Developer!

Backblaze is a 10 year old company. Providing great customer experiences is the “secret sauce” that enables us to successfully compete against some of technology’s giants. We’ll finish the year at ~$20MM ARR and are a profitable business. This is an opportunity to have your work shine at scale in one of the fastest growing verticals in tech — Cloud Storage.

You will utilize HTML, ReactJS, CSS and jQuery to develop intuitive, elegant user experiences. As a member of our Front End Dev team, you will work closely with our web development, software design, and marketing teams.

On a day to day basis, you must be able to convert image mockups to HTML or ReactJS – There’s some production work that needs to get done. But you will also be responsible for helping build out new features, rethink old processes, and enabling third party systems to empower our marketing, sales, and support teams.

Our Associate Front End Developer must be proficient in:

  • HTML, CSS, Javascript (ES5)
  • jQuery, Bootstrap (with responsive targets)
  • Understanding of ensuring cross-browser compatibility and browser security for features
  • Basic SEO principles and ensuring that applications will adhere to them
  • Familiarity with ES2015+, ReactJS, unit testing
  • Learning about third party marketing and sales tools through reading documentation. Our systems include Google Tag Manager, Google Analytics, Salesforce, and Hubspot.
  • React Flux, Redux, SASS, Node experience is a plus

We’re looking for someone that is:

  • Passionate about building friendly, easy to use Interfaces and APIs.
  • Likes to work closely with other engineers, support, and marketing to help customers.
  • Is comfortable working independently on a mutually agreed upon prioritization queue (we don’t micromanage, we do make sure tasks are reasonably defined and scoped).
  • Diligent with quality control. Backblaze prides itself on giving our team autonomy to get work done, do the right thing for our customers, and keep a pace that is sustainable over the long run. As such, we expect everyone that checks in code that is stable. We also have a small QA team that operates as a secondary check when needed.

Backblaze Employees Have:

  • Good attitude and willingness to do whatever it takes to get the job done.
  • Strong desire to work for a small fast, paced company.
  • Desire to learn and adapt to rapidly changing technologies and work environment.
  • Comfort with well behaved pets in the office.

This position is located in San Mateo, California. Regular attendance in the office is expected.

Backblaze is an Equal Opportunity Employer and we offer competitive salary and benefits, including our no policy vacation policy.

If this sounds like you…
Send an email to: jobscontact@backblaze.com with:

  1. Associate Front End Dev in the subject line
  2. Your resume attached
  3. An overview of your relevant experience

The post Needed: Associate Front End Developer appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Security updates for Wednesday

Post Syndicated from jake original https://lwn.net/Articles/739858/rss

Security updates have been issued by Arch Linux (roundcubemail), Debian (optipng, samba, and vlc), Fedora (compat-openssl10, fedpkg, git, jbig2dec, ldns, memcached, openssl, perl-Net-Ping-External, python-copr, python-XStatic-jquery-ui, rpkg, thunderbird, and xen), SUSE (tomcat), and Ubuntu (db, db4.8, db5.3, linux, linux-raspi2, linux-aws, linux-azure, linux-gcp, and samba).

Building a Multi-region Serverless Application with Amazon API Gateway and AWS Lambda

Post Syndicated from Stefano Buliani original https://aws.amazon.com/blogs/compute/building-a-multi-region-serverless-application-with-amazon-api-gateway-and-aws-lambda/

This post written by: Magnus Bjorkman – Solutions Architect

Many customers are looking to run their services at global scale, deploying their backend to multiple regions. In this post, we describe how to deploy a Serverless API into multiple regions and how to leverage Amazon Route 53 to route the traffic between regions. We use latency-based routing and health checks to achieve an active-active setup that can fail over between regions in case of an issue. We leverage the new regional API endpoint feature in Amazon API Gateway to make this a seamless process for the API client making the requests. This post does not cover the replication of your data, which is another aspect to consider when deploying applications across regions.

Solution overview

Currently, the default API endpoint type in API Gateway is the edge-optimized API endpoint, which enables clients to access an API through an Amazon CloudFront distribution. This typically improves connection time for geographically diverse clients. By default, a custom domain name is globally unique and the edge-optimized API endpoint would invoke a Lambda function in a single region in the case of Lambda integration. You can’t use this type of endpoint with a Route 53 active-active setup and fail-over.

The new regional API endpoint in API Gateway moves the API endpoint into the region and the custom domain name is unique per region. This makes it possible to run a full copy of an API in each region and then use Route 53 to use an active-active setup and failover. The following diagram shows how you do this:

Active/active multi region architecture

  • Deploy your Rest API stack, consisting of API Gateway and Lambda, in two regions, such as us-east-1 and us-west-2.
  • Choose the regional API endpoint type for your API.
  • Create a custom domain name and choose the regional API endpoint type for that one as well. In both regions, you are configuring the custom domain name to be the same, for example, helloworldapi.replacewithyourcompanyname.com
  • Use the host name of the custom domain names from each region, for example, xxxxxx.execute-api.us-east-1.amazonaws.com and xxxxxx.execute-api.us-west-2.amazonaws.com, to configure record sets in Route 53 for your client-facing domain name, for example, helloworldapi.replacewithyourcompanyname.com

The above solution provides an active-active setup for your API across the two regions, but you are not doing failover yet. For that to work, set up a health check in Route 53:

Route 53 Health Check

A Route 53 health check must have an endpoint to call to check the health of a service. You could do a simple ping of your actual Rest API methods, but instead provide a specific method on your Rest API that does a deep ping. That is, it is a Lambda function that checks the status of all the dependencies.

In the case of the Hello World API, you don’t have any other dependencies. In a real-world scenario, you could check on dependencies as databases, other APIs, and external dependencies. Route 53 health checks themselves cannot use your custom domain name endpoint’s DNS address, so you are going to directly call the API endpoints via their region unique endpoint’s DNS address.

Walkthrough

The following sections describe how to set up this solution. You can find the complete solution at the blog-multi-region-serverless-service GitHub repo. Clone or download the repository locally to be able to do the setup as described.

Prerequisites

You need the following resources to set up the solution described in this post:

  • AWS CLI
  • An S3 bucket in each region in which to deploy the solution, which can be used by the AWS Serverless Application Model (SAM). You can use the following CloudFormation templates to create buckets in us-east-1 and us-west-2:
    • us-east-1:
    • us-west-2:
  • A hosted zone registered in Amazon Route 53. This is used for defining the domain name of your API endpoint, for example, helloworldapi.replacewithyourcompanyname.com. You can use a third-party domain name registrar and then configure the DNS in Amazon Route 53, or you can purchase a domain directly from Amazon Route 53.

Deploy API with health checks in two regions

Start by creating a small “Hello World” Lambda function that sends back a message in the region in which it has been deployed.


"""Return message."""
import logging

logging.basicConfig()
logger = logging.getLogger()
logger.setLevel(logging.INFO)

def lambda_handler(event, context):
    """Lambda handler for getting the hello world message."""

    region = context.invoked_function_arn.split(':')[3]

    logger.info("message: " + "Hello from " + region)
    
    return {
		"message": "Hello from " + region
    }

Also create a Lambda function for doing a health check that returns a value based on another environment variable (either “ok” or “fail”) to allow for ease of testing:


"""Return health."""
import logging
import os

logging.basicConfig()
logger = logging.getLogger()
logger.setLevel(logging.INFO)

def lambda_handler(event, context):
    """Lambda handler for getting the health."""

    logger.info("status: " + os.environ['STATUS'])
    
    return {
		"status": os.environ['STATUS']
    }

Deploy both of these using an AWS Serverless Application Model (SAM) template. SAM is a CloudFormation extension that is optimized for serverless, and provides a standard way to create a complete serverless application. You can find the full helloworld-sam.yaml template in the blog-multi-region-serverless-service GitHub repo.

A few things to highlight:

  • You are using inline Swagger to define your API so you can substitute the current region in the x-amazon-apigateway-integration section.
  • Most of the Swagger template covers CORS to allow you to test this from a browser.
  • You are also using substitution to populate the environment variable used by the “Hello World” method with the region into which it is being deployed.

The Swagger allows you to use the same SAM template in both regions.

You can only use SAM from the AWS CLI, so do the following from the command prompt. First, deploy the SAM template in us-east-1 with the following commands, replacing “<your bucket in us-east-1>” with a bucket in your account:


> cd helloworld-api
> aws cloudformation package --template-file helloworld-sam.yaml --output-template-file /tmp/cf-helloworld-sam.yaml --s3-bucket <your bucket in us-east-1> --region us-east-1
> aws cloudformation deploy --template-file /tmp/cf-helloworld-sam.yaml --stack-name multiregionhelloworld --capabilities CAPABILITY_IAM --region us-east-1

Second, do the same in us-west-2:


> aws cloudformation package --template-file helloworld-sam.yaml --output-template-file /tmp/cf-helloworld-sam.yaml --s3-bucket <your bucket in us-west-2> --region us-west-2
> aws cloudformation deploy --template-file /tmp/cf-helloworld-sam.yaml --stack-name multiregionhelloworld --capabilities CAPABILITY_IAM --region us-west-2

The API was created with the default endpoint type of Edge Optimized. Switch it to Regional. In the Amazon API Gateway console, select the API that you just created and choose the wheel-icon to edit it.

API Gateway edit API settings

In the edit screen, select the Regional endpoint type and save the API. Do the same in both regions.

Grab the URL for the API in the console by navigating to the method in the prod stage.

API Gateway endpoint link

You can now test this with curl:


> curl https://2wkt1cxxxx.execute-api.us-west-2.amazonaws.com/prod/helloworld
{"message": "Hello from us-west-2"}

Write down the domain name for the URL in each region (for example, 2wkt1cxxxx.execute-api.us-west-2.amazonaws.com), as you need that later when you deploy the Route 53 setup.

Create the custom domain name

Next, create an Amazon API Gateway custom domain name endpoint. As part of using this feature, you must have a hosted zone and domain available to use in Route 53 as well as an SSL certificate that you use with your specific domain name.

You can create the SSL certificate by using AWS Certificate Manager. In the ACM console, choose Get started (if you have no existing certificates) or Request a certificate. Fill out the form with the domain name to use for the custom domain name endpoint, which is the same across the two regions:

Amazon Certificate Manager request new certificate

Go through the remaining steps and validate the certificate for each region before moving on.

You are now ready to create the endpoints. In the Amazon API Gateway console, choose Custom Domain Names, Create Custom Domain Name.

API Gateway create custom domain name

A few things to highlight:

  • The domain name is the same as what you requested earlier through ACM.
  • The endpoint configuration should be regional.
  • Select the ACM Certificate that you created earlier.
  • You need to create a base path mapping that connects back to your earlier API Gateway endpoint. Set the base path to v1 so you can version your API, and then select the API and the prod stage.

Choose Save. You should see your newly created custom domain name:

API Gateway custom domain setup

Note the value for Target Domain Name as you need that for the next step. Do this for both regions.

Deploy Route 53 setup

Use the global Route 53 service to provide DNS lookup for the Rest API, distributing the traffic in an active-active setup based on latency. You can find the full CloudFormation template in the blog-multi-region-serverless-service GitHub repo.

The template sets up health checks, for example, for us-east-1:


HealthcheckRegion1:
  Type: "AWS::Route53::HealthCheck"
  Properties:
    HealthCheckConfig:
      Port: "443"
      Type: "HTTPS_STR_MATCH"
      SearchString: "ok"
      ResourcePath: "/prod/healthcheck"
      FullyQualifiedDomainName: !Ref Region1HealthEndpoint
      RequestInterval: "30"
      FailureThreshold: "2"

Use the health check when you set up the record set and the latency routing, for example, for us-east-1:


Region1EndpointRecord:
  Type: AWS::Route53::RecordSet
  Properties:
    Region: us-east-1
    HealthCheckId: !Ref HealthcheckRegion1
    SetIdentifier: "endpoint-region1"
    HostedZoneId: !Ref HostedZoneId
    Name: !Ref MultiregionEndpoint
    Type: CNAME
    TTL: 60
    ResourceRecords:
      - !Ref Region1Endpoint

You can create the stack by using the following link, copying in the domain names from the previous section, your existing hosted zone name, and the main domain name that is created (for example, hellowordapi.replacewithyourcompanyname.com):

The following screenshot shows what the parameters might look like:
Serverless multi region Route 53 health check

Specifically, the domain names that you collected earlier would map according to following:

  • The domain names from the API Gateway “prod”-stage go into Region1HealthEndpoint and Region2HealthEndpoint.
  • The domain names from the custom domain name’s target domain name goes into Region1Endpoint and Region2Endpoint.

Using the Rest API from server-side applications

You are now ready to use your setup. First, demonstrate the use of the API from server-side clients. You can demonstrate this by using curl from the command line:


> curl https://hellowordapi.replacewithyourcompanyname.com/v1/helloworld/
{"message": "Hello from us-east-1"}

Testing failover of Rest API in browser

Here’s how you can use this from the browser and test the failover. Find all of the files for this test in the browser-client folder of the blog-multi-region-serverless-service GitHub repo.

Use this html file:


<!DOCTYPE HTML>
<html>
<head>
    <meta charset="utf-8"/>
    <meta http-equiv="X-UA-Compatible" content="IE=edge"/>
    <meta name="viewport" content="width=device-width, initial-scale=1"/>
    <title>Multi-Region Client</title>
</head>
<body>
<div>
   <h1>Test Client</h1>

    <p id="client_result">

    </p>

    <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script>
    <script src="settings.js"></script>
    <script src="client.js"></script>
</body>
</html>

The html file uses this JavaScript file to repeatedly call the API and print the history of messages:


var messageHistory = "";

(function call_service() {

   $.ajax({
      url: helloworldMultiregionendpoint+'v1/helloworld/',
      dataType: "json",
      cache: false,
      success: function(data) {
         messageHistory+="<p>"+data['message']+"</p>";
         $('#client_result').html(messageHistory);
      },
      complete: function() {
         // Schedule the next request when the current one's complete
         setTimeout(call_service, 10000);
      },
      error: function(xhr, status, error) {
         $('#client_result').html('ERROR: '+status);
      }
   });

})();

Also, make sure to update the settings in settings.js to match with the API Gateway endpoints for the DNS-proxy and the multi-regional endpoint for the Hello World API: var helloworldMultiregionendpoint = "https://hellowordapi.replacewithyourcompanyname.com/";

You can now open the HTML file in the browser (you can do this directly from the file system) and you should see something like the following screenshot:

Serverless multi region browser test

You can test failover by changing the environment variable in your health check Lambda function. In the Lambda console, select your health check function and scroll down to the Environment variables section. For the STATUS key, modify the value to fail.

Lambda update environment variable

You should see the region switch in the test client:

Serverless multi region broker test switchover

During an emulated failure like this, the browser might take some additional time to switch over due to connection keep-alive functionality. If you are using a browser like Chrome, you can kill all the connections to see a more immediate fail-over: chrome://net-internals/#sockets

Summary

You have implemented a simple way to do multi-regional serverless applications that fail over seamlessly between regions, either being accessed from the browser or from other applications/services. You achieved this by using the capabilities of Amazon Route 53 to do latency based routing and health checks for fail-over. You unlocked the use of these features in a serverless application by leveraging the new regional endpoint feature of Amazon API Gateway.

The setup was fully scripted using CloudFormation, the AWS Serverless Application Model (SAM), and the AWS CLI, and it can be integrated into deployment tools to push the code across the regions to make sure it is available in all the needed regions. For more information about cross-region deployments, see Building a Cross-Region/Cross-Account Code Deployment Solution on AWS on the AWS DevOps blog.

JavaScript got better while I wasn’t looking

Post Syndicated from Eevee original https://eev.ee/blog/2017/10/07/javascript-got-better-while-i-wasnt-looking/

IndustrialRobot has generously donated in order to inquire:

In the last few years there seems to have been a lot of activity with adding emojis to Unicode. Has there been an equal effort to add ‘real’ languages/glyph systems/etc?

And as always, if you don’t have anything to say on that topic, feel free to choose your own. :p

Yes.

I mean, each release of Unicode lists major new additions right at the top — Unicode 10, Unicode 9, Unicode 8, etc. They also keep fastidious notes, so you can also dig into how and why these new scripts came from, by reading e.g. the proposal for the addition of Zanabazar Square. I don’t think I have much to add here; I’m not a real linguist, I only play one on TV.

So with that out of the way, here’s something completely different!

A brief history of JavaScript

JavaScript was created in seven days, about eight thousand years ago. It was pretty rough, and it stayed rough for most of its life. But that was fine, because no one used it for anything besides having a trail of sparkles follow your mouse on their Xanga profile.

Then people discovered you could actually do a handful of useful things with JavaScript, and it saw a sharp uptick in usage. Alas, it stayed pretty rough. So we came up with polyfills and jQuerys and all kinds of miscellaneous things that tried to smooth over the rough parts, to varying degrees of success.

And… that’s it. That’s pretty much how things stayed for a while.


I have complicated feelings about JavaScript. I don’t hate it… but I certainly don’t enjoy it, either. It has some pretty neat ideas, like prototypical inheritance and “everything is a value”, but it buries them under a pile of annoying quirks and a woefully inadequate standard library. The DOM APIs don’t make things much better — they seem to be designed as though the target language were Java, rarely taking advantage of any interesting JavaScript features. And the places where the APIs overlap with the language are a hilarious mess: I have to check documentation every single time I use any API that returns a set of things, because there are at least three totally different conventions for handling that and I can’t keep them straight.

The funny thing is that I’ve been fairly happy to work with Lua, even though it shares most of the same obvious quirks as JavaScript. Both languages are weakly typed; both treat nonexistent variables and keys as simply false values, rather than errors; both have a single data structure that doubles as both a list and a map; both use 64-bit floating-point as their only numeric type (though Lua added integers very recently); both lack a standard object model; both have very tiny standard libraries. Hell, Lua doesn’t even have exceptions, not really — you have to fake them in much the same style as Perl.

And yet none of this bothers me nearly as much in Lua. The differences between the languages are very subtle, but combined they make a huge impact.

  • Lua has separate operators for addition and concatenation, so + is never ambiguous. It also has printf-style string formatting in the standard library.

  • Lua’s method calls are syntactic sugar: foo:bar() just means foo.bar(foo). Lua doesn’t even have a special this or self value; the invocant just becomes the first argument. In contrast, JavaScript invokes some hand-waved magic to set its contextual this variable, which has led to no end of confusion.

  • Lua has an iteration protocol, as well as built-in iterators for dealing with list-style or map-style data. JavaScript has a special dedicated Array type and clumsy built-in iteration syntax.

  • Lua has operator overloading and (surprisingly flexible) module importing.

  • Lua allows the keys of a map to be any value (though non-scalars are always compared by identity). JavaScript implicitly converts keys to strings — and since there’s no operator overloading, there’s no way to natively fix this.

These are fairly minor differences, in the grand scheme of language design. And almost every feature in Lua is implemented in a ridiculously simple way; in fact the entire language is described in complete detail in a single web page. So writing JavaScript is always frustrating for me: the language is so close to being much more ergonomic, and yet, it isn’t.

Or, so I thought. As it turns out, while I’ve been off doing other stuff for a few years, browser vendors have been implementing all this pie-in-the-sky stuff from “ES5” and “ES6”, whatever those are. People even upgrade their browsers now. Lo and behold, the last time I went to write JavaScript, I found out that a number of papercuts had actually been solved, and the solutions were sufficiently widely available that I could actually use them in web code.

The weird thing is that I do hear a lot about JavaScript, but the feature I’ve seen raved the most about by far is probably… built-in types for working with arrays of bytes? That’s cool and all, but not exactly the most pressing concern for me.

Anyway, if you also haven’t been keeping tabs on the world of JavaScript, here are some things we missed.

let

MDN docs — supported in Firefox 44, Chrome 41, IE 11, Safari 10

I’m pretty sure I first saw let over a decade ago. Firefox has supported it for ages, but you actually had to opt in by specifying JavaScript version 1.7. Remember JavaScript versions? You know, from back in the days when people actually suggested you write stuff like this:

1
<SCRIPT LANGUAGE="JavaScript1.2" TYPE="text/javascript">

Yikes.

Anyway, so, let declares a variable — but scoped to the immediately containing block, unlike var, which scopes to the innermost function. The trouble with var was that it was very easy to make misleading:

1
2
3
4
5
6
// foo exists here
while (true) {
    var foo = ...;
    ...
}
// foo exists here too

If you reused the same temporary variable name in a different block, or if you expected to be shadowing an outer foo, or if you were trying to do something with creating closures in a loop, this would cause you some trouble.

But no more, because let actually scopes the way it looks like it should, the way variable declarations do in C and friends. As an added bonus, if you refer to a variable declared with let outside of where it’s valid, you’ll get a ReferenceError instead of a silent undefined value. Hooray!

There’s one other interesting quirk to let that I can’t find explicitly documented. Consider:

1
2
3
4
5
6
7
let closures = [];
for (let i = 0; i < 4; i++) {
    closures.push(function() { console.log(i); });
}
for (let j = 0; j < closures.length; j++) {
    closures[j]();
}

If this code had used var i, then it would print 4 four times, because the function-scoped var i means each closure is sharing the same i, whose final value is 4. With let, the output is 0 1 2 3, as you might expect, because each run through the loop gets its own i.

But wait, hang on.

The semantics of a C-style for are that the first expression is only evaluated once, at the very beginning. So there’s only one let i. In fact, it makes no sense for each run through the loop to have a distinct i, because the whole idea of the loop is to modify i each time with i++.

I assume this is simply a special case, since it’s what everyone expects. We expect it so much that I can’t find anyone pointing out that the usual explanation for why it works makes no sense. It has the interesting side effect that for no longer de-sugars perfectly to a while, since this will print all 4s:

1
2
3
4
5
6
7
8
9
closures = [];
let i = 0;
while (i < 4) {
    closures.push(function() { console.log(i); });
    i++;
}
for (let j = 0; j < closures.length; j++) {
    closures[j]();
}

This isn’t a problem — I’m glad let works this way! — it just stands out to me as interesting. Lua doesn’t need a special case here, since it uses an iterator protocol that produces values rather than mutating a visible state variable, so there’s no problem with having the loop variable be truly distinct on each run through the loop.

Classes

MDN docs — supported in Firefox 45, Chrome 42, Safari 9, Edge 13

Prototypical inheritance is pretty cool. The way JavaScript presents it is a little bit opaque, unfortunately, which seems to confuse a lot of people. JavaScript gives you enough functionality to make it work, and even makes it sound like a first-class feature with a property outright called prototype… but to actually use it, you have to do a bunch of weird stuff that doesn’t much look like constructing an object or type.

The funny thing is, people with almost any background get along with Python just fine, and Python uses prototypical inheritance! Nobody ever seems to notice this, because Python tucks it neatly behind a class block that works enough like a Java-style class. (Python also handles inheritance without using the prototype, so it’s a little different… but I digress. Maybe in another post.)

The point is, there’s nothing fundamentally wrong with how JavaScript handles objects; the ergonomics are just terrible.

Lo! They finally added a class keyword. Or, rather, they finally made the class keyword do something; it’s been reserved this entire time.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
class Vector {
    constructor(x, y) {
        this.x = x;
        this.y = y;
    }

    get magnitude() {
        return Math.sqrt(this.x * this.x + this.y * this.y);
    }

    dot(other) {
        return this.x * other.x + this.y * other.y;
    }
}

This is all just sugar for existing features: creating a Vector function to act as the constructor, assigning a function to Vector.prototype.dot, and whatever it is you do to make a property. (Oh, there are properties. I’ll get to that in a bit.)

The class block can be used as an expression, with or without a name. It also supports prototypical inheritance with an extends clause and has a super pseudo-value for superclass calls.

It’s a little weird that the inside of the class block has its own special syntax, with function omitted and whatnot, but honestly you’d have a hard time making a class block without special syntax.

One severe omission here is that you can’t declare values inside the block, i.e. you can’t just drop a bar = 3; in there if you want all your objects to share a default attribute. The workaround is to just do this.bar = 3; inside the constructor, but I find that unsatisfying, since it defeats half the point of using prototypes.

Properties

MDN docs — supported in Firefox 4, Chrome 5, IE 9, Safari 5.1

JavaScript historically didn’t have a way to intercept attribute access, which is a travesty. And by “intercept attribute access”, I mean that you couldn’t design a value foo such that evaluating foo.bar runs some code you wrote.

Exciting news: now it does. Or, rather, you can intercept specific attributes, like in the class example above. The above magnitude definition is equivalent to:

1
2
3
4
5
6
7
Object.defineProperty(Vector.prototype, 'magnitude', {
    configurable: true,
    enumerable: true,
    get: function() {
        return Math.sqrt(this.x * this.x + this.y * this.y);
    },
});

Beautiful.

And what even are these configurable and enumerable things? It seems that every single key on every single object now has its own set of three Boolean twiddles:

  • configurable means the property itself can be reconfigured with another call to Object.defineProperty.
  • enumerable means the property appears in for..in or Object.keys().
  • writable means the property value can be changed, which only applies to properties with real values rather than accessor functions.

The incredibly wild thing is that for properties defined by Object.defineProperty, configurable and enumerable default to false, meaning that by default accessor properties are immutable and invisible. Super weird.

Nice to have, though. And luckily, it turns out the same syntax as in class also works in object literals.

1
2
3
4
5
6
Vector.prototype = {
    get magnitude() {
        return Math.sqrt(this.x * this.x + this.y * this.y);
    },
    ...
};

Alas, I’m not aware of a way to intercept arbitrary attribute access.

Another feature along the same lines is Object.seal(), which marks all of an object’s properties as non-configurable and prevents any new properties from being added to the object. The object is still mutable, but its “shape” can’t be changed. And of course you can just make the object completely immutable if you want, via setting all its properties non-writable, or just using Object.freeze().

I have mixed feelings about the ability to irrevocably change something about a dynamic runtime. It would certainly solve some gripes of former Haskell-minded colleagues, and I don’t have any compelling argument against it, but it feels like it violates some unwritten contract about dynamic languages — surely any structural change made by user code should also be able to be undone by user code?

Slurpy arguments

MDN docs — supported in Firefox 15, Chrome 47, Edge 12, Safari 10

Officially this feature is called “rest parameters”, but that’s a terrible name, no one cares about “arguments” vs “parameters”, and “slurpy” is a good word. Bless you, Perl.

1
2
3
function foo(a, b, ...args) {
    // ...
}

Now you can call foo with as many arguments as you want, and every argument after the second will be collected in args as a regular array.

You can also do the reverse with the spread operator:

1
2
3
4
5
let args = [];
args.push(1);
args.push(2);
args.push(3);
foo(...args);

It even works in array literals, even multiple times:

1
2
let args2 = [...args, ...args];
console.log(args2);  // [1, 2, 3, 1, 2, 3]

Apparently there’s also a proposal for allowing the same thing with objects inside object literals.

Default arguments

MDN docs — supported in Firefox 15, Chrome 49, Edge 14, Safari 10

Yes, arguments can have defaults now. It’s more like Sass than Python — default expressions are evaluated once per call, and later default expressions can refer to earlier arguments. I don’t know how I feel about that but whatever.

1
2
3
function foo(n = 1, m = n + 1, list = []) {
    ...
}

Also, unlike Python, you can have an argument with a default and follow it with an argument without a default, since the default default (!) is and always has been defined as undefined. Er, let me just write it out.

1
2
3
function bar(a = 5, b) {
    ...
}

Arrow functions

MDN docs — supported in Firefox 22, Chrome 45, Edge 12, Safari 10

Perhaps the most humble improvement is the arrow function. It’s a slightly shorter way to write an anonymous function.

1
2
3
(a, b, c) => { ... }
a => { ... }
() => { ... }

An arrow function does not set this or some other magical values, so you can safely use an arrow function as a quick closure inside a method without having to rebind this. Hooray!

Otherwise, arrow functions act pretty much like regular functions; you can even use all the features of regular function signatures.

Arrow functions are particularly nice in combination with all the combinator-style array functions that were added a while ago, like Array.forEach.

1
2
3
[7, 8, 9].forEach(value => {
    console.log(value);
});

Symbol

MDN docs — supported in Firefox 36, Chrome 38, Edge 12, Safari 9

This isn’t quite what I’d call an exciting feature, but it’s necessary for explaining the next one. It’s actually… extremely weird.

symbol is a new kind of primitive (like number and string), not an object (like, er, Number and String). A symbol is created with Symbol('foo'). No, not new Symbol('foo'); that throws a TypeError, for, uh, some reason.

The only point of a symbol is as a unique key. You see, symbols have one very special property: they can be used as object keys, and will not be stringified. Remember, only strings can be keys in JavaScript — even the indices of an array are, semantically speaking, still strings. Symbols are a new exception to this rule.

Also, like other objects, two symbols don’t compare equal to each other: Symbol('foo') != Symbol('foo').

The result is that symbols solve one of the problems that plauges most object systems, something I’ve talked about before: interfaces. Since an interface might be implemented by any arbitrary type, and any arbitrary type might want to implement any number of arbitrary interfaces, all the method names on an interface are effectively part of a single global namespace.

I think I need to take a moment to justify that. If you have IFoo and IBar, both with a method called method, and you want to implement both on the same type… you have a problem. Because most object systems consider “interface” to mean “I have a method called method, with no way to say which interface’s method you mean. This is a hard problem to avoid, because IFoo and IBar might not even come from the same library. Occasionally languages offer a clumsy way to “rename” one method or the other, but the most common approach seems to be for interface designers to avoid names that sound “too common”. You end up with redundant mouthfuls like IFoo.foo_method.

This incredibly sucks, and the only languages I’m aware of that avoid the problem are the ML family and Rust. In Rust, you define all the methods for a particular trait (interface) in a separate block, away from the type’s “own” methods. It’s pretty slick. You can still do obj.method(), and as long as there’s only one method among all the available traits, you’ll get that one. If not, there’s syntax for explicitly saying which trait you mean, which I can’t remember because I’ve never had to use it.

Symbols are JavaScript’s answer to this problem. If you want to define some interface, you can name its methods with symbols, which are guaranteed to be unique. You just have to make sure you keep the symbol around somewhere accessible so other people can actually use it. (Or… not?)

The interesting thing is that JavaScript now has several of its own symbols built in, allowing user objects to implement features that were previously reserved for built-in types. For example, you can use the Symbol.hasInstance symbol — which is simply where the language is storing an existing symbol and is not the same as Symbol('hasInstance')! — to override instanceof:

1
2
3
4
5
6
7
8
// oh my god don't do this though
class EvenNumber {
    static [Symbol.hasInstance](obj) {
        return obj % 2 == 0;
    }
}
console.log(2 instanceof EvenNumber);  // true
console.log(3 instanceof EvenNumber);  // false

Oh, and those brackets around Symbol.hasInstance are a sort of reverse-quoting — they indicate an expression to use where the language would normally expect a literal identifier. I think they work as object keys, too, and maybe some other places.

The equivalent in Python is to implement a method called __instancecheck__, a name which is not special in any way except that Python has reserved all method names of the form __foo__. That’s great for Python, but doesn’t really help user code. JavaScript has actually outclassed (ho ho) Python here.

Of course, obj[BobNamespace.some_method]() is not the prettiest way to call an interface method, so it’s not perfect. I imagine this would be best implemented in user code by exposing a polymorphic function, similar to how Python’s len(obj) pretty much just calls obj.__len__().

I only bring this up because it’s the plumbing behind one of the most incredible things in JavaScript that I didn’t even know about until I started writing this post. I’m so excited oh my gosh. Are you ready? It’s:

Iteration protocol

MDN docs — supported in Firefox 27, Chrome 39, Safari 10; still experimental in Edge

Yes! Amazing! JavaScript has first-class support for iteration! I can’t even believe this.

It works pretty much how you’d expect, or at least, how I’d expect. You give your object a method called Symbol.iterator, and that returns an iterator.

What’s an iterator? It’s an object with a next() method that returns the next value and whether the iterator is exhausted.

Wait, wait, wait a second. Hang on. The method is called next? Really? You didn’t go for Symbol.next? Python 2 did exactly the same thing, then realized its mistake and changed it to __next__ in Python 3. Why did you do this?

Well, anyway. My go-to test of an iterator protocol is how hard it is to write an equivalent to Python’s enumerate(), which takes a list and iterates over its values and their indices. In Python it looks like this:

1
2
3
4
5
for i, value in enumerate(['one', 'two', 'three']):
    print(i, value)
# 0 one
# 1 two
# 2 three

It’s super nice to have, and I’m always amazed when languages with “strong” “support” for iteration don’t have it. Like, C# doesn’t. So if you want to iterate over a list but also need indices, you need to fall back to a C-style for loop. And if you want to iterate over a lazy or arbitrary iterable but also need indices, you need to track it yourself with a counter. Ridiculous.

Here’s my attempt at building it in JavaScript.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
function enumerate(iterable) {
    // Return a new iter*able* object with a Symbol.iterator method that
    // returns an iterator.
    return {
        [Symbol.iterator]: function() {
            let iterator = iterable[Symbol.iterator]();
            let i = 0;

            return {
                next: function() {
                    let nextval = iterator.next();
                    if (! nextval.done) {
                        nextval.value = [i, nextval.value];
                        i++;
                    }
                    return nextval;
                },
            };
        },
    };
}
for (let [i, value] of enumerate(['one', 'two', 'three'])) {
    console.log(i, value);
}
// 0 one
// 1 two
// 2 three

Incidentally, for..of (which iterates over a sequence, unlike for..in which iterates over keys — obviously) is finally supported in Edge 12. Hallelujah.

Oh, and let [i, value] is destructuring assignment, which is also a thing now and works with objects as well. You can even use the splat operator with it! Like Python! (And you can use it in function signatures! Like Python! Wait, no, Python decided that was terrible and removed it in 3…)

1
let [x, y, ...others] = ['apple', 'orange', 'cherry', 'banana'];

It’s a Halloween miracle. 🎃

Generators

MDN docs — supported in Firefox 26, Chrome 39, Edge 13, Safari 10

That’s right, JavaScript has goddamn generators now. It’s basically just copying Python and adding a lot of superfluous punctuation everywhere. Not that I’m complaining.

Also, generators are themselves iterable, so I’m going to cut to the chase and rewrite my enumerate() with a generator.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
function enumerate(iterable) {
    return {
        [Symbol.iterator]: function*() {
            let i = 0;
            for (let value of iterable) {
                yield [i, value];
                i++;
            }
        },
    };
}
for (let [i, value] of enumerate(['one', 'two', 'three'])) {
    console.log(i, value);
}
// 0 one
// 1 two
// 2 three

Amazing. function* is a pretty strange choice of syntax, but whatever? I guess it also lets them make yield only act as a keyword inside a generator, for ultimate backwards compatibility.

JavaScript generators support everything Python generators do: yield* yields every item from a subsequence, like Python’s yield from; generators can return final values; you can pass values back into the generator if you iterate it by hand. No, really, I wasn’t kidding, it’s basically just copying Python. It’s great. You could now built asyncio in JavaScript!

In fact, they did that! JavaScript now has async and await. An async function returns a Promise, which is also a built-in type now. Amazing.

Sets and maps

MDN docs for MapMDN docs for Set — supported in Firefox 13, Chrome 38, IE 11, Safari 7.1

I did not save the best for last. This is much less exciting than generators. But still exciting.

The only data structure in JavaScript is the object, a map where the strings are keys. (Or now, also symbols, I guess.) That means you can’t readily use custom values as keys, nor simulate a set of arbitrary objects. And you have to worry about people mucking with Object.prototype, yikes.

But now, there’s Map and Set! Wow.

Unfortunately, because JavaScript, Map couldn’t use the indexing operators without losing the ability to have methods, so you have to use a boring old method-based API. But Map has convenient methods that plain objects don’t, like entries() to iterate over pairs of keys and values. In fact, you can use a map with for..of to get key/value pairs. So that’s nice.

Perhaps more interesting, there’s also now a WeakMap and WeakSet, where the keys are weak references. I don’t think JavaScript had any way to do weak references before this, so that’s pretty slick. There’s no obvious way to hold a weak value, but I guess you could substitute a WeakSet with only one item.

Template literals

MDN docs — supported in Firefox 34, Chrome 41, Edge 12, Safari 9

Template literals are JavaScript’s answer to string interpolation, which has historically been a huge pain in the ass because it doesn’t even have string formatting in the standard library.

They’re just strings delimited by backticks instead of quotes. They can span multiple lines and contain expressions.

1
2
console.log(`one plus
two is ${1 + 2}`);

Someone decided it would be a good idea to allow nesting more sets of backticks inside a ${} expression, so, good luck to syntax highlighters.

However, someone also had the most incredible idea ever, which was to add syntax allowing user code to do the interpolation — so you can do custom escaping, when absolutely necessary, which is virtually never, because “escaping” means you’re building a structured format by slopping strings together willy-nilly instead of using some API that works with the structure.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
// OF COURSE, YOU SHOULDN'T BE DOING THIS ANYWAY; YOU SHOULD BUILD HTML WITH
// THE DOM API AND USE .textContent FOR LITERAL TEXT.  BUT AS AN EXAMPLE:
function html(literals, ...values) {
    let ret = [];
    literals.forEach((literal, i) => {
        if (i > 0) {
            // Is there seriously still not a built-in function for doing this?
            // Well, probably because you SHOULDN'T BE DOING IT
            ret.push(values[i - 1]
                .replace(/&/g, '&amp;')
                .replace(/</g, '&lt;')
                .replace(/>/g, '&gt;')
                .replace(/"/g, '&quot;')
                .replace(/'/g, '&apos;'));
        }
        ret.push(literal);
    });
    return ret.join('');
}
let username = 'Bob<script>';
let result = html`<b>Hello, ${username}!</b>`;
console.log(result);
// <b>Hello, Bob&lt;script&gt;!</b>

It’s a shame this feature is in JavaScript, the language where you are least likely to need it.

Trailing commas

Remember how you couldn’t do this for ages, because ass-old IE considered it a syntax error and would reject the entire script?

1
2
3
4
5
{
    a: 'one',
    b: 'two',
    c: 'three',  // <- THIS GUY RIGHT HERE
}

Well now it’s part of the goddamn spec and if there’s anything in this post you can rely on, it’s this. In fact you can use AS MANY GODDAMN TRAILING COMMAS AS YOU WANT. But only in arrays.

1
[1, 2, 3,,,,,,,,,,,,,,,,,,,,,,,,,]

Apparently that has the bizarre side effect of reserving extra space at the end of the array, without putting values there.

And more, probably

Like strict mode, which makes a few silent “errors” be actual errors, forces you to declare variables (no implicit globals!), and forbids the completely bozotic with block.

Or String.trim(), which trims whitespace off of strings.

Or… Math.sign()? That’s new? Seriously? Well, okay.

Or the Proxy type, which lets you customize indexing and assignment and calling. Oh. I guess that is possible, though this is a pretty weird way to do it; why not just use symbol-named methods?

You can write Unicode escapes for astral plane characters in strings (or identifiers!), as \u{XXXXXXXX}.

There’s a const now? I extremely don’t care, just name it in all caps and don’t reassign it, come on.

There’s also a mountain of other minor things, which you can peruse at your leisure via MDN or the ECMAScript compatibility tables (note the links at the top, too).

That’s all I’ve got. I still wouldn’t say I’m a big fan of JavaScript, but it’s definitely making an effort to clean up some goofy inconsistencies and solve common problems. I think I could even write some without yelling on Twitter about it now.

On the other hand, if you’re still stuck supporting IE 10 for some reason… well, er, my condolences.

Wanted: Front End Developer

Post Syndicated from Yev original https://www.backblaze.com/blog/wanted-front-end-developer/

Want to work at a company that helps customers in over 150 countries around the world protect the memories they hold dear? Do you want to challenge yourself with a business that serves consumers, SMBs, Enterprise, and developers? If all that sounds interesting, you might be interested to know that Backblaze is looking for a Front End Developer​!

Backblaze is a 10 year old company. Providing great customer experiences is the “secret sauce” that enables us to successfully compete against some of technology’s giants. We’ll finish the year at ~$20MM ARR and are a profitable business. This is an opportunity to have your work shine at scale in one of the fastest growing verticals in tech – Cloud Storage.

You will utilize HTML, ReactJS, CSS and jQuery to develop intuitive, elegant user experiences. As a member of our Front End Dev team, you will work closely with our web development, software design, and marketing teams.

On a day to day basis, you must be able to convert image mockups to HTML or ReactJS – There’s some production work that needs to get done. But you will also be responsible for helping build out new features, rethink old processes, and enabling third party systems to empower our marketing/sales/ and support teams.

Our Front End Developer must be proficient in:

  • HTML, ReactJS
  • UTF-8, Java Properties, and Localized HTML (Backblaze runs in 11 languages!)
  • JavaScript, CSS, Ajax
  • jQuery, Bootstrap
  • JSON, XML
  • Understanding of cross-browser compatibility issues and ways to work around them
  • Basic SEO principles and ensuring that applications will adhere to them
  • Learning about third party marketing and sales tools through reading documentation. Our systems include Google Tag Manager, Google Analytics, Salesforce, and Hubspot

Struts, Java, JSP, Servlet and Apache Tomcat are a plus, but not required.

We’re looking for someone that is:

  • Passionate about building friendly, easy to use Interfaces and APIs.
  • Likes to work closely with other engineers, support, and marketing to help customers.
  • Is comfortable working independently on a mutually agreed upon prioritization queue (we don’t micromanage, we do make sure tasks are reasonably defined and scoped).
  • Diligent with quality control. Backblaze prides itself on giving our team autonomy to get work done, do the right thing for our customers, and keep a pace that is sustainable over the long run. As such, we expect everyone that checks in code that is stable. We also have a small QA team that operates as a secondary check when needed.

Backblaze Employees Have:

  • Good attitude and willingness to do whatever it takes to get the job done
  • Strong desire to work for a small fast, paced company
  • Desire to learn and adapt to rapidly changing technologies and work environment
  • Comfort with well behaved pets in the office

This position is located in San Mateo, California. Regular attendance in the office is expected. Backblaze is an Equal Opportunity Employer and we offer competitive salary and benefits, including our no policy vacation policy.

If this sounds like you
Send an email to [email protected] with:

  1. Front End Dev​ in the subject line
  2. Your resume attached
  3. An overview of your relevant experience

The post Wanted: Front End Developer appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Get wordy with our free resources

Post Syndicated from Alex Bate original https://www.raspberrypi.org/blog/get-wordy-with-our-free-resources/

Here at the Raspberry Pi Foundation, we take great pride in the wonderful free resources we produce for you to use in classes, at home and in coding clubs. We publish them under a Creative Commons licence, and they’re an excellent way to develop your digital-making skills.

With yesterday being World Poetry Day (I’m a day late to the party. Shhh), I thought I’d share some wordy-themed [wordy-themed? Are you sure? – Ed] resources for you all to have a play with.

Shakespearean Insult Generator

Raspberry Pi Free Resources Shakespearean Insult Generator

Have you ever found yourself lost for words just when the moment calls for your best comeback? With the Shakespearean Insult Generator, your mumbled retorts to life’s awkward situations will have the lyrical flow of our nation’s most beloved bard.

Thou sodden-witted lord! Thou hast no more brain than I have in mine elbows!

Not only will the generator provide you with hours of potty-mouthed fun, it’ll also teach you how to read and write data in CSV format using Python, how to manipulate lists, and how to choose a random item from a list.

Talk like a Pirate

Raspberry Pi Free Resources Talk Like a Pirate

Ye’ll never be forced t’walk the plank once ye learn how to talk like a scurvy ol’ pirate… yaaaarrrgh!

The Talk like a Pirate speech generator teaches you how to use jQuery to cause live updates on a web page, how to write regular expressions to match patterns and words, and how to create a web page to input text and output results.

Once you’ve mastered those skills, you can use them to create other speech generators. How about a speech generator that turns certain words into their slang counterparts? Or one that changes words into txt speak – laugh into LOL, and see you into CU?

Secret Agent Chat

Raspberry Pi Free Resources Secret Agent Chat

So you’ve already mastered insults via list manipulation and random choice, and you’ve converted words into hilarious variations through matching word patterns and input/output. What’s next?

The Secret Agent Chat resource shows you how random numbers can be used to encrypt messages, how iteration can be used to encrypt individual characters, and, to make sure nobody cracks your codes, the importance of keeping your keys secret. And with these new skills under your belt, you can write and encrypt messages between you and your friends, ensuring that nobody will be able to read your secrets.

Unlocking your transferable skill set

One of the great things about building projects like these is the way it expands your transferable skill set. When you complete a project using one of our resources, you gain abilities that can be transferred to other projects and situations. You might never need to use a ‘Talk like a Pirate’ speech generator, but you might need to create a way to detect and alter certain word patterns in a document. And while you might be able to coin your own colourful insults, making the Shakespearean Insult Generator gives you the ability to select words from lists at random, allowing you to write a program that picks names to create sports or quiz teams without bias.

All of our resources are available for free on our website, and we continually update them to offer you more opportunities to work on your skills, whatever your age and experience.

Have you built anything from our resources? Let us know in the comments.

The post Get wordy with our free resources appeared first on Raspberry Pi.

Skillz: editing a web page

Post Syndicated from Robert Graham original http://blog.erratasec.com/2017/02/skillz-editing-web-page.html

So one of the skillz you ought to have in cybersec is messing with web-pages client-side using Chrome’s Developer Tools. Web-servers give you a bunch of HTML and JavaScript code which, once it reaches your browser, is yours to change and play with. You can do a lot with web-sites that they don’t intend by changing that code.

Let me give you an example. It’s only an example — touching briefly on steps to give you an impression what’s going on. It’s not a ground up explanation of everything, which you may find off-putting. Click on the images to expand them so you can see fully what’s going on.

Today is the American holiday called “Presidents Day”. It’s actually not a federal holiday, but a holiday in all 50 states. Originally it was just Washington’s birthday (February 22), but some states choose to honor other presidents as well, hence “Presidents Day”.
Thus of us who donated to Donald Trump’s campaign (note: I donated to all candidates campaigns back in 2015) received an email today suggesting that to honor Presidents Day, we should “sign a card” for Trump. It’s a gross dis-honoring of the Presidents the day is supposed to commemorate, but whatever, it’s the 21st century.
Okay, let’s say we want to honor the current President with a bunch of 🖕🖕🖕🖕 in order to point out his crassness of exploiting this holiday, and clicked on the URL [*], and filled it in as such (with multiple skin tones for the middle finger, just so he knows its from all of us):
Okay, now we hit the submit button “Add My Name” in order to send this to his campaign. The only problem is, the web page rejects us, telling us “Please enter a valid name” (note, I’m changing font sizes in these screen shots so you can see the message):
This is obviously client side validation of the field. It’s at this point that we go into Developer Tools in order to turn it off. One way is to [right-click] on that button, and from the popup menu, select “Inspect”, which gets you this screen (yes, the original page is squashed to the left-hand side):
We can edit the HTML right there and add the “novalidate” flag, as shown below, then hit the “Add My Name” button again:
This doesn’t work. The scripts on the webpage aren’t honoring the HTML5 “novalidate” flag. Therefore, we’ll have to go edit those scripts. We do that by clicking on the Sources tab, then press [ctrl-shift-f] to open the ‘find’ window in the sources, and type “Please enter a valid name”, and you’ll find the JavaScript source file (validation.js) where the validation function is located:
If at this point you find all these windows bewildering, then yes, you are on the right track. We typed in the search there near the bottom next to the classic search icon 🔍. Then right below that we got the search results. We clicked on the search results, then up above popped up the source file (validation.js) among all the possible source files with the line selected that contains our search term. Remember: when you pull down a single HTML page, like the one from donaldtrump.com, it can pull in a zillion JavaScript files as well.
Unlike the HTML, we can’t change the JavaScript on the fly (at least, I don’t know how to). Instead, we have to run more JavaScript. Specifically, we need to run a script that registers a new validation function. If you look in the original source, it contains a function that validates the input by making sure it matches a regular expression:
  1. jQuery.validator.addMethod(“isname”, function(value, element) {
  2.     return this.optional(element) || (/^[a-zA-Z]+[ ]+(([‘,. -][a-zA-Z ])?[a-zA-Z]*)+.?$/.test(value.trim()));
  3. }, “Please enter a valid name”);
From the console, we are going to call the addMethod function ourselves to register a different validation function for isname, specifically a validation function that always returns true, meaning the input is valid. This will override the previously registered function. As the Founders of our country say, the solution to bad JavaScript is not to censor it, but to add more JavaScript.
  1. jQuery.validator.addMethod(“isname”, function () {
  2.     return true});
We just type that in the Console as shown below (in the bottom window where Search used to be) and hit [enter]. It gives us the response “undefined”, but that’s OK. (Note: in the screenshot I misspelled it as isName, it should instead be all lowercase isname).
Now we can close Developer Tools and press the “Add My Name” button, and we get the following response:
Darn, foiled again. But at least this time, our request went to the server. It was on the server side that the request was rejected. We successfully turned off client-side checking. Had the server accepted our Unicode emoji, we would’ve reached the next step, where it asks for donations. (By the way, the entire purpose of “sign this card” is to get users to donate, nothing else).

Conclusion

So we didn’t actually succeed at doing anything here, but I thought I’d write it up anyway. Editing the web-page client-side, or mucking around with JavaScript client-side, is a skill that every cybersec professional should have. Hopefully, this is an amusing enough example that people will follow the steps to see how this is done.

Friday’s security updates

Post Syndicated from jake original http://lwn.net/Articles/712154/rss

Arch Linux has updated php (three
vulnerabilities), powerdns (MV), and powerdns-recursor (three vulnerabilities).

Debian has updated mysql-5.5
(multiple unspecified vulnerabilities).

Debian-LTS has updated libphp-swiftmailer (code execution).

Gentoo has updated curl (MV, two
from 2014), cvs (code execution from 2012),
icedtea-bin (MV), irssi (MV), and nss (MV, three from 2015).

openSUSE has updated pdns-recursor (42.2, 42.1: denial of service)
and squid (42.1: two vulnerabilities, one
from 2014).

Red Hat has updated java-1.8.0-openjdk (RHEL7&6: MV),
openstack-cinder (OSP6.0 for RHEL7; OSP5.0 for RHEL7; OSP5.0 for
RHEL6
: denial of service from 2015), and python-XStatic-jquery-ui (OSP7.0 for RHEL7:
cross-site scripting).

SUSE has updated gstreamer-0_10-plugins-good (SLE12SP2: MV).

Security updates for Friday

Post Syndicated from jake original http://lwn.net/Articles/710355/rss

Debian has updated dcmtk (code
execution from 2015).

Debian-LTS has updated curl (code
execution) and libxi (regression in
previous update).

Fedora has updated js-jquery
(F24: cross-site scripting), js-jquery1 (F25; F24:
cross-site scripting), smack (F25: TLS
bypass), and tracker (F24: adding sandboxing).

Gentoo has updated mod_wsgi
(privilege escalation from 2014).

Mageia has updated game-music-emu
(multiple vulnerabilities), gstreamer1.0-plugins-good (multiple vulnerabilities), hdf5 (multiple vulnerabilities), kernel,
kmod
(three vulnerabilities), libgsf
(denial of service), openjpeg2 (multiple vulnerabilities), roundcubemail (code execution), and samba (authentication bypass).

openSUSE has updated irc-otr
(42.2: information disclosure).

Slackware has updated python (two
vulnerabilities) and samba (three vulnerabilities).

SUSE has updated gstreamer-plugins-bad (SLE12: multiple vulnerabilities) and gstreamer-plugins-good (SLE12: multiple vulnerabilities).

Security advisories for Wednesday

Post Syndicated from ris original http://lwn.net/Articles/710277/rss

Arch Linux has updated openfire (multiple vulnerabilities).

Debian-LTS has updated libcrypto++ (denial of service).

Fedora has updated community-mysql (F25; F24:
multiple unspecified vulnerabilities), curl
(F25: buffer overflow), hdf5 (F25: multiple
vulnerabilities), js-jquery (F25:
cross-site scripting), nagios-plugins (F25; F24:
multiple vulnerabilities), python-wikitcms (F25; F24: code
execution), and xen (F25: multiple vulnerabilities).

Gentoo has updated firejail-lts (denial of service).

openSUSE has updated ntp
(42.2, 42.1: multiple vulnerabilities) and tor (42.2; 42.1, 13.2: denial of service).

SUSE has updated openjpeg2
(SLE12-SP2: multiple vulnerabilities) and xen (SOSC5, SMP2.1, SM2.1, SLE11-SP3: multiple vulnerabilities).

Security advisories for Friday

Post Syndicated from jake original http://lwn.net/Articles/708651/rss

Arch Linux has updated jasper
(multiple vulnerabilities, two from 2015) and linux-zen (code execution).

Debian-LTS has updated roundcube
(code execution) and spip (?:).

Fedora has updated httpd (F25:
denial of service).

Mageia has updated phpmyadmin (multiple vulnerabilities).

openSUSE has updated GraphicsMagick (42.2: multiple vulnerabilities, many from 2014), kernel (13.2: multiple vulnerabilities, two from 2015), and libXfixes (13.2: denial of service).

Red Hat has updated python-XStatic-jquery-ui (RHOSP 9.0; RHOSP
8.0
: cross-site scripting), rh-mariadb100-mariadb (RHSC: multiple vulnerabilities), and rh-mariadb101-mariadb (RHSC: multiple vulnerabilities).

SUSE has updated kernel (SLE12:
three vulnerabilities).

Ubuntu has updated oxide-qt
(16.10, 16.04, 14.04: multiple vulnerabilities).

A Rebuttal For Python 3

Post Syndicated from Eevee original https://eev.ee/blog/2016/11/23/a-rebuttal-for-python-3/

Zed Shaw, of Learn Python the Hard Way fame, has now written The Case Against Python 3.

I’m not involved with core Python development. The only skin I have in this game is that I like Python 3. It’s a good language. And one of the big factors I’ve seen slowing its adoption is that respected people in the Python community keep grouching about it. I’ve had multiple newcomers tell me they have the impression that Python 3 is some kind of unusable disaster, though they don’t know exactly why; it’s just something they hear from people who sound like they know what they’re talking about. Then they actually use the language, and it’s fine.

I’m sad to see the Python community needlessly sabotage itself, but Zed’s contribution is beyond the pale. It’s not just making a big deal about changed details that won’t affect most beginners; it’s complete and utter nonsense, on a platform aimed at people who can’t yet recognize it as nonsense. I am so mad.

The Case Against Python 3

I give two sets of reasons as I see them now. One for total beginners, and another for people who are more knowledgeable about programming.

Just to note: the two sets of reasons are largely the same ideas presented differently, so I’ll just weave them together below.

The first section attempts to explain the case against starting with Python 3 in non-technical terms so a beginner can make up their own mind without being influenced by propaganda or social pressure.

Having already read through this once, this sentence really stands out to me. The author of a book many beginners read to learn Python in the first place is providing a number of reasons (some outright fabricated) not to use Python 3, often in terms beginners are ill-equipped to evaluate, but believes this is a defense against propaganda or social pressure.

The Most Important Reason

Before getting into the main technical reasons I would like to discuss the one most important social reason for why you should not use Python 3 as a beginner:

THERE IS A HIGH PROBABILITY THAT PYTHON 3 IS SUCH A FAILURE IT WILL KILL PYTHON.

Python 3’s adoption is really only at about 30% whenever there is an attempt to measure it.

Wait, really? Wow, that’s fantastic.

I mean, it would probably be higher if the most popular beginner resources were actually teaching Python 3, but you know.

Nobody is all that interested in finding out what the real complete adoption is, despite there being fairly simple ways to gather metrics on the adoption.

This accusatory sentence conspicuously neglects to mention what these fairly simple ways are, a pattern that repeats throughout. The trouble is that it’s hard to even define what “adoption” means — I write all my code in Python 3 now, but veekun is still Python 2 because it’s in maintenance mode, so what does that say about adoption? You could look at PyPI download stats, but those are thrown way off by caches and system package managers. You could look at downloads from the Python website, but a great deal of Python is written and used on Unix-likes, where Python itself is either bundled or installed from the package manager.

It’s as simple as that. If you learn Python 2, then you can still work with all the legacy Python 2 code in existence until Python dies or you (hopefully) move on. But if you learn Python 3 then your future is very uncertain. You could really be learning a dead language and end up having to learn Python 2 anyway.

You could use Python 2, until it dies… or you could use Python 3, which might die. What a choice.

By some definitions, Python 2 is already dead — it will not see another major release, only security fixes. Python 3 is still actively developed, and its seventh major release is next month. It even contains a new feature that Zed later mentions he prefers to Python 2’s offerings.

It may shock you to learn that I know both Python 2 and Python 3. Amazingly, two versions of the same language are much more similar than they are different. If you learned Python 3 and then a wizard cast a spell that made it vanish from the face of the earth, you’d just have to spend half an hour reading up on what had changed from Python 2.

Also, it’s been over a decade, maybe even multiple decades, and Python 3 still isn’t above about 30% in adoption. Even among the sciences where Python 3 is touted as a “success” it’s still only around 25-30% adoption. After that long it’s time to admit defeat and come up with a new plan.

Python 3.0 came out in 2008. The first couple releases ironed out some compatibility and API problems, so it didn’t start to gain much traction until Python 3.2 came out in 2011. Hell, Python 2.0 came out in 2000, so even Python 2 isn’t multiple decades old. It would be great if this trusted beginner reference could take two seconds to check details like this before using them to scaremonger.

The big early problem was library compatibility: it’s hard to justify switching to a new version of the language if none of the libraries work. Libraries could only port once their own dependencies had ported, of course, and it took a couple years to figure out the best way to maintain compatibility with both Python 2 and Python 3. I’d say we only really hit critical mass a few years ago — for instance, Django didn’t support Python 3 until 2013 — in which case that 30% is nothing to sneeze at.

There are more reasons beyond just the uncertain future of Python 3 even decades later.

In one paragraph, we’ve gone from “maybe even multiple decades” to just “decades”, which is a funny way to spell “eight years”.

Not In Your Best Interests

The Python project’s efforts to convince you to start with Python 3 are not in your best interest, but, rather, are only in the best interests of the Python project.

It’s bad, you see, for the Python project to want people to use the work it produced.

Anyway, please buy Zed Shaw’s book.

Anyway, please pledge to my Patreon.

Ultimately though, if Python 3 were good they wouldn’t need to do any convincing to get you to use it. It would just naturally work for you and you wouldn’t have any problems. Instead, there are serious issues with Python 3 for beginners, and rather than fix those issues the Python project uses propaganda, social pressure, and marketing to convince you to use it. In the world of technology using marketing and propaganda is immediately a sign that the technology is defective in some obvious way.

This use of social pressure and propaganda to convince you to use Python 3 despite its problems, in an attempt to benefit the Python project, is morally unconscionable to me.

Ten paragraphs in, Zed is telling me that I should be suspicious of anything that relies on marketing and propaganda. Meanwhile, there has yet to be a single concrete reason why Python 3 is bad for beginners — just several flat-out incorrect assertions and a lot of handwaving about how inexplicably nefarious the Python core developers are. You know, the same people who made Python 2. But they weren’t evil then, I guess.

You Should Be Able to Run 2 and 3

In the programming language theory there is this basic requirement that, given a “complete” programming language, I can run any other programming language. In the world of Java I’m able to run Ruby, Java, C++, C, and Lua all at the same time. In the world of Microsoft I can run F#, C#, C++, and Python all at the same time. This isn’t just a theoretical thing. There is solid math behind it. Math that is truly the foundation of computer science.

The fact that you can’t run Python 2 and Python 3 at the same time is purely a social and technical decision that the Python project made with no basis in mathematical reality. This means you are working with a purposefully broken platform when you use Python 3, and I personally can’t condone teaching people to use something that is fundamentally broken.

The programmer-oriented section makes clear that the solid math being referred to is Turing-completeness — the section is even titled “Python 3 Is Not Turing Complete”.

First, notice a rhetorical trick here. You can run Ruby, Java, C++, etc. at the same time, so why not Python 2 and Python 3?

But can you run Java and C# at the same time? (I’m sure someone has done this, but it’s certainly much less popular than something like Jython or IronPython.)

Can you run Ruby 1.8 and Ruby 2.3 at the same time? Ah, no, so I guess Ruby 2.3 is fundamentally and purposefully broken.

Can you run Lua 5.1 and 5.3 at the same time? Lua is a spectacular example, because Lua 5.2 made a breaking change to how the details of scope work, and it’s led to a situation where a lot of programs that embed Lua haven’t bothered upgrading from Lua 5.1. Was Lua 5.2 some kind of dark plot to deliberately break the language? No, it’s just slightly more inconvenient than expected for people to upgrade.

Anyway, as for Turing machines:

In computer science a fundamental law is that if I have one Turing Machine I can build any other Turing Machine. If I have COBOL then I can bootstrap a compiler for FORTRAN (as disgusting as that might be). If I have FORTH, then I can build an interpreter for Ruby. This also applies to bytecodes for CPUs. If I have a Turing Complete bytecode then I can create a compiler for any language. The rule then can be extended even further to say that if I cannot create another Turing Machine in your language, then your language cannot be Turing Complete. If I can’t use your language to write a compiler or interpreter for any other language then your language is not Turing Complete.

Yes, this is true.

Currently you cannot run Python 2 inside the Python 3 virtual machine. Since I cannot, that means Python 3 is not Turing Complete and should not be used by anyone.

And this is completely asinine. Worse, it’s flat-out dishonest, and relies on another rhetorical trick. You only “cannot” run Python 2 inside the Python 3 VM because no one has written a Python 2 interpreter in Python 3. The “cannot” is not a mathematical impossibility; it’s a simple matter of the code not having been written. Or perhaps it has, but no one cares anyway, because it would be comically and unusably slow.

I assume this was meant to be sarcastic on some level, since it’s followed by a big blue box that seems unsure about whether to double down or reverse course. But I can’t tell why it was even brought up, because it has absolutely nothing to do with Zed’s true complaint, which is that Python 2 and Python 3 do not coexist within a single environment. Implementing language X using language Y does not mean that X and Y can now be used together seamlessly.

The canonical Python release is written in C (just like with Ruby or Lua), but you can’t just dump a bunch of C code into a Python (or Ruby or Lua) file and expect it to work. You can talk to C from Python and vice versa, but defining how they communicate is a bit of a pain in the ass and requires some level of setup.

I’ll get into this some more shortly.

No Working Translator

Python 3 comes with a tool called 2to3 which is supposed to take Python 2 code and translate it to Python 3 code.

I should point out right off the bat that this is not actually what you want to use most of the time, because you probably want to translate your Python 2 code to Python 2/3 code. 2to3 produces code that most likely will not work on Python 2. Other tools exist to help you port more conservatively.

Translating one programming language into another is a solidly researched topic with solid math behind it. There are translators that convert any number of languages into JavaScript, C, C++, Java, and many times you have no idea the translation is being done. In addition to this, one of the first steps when implementing a new language is to convert the new language into an existing language (like C) so you don’t have to write a full compiler. Translation is a fully solved problem.

This is completely fucking ludicrous. Translating one programming language to another is a common task, though “fully solved” sounds mighty questionable. But do you know what the results look like?

I found a project called “Transcrypt”, which puts Python in the browser by “translating” it to JavaScript. I’ve never used or heard of this before; I just googled for something to convert Python to JavaScript. Here’s their first sample, a demo using jQuery:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
def start ():
    def changeColors ():
        for div in S__divs:
            S (div) .css ({
                'color': 'rgb({},{},{})'.format (* [int (256 * Math.random ()) for i in range (3)]),
            })

    S__divs = S ('div')
    changeColors ()
    window.setInterval (changeColors, 500)

And here’s the JavaScript code it compiles to:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
(function () {
    var start = function () {
        var changeColors = function () {
            var __iterable0__ = $divs;
            for (var __index0__ = 0; __index0__ < __iterable0__.length; __index0__++) {
                var div = __iterable0__ [__index0__];
                $ (div).css (dict ({'color': 'rgb({},{},{})'.format.apply (null, function () {
                    var __accu0__ = [];
                    for (var i = 0; i < 3; i++) {
                        __accu0__.append (int (256 * Math.random ()));
                    }
                    return __accu0__;
                } ())}));
            }
        };
        var $divs = $ ('div');
        changeColors ();
        window.setInterval (changeColors, 500);
    };
    __pragma__ ('<all>')
        __all__.start = start;
    __pragma__ ('</all>')
}) ();

Well, not quite. That’s actually just a small piece at the end of the full 1861-line file.

You may notice that the emitted JavaScript effectively has to emulate the Python for loop, because JavaScript doesn’t have anything that works exactly the same way. And this is a basic, common language feature translated between two languages in the same general family! Imagine how your code would look if you relied on gritty details of how classes are implemented.

Is this what you want 2to3 to do to your code?

Even if something has been proven to be mathematically possible, that doesn’t mean it’s easy, and it doesn’t mean the results will be pretty (or fast).

The 2to3 translator fails on about 15% of the code it attempts, and does a poor job of translating the code it can handle. The motivations for this are unclear, but keep in mind that a group of people who claim to be programming language experts can’t write a reliable translator from one version of their own language to another. This is also a cause of their porting problems, which adds up to more evidence Python 3’s future is uncertain.

Writing a translator from one language to another is a fully proven and fundamental piece of computer science. Yet, the 2to3 translator cannot translate code 100%. In my own tests it is only about 85% effective, leaving a large amount of code to translate manually. Given that translation is a solved problem this seems to be a decision bordering on malice rather than incredible incompetence.

The programmer-oriented section doubles down on this idea with a title of “Purposefully Crippled 2to3 Translator” — again, accusing the Python project of sabotaging everyone. That doesn’t even make sense; if their goal is to make everyone use Python 3 at any cost, why would they deliberately break their tool that reduces the amount of Python 2 code and increases the amount of Python 3 code?

2to3 sucks because its job is hard. Python is dynamically typed. If it sees d.iteritems(), it might want to change that to d.items(), as it’s called in Python 3 — but it can’t always be sure that d is actually a dict. If d is some user-defined type, renaming the method is wrong.

But hey, Turing-completeness, right? It must be mathematically possible. And it is! As long as you’re willing to see this:

1
2
for key, value in d.iteritems():
    ...

Get translated to this:

1
2
3
__d = d
for key, value in (__d.items() if isinstance(__d, dict) else __d.iteritems()):
    ...

Would Zed be happier with that, I wonder?

The JVM and CLR Prove It’s Pointless

Yet, for some reason, the Python 3 virtual machine can’t run Python 2? Despite the solidly established mathematics disproving this, the countless examples of running one crazy language inside a Russian doll cascade of other crazy languages, and huge number of languages that can coexist in nearly every other virtual machine? That makes no sense.

This, finally, is the real complaint. It’s not a bad one, and it comes up sometimes, but… it’s not this easy.

The Python 3 VM is fairly similar to the Python 2 VM. The problem isn’t the VM, but the core language constructs and standard library.

Consider: what happens when a Python 2 old-style class instance gets passed into Python 3, which has no such concept? It seems like a value would have to always have the semantics of the language version it came from — that’s how languages usually coexist on the same VM, anyway.

Now, I’m using Python 3, and I load some library written for Python 2. I call a Python 2 function that deals with bytestrings, and I pass it a Python 3 bytestring. Oh no! It breaks because Python 3 bytestrings iterate as integers, whereas the Python 2 library expects them to iterate as characters.

Okay, well, no big deal, you say. Maybe Python 2 libraries just need to be updated to work either way, before they can be used with Python 3.

But that’s exactly the situation we’re in right now. Syntax changes are trivially fixed by 2to3 and similar tools. It’s libraries that cause the subtler issues.

The same applies the other way, too. I write Python 3 code, and it gets an int from some Python 2 library. I try to use the .to_bytes method on it, but that doesn’t exist on Python 2 integers. So my Python 3 code, written and intended purely for Python 3, now has to deal with Python 2 integers as well.

Perhaps “primitive” types should convert automatically, on the boundary? Okay, sure. What about the Python 2 buffer type, which is C-backed and replaced by memoryview in Python 3?

Or how about this very fundamental problem: names of methods and other attributes are str in both versions, but that means they’re bytestrings in Python 2 and text in Python 3. If you’re in Python 3 land, and you call obj.foo() on a Python 2 object, what happens? Python 3 wants a method with the text name foo, but Python 2 wants a method with the bytes name foo. Text and bytes are not implicitly convertible in Python 3. So does it error? Somehow work anyway? What about the other way around?

What about the standard library, which has had a number of improvements in Python 3 that don’t or can’t exist in Python 2? Should Python ship two entire separate copies of its standard library? What about modules like logging, which rely on global state? Does Python 2 and Python 3 code need to set up logging separately within the same process?

There are no good solutions here. The language would double in size and complexity, and you’d still end up with a mess at least as bad as the one we have now when values leak from one version into the other.

We either have two situations here:

  1. Python 3 has been purposefully crippled to prevent Python 2’s execution alongside Python 3 for someone’s professional or ideological gain.
  2. Python 3 cannot run Python 2 due to simple incompetence on the part of the Python project.

I can think of a third.

Difficult To Use Strings

The strings in Python 3 are very difficult to use for beginners. In an attempt to make their strings more “international” they turned them into difficult to use types with poor error messages.

Why is “international” in scare quotes?

Every time you attempt to deal with characters in your programs you’ll have to understand the difference between byte sequences and Unicode strings.

Given that I’m reading part of a book teaching Python, this would be a perfect opportunity to drive this point home by saying “Look! Running exercise N in Python 3 doesn’t work.” Exercise 1, at least, works fine for me with a little extra sprinkle of parentheses:

1
2
3
4
5
6
7
print("Hello World!")
print("Hello Again")
print("I like typing this.")
print("This is fun.")
print('Yay! Printing.')
print("I'd much rather you 'not'.")
print('I "said" do not touch this.')

Contrast with the actual content of that exercise — at the bottom is a big red warning box telling people from “another country” (relative to where?) that if they get errors about ASCII encodings, they should put an unexplained magical incantation at the top of their scripts to fix “Unicode UTF-8”, whatever that is. I wonder if Zed has read his own book.

Don’t know what that is? Exactly.

If only there were a book that could explain it to beginners in more depth than “you have to fix this if you’re foreign”.

The Python project took a language that is very forgiving to beginners and mostly “just works” and implemented strings that require you to constantly know what type of string they are. Worst of all, when you get an error with strings (which is very often) you get an error message that doesn’t tell you what variable names you need to fix.

The complaint is that this happens in Python 3, whereas it’s accepted in Python 2:

1
2
3
4
>>> b"hello" + "hello"
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: can't concat bytes to str

The programmer section is called “Statically Typed Strings”. But this is not static typing. That’s strong typing, a property that sets Python’s type system apart from languages like JavaScript. It’s usually considered a good thing, because the alternative is to silently produce nonsense in some cases, and then that nonsense propagates through your program and is hard to track down when it finally causes problems.

If they’re going to require beginners to struggle with the difference between bytes and Unicode the least they could do is tell people what variables are bytes and what variables are strings.

That would be nice, but it’s not like this is a new problem. Try this in Python 2.

1
2
3
4
>>> 3 + "hello"
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for +: 'int' and 'str'

How would Python even report this error when I used literals instead of variables? How could custom types hook into such a thing? Error messages are hard.

By the way, did you know that several error messages are much improved in Python 3? Python 2 is somewhat notorious for the confusing errors it produces when an argument is missing from a method call, but Python 3 is specific about the problem, which is much friendlier to beginners.

However, when you point out that this is hard to use they try to claim it’s good for you. It is not. It’s simple blustering covering for a poor implementation.

I don’t know what about this is hard. Why do you have a text string and a bytestring in the first place? Why is it okay to refuse adding a number to a string, but not to refuse adding bytes to a string?

Imagine if one of the Python core developers were just getting into Python 2 and messing around.

1
2
3
# -*- coding: utf8 -*-
print "Hi, my name is Łukasz Langa."
print "Hi, my name is Łukasz Langa."[::-1]
1
2
Hi, my name is Łukasz Langa.
.agnaL zsaku�� si eman ym ,iH

Good luck figuring out how to fix that.

This isn’t blustering. Bytes are not text; they are binary data that could encode anything. They happen to look like text sometimes, and you can get away with thinking they’re text if you’re not from “another country”, but that mindset will lead you to write code that is wrong. The resulting bugs will be insidious and confusing, and you’ll have a hard time even reasoning about them because it’ll seem like “Unicode text” is somehow a different beast altogether from “ASCII text”.

Exercise 11 mentions at the end that you can use int() to convert a number to an integer. It’s no more complicated to say that you convert bytes to a string using .decode(). It shouldn’t even come up unless you’re explicitly working with binary data, and I don’t see any reading from sockets in LPTHW.

It’s also not statically compiled as strongly as it could be, so you can’t find these kinds of type errors until you run the code.

This comes a scant few paragraphs after “Dynamic typing is what makes Python easy to use and one of the reasons I advocate it for beginners.”

You can’t find any kinds of type errors until you run the code. Welcome to dynamic typing.

Strings are also most frequently received from an external source, such as a network socket, file, or similar input. This means that Python 3’s statically typed strings and lack of static type safety will cause Python 3 applications to crash more often and have more security problems when compared with Python 2.

On the contrary — Python 3 applications should crash less often. The problem with silently converting between bytestrings and text in Python 2 is that it might fail, depending on the contents. "cafe" + u"hello" works fine, but "café" + u"hello" raises a UnicodeDecodeError. Python 2 makes it very easy to write code that appears to work when tested with ASCII data, but later breaks with anything else, even though the values are still the same types. In Python 3, you get an error the first time you try to run such code, regardless of what’s in the actual values. That’s the biggest reason for the change: it improves things from being intermittent value errors to consistent type errors.

More security problems? This is never substantiated, and seems to have been entirely fabricated.

Too Many Formatting Options

In addition to that you will have 3 different formatting options in Python 3.6. That means you’ll have to learn to read and use multiple ways to format strings that are all very different. Not even I, an experienced professional programmer, can easily figure out these new formatting systems or keep up with their changing features.

I don’t know what on earth “keep up with their changing features” is supposed to mean, and Zed doesn’t bother to go into details.

Python 3 has three ways to format strings: % interpolation, str.format(), and the new f"" strings in Python 3.6. The f"" strings use the same syntax as str.format(); the difference is that where str.format() uses numbers or names of keyword arguments, f"" strings just use expressions. Compare:

1
2
3
number = 133
print("{n:02x}".format(n=number))
print(f"{number:02x}")

This isn’t “very different”. A frequently-used method is being promoted to syntax.

I really like this new style, and I have no idea why this wasn’t the formatting for Python 3 instead of that stupid .format function. String interpolation is natural for most people and easy to explain.

The problem is that beginner will now how to know all three of these formatting styles, and that’s too many.

I could swear Zed, an experienced professional programmer, just said he couldn’t easily figure out these new formatting systems. Note also that str.format() has existed in Python 2 since Python 2.6 was released in 2008, so I don’t know why Zed said “new formatting systems“, plural.

This is a truly bizarre complaint overall, because the mechanism Zed likes best is the newest one. If Python core had agreed that three mechanisms was too many, we wouldn’t be getting f"" at all.

Even More Versions of Strings

Finally, I’m told there is a new proposal for a string type that is both bytes and Unicode at the same time? That’d be fantastic if this new type brings back the dynamic typing that makes Python easy, but I’m betting it will end up being yet another static type to learn. For that reason I also think beginners should avoid Python 3 until this new “chimera string” is implemented and works reliably in a dynamic way. Until then, you will just be dealing with difficult strings that are statically typed in a dynamically typed language.

I have absolutely no idea what this is referring to, and I can’t find anyone who does. I don’t see any recent PEPs mentioning such a thing, nor anything in the last several months on the python-dev mailing list. I don’t see it in the Python 3.6 release notes.

The closest thing I can think of is the backwards-compatibility shenanigans for PEP 528 and PEP 529 — they switch to the Windows wide-string APIs for console and filesystem encoding, but pretend under the hood that the APIs take UTF-8-encoded bytes to avoid breaking libraries like Twisted. That’s a microscopic detail that should never matter to anyone but authors of Twisted, and is nothing like a new hybrid string type, but otherwise I’m at a loss.

This paragraph really is a perfect summary of the whole article. It speaks vaguely yet authoritatively about something that doesn’t seem to exist, it doesn’t bother actually investigating the thing the entire section talks about, it conjectures that this mysterious feature will be hard just because it’s in Python 3, and it misuses terminology to complain about a fundamental property of Python that’s always existed.

Core Libraries Not Updated

Many of the core libraries included with Python 3 have been rewritten to use Python 3, but have not been updated to use its features. How could they given Python 3’s constant changing status and new features?

What “constant changing status”? The language makes new releases; is that bad? The only mention of “changing” so far was with string formatting, which makes no sense to me, because the only major change has been the addition of syntax that Zed prefers.

There are several libraries that, despite knowing the encoding of data, fail to return proper strings. The worst offender seems to be any libraries dealing with the HTTP protocol, which does indicate the encoding of the underlying byte stream in many cases.

In many cases, yes. Not in all. Some web servers don’t send back an encoding. Some files don’t have an encoding, because they’re images or other binary data. HTML allows the encoding to be given inside the document, instead. urllib has always returned bytes, so it’s not all that unreasonable to keep doing that, rather than… well, I’m not quite sure what this is proposing. Return strings sometimes?

The documentation for urllib.request and http.client both advise using the higher-level Requests library instead, in a prominent yellow box right at the top. Requests has distinct mechanisms for retrieving bytes versus text and is vastly easier to use overall, though I don’t think even it understands reading encodings from HTML. Alas, computers.

Good luck to any beginner figuring out how to install Requests on Python 2 — but thankfully, Python 3 now comes bundled with pip, which makes installing libraries much easier. Contrast with the beginning of exercise 46, which apologizes for how difficult this is to explain, lists four things to install, warns that it will be frustrating, and advises watching a video to help figure it out.

What’s even more idiotic about this is Python has a really good Chardet library for detecting the encoding of byte streams. If Python 3 is supposed to be “batteries included” then fast Chardet should be baked into the core of Python 3’s strings making it cake to translate strings to bytes even if you don’t know the underlying encoding. … Call the function whatever you want, but it’s not magic to guess at the encoding of a byte stream, it’s science. The only reason this isn’t done for you is that the Python project decided that you should be punished for not knowing about bytes vs. Unicode, and their arrogance means you have difficult to use strings.

Guessing at the encoding of a byte stream isn’t so much science as, well, guessing. Guessing means that sometimes you’re wrong. Sometimes that’s what you want, and I’m honestly ambivalent about having chardet in the standard library, but it’s hardly arrogant to not want to include a highly-fallible heuristic in your programming language.

Conclusions and Warnings

I have resisted writing about these problems with Python 3 for 5 versions because I hoped it would become usable for beginners. Each year I would attempt to convert some of my code and write a couple small tests with Python 3 and simply fail. If I couldn’t use Python 3 reliably then there’s no way a total beginner could manage it. So each year I’d attempt it, and fail, and wait until they fix it. I really liked Python and hoped the Python project would drop their stupid stances on usability.

Let us recap the usability problems seen thusfar.

  • You can’t add b"hello" to "hello".
  • TypeErrors are phrased exactly the same as they were in Python 2.
  • The type system is exactly as dynamic as it was in Python 2.
  • There is a new formatting mechanism, using the same syntax as one in Python 2, that Zed prefers over the ones in Python 2.
  • urllib.request doesn’t decode for you, just like in Python 2.
  • 档牡敤㽴 isn’t built in. Oh, sorry, I meant chardet.

Currently, the state of strings is viewed as a Good Thing in the Python community. The fact that you can’t run Python 2 inside Python 3 is seen as a weird kind of tough love. The brainwashing goes so far as to outright deny the mathematics behind language translation and compilation in an attempt to motivate the Python community to brute force convert all Python 2 code.

Which is probably why the Python project focuses on convincing unsuspecting beginners to use Python 3. They don’t have a switching cost, so if you get them to fumble their way through the Python 3 usability problems then you have new converts who don’t know any better. To me this is morally wrong and is simply preying on people to prop up a project that needs a full reset to survive. It means beginners will fail at learning to code not because of their own abilities, but because of Python 3’s difficulty.

Now that we’re towards the end, it’s a good time to say this: Zed Shaw, your behavior here is fucking reprehensible.

Half of what’s written here is irrelevant nonsense backed by a vague appeal to “mathematics”. Instead of having even the shred of humility required to step back and wonder if there are complicating factors beyond whether something is theoretically possible, you have invented a variety of conflicting and malicious motivations to ascribe to the Python project.

It’s fine to criticize Python 3. The string changes force you to think about what you’re doing a little more in some cases, and occasionally that’s a pain in the ass. I absolutely get it.

But you’ve gone out of your way to invent a conspiracy out of whole cloth and promote it on your popular platform aimed at beginners, who won’t know how obviously full of it you are. And why? Because you can’t add b"hello" to "hello"? Are you kidding me? No one can even offer to help you, because instead of examples of real problems you’ve had, you gave two trivial toys and then yelled a lot about how the whole Python project is releasing mind-altering chemicals into the air.

The Python 3 migration has been hard enough. It’s taken a lot of work from a lot of people who’ve given enough of a crap to help Python evolve — to make it better to the best of their judgment and abilities. Now we’re finally, finally at the point where virtually all libraries support Python 3, a few new ones only support Python 3, and Python 3 adoption is starting to take hold among application developers.

And you show up to piss all over it, to propagate this myth that Python 3 is hamstrung to the point of unusability, because if the Great And Wise Zed Shaw can’t figure it out in ten seconds then it must just be impossible.

Fuck you.

Sadly, I doubt this will happen, and instead they’ll just rant about how I don’t know what I’m talking about and I should shut up.

This is because you don’t know what you’re talking about, and you should shut up.

A Rebuttal For Python 3

Post Syndicated from Eevee original https://eev.ee/blog/2016/11/23/a-rebuttal-for-python-3/

Zed Shaw, of Learn Python the Hard Way fame, has now written The Case Against Python 3.

I’m not involved with core Python development. The only skin I have in this game is that I like Python 3. It’s a good language. And one of the big factors I’ve seen slowing its adoption is that respected people in the Python community keep grouching about it. I’ve had multiple newcomers tell me they have the impression that Python 3 is some kind of unusable disaster, though they don’t know exactly why; it’s just something they hear from people who sound like they know what they’re talking about. Then they actually use the language, and it’s fine.

I’m sad to see the Python community needlessly sabotage itself, but Zed’s contribution is beyond the pale. It’s not just making a big deal about changed details that won’t affect most beginners; it’s complete and utter nonsense, on a platform aimed at people who can’t yet recognize it as nonsense. I am so mad.

The Case Against Python 3

I give two sets of reasons as I see them now. One for total beginners, and another for people who are more knowledgeable about programming.

Just to note: the two sets of reasons are largely the same ideas presented differently, so I’ll just weave them together below.

The first section attempts to explain the case against starting with Python 3 in non-technical terms so a beginner can make up their own mind without being influenced by propaganda or social pressure.

Having already read through this once, this sentence really stands out to me. The author of a book many beginners read to learn Python in the first place is providing a number of reasons (some outright fabricated) not to use Python 3, often in terms beginners are ill-equipped to evaluate, but believes this is a defense against propaganda or social pressure.

The Most Important Reason

Before getting into the main technical reasons I would like to discuss the one most important social reason for why you should not use Python 3 as a beginner:

THERE IS A HIGH PROBABILITY THAT PYTHON 3 IS SUCH A FAILURE IT WILL KILL PYTHON.

Python 3’s adoption is really only at about 30% whenever there is an attempt to measure it.

Wait, really? Wow, that’s fantastic.

I mean, it would probably be higher if the most popular beginner resources were actually teaching Python 3, but you know.

Nobody is all that interested in finding out what the real complete adoption is, despite there being fairly simple ways to gather metrics on the adoption.

This accusatory sentence conspicuously neglects to mention what these fairly simple ways are, a pattern that repeats throughout. The trouble is that it’s hard to even define what “adoption” means — I write all my code in Python 3 now, but veekun is still Python 2 because it’s in maintenance mode, so what does that say about adoption? You could look at PyPI download stats, but those are thrown way off by caches and system package managers. You could look at downloads from the Python website, but a great deal of Python is written and used on Unix-likes, where Python itself is either bundled or installed from the package manager.

It’s as simple as that. If you learn Python 2, then you can still work with all the legacy Python 2 code in existence until Python dies or you (hopefully) move on. But if you learn Python 3 then your future is very uncertain. You could really be learning a dead language and end up having to learn Python 2 anyway.

You could use Python 2, until it dies… or you could use Python 3, which might die. What a choice.

By some definitions, Python 2 is already dead — it will not see another major release, only security fixes. Python 3 is still actively developed, and its seventh major release is next month. It even contains a new feature that Zed later mentions he prefers to Python 2’s offerings.

It may shock you to learn that I know both Python 2 and Python 3. Amazingly, two versions of the same language are much more similar than they are different. If you learned Python 3 and then a wizard cast a spell that made it vanish from the face of the earth, you’d just have to spend half an hour reading up on what had changed from Python 2.

Also, it’s been over a decade, maybe even multiple decades, and Python 3 still isn’t above about 30% in adoption. Even among the sciences where Python 3 is touted as a “success” it’s still only around 25-30% adoption. After that long it’s time to admit defeat and come up with a new plan.

Python 3.0 came out in 2008. The first couple releases ironed out some compatibility and API problems, so it didn’t start to gain much traction until Python 3.2 came out in 2011. Hell, Python 2.0 came out in 2000, so even Python 2 isn’t multiple decades old. It would be great if this trusted beginner reference could take two seconds to check details like this before using them to scaremonger.

The big early problem was library compatibility: it’s hard to justify switching to a new version of the language if none of the libraries work. Libraries could only port once their own dependencies had ported, of course, and it took a couple years to figure out the best way to maintain compatibility with both Python 2 and Python 3. I’d say we only really hit critical mass a few years ago — for instance, Django didn’t support Python 3 until 2013 — in which case that 30% is nothing to sneeze at.

There are more reasons beyond just the uncertain future of Python 3 even decades later.

In one paragraph, we’ve gone from “maybe even multiple decades” to just “decades”, which is a funny way to spell “eight years”.

Not In Your Best Interests

The Python project’s efforts to convince you to start with Python 3 are not in your best interest, but, rather, are only in the best interests of the Python project.

It’s bad, you see, for the Python project to want people to use the work it produced.

Anyway, please buy Zed Shaw’s book.

Anyway, please pledge to my Patreon.

Ultimately though, if Python 3 were good they wouldn’t need to do any convincing to get you to use it. It would just naturally work for you and you wouldn’t have any problems. Instead, there are serious issues with Python 3 for beginners, and rather than fix those issues the Python project uses propaganda, social pressure, and marketing to convince you to use it. In the world of technology using marketing and propaganda is immediately a sign that the technology is defective in some obvious way.

This use of social pressure and propaganda to convince you to use Python 3 despite its problems, in an attempt to benefit the Python project, is morally unconscionable to me.

Ten paragraphs in, Zed is telling me that I should be suspicious of anything that relies on marketing and propaganda. Meanwhile, there has yet to be a single concrete reason why Python 3 is bad for beginners — just several flat-out incorrect assertions and a lot of handwaving about how inexplicably nefarious the Python core developers are. You know, the same people who made Python 2. But they weren’t evil then, I guess.

You Should Be Able to Run 2 and 3

In the programming language theory there is this basic requirement that, given a “complete” programming language, I can run any other programming language. In the world of Java I’m able to run Ruby, Java, C++, C, and Lua all at the same time. In the world of Microsoft I can run F#, C#, C++, and Python all at the same time. This isn’t just a theoretical thing. There is solid math behind it. Math that is truly the foundation of computer science.

The fact that you can’t run Python 2 and Python 3 at the same time is purely a social and technical decision that the Python project made with no basis in mathematical reality. This means you are working with a purposefully broken platform when you use Python 3, and I personally can’t condone teaching people to use something that is fundamentally broken.

The programmer-oriented section makes clear that the solid math being referred to is Turing-completeness — the section is even titled “Python 3 Is Not Turing Complete”.

First, notice a rhetorical trick here. You can run Ruby, Java, C++, etc. at the same time, so why not Python 2 and Python 3?

But can you run Java and C# at the same time? (I’m sure someone has done this, but it’s certainly much less popular than something like Jython or IronPython.)

Can you run Ruby 1.8 and Ruby 2.3 at the same time? Ah, no, so I guess Ruby 2.3 is fundamentally and purposefully broken.

Can you run Lua 5.1 and 5.3 at the same time? Lua is a spectacular example, because Lua 5.2 made a breaking change to how the details of scope work, and it’s led to a situation where a lot of programs that embed Lua haven’t bothered upgrading from Lua 5.1. Was Lua 5.2 some kind of dark plot to deliberately break the language? No, it’s just slightly more inconvenient than expected for people to upgrade.

Anyway, as for Turing machines:

In computer science a fundamental law is that if I have one Turing Machine I can build any other Turing Machine. If I have COBOL then I can bootstrap a compiler for FORTRAN (as disgusting as that might be). If I have FORTH, then I can build an interpreter for Ruby. This also applies to bytecodes for CPUs. If I have a Turing Complete bytecode then I can create a compiler for any language. The rule then can be extended even further to say that if I cannot create another Turing Machine in your language, then your language cannot be Turing Complete. If I can’t use your language to write a compiler or interpreter for any other language then your language is not Turing Complete.

Yes, this is true.

Currently you cannot run Python 2 inside the Python 3 virtual machine. Since I cannot, that means Python 3 is not Turing Complete and should not be used by anyone.

And this is completely asinine. Worse, it’s flat-out dishonest, and relies on another rhetorical trick. You only “cannot” run Python 2 inside the Python 3 VM because no one has written a Python 2 interpreter in Python 3. The “cannot” is not a mathematical impossibility; it’s a simple matter of the code not having been written. Or perhaps it has, but no one cares anyway, because it would be comically and unusably slow.

I assume this was meant to be sarcastic on some level, since it’s followed by a big blue box that seems unsure about whether to double down or reverse course. But I can’t tell why it was even brought up, because it has absolutely nothing to do with Zed’s true complaint, which is that Python 2 and Python 3 do not coexist within a single environment. Implementing language X using language Y does not mean that X and Y can now be used together seamlessly.

The canonical Python release is written in C (just like with Ruby or Lua), but you can’t just dump a bunch of C code into a Python (or Ruby or Lua) file and expect it to work. You can talk to C from Python and vice versa, but defining how they communicate is a bit of a pain in the ass and requires some level of setup.

I’ll get into this some more shortly.

No Working Translator

Python 3 comes with a tool called 2to3 which is supposed to take Python 2 code and translate it to Python 3 code.

I should point out right off the bat that this is not actually what you want to use most of the time, because you probably want to translate your Python 2 code to Python 2/3 code. 2to3 produces code that most likely will not work on Python 2. Other tools exist to help you port more conservatively.

Translating one programming language into another is a solidly researched topic with solid math behind it. There are translators that convert any number of languages into JavaScript, C, C++, Java, and many times you have no idea the translation is being done. In addition to this, one of the first steps when implementing a new language is to convert the new language into an existing language (like C) so you don’t have to write a full compiler. Translation is a fully solved problem.

This is completely fucking ludicrous. Translating one programming language to another is a common task, though “fully solved” sounds mighty questionable. But do you know what the results look like?

I found a project called “Transcrypt”, which puts Python in the browser by “translating” it to JavaScript. I’ve never used or heard of this before; I just googled for something to convert Python to JavaScript. Here’s their first sample, a demo using jQuery:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
def start ():
    def changeColors ():
        for div in S__divs:
            S (div) .css ({
                'color': 'rgb({},{},{})'.format (* [int (256 * Math.random ()) for i in range (3)]),
            })

    S__divs = S ('div')
    changeColors ()
    window.setInterval (changeColors, 500)

And here’s the JavaScript code it compiles to:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
(function () {
    var start = function () {
        var changeColors = function () {
            var __iterable0__ = $divs;
            for (var __index0__ = 0; __index0__ < __iterable0__.length; __index0__++) {
                var div = __iterable0__ [__index0__];
                $ (div).css (dict ({'color': 'rgb({},{},{})'.format.apply (null, function () {
                    var __accu0__ = [];
                    for (var i = 0; i < 3; i++) {
                        __accu0__.append (int (256 * Math.random ()));
                    }
                    return __accu0__;
                } ())}));
            }
        };
        var $divs = $ ('div');
        changeColors ();
        window.setInterval (changeColors, 500);
    };
    __pragma__ ('<all>')
        __all__.start = start;
    __pragma__ ('</all>')
}) ();

Well, not quite. That’s actually just a small piece at the end of the full 1861-line file.

You may notice that the emitted JavaScript effectively has to emulate the Python for loop, because JavaScript doesn’t have anything that works exactly the same way. And this is a basic, common language feature translated between two languages in the same general family! Imagine how your code would look if you relied on gritty details of how classes are implemented.

Is this what you want 2to3 to do to your code?

Even if something has been proven to be mathematically possible, that doesn’t mean it’s easy, and it doesn’t mean the results will be pretty (or fast).

The 2to3 translator fails on about 15% of the code it attempts, and does a poor job of translating the code it can handle. The motivations for this are unclear, but keep in mind that a group of people who claim to be programming language experts can’t write a reliable translator from one version of their own language to another. This is also a cause of their porting problems, which adds up to more evidence Python 3’s future is uncertain.

Writing a translator from one language to another is a fully proven and fundamental piece of computer science. Yet, the 2to3 translator cannot translate code 100%. In my own tests it is only about 85% effective, leaving a large amount of code to translate manually. Given that translation is a solved problem this seems to be a decision bordering on malice rather than incredible incompetence.

The programmer-oriented section doubles down on this idea with a title of “Purposefully Crippled 2to3 Translator” — again, accusing the Python project of sabotaging everyone. That doesn’t even make sense; if their goal is to make everyone use Python 3 at any cost, why would they deliberately break their tool that reduces the amount of Python 2 code and increases the amount of Python 3 code?

2to3 sucks because its job is hard. Python is dynamically typed. If it sees d.iteritems(), it might want to change that to d.items(), as it’s called in Python 3 — but it can’t always be sure that d is actually a dict. If d is some user-defined type, renaming the method is wrong.

But hey, Turing-completeness, right? It must be mathematically possible. And it is! As long as you’re willing to see this:

1
2
for key, value in d.iteritems():
    ...

Get translated to this:

1
2
3
__d = d
for key, value in (__d.items() if isinstance(__d, dict) else __d.iteritems()):
    ...

Would Zed be happier with that, I wonder?

The JVM and CLR Prove It’s Pointless

Yet, for some reason, the Python 3 virtual machine can’t run Python 2? Despite the solidly established mathematics disproving this, the countless examples of running one crazy language inside a Russian doll cascade of other crazy languages, and huge number of languages that can coexist in nearly every other virtual machine? That makes no sense.

This, finally, is the real complaint. It’s not a bad one, and it comes up sometimes, but… it’s not this easy.

The Python 3 VM is fairly similar to the Python 2 VM. The problem isn’t the VM, but the core language constructs and standard library.

Consider: what happens when a Python 2 old-style class instance gets passed into Python 3, which has no such concept? It seems like a value would have to always have the semantics of the language version it came from — that’s how languages usually coexist on the same VM, anyway.

Now, I’m using Python 3, and I load some library written for Python 2. I call a Python 2 function that deals with bytestrings, and I pass it a Python 3 bytestring. Oh no! It breaks because Python 3 bytestrings iterate as integers, whereas the Python 2 library expects them to iterate as characters.

Okay, well, no big deal, you say. Maybe Python 2 libraries just need to be updated to work either way, before they can be used with Python 3.

But that’s exactly the situation we’re in right now. Syntax changes are trivially fixed by 2to3 and similar tools. It’s libraries that cause the subtler issues.

The same applies the other way, too. I write Python 3 code, and it gets an int from some Python 2 library. I try to use the .to_bytes method on it, but that doesn’t exist on Python 2 integers. So my Python 3 code, written and intended purely for Python 3, now has to deal with Python 2 integers as well.

Perhaps “primitive” types should convert automatically, on the boundary? Okay, sure. What about the Python 2 buffer type, which is C-backed and replaced by memoryview in Python 3?

Or how about this very fundamental problem: names of methods and other attributes are str in both versions, but that means they’re bytestrings in Python 2 and text in Python 3. If you’re in Python 3 land, and you call obj.foo() on a Python 2 object, what happens? Python 3 wants a method with the text name foo, but Python 2 wants a method with the bytes name foo. Text and bytes are not implicitly convertible in Python 3. So does it error? Somehow work anyway? What about the other way around?

What about the standard library, which has had a number of improvements in Python 3 that don’t or can’t exist in Python 2? Should Python ship two entire separate copies of its standard library? What about modules like logging, which rely on global state? Does Python 2 and Python 3 code need to set up logging separately within the same process?

There are no good solutions here. The language would double in size and complexity, and you’d still end up with a mess at least as bad as the one we have now when values leak from one version into the other.

We either have two situations here:

  1. Python 3 has been purposefully crippled to prevent Python 2’s execution alongside Python 3 for someone’s professional or ideological gain.
  2. Python 3 cannot run Python 2 due to simple incompetence on the part of the Python project.

I can think of a third.

Difficult To Use Strings

The strings in Python 3 are very difficult to use for beginners. In an attempt to make their strings more “international” they turned them into difficult to use types with poor error messages.

Why is “international” in scare quotes?

Every time you attempt to deal with characters in your programs you’ll have to understand the difference between byte sequences and Unicode strings.

Given that I’m reading part of a book teaching Python, this would be a perfect opportunity to drive this point home by saying “Look! Running exercise N in Python 3 doesn’t work.” Exercise 1, at least, works fine for me with a little extra sprinkle of parentheses:

1
2
3
4
5
6
7
print("Hello World!")
print("Hello Again")
print("I like typing this.")
print("This is fun.")
print('Yay! Printing.')
print("I'd much rather you 'not'.")
print('I "said" do not touch this.')

Contrast with the actual content of that exercise — at the bottom is a big red warning box telling people from “another country” (relative to where?) that if they get errors about ASCII encodings, they should put an unexplained magical incantation at the top of their scripts to fix “Unicode UTF-8”, whatever that is. I wonder if Zed has read his own book.

Don’t know what that is? Exactly.

If only there were a book that could explain it to beginners in more depth than “you have to fix this if you’re foreign”.

The Python project took a language that is very forgiving to beginners and mostly “just works” and implemented strings that require you to constantly know what type of string they are. Worst of all, when you get an error with strings (which is very often) you get an error message that doesn’t tell you what variable names you need to fix.

The complaint is that this happens in Python 3, whereas it’s accepted in Python 2:

1
2
3
4
>>> b"hello" + "hello"
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: can't concat bytes to str

The programmer section is called “Statically Typed Strings”. But this is not static typing. That’s strong typing, a property that sets Python’s type system apart from languages like JavaScript. It’s usually considered a good thing, because the alternative is to silently produce nonsense in some cases, and then that nonsense propagates through your program and is hard to track down when it finally causes problems.

If they’re going to require beginners to struggle with the difference between bytes and Unicode the least they could do is tell people what variables are bytes and what variables are strings.

That would be nice, but it’s not like this is a new problem. Try this in Python 2.

1
2
3
4
>>> 3 + "hello"
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for +: 'int' and 'str'

How would Python even report this error when I used literals instead of variables? How could custom types hook into such a thing? Error messages are hard.

By the way, did you know that several error messages are much improved in Python 3? Python 2 is somewhat notorious for the confusing errors it produces when an argument is missing from a method call, but Python 3 is specific about the problem, which is much friendlier to beginners.

However, when you point out that this is hard to use they try to claim it’s good for you. It is not. It’s simple blustering covering for a poor implementation.

I don’t know what about this is hard. Why do you have a text string and a bytestring in the first place? Why is it okay to refuse adding a number to a string, but not to refuse adding bytes to a string?

Imagine if one of the Python core developers were just getting into Python 2 and messing around.

1
2
3
# -*- coding: utf8 -*-
print "Hi, my name is Łukasz Langa."
print "Hi, my name is Łukasz Langa."[::-1]
1
2
Hi, my name is Łukasz Langa.
.agnaL zsaku�� si eman ym ,iH

Good luck figuring out how to fix that.

This isn’t blustering. Bytes are not text; they are binary data that could encode anything. They happen to look like text sometimes, and you can get away with thinking they’re text if you’re not from “another country”, but that mindset will lead you to write code that is wrong. The resulting bugs will be insidious and confusing, and you’ll have a hard time even reasoning about them because it’ll seem like “Unicode text” is somehow a different beast altogether from “ASCII text”.

Exercise 11 mentions at the end that you can use int() to convert a number to an integer. It’s no more complicated to say that you convert bytes to a string using .decode(). It shouldn’t even come up unless you’re explicitly working with binary data, and I don’t see any reading from sockets in LPTHW.

It’s also not statically compiled as strongly as it could be, so you can’t find these kinds of type errors until you run the code.

This comes a scant few paragraphs after “Dynamic typing is what makes Python easy to use and one of the reasons I advocate it for beginners.”

You can’t find any kinds of type errors until you run the code. Welcome to dynamic typing.

Strings are also most frequently received from an external source, such as a network socket, file, or similar input. This means that Python 3’s statically typed strings and lack of static type safety will cause Python 3 applications to crash more often and have more security problems when compared with Python 2.

On the contrary — Python 3 applications should crash less often. The problem with silently converting between bytestrings and text in Python 2 is that it might fail, depending on the contents. "cafe" + u"hello" works fine, but "café" + u"hello" raises a UnicodeDecodeError. Python 2 makes it very easy to write code that appears to work when tested with ASCII data, but later breaks with anything else, even though the values are still the same types. In Python 3, you get an error the first time you try to run such code, regardless of what’s in the actual values. That’s the biggest reason for the change: it improves things from being intermittent value errors to consistent type errors.

More security problems? This is never substantiated, and seems to have been entirely fabricated.

Too Many Formatting Options

In addition to that you will have 3 different formatting options in Python 3.6. That means you’ll have to learn to read and use multiple ways to format strings that are all very different. Not even I, an experienced professional programmer, can easily figure out these new formatting systems or keep up with their changing features.

I don’t know what on earth “keep up with their changing features” is supposed to mean, and Zed doesn’t bother to go into details.

Python 3 has three ways to format strings: % interpolation, str.format(), and the new f"" strings in Python 3.6. The f"" strings use the same syntax as str.format(); the difference is that where str.format() uses numbers or names of keyword arguments, f"" strings just use expressions. Compare:

1
2
3
number = 133
print("{n:02x}".format(n=number))
print(f"{number:02x}")

This isn’t “very different”. A frequently-used method is being promoted to syntax.

I really like this new style, and I have no idea why this wasn’t the formatting for Python 3 instead of that stupid .format function. String interpolation is natural for most people and easy to explain.

The problem is that beginner will now how to know all three of these formatting styles, and that’s too many.

I could swear Zed, an experienced professional programmer, just said he couldn’t easily figure out these new formatting systems. Note also that str.format() has existed in Python 2 since Python 2.6 was released in 2008, so I don’t know why Zed said “new formatting systems“, plural.

This is a truly bizarre complaint overall, because the mechanism Zed likes best is the newest one. If Python core had agreed that three mechanisms was too many, we wouldn’t be getting f"" at all.

Even More Versions of Strings

Finally, I’m told there is a new proposal for a string type that is both bytes and Unicode at the same time? That’d be fantastic if this new type brings back the dynamic typing that makes Python easy, but I’m betting it will end up being yet another static type to learn. For that reason I also think beginners should avoid Python 3 until this new “chimera string” is implemented and works reliably in a dynamic way. Until then, you will just be dealing with difficult strings that are statically typed in a dynamically typed language.

I have absolutely no idea what this is referring to, and I can’t find anyone who does. I don’t see any recent PEPs mentioning such a thing, nor anything in the last several months on the python-dev mailing list. I don’t see it in the Python 3.6 release notes.

The closest thing I can think of is the backwards-compatibility shenanigans for PEP 528 and PEP 529 — they switch to the Windows wide-string APIs for console and filesystem encoding, but pretend under the hood that the APIs take UTF-8-encoded bytes to avoid breaking libraries like Twisted. That’s a microscopic detail that should never matter to anyone but authors of Twisted, and is nothing like a new hybrid string type, but otherwise I’m at a loss.

This paragraph really is a perfect summary of the whole article. It speaks vaguely yet authoritatively about something that doesn’t seem to exist, it doesn’t bother actually investigating the thing the entire section talks about, it conjectures that this mysterious feature will be hard just because it’s in Python 3, and it misuses terminology to complain about a fundamental property of Python that’s always existed.

Core Libraries Not Updated

Many of the core libraries included with Python 3 have been rewritten to use Python 3, but have not been updated to use its features. How could they given Python 3’s constant changing status and new features?

What “constant changing status”? The language makes new releases; is that bad? The only mention of “changing” so far was with string formatting, which makes no sense to me, because the only major change has been the addition of syntax that Zed prefers.

There are several libraries that, despite knowing the encoding of data, fail to return proper strings. The worst offender seems to be any libraries dealing with the HTTP protocol, which does indicate the encoding of the underlying byte stream in many cases.

In many cases, yes. Not in all. Some web servers don’t send back an encoding. Some files don’t have an encoding, because they’re images or other binary data. HTML allows the encoding to be given inside the document, instead. urllib has always returned bytes, so it’s not all that unreasonable to keep doing that, rather than… well, I’m not quite sure what this is proposing. Return strings sometimes?

The documentation for urllib.request and http.client both advise using the higher-level Requests library instead, in a prominent yellow box right at the top. Requests has distinct mechanisms for retrieving bytes versus text and is vastly easier to use overall, though I don’t think even it understands reading encodings from HTML. Alas, computers.

Good luck to any beginner figuring out how to install Requests on Python 2 — but thankfully, Python 3 now comes bundled with pip, which makes installing libraries much easier. Contrast with the beginning of exercise 46, which apologizes for how difficult this is to explain, lists four things to install, warns that it will be frustrating, and advises watching a video to help figure it out.

What’s even more idiotic about this is Python has a really good Chardet library for detecting the encoding of byte streams. If Python 3 is supposed to be “batteries included” then fast Chardet should be baked into the core of Python 3’s strings making it cake to translate strings to bytes even if you don’t know the underlying encoding. … Call the function whatever you want, but it’s not magic to guess at the encoding of a byte stream, it’s science. The only reason this isn’t done for you is that the Python project decided that you should be punished for not knowing about bytes vs. Unicode, and their arrogance means you have difficult to use strings.

Guessing at the encoding of a byte stream isn’t so much science as, well, guessing. Guessing means that sometimes you’re wrong. Sometimes that’s what you want, and I’m honestly ambivalent about having chardet in the standard library, but it’s hardly arrogant to not want to include a highly-fallible heuristic in your programming language.

Conclusions and Warnings

I have resisted writing about these problems with Python 3 for 5 versions because I hoped it would become usable for beginners. Each year I would attempt to convert some of my code and write a couple small tests with Python 3 and simply fail. If I couldn’t use Python 3 reliably then there’s no way a total beginner could manage it. So each year I’d attempt it, and fail, and wait until they fix it. I really liked Python and hoped the Python project would drop their stupid stances on usability.

Let us recap the usability problems seen thusfar.

  • You can’t add b"hello" to "hello".
  • TypeErrors are phrased exactly the same as they were in Python 2.
  • The type system is exactly as dynamic as it was in Python 2.
  • There is a new formatting mechanism, using the same syntax as one in Python 2, that Zed prefers over the ones in Python 2.
  • urllib.request doesn’t decode for you, just like in Python 2.
  • 档牡敤㽴 isn’t built in. Oh, sorry, I meant chardet.

Currently, the state of strings is viewed as a Good Thing in the Python community. The fact that you can’t run Python 2 inside Python 3 is seen as a weird kind of tough love. The brainwashing goes so far as to outright deny the mathematics behind language translation and compilation in an attempt to motivate the Python community to brute force convert all Python 2 code.

Which is probably why the Python project focuses on convincing unsuspecting beginners to use Python 3. They don’t have a switching cost, so if you get them to fumble their way through the Python 3 usability problems then you have new converts who don’t know any better. To me this is morally wrong and is simply preying on people to prop up a project that needs a full reset to survive. It means beginners will fail at learning to code not because of their own abilities, but because of Python 3’s difficulty.

Now that we’re towards the end, it’s a good time to say this: Zed Shaw, your behavior here is fucking reprehensible.

Half of what’s written here is irrelevant nonsense backed by a vague appeal to “mathematics”. Instead of having even the shred of humility required to step back and wonder if there are complicating factors beyond whether something is theoretically possible, you have invented a variety of conflicting and malicious motivations to ascribe to the Python project.

It’s fine to criticize Python 3. The string changes force you to think about what you’re doing a little more in some cases, and occasionally that’s a pain in the ass. I absolutely get it.

But you’ve gone out of your way to invent a conspiracy out of whole cloth and promote it on your popular platform aimed at beginners, who won’t know how obviously full of it you are. And why? Because you can’t add b"hello" to "hello"? Are you kidding me? No one can even offer to help you, because instead of examples of real problems you’ve had, you gave two trivial toys and then yelled a lot about how the whole Python project is releasing mind-altering chemicals into the air.

The Python 3 migration has been hard enough. It’s taken a lot of work from a lot of people who’ve given enough of a crap to help Python evolve — to make it better to the best of their judgment and abilities. Now we’re finally, finally at the point where virtually all libraries support Python 3, a few new ones only support Python 3, and Python 3 adoption is starting to take hold among application developers.

And you show up to piss all over it, to propagate this myth that Python 3 is hamstrung to the point of unusability, because if the Great And Wise Zed Shaw can’t figure it out in ten seconds then it must just be impossible.

Fuck you.

Sadly, I doubt this will happen, and instead they’ll just rant about how I don’t know what I’m talking about and I should shut up.

This is because you don’t know what you’re talking about, and you should shut up.

Elegance

Post Syndicated from Eevee original https://eev.ee/blog/2016/04/21/elegance/

Programmers sometimes like to compliment code as elegant, yet I can’t recall ever seeing a satisfying explanation of what “elegant code” is. Perhaps it’s telling that I see “elegant” used much less often by more experienced programmers, who opt for more concrete commentary.

Surely elegance is a quality to strive for, but how are we to strive for something we can’t define? “I know it when we see it” isn’t good enough.

I think about this from time to time. Here’s what I’ve come up with.

Some definitions

I get a gut feeling when something is elegant, and a different gut feeling altogether when something is hacky; I suspect most programmers experience the same. The strongest pattern I’ve found is this:

Elegance is about expressing exactly what you mean — no more, no less.

Conversely, I could define a hack as something that doesn’t remotely express what you mean, but happens to have a close-enough effect.

That’s not to say all code lies on a linear spectrum between two extremes. There’s some complexity here, because “what you mean” is less concrete than the shape of your code or what happens when it executes.

Consider my recent example of links recreated in JavaScript. You might implement such a faux link with some jQuery.

1
2
3
$('#link').click(function() {
    window.location = 'http://www.google.com/';
});

Isn’t that elegant? It’s short, it’s sweet, and it does exactly as it says: when the link element is clicked, navigate to Google.

No, of course not. jQuery is elegant, perhaps, for some set of simple operations. This code is a hack, but in a way that only a human could reckon. What the author actually meant was a link — not “an element that navigates when clicked upon”, but a link, the fundamental concept that makes the Web what it is. The concrete impact of this is that a bunch of stuff humans expect from links, like the ability to middle click, is conspicuously missing.

Okay, what if you could reproduce all of that extra functionality? What if you painstakingly researched how links behave on every operating system and in every browser, and recreated that with JavaScript? No one would look at the resulting pile of special cases and call it elegant. And yet it wouldn’t really be a hack, either. The code would express “simulate everything a link does”, and while that’s not the same as having a link, it’s at least fairly close. It’d fall into a third unnamed category where a problem is solved fairly rigorously, but the outcome isn’t pretty.

The trick here is, again, all about meaning. We like to pretend that programming is a purely abstract thing, and perhaps some of the ideas are, but the languages and tools are all ultimately designed for humans. They’re designed to make sense to humans (as much as possible within given constraints, anyway), and they’re designed to solve problems humans have.

Elegance is what happens when we find a way to express what we mean with the units of meaning that our tools provide.

Sometimes that’s not possible, so the rest of this post — spoilers! — will be some concrete examples that have crossed my path recently. Maybe they’ll give you a better idea of when and why I frown at computers.

ZDoom and PickActor

Every live object in ZDoom — monsters, weapons, pickups, etc. — is called an “actor” (or sometimes a “thing”). ZDoom’s scripting language has a PickActor function for, essentially, finding what actor another actor is looking at. You might use this to find what monster is under the player’s crosshair and show a health bar over it, say.

There’s a catch. ZDoom’s scripting language can not manipulate actors directly; there is no actor type. Instead, an actor can have a “TID” (“thing ID”). Most actor-related functions thus accept a TID and affect every actor with that TID simultaneously. Appropriately, PickActor doesn’t (can’t!) return the actor it finds, but assigns that actor a TID so you can manipulate it indirectly.

By default, PickActor will refuse to overwrite an existing TID, and returns 1 to indicate it found a target or 0 to indicate it didn’t. It has two flags that can change its behavior: one to force overwriting an existing TID, and one to change the return value to the found actor’s TID.

This is everything you need to know to understand the problem, which is: how do you use PickActor to pick an actor?

The actor might already have a TID, intended for some other purpose. If you use no flags, the function will return 1 to indicate success, but the target won’t have your chosen TID, so any attempts to manipulate the target will silently do nothing. If you use the flag that forces changing a TID, you’ll almost certainly break some other effect that needed to be able to identify that actor. If you use the flag that returns an existing TID, you might end up manipulating multiple actors, because actors can share a TID.

It seems that there’s no way at all to use this function correctly!

In truth, there is, but it relies on a little outside knowledge: scripts block the entire simulation. Unless you explicitly sleep (or run for far too long and get forcibly killed), the game state cannot change out from under you.

With that in mind, the solution is:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
    // Get the existing TID, if there is one.
    // A TID of 0 is the same as not having a TID, so "setting" it to 0 will not
    // change anything's TID.
    int old_tid = PickActor(..., tid=0, flags=PICKAF_RETURNTID);

    // Find a TID that's not currently in use.
    int new_tid = UniqueTID();

    // Do the "real" call, which both forcibly sets the TID and returns true if
    // something was actually found.
    if (PickActor(..., tid=new_tid, flags=PICKAF_FORCETID)) {
        do_some_stuff_with(new_tid);
        do_some_other_stuff_with(new_tid);

        // Restore the actor's TID to its original value.
        Thing_ChangeTID(new_tid, old_tid);
    }

This relies on calling PickActor twice: once to get the target’s old TID, and once to change it. As long as both calls have the same arguments, the result must be the same, because the game state is frozen for the duration of this code. If you need to operate on the target for more than one frame, well… you have some more work to do.

The workaround is certainly not elegant. “Look for an actor in this direction, twice” is not what I wanted to express. And yet it’s not a hack, either. The code above demonstrably does the correct thing in all cases, and is suitable as a permanent solution. It occupies that nebulous third category of “complete, but not pretty”.

PickActor, on the other hand, is a shambles. I don’t know if you can really call an API a “hack”, but if you can, I would definitely like to do so right here. The function alone does the wrong thing, the “force” flag does the wrong thing, and the correct solution of calling it twice is not remotely obvious.


I care about this because modding an ancient game is largely a hobbyist affair, and there are plenty of people in the Doom community for whom tinkering with ZDoom is their first exposure to programming. They aren’t going to realize the caveats of this function, let alone know how to fix them.

I don’t want to rag on anyone in particular who’s just trying to make a fun game mod, but there is a whole lot of bad ZDoom code floating around. People try stuff until it looks like it works, and then they leave it be. I don’t blame non-professionals for that. I blame tools that don’t offer the right building blocks for modders to express what they mean.

Tool design is important! It’s why I pick on programming languages. If the fundamental pieces at your disposal are awkwardly-shaped, you’ll have a much harder time expressing what you actually intended.

Inform 7

More spoilers: these are all going to be from video games.

Video games are surprisingly difficult to express. Programming languages are languages, so they usually inherit some of the properties of human languages: code is read (executed) in sequence, and large amounts of code can be broken into a hierarchy for easier understanding. For a great many applications, that works great.

For simulations (a superset of video games), that all goes to hell. You have a great many things all acting independently and simultaneously, each with slightly different concerns and behavior. Expressing that elegantly in a linear, hierarchical language is a lost cause right from the start. Operating systems could be described in much the same way, and I’ve never heard anyone call OS code particularly elegant either. We’ve barely even figured out how to expose threads in a language, without the associated mountain of gotchas and pitfalls.

Inform 7 is interesting because it’s explicitly designed for games, and even a particular kind of game. (Contrast with the more popular style of “something like C, but made to fit my engine”.) I should do a full post on it sometime, but here’s the tl;dr.

Inform 7 is intended for writing interactive fiction games (like Zork). The syntax resembles written English, roughly mirroring how the resulting game is played. For example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
The mysterious cavern is a room.
"Numerous stalagtites cast eerie shadows on the walls."

A golden crown is a wearable thing in the mysterious cavern.
"A golden glint pierces the darkness."
The description is "It's a golden crown."
Understand "glint" as the golden crown.

After wearing the golden crown:
    end the story finally saying "All hail the new queen!"

This is a full and complete game, which you can win by wearing the crown. There’s a lot of stuff crammed in here, and hopefully you get the general idea: objects are defined with English statements of fact, not by fenced-off blocks.

Inform 7 is particularly interesting because it attempts to address an ancient question of game design: how do you define the interaction between two mechanics?

This game’s win condition is wearing the crown — i.e., wearing the crown triggers the effect of winning the game. That code has to go somewhere, but does it go in the implementation of wearing or the implementation of the crown? Probably in the crown, since it’s a oneoff effect.

Consider, then, if I had a whole set of armor items with different defense ratings. Is that the concern of the action or the individual objects? It depends. If the vast majority of wearable objects are armor, then surely the act of wearing is concerned with handling armor. If there are only a few wearables that act as armor, they’re outliers, and should implement their own handling. Between these two extremes lies an ambiguous middle ground where it may not be clear who should really be responsible.

Every interaction in a simulated world has at least three possible “parents”: the action, who’s doing the action, and what the action is being done to. Expressing this in a language designed around hierarchy can feel clumsy when the “parent” is ambiguous, because a hierarchy necessarily requires that code only have one parent. (I’m not talking just about inheritance hierarchies, either; your actual source code can only exist in one place, nested inside one class, which is nested inside a namespace, which is nested inside a file, and so on.)

Inform 7 mostly throws the problem out the window. We can’t escape hierarchy entirely, because it’s how we as humans tend to make sense of the world. But instead of forcing the design of your game into a series of nested blocks, Inform 7 more or less lets you define anything anywhere. You can write “After” rules that apply to particular actions with particular objects, and you can put those rules anywhere in your source code: alongside the definition of wearing (well, not in this case, because that’s in the standard library), alongside the definition of the crown, or even somewhere else entirely that deals with all the ways to end the game. It’s completely up to you, and the compiler largely doesn’t care. You can even put the rule before you announce that the golden crown is an object at all, and the compiler will do a pretty good job of figuring it out.

You can still have a hierarchy, but it’s imposed entirely by you. Inform 7 lets you split your source text into a tree of volumes, books, parts, chapters, and sections, named and arranged however you like. You can see this in the source text for the example game, Bronze by Emily Short.


I tell you all this so I can compare it to the other extreme: NetHack. NetHack, written in C, is what you might call “data-oriented”. Consider for example peffects(), a 500-line function that single-handedly implements every possible effect a potion can have. polymon(), the function responsible for polymorphing the player into a monster, contains all manner of little details like halting the process of stoning if you polymorph into a stoning-resistant monster.

I once tried writing a roguelike that used entity component to avoid this kind of mess — no small feat, since it involves a whole lot of fighting back against the language design. Having beaten on that for a while, I can safely say I vastly prefer the Inform 7 model. It’s flexible enough for most purposes, and it has multiple layers of overriding, so you can make an object that prevents the golden crown from winning the game without having to touch the existing rule!

Given the constraints of how we read and write text (and code), and given that the intended usage is still fairly broad, I daresay Inform 7’s approach is fairly elegant. It allows me to clearly say what I mean in most cases, and that’s often enough.

Of course, I wouldn’t have brought it up without some more specific anecdotes.

Locksmith by Emily Short

That’s the name of an extension bundled with Inform 7. Extension names always include the name of the author, which is charming.

Locks and keys are a staple of interactive fiction, and Inform 7 has some built-in support for them. However, it requires you to explicitly name which key you’re using to unlock something. If this is part of a puzzle, it might be appropriate; in the common case of “iron key” unlocking “iron door”, it’s a little tedious.

Locksmith by Emily Short addresses this problem by splitting the “LOCK” and “UNLOCK” actions into two parts. If you say “UNLOCK DOOR WITH KEY”, that’s regular old unlocking. If you only say “UNLOCK DOOR”, that’s a new action called unlocking keylessly, and it tries to find a matching key in your inventory. “LOCK” works similarly.

Now, I want to make a padlock. You don’t need a key to close a padlock, so I want to hijack “locking keylessly” to work without needing a key at all. Both “CLOSE PADLOCK” and “LOCK PADLOCK” should do the same thing, so I use an instead rule to redirect both actions. I also want to make the padlock support the standard “open” property, but I define it in terms of whether it’s currently locked, since that’s how a padlock works.

1
2
3
4
5
6
Definition: the padlock is open rather than closed when it is not locked.
Instead of closing the padlock:
    try locking keylessly the padlock.
Instead of locking keylessly the padlock:
    now the padlock is locked;
    say "You snap the padlock shut.  It makes a chunky, satisfying click."

I compile this and type “CLOSE PADLOCK” and… the game freezes, trapped forever in an infinite loop. Or maybe the stack would overflow eventually.

What’s the problem? It’s not quite in my code; it’s an interaction with the extension, which has some convenience features. If you try to keylessly lock something that’s open (usually a door!), a before rule kicks in and makes you automatically close it first. Before rules happen before instead rules, so we have the following sequence of events.

  1. The player tries to close the (open) padlock.
  2. My code redirects this into having the player lock the padlock instead.
  3. The extension sees that the player is trying to lock something that’s open, and has the player try to close it first.
  4. GOTO 1

This is the downside of having a lot of override mechanisms. Arguably, the extension’s convenience behavior shouldn’t be in the actual implementation of “locking keylessly” rather than in before rules, but there are decent arguments to be made both ways.

The solution is remarkably simple: just swap the actions.

1
2
3
4
5
6
Definition: the padlock is open rather than closed when it is not locked.
Instead of locking keylessly the padlock:
    try closing the padlock.
Instead of closing the padlock:
    now the padlock is locked;
    say "You snap the padlock shut.  It makes a chunky, satisfying click."

And now I get exactly what I want:

> close padlock

You snap the padlock shut. It makes a chunky, satisfying click.

Simple, but perhaps not obvious. Something here is inelegant, yet everyone involved was pretty clearly expressing what they meant. I wonder what the lesson is.

Doors

Ah, good, more Inform 7 stuff, and more door stuff as well!

Doors are pretty straightforward in Inform 7: they have two sides, and each side goes in a room. The player can go through the door to get from one side to the other. Easy.

I want to have a door that can lead to several different places. In other words, the front side is fixed in place, but the back side can move around.

Inform 7 won’t let you do this, full stop. That means it’s time for an adventure.


The obvious solution is to fake it.

  1. Make a bunch of identical fake-door objects (which aren’t actually doors) and put one in each place the door can appear.
  2. Hide them from everywhere the door isn’t supposed to be.
  3. Override all the actions that could conceivably take the player through the door to just teleport them instead.

This is pretty clearly a hack. What I mean is a door, and this is as far away from a door as is possible to get. As is common with hacks, there are also thorny downsides to every step.

  1. Identical objects can be slightly awkward. In particular, if you try to use a custom command that works on objects you have seen but can’t currently see (like “remember”, perhaps), you may get a rather awful disambiguation prompt like: “Which do you mean? The green door, the green door, the green door, or the green door?”

    Also, since they aren’t doors, no door-specific behavior will work on them. Pathfinding won’t work, because the spaces aren’t actually connected. I want to implement looking through a door, and that won’t work. Built-in mechanisms like “Instead of going through the green door” won’t work.

    You might have slightly more luck if you used real doors that all lead nowhere, and very carefully hooked only the handling for where a door leads. But that has problems with part 2.

  2. Hiding objects is more complicated than you’d think! The most common suggestion I’ve seen is to do this:

    1
    2
    Instead of doing something with the hidden item:
        say "You can't see any such thing."
    

    That is, of course, the default response when you try to refer to an object that doesn’t exist. And this is a terrible hack! If you change that default response, this hack will get out of sync. If you use one of the available extensions to improve the error handling, this will stick out like a sore thumb. And just like with the padlock, a before rule will still trigger first.

    An easier option is to actually move the dummy objects around. Of course, if you decide to use real single-sided doors, you’re back to the original problem: you can’t move a door.

  3. Inform 7’s movement is remarkably complex. You can push objects from room to room and ride in vehicles. Merely teleporting the player would break these; you would need to reimplement them in some fashion.

None of these problems are insurmountable, but certainly this approach is difficult to implement correctly.


The solution I went for was to make it work by force.

You see, Inform 7 actually compiles to a very different language called Inform 6, which is more traditional and C-like (though with some pretty funky syntax in places). Parts of the standard library are implemented in Inform 6, and the detailed comments in those files reveal some interesting details.

One, Inform 6 models the entire world as… a hierarchy. Everything is one big tree. The room contains the player, and the player contains their inventory — the only thing distinguishing an inventory is that the parent is a person. Even worn objects are children like any other, except that they have an invisible “worn” flag.

This model presents a problem with doors, which exist in two places at once. The secret is… they actually don’t. The physical door object moves to follow the player! There’s some special consideration for doors in places to make them appear to be in both places from other characters’ points of view, so you’d never notice the difference.

Two, Inform 6 actually had an entire feature for objects that faked being in more than one room, called “floating”. These were apparently such a huge and buggy pain in the ass that Inform 7 drastically limited floating objects to only two specific categories: doors, and a certain kind of non-interactive decorations.

Three, every Inform 7 door ends up with a found_in property in Inform 6, which is an array of the two places the door should exist.

Well, hey, that sounds easy. You can embed chunks of Inform 6 code in Inform 7, so I wrote some code that would edit that array directly.

Foiled! It didn’t work. The problem is that going through a door takes you to the “other side”. “Other side” is a real physical property — an Inform 6 function that picks a side based on where the player currently is. Alas, rather than consulting the array, this function has the two sides hardcoded. It looks like this:

1
2
3
4
5
6
with door_to [ 
    loc ! room of actor
    ;
    loc = location;
    if (loc == thedark) loc = real_location;
    if (loc == I483_red_room) return I384_green_room; return I483_red_room;],

door_to is just the Inform 6 name of the “other side” property. loc is an argument to the function bounded by square brackets. Like I said, funky syntax.

I did try changing “the other side of the green door” from Inform 7 land, but it seems the compiler special-cases this one particular property name and intercepts any attempt to change it dynamically. Foiled again!

The solution I settled on was to rewrite the “other side” function for this particular door entirely. I ended up with:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
Include (-
[ HackedOtherSide loc;
    loc = location;
    if (loc == thedark) loc = real_location;
    if (loc == (+ the green door +).&found_in-->0) return (+ the green door +).&found_in-->1; return (+ the green door +).&found_in-->0;
];
[ HackedDirection loc;
    loc = location;
    if (loc == thedark) loc = real_location;
    if (loc == (+ the green door +).&found_in-->0) return (+ southwest +); return (+ northeast +);
];
-).

To hackily override the door props:
    (-
    (+ the green door +).door_dir = HackedDirection;
    (+ the green door +).door_to = HackedOtherSide;
    -).

When play begins:
    hackily override the door_to.

To hackily force the back side of (portal - a door) to (where - an object):
    (- ({portal}.&found_in)-->1 = {where}; -).

(- ... -) denotes embedded Inform 6 code; (+ ... +) denotes embedded Inform 7 object names; foo-->0 is an array subscript. door_to is another property that tells you what direction the door is in, based on what room you’re in.

Right. So.

This is definitely… uglier, at a glance, than faking the door. On the other hand, it’s pretty much transparent to the rest of the language and standard library. The door is a real door that genuinely moves from place to place. Anything that operates on a door goes through the code I’ve overridden here, so it’s fairly bulletproof.

The one issue not addressed here is route-finding — the standard library can cache information about connections between rooms to make route-finding faster, and after moving the door, that cache is no longer valid. A quick read of the comments reveals a trivial fix: call SignalMapChange() after moving the door around.

So. Is this elegant?

It sure doesn’t look pretty, and it requires meddling with guts that I’m not even supposed to know about. That’s not particularly elegant.

And yet! I only did this in the first place because it lets all the rest of my code express exactly what I want. I don’t have to keep remembering that my doors are fake and sticking special cases everywhere. This code is self-contained, robust, and forgettable. It sacrifices elegance so that other code doesn’t have to.

I think that’s something to value.

Starbound versus Aseprite

My door hack is not so much about code elegance, but interface elegance. It does some strange things in order to preserve all the guarantees and semantics of a fundamental existing interface: the door.

I’d like to close with a more literal example of that.

Hardware-accelerated programs generally don’t use native GUI controls. This generally means: video games implement their own buttons, textboxes, etc. I’ve heard rumors that it’s possible to mash native GUI controls into an accelerated canvas, but I can’t recall ever having seen it done.

I got into Starbound for a while last year. I even ran a little private server for some friends. And one constant aggravation I had was that Tab doesn’t move the focus between textboxes in dialogs. The “connect to server” dialog is just three boxes: hostname, username, password. It seems so obvious that you should be able to tab between these, but you cannot. Whoever built the GUI didn’t implement tabbing.

Some other common keyboard shortcuts are wrong, which trips me up from time to time. Ctrlarrow moves the cursor only one character at a time, rather than one word at a time. Shiftarrow moves one word at a time, rather than selecting a character. CtrlBackspace appears to clear an entire textbox, rather than deleting the previous word.

It’s not surprising that a few things would slip through the cracks. Game GUI libraries have to be reimplemented from scratch, and that’s an awful lot of work just to get the buttons and textboxes that other developers get to take for granted.

A few days ago, I tried Aseprite, a pixel art editor. The entire interface is custom-drawn (in a pixellated style, no less), which had me worried that I’d encounter the same problems as in Starbound.

To my great surprise, I didn’t! All the keys I’m used to work just fine. I can even double-click a word to select it. I don’t know if this is a third-party library or something the Aseprite developer built themselves, but someone put a lot of work into making this feel natural.

I don’t usually like custom UIs at all. Between the very appropriate design and the attention to detail, I’m pretty sold on Aseprite’s.

I might even call it… elegant.

It doesn’t matter that I haven’t seen the code. Maybe the code is a mess. In fact it almost certainly is. The end result is what’s elegant here, because it completely disappears. I can forget I’m not using native controls.

Maybe that’s a better definition of elegance, then: elegance is transparent. It’s so natural that it disappears entirely. A hack stands out because it’s unusual and jarring; an elegant solution (or interface, or whatever) fits together so smoothly that you forget the problem ever existed.

Germany Trip: Samba XP Keynote and LinuxTag Keynote

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2011/05/18/germany.html

I just returned a few days ago to the USA after one week in Germany. I
visited Göttingen for my keynote at Samba XP (which
I already
blogged about
).
Attending Samba XP was
an excellent experience, and I
thank SerNet for sponsoring my
trip there. Since going full-time at Conservancy last year, I have been
trying to visit the conferences of each of Conservancy’s member
projects. It will probably take me years to do this, but given that
Samba is one of Conservancy’s
charter members,
it’s good that I have finally visited Samba’s annual conference. It was
even better that they asked me to give
a keynote talk
at Samba XP
.

I must admit that I didn’t follow the details many of the talks other
than Tridge’s Samba 4 Status Report talk and
Jeremy’s The Death of File Protocols. This time I really mean
it! talk. The rest, unsurprisingly, were highly specific and
detailed about Samba, and since I haven’t been a regular Samba user
myself since 1996, I didn’t have the background information required to
grok the talks fully. But I did see a lot of excited developers, and it
was absolutely wonderful to meet the entire Samba Team for the first
time after exchanging email with them for so many years.

It’s funny to see how different communities tend to standardize around
the same kinds of practices with minor tweaks. Having visited a lot of
project-specific conferences for Conservancy’s members, I’m seeing how
each community does their conference, and one key thing all projects
have in common is the same final conference session: a panel discussion
with all the core developers.

The Samba Team has their own little tweak on this.
First, John Terpstra asks all
speakers at the conference (which included me this year) to join the
Samba Team and stand up in front of the audience. Then, the audience
can ask any final questions of all speakers (this year, the attendees
had none). Then, the Samba Team stands up in front of the crowd and
takes questions.

The Samba tweak on this model is that the Samba Team is not permitted
to sit down during the Q&A. This year, it didn’t last that long,
but it was still rather amusing. I’ve never seen a developers’ panel
before where the developers couldn’t sit down!

After Samba XP, I headed “back” to Berlin (my
flight had landed there on Saturday and I’d taken the Deutsche Bahn ICE
train to Göttingen for Samba XP), and arrived just in
time to
attend LinuxNacht,
the LinuxTag annual party
. (WARNING: name dropping follows!) It was excellent to
see Vincent
Untz
, Lennart Poettering,
Michael Meeks and
Stefano Zacchiroli at the party
(listed in order I saw them at the party).

The next day I
attended Vincent’s
talk, which was about cross-distribution collaboration
. It was a
good talk, although, I think Vincent glossed over too much the fact that
many distributions (Fedora, Ubuntu, and OpenSUSE, specifically) are
controlled by companies and that cross-distribution collaboration has
certain complications because of this corporate influence. I talked
with Vincent in more detail about this later, and he argued that the
developers at the companies in question have a lot of freedom to
operate, but I maintain there are subtle (and sometimes, not so subtle)
influences that cause problems for cross-distribution collaboration. I
also encouraged Vincent to listen
to Richard Fontana‘s talk, Open
Source Projects and Corporate Entanglement, that Karen and I
released as an episode
of the
FaiF oggcast
.

I also attended Martin
Michlmayr
‘s talk
on SPDX
. I kibitzed more than I should have from the audience,
pointing out that while SPDX is a good “first start”, it’s a
bit of a “too little, too late” attempt to address and
prevent the flood of GPL violations that are now all too common. I
believe SPDX is a great tool for those who already are generally in
compliance, but it isn’t very likely to impact the more common
violations, wherein the companies just ignore their GPL obligations. A
lively debate ensued on this topic. I frankly hope to be proved wrong
on this; if SPDX actually ends or reduces GPL violations, I’ll be happy
to work on something else instead.

On Friday afternoon, I gave
my second
keynote of the week, which was an updated version of my talk, 12
Years of GPL Compliance: A Historical Perspective
. It went
well, although I misunderstood and thought I had a full hour slot, but
only actually had a 50 minute slot, so I had to rush a bit at the end. I
really do hate rushing at the end when speaking primarily to a
non-native-English-speaking audience, as I know I’m capable of speaking
English way too fast (a problem that I am constantly vigilant
about under normal public speaking circumstances).

The talk was nevertheless pretty well received, and afterward, I was
surrounded by a gaggle of interested copyleft enthusiasts, who, as
always, were asking what more can be done to enforce the GPL. My talks
on enforcement always tend to elicit this reaction, since my final
slides are a
bit depressing with regard to the volume of GPL enforcement that’s
currently occurring.

Meanwhile, I also decided I should also start putting up my slides from
talks in a more accessible fashion. Since I
use S5 (although I hope
to switch to jQuery
S5
RSN), my slides are trivially web-publishable anyway. While
I’ve generally
published the source code to my slides
, it makes sense to also make
compiled, quickly viewable
versions of my slides
on my website too. Finally, I realized I
should also put my upcoming public
speaking events on my frontpage
and have done so.

After a late lunch on Friday, I saw only the very end
of Lennart’s
talk on systemd
, and then I visited for a while
with Claudia
Rauch
, Business
Manager of KDE, e.V.
in the KDE booth. Claudia kindly helped me
practice my German a bit by speaking slowly enough that I could actually
parse the words.

I must admit I was pretty frustrated all week that my German is now so
poor. I studied German for two years in High School and one semester in
college. I even participated in a three-week student exchange trip to a
Gymnasium (the German term for college-prep high school) in Munich in
1990. Yet, German speaking skills are just a degraded version of what
they once were.

Meanwhile, I did rather like Berlin’s Tegel airport (TXL). It’s a
pretty small airport, but I really like its layout. Because of its
small size, each check-in area is attached to a security checkpoint,
which is then directly connected to the gate. While this might seem a
bit tight, it makes it very easy to check-in, go through security, and
then be right at your gate. I can understand why an airport this small
would have to be closed (it’s slated for closure in 2012), but I am glad
that I got a chance to travel to it (and probably again, for the Desktop
Summit) before it closes.

Conservancy Activity Summary, 2010-10-01 to 2010-12-31

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2011/01/02/conservancy-1.html

[ Crossposted
from Conservancy’s
blog
. ]

I had hoped to blog more regularly about my work at Conservancy, and
hopefully I’ll do better in the coming year. But now seems a good time
to summarize what has happened with Conservancy since I started my
full-time volunteer stint as Executive Director from 2010-10-01 until
2010-12-31.

New Members

We excitedly announced in the last few months two new Conservancy
member
projects, PyPy
and Git.
Thinking of PyPy connects me back to my roots in Computer Science: in
graduate school, I focused on research about programming language
infrastructure and, in particular, virtual machines and language
runtimes. PyPy is a project that connects Conservancy to lots of
exciting programming language research work of that nature, and I’m glad
they’ve joined.

For its part, Git rounds out a group of three DVCS projects that are
now Conservancy members; Conservancy is now the home of Darcs, Git, and
Mercurial. Amusingly, when I reminded the Git developers when they
applied that their “competition” were members, the Git
developers told me that they were inspired to apply because these other
DVCS’ had been happy in Conservancy. That’s a reminder that the
software freedom community remains a place where projects — even
that might seem on the surface as competitors — seek to get along
and work together whenever possible. I’m glad Conservancy now hosts all
these projects together.

Meanwhile, I remain in active discussions with five projects that have
been offered membership in Conservancy. As I always tell new projects,
joining Conservancy is a big step for a project, so it often takes time
for communities to discuss the details of Conservancy’s Fiscal
Sponsorship Agreement. It may be some time before these five projects
join, and perhaps they’ll ultimately decide not to join. However, I’ll
continue to help them make the right decision for their project, even if
joining a different fiscal sponsor (or not joining one at all) is the
ultimately right choice.

Also, about once every two weeks, another inquiry about joining
Conservancy comes in. We won’t be able to accept all the projects that
are interested, but hopefully many can become members of
Conservancy.

Annual Filings

In the late fall, I finished up Conservancy’s 2010 filings. Annual
filings for a non-profit can be an administrative rat-hole at times, but
the level of transparency they create for an organization makes them worth
it.
Conservancy’s FY
2009 Federal Form 990

and FY
2009 New York CHAR-500
are up
on Conservancy’s filing
page
. I always make the filings available on our own website; I wish
other non-profits would do this. It’s so annoying to have to go to a
third-party source to grab these documents. (Although New York State, to
its credit, makes all
the NY
NPO filings available on its website
.)

Conservancy filed a Form 990-EZ in FY 2009. If you take a look, I’d
encourage you to direct the most attention to Part III (which is on the
top of page 2) to see most of Conservancy’s program activities between
2008-03-01 to 2009-02-28.

In FY 2010, Conservancy will move from the New York State requirement
of “limited financial review” to “full audit“
(see page 4 of the CHAR-500 for the level requirements). Conservancy
had so little funds in FY 2007 that it wasn’t required to file a Form 990 at all.
Now, just three years later, there is enough revenue to warrant a full
audit. However, I’ve already begun preparing myself for all the
administrative work that will entail.

Project Growth and Funding

Those increases in revenue are related to growth in many of
Conservancy’s projects. 2010 marked the beginning of the first
full-time funding of a developer by Conservancy. Specifically, since
June, Matt
Mackall has been funded through directed donations to Conservancy to
work full-time on Mercurial
.
Matt blogs once a month (under
topic of Mercurial Fellowship Update)
about his work,
but, more directly,
the hundreds
of changesets that Matt’s committed really show
the advantages of
funding projects through Conservancy.

Conservancy is also collecting donations and managing funding for
various part-time development initiatives by many developers.
Developers of jQuery, Sugar Labs, and Twisted have all recently received
regular development funding through Conservancy. An important part of
my job is making sure these developers receive funding and report the
work clearly and fully to the community of donors (and the general
public) that fund this work.

But, as usual with Conservancy, it’s handling of the “many little
things” for projects that make a big difference and sometimes
takes the most time. In late 2010, Conservancy handled funding for Code
Sprints and conferences for
the Mercurial, Darcs,
and jQuery. In addition, jQuery
held a conference in
Boston in October
, for which Conservancy handled all the financial
details. I was fortunate to be able to attend the conference and meet
many of the jQuery developers in person for the first time. Wine also
held their annual conference in November 2010, and Conservancy handled
the venue details and reimbursements to many of travelers to the
conference.

Also, as always, Conservancy project contributors regularly attend
other conferences related to their projects. At least a few times a
month, Conservancy reimburses developers for travel to speak and attend
important conferences related to their projects.

Google Summer of Code

Since its inception, Google’s Summer of Code (SoC) program has been one
of the most important philanthropy programs for Open Source and Free
Software projects. In 2010, eight Conservancy projects (and 5% of the
entire SoC program) participated in SoC. The SoC program funds college
students for the summer to contribute to the projects, and an
experienced contributor to project mentors each student. A $500 stipend
is paid to the non-profit organization of the project for each project
contributor who mentors a student.

Furthermore, there’s an annual conference, in October, of all the
mentors, with travel funded by Google. This is a really valuable
conference, since it’s one of the few places where very disparate Free
Software projects that usually wouldn’t interact can meet up in one
place. I attended this year’s Soc Mentor Summit and hope to attend
again next year.

I’m really going to be urging all Conservancy’s projects to take
advantage of the SoC program in 2011. The level of funding given out by
Google for this program is higher than any other open-application
funding program for
FLOSS.
While Google’s selfish motives are clear (the program presumably helps
them recruit young programmers to hire), the benefit to Free Software
community of the program can nevertheless not be ignored.

GPL Enforcement

GPL Enforcement,
primarily for our BusyBox member
project, remains an active focus of Conservancy. Work regarding the
lawsuit continues. It’s been more than a year since Conservancy filed a
lawsuit against fourteen defendants who manufacture embedded devices
that included BusyBox without source nor an offer for source. Some of
those have come into compliance with the GPL and settled, but a number
remain and are out of compliance; our litigation efforts continue.
Usually, our lawyers encourage us not to comment on ongoing litigation,
but we did put up
a news
item in August when the Court granted Conservancy a default judgment
against one of the defendants, Westinghouse
.

Meanwhile, in the coming year, Conservancy hopes to expand efforts to
enforce the GPL. New violation reports on BusyBox arrive almost daily
that need attention.

More Frequent Blogging

As noted at the start of this post, my hope is to update Conservancy’s
blog more regularly with information about our activities.

This blog post was covered on
LWN
and
on lxnews.org.

Does “Open Core” Actually Differ from Proprietary Relicensing?

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2010/10/19/proprietary-relicensing.html

I’ve been criticized
quite
a bit
this week,
but before
that
too — for using the
term “Open
Core”
as a shortcut for the phrase “proprietary
relicensing0
that harms software freedom”.
Meanwhile, Matt
Aslett points

to Andrew
Lampitt’s “Open Core” definition as canonical
. I admit
I wasn’t aware of Lampitt’s definition before, but I dutifully read it
when Aslett linked to it, and I quote it here:

[Lampitt] propose[s] the following for the Open Core Licensing business model:

core is GPL: if you embed the GPL in closed source, you pay a fee

technical support of GPL product may be offered for a fee (up for
debate as to whether it must be offered)

annual commercial subscription includes: indemnity, technical support,
and additional features and/or platform support. (Additional commercial
features having viewable or closed source, becoming GPL after timebomb
period are both up for debate).

professional services and training are for a fee.

The amusing fact about this definition is that half the things on it
(i.e., technical support, services/training, indemnity, tech support)
can be part of any
FLOSS business
model and do not require the offering company to hold the exclusive
right of proprietary relicensing. Meanwhile, the rest of the items on the list are definitely
part of what was traditionally called the “proprietary relicensing
business“ dating back to the late 1990s: namely, customers can buy
their way out of GPL obligations, and a single company can exclusively
offer proprietary add-ons. For example, this is precisely what Ximian did
with their Microsoft Exchange Connector for Evolution, which predated the
first use of the term “Open Core” by nearly a
decade. Cygnus
also used this model for Cygwin
, which has unfortunately continued at
Red Hat
(although Richard
Fontana of Red Hat wants to end the copyright assignment of
Cygwin
).

In my opinion, mass terminology confusion exists on this point simply
because there is a spectrum1 of behaviors that
are all under the banner of “proprietary relicensing”.
Moreover, these behaviors get progressively worse for software freedom
as you continue down the spectrum. Nearly the entire spectrum consists
of activities that are harmful to software freedom (to varying degrees),
but the spectrum does begin with a practice that is barely
legitimate.

That practice is one that
RMS’ himself began
calling barely legitimate in the early 2000s. RMS specifically and
carefully coined his own term for it:
selling
exceptions to the GPL
. This practice is a form of proprietary
relicensing that never permits the seller to create their own
proprietary fork of the code and always releases all
improvements done by the sole proprietary licensee itself to the general public.
If this practice is barely legitimate, it stands to reason that anything that goes
even just a little bit further crosses the line into illegitimacy.

From that perspective, I view this spectrum of proprietary relicensing
thusly: on the narrow benign end of the spectrum we find what RMS calls
“exception selling” and on the other end, we find GPL’d
demoware that is merely functional enough to convince customers to call
up the company to ask to buy more. Everything beyond “selling
exceptions” in harmful to software freedom, getting progressively
more harmful as you move further down the spectrum. Also,
notwithstanding Lampitt’s purportedly canonical definition, “Open
Core” doesn’t really have a well-defined meaning. The best we can
say is that “Open Core” must be something beyond
“selling exceptions” and therefore lives somewhere outside
of the benign areas of “proprietary relicensing”. So, from
my point of view, it’s not a question of whether or not
“Open Core” is a benign use of GPL: it clearly isn’t. The
only question to be asked is: how bad is it for software freedom, a
little or a lot? Furthermore, I don’t really care that much how
far a company gets into “proprietary relicensing”, because
I believe it’s already likely to be harmful to software freedom. Thus,
focusing debate only on how bad is it? seems to be missing the
primary point: we should shun nearly all proprietary relicensing models entirely.

Furthermore, I believe that once a company starts down the path of this
proprietary relicensing spectrum, it becomes a slippery slope. I have never seen the benign
“exception selling” last for very long in practice. Perhaps
a truly ethical company might stick to the principle, and would thus
use an
additional promise-back as RMS’ suggests
to prove to the community
they will never veer from it. RMS’ suggested texts have only been
available for less than a month, so more time is needed to see if they
are actually adopted. Of course, I call on any company asking for a
CLA and/or
CAA to adopt
RMS’ texts, and I will laud any company that
does.

But, pragmatically, I admit I’ll be (pleasantly) surprised if most
CAA/CLA-requesting companies come forward to adopt RMS’
suggested texts. We have a long historical list of examples of for-profit
corporate CAAs and CLAs being used for more nefarious purposes than
selling exceptions, even when that wasn’t the original intent. For
example2, When MySQL AB switched to GPL, they started benignly selling
exceptions, but, by the end of their reign, part of their marketing was
telling potential “customers” that they’d violated the GPL
even when they hadn’t — merely to manipulate the customer into
buying a proprietary license. Ximian initially had no plans to make
proprietary add-ons to Evolution, but nevertheless made use of their
copyright assignment to make the Microsoft Exchange Connector.
Sourceforge, Inc. (named VA Linux at the time) even went so far as to
demand copyright assignments on the Sourceforge code after the fact
(writing out changes by developers who refused) so they could move to an
“Open Core”-style business model. (Ultimately,
Sourceforge.net became merely demoware for a proprietary product.)

In short, handing over copyright assignment to a company gives that
company a lot of power, and it’s naïve to believe a for-profit
company won’t use every ounce of that power to make a buck when it’s not
turning a profit otherwise. Non-profit assignors, for their part,
mitigate the situation by making firm promises back regarding what will
and won’t be done with the code, and also (usually) have well-defined
non-profit missions that prevent them from moving in troubling
directions. For profit companies don’t usually have either.

Without strong assurances in the
agreement, like
the ones RMS suggests
, individual developers simply must assume the
worst when assigning copyright and/or giving a broad CLA to a for-profit
company. Whether we can ever determine what is or is not “Open
Core”, history shows us that for-profit companies with exclusive
proprietary relicensing power eventually move away from the (extremely
narrow) benign end of the proprietary relicensing spectrum.

0Most
pundits will prefer the term “dual licensing” for
what I call “proprietary relicensing”. I urge
avoidance of the term “dual licensing”.
“Dual licensing” also has a completely
orthogonal denotative usage: a Free Software license that has
two branches, like jQuery’s
license of (GPLv2-or-later|MIT)
. That terminology usage
was quite common before even the first “proprietary
relicensing” business model was dreamed of, and therefore
it only creates confusion to overload that term further.

1BTW, Lampitt
does deserve some credit here. His August 2008 post hints at
this spectrum idea of proprietary licensing models. His post
doesn’t consider the software-freedom implications of the
various types, but it seems to me that post was likely ahead of
its time for two years ago, and I wish I’d seen it sooner.

2I give
here just of a few of the many examples, which actually name
names. Although he doesn’t name
names, Michael
Meeks, in his Some Thoughts on Copyright
Assignment
, gives quite a good laundry list of all
the software-freedom-unfriendly things that have historically
happened in situations where CAA/CLAs without adequate
promises back were used.