All posts by esr

Looking for C-to-anything transpilers

Post Syndicated from esr original http://esr.ibiblio.org/?p=8705

I’m looking for languages that have three properties:

(1) Must have weak memory safety. The language is permitted to crash on an out -of-bounds array reference or null pointer, but may not corrupt or overwrite memory as a result.

(2) Must have a transpiler from C that produces human-readable, maintainable code that preserves (non-perverse) comments. The transpiler is allowed to not do a 100% job, but it must be the case that (a) the parts it does translate are correct, and (b) the amount of hand-fixup required to get to complete translation is small.

(3) Must not be Go, Rust, Ada, or Nim. I already know about these languages and their transpilers.

Designing tasteful CLIs: a case study

Post Syndicated from esr original http://esr.ibiblio.org/?p=8697

Yesterday evening my apprentice, Ian Bruene, tossed a design question at me.

Ian is working on a utility he calls “igor” intended to script interactions with GitLab, a major public forge site. Like many such sites, it has a sort of remote-procedure-call interface that allows you, as an alternative to clicky-dancing on the visible Web interface, to pass it JSON datagrams and get back responses that do useful things like – for example – publishing a release tarball of a project where GitLab users can easily find it.

Igor is going to have (actually, already has) one mode that looks like a command interpreter for a little minilanguage, with each command being an action verb like “upload” or “release”. The idea is not so much for users to drive this manually as for them to be able to write scripts in the minilanguage which become part of a project’s canned release procedure. (This is why GUIs are irrelevant to this whole discussion; you can’t script a GUI.)

Ian, quite reasonably, also wants users to be able to run simple igor commands in a fire-and-forget mode by typing “igor” followed by command-line arguments. Now, classically, under Unix, you would expect a single-line “release” command to be designed to look something like this:

$ igor -r -n fooproject -t 1.2.3 foo-1.2.3.tgz

(To be clear, the dollar sign on the left is a shell prompt, put in to emphasize that this is something you type direct to a shell.)

In this invocation, the “-r” option says “I want to do a release”, the -n option says “This is the GitLab name of the project I’m shipping a release of”, the -t option specifies a release tag, and the following filename argument is the name of the tarball you want to publish.

It might not look exactly like this. Maybe there’d be yet another switch that lets you attach a release notes file. Maybe you’d have the utility deduce the project name from the directory it’s running in. But the basic style of this CLI (= Command Line Interface), with option flags like -r that act as command verbs and other flags that exist to attach their arguments to the request, is very familiar to any Unix user. This what most Unix system commands look like.

One of the design rules of the old-school style is that the first token on the line that is not a switch argument terminates recognition of switches. It, and all tokens after it, are treated as arguments to be passed to the program and are normally expected to be filenames (or, in the 21st century, filename-like things like URLs).

Another characteristic of this style is that the order of the switch clauses is not fixed. You could write

$ igor -t 1.2.3 -n fooproject -r foo-1.2.3.tgz

and it would mean the same thing. (Order of the following arguments, on the other hand, will usually be significant if there is more than one.)

For purposes of this post I’m going to call this style old-school UNIX CLI, because Ian’s puzzlement comes from a collision he’s having with a newer style of doing things. And, actually, with a third interface style, also ancient but still vigorous.

When those of us in Unix-land only had the old-school CLI style as a model it was difficult to realize that all of those switches, though compact and easy to type, imposed a relatively high cognitive load. They were, and still are, difficult to remember. But we couldn’t really notice this until we had something to contrast it with that met similar functional requirements with lower cognitive effort.

Though there may have been earlier precedents, the first well-known program to use something recognizably like what I will call new-school CLI was the CVS version control system. The distinguishing trope was this: Each CVS command begins with a subcommand verb, like “cvs update” or “cvs checkout”. If there are switches, they normally follow the subcommand rather than preceding it. And there are fewer switches.

Later version-control systems like Subversion and Mercurial picked up on the subcommand idea and used it to further reduce the number of arbitrary-looking switches users had to remember. In Subversion, especially, your normal workflow could consist of a sequence of svn add, svn update, svn status, and svn commit commands during which you’d never type anything that looked like an old-school Unixy switch at all. This was easy to remember, easy to document, and users liked it.

Users liked it because humans are used to remembering associations between actions and natural-language verbs; “release” is less of a memory load than “-r” even if it takes longer to type. Which illuminates one of the drivers of the old-school style; it was shaped back in the 1970s by 110-baud Teletypes on which terseness and only having to type few characters was a powerful virtue.

After Subversion and Mercurial Git came along, with its CLI written in a style that, though it uses leading subcommand verbs, is rather more switch-heavy. From the point of view of being comfortable for users (especially new users), this was a pretty serious regression from Subversion. But then the CLI of git wasn’t really a design at all, it was an accretion of features that there was little attempt to simplify or systematize. It’s fair to say that git has succeeded despite its rather spiky UI rather than because of it.

Git is, however a digression here; I’ve mainly described it to make clear that you can lose the comfort benefits of the new-school CLI if a lot of old-school-style switches crowd in around the action verbs.

Next we need to look at a third UI style, which I’m going to call “GDB style” because the best-known program that uses it today is the GNU symbolic debugger. It’s almost as ancient as old-school CLIs, going back to the early 1980s at least.

A program like GDB is almost never invoked as a one-liner at all; a command is something you type to its internal command prompt, not the shell. As with new-school CLIs like Subversuon’s, all commands begin with an action verb, but there are no switches. Each space-separated token after the verb on the command line is passed to the command handler as a positional argument.

Part of Igor’s interface is intended to be a GDB-style interpreter. In that, the release command should logically look something like this, with igor’s command prompt at the left margin.

igor> release fooproject 1.2.3 foo-1.2.3.tgz

Note that this is the same arguments in the same order as our old-school “igor -r” command, but now -r has been replaced by a command verb and the order of what follows it is fixed. If we were designing Igor to be Subversion-like, with a fire-and-forget interface and no internal command interpreter at all, it would correspond to a shell command line like this:

$ igor release fooproject 1.2.3 foo-1.2.3.tgz

This is where we get to the collision of design styles I referred to earlier. What was really confusing Ian, I think, is that part of his experience was pulling for old-school fire-and-forget with switches, part of his experience was pulling for new-school as filtered through git’s rather botched version of it, and then there is this internal GDB-like interpreter to reconcile with how the command line works.

My apprentice’s confusion was completely reasonable. There’s a real question here which the tradition he’s immersed in has no canned, best-practices answer for. Git and GDB evade it in equal and opposite ways – Git by not having any internal interpreter like GDB, GDB by not being designed to do anything in a fire-and-forget mode without going through its internal interpreter.

The question is: how do you design a tool that (a) has a GDB like internal interpreter for a command minilanguage, (b) also allows you to write useful fire-and-forget one-liners in the shell without diving into that interpreter, (c) has syntax for those one liners that looks like an old-school CLI, and (d) has only one syntax for each command?

And the answer is: you can’t actually satisfy all four of those constraints at once. One of them has to give. It’s trivially true that if you abandon (a) or (b) you evade the problem, the way Git and GDB do. The real problem is that an old-school CLI wants to have terse switch clauses with flexible order, a GDB-style minilanguage wants to have more verbose commands with positional arguments, and never these twain shall meet.

The only one-syntax-for-each-command choice you can make is to have the same command interpreter parse your command line and what the user types to the internal prompt.

I bit this bullet when I designed reposurgeon, which is why a fire-and-forget command to read a stream dump of a Subversion repository and build a live repository from it looks like this:

$ reposurgeon "read <project .svn" "prefer git" "rebuild ../overthere"

Each of those string arguments is just fed to reposurgeon’s internal interpreter; any attempt to look like an old-school CLI has been abandoned. This way, I can fire and forget multiple reposurgeon commands; for Igor, it might be more appropriate to pass all the tokens on the command line as a single command.

The other possible way Igor could go is to have a command language for the internal interpreter in which each line looks like a new-school shell command with a command verb followed by switch clusters:

$ release -t 1.2.3 -n fooproject foo-1.2.3.tgz

Which is fine except that now we’ve violated some of the implicit rules of the GDB style. Those aren’t simple positional arguments, and we’re back to the higher cognitive load of having to remember cryptic switches.

But maybe that’s what your prospective users would be comfortable with, because it fits their established habits! This seems to me unlikely but possible.

Design questions like these generally come down to having some sense of who your audience is. Who are they? What do they want? What will surprise them the least? What will fit into their existing workflows and tools best?

I could launch into a panegyric on the agile-programming practice of design-by-user-story at this point; I think this is one of the things agile most clearly gets right. Instead, I’ll leave the reader with a recommendation to read up on that idea and learn how to do it right. Your users will be grateful.

Two graceful finishes

Post Syndicated from esr original http://esr.ibiblio.org/?p=8694

I’m having a rather odd feeling.

Reposurgeon. It’s…done; it’s a finished tool, fully fit for its intended purposes. After nine years of work and thinking, there’s nothing serious left on the to-do list. Nothing to do until someone files a bug or something in its environment changes, like someone writing an exporter/importer pair it doesn’t know about and should.

When you wrestle with a problem that is difficult and worthy for long enough, the problem becomes part of you. Having that go away is actually a bit disconcerting, like putting your foot on a step that’s not there. But it’s OK; there are lots of other interesting problems out there and I’m sure one will find me to replace reposurgeon’s place in my life.

I might try to write a synoptic look back on the project at some point.

Looking over some back blog posts on reposurgeon, I became aware that I never told my blog audience the last bit of the saga following my ankle surgery. That’s because there was no drama. The ankle is now fully healed and as solidly functional as though I never injured it at all – I’ve even stopped having residual aches in damp weather.

Evidently the internal cartilage healed up completely, which is far from a given with this sort of injury. My thanks to everyone who was supportive when I literally couldn’t walk.

Term of the day: builder gloves

Post Syndicated from esr original http://esr.ibiblio.org/?p=8688

Another in my continuing series of attempts to coin, or popularize, terms that software engineers don’t know they need yet. This one comes from my apprentice, Ian Bruene.

“Builder gloves” is the special knowledge possessed by the builder of a tool which allows the builder to use it without getting fingers burned.

Software that requires builder gloves to use is almost always faulty. There are rare exceptions to this rule, when the application area of the software is so arcane that the builder’s specialist knowledge is essential to driving it. But usually the way to bet is that if your code requires builder gloves it is half-baked, buggy, has a poorly designed UI or is poorly documented.

When you ship software that you know requires builder gloves, or someone else tells you that it seems to require builder gloves, it could ruin someone else’s day and reflect badly on you. But if you believe in releasing early and often, sometimes half-baked is going to happen. Here’s how to mitigate the problem.

1. Warn the users what’s buggy and unstable in your release notes and the rest of your documentation.

2. Document your assumptions where the user can see them,

3. Work harder at not being a terrible UI designer.

Becoming really good at software engineering requires that you care about the experience the user sees, not just the code you can see.

This is your final warning

Post Syndicated from esr original http://esr.ibiblio.org/?p=8685

Earlier today, armed demonstrators stormed the Michigan State House protesting the state’s stay-at-home order.

I’m not going to delve in to the specific politics around the stay-at-home order, or whether I think it’s a good idea or a bad one, because there is a more important point to be made here. Actually, two important points.

(1) Nobody got shot. These protesters were not out-of-control yahoos intent on violence. This was a carefully calibrated and very controlled demonstration.

(2) This is the American constitutional system working correctly and as designed by the Founders. When the patience of the people has been pushed past its limit by tyranny and usurpation, armed revolt is what is supposed to happen. The threat of popular armed revolt is an intentional and central part of our system of checks and balances.

We aren’t at that point yet, though. The Michigan legislators should consider that they have received a final warning before actual shooting. The protesters demonstrated and threatened just as George Washington, Thomas Jefferson, Patrick Henry, and other Founders expected and wanted citizens to demonstrate and threaten in like circumstances.

I am sure there will be calls from the usual suspects to tighten gun laws and arrest the protesters as domestic terrorists. All of which will miss the point. Nobody got shot – this was the last attempt, within the norms of the Constitutional system as designed, to avoid violence.

If the Michigan state government responds to this demonstration with repression or violence, citizens will have the right – indeed, they will have a Constitutional duty – to correct the arrogance of power via armed revolt.

This was your final warning, legislators. Choose wisely.

The feel for weapons

Post Syndicated from esr original http://esr.ibiblio.org/?p=8680

I read Scientists Have Recreated Ancient Battles to Solve Debate Over Ancient Bronze Swords and was annoyed.

Not because the study wasn’t worth doing for its own sake – I applaud archeologists with the good sense to use historical reenactors to learn more about how combat in bygone times must have worked. But it seems to have been done to refute a theory that shouldn’t have been entertained seriously for three seconds – namely that Bronze Age swords were mainly ceremonial items with little use in actual combat.

I am, as Charles Babbage might have said, not able rightly to apprehend the kind of confusion of ideas that could provoke such a theory.

They could have used me in this study. I have trained pretty extensively with pre-gunpowder contact weapons, especially swords and daggers and polearms. I’m not sure I can claim to be an expert swordsman, but people who are undoubtedly experts take me seriously when I fight.

I’m laying out these credentials because I want to make a point about the premise of the study. When you’ve trained with enough different varieties of swords – and my experience spans everything from very late dueling rapiers back to Viking-era cut-and-thrust weapons built in ways that go back to very similar prototypes in the early iron age – you learn to read a weapon.

By “read a weapon” I mean look at it, swing it, thrust it, find its balance point, and feel out the motions that are natural for that weight, length, and distribution of metal. If you know how to ask, the sword will tell you what its affordances are.

Here is a rule you can bank on: In a warrior culture, every affordance of a weapon will be used in its technique. Because why wouldn’t they? The more varied your toolkit, the more likely you are to be able to spring a deadly surprise on an enemy. Techniques that the weight and moment arm of the weapon make efficient will be used a lot. Weapon form and technique will coevolve under the selective pressure of combat experience in such a way that the match between technique and physical affordances stays close evebn as both change.

With this rule in mind, you can feel out the affordances of a weapon and deduce a lot of the technique that went with it. That’s why the medieval recreationists were valuable – it wasn’t just their experience at simulated medieval combat, it was that their feel for weapons could extend to earlier blade forms they hadn’t necessarily fought with extensively themselves yet.

I have in fact handled exact replicas of Bronze-age swords – and possibly one real one, though I suspect my buddy who owned it got conned by a forger. It is not difficult to tell what kind of moves they’re designed for – those front-heavy leaf-shaped blades clearly indicate chopping weapons implying a somewhat shorter engagement range than the longer weapons that would become typical in medieval Europe.

These were not cult objects, not ornamented gewgaws. They were spare and elegant killing weapons, probably carried for close-quarters use by fighters who used spears for combat at longer engagement ranges. Because spears suck in forest country or on rough ground, also indoors. In those circumstances the greater maneuverability and agility of a swordsman gives an advantage.

I speak from experience here, having fought in varied terrain with both swords and polearms. I’m not even a bit surprised that recovered Bronze-Age swords show patterns of hard wear resembling what reenactors did to replicas when they fought with them. The affordances of the weapons haven’t changes since they were forged; there is every reason to expect a priori that a skilled martial artist to feel out the same techniques our distant ancestors used.

“Ceremonial objects”, my ass. That shouldn’t even have been a question. What I think is really interesting about this is that it looks like a posteriori confirmation that re-enactors do in fact naturally rediscover ancient techniques; if they didn’t the wear patterns wouldn’t match.

Lassie errors

Post Syndicated from esr original http://esr.ibiblio.org/?p=8674

I didn’t invent this term, but boosting the signal gives me a good excuse for a rant against its referent.

Lassie was a fictional dog. In all her literary, film, and TV adaptations the most recurring plot device was some character getting in trouble (in the print original, two brothers lost in a snowstorm; in popular false memory “Little Timmy fell in a well”, though this never actually happened in the movies or TV series) and Lassie running home to bark at other humans to get them to follow her to the rescue.

In software, “Lassie error” is a diagnostic message that barks “error” while being comprehensively unhelpful about what is actually going on. The term seems to have first surfaced on Twitter in early 2020; there is evidence in the thread of at least two independent inventions, and I would be unsurprised to learn of others.

In the Unix world, a particularly notorious Lassie error is what the ancient line-oriented Unix editor “ed” does on a command error. It says “?” and waits for another command – which is especially confusing since ed doesn’t have a command prompt. Ken Thompson had an almost unique excuse for extreme terseness, as ed was written in 1973 to run on a computer orders of magnitude less capable than the embedded processor in your keyboard.

Herewith the burden of my rant: You are not Ken Thompson, 1973 is a long time gone, and all the cost gradients around error reporting have changed. If you ever hear this term used about one of your error messages, you have screwed up. You should immediately apologize to the person who used it and correct your mistake.

Part of your responsibility as a software engineer, if you take your craft seriously, is to minimize the costs that your own mistakes or failures to anticipate exceptional conditions inflict on others. Users have enough friction costs when software works perfectly; when it fails, you are piling insult on that injury if your Lassie error leaves them without a clue about how to recover.

Really this term is unfair to Lassie, who as a dog didn’t have much of a vocabulary with which to convey nuances. You, as a human, have no such excuse. Every error message you write should contain a description of what went wrong in plain language, and – when error recovery is possible – contain actionable advice about how to recover.

This remains true when you are dealing with user errors. How you deal with (say) a user mistake in configuration-file syntax is part of the user interface of your program just as surely as the normally visible controls are. It is no less important to get that communication right; in fact, it may be more important – because a user encountering an error is a user in trouble that he needs help to get out of. When Little Timmy falls down a well you constructed and put in his path, your responsibility to say something helpful doesn’t lessen just because Timmy made the immediate mistake.

A design pattern I’ve seen used successfully is for immediate error messages to include both a one-line summary of the error and a cookie (like “E2317”) which can be used to look up a longer description including known causes of the problem and remedies. In a hypothetical example, the pair might look like this:

Out of memory during stream parsing (E1723)

E1723: Program ran out of memory while building the deserialized internal representation of a stream dump. Try lowering the value of GOGC to cause more frequent garbage collections, increasing the size if your swap partition, or moving to hardware with more RAM.

The key point here is that the user is not left in the lurch. The messages are not a meaningless bark-bark, but the beginning of a diagnosis and repair sequence.

If the thought of improving user experience in general leaves you unmoved, consider that the pain you prevent with an informative error message is rather likely to be your own, as you use your software months or years down the road or are required to answer pesky questions about it.

As with good comments in your code, it is perhaps most motivating to think of informative error messages as a form of anticipatory mercy towards your future self.

Payload, singleton, and stride lengths

Post Syndicated from esr original http://esr.ibiblio.org/?p=8663

Once again I’m inventing terms for useful distinctions that programmers need to make and sometimes get confused about because they lack precise language.

The motivation today is some issues that came up while I was trying to refactor some data representations to reduce reposurgeon’s working set. I realized that there are no fewer than three different things we can mean by the “length” of a structure in a language like C, Go, or Rust – and no terms to distinguish these senses.

Before reading these definitions, you might to do a quick read through The Lost Art of Structure Packing.

The first definition is payload length. That is the sum of the lengths of all the data fields in the structure.

The second is stride length. This is the length of the structure with any interior padding and with the trailing padding or dead space required when you have an array of them. This padding is forced by the fact that on most hardware, an instance of a structure normally needs to have the alignment of its widest member for fastest access. If you’re working in C, sizeof gives you back a stride length in bytes.

I derived the term “stride length” for individual structures from a well-established traditional use of “stride” for array programming in PL/1 and FORTRAN that is decades old.

Stride length and payload length coincide if the structure has no interior or trailing padding. This can sometimes happen when you get an arrangement of fields exactly right, or your compiler might have a pragma to force tight packing even though fields may have to be accessed by slower multi-instruction sequences.

“Singleton length” is the term you’re least likely to need. It’s the length of a structure with interior padding but without trailing padding. The reason I’m dubbing it “singleton” length is that it might be relevant in situations where you’re declaring a single instance of a struct not in an array.

Consider the following declarations in C on a 64-bit machine:

struct {int64_t a; int32_t b} x;
char y

That structure has a payload length of 12 bytes. Instances of it in an array would normally have a stride length of 16 bytes, with the last two bytes being padding. But in this situation, with a single instance, your compiler might well place the storage for y in the byte immediately following x.b, where there would trailing padding in an array element.

This struct has a singleton length of 12, same as its payload length. But these are not necessarily identical, Consider this:

struct {int64_t a; char b[6]; int32_t c} x;

The way this is normally laid out in memory it will have two bytes of interior padding after b, then 4 bytes of trailing padding after c. Its payload length is 8 + 6 + 4 = 20; its stride length is 8 + 8 + 8 = 24; and its singleton length is 8 + 6 + 2 + 4 = 22.

To avoid confusion, you should develop a habit: any time someone speaks or writes about the “length” of a structure, stop and ask: is this payload length, stride length, or singleton length?

Most usually the answer will be stride length. But someday, most likely when you’re working close to the metal on some low-power embedded system, it might be payload or singleton length – and the difference might actually matter.

Even when it doesn’t matter, having a more exact mental model is good for reducing the frequency of times you have to stop and check yourself because a detail is vague. The map is not the territory, but with a better map you’ll get lost less often.

Insights need you to keep your nerve

Post Syndicated from esr original http://esr.ibiblio.org/?p=8657

This is a story I’ve occasionally told various friends when one of the subjects it touches comes up. I told it again last night, and it occurred to me that I ought to put in the blog. It’s about how, if you want to have productive insights, you need a certain kind of nerve or self-belief.

Many years ago – possibly as far back as the late 80s – I happened across a film of a roomful of Sufi dervishes performing a mystical/devotional exercise called “dhikr”. The film was very old, grainy B&W footage from the early 20th century. It showed a roomful of bearded, turbaned, be-robed men swaying, spinning, and chanting. Some were gazing at bright objects that might have been lamps, or polished metal or jewelry reflecting other lamps – it wasn’t easy to tell from the footage.

I can’t find the footage I saw, but the flavor was a bit like this. No unison movement in what I saw, though – individuals doing different things and ignoring each other, more inward-focused.

The text accompanying the film explained that the intention of “dhikr” is to shut out the imperfect sensory world so the dervish can focus on the pure and holy name of Allah. “Right,” I thought, already having had quite a bit of experience as an experimental mystic myself, “I get this. In Zen language, they’re shutting down the drunken monkeys. Autohypnosis inducing a serene mind, nothing surprising here.”

But there was something else. Something about the induction methods they were using. It all seemed oddly familiar, more than it ought to. I had seen behaviors like this before somewhere, from people who weren’t wearing pre-Kemalist Turkish garb. I watched the film…and it hit me. This was exactly like watching a roomful of people with serious autism!

The rocking. The droning. The fixated behavior, or in the Sufi case the behavior designed to induce fixation. Which immediately led to the next question: why? I think the least hypothesis in cases where you observe parallel behaviors is that they have parallel causation. We know what the Sufis tell us about what they’re doing; might it tell us what the autists are doing what they’re doing?

The Sufis are trying to shut out sense data. What if the autists are too? That would imply that the autists live in a state of what is, for them, perpetual sensory overload. Their dhikr-like behaviors are a coping mechanism, an attempt to turn down the gain on their sensors so they can have some peace inside their own skulls.

The first applications of nerve I want to talk about here are (a) the nerve to believe that autistic behaviors have an explanation more interesting than “uhhh…those people are randomly broken”, and (b) the nerve to believe that you can apply a heuristic like “parallel behavior, parallel causes” to humans when you picked it up from animal ethology.

Insights need creativity and mental flexibility, but they also need you to keep your nerve. I think there are some very common forms of failing to keep your nerve that people who would like to have good and novel ideas self-sabotage with. One is “If that were true, somebody would have noticed it years ago”. Another is “Only certified specialists in X are likely to have good novel ideas about X, and I’m not a specialist in X, so it’s a bad risk to try following through.”

You, dear reader, are almost certainly browsing this blog because I’m pretty good at not falling victim to those, and duly became famous by having a few good ideas that I didn’t drop on the floor. However, in this case, I failed to keep my nerve in another bog-standard way: I believed an expert who said my idea was silly.

That was decades ago. Nowadays, the idea that autists have a sensory-overload problem is not even controversial – in fact it’s well integrated into therapeutic recommendations. I don’t know when that changed, because I haven’t followed autism research closely enough. Might even be the case that somewhere in the research literature, someone other than me has noticed the similarity between semi-compulsive autistic behaviors and Sufi dhikr, or other similar autohypnotic practices associated with mystical schools.

But I got there before the experts did. And dropped the idea because my nerve failed.

Now, it can be argued that there were good reasons for me not to have pursued it. Getting a real hearing for a heterodox idea is difficult in fields where the experts all have their own theories they’re heavily invested in, and success is unlikely enough that perhaps it wasn’t an efficient use of my time to try. That’s a sad reason, but in principle a sound one.

But losing my nerve because an expert laughed at me, that was not sound. I think I wouldn’t make that mistake today; I’m tougher and more confident than I used to be, in part because I’ve had “crazy” ideas that I’ve lived to see become everyone’s conventional wisdom.

You can read this as a variation on a theme I developed in Eric and the Quantum Experts: A Cautionary Tale. But it bears repeating. If you want to be successfully creative, your insights need you to keep your nerve.

The implausibility of a war with China

Post Syndicated from esr original http://esr.ibiblio.org/?p=8643

In the wake of the PRC’s actions around the COVID-19 pandemic, there has been increasing speculation in some circles that the PRC might be preparing to wage war against the United States, or at least some sort of regional war (such as an invasion of Taiwan) in which treaty obligations would involve the U.S.

I’ve actually been considering this possibility, from my perspective as a wargamer and military-history buff, for over a decade – ever since China began seriously flexing its muscles in the South China Sea. And the risk of war has undoubtedly been rising recently.

The PRC has given U.S. and other trade partners ample reason to conclude that they need to decouple their economies from Chinese supply chains. Threats by China to use its control of most of rare-earth production for economic blackmail have been followed by much more serious threats to use its dominance of the manufacturing of basic pharmaceuticals as a weapon.

Post-COVID-19, it’s now strategically vital for other nations to develop supply chains for critical goods that are domestic, or at least better guarded against the political and epidemiological risk of relying on Chinese manufacturing. While necessary, this shift does mean the PRC has less to lose in the event of going to war.

Nevertheless, I continue to judge that the odds of China launching a war are very low. Nobody can entirely rule out enraged, irrational behavior by the PRC, but in the remainder of this post I will attempt to demonstrate why the war options available to the PRC hold out little or no prospect of a satisfying victory and entail severe terminal risks.

To wage a winning war, you need to formulate a set of war aims that are achievable with the tools and resources you have. Your strategy derives from your war aims, which have to be grounded in some notion of how you will manage the peace following a military victory to your advantage.

Historically, the overwhelmingly most common sort of aggressive war is a war of conquest. In these the war aim is simple – to conquer and annex some territory, and then integrate it into your state structure following victory.

In more sophisticated versions of this game you may be satisfied with the creation of a compliant client state from your conquest.

A step further away from raw conquest is war to maintain position as the dominant power (the hegemon) of a trade network that gathers wealth for your nation even without exerting formal control of the other polities in the network. Many wars that at first sight appear to be ideologically motivated can be understood this way, with political or religious ideology providing a rationale for hegemony that the entire trade network accepts – or can at least be made to echo.

Broadly speaking, land power tend to wage wars of conquest, while maritime powers wage wars of hegemony. There have been exceptions in both directions.

In all these cases, a set of war aims needs to hold out a better than even chance that the gains of war will outweigh the costs.

To understand how limited the PRC’s war options are, we can start with a grasp on how difficult and unsatisfying any war of conquest would be due to the geographic box China is in. The obstacles around it are formidable.

To the south, the Himalayan massif makes all of South Asia other than a narrow coastal plain on the Southeast Asian peninsula inaccessible to serious troops movements. There are no roads or rail links. The last time the Chinese tried pushing in that direction, in 1979, they were unable to sustain an offensive at any distance from their railheads and withdrew after less than a month. Their war aim – forcing the North Vietnamese to withdraw its troops from Cambodia – failed.

To the west, the vastness and comparatively undeveloped state of China’a western hinterland is a serious logistical problem before one even gets to the border. At the borders, the Tien Shan and Pamir ranges present a barrier almost as formidable as the Himalayas. External road and rail links are poor and would be easily interdicted.

To the north, movement would be easier. It might be just within logistical possibility for the PLA to march into Siberia. The problem with this idea is that once you’ve conquered Siberia, what you have is…Siberia. Most of it, except for a small area in the south coastal region of Primorsky Kraye, is so cold that cities aren’t viable without food imports from outside the region. Set this against the risks of invading a nuclear-armed Russia and you don’t have a winning proposition.

To the east is the South China Sea. The brute fact constraining the PRC’s ambitions in that direction is that mass movement of troops by sea is risky and difficult. I recently did the math on Chinese sealift craft and despite an expensive buildup since the 1980s they don’t have the capacity to move even a single division-sized formation over ocean. Ain’t nobody going to take Taiwan with one division, they’ve has too much time to prepare and fortify over the last 60 years.

The PRC leadership is evil and ruthless, but it’s also cautious and historically literate and can read maps. Accordingly, the People’s Liberation Army is designed not to take territory but to hold the territory the PRC already has. Its mission is not conquest but the suppression of regional warlordism inside China itself. The capability for it to wage serious expeditionary warfare doesn’t exist, and can’t be built in the near-term future.

It’s often said that the danger of aggressive war by China is a function of the huge excess of young men produced by covert sexual selection and the one-child policy. But to expend those young men usefully you need to get them to where they can fight and are motivated by some prospect of seizing the wives unavailable for them at home. The PRC can’t do that.

The military threat from China is, therefore, a function of what it can do with its navy, its airpower, and its missiles. And what it can do with those against the U.S. is upper-bounded by the fact that the U.S. has nuclear weapons and would be certain to respond to a PRC nuclear or EMP attack on the U.S. mainland by smashing Chinese cities into radioactive rubble.

Within the constraints of conventional warfare waged by navy and air force it is difficult to imagine an achievable set of PRC war aims that gains more than it costs.

This isn’t to say the PRC couldn’t do a lot of damage if it wants to. Anybody with a brain has to worry about U.S. carriers looking like big, fat, slow targets in the modern naval battlespace. There’s intelligence that the PRC is working hard on hypersonic ship-killer missiles, and I certainly would be in their shoes. It may already be unsafe for hostile carriers to operate inside the first island chain.

The problem is this: after you’ve surprised and sunk a couple of U.S. fleet carriers, what do you do for an encore? How do you convert that tactical victory into strategic gains? You’re not going to do it with your army, which can’t get anywhere more interesting after the sinkings than it could before.

Your problems are compounded by China’s extreme import dependence. You need a constant high volume of imports of coal, oil, and steel to keep your economy running. These have to be imported through sealanes that are extremely vulnerable to interdiction, notably at the Malacca Straits and in the Persian Gulf.

In a lot of ways your strategic situation is like a scaled-up version of Japan’s in 1941 – you could seize the initiative with a Pearl-Harbor-like initial shock, but you can’t wage a long war because without sealane control you’ll run out of key feedstocks and even food rather rapidly. And unlike the Japanese in 1941, you don’t have the kind of serious blue-water navy that you’d need for sealane control outside the First Island Chain – not with just two carriers you don’t.

There is one way an aggressive naval war could work out in your favor anyway. You can count on the U.S.’s media establishment to be pulling for the U.S. to lose any war it’s in, especially against a Communist or Socialist country. If your war goals are limited to ending U.S. naval power projection in the Western Pacific, playing for a rapid morale collapse orchestrated by agents of influence in the U.S. is not completely unrealistic.

It’s playing with fire, though. One problem is that before you launch your attack you don’t know that your sucker punch will actually work. Another is that, as the Japanese found out after Pearl Harbor, the American public may react to tragic losses with Jacksonian fury. If that happens, you’re seriously screwed. The war will end with your unconditional surrender, and not sooner.

You’re probably screwed anyway. Given even minimal spine in the U.S.’s civilian leadership, the U.S. Navy can strangle your economy in a matter of months by interdicting a handful of chokepoints well outside of the area where you can sustain naval operations at a wartime tempo. Those hypersonic missiles are all very well if you actually have them, but even if you could could reach out and touch the Malacca Straits with them they’re not going to do much against attack submarines.

Again I note that the PRC leadership can read maps. It is probably more aware and less self-deluding about the economic precariousness of its situation than American politicians would be if the positions were reversed, because Marxist doctrine insists that politics is an epiphenomenon of economics.

The PRC can start a war, but they don’t have the capability to win one. That’s why, barring a Hitler-scale episode of insanity in the PRC leadership, it’s not going to happen.

Choosing your weapons wisely

Post Syndicated from esr original http://esr.ibiblio.org/?p=8632

In the comments to my last post advising people not to panic-buy guns because of COVID-19, I got a request from a regular wondering how to choose a first firearm wisely, and about safe storage practices.

He said: “I’m thinking in the next year of getting a gun for home defense, and I’d like myself and my spouse to train with it. […] I have young kids, and want to make sure the gun is accessible enough to be reachable in the event of a break-in but hard enough to access that my daughter doesn’t get into it.”

Credentials for anybody new here: I have several decades of experience as a self-defense and firearms instructor. I’m grateful that I haven’t had to shoot a human being yet. I’m not a professional in this stuff, but people who are treat me as a peer. As you keep reading, I think you will recognize the common sense in my advice.

Content warning: if you are easily offended by cold-blooded consideration of violent outcomes or Damned Facts about statistical patterns in criminality, this post will offend you.

I’m going to address the second sentence first. For basic physical security, you may want to consider getting a pistol-sized gun safe with a biometric lock. However…and I cannot emphasize this enough…do not rely on this to protect your children. Children are curious and ingenious and if they consider your security a challenge to be defeated you could have a tragedy.

The only safety lock that reliably protects your child is the one you install in your child’s head by teaching him or her that a gun is a dangerous tool that should only be used with adult supervision. Explain the danger. Do not make your weapons taboo forbidden fruit or surround them with mystery; if your child is curious, take him or her to the range with you.

If your child becomes very interested, this is good. Shooting sports are an effective way to develop discipline and concentration. And very safe (safer than golf, for example) except in the extremely unlikely case that you’ve raised a sociopath or some other kind of minimal-brain-damage victim, in which case you have larger problems than I’m going to try to address here.

Now I’ll talk about intelligent choice of weapons. This depends on your threat model and where you live.

I’m going to go into different threat models more later in this post, but I’ll start with advice that is common to all of them. The single most universally useful firearm you can have – and the least dangerous in case of accident or misuse – is a reliable carry pistol which you do, in fact, carry daily.

Do not get hung up on caliber or type. Gunfolks love to argue about stopping power and bullet ballistics but it turns out that once you get out of the mouse-gun range (.22, .25 and .30) all pistol calibers have essentially indistinguishable statistics on two-shot stops.

Therefore, keep it simple. Rent several different pistols at a range. To use your time efficiently, exclude monster hand cannons like .44 Magnum; that is certainly not a good beginner’s choice. You should be looking at calibers from 9mm up to .45ACP (11mm). Shoot them all, and pay attention to which one fits your hand the best and feels most comfortable for you to shoot. That is almost certainly the one you should buy.

I myself prefer medium-caliber semiautomatics like a .40 or .45 because I don’t enjoy the snappy recoil of a 9mm. But other people can be best suited for lighter-caliber pistols or revolvers; there are a lot of relevant variables including the shape and size of your hands and what kind of upper-body strength you have.

For home defense, it’s probably a good idea to fit a laser sight on your pistol; I got an aftermarket one recently for my .45. Then you can train in point shooting using the laser – makes you faster responding because you don’t have to pause to get a sight picture.

Because this post is about choice of weapons, I’m not going to talk a lot about training methods except to say “do one”. Train, train, train. Get comfortable with firing your weapon, learn how to be accurate at normal pistol engagement ranges of 7-10 feet.

That’s feet, not yards. It’s pretty close. Accuracy at that range is easy. More important than crisply perfect technique is the ability to handle the psychological stress of a clutch situations so your accuracy doesn’t go to shit when you’re tired, rattled, and in low-light conditions. Read up on “stress inoculation” and try to get some.

Don’t be daunted by the thought that it takes years to master shooting. As with all skills, the more you put into it, the more you can get out. But any competent instructor can teach you how to handle firearms safely in 20 minutes, and you can develop the competence for basic self-defense shooting in a few hours.

You should lock that in with at least semi-regular practice, though. The newer you are, the more regular it should be; eventually (after years) you may get to the point where your muscle memory is solid enough to weather long periods without practice.

You’ll need a holster so you can carry. A gunbelt – which is just an extra-stiff leather belt that helps distribute the weight of you weapon – is a good investment. Alas, choosing good gunleather is an entire topic in itself. Expect that the first holster you buy will not be optimal and that you’ll probably need to experiment a few times before finding one that suits you for long-term use.

One area in which I think the gun culture can be unhelpful is in helping you judge how much ammunition to keep around. The problem is that a lot of us gunfolks end to treat the size of our ammo stockpiles as a sort of tongue-in-cheek competitive studliness display. The truth is that for almost everyone a 250-round reserve per weapon (exclusive of what you buy and shoot at the range) is just fine – generous, in fact.

Beyond that first pistol, what else you should buy starts to diverge based on where you live and what your threat model is. I’m going to start by assuming the most common and simplest one, which I’ll call the Standard Threat Model: you want to defend yourself and your family against low-level criminal violence, with a side order of hedging against a temporary (on the order of a few days) condition of civil disorder due to, e.g., natural disaster.

In that case a lot depends on whether you live in Switzerland or Swaziland. Most of the U.S. has violent-crime statistics like Switzerland – very low base rate of crime, law-abiding neighbors, high levels of legal gun ownership. In Switzerland, even temporary disaster conditions don’t induce looting, arson, and crime spikes. Therefore they do not raise your threat level much.

Unfortunately, parts of the U.S. – some major urban cores, and some drug-corridor rural areas near the Mexican border – are Swaziland. In Swaziland base violent-crime rates are high. Rates of legal gun ownership are low. Your neighbors are unhelpful, and a high-deviant cohort of them is actively dangerous.

If you live in Switzerland (easily 95% of the U.S. by land area), rational assessment of the Standard Threat Model does not require you to be heavily armed. I’d start with a carry pistol for each adult household member, and one shotgun for fixed-point defense. Whether you should also get a rifle depends on where you live. If you’re urban or suburban there’s not a lot of physical point to it because you won’t have long enough sightlines for distance shooting to matter much.

If you’re rural, on the other hand, you want a rifle. How serious a rifle depends on whether you have dangerous critters like bears or mountain lions in your area. Most people can get away with what gunfolks call a “varmint rifle” – a light-caliber rifle that shoots cheap ammo like .22LR. This is fine if your typical animal threat is something like a rabid skunk. It will take care of threatening humans too, in the extremely unlikely event you assess enough threat to have to shoot them at distance.

If, on the other hand, you have heavy threats like cougars or bears, you need a heavier rifle and a bigger bullet. Detailed discussion of these is out of scope for this post. Besides, I don’t know much about heavy rifles and wouldn’t want to give bad advice.

If you live in Swaziland, your threat profile is entirely different. Here’s how to tell if you do: (1) you live in a rural area with the Mexican border or concentrations of illegal immigrants within a two-hour drive. (2) you live in an urban area and within 20 minutes’ walk of you are places where groups of black or Hispanic males aged 15-35 carrying intoxicants routinely gather.

Yes, I can hear you lefties screaming already. All I have to say is: study the crime statistics. We can tell all kinds of stories about why those numbers look the way they do. Some of the stories we could tell are racist and irrational (but I repeat myself). The fact that shitty people tell toxic stories about the numbers doesn’t change the numbers, and it doesn’t change what the rational response to the numbers should be.

In American Swaziland, unlike African Swaziland, there’s no dangerous-animal threat at all, so you don’t need a heavy rifle. However, you have a banditry problem – not just individual muggers and home invaders but gangs of feral predators who routinely commit crimes ranging from mass shoplifting raids upwards to savage
monkey-dance beatings that cripple or kill their victims. Civil disorder in Swaziland is quite dangerous, not only because of direct threat from mobs of ferals but due to indirect threats like arson.

In Swaziland you also need to assume that any assailant will be high off his ass on something like PCP or bath salts – a disassociative anesthetic. Pistol rounds do not reliably stop such a person before they can get close enough to kill you unless you luck out with a heart or brain shot.

If you’re living in Swaziland, the best thing you can do for yourself and society is arm up to the level where you pose a credible threat not just to individual criminals but to a mob of drunk or drugged ferals with a low average IQ and poor impulse control. Because riots or natural disaster could require you to step up like a roof Korean.

That means we’re in scary-black-rifle territory. You want an AR-15 or something like one. Understand that functionally an AR-15 is not very different in how accurately can shoot and what it can stop from your granddad’s hunting rifle. However, what it does to a mob’s threat assessment is very different.

Granddad’s hunting rifle says to a mob “Stupid ofay probably hasn’t fired that thing in years.” Black rifle says “Uh oh, gun nut. Prepared. Would probably rather shoot than not.” Ironically, this means that if what you’re showing is granddad’s hunting rifle, you’re more likely to have to actually shoot it. I’d consider actually having to shoot a less than optimal outcome; if you don’t, I probably don’t want to know you.

And that pretty much wraps up the Standard Threat Model. Now I’ll briefly cover a couple of other possibilities you might want to arm against which group together because they push your weapons mix in a similar direction.

One is longer-term civil disorder, ranging upwards to what gunfolks and preppers call “SHTF” (Shit Hits The Fan) scenarios. Worrying about these changes your optimal weapons mix – basically, you have to assume mob-feral violence as a prompt threat even in Switzerland. You’ll want scary black rifles, at least one per military-age household member.

However…I urge you not to worry about the weapons themselves so much that you neglect other needs. One is ammunition. Anywhere near SHTF conditions ammunition is going to become scarce and valuable; you want at least a thousand rounds per weapon and 10K would not be excessive.

More importantly, however, you need to lay in serious amounts of food and medical supplies before going on any gun-buying sprees. You can’t eat bullets, and raiding your neighbors for food would get terminally risky pretty fast.

I myself do not prep for SHTF very seriously, for reasons which I could explain but which are beyond the scope of this post. However, there is a different reason for me to have a SHTF-like weapons mix: the Second Amendment. I take my Constitutional duty to be part of the nation in arms seriously, and I insist on having the weapons that would-be tyrants foreign and domestic fear and want to take way from me precisely because they want to take them.

Generalizing, a sufficient reason to keep weapons beyond what the Standard Threat Model requires is as a move in the political power game, with the goal of ensuring that they are never actually needed.

PSA: COVID-19 is a bad reason to get a firearm

Post Syndicated from esr original http://esr.ibiblio.org/?p=8627

I’m a long-time advocate of more ordinary citizens getting themselves firearms and learning to use them safely and competently. But this is a public-service announcement: if you’re thinking of running out to buy a gun because of COVID-19, please don’t.

There are disaster scenarios in which getting armed up in a hurry makes sense; the precondition for all of them is a collapse of civil order. That’s not going to happen with COVID-19 – the mortality rate is too low.

Be aware that the gun culture doesn’t like and doesn’t trust panic buyers; they tend to be annoying flake cases who are more of liability than an asset. We prefer a higher-quality intake than we can get in the middle of a plague panic. Slow down. Think. And if you’ve somehow formed the idea that you’re in a zombie movie or a Road Warrior sequel, chill. That’s not a useful reaction; it can lead to panic shootings and those are never good.

I don’t mean to discourage anyone from buying guns in the general case – more armed citizens are a good thing on multiple levels. After we’re through the worst of this would be a good time for it. But do it calmly, learn the Four Rules of Firearms Safety first, and train, train, train. Get good with your weapons, and confident enough not to shoot unless you have to, before the next episode of shit-hits-the-fan.

shellcheck: boosting the signal

Post Syndicated from esr original http://esr.ibiblio.org/?p=8622

I like code-validation tools, because I hate defects in my software and I know that there are lots of kinds of defects that are difficult for an unaided human brain to notice.

On my projects, I throw every code validater I can find at my code. Standbys are cppcheck for C code, pylint for Python, and go lint for Go code. I run these frequently – usually they’re either part of the “make check” I use to run regression tests, or part of the hook script run when I push changes to the public repository.

A few days ago I found another validator that I now really like: shellcheck Yes, it’s a lint/validator for shell scripts – and in retrospect shell, as spiky and irregular and suffused with multilevel quoting as it is, has needed something like this for a long time.

I haven’t done a lot of shell scripting in the last couple of decades. It’s not a good language for programming at larger orders of magnitude than 10 lines or so – too many tool dependencies, too difficult to track what’s going on. These problems are why Perl and later scripting language became important; if shell had scaled up better the space they occupy would have been shell code as far as they eye can see.

But sometimes you write a small script, and then it starts to grow, and you can end up in an awkward size range where it isn’t quite unmanageable enough to drive you to port it to (say) Python yet. I have some cases like this in the reposurgeon suite.

For this sort of thing a shell validater/linter can be a real boon, enabling you to have much more confidence that you’ll catch silly errors when you modify the script, and actually increasing the upper limit of source-line count at which shell remains a viable programming language.

So it is an excellent thing that shellcheck is a solid and carefully-thought-out piece of work. It does catch lot of nits and potential errors, hardening your script against cases you probably haven’t tested yet. For example. it’s especially good at flagging constructs that will break if a shell variable like $1 gets set to a value with embedded whitspace.

It has other features you want in a code validator, too. You can do line-by-line suppression of specific spellcheck warnings with magic comments, telling the tool “Yes, I really meant to do that” so it will shut up. This means when you get new warnings they are really obvious.

Also, it’s fast. Fast enough that you can run it on all your shellscripts up front of all your regular regression tests and probably barely ever notice the time cost.

It’s standard practice for me to have a “make check” that runs code validators and then the regression tests. I’m going back and adding shellcheck validation to those check productions on all my projects that ship shell scripts. I recommend this as a good habit to everybody.

The right to be rude

Post Syndicated from esr original http://esr.ibiblio.org/?p=8609

The historian Robert Conquest once wrote: “The behavior of any bureaucratic organization can best be understood by assuming that it is controlled by a secret cabal of its enemies.”

Today I learned that the Open Source Initiative has reached that point of bureaucratization. I was kicked off their lists for being too rhetorically forceful in opposing certain recent attempts to subvert OSD clauses 5 and 6. This despite the fact that I had vocal support from multiple list members who thanked me for being willing to speak out.

It shouldn’t be news to anyone that there is an effort afoot to change – I would say corrupt – the fundamental premises of the open-source culture. Instead of meritocracy and “show me the code”, we are now urged to behave so that no-one will ever feel uncomfortable.

The effect – the intended effect, I should say, is to diminish the prestige and autonomy of people who do the work – write the code – in favor of self-appointed tone-policers. In the process, the freedom to speak necessary truths even when the manner in which they are expressed is unpleasant is being gradually strangled.

And that is bad for us. Very bad. Both directly – it damages our self-correction process – and in its second-order effects. The habit of institutional tone policing, even when well-intentioned, too easily slides into the active censorship of disfavored views.

The cost of a culture in which avoiding offense trumps the liberty to speak is that crybullies control the discourse. To our great shame, people who should know better – such as the OSI list moderators and BOD – have internalized anticipatory surrender to crybullying. They no longer even wait for the soi-disant victims to complain before wielding the ban-hammer.

We are being social-hacked from being a culture in which freedom is the highest value to one in which it is trumped by the suppression of wrongthink and wrongspeak. Our enemies – people like Coraline Ada-Ehmke – do not even really bother to hide this objective.

Our culture is not fatally damaged yet, but the trend is not good. OSI has been suborned and is betraying its founding commitment to freedom. “Codes of Conduct” that purport to regulate even off-project speech have become all too common.

Wake up and speak out. Embrace the right to be rude – not because “rude” in itself is a good thing, but because the degenerative slide into suppression of disfavored opinions has to be stopped right where it starts, at the tone policing.

Reposurgeon defeats all monsters!

Post Syndicated from esr original http://esr.ibiblio.org/?p=8607

On January 12th 2020, reposurgeon performed a successful conversion of its biggest repository ever – the entire history of the GNU Compiler Collection, 280K commits with a history stretching back through 1987. Not only were some parts CVS, the earliest portions predated CVS and had been stored in RCS.

I waited this long to talk about it to give the dust time to settle on the conversion. But it’s been 5 weeks now and I’ve heard nary a peep from the GCC developers about any problems, so I think we can score this as reposurgeon’s biggest victory yet.

The Go port really proved itself. Those 280K commits can be handled on the 128GB Great Beast with a load time of about two hours. I have to tell the Go garbage collector to be really aggressive – set GOGC=30 – but that’s exactly what GOGC is for.

The Go language really proved itself too. The bet I made on it a year ago paid off handsomely – the increase in throughput from Python is pretty breathtaking, at least an order of magnitude and would have been far more if it weren’t constrained by the slowness of svnadmin dump. Some of that was improved optimization of the algorithms – we knocked out one O(n**2) after translation. More of it, I think, was the combined effect of machine-code speed and much smaller object sizes – that reduced working set a great deal, meaning cache miss penalties got less frequent.

Also we got a lot of speedup out of various parallelization tricks. This deserves mention because Go made it so easy. I wrote – and Julien Rivaud later improved – a function that would run a specified functional hook on the entire set of repository events, multithreading them from a worker pool optimally sized from your machine’s number of processors, or (with the “serialize” debug switch on) running them serially.

That is 35 lines of easily readable code in Go, and we got no fewer than 9 uses out of it in various parts of the code! I have never before used a language in which parallelism is so easy to manage – Go’s implementation of Communicating Sequential Processes is nothing short of genius and should be a model for how concurrency primitives are done in future languages.

Thanks where thanks are due: when word about the GCC translation deadline got out, some of my past reposurgeon contributors – notably Edward Cree, Daniel Brooks, and Julien Rivaud – showed up to help. These guys understood the stakes and put in months of steady, hard work along with me to make the Go port both correct and fast enough to be a practical tool for a 280K-commit translation. Particular thanks to Julien, without whose brilliance and painstaking attention to detail I might never have gotten the Subversion dump reader quite correct.

While I’m giving out applause I cannot omit my apprentice Ian Bruene, whose unobtrusively excellent work on Kommandant provided a replacement for the Cmd class I used in the original Python. The reposurgeon CLI wouldn’t work without it. I recommend it to anyone else who needs to build a CLI in Go.

These guys exemplify the best in what open-source collegiality can be. The success of the GCC lift is almost as much their victory as it is mine.

The reality of the lizard people

Post Syndicated from esr original http://esr.ibiblio.org/?p=8600

One of the loonier and more wonderful conspiracy theories floating around the Internet is it many of the world’s elite are shapeshifting lizardoid extraterrestrials. This explainer seems to sum it up pretty well.

When I first encountered this idea I was gobsmacked. How could anyone actually believe such a thing? And yet, apparently, many people do – millions of them, if polls on the topic are to be believed.

How humans form and maintain insane beliefs when there are plenty of objective reasons to know better is, I fear, a topic of continuing fascination to me. If only because when contagious and totalizing forms of insanity like Marxism or supernaturalist religions motivate the behavior of mobs they pose a significant threat to my survival.

The lizard-people theory isn’t in that class of danger, but I think cases like it and (for example) flat-Earthism are worth analysis precisely because they’re so implausible and still manage to attract adherents. Extremes like this can be revealing about mechanisms that are harder to see closer to the ordinary.

And indeed when I was mulling over lizard-people theory a few years ago I think I really did get a significant insight about the psychology of belief and what lizard-people conspiracy theory actually means.

Many years ago I read a penetrating analysis of UFOlogy arguing that the reports of people who believed themselves UFO contactees or witnesses were expressing the same sorts of psychological drama that in past centuries would gave been coded as religious experiences – eruptions of nigh-incomprehensible powers into the mundane world.

In this telling, to understand UFO reports and the weird little subculture that has grown up around their believers, you need to understand that for those people the imagery of the cheesiest-sort of B-movie science fiction has taken up the same receptors in their minds that religious mythology does for the conventionally devout. What they are really grappling with is mystical experience – altered states of consciousness, mindstorms that are very real phenomena in themselves but one which they lack any context to understand in a rational and generative way.

When I remembered this, I found a fruitful question to ask. That is, what is it about their experience of reality that believers in lizard people are coding in this cheesy SFnal imagery? What are they actually trying to talk about?

The answer came to me almost immediately once I managed to formulate the question. That was the moment at which I realized that, barring one unimportant detail, lizard-people theory is actually true.

The unimportant detail is the part about the lizard people being actual extraterrestrials. But let’s look at the rest of it. The believer says: Our elites behave as though they are heavily infiltrated by beings hostile to the interests of ordinary humans. They hide behind a mask of humanity but they have alien minds. They are predators and exploiters, cunning at hiding their nature – but sometimes the mask slips.

Nothing about this is in any way wrong, once you realize that “lizard” is code for “sociopath”. Sociopaths do, differentially, seek power over others, and are rather good at getting it. The few studies that have dared to look have found they are concentrated in political and business elites where drive and ruthlessness are rewarded.

“Lizard” is actually a rather clever code, if you happen to know your evolutionary neuroanatomy. Oversimplifying a little, humans have an exceptionally elaborate neocortex wrapped around a monkey brain wrapped around a lizard brain. The neocortex does what we are pleased to consider higher cognitive functions, the monkey brain does emotions and social behavior, and the lizard brain does territoriality/aggression/dominance.

What is wrong with sociopaths (and psychopaths – these categories are not clearly distinguished) is not entirely clear, but it is certain that their ability to experience emotions is damaged. The monkey brain is compromised; sociopaths live more in their lizard brain and display a lizard-like ability to go from flat affect to aggressive violence and back again in two blinks of an eye.

So, yeah, aliens. We have a live conspiracy theory because a lot of people can sense the alienness in their sociopathic/psychopathic bosses and politicians – and sometimes the mask slips. Not having any grasp of the language of abnormal psychology, they reach for the nearest metaphor handy. There’s a close relative of lizard-people theory in which which many of our elites are held to be members of an ancient Satanic conspiracy and have become demon-ridden; this is different language carrying the same freight.

It’s not really clear that knowing the language of abnormal psychology would help much. Yes, it might enable people to shed their incidentally nutty beliefs about actual extraterrestrials or demons, but it wouldn’t solve the actual problem that either kind of nutty language is pointing at.

Actually, “lizardoid alien” has useful connotative freight that “sociopath” lacks. Sociopaths are good at manipulating the rest of us precisely because they have an outside view of human emotion – it’s mostly just mechanism to them, knobs and levers they’re not involved in. The SFnal mythology encodes knowledge that the sterile psychological term does not.

Still, neither term solves the actual problem. Which is that we don’t keep the sociopaths, the hostile aliens, away from the levers of power. Indeed, human power hierarchies often reward sociopathic behavior with competitive success.

Maybe we can make a start on that exclusions by understanding the sociopaths to be hostile aliens who need to be identified, othered, and thwarted. So, since explaining this to some of my friends, I’ve taken to using the term “lizard people” when I mean sociopaths, the uncaring predatory monsters who reveal themselves by seeking domination over others.

It’s a start, anyway.

Chinese bioweapon II: Electric Boogaloo

Post Syndicated from esr original http://esr.ibiblio.org/?p=8587

Yikes. Despite the withdrawal of the Indian paper arguing that the Wuhan virus showed signs of engineering, the hypothesis that that it’s an escaped bioweapon looks stronger than ever.

Why do I say this? Because it looks like my previous inclination to believe the rough correctness of the official statistics – as conveyed by the Johns Hopkins tracker – was wrong. I now think the Chinese are in way deeper shit than they’re admitting.

My willingness to believe the official line didn’t stem from any credulity about what the Chinese government would do if it believed the truth wouldn’t serve. As Communists they are lying evil scum pretty much by definition, and denial would have been politically attractive for as long as they thought they could nip the pandemic in the bud. I thought their incentives had flipped and they would now be honest as a way of assisting their own countermeasures and seeking international help.

My first clue that I was wrong about that came from a friend who is plugged into the diaspora Chinese community. According to him, there is terrifying video being sent from Chinese clans to the overseas branches they planted in the West to prepare a soft landing in case they have to bail out of China. Video of streets littered with corpses. And of living victims exhibiting symptoms like St. Vitus’s Dance (aka Sydenham’a chorea), which means the virus is attacking central nervous systems.

My second clue was the Tencent leak. Read about it here; the takeaway is that there is now reason to believe that as of Feb 1st the actual coronavirus toll looked like this: confirmed cases 154023, suspected cases 79808, cured 269, deaths 24589.

Compare that with the Johns Hopkins tracker numbers for today, a week later: Confirmed cases 31207, cured 1733, deaths 638. Allowing for the Tencent leak being roughly one doubling period earlier, the official statistics have been lowballing the confirmed case number by a factor of about 8 and the deaths by a factor of about 80. And then inflating cures by a factor of about 12.

Even given what I’d heard about the video, I might have remained skeptical about the leak numbers if someone (don’t remember who or where) hadn’t pointed out that the ratio between reported cases and deaths has been suspiciously constant in the official Chinese statistics. In uncooked statistics one would expect more noise in that ratio, if only because of reporting problems.

So my present judgment, subject to change on further evidence, is that the Tencent-leak numbers are the PRC’s actual statistics. And that has a lot of grim implications.

One is that the Wuhan virus has at least a 15% fatality rate in confirmed cases – and most ways the PRC’s own statistics could be off due to reporting problems would drive it higher. Another is that containment in China has failed. Even in the cooked official statistics first derivative has not fallen; the doubling time is on the close order of five days now and may decrease.

We are already well past any even theoretical coping capability of China’s medical infrastructure. For that matter, it isn’t likely that there are enough trained medical personnel on the entire planet to get on top of a pandemic this size with a 5-day doubling time.

Which means this thing is probably not going to top out in China until it saturates the percentage of the population without natural immunity and kills at least 15% of them. The big, grim question is how many natural immunes there are. The history of past natural pandemics does not conduce to any optimism at all about that.

A very safe prediction is that a whole lot of elderly Chinese people are going to die because their immune systems are pre-compromised.

China’s population is about 1.4 billion. Conservatively, therefore, we can already expect this plague to kill more people in China than the Black Death did in Europe. At its present velocity we can expect that in about 12 doubling periods, or approximately 60 days.

Meanwhile, coronavirus spread outside China enters a critical time.

Based on what we think we know about the incubation period (about 14 days), if there’s going to be a pandemic breakout outside of China due to asymptomatic carriers, we should start to see a slope change in the overseas incidence curve during the next week. It’s been long enough for that now.

If that doesn’t happen, either the rest of the world dodged the pandemic bullet (optimistic) or the low end of the incubation period is longer than has been thought (pessimistic). On the basis of previous experience with SARS and MERS, I think the optimistic read is more likely to be correct.

Now back to the bioweapon hypothesis. Does recent data strengthen or weaken it? Consider:

* 645 Indian evacuees from Wuhan all tested negative.

* The only death outside China has been an ethnic-Chinese traveler from Wuhan.

The evidence that this virus likes to eat Han Chinese and almost ignores everybody else is mounting. That’s bioweapon-like selectivity.

One of my previous objections to the bioweapon hypothesis was that the Wuhan virus’s lethality wasn’t high enough. At 15% or higher I withdraw that objection.

And that St. Vitus’s Dance thing – coronaviruses don’t do that. But it’s exactly the kind of thing you’d engineer into a terror weapon intended not just to kill a chunk of your target population but break the morale of the rest.

Finally, my friend Phil Salkie tells me that on Google Maps the reported location of the Wuhan Institute of Virology has been jumping around like a Mexican flea. That’s guilty behavior, that is.

Build engines suck. Help GPSD select a new one.

Post Syndicated from esr original http://esr.ibiblio.org/?p=8581

One of the eternal mysteries of software is why build engines suck so badly.

Makefiles weren’t terrible as a first try, except for the bizarre decision to make the difference between tabs and spaces critically different so you can screw up your recipe invisibly.

GNU autotools is a massive, gnarly, hideous pile of cruft with far too many moving parts. Troubleshooting any autotools recipe of nontrivial capacity will make you want to ram your forehead repeatedly into a brick wall until the pain stops.

Scons, among the first of the new wave of build engines to arise when mature scripting languages make writing them easier, isn’t bad. Except for the part where the development team was unresponsive at the best of times and development has stagnated for years now.

Waf is also not bad, except for being somewhat cryptic to write and having documentation that would hardly be less comprehensible if it had been written in hieroglyphics.

cmake and meson, two currently popular engines, share one of the (many) fatal flaws of autotools. They don’t run recipes directly, they compile recipes to some simpler form to be run by a back-end builder (typically, but not always, such systems generate makefiles). Recipe errors thrown by the back end don’t necessarily have any direct relationship to an identifiable part of the recipe you give your front end, making troubleshooting unnecessarily painful and tedious.

Wikipedia has a huge list of build-automation systems. Most seem to be relatively new (last decade) and if any of them had distinguished itself enough to be a cler winning choice I’m pretty sure I’d already know it.

Scons is where I landed after getting fed up with GPSD’s autotools build. In retrospect it is still probably the best choice we could have made at the time (around 2006 IIRC), and I think I would be very happy with it if it had lived up to its early promise. It didn’t. Time to move on.

Waf is what NTPsec uses, and while it has served us well the abysmally bad state of the documentation and the relatively high complexity cost of starting a recipe from a blank sheet of paper make it a questionable choice going forward. It doesn’t help that the project owner, while brilliant, does not communicate with lesser mortals very well.

Is there anything in this space that doesn’t have awful flaws?

Here’s what we want:

– works on any reasonable basically-POSIX system for bulding C programs

– there is no requirement for native Windows builds, and perhaps no requirement for Windows at all.

– has a relatively compact expression of build rules, which more or less means declarative notation rather than writing code to do builds

– has a track record of being maintained, and enough usage by other projects that there is a reasonable expectation that the build system will continue to be a reasonable choice for at least 5 years, and ideally longer

– doesn’t impose difficult requirements beyond what gpsd does already (e.g., needing C++11 for a build tool would be bad)

– has a notion of feature tests, rather than ifdefing per operating system, and promotes a culture of doing it that way

– supports setting build prefix, and enabling/disabling options, etc.

– supports dealing with -L/-R flags as expected on a variety of systems

– supports running tests

– supports running the just-built code before installing so that it uses the built libs and does not pick up the already-in-system libs (necessary for tests)

– supports cross compiling

– supports building placing the objects in a specified location outside the source tree

– supports creating distfiles (suffisient if it can shell out to tar commands).

– supports testing the created distfile by unpacking it, doing an out-of-source build and then running tests

– is not a two-phase generator system like autotools, cmake, and meson.

What in Wikipedia’s huge list should we be looking at?

Three reasons to believe the Wuhan virus is a bioweapon

Post Syndicated from esr original http://esr.ibiblio.org/?p=8566

I know it sounds like tinfoil-hat territory, but there are now three pieces of evidence pointing at the conclusion that the Wuhan coronavirus is an accidentally released bioweapon.

First point: The propagation statistics of this virus have looked deeply weird from the get-go. In particular, the differential in spread and mortality rates between China and the rest of the world.

This map and accompanying statistics story the story. Inside China, the disease is propagating like crazy and has 2.3% mortality among those showing overt symptoms (which sounds low, but it’s about the same as the devastating 1918 flu virus). Outside of China, the disease has spread very very slowly and there have been zero deaths. If it had comparable lethality among overseas populations to what it has exhibited in China we should have seen approximately 4 overseas deaths by now.

In general, infectious diseases only have this kind of behavioral split when for whatever reason you’re looking at two populations with greatly differing vulnerability to the pathogen. The classic example is Hansen’s disease – leprosy – which is nearly asymptomatic in most of the population but has horrifying symptoms if you are in a minority with exactly the wrong alleles at six identified genetic loci.

In the case of the Wuhan virus, infectivity and lethality may be related to a particular protein called ACE II which, when present in lung cells, acts as a receptor for the virus. This receptor is, apparently, common in East Asians but not in other populations.

If you were designing a targeted bioweapon, this is exactly the kind of discriminator you’d look for – a way to make it highly lethal to a specific target population and relatively harmless to others.

Second point: The Wuhan virus shows signs of having been engineered. This paper says there are four sequences in the Wuhan virus’s spike protein that are suspiciously similar to those of the AIDS virus. The abstract observes the similarities are “unlikely to be fortuitous in nature”.

Third point: Laboratory facilities that are graded for the most dangerous category of research into infectious diseases are called P4 or BSL4 labs. If you are doing research into biological warfare, a P4 lab is where you are doing it, because it’s too dangerous to pursue anywhere else.

Worldwide, there are approximately 55 such laboratories. The U.S. has 15 of them. The People’s Republic of China has one. It’s the Wuhan Institute of Virology and yes, it’s located at geographical ground zero of the outbreak.

Awfully damn coincidental, isn’t that?

Let’s review. Virus shows signs of engineering. There’s a P4 lab dealing in virus research at ground zero of the epidemic. And the virus exhibits weirdly sharp selectivity about who it will infect and kill.

There are problems with the bioweapon theory. One is the obvious question of why the PRC would design a bioweapon genetically targeted at its own people. Another is that the Wuhan virus is missing some traits you’d want in a production bioweapon, like much higher lethality and a reliable vaccine. If the PRC had one they would surely have deployed it by now.

But there is a plausible theory that fits all these facts. It’s this: the Wuhan virus was not a deliberate release of a production bioweapon, it was an accidental release of research in progress – the kind of nasty “Ooops!” that P4 containment is designed to prevent. Someone screwed up big time.

And why targeted on East Asians? Possibly no reason at all other than they were experimenting with genetic targeting and it was intended as a step towards a virus that could selectively target gweilos. Or Japanese. Or Indians.

But I can think of a blood-chillingly good reason for the Communists to keep a shot like this in their locker. And that reason is the island of Taiwan. Release the virus there, wait for it to do its thing, then when the Taiwanese are screaming for help offer humanitarian aid. Delivered by the Peoples’ Liberation Army.

No, I do not think this was intended for Hong Kong. The PRC leadership are not gamblers; I don’t think they’d be down with releasing a bioweapon in a city that has close overland links with the Chinese industrial heartland. But Taiwan is an island; if the plague got too hot for available countermeasures you could just interdict the island until it cooled down.

A final note: if the Chinese people become convinced that the PRC government brewed up this virus, it’s done. Kaput. Serious plagues are historically considered a sign that a dynasty has lost the Mandate of Heaven anyway, and the self-infliction of the wound would make the legitimacy collapse harder. Every couple of centuries there are popular revolts severe enough to topple dynasties; we could see one over this.

Head-voice vs. quiet-mind

Post Syndicated from esr original http://esr.ibiblio.org/?p=8558

I’m utterly boggled. Yesterday, out of nowhere, I learned of a fundamental divide in how peoples’ mental lives work about which I had had no previous idea at all.

From this: Today I Learned That Not Everyone Has An Internal Monologue And It Has Ruined My Day.

My reaction to that title can be rendered in language as – “Wait. People actually have internal monologues? Those aren’t just a cheesy artistic convention used to concretize the welter of pre-verbal feelings and images and maps bubbling in peoples’ brains?”

Apparently not. I’m what I have now learned to call a quiet-mind. I don’t have an internal narrator constantly expressing my thinking in language; in shorthand. I’m not a head-voice person. So much not so that when I follow the usual convention of rendering quotes from my thinking as though they were spoken to myself, I always feel somewhat as though I’m lying, fabulating to my readers. It’s not like that at all! I justify doing that only because the full multiordinality of my actual thought-forms won’t fit through my fingers.

But, apparently, for others it often is like that. Yesterday I learned that the world is full of head-voice people who report that they don’t know what they’re thinking until the narratizer says it. Judging by the reaction to the article it seems us quiet-minds are a minority, one in five or fewer. And that completely messes with my head.

What’s the point? Why do you head-voice people need a narrator to tell you what your own mind is doing? I fully realize this question could be be reflected with “Why don’t you need one, Eric?” but it is quite disturbing in either direction.

So now I’m going to report some interesting detail. There are exactly two circumstances under which I have head-voice. One is when I’m writing or consciously framing spoken communication. Then, my compositional output does indeed manifest as narratizing head-voice. The other circumstances is the kind of hypnogogic experience I reported in Sometimes I hear voices.

Outside of those two circumstances, no head-voice. Instead, my thought forms are a jumble of words, images, and things like diagrams (a commenter on Instapundit spoke of “concept maps” and yeah, a lot of it is like that). To speak or write I have to down-sample this flood of pre-verbal stuff into language, a process I am not normally aware of except as an occasional vague and uneasy sense of how much I have thrown away.

(A friend reports Richard Feynman observing that ‘You don’t describe the shape of a camshaft to yourself.” No; you visualize a camshaft, then work with that visualization in your head. Well, if you can – some people can’t. I therefore dub the pre-verbal level “camshaft thinking.”)

To be fully aware of that pre-verbal, camshaft-thinking level I have to go into a meditative or hypnogogic state. Then I can observe that underneath my normal mental life is a vast roar of constant free associations, apparently random memory retrievals, and weird spurts of logic connecting things, only some of which passes filters to present to my conscious attention.

I don’t think much or any of this roar is language. What it probably is, is the shock-front described in the predictive-processing model of how the brain works – where the constant inrush of sense-data meets the brain’s attempt to fit it to prior predictive models.

So for me there are actually three levels: (1) the roaring flood of free association, which I normally don’t observe; (2) the filtered pre-verbal stream of consciousness, mostly camshaft thinking, that is my normal experience of self, and (3) narratized head-voice when I’m writing or thinking about what to say to other people.

I certainly do not head-voice when I program. No, that’s all camshaft thinking – concept maps of data structures, chains of logic. processing that is like mathematical reasoning though not identical to it. After the fact I can sometimes describe parts of this process in language, but it doesn’t happen in language.

Learning that other people mostly hang out at (3), with a constant internal monologue…this is to me unutterably bizarre. A day later I’m still having trouble actually believing it. But I’ve been talking with wife and friends, and the evidence is overwhelming that it’s true.

Language…it’s so small. And linear. Of course camshaft thinking is intrinsically limited by the capabilities of the brain and senses, but less so. So why do most people further limit themselves by being in head-voice thinking most of the time? What’s the advantage to this? Why are quiet-minds a minority?

I think the answers to these questions might be really important.