Tag Archives: software

Last phase of the desktop wars?

Post Syndicated from Armed and Dangerous original http://esr.ibiblio.org/?p=8764

The two most intriguing developments in the recent evolution of the Microsoft Windows operating system are Windows System for Linux (WSL) and the porting of their Microsoft Edge browser to Ubuntu.

For those of you not keeping up, WSL allows unmodified Linux binaries to run under Windows 10. No emulation, no shim layer, they just load and go.

Microsoft developers are now landing features in the Linux kernel to improve WSL. And that points in a fascinating technical direction. To understand why, we need to notice how Microsoft’s revenue stream has changed since the launch of its cloud service in 2010.

Ten years later, Azure makes Microsoft most of its money. The Windows monopoly has become a sideshow, with sales of conventional desktop PCs (the only market it dominates) declining. Accordingly, the return on investment of spending on Windows development is falling. As PC volume sales continue to fall off , it’s inevitably going to stop being a profit center and turn into a drag on the business.

Looked at from the point of view of cold-blooded profit maximization, this means continuing Windows development is a thing Microsoft would prefer not to be doing. Instead, they’d do better putting more capital investment into Azure – which is widely rumored to be running more Linux instances than Windows these days.

Our third ingredient is Proton. Proton is the emulation layer that allows Windows games distributed on Steam to run over Linux. It’s not perfect yet, but it’s getting close. I myself use it to play World of Warships on the Great Beast.

The thing about games is that they are the most demanding possible stress test for a Windows emulation layer, much more so than business software. We may already be at the point where Proton-like technology is entirely good enough to run Windows business software over Linux. If not, we will be soon.

So, you’re a Microsoft corporate strategist. What’s the profit-maximizing path forward given all these factors?

It’s this: Microsoft Windows <em>becomes</em> a Proton-like emulation layer over a Linux kernel, with the layer getting thinner over time as more of the support lands in the mainline kernel sources. The economic motive is that Microsoft sheds an ever-larger fraction of its development costs as less and less has to be done in-house.

If you think this is fantasy, think again. The best evidence that it’s already the plan is that Microsoft has already ported Edge to run under Linux. There is only one way that makes any sense, and that is as a trial run for freeing the rest of the Windows utility suite from depending on any emulation layer.

So, the end state this all points at is: New Windows is mostly a Linux kernel, there’s an old-Windows emulation over it, but Edge and the rest of the Windows user-land utilities <em>don’t use the emulation.</em> The emulation layer is there for games and other legacy third-party software.

Economic pressure will be on Microsoft to deprecate the emulation layer. Partly because it’s entirely a cost center. Partly because they want to reduce the complexity cost of running Azure. Every increment of Windows/Linux convergence helps with that – reduces administration and the expected volume of support traffic.

Eventually, Microsoft announces upcoming end-of-life on the Windows emulation. The OS itself , and its userland tools, has for some time already been Linux underneath a carefully preserved old-Windows UI. Third-party software providers stop shipping Windows binaries in favor of ELF binaries with a pure Linux API…

…and Linux finally wins the desktop wars, not by displacing Windows but by co-opting it. Perhaps this is always how it had to be.

Documentation as knowledge capture

Post Syndicated from esr original http://esr.ibiblio.org/?p=8741

Maybe you’re one of the tiny minority of programmers that, like me, already enjoys writing documentation and works hard at doing it right. If so,the rest of this essay is not for you and you can skip it.

Otherwise, you might want to re-read (or at least re-skim) Ground-Truth Documents before continuing. Because ground-truth documents are a special case of a more general reason why you might want to try to change your mindset about documentation.

In that earlier essay I used the term “knowledge capture” in passing. This is a term of art from AI; it refers to the process of extracting domain knowledge from the heads of human experts into a form that can be expressed as an algorithm executable by the literalistic logic of a computer.

What I invite you to think about now is how writing documentation for software you are working on can save you pain and effort by (a) capturing knowledge you have but don’t know you have, and (b) eliciting knowledge that you have not yet developed.

Humans, including me and you, are sloppy and analogical thinkers who tend to solve problems by pattern-matching against noisy data first and checking our intuitions with logic after the fact (if we actually get that far). There’s no point in protesting that it shouldn’t be that way, that we should use rigorous logic all the way down, because our brains simply aren’t wired for that. Evolved cognition is a kludge – more properly, multiple stacks of kludges – developed under selection to be just barely adequate at coping.

This kludginess is revealed by, for example, optical illusions. And by the famous 7±2 result about the very limited sized of the human working set. And the various well-documented ways that human beings are extremely bad at statistical reasoning. And in many other ways…

When you do work that is as demanding of rigor as software engineering, one of your central challenges is hacking around the limitations of your own brain. Sometimes this develops in very obvious ways; the increasing systematization of testing during development during the last couple of decades, for example.

Other brain hacks are more subtle. Which is why I am here to suggest that you try to stop thinking of documentation as a chore you do for others, and instead think of it as a way to explore your problem space. and the space in your head around your intuitions about the problem, so you can shine light into the murkier corners of both. Writing documentation can function as valuable knowledge capture about your problem domain even when you are the only expert about what you are trying to do.

This is why my projects often have a file called “designer’s notes” or “hacking guide”. Early in a project these may just be random jottings that are an aid to my own memory about why things are the way they are. They tend to develop into a technical briefing about the code internals for future contributors. This is a good habit to form if you want to have future contributors!

But even though the developed version of a “designer’s notes” looks other-directed, it’s really a thing I do to reduce my own friction costs. And not just in the communication-to-my-future-self way either. Yes, it’s tremendously valuable to have a document that, months or years after I wrote it, reminds me of my assumptions when I have half-forgotten them. And yes, a “designer’s notes” file is good practice for that reason alone. But its utility does not even start there, let alone end there.

Earlier, I wrote of (a) capturing knowledge you have but don’t know you have, and (b) eliciting knowledge that you have not yet developed. The process of writing your designer’s notes can be powerful and catalytic that way even if they’re never communicated. The thing you have to do in your brain to narratize your thoughts so they can be written down is itself an exploratory tool.

As with “designer’s notes” so with every other form of documentation from the one-line code comment to a user-oriented HOWTO. When you achieve right mindset about these they are no longer burdens; instead they become an integral part of your creative process, enabling you to design better and write better code with less total effort.

I understand that to a lot of programmers who now experience writing prose as difficult work this might seem like impossible advice. But I think there is a way from where you are to right mindset. That way is to let go of the desire for perfection in your prose, at least early on. Sentence fragments are OK. Misspellings are OK. Anything you write that explores the space is OK, no matter how barbarous it would look to your third-grade grammar teacher or the language pedants out there (including me).

It is more important to do the discovery process implied by writing down your ideas than it is for the result to look polished. If you hold on to that thought, get in the habit of this kind of knowledge capture, and start benefiting from it, then you might find that over time your standards rise and it gets easier to put more effort into polishing.

If that happens, sure; let it happen – but it’s not strictly necessary. The only thing that is necessary is that you occasionally police what you’ve recorded so it doesn’t drift into reporting something the software no longer does. That sort of thing is a land-mine for anyone else who might read your notes and very bad form.

Other than that, though, the way to get to where you do the documentation-as-knowledge-capture thing well is by starting small; allow a low bar for polish and completeness and grow the capability organically. You will know you have won when it starts being fun.

A user story about user stories

Post Syndicated from esr original http://esr.ibiblio.org/?p=8720

The way I learned to use the term “user story”, back in the late 1990s at the beginnings of what is now called “agile programming”, was to describe a kind of roleplaying exercise in which you imagine a person and the person’s use case as a way of getting an outside perspective on the design, the documentation, and especially the UI of something you’re writing.

For example:

Meet Joe. He works for Randomcorp, who has a nasty huge old Subversion repository they want him to convert to Git. Joe is a recent grad who got thrown at the problem because he’s new on the job and his manager figures this is a good performance test in a place where the damage will be easily contained if he screws up. Joe himself doesn’t know this, but his teammates have figured it out.

Joe is smart and ambitious but has little experience with large projects yet. He knows there’s an open-source culture out there, but isn’t part of it – he’s thought about running Linux at home because the more senior geeks around him all seem to do that, but hasn’t found a good specific reason to jump yet. In truth most of what he does with his home machine is play games. He likes “Elite: Dangerous” and the Bioshock series.

Joe knows Git pretty well, mainly through the Tortoise GUI under Windows; he learned it in school. He has only used Subversion just enough to know basic commands. He found reposurgeon by doing web searches. Joe is fairly sure reposurgeon can do the job he needs and has told his boss this, but he has no idea where to start.

What does Joe’s discovery process looks like? Read the first two chapters of “Repository Editing with Reposurgeon” using Joe’s eyes. Is he going to hit this wall of text and bounce? If so, what could be done to make it more accessible? Is there some way to write a FAQ that would help him? If so, can we start listing the questions in the FAQ?

Joe has used gdb a little as part of a class assignment but has not otherwise seen programs with a CLI resembling reposurgeon’s. When he runs it, what is he likely to try to do first to get oriented? Is that going to help him feel like he knows what’s going on, or confuse him?

“Repository Editing…” says he ought to use repotool to set up a Makefile and stub scripts for the standard conversion workflow. What will Joe’s eyes tell him when he looks at the generated Makefile? What parts are likeliest to confuse him? What could be done to fix that?

Joe, my fictional character, is about as little like me as as is plausible at a programming shop in 2020, and that’s the point. If I ask abstractly “What can I do to improve reposurgeon’s UI?”, it is likely I will just end up spinning my wheels; if, instead, I ask “What does Joe see when he looks at this?” I am more likely to get a useful answer.

It works even better if, even having learned what you can from your imaginary Joe, you make up other characters that are different from you and as different from each other as possible. For example, meet Jane the system administrator, who got stuck with the conversion job because her boss thinks of version-control systems as an administrative detail and doesn’t want to spend programmer time on it. What do her eyes see?

In fact, the technique is so powerful that I got an idea while writing this example. Maybe in reposurgeon’s interactive mode it should issue a first like that says “Interactive help is available; type ‘help’ for a topic menu.”

However. If you search the web for “design by user story”, what you are likely to find doesn’t resemble my previous description at all. Mostly, now twenty years after the beginnings of “agile programming”, you’ll see formulaic stuff equating “user story” with a one-sentence soundbite of the form “As an X, I want to do Y”. This will be surrounded by a lot of talk about processes and scrum masters and scribbling things on index cards.

There is so much gone wrong with this it is hard to even know where to begin. Let’s start with the fact that one of the original agile slogans was “Individuals and Interactions Over Processes and Tools”. That slogan could be read in a number of different ways, but under none of them at all does it make sense to abandon a method for extended insight into the reactions of your likely users for a one-sentence parody of the method that is surrounded and hemmed in by bureaucratic process-gabble.

This is embedded in a larger story about how “agile” went wrong. The composers of the Agile Manifesto intended it to be a liberating force, a more humane and effective way to organize software development work that would connect developers to their users to the benefit of both. A few of the ideas that came out of it were positive and important – besides design by user story, test-centric development and refactoring leap to mind,

Sad to say, though, the way “user stories” became trivialized in most versions of agile is all too representative of what it has often become under the influence of two corrupting forces. One is fad-chasers looking to make a buck on it, selling it like snake oil to managers forever perplexed by low productivity, high defect rates, and inability to make deadlines. Another is the managers’ own willingness to sacrifice productivity gains for the illusion of process control.

It may be too late to save “agile” in general from becoming a deadening parody of what it was originally intended to be, but it’s not too late to save design by user story. To do this, we need to bear down on some points that its inventors and popularizers were never publicly clear about, possibly because they themselves didn’t entirely understand what they had found.

Point one is how and why it works. Design by user story is a trick you play on your social-monkey brain that uses its fondness for narrative and characters to get you to step out of your own shoes.

Yes, sure, there’s a philosophical argument that stepping out of your shoes in this sense is impossible; Joe, being your fiction, is limited by what you can imagine. Nevertheless, this brain hack actually works. Eppure, si muove; you can generate insights with it that you wouldn’t have had otherwise.

Point two is that design by user story works regardless of the rest of your methodology. You don’t have to buy any of the assumptions or jargon or processes that usually fly in formation with it to get use out of it.

Point three is that design by user story is not a technique for generating code, it’ s a technique for changing your mind. If you approach it in an overly narrow and instrumental way, you won’t imagine apparently irrelevant details like what kinds of video games Joe likes. But you should do that sort of thing; the brain hack works in exact proportion to how much imaginative life you give your characters.

(Which in particular, is why “As an X, I want to do Y” is such a sadly reductive parody. This formula is designed to stereotype the process, but stereotyping is the enemy of novelty, and novelty is exactly what you want to generate.)

A few of my readers might have the right kind of experience for this to sound familiar. The mental process is similar to what in theater and cinema is called “method acting.” The goal is also similar – to generate situational responses that are outside your normal habits.

Once again: you have to get past tools and practices to discover that the important part of software design – the most difficult and worthwhile part – is mindset. In this case, and temporarily, someone else’s.

Looking for C-to-anything transpilers

Post Syndicated from esr original http://esr.ibiblio.org/?p=8705

I’m looking for languages that have three properties:

(1) Must have weak memory safety. The language is permitted to crash on an out -of-bounds array reference or null pointer, but may not corrupt or overwrite memory as a result.

(2) Must have a transpiler from C that produces human-readable, maintainable code that preserves (non-perverse) comments. The transpiler is allowed to not do a 100% job, but it must be the case that (a) the parts it does translate are correct, and (b) the amount of hand-fixup required to get to complete translation is small.

(3) Must not be Go, Rust, Ada, or Nim. I already know about these languages and their transpilers.

How to work from home with Raspberry Pi | The Magpi 93

Post Syndicated from Ashley Whittaker original https://www.raspberrypi.org/blog/how-to-work-from-home-with-raspberry-pi-the-magpi-93/

If you find yourself working or learning, or simply socialising from home, Raspberry Pi can help with everything from collaborative productivity to video conferencing. Read more in issue #92 of The MagPi, out now.

01 Install the camera

If you’re using a USB webcam, you can simply insert it into a USB port on Raspberry Pi. If you’re using a Raspberry Pi Camera Module, you’ll need to unpack it, then find the ‘CAMERA’ port on the top of Raspberry Pi – it’s just between the second micro-HDMI port and the 3.5mm AV port. Pinch the shorter sides of the port’s tab with your nails and pull it gently upwards. With Raspberry Pi positioned so the HDMI ports are at the bottom, insert one end of the camera’s ribbon cable into the port so the shiny metal contacts are facing the HDMI port. Hold the cable in place, and gently push the tab back home again.

If the Camera Module doesn’t have the ribbon cable connected, repeat the process for the connector on its underside, making sure the contacts are facing downwards towards the module. Finally, remove the blue plastic film from the camera lens.

02 Enable Camera Module access

Before you can use your Raspberry Pi Camera Module, you need to enable it in Raspbian. If you’re using a USB webcam, you can skip this step. Otherwise, click on the raspberry menu icon in Raspbian, choose Preferences, then click on Raspberry Pi Configuration.

When the tool loads, click on the Interfaces tab, then click on the ‘Enabled’ radio button next to Camera. Click OK, and let Raspberry Pi reboot to load your new settings. If you forget this step, Raspberry Pi won’t be able to communicate with the Camera Module.

03 Set up your microphone

If you’re using a USB webcam, it may come with a microphone built-in; otherwise, you’ll need to connect a USB headset, a USB microphone and separate speakers, or a USB sound card with analogue microphone and speakers to Raspberry Pi. Plug the webcam into one of Raspberry Pi’s USB 2.0 ports, furthest away from the Ethernet connector and marked with black plastic inners.

Right-click on the speaker icon at the top-right of the Raspbian desktop and choose Audio Inputs. Find your microphone or headset in the list, then click it to set it as the default input. If you’re using your TV or monitor’s speakers, you’re done; if you’re using a headset or separate speakers, right-click on the speaker icon and choose your device from the Audio Outputs menu as well.

04 Set access permissions

Click on the Internet icon next to the raspberry menu to load the Chromium web browser. Click in the address box and type hangouts.google.com. When the page loads, click ‘Sign In’ and enter your Google account details; if you don’t already have a Google account, you can sign up for one free of charge.

When you’ve signed in, click Video Call. You’ll be prompted to allow Google Hangouts to access both your microphone and your camera. Click Allow on the prompt that appears. If you Deny access, nobody in the video chat will be able to see or hear you!

05 Invite friends or join a chat

You can invite friends to your video chat by writing their email address in the Invite People box, or copying the link and sending it via another messaging service. They don’t need their own Raspberry Pi to participate – you can use Google Hangouts from a laptop, desktop, smartphone, or tablet. If someone has sent you a link to their video chat, open the message on Raspberry Pi and simply click the link to join automatically.

You can click the microphone or video icons at the bottom of the window to temporarily disable the microphone or camera; click the red handset icon to leave the call. You can click the three dots at the top-right to access more features, including switching the chat to full-screen view and sharing your screen – which will allow guests to see what you’re doing on Raspberry Pi, including any applications or documents you have open.

06 Adjust microphone volume

If your microphone is too quiet, you’ll need to adjust the volume. Click the Terminal icon at the upper-left of the screen, then type alsamixer followed by the ENTER key. This loads an audio mixing tool; when it opens, press F4 to switch to the Capture tab and use the up-arrow and down-arrow keys on the keyboard to increase or decrease the volume. Try small adjustments at first; setting the capture volume too high can cause the audio to ‘clip’, making you harder to hear. When finished, press CTRL+C to exit AlsaMixer, then click the X at the top-right of the Terminal to close it.

Adjust your audio volume settings with the AlsaMixer tool

Work online with your team

Just because you’re not shoulder-to-shoulder with colleagues doesn’t mean you can’t collaborate, thanks to these online tools.

Google Docs

Google Docs is a suite of online productivity tools linked to the Google Drive cloud storage platform, all accessible directly from your browser. Open the browser and go to drive.google.com, then sign in with your Google account – or sign up for a new account if you don’t already have one – for 15GB of free storage plus access to the word processor Google Docs, spreadsheet Google Sheets, presentation tool Google Slides, and more. Connect with colleagues and friends to share files or entire folders, and collaborate within documents with simultaneous multi-user editing, comments, and change suggestions.


Designed for business, Slack is a text-based instant messaging tool with support for file transfer, rich text, images, video, and more. Slack allows for easy collaboration in Teams, which are then split into multiple channels or rooms – some for casual conversation, others for more focused discussion. If your colleagues or friends already have a Slack team set up, ask them to send you an invite; if not, you can head to app.slack.com and set one up yourself for free.


Built more for casual use, Discord offers live chat functionality. While the dedicated Discord app includes voice chat support, this is not yet supported on Raspberry Pi – but you can still use text chat by opening the browser, going to discord.com, and choosing the ‘Open Discord in your browser’ option. Choose a username, read and agree to the terms of service, then enter an email address and password to set up your own free Discord server. Alternatively, if you know someone on Discord already, ask them to send you an invitation to access their server.

Firefox Send

If you need to send a document, image, or any other type of file to someone who isn’t on Google Drive, you can use Firefox Send – even if you’re not using the Firefox browser. All files transferred via Firefox Send are encrypted, and can be protected with an optional password, and are automatically deleted after a set number of downloads or length of time. Simply open the browser and go to send.firefox.com; you can send files up to 1GB without an account, or sign up for a free Firefox account to increase the limit to 2.5GB.


For programmers, GitHub is a lifesaver. Based around the Git version control system, GitHub lets teams work on a project regardless of distance using repositories of source code and supporting files. Each programmer can have a local copy of the program files, work on them independently, then submit the changes for inclusion in the master copy – complete with the ability to handle conflicting changes. Better still, GitHub offers additional collaboration tools including issue tracking. Open the browser and go to github.com to sign up, or sign in if you have an existing account, and follow the getting started guide on the site.

Read The MagPi for free!

Find more fantastic projects, tutorials, and reviews in The MagPi #93, out now! You can get The MagPi #92 online at our store, or in print from all good newsagents and supermarkets. You can also access The MagPi magazine via our Android and iOS apps.

Don’t forget our super subscription offers, which include a free gift of a Raspberry Pi Zero W when you subscribe for twelve months.

And, as with all our Raspberry Pi Press publications, you can download the free PDF from our website.

The post How to work from home with Raspberry Pi | The Magpi 93 appeared first on Raspberry Pi.

Lassie errors

Post Syndicated from esr original http://esr.ibiblio.org/?p=8674

I didn’t invent this term, but boosting the signal gives me a good excuse for a rant against its referent.

Lassie was a fictional dog. In all her literary, film, and TV adaptations the most recurring plot device was some character getting in trouble (in the print original, two brothers lost in a snowstorm; in popular false memory “Little Timmy fell in a well”, though this never actually happened in the movies or TV series) and Lassie running home to bark at other humans to get them to follow her to the rescue.

In software, “Lassie error” is a diagnostic message that barks “error” while being comprehensively unhelpful about what is actually going on. The term seems to have first surfaced on Twitter in early 2020; there is evidence in the thread of at least two independent inventions, and I would be unsurprised to learn of others.

In the Unix world, a particularly notorious Lassie error is what the ancient line-oriented Unix editor “ed” does on a command error. It says “?” and waits for another command – which is especially confusing since ed doesn’t have a command prompt. Ken Thompson had an almost unique excuse for extreme terseness, as ed was written in 1973 to run on a computer orders of magnitude less capable than the embedded processor in your keyboard.

Herewith the burden of my rant: You are not Ken Thompson, 1973 is a long time gone, and all the cost gradients around error reporting have changed. If you ever hear this term used about one of your error messages, you have screwed up. You should immediately apologize to the person who used it and correct your mistake.

Part of your responsibility as a software engineer, if you take your craft seriously, is to minimize the costs that your own mistakes or failures to anticipate exceptional conditions inflict on others. Users have enough friction costs when software works perfectly; when it fails, you are piling insult on that injury if your Lassie error leaves them without a clue about how to recover.

Really this term is unfair to Lassie, who as a dog didn’t have much of a vocabulary with which to convey nuances. You, as a human, have no such excuse. Every error message you write should contain a description of what went wrong in plain language, and – when error recovery is possible – contain actionable advice about how to recover.

This remains true when you are dealing with user errors. How you deal with (say) a user mistake in configuration-file syntax is part of the user interface of your program just as surely as the normally visible controls are. It is no less important to get that communication right; in fact, it may be more important – because a user encountering an error is a user in trouble that he needs help to get out of. When Little Timmy falls down a well you constructed and put in his path, your responsibility to say something helpful doesn’t lessen just because Timmy made the immediate mistake.

A design pattern I’ve seen used successfully is for immediate error messages to include both a one-line summary of the error and a cookie (like “E2317”) which can be used to look up a longer description including known causes of the problem and remedies. In a hypothetical example, the pair might look like this:

Out of memory during stream parsing (E1723)

E1723: Program ran out of memory while building the deserialized internal representation of a stream dump. Try lowering the value of GOGC to cause more frequent garbage collections, increasing the size if your swap partition, or moving to hardware with more RAM.

The key point here is that the user is not left in the lurch. The messages are not a meaningless bark-bark, but the beginning of a diagnosis and repair sequence.

If the thought of improving user experience in general leaves you unmoved, consider that the pain you prevent with an informative error message is rather likely to be your own, as you use your software months or years down the road or are required to answer pesky questions about it.

As with good comments in your code, it is perhaps most motivating to think of informative error messages as a form of anticipatory mercy towards your future self.

Payload, singleton, and stride lengths

Post Syndicated from esr original http://esr.ibiblio.org/?p=8663

Once again I’m inventing terms for useful distinctions that programmers need to make and sometimes get confused about because they lack precise language.

The motivation today is some issues that came up while I was trying to refactor some data representations to reduce reposurgeon’s working set. I realized that there are no fewer than three different things we can mean by the “length” of a structure in a language like C, Go, or Rust – and no terms to distinguish these senses.

Before reading these definitions, you might to do a quick read through The Lost Art of Structure Packing.

The first definition is payload length. That is the sum of the lengths of all the data fields in the structure.

The second is stride length. This is the length of the structure with any interior padding and with the trailing padding or dead space required when you have an array of them. This padding is forced by the fact that on most hardware, an instance of a structure normally needs to have the alignment of its widest member for fastest access. If you’re working in C, sizeof gives you back a stride length in bytes.

I derived the term “stride length” for individual structures from a well-established traditional use of “stride” for array programming in PL/1 and FORTRAN that is decades old.

Stride length and payload length coincide if the structure has no interior or trailing padding. This can sometimes happen when you get an arrangement of fields exactly right, or your compiler might have a pragma to force tight packing even though fields may have to be accessed by slower multi-instruction sequences.

“Singleton length” is the term you’re least likely to need. It’s the length of a structure with interior padding but without trailing padding. The reason I’m dubbing it “singleton” length is that it might be relevant in situations where you’re declaring a single instance of a struct not in an array.

Consider the following declarations in C on a 64-bit machine:

struct {int64_t a; int32_t b} x;
char y

That structure has a payload length of 12 bytes. Instances of it in an array would normally have a stride length of 16 bytes, with the last two bytes being padding. But in this situation, with a single instance, your compiler might well place the storage for y in the byte immediately following x.b, where there would trailing padding in an array element.

This struct has a singleton length of 12, same as its payload length. But these are not necessarily identical, Consider this:

struct {int64_t a; char b[6]; int32_t c} x;

The way this is normally laid out in memory it will have two bytes of interior padding after b, then 4 bytes of trailing padding after c. Its payload length is 8 + 6 + 4 = 20; its stride length is 8 + 8 + 8 = 24; and its singleton length is 8 + 6 + 2 + 4 = 22.

To avoid confusion, you should develop a habit: any time someone speaks or writes about the “length” of a structure, stop and ask: is this payload length, stride length, or singleton length?

Most usually the answer will be stride length. But someday, most likely when you’re working close to the metal on some low-power embedded system, it might be payload or singleton length – and the difference might actually matter.

Even when it doesn’t matter, having a more exact mental model is good for reducing the frequency of times you have to stop and check yourself because a detail is vague. The map is not the territory, but with a better map you’ll get lost less often.

shellcheck: boosting the signal

Post Syndicated from esr original http://esr.ibiblio.org/?p=8622

I like code-validation tools, because I hate defects in my software and I know that there are lots of kinds of defects that are difficult for an unaided human brain to notice.

On my projects, I throw every code validater I can find at my code. Standbys are cppcheck for C code, pylint for Python, and go lint for Go code. I run these frequently – usually they’re either part of the “make check” I use to run regression tests, or part of the hook script run when I push changes to the public repository.

A few days ago I found another validator that I now really like: shellcheck Yes, it’s a lint/validator for shell scripts – and in retrospect shell, as spiky and irregular and suffused with multilevel quoting as it is, has needed something like this for a long time.

I haven’t done a lot of shell scripting in the last couple of decades. It’s not a good language for programming at larger orders of magnitude than 10 lines or so – too many tool dependencies, too difficult to track what’s going on. These problems are why Perl and later scripting language became important; if shell had scaled up better the space they occupy would have been shell code as far as they eye can see.

But sometimes you write a small script, and then it starts to grow, and you can end up in an awkward size range where it isn’t quite unmanageable enough to drive you to port it to (say) Python yet. I have some cases like this in the reposurgeon suite.

For this sort of thing a shell validater/linter can be a real boon, enabling you to have much more confidence that you’ll catch silly errors when you modify the script, and actually increasing the upper limit of source-line count at which shell remains a viable programming language.

So it is an excellent thing that shellcheck is a solid and carefully-thought-out piece of work. It does catch lot of nits and potential errors, hardening your script against cases you probably haven’t tested yet. For example. it’s especially good at flagging constructs that will break if a shell variable like $1 gets set to a value with embedded whitspace.

It has other features you want in a code validator, too. You can do line-by-line suppression of specific spellcheck warnings with magic comments, telling the tool “Yes, I really meant to do that” so it will shut up. This means when you get new warnings they are really obvious.

Also, it’s fast. Fast enough that you can run it on all your shellscripts up front of all your regular regression tests and probably barely ever notice the time cost.

It’s standard practice for me to have a “make check” that runs code validators and then the regression tests. I’m going back and adding shellcheck validation to those check productions on all my projects that ship shell scripts. I recommend this as a good habit to everybody.

Reposurgeon defeats all monsters!

Post Syndicated from esr original http://esr.ibiblio.org/?p=8607

On January 12th 2020, reposurgeon performed a successful conversion of its biggest repository ever – the entire history of the GNU Compiler Collection, 280K commits with a history stretching back through 1987. Not only were some parts CVS, the earliest portions predated CVS and had been stored in RCS.

I waited this long to talk about it to give the dust time to settle on the conversion. But it’s been 5 weeks now and I’ve heard nary a peep from the GCC developers about any problems, so I think we can score this as reposurgeon’s biggest victory yet.

The Go port really proved itself. Those 280K commits can be handled on the 128GB Great Beast with a load time of about two hours. I have to tell the Go garbage collector to be really aggressive – set GOGC=30 – but that’s exactly what GOGC is for.

The Go language really proved itself too. The bet I made on it a year ago paid off handsomely – the increase in throughput from Python is pretty breathtaking, at least an order of magnitude and would have been far more if it weren’t constrained by the slowness of svnadmin dump. Some of that was improved optimization of the algorithms – we knocked out one O(n**2) after translation. More of it, I think, was the combined effect of machine-code speed and much smaller object sizes – that reduced working set a great deal, meaning cache miss penalties got less frequent.

Also we got a lot of speedup out of various parallelization tricks. This deserves mention because Go made it so easy. I wrote – and Julien Rivaud later improved – a function that would run a specified functional hook on the entire set of repository events, multithreading them from a worker pool optimally sized from your machine’s number of processors, or (with the “serialize” debug switch on) running them serially.

That is 35 lines of easily readable code in Go, and we got no fewer than 9 uses out of it in various parts of the code! I have never before used a language in which parallelism is so easy to manage – Go’s implementation of Communicating Sequential Processes is nothing short of genius and should be a model for how concurrency primitives are done in future languages.

Thanks where thanks are due: when word about the GCC translation deadline got out, some of my past reposurgeon contributors – notably Edward Cree, Daniel Brooks, and Julien Rivaud – showed up to help. These guys understood the stakes and put in months of steady, hard work along with me to make the Go port both correct and fast enough to be a practical tool for a 280K-commit translation. Particular thanks to Julien, without whose brilliance and painstaking attention to detail I might never have gotten the Subversion dump reader quite correct.

While I’m giving out applause I cannot omit my apprentice Ian Bruene, whose unobtrusively excellent work on Kommandant provided a replacement for the Cmd class I used in the original Python. The reposurgeon CLI wouldn’t work without it. I recommend it to anyone else who needs to build a CLI in Go.

These guys exemplify the best in what open-source collegiality can be. The success of the GCC lift is almost as much their victory as it is mine.

Build engines suck. Help GPSD select a new one.

Post Syndicated from esr original http://esr.ibiblio.org/?p=8581

One of the eternal mysteries of software is why build engines suck so badly.

Makefiles weren’t terrible as a first try, except for the bizarre decision to make the difference between tabs and spaces critically different so you can screw up your recipe invisibly.

GNU autotools is a massive, gnarly, hideous pile of cruft with far too many moving parts. Troubleshooting any autotools recipe of nontrivial capacity will make you want to ram your forehead repeatedly into a brick wall until the pain stops.

Scons, among the first of the new wave of build engines to arise when mature scripting languages make writing them easier, isn’t bad. Except for the part where the development team was unresponsive at the best of times and development has stagnated for years now.

Waf is also not bad, except for being somewhat cryptic to write and having documentation that would hardly be less comprehensible if it had been written in hieroglyphics.

cmake and meson, two currently popular engines, share one of the (many) fatal flaws of autotools. They don’t run recipes directly, they compile recipes to some simpler form to be run by a back-end builder (typically, but not always, such systems generate makefiles). Recipe errors thrown by the back end don’t necessarily have any direct relationship to an identifiable part of the recipe you give your front end, making troubleshooting unnecessarily painful and tedious.

Wikipedia has a huge list of build-automation systems. Most seem to be relatively new (last decade) and if any of them had distinguished itself enough to be a cler winning choice I’m pretty sure I’d already know it.

Scons is where I landed after getting fed up with GPSD’s autotools build. In retrospect it is still probably the best choice we could have made at the time (around 2006 IIRC), and I think I would be very happy with it if it had lived up to its early promise. It didn’t. Time to move on.

Waf is what NTPsec uses, and while it has served us well the abysmally bad state of the documentation and the relatively high complexity cost of starting a recipe from a blank sheet of paper make it a questionable choice going forward. It doesn’t help that the project owner, while brilliant, does not communicate with lesser mortals very well.

Is there anything in this space that doesn’t have awful flaws?

Here’s what we want:

– works on any reasonable basically-POSIX system for bulding C programs

– there is no requirement for native Windows builds, and perhaps no requirement for Windows at all.

– has a relatively compact expression of build rules, which more or less means declarative notation rather than writing code to do builds

– has a track record of being maintained, and enough usage by other projects that there is a reasonable expectation that the build system will continue to be a reasonable choice for at least 5 years, and ideally longer

– doesn’t impose difficult requirements beyond what gpsd does already (e.g., needing C++11 for a build tool would be bad)

– has a notion of feature tests, rather than ifdefing per operating system, and promotes a culture of doing it that way

– supports setting build prefix, and enabling/disabling options, etc.

– supports dealing with -L/-R flags as expected on a variety of systems

– supports running tests

– supports running the just-built code before installing so that it uses the built libs and does not pick up the already-in-system libs (necessary for tests)

– supports cross compiling

– supports building placing the objects in a specified location outside the source tree

– supports creating distfiles (suffisient if it can shell out to tar commands).

– supports testing the created distfile by unpacking it, doing an out-of-source build and then running tests

– is not a two-phase generator system like autotools, cmake, and meson.

What in Wikipedia’s huge list should we be looking at?

30 Days in the Hole

Post Syndicated from esr original http://esr.ibiblio.org/?p=8543

Yes, it’s been a month since I posted here. To be more precise, 30 Days in the Hole – I’ve been heads-down on a project with a deadline which I just barely met. and then preoccupied with cleanup from that effort.

The project was reposurgeon’s biggest conversion yet, the 280K-commit history of the Gnu Compiler Collection. As of Jan 11 it is officially lifted from Subversion to Git. The effort required to get that done was immense, and involved one hair-raising close call.

I was still debugging the Go translation of the code four months ago when the word came from the GCC team that they has a firm deadline of December 16 to choose between reposurgeon and a set of custom scripts written by a GCC hacker named Maxim Kyurkov. Which I took a look at – and promptly recoiled from in horror.

The problem wasn’t the work of Kyurkov himself; his scripts looked pretty sane to me, But they relied on git-svn, and that was very bad. It works adequately for live gatewaying to a Subversion repository, but if you use it for batch conversions it has any number of murky bugs including a tendency to badly screw up the location of branch joins.

The problem I was facing was that Kyurkov and the GCC guys, never having had their noses rubbed in these problems as I had, might be misled by git-svn’s surface plausibility into using it, and winding up with a subtly damaged conversion and increased friction costs for the rest of time. To head that off, I absolutely had to win on 16 Dec.

Which wan’t going to be easy. My Subversion dump analyzer had problems of it own. I had persistent failures on some particularly weird cases in my test suite, and the analyzer itself was a hairball that tended to eat RAM at prodigious rates. Early on, it became apparent that the 128GB Great Beast II was actually too small for the job!

But a series of fortunate occurrences followed. One was that friend at Amazon was able to lend me access to a really superpowered load machine with 512GB. The second and much more important was in mid-October when a couple of occasional reposurgeon contributors, Julien “__FrnchFrgg__” Rivaud and Daniel Brooks showed up to help – Daniel having wangled his boss’s permission to go full-time on this until it was done. (His boss;s who;e company is critically depended on GCC flourishing.)

Many, many hours of hard work followed – profiling, smashing out hidden O(n**2) loops that exploded on a repo this size, reducing working set, fixing analyzer bugs. I doubled my lifetime consumption of modafinil. And every time I scoped what was left to do I came up with the same answer: we would just barely make the deadline. Probably.

Until…until I had a moment of perspective after three week of futile attempts to patch the latest round of Subversion-dump analyzer bugs and realized that trying to patch-and-kludge my way around the last 5% of weird cases was probably not going to work. The code had become a rubble pile; I couldn’t change anything without breaking anything.

It looked like time to scrap everything downstream of the first-stage stream parser (the simplest part, and the only one I was completely sure was correct) and rebuild the analyzer from first principles using what I had learned from all the recent failures.

Of course the risk I was taking was that come deadline time the analyzer wouldn’t be 95% right but rather catastrophically broken – that there simply wouldn’t be time to get the cleaner code working and qualified. But after thinking about the odds a great deal, I swallowed hard and pulled the trigger on a rewrite.

I made the fateful decision on 29 Nov 2019 and as the Duke of Wellington famously said, “It was a damned near-run thing.” If I had waited even a week longer to pull that trigger, we would probably have failed.

Fortunately, what actually happened was this: I was able to factor the new analyzer into a series of passes, very much like code-analysis phases in a compiler. The number fluctuated, there ended up being 14 of them, but – and this is the key point – each pass was far simpler than the old code, and the relationships between then well-defined. Several intermediate state structures that had become more complication than help were scrapped.

Eventually Julien took over two of the trickier intermediate passes so I could concentrate on the worst of the bunch. Meanwhile, Daniel was unobtrusively finding ways to speed the code and slim its memory usage down. And – a few days before the deadline – the GCC project lead and a sidekick showed up on our project channel to work on improving the conversion recipe.

After that formally getting the nod to do the conversion was not a huge surprise. But there was a lot of cleanup, verification, and tuning to be done before the official repository cutover on Jan 11. What with one thing and another in was Jan 13 before I could declare victory and ship 4.0.

After which I promptly…collapsed. Having overworked myself, I picked up a cold. Normally for me this is no big deal; I sniffle and sneeze for a few days and it barely slows me down. Not this time – hacking cough, headaches, flu-like symptoms except with no fever at all, and even the occasional dizzy spell because the trouble spread to my left ear canal.

I’m getting better now. But I had planned to go to the big pro-Second Amendment demonstration in Richmond on Jan 20th and had to bail at the last minute because I was too sick to travel.

Anyway, the mission got done. GCC has a really high-quality Git repository now. And there will be a sequel to this, my first GCC compiler mod.

And posting at something like my usual frequency will resume. I have a couple of topics queued up.

Segfaults and Twitter monkeys: a tale of pointlessness

Post Syndicated from esr original http://esr.ibiblio.org/?p=8394

For a few years in the 1990s, when PNG was just getting established as a Web image format, I was a developer on the libpng team.

One reason I got involved is that the compression patent on GIFs was a big deal at the time. I had been the maintainer of GIFLIB since 1989; it was on my watch that Marc Andreesen chose that code for use in the first graphics-capable browser in ’94. But I handed that library off to a hacker in Japan who I thought would be less exposed to the vagaries of U.S. IP law. (Years later, after the century had turned and the LZW patents expired, it came back to me.)

Then, sometime within a few years of 1996, I happened to read the PNG standard, and thought the design of the format was very elegant. So I started submitting patches to libpng and ended up writing the support for six of the minor chunk types, as well as implementing the high-level interface to the library that’s now in general use.

As part of my work on PNG, I volunteered to clean up some code that Greg Roelofs had been maintaining and package it for release. This was “gif2png” and it was more or less the project’s official GIF converter.

(Not to be confused, though, with the GIFLIB tools that convert to and from various other graphics formats, which I also maintain. Those had a different origin, and were like libgif itself rather better code.)

gif2pngs’s role then was more important than it later became. ImageMagick already existed, but not in anything like its current form; GIMP had barely launched, and the idea of a universal image converter hadn’t really taken hold yet. The utilities I ship with GIFLIB also had an importance then that they would later lose as ImageMagick’s “convert” became the tool everyone learned to reach for by reflex.

It has to be said that gif2png wasn’t very good code by today’s standards. It had started life in 1995 as a dorm-room project written in journeyman C, with a degree of carelessness about type discipline and bounds checking that was still normal in C code of the time. Neither Greg nor I gave it the thorough rewrite it perhaps should have gotten because, after all, it worked on every well-formed GIF we ever threw at it. And we had larger problems to tackle.

Still, having taken responsibility for it in ’99. I kept it maintained even as it was steadily decreasing in importance. ImageMagick convert(1) had taken over; I got zero bug reports or RFEs for six years between 2003 and 2009.

I did some minor updating in 2010, but more out of completism than anything else; I was convinced that the user constituency for the tool was gone. And that was fine with me – convert(1) had more eyes on it and was almost certainly better code. So gif2png fell to near the bottom of my priority list and stayed there.

A few years after that, fuzzer attacks on programs started to become a serious thing. I got one against GIFLIB, which was issued a CVE and I took very seriously – rogue code execution in a ubiquitous service library is baaaad. A couple of others in GIFLIB’s associated utility programs, which I took much less seriously as I wasn’t convinced anyone still used them at all. You’re going to exploit these…how?

And, recently, two segfaults in gif2png. Which was absolutely at the bottom of my list of security concerns. Standalone program, designed to be used in input files you trust to be reasonably close to well-formed GIFs (there was a ‘recover’ option that could salvage certain malformed ones if you were very lucky). Next to no userbase since around 2003. Again, you’re going to exploit this…how?

Now, I’m no infosec specialist, but there is real-world evidence that I know how to get my priorities right. I’ve led the the NTPsec project for nearly five years now, reworking its code so thoroughly that its size has shrunk by a factor of 4. NTP implementations are a prime attack target because the pre-NTPsec reference version used to be so easy to subvert. And you know what the count of CVEs against our code (as opposed to what we inherited) is?

Zero. Zip. Zilch. Nobody has busted my code or my team’s. Despite half the world’s academics and security auditors running attacks on it. Furthermore, we have a record of generally having plugged about four out of five CVEs in the legacy code by the time they’re issued.

That’s how the security of my code looks when I think it’s worth the effort. For GIFLIB I’ll spend that effort willingly. For the GIFLIB tools, less willingly. But for gif2png, that seemed pointless. I was tired of spending effort to deal with the 47,000th CS student thinking “I know! I’ll run a fuzzer on !” and thinking a crash was a big deal when the program was a superannuated standalone GIF filter that hasn’t seen any serious use since J. Random Student was in diapers.

So two days ago I marked two crashes on malformed input in gif2png won’t-fix, put in in a segfault handler so it would die gracefully no matter what shit you shoved at it, and shipped it…

…only to hear a few ours later, from my friend Perry Metzger, that there was a shitstorm going down on Twitter about how shockingly incompetent this was.

Really? They really thought this program was an attack target, and that you could accomplish anything by running rogue code from inside it?

Narrator voice: No, they didn’t. There are some people for whom any excuse to howl and fling feces will do.

A similar bug in libgif or NTPsec would have been a serious matter. But I’m pretty good at not allowing serious bugs to happen in those. In a quarter century of writing critical service code my CVE count is, I think, two (one long ago in fetchmail) with zero exploits in the wild.

This? This ain’t nothin’. Perry did propose a wildly unlikely scenario in which the gif2png binary somehow got wedged in the middle of somebody’s web framework on a server and allowed to see ill-formed input, allowing a remote exploit, but I don’t believe it.

Alas, if I’ve learned anything about living on the modern Internet it’s that arguing that sort of point with the howler monkeys on Twitter is a waste of time. (Actually, arguing anything with the howler monkeys on Twitter is a waste of time.) Besides, the code may not be an actual security hazard, but it has been kind of embarrassing to drag around ever since I picked it up.

So, rather than patch the C and deal with yet another round of meaningless fuzzer bugs in the future, I’ve rewritten it in Go. Here it is, and now that it’s in a type-safe language with access bounds checking I don’t ever have to worry about that class of problem again.

One good thing may come of this episode (other than lifting code out of C, which is always a plus). I notice that the GIF and PNG libraries in Go are, while serviceable for basic tasks, rather limited. You can convert with them, but you can’t do lossless editing with them. Neither one deserializes the entire ontology of its file format.

As the maintainer of GIFLIB and a past libpng core developer, I don’t know where I’d find a better-qualified person to fix this than me. So now on my to-do list, though not at high priority: push some patches upstream to improve these libraries.

Fear of COMITment

Post Syndicated from Eric Raymond original http://esr.ibiblio.org/?p=8375

I shipped the first release of another retro-language revival today: COMIT. Dating from 1957 (coincidentally the year I was born) this was the first string-processing language, ancestral to SNOBOL and sed and ed and Unix shell. One of the notational conventions invented in COMIT, the use of $0, $1…etc. as substitution variables, survives in all these languages.

I actually wrote most of the interpreter three years ago, when a copy of the COMIT book fell into my hands (I think A&D regular Daniel Franke was responsible for that). It wasn’t difficult – 400-odd lines of Python, barely enough to make me break a sweat. That is, until I hit the parts when the book’s description of the language is vague and inadequate.

It was 1957 and nobody knew hardly anything about how to describe computer language systematically, so I can’t fault Dr. Victor Yngve too harshly. Where I came a particular cropper was trying to understand the intended relationship between indices into the workspace buffer and “right-half relative constituent numbers”. That defeated me, so I went off and did other things.

Over the last couple days, as part of my effort to promote my Patreon feed to a level where my medical expenses are less seriously threatening, I’ve been rebuilding all my project pages to include a Patreon button and an up-to-date list of Bronze and Institutional patrons. While doing this I tripped over the unshipped COMIT code and pondered what to do with it.

What I decided to do was ship it with an 0.1 version number as is. The alternative would have been to choose from several different possible interpretations of the language book and quite possibly get it wrong.

I think a good rule in this kind of situation is “First, do no harm”. I’d rather ship an incomplete implementation that can be verified by eyeball, and that’s what I’ve done – I was able to extract a pretty good set of regression tests for most of the features from the language book.

If someone else cares enough, some really obsessive forensics on the documentation and its code examples might yield enough certainty about the author’s intentions to support a full reconstruction. Alas, we can’t ask him for help, as he died in 2012.

A lot of the value in this revival is putting the language documentation and a code chrestomathy in a form that’s easy to find and read, anyway. Artifacts like COMIT are interesting to study, but actually using it for anything would be perverse.

The dangerous folly of “Software as a Service”

Post Syndicated from Eric Raymond original http://esr.ibiblio.org/?p=8338

Comes the word that Saleforce.com has announced a ban on its customers selling “military-style rifles”.

The reason this ban has teeth is that the company provides “software as a service”; that is, the software you run is a client for servers that the provider owns and operates. If the provider decides it doesn’t want your business, you probably have no real recourse. OK, you could sue for tortious interference in business relationships, but that’s chancy and anyway you didn’t want to be in a lawsuit, you wanted to conduct your business.

This is why “software as a service” is dangerous folly, even worse than old-fashioned proprietary software at saddling you with a strategic business risk. You don’t own the software, the software owns you.

It’s 2019 and I feel like I shouldn’t have to restate the obvious, but if you want to keep control of your business the software you rely on needs to be open-source. All of it. All of it. And you can’t afford it to be tethered to a service provider even if the software itself is nominally open source.

Otherwise, how do you know some political fanatic isn’t going to decide your product is unclean and chop you off at the knees? It’s rifles today, it’ll be anything that can be tagged “hateful” tomorrow – and you won’t be at the table when the victim-studies majors are defining “hate”. Even if you think you’re their ally, you can’t count on escaping the next turn of the purity spiral.

And that’s disregarding all the more mundane risks that come from the fact that your vendor’s business objectives aren’t the same as yours. This is ground I covered twenty years ago, do I really have to put on the Mr. Famous Guy cape and do the rubber-chicken circuit again? Sigh…

Business leaders should to fear every piece of proprietary software and “service” as the dangerous addictions they are. If Salesforce.com’s arrogant diktat teaches that lesson, it will have been a service indeed.

Contributor agreements considered harmful

Post Syndicated from Eric Raymond original http://esr.ibiblio.org/?p=8287

Yesterday I got email from a project asking me to wear my tribal-elder hat, looking for advice on how to re-invent its governance structure. I’m not going to name the project because they haven’t given me permission to air their problems in public, but I need to write about something that came up during the discussion, when my querent said they were thinking about requiring a contributor release form from people submitting code, “the way Apache does”.

“Don’t do it!” I said. Please don’t go the release-form route. It’s bad for the open-source community’s future every time someone does that. In the rest of this post I’ll explain why.

Every time a project says “we need you to sign a release before we’ll take your code”, it helps create a presumption that such releases are necessary – as opposed to the opposite theory, which is that the act of donating code to an open-source project constitutes in itself a voluntary cession of the project’s right to use it under terms implied by the open-source license of the project.

Obviously one of those theories is better for open source – no prize for guessing which.

Here is the language NTPsec uses in its hacking guide:

By submitting patches to this project, you agree to allow them to be redistributed under the project’s license according to the normal forms and usages of the open-source community.

There is as much legal ground for the cession theory of contribution as there is for any idea that contributor releases are required for some nebulous kind of legal safety. There’s no governing statute and no case law on this; no dispute over an attempt to revoke a contribution has yet been adjudicated.

And here’s the ironic part: if it ever comes to a court case, one of the first things the judge is going to look at is community expectations and practice around our licenses. A jurist is supposed to do this in contract and license cases; there’s some famous case law about the interpretation of handshake contracts among Hasidic Jewish diamond merchants in New York City that makes this very clear and explicit. Where there is doubt about interpretation and no overriding problem of of equity, the norms of the community within which the license/contract was arrived at should govern.

So, if the judge thinks that we expect contribution permissions to fail closed unless explicitly granted, he/she is more likely to make that happen. On the other hand, if he/she thinks that community norms treat contribution as an implied cession of certain rights in exchange for the benefits of participating in the project, that is almost certainly how the ruling will come out.

I say, therefore, that Apache and the FSF and the Golang project and everybody else requiring contributor releases are wrong. Because there is no governing law on the effect of these release forms, they are not actually protection against any risk, just a sort of ritual fundament-covering that a case of first impression could toss out in a heartbeat. Furthermore, the way they’ve gone wrong is dangerous; this ritual fundament-covering could someday bring about the very harm it was intended to prevent.

If your project has a contributor release, do our whole community a favor and scrap it. Any lawyer who tells you such a thing is necessary is talking out his ass – he doesn’t know that, and at the present state of the law he can’t know it.

(My wife Cathy, the attorney, concurs. So this advice isn’t just a layperson vaporing in ignorance.)

Instead, post a contract of adhesion on your website or in your guide for contributors. Use my language, or edit to taste. The one thing you should be sure stays in is some language equivalent to this part: “according to the normal forms and usages of the open-source community”.

That is important because, if it ever comes to a court case, we want to be able to point the judge at that as a clue that there are normal forms and usages and he/she can do what he’s supposed to and almost certainly wants to do by understanding them and applying them.

Am I really shipper’s only deployment case?

Post Syndicated from Eric Raymond original http://esr.ibiblio.org/?p=8281

I released shipper 1.14 just now. It takes advantage of the conventional asciidoc extension – .adoc – that GitHub and GitLab have established, to do a useful little step if it can detect that your project README and NEWS files are asciidoc.

And I wondered, as I usually do when I cut a shipper release: am I really the only user this code has? My other small projects (things like SRC and irkerd) tend to attract user communities that stick with them, but I’ve never seen any sign of that with shipper – no bug reports or RFEs coming in over the transom.

This time, it occurred to me that if I am shipper’s only user, then maybe the typical work practices of the open-source community are rather different than I thought they were. That’s a question worth raising in public, so I’m posting it here to attract some comment.

The problem shipper solves for me is this: I run, by exact count that I checked just now, 52 small projects. By ‘small’ I really mean “not NTPsec or GPSD”; some, like reposurgeon, are relatively large as codebases. What really distinguishes GPSD and NTPsec is their need to have custom websites full of resource pages, blogs, and other impedimenta; a small project, on the other hand, can do with one page existing mainly to carry a description, recent project news, and a place to download the latest release tarball.

About a quarter of these projects are pretty dormant. On the other hand, among the dozen most active of them it’s not unusual for me to ship two or three point releases a week. Without tooling that would be a really annoying amount of hand-work – making distribution tarballs, copying them to my personal website or places like SourceForge that carry tarballs, editing web pages to publish the latest project news, emailing distribution packagers to tell them there’s a new release, posting announcements to IRC channels. Inevitably I’d make errors, for example by forgetting to email people who have asked to be notified when I ship Project X.

That’s the problem shipper solves. A shipperized project contains a control file that gathers all the metadata required to ship a release. Then its makefile (or equivalent) can have a “release” production that runs the project’s build and regression test to make sure it’s in a sane state, and if so proceeds to ship tarballs to their public download sites, template a web page with project news and documentation/download links and ship that to where it needs to live, and finally send out notifications on all required email and IRC channels.

Here’s what a typical control file looks like:

# This is not a real Debian control file, though the syntax is compatible.
# It's project metadata for the shipper tool

Package: shipper

Description: Automated shipping of open-source project releases.
 shipper is a power distribution tool for developers with multiple
 projects who do frequent releases.  It automates the tedious process
 of shipping a software release and (if desired) templating a project
 web page. It can deliver releases in correct form to SourceForge,

Homepage: http://www.catb.org/~esr/shipper

XBS-IRC-Channel: irc://chat.freenode.net/shipper

XBS-Repository-URL: https://gitlab.com/esr/shipper

XBS-Logo: shipper-logo.png

#XBS-Project-Tags: packaging, distribution

XBS-Validate: make check

The shipper code figures out everything it needs to do from that or from looking at the files in your source tree – for example, it assumes that any file with an “.html” extension that exists in the top level of your project tree at the time it’s run should be shipped to the project website along with the templated index page. About all you need to pass it on each invocation is a release number.

From the user’s point of view, the good thing about a shipperized project is that the templated page puts the project blurb, links to up-to-date documentation, and project news in one place that’s easy to find – no need to go digging for them.

My shipperized projects also have a “refresh” production that just updates the project page. This is useful, for example, when I find a typo in the documentation or blurb and want to correct that without shipping an actual code release. Fix it, run “make refresh”, done.

The only real cost this tool imposes other than making the initial control file is that your NEWS file has to be in a particular format in order for it to extract the latest item: stanzas separated by blank lines, each with a recognizable date on the first line, newest at the top.

Over the years this little tool has saved me from immense amounts of process friction and substantially reduced the defect rate in what I actually ship. I can release more frequently, being more responsive to bugs and RFEs, because doing so isn’t much work. It’s easy to use, it’s well documented, it’s very customizable (you can set up your own web page template, etc). So…why isn’t anybody else using it? Why do I never get bug reports or RFEs the way I so frequently do on my other projects?

The possibility that occurs to me now is that maybe it solves a problem nobody else really has to the same extent. While shipper is useful for eliminating stupid finger errors while shipping even if you have just one project, it really comes into its own if you have a dozen. So maybe people aren’t doing that? Is being a lead on over a dozen projects more of an outlier than I thought?

The obvious alternative hypothesis is that repository-centered forges like GitHub and GitLab have eaten the role small-project pages used to have – your project page is now the README on its forge. I could almost believe that, except that (a) “stable release” is an important distinction for a lot of projects, those tarballs have to live somewhere, (b) forge READMEs also aren’t so good for advertising things like project mailing lists, and (c) because of a and b I see a lot of small-project web pages when I go browsing.

Another hypothesis is that there are lots of people who would find shipper useful, but they don’t know it exists. Also possible – but when I write other things that seem comparably useful, like SRC or irkerd or deheader, they don’t seem to have any trouble being found by their natural constituents. My project issue trackers tell the tale!

Is it really normal for hackers to be working on so few projects at a time, and releasing so infrequently, that they never experience release-process friction as a problem? Perhaps so – but I’m curious what my commenters think.

Declarative is greater than imperative

Post Syndicated from Eric Raymond original http://esr.ibiblio.org/?p=8270

Sometimes I’m a helpless victim of my urges.

A while back -very late in 2016 – I started work on a program called loccount. This project originally had two purposes.

One is that I wanted a better, faster replacement for David Wheeler’s sloccount tool, which I was using to collect statistics on the amount of virtuous code shrinkage in NTPsec. David is good people and sloccount is a good idea, but internally it’s a slow and messy pile of kludges – so much so that it seems to have exceed his capacity to maintain, at time of writing in 2019 it hadn’t been updated since 2004. I’d been thinking about writing a better replacement, in Python, for a while.

Then I realized this problem was about perfectly sized to be my learn-Go project. Small enough to be tractable, large enough to not be entirely trivial. And there was the interesting prospect of using channels/goroutines to parallelize the data collection. I got it working well enough for NTP statistics pretty quickly, though I didn’t issue a first public release until a little over a month later (mainly because I wanted to have a good test corpus in place to demonstrate correctness). And the parallelized code was both satisfyingly fast and really pretty. I was quite pleased.

The only problem was that the implementation, having been a deliberately straight translation of sloccount’s algorithms in order to preserve comparability of the reports, was a bit of a grubby pile internally. Less so than sloccount’s because it was all in one language. but still. It’s difficult for me to leave that kind of thing alone; the urge to clean it up becomes like a maddening itch.

The rest of this post is about what happened when I succumbed. I got two lessons from this experience: one reinforcement of a thing I already knew, and one who-would-have-thought-it-could-go-this-far surprise. I also learned some interesting things about the landscape of programming languages.

For those of you who haven’t used sloccount, it counts source lines of code (SLOC) in a code tree. This is lines that are non-blank and not comments. SLOC counts are useful for complexity estimation and predicting defect incidence. SLOC is especially good for tracking how the complexity (and hence, for example, the potential attack surface) of a codebase has changed over time. There are more elaborate ways of making numbers that aim at similar usefuleness; the best known is called “cyclomatic complexity” and there’s another metric called LLOC (logical lines of code) that loccount can also tally for many languages. But it turns out empirically that it’s quite difficult to do better than SLOC for forecasting defect incidence, at least at the present state of our knowledge. More complex measures don’t seem to gain much; there is some evidence that LLOC does but even that is disputable.

I wanted to use loccount to post hard numbers about the continuing SLOC shrinkage of NTPSec. We’ve shrunk it by a little better than 4:1 since we forked it, from 231 KLOC to 55 KLOC, and that’s a big deal – a large reduction in attack surface and an equally large improvement in auditability. People have trouble believing a change that dramatic that unless you can show them numbers and how to reproduce those numbers; I wanted to be able to say “Here’s loccount and its test suite. Here’s our repository. See for yourself.”

But the code was still grubby, with multiple different rather ad-hoc parsers in it inherited from sloccount for different categories of languages; sloccount supports 27 in total. I didn’t like that, and over the next two years I would occasionally return to the code, gradually refactoring and simplifying.

Why parsers? Think about what you have to do to count SLOC. You need to run through a source file recognizing the following events:

  • Start of comment.
  • End of comment
  • Newline (so you can bump the SLOC counter when you’re outside a comment)
  • Start of string literal (so you start ignoring what would otherwise look like a comment start)
  • End of string (you stop ignoring comment starts)

This is complicated by the fact that in any given language there may be multiple syntaxes for start or end comment and start or end string literal. C and most other languages have two styles of comment; /* the block comment, which can contain newlines and has an explicit end delimiter */, versus // the winged comment, which can’t contain newlines and ends at the next one.

The most naive way to handle this simple grammar would be to write a custom parser for each language. Sometimes, when the language syntax is especially grotty you have to do that. There’s still a bespoke parser for Perl inside loccount. But most languages fall into fairly large syntactic families in which all the members can be parsed alike. Sloccount, supporting 27 languages, had at least four of these; C-like, Pascal-like, scripting-language-like. and Fortran-like.

Now, if you’re me, learning that most languages fall into families like that is interesting in itself. But the number of outliers that don’t fit, like Lisp and assembler (which sloccount can tally) and really odd ones like APL (which it can’t) is annoying. I can’t look at code like that without wanting to clean it up, simplify it, fold special cases into general ones, and learn about the problem space as I do so. Victim of my urges, I am.

I’ve written more parsers for messy data than I can remember, so I know the strategies for this. Let’s call the bespoke-parser-for-every-language level 0, and recognizing language families level 1. Sloccount was already at level 1 and loccount inherited that. Level 2 would be reducing the number of distinct parsers, ideally to one. How do we do that?

Let’s consider two specific, important family parsers: the C family (C, C++, Objective-C, Java, C#, Jana, Javascript, Scala) and the Pascal family. The main differences are (1) the block comment syntax is /* */ in C but (* *) in Pascal, (2) Pascal has no winged comments like C //, and (3) C double quotes around string literals vs. Pascal single quotes.

So instead of two parsers with two functions “is there a a /* next?” and “is there a (* next?”, you write one function “is there a block comment start next?” which dispatches to looking for /* or (*. Your “Is there a winged comment start?” file looks for // in a source file with a .c extension and always returns false in a file with a .pas extension.

That’s level 2. You’ve replaced “look for this family-specific syntax” with “look for the start-block-comment terminal” with the token-getter figuring out what to match based on which family the file contents is in. But that doesn’t handle idiosyncratic languages like – say – D, which uses /+ +/. Well, not unless you’re willing to let your start-block-comment matcher be a huge ugly switch statement with an arm not just for each family but for each outlier file extension.

And you’d need parallel big ugly switches in every other terminal – the one for end of block comment, the one for winged comment, the one for string delimiter, et tiresome cetera. Don’t do that; it’s ugly and hard to maintain.

Level 3 is where you change the big switch/case into a traits table. At start of processing you use the file extension to figure out which traits entry to use. You read your start and end block comment from that entry. It may have flags in it which declare, for example “this language uses single-quoted strings”, or “block comments nest in this language”.

This move from executable code to a traits table is really important, because it drastically reduces the complexity cost of adding support for new languages. Usually, now, each new one is just a table entry that can be easily composed from a read of the syntax part of the language specification. Declarative is greater than imperative!

The project NEWS file and repo history tell us how dramatic this effect is. In the first two years of the project’s lifetime I added about 14 languages to the original set of 27, less than one a month. Late last month I got the ontology of the traits table about right after staring long enough at the languages I’d been supporting. I immediately added five languages on one day.

Then I hit a bump because I needed more trait flags to handle the next couple of languages. But as I added the next half-dozen or so (Algol 60, F#, Dart, Cobra, VRML, Modula-2, and Kotlin) I noticed something interesting; the frequency with which I needed to add trait flags was dropping as I went. I was beginning to mine out the space of relevant syntactic variations.

Now I want to draw a contrast with the project I used to define the term Zeno tarpit. On doclifter I can always make more progress towards lifting the last few percent of increasingly weird malformed legacy documents out of the nroff muck if I’m willing to push hard enough, but the cost is that the pain never ends. Each increment of progress normally requires more effort than the last.

In loccount I’m clearly heading in the opposite direction. In the last week (and remember most of my attention is on NTPsec; this is a side project) I’ve added no fewer than 34 new languages and markup formats to loccount. My time to put in a new one is down to minutes – skim the spec, fill in a new table entry, drop a sample into the test directory, eyeball the results to make sure they’re sane, and update the news file.

The contrast with the effort required to get another 0.1% down the long tail of the man-page corpus is really extreme. What makes the difference?

OK, yes, the nroff-lifting problem is more complicated. You’re parsing for many more different kinds of terminals and relations among them. That’s true but it’s not a “true” that generates much insight. No; the real difference is that in that domain, I still have to write a parser extension – executable code – in order to make progress. In loccount, on the other hand, I’m just composing new table entries. Declarative is greater than imperative!

Actually it has become a game; I’ve made a couple of passes through Wikipedia’s List of programming languages looking for plausible candidates. I’ve put in historical relics like SNOBOL4 and B simply because I could. When you’re on a stamp-collecting streak like this, it’s natural to start wondering just how far you can push it. A victim of my urges sometimes, am I.

I do have some limits, though. I haven’t gone after esoteric languages, because those are often deliberately constructed to be so syntactically bizarre that upgrding my parser would be more like work than fun. And I really don’t care about the 17th dialect of Business BASIC or yet another proprietary database language, thank you. Also I have in general not added languages that I judged to be academic toys intended moe to generate research papers rather than production code, though I gave in on a few I thought to be of particular historical interest.

The lesson here is one I know I’ve written about before, but it deserves reinforcing. Declarative is greater than imperative. Simple code plus smart data is better than enough smart code to do the same job. The move from four parsers to one parser and a big trait table was a win on every level – the result is easier to understand and much easier to extend. This is still an underutilized design strategy.

There is a level 4. I may yet do something like what we did with Open Adventure and move all that smart data from a table initializer to a YAML file compiled back into the trait table at build time. Then adding new languages would usually just be an edit to the YAML, not touching the Go code at all.

I think at this point I’ve come pretty close to entirely excavating the space of production languages that might reasonably appear on a Unix system today. Now prove me wrong. Throw a language loccount -s doesn’t already list at me and let’s see if my generic parser and quirks can cope.

Some restrictions: No esolangs, please; nothing with only closed-source implementations; nothing outright extinct; no fad-of-the-week webdev crap; no academic toys only ever touched by the designer’s research group. Exceptions to these restrictions may be made in cases of exceptional historical interest – it amused me to add B and SNOBOL4 and I’d do it again.

Oh, and the current count of languages? 117 The real surprise is that 117 languages was this easy. There is less variation out there than one might suppose.

How not to design a wire protocol

Post Syndicated from Eric Raymond original http://esr.ibiblio.org/?p=8254

A wire protocol is a way to pass data structures or aggregates over a serial channel between different computing environments. At the very lowest level of networking there are bit-level wire protocols to pass around data structures called “bytes”; further up the stack streams of bytes are used to serialize more complex things, starting with numbers and working up to aggregates more conventionally thought of as data structures. The one thing you generally cannot successfully pass over a wire is a memory address, so no pointers.

Designing wire protocols is, like other kinds of engineering, an art that responds to cost gradients. It’s often gotten badly wrong, partly because of clumsy technique but mostly because people have poor intuitions about those cost gradients and optimize for the wrong things. In this post I’m going to write about those cost gradients and how they push towards different regions of the protocol design space.

My authority for writing about this is that I’ve implemented endpoints for nearly two dozen widely varying wire protocols, and designed at least one wire protocol that has to be considered widely deployed and successful by about anybody’s standards. That is the JSON profile used by many location-aware applications to communicate with GPSD and thus deployed on a dizzying number of smartphones and other embedded devices.

I’m writing about this now because I’m contemplating two wire-protocol redesigns. One is of NTPv4, the packet format used to exchange timestamps among cooperating time-service programs. The other is an unnamed new protocol in IETF draft, deployed in prototype in NTPsec and intended to be used for key exchange among NTP daemons authenticating to each other.

Here’s how not to do it…

NTPv4 is a really classic example of one extreme in wire protocol design. A base NTP packet is 48 bytes of packed binary blob that looks like this:

       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      |LI | VN  |Mode |    Stratum    |     Poll      |  Precision    |
      |                         Root Delay                            |
      |                         Root Dispersion                       |
      |                          Reference ID                         |
      |                                                               |
      +                     Reference Timestamp (64)                  +
      |                                                               |
      |                                                               |
      +                      Origin Timestamp (64)                    +
      |                                                               |
      |                                                               |
      +                      Receive Timestamp (64)                   +
      |                                                               |
      |                                                               |
      +                      Transmit Timestamp (64)                  +
      |                                                               |

The numbers are bit widths. If I showed you an actual packet dump it would be a random-looking blob of characters with no significance at the character level; only the bits matter.

It’s not very relevant to this episode what the detailed semantics of those fields are, though you can make some guesses from the names and probably be right; just think of it as a clock sample being passed around. The only two we’re going to care about here is VN, which is a three-bit protocol version field normally set to 0b100 = 4, and mode – three more bits of packet type. Most of the others are interpreted as binary numbers except for “Reference ID”, which is either an IPv4 address or a 4-character string.

Here’s a GPSD report that exemplifies the opposite extreme in wire-protocol design. This is an actual if somewhat old Time-Position-Velocity packet capture from the GPSD documentation:


Those of you with a web-services background will recognize this as a JSON profile.

You don’t have to guess what the principal fields in this report mean; they have tags that tell you. I’ll end the suspense by telling you that “track” is a course bearing and the fields beginning with “e” are 95%-confidence error bars for some of the others. But again, the detailed field semantics don’t matter much to this episode; what we want to do here is focus on the properties of the GPSD protocol itself and how they contrast with NTPv4.

The most obvious difference is discoverability. Unless you know you’re looking at an NTP packet in advance, seeing the data gives you no clue what it means. On the other hand, a GPSD packet is full of meaning to the naked eye even if you’ve never seen one before, and the rest is pretty transparent once you know what the field tags mean.

Another big difference is bit density. Every bit in an NTPv4 packet is significant; you’re squeezing the information into as short a packet as is even theoretically possible. The GPSD packet, on the other hand, has syntactic framing and tags that tell you about itself, not its payload.

These two qualities are diametrically opposed. The bits you spend on making a wire protocol discoverable are bits you’re not spending on payload. That both extremes exist in the world is a clue: it means there’s no one right way to do things, and the cost gradients around wire protocols differ wildly in different deployments.

Before I get to a direct examination of those cost gradients I’m going to point out a couple of other contrasting properties. One is that the base NTPv4 packet has a fixed length locked in; it’s 48 bytes, it’s never going to be anything but 48 bytes, and the 32- or 64-bit precision of the numeric fields can never change. The GPSD packet embodies the opposite choice; on the one hand it is variable-length as the number of decimal digits in the data items change, on the other hand it is quite easy to see how to ship more precision in the GPSD packet if and when it’s available.

Hardware independence is another important difference. A decimal digit string is a decimal digit string; there’s no real ambiguity about how to interpret it, certainly not if you’ve ever seen a JSON-based protocol before. The binary words in an NTPv4 packet, on the other hand, may need to be byte-swapped to turn into local machine words, and the packet itself does not imply what the right decoding is. You need to have prior knowledge that they’re big-endian…and getting this kind of detail wrong (byte-swapping when you shouldn’t, or failing to when you should) is a really notorious defect attractor.

More generally, these protocols differ greatly in two related qualities; extensibility is one. The other doesn’t have a term of art; it’s whether data encoded in the protocol can mix gracefully with other payload types traveling on the same wire. I’ll call it “sociability”.

(And why does sociability matter? One reason is because the friction cost of poking new holes for new protocols in network firewalls is considerable; it triggers security concerns. This is why so much stuff is multiplexed on HTTP port 80 these days; it isn’t only for convenience with browsers.)

Adding a new field to a JSON datagram, or more generally any other kind of self-describing protocol), is not difficult. Even if you’ve never seen JSON before, it’s pretty easy to see how a new field named (say) “acceleration” with a numeric value would fit in. Having different kinds of datagrams on the wire is also no problem because there’s a class field. GPSD actually ships several other reports besides TPV over the same service port.

It’s trickier to see how to do the analogous things with an NTPv4 packet. It is possible, and I’m now going to walk you through some fairly painful details not because they’re so important in themselves but because they illustrate some systematic problems with packed binary protocols in general. There will be no quiz afterwards and you can forget them once you’ve absorbed the general implications.

In fact NTPv4 has an extension-field mechanism, but it depends on a quirk of the transmission path: NTPv4 packets are UDP datagrams and arrive with a length. This gives you a dodge; if you see a length longer than 48 bytes, you can assume the rest is a sequence of extension fields. Here’s what those look like:

       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      |         Type field             |      Payload length          |
      |                                                               |
      |                        Payload (variable)                     |
      |                                                               |

Good luck eyeballing that! It’s simple in concept, but it’s more binary blob. As with the base packet, you need a special tool like wireshark and a detailed spec in front of you just to interpret the type fields, let alone whatever wacky private encodings get invented for the payload parts.

Actually, this last section was partly a fib. Detecting NTPv4 extension fields is tricky because it interacts with a different, older extension – an optional cryptographic signature which can itself have two different lengths:

       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      |                          Key Identifier                       |
      |                                                               |
      |                        dgst (128 or 160)                      |
      |                                                               |

It is possible to work out whether one or both kinds of extension are present by doing some tricky modular arithmetic, but I’ve tortured you enough without getting into exactly how. The thing to take away is that gymnastics are required compared to what it takes to add extensions to a JSON-based protocol, and this isn’t any accident or evidence that NTPv4 is especially ill-designed. This kind of complexity is generic to packed binary protocols, and that has implications we’ll focus in on when we got to cost gradients.

In fact NTPv4 was not badly designed for its time – the Internet protocol design tradition is pretty healthy. I’ve seen (and been forced by standards to implement) much worse. For please-make-it-stop awfulness not much beats, for example, the binary packet protocol used in Marine AIS (Automatic Identification system). One of its packet types, 22 (Channel Management), even has a critical mode bit controlling the interpretation of an address field located after the address field rather than before. That is wrap-a-tire-iron-around-somebody’s-head stupid; it complicates writing a streaming decoder and is certain to attract bugs. By comparison the NTPv4 design is, with all its quirks, quite graceful.

It is also worth noting that we had a narrow escape here. UDP protocols are now unusual, because they have no retransmission guarantees. Under TCP, you don’t get a whole datagram and a length when you read off the network. A TCP equivalent of the NTPv4 packet protocol would either have been fixed at 48 bits no extension forever or have needed to give you some way to compute the expected packet length from data that’s within a minimum-size distance of the start of packet.

JSON evades this whole set of complications by having an unambiguous end delimiter. In general under TCP your packets need to have either that or an early length field. Computing a length from some constellation of mode bits is also available in principle, but it’s asking for trouble. It is…say it with me now…a defect attractor. In fact it took six years after the NTPv4 RFC to issue a correction that clarified the edge cases in combination of crypto-signature and extensions.

What about sociability? The key to it is those version and mode fields. They’re at fixed locations in the packet’s first 32-bit word. We could use them to dispatch among different ways of interpreting everything part those first 8 bits, allowing the field structure and packet length to vary.

NTPv4 does in fact do this. You might actually see two different kinds of packet structure on the wire. The diagram above shows a mode 2 or 3 packet; there’s a mode 6 packet used for control and monitoring that (leaving out an optional authentication trailer) looks like this instead:

       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      |LI | VN  | 6   |R|E|M|  Opcode  |          Sequence            |
      |               Status           |       Association ID         |
      |               Offset           |            Count             |
      |                                                               |
      .                                                               .
      .                        Payload (variable)                     .
      .                                                               .
      |                                                               |

The count field tells you the length of the variable part. Self-description!

Two packet structures, seven potential mode values. You might be wondering what happened to the other five – and in fact this illustrates one of the problems with the small fixed-length fields in packed-binary formats. Here’s the relevant table from RFC5905:

                      | Value | Meaning                  |
                      | 0     | reserved                 |
                      | 1     | symmetric active         |
                      | 2     | symmetric passive        |
                      | 3     | client                   |
                      | 4     | server                   |
                      | 5     | broadcast                |
                      | 6     | NTP control message      |
                      | 7     | reserved for private use |

You don’t have to know the detailed meanings of all of these to get that the mode field mixes information about the packet structure with other control bits. In fact values 1 through 5 all have the same structure, mode 6 has the one I just diagrammed, and all bets are off with mode 7.

When you’re optimizing for highest bit tendency – which is what they were doing in 1985 when this protocol was originally designed – the temptation to do this sort of thing is pretty near irresistible. The result, 34 years later, is that all the bits are taken! We can’t hardly get any more multivalent with this field without committing a backward incompatibility – not a really safe thing to do when there are lots of big-iron legacy implementations still out there, pinned in place by certification requirements and sheer bureaucratic inertia.

OK, in theory we could claim mode 0. But I’ve seen several of the decoders out there and I would not warrant that a mode 0 packet won’t ever slip past anyone’s sanity checks to be misinterpreted as something else. On the other hand, decoders do check the version field; they have to, because versions 0 to 3 have existed and there could be ancient time servers out there still speaking them. So the version field gives us a way out; as long as the version field reads 5, 6, or 7, the rest of the packet part that first byte could look like anything we like and can write an RFC for.

I’ve walked you through this maze to illustrate an important point: packed binary formats are extremely brittle under the pressure of changing requirements. They’re unsociable, difficult to extend without defect-attracting trickery, and eventually you run into hard limits due to the fixed field sizes.

NTP has a serious functional problem that stems directly from this. Its timestamps are 64-bit but only half of that is whole seconds; those counters are going to wrap around in 2036, a couple of years before the more widely anticipated Unix-timestamp turnover in 2038. In theory the existing implementations will cope with this smoothly using more clever modular arithmetic. In practice, anybody who knows enough to have gamed out the possible failure scenarios is nervous…and the more we know the more nervous-making it gets.

This is why I’m thinking about NTPv5 now. 2019 is not too soon. Closing the circle, all this would have been avoided if NTP timestamps had looked like “2010-04-30T11:48:20.10Z”, with variable-length integer and decimal parts, from the beginning. So why wasn’t it done that way?

To address that question let’s start by looking at where the advantages in self-describing textual formats vs. packed binary ones stack up. For self-describing: auditability, hardware independence, extensibility, and sociability. For packed binary: highest possible bit density.

A lot of people would adder “faster, simpler decoding” to the list of advantages for packed binary. But this (at least in the “simpler” part) is exactly where peoples’ design intuitions often start to go wrong, and the history of NTPv4 demonstrates why. Packed protocols start out with “simpler”, but they don’t stay that way. In the general case you always end up doing things like tricky modular arithmetic to get around those fixed limits. You always exhaust your mode-bit space eventually. The “faster” advantage does tend to be stable over time, but the “simpler” does not.

(And oh, boy, will you ever have this lesson pounded home to you if, as I did, you spend a decade on GPSD implementing decoders for at least nineteen different GPS wire protocols.)

If you are a typically arrogant software engineer or EE, you may be thinking at this point “But I’m smarter! And I’ve learned from the past! I can design an optimum-bit-density wire-format that avoids these problems!”

And this is my forty years of field experience, with specific and proven domain expertise in wire-protocol design, telling you: This. Never. Happens. The limitations of that style of protocol are inherent, and they are more binding than you understand. You aren’t smart enough to evade them, I’m not, and nobody else is either.

Which brings us back to the question of why NTPv4 was designed the way it was. And when it is still appropriate to design wire protocols in that packed-binary style. Which means that now it’s time to look at cost gradients and deployment environments.

One clue is that the NTP wire protocol was designed decades ago when computing cycles and bits-per-second on the wire were vastly more expensive than they are now. We can put numbers on that. NTP was designed under the cost profile of early ARPANET to operate well with connection speeds not much higher than 50Kbps. Today (2019) average U.S broadband speeds are 64Mps. That’s a factor of 10^3 difference. Over the same period processor speeds have gone up by about 10^3-10^4. There’s room for argument there based on different performance measures, but assuming the low end of that range we’re still looking at about the same cost change as bits on the wire.

Now let me throw an interesting number at you that I hope brings home the implications of that change. A few weeks ago we at NTPsec had an email conversation with a guy who is running time service out of the National Metrology Institute in Germany. This is undoubtedly Deutschland’s most heavily loaded Stratum 1 NTP provider.

We were able to get his requests per-second figure, do a bit of back-of-the-envelope calculation, and work out that the production NTP load on a a national time authority for the most populous nation in Europe (excluding transcontinental Russia) wouldn’t come even close to maxing out my home broadband or even just one of the Raspberry Pi 3s on the windowsill above my desk. With all six of them and only a modest bandwidth increase I could probably come pretty close to servicing the Stratum 2 sites of the entire planet in a pinch, if only because time service demand per head is so much lower outside North America/Europe/Japan.

Now that I have your attention, let’s look at the fundamentals behind this. That 10^3 drop tracks the change in one kind of protocol cost that is basically thermodynamic. How much power do you have to use, what kind of waste heat do you generate, if you throw enough hardware at your application to handle its expected transaction load? Most of what are normally thought of as infrastructure costs (equipping your data center, etc.) are derivative of that thermodynamic cost. And that is the cost you are minimizing with a packed binary format.

In the case of NTP, we’ve just seen that cost is trivial. The reason for this is instructive and not difficult to work out. It’s because NTP transaction loads per user are exceptionally low. This ain’t streaming video, folks – what it takes to keep two machines synchronized is a 48-byte call and a 48-byte response at intervals which (as I look at a live peers display just now) average about 38 minutes.

There’s just a whisper of a nuance of a hint there that just mmmaybe, three decades after it was first deployed, optimizing NTP for bit density on the wire might not be the most productive use of our effort!

Maybe, in another application with 10^3 more transaction volume per user, or with a 10^3 increase in userbase numbers, we’d incur as much thermodynamic cost as landed on a typical NTP server in 1981, and a packed binary format would make the kind of optimization sense it did then. But that was then, this is now, and peoples’ intuitions about this tend to be grossly out of whack. It’s almost as though a lot of software engineers and EEs who really ought to know better are still living in 1981 even when they weren’t born yet.

OK, so what should we be optimizing NTP for instead? Take a moment to think about this before you keep reading, because the answer is really stupid obvious.

We should be designing to minimize the cost of human attention. Like thermodynamic cost, attention cost unifies a lot of things we normally think of as separate line items. Initial development. Test. Debugging. Coping with the long-term downstream defect rate. Configuring working setups. Troubleshooting not-quite-working setups. And – this is where you should start to hear triumphant music – dealing gracefully with changes in requirements.

It is also a relevant fact that the cost of human attention has not dropped by 10^3 along with thermodynamic cost per unit of information output since 1981. To a good first approximation it has held constant. Labor-hours are labor-hours are labor-hours.

Now let’s review where the advantages of discoverable/textual formats are. Auditability. Hardware independence. Sociability. Extensibility. These are all attention-cost minimizers. They’re, very specifically, enablers of forward design. In the future you’ll always need what a Go player calls “aji” (potential to extend and maneuver). Discoverable textual wire protocols are good at that; packed binary protocols are bad at it.

But I’m absolutely not here to propose a cost model under which discoverability is in a simple linear struggle with bit-density that discoverability always wins in the end. That’s what you might think if you notice that the ratio between attention cost and thermodynamic cost keeps shifting to favor discoverability as thermodynamic cost falls while attention cost stays near constant. But there’s a third factor that our NTP estimation has already called out.

That factor is transaction volume. If you pull that low enough, your thermodynamic costs nearly vanish and packed binary formats look obviously idiotic. That’s where we are with NTP service today. Consequently, my design sketch for NTPv5 is a JSON profile.

On the other hand, suppose you’re running a Google-sized data center, the kind that’s so big you need to site it very near cheap power as though it were an aluminum smelter. Power and heat dissipation are your major running costs; it’s all about the thermodynamics, baby.

Even that kind of deployment, NTP service will still be thermodynamically cheap. But there will be lots of other wire protocols in play that have transaction volumes many orders of magnitude higher…and now you know why protocol buffers, which are sure enough packed binary, are a good idea.

The thought I want to leave you all with is this: to design wire protocols well, you need to know what your cost drivers really are, how their relative magnitudes stack up. And – I’m sorry, but this needs emphasizing – I constantly run into engineers (even very bright and capable ones) whose intuitions about this are spectacularly, ludicrously wrong.

You, dear reader, might be one of them. If it surprised you that a credit-card-sized hobby computer could supply Stratum 1 service for a major First-World country, you are one of them. Time to check your assumptions.

I think I know why people get stuck this way. It’s what Frederic Bastiat called a “things seen versus things not seen” problem in economic estimation. We over-focus on metrics we can visualize, measure, and estimate crisply; thermodynamic costs and the benefits of perfect bit density tends to be like that. Attention costs are squishier and more contingent, it’s more difficult to value options, and it’s especially easy to underestimate the attention cost of having to do a major re-engineering job in the future because the design you’re sketching today is too rigid and didn’t leave you the option to avoid a disruption.

One of the deeper meanings of the quote “Premature optimization is the root of all evil” (often misattributed to Donald Knuth but actually by Tony Hoare), is that you should constantly beware of doing that. Nassem Taleb, the “Black Swan” guy, would rightly call it fragilista behavior, over-planner’s arrogance. In the real world, aji usually beats arrogance – not every time, but that’s the way to bet.

Announcing loccount 2.0 – now up to 74 languages

Post Syndicated from Eric Raymond original http://esr.ibiblio.org/?p=8250

I just released the 2.0 version of loccount.

This is a major release with many new features and upgrades. It’s gone well beyond just being a faster, cleaner, bug-fixed port of David A. Wheeler’s sloccount. The count of supported languages is now up to 74 from sloccount’s 30. But the bigger change is that for 33 of those languages the tool can now deliver a statement count (LLOC = Logical Lines Of Code) as well as opposed to a line count (SLOC = Source Lines of Code, ignoring whitespace and comments)

To go with this, the tool can now perform COCOMO II cost and schedule estimation based on LLOC as well as COCOMO I based on SLOC.

The manual page includes the following cautions:

SLOC/LLOC figures should be used with caution. While they do predict project costs and defect incidence reasonably well, they are not appropriate for use as ‘productivity’ measures; good code is often less bulky than bad code. Comparing SLOC across languages is also dubious, as differing languages can have very different complexity per line.

With these qualifications, SLOC/LLOC does have some other uses. It is quite effective for tracking changes in complexity and attack surface as a codebase evolves over time.

That’s how I’ve used it – to track the drastic reduction in NTPsec’s codebase size. It was also my finger exercise in learning Go. For those of you who enjoy reading code, there are a couple of points of interest…

One is how much of the program’s intelligence is in tables rather than executable code. Adding support for a new language is usually as simple as adding a new table entry and a test load. Some of the internal changes in 2.0 relate to enriching the table format – for example, two things you can now declare are that a language uses the C preprocessor and that C-style backslash escapes are implemented in its string constants. I’ve often advocated this kind of programming by declarative specification in my writings; here you can see it in action.

The contrast with sloccount is extreme; sloccount was clever, but it’s a messy pile of scripting-language hacks that makes adding a new language – or, really, changing anything about it at all – rather difficult. This is probably why it hasn’t been updated since 2004.

Another point of interest is the way the program uses Go’s CSP concurrency primitives. It’s a pattern that could generalize to other ways of data-mining file trees. Walk the tree, spawning a thread to gather stats on each file; each thread writes a summary to one single rendezvous channel; the main thread reads summary blocks off the rendezvous channel and aggregates them into a report. There’s no explicit lock or release anywhere, all the synchronization is supplied by the channel’s atomicity guarantees.

That’s pretty simple, but as a readily accessible demonstration of why CSP rocks compared to conventional mailbox-and-mutex synchronization its very simplicity makes it hard to beat.

Ironically, one of the source languages this tool written in Go cannot deliver LLOC reports for is Go itself. That’s because Go doesn’t have an explicit end-of-statement marker; counting statements would in principle require a full parse. Something might be done with the assumption that the input source in in the canonical form that go fmt produces.

Plans for future releases:

* Beat Go into not barfing on ABC comments? (We have a test load)

* Tell Objective-C .m from MUMPS .m with a verifier?

* sloccount handles different asm comment conventions. We should too.

* How to weight EAF: https://dwheeler.com/sloccount/sloccount.html#cocomo

* Check that we handle every extension in sloccount’s list.

* Detect inline assembler in C? https://en.wikipedia.org/wiki/Inline_assembler.

Here’s the set of languages supported: ada algol60 asm autotools awk c c# c++ c-header clojure clojurescript clu cobol cobra csh css d dart eiffel elisp erlang expect f# fortran fortran03 fortran90 fortran95 go haskell icon java javascript kotlin lex lisp lua m4 makefile ml modula2 modula3 mumps oberon objective-c occam pascal perl php php3 php4 php5 php6 php7 pl/1 pop11 prolog python rebol ruby rust sather scheme scons sed shell simula sql swift tcl verilog vhdl vrml waf yacc.

If something you actually use isn’t on there, send me a code sample with the name of the language in it. The code samples should contain a block comment and a winged comment as well as code comprising at least one full statement. “Hello, world” will do.

NEW – Machine Learning algorithms and model packages now available in AWS Marketplace

Post Syndicated from Shaun Ray original https://aws.amazon.com/blogs/aws/new-machine-learning-algorithms-and-model-packages-now-available-in-aws-marketplace/

At AWS, our mission is to put machine learning in the hands of every developer. That’s why in 2017 we launched Amazon SageMaker. Since then it has become one of the fastest growing services in AWS history, used by thousands of customers globally. Customers using Amazon SageMaker can use optimized algorithms offered in Amazon SageMaker, to run fully-managed MXNet, TensorFlow, PyTorch, and Chainer algorithms, or bring their own algorithms and models. When it comes to building their own machine learning model, many customers spend significant time developing algorithms and models that are solutions to problems that have already been solved.


Introducing Machine Learning in AWS Marketplace

I am pleased to announce the new Machine Learning category of products offered by AWS Marketplace, which includes over 150+ algorithms and model packages, with more coming every day. AWS Marketplace offers a tailored selection for vertical industries like retail (35 products), media (19 products), manufacturing (17 products), HCLS (15 products), and more. Customers can find solutions to critical use cases like breast cancer prediction, lymphoma classifications, hospital readmissions, loan risk prediction, vehicle recognition, retail localizer, botnet attack detection, automotive telematics, motion detection, demand forecasting, and speech recognition.

Customers can search and browse a list of algorithms and model packages in AWS Marketplace. Once customers have subscribed to a machine learning solution, they can deploy it directly from the SageMaker console, a Jupyter Notebook, the SageMaker SDK, or the AWS CLI. Amazon SageMaker protects buyers data by employing security measures such as static scans, network isolation, and runtime monitoring.

The intellectual property of sellers on the AWS Marketplace is protected by encrypting the algorithms and model package artifacts in transit and at rest, using secure (SSL) connections for communications, and ensuring role based access for deployment of artifacts. AWS provides a secure way for the sellers to monetize their work with a frictionless self-service process to publish their algorithms and model packages.


Machine Learning category in Action

Having tried to build my own models in the past, I sure am excited about this feature. After browsing through the available algorithms and model packages from AWS Marketplace, I’ve decided to try the Deep Vision vehicle recognition model, published by Deep Vision AI. This model will allow us to identify the make, model and type of car from a set of uploaded images. You could use this model for insurance claims, online car sales, and vehicle identification in your business.

I continue to subscribe and accept the default options of recommended instance type and region. I read and accept the subscription contract, and I am ready to get started with our model.

My subscription is listed in the Amazon SageMaker console and is ready to use. Deploying the model with Amazon SageMaker is the same as any other model package, I complete the steps in this guide to create and deploy our endpoint.

With our endpoint deployed I can start asking the model questions. In this case I will be using a single image of a car; the model is trained to detect the model, maker, and year information from any angle. First, I will start off with a Volvo XC70 and see what results I get:


{'result': [{'mmy': {'make': 'Volvo', 'score': 0.97, 'model': 'Xc70', 'year': '2016-2016'}, 'bbox': {'top': 146, 'left': 50, 'right': 1596, 'bottom': 813}, 'View': 'Front Left View'}]}

My model has detected the make, model and year correctly for the supplied image. I was recently on holiday in the UK and stayed with a relative who had a McLaren 570s supercar. The thought that crossed my mind as the gulf-wing doors opened for the first time and I was about to be sitting in the car, was how much it would cost for the insurance excess if things went wrong! Quite apt for our use case today.


{'result': [{'mmy': {'make': 'Mclaren', 'score': 0.95, 'model': '570S', 'year': '2016-2017'}, 'bbox': {'top': 195, 'left': 126, 'right': 757, 'bottom': 494}, 'View': 'Front Right View'}]}

The score (0.95) measures how confident the model is that the result is right. The range of the score is 0.0 to 1.0. My score is extremely accurate for the McLaren car, with the make, model and year all correct. Impressive results for a relatively rare type of car on the road. I test a few more cars given to me by the launch team who are excitedly looking over my shoulder and now it’s time to wrap up.

Within ten minutes, I have been able to choose a model package, deploy an endpoint and accurately detect the make, model and year of vehicles, with no data scientists, expensive GPU’s for training or writing any code. You can be sure I will be subscribing to a whole lot more of these models from AWS Marketplace throughout re:Invent week and trying to solve other use cases in less than 15 minutes!

Access for the machine learning category in AWS Marketplace can be achieved through the Amazon SageMaker console, or directly through AWS Marketplace itself. Once an algorithm or model has been successfully subscribed to, it is accessible via the console, SDK, and AWS CLI. Algorithms and models from the AWS Marketplace can be deployed just like any other model or algorithm, by selecting the AWS Marketplace option as your package source. Once you have chosen an algorithm or model, you can deploy it to Amazon SageMaker by following this guide.


Availability & Pricing

Customers pay a subscription fee for the use of an algorithm or model package and the AWS resource fee. AWS Marketplace provides a consolidated monthly bill for all purchased subscriptions.

At launch, AWS Marketplace for Machine Learning includes algorithms and models from Deep Vision AI Inc, Knowledgent, RocketML, Sensifai, Cloudwick Technologies, Persistent Systems, Modjoul, H2Oai Inc, Figure Eight [Crowdflower], Intel Corporation, AWS Gluon Model Zoos, and more with new sellers being added regularly. If you are interested in selling machine learning algorithms and model packages, please reach out to [email protected]