Why cybersecurity certifications suck

Post Syndicated from Robert Graham original http://blog.erratasec.com/2016/10/why-cybersecurity-certifications-suck.html

Here’s a sample question from a GIAC certification test. It demonstrates why such tests suck.

The important deep knowledge you should know about traceroute how it send packets with increasing TTLs to trace the route.

But that’s not what the question is asking. Instead, it’s asking superfluous information about the default behavior, namely about Linux defaults. It’s a trivia test, not a knowledge test. If you’ve recently studied the subject, your course book probably tells you that Linux traceroute defaults to UDP packets on transmit. So, those who study for the test will do well on the question.

But those with either a lot of deep knowledge or practical experience will find this question harder. Windows and Linux use different defaults (Windows uses ICMP ECHOs, Linux uses UDP). Personally, I’m not sure which is which (well, I am now, ’cause I looked it up, but I’m likely to forget it again soon, because it’s a relatively unimportant detail).

Those with deep learning have another problem with the word “protocol”. This question uses “protocol” in one sense, where only UDP, TCP, and ICMP are valid “protocols”.

But the word can be used in another sense, where “Echo” and “TTL” are also valid “protocols”. A protocol is a set of rules that govern things. Thus we say phrases like “slow start protocol” for how TCP handles initial congestion, even though this “protocol” has no protocol header or particular fields. In much the same way, TTL is a “protocol” or “set of rules” for handling routing loops that traceroute exploits. That Linux uses the TTL protocol when transmitting packets is a perfectly valid answer to this question, albeit not the conventional one.

Exams suck because those writing the exams themselves often lack experience and deep knowledge. They are only one short step ahead of their students.

This leaves such test prejudiced toward those who have recent read (and who are likely soon to forget) a textbook. The tests are prejudiced against those who the tests are intended to highlight, those with experience and deep knowledge.

I’m not really trying to beat up on the GIAC tests here. I’m simply demonstrating the problem in our industry. We want to be able to certify people like doctors and lawyers, real “professions” where if things go wrong, people’s lives can be ruined. We are far from that. All certification tests are entry-level only. Our trade has not existed long enough to become a full trustworthy “profession”.