The Sceptical Competent
April 23rd, 2013 by ravi

The Sceptical Competent

Warning and apologies: the train metaphor employed within might drive you off the rails.

Software development has hidden hierarchies that play out not so much in titles or skills as in echo chambers. A person working up the programming ladder is corralled into pre-approved positions by the din of expert opinion repeated many times over in text books, blog posts, Twitter, conferences and sundry outlets. It is only on occasions when there is a clash of propaganda or when a glimpse of history raises question about the grand narratives, that the adrift novice coder begins to question the emperor’s attire. The smarter among such novices quickly learn to swim with the current; or whatever happens to be current. The rest stay adrift, unable to reconcile the reality of the sausage making with the extravagant theorising about it.

What’s my point?

The Expert Beginner

Consider Erik Dietrich’s insightful blog post about the The Rise of the Expert Beginner, which can be summarised as the reasonable thesis that somewhere on the path from Novice through Advanced Beginner, Competent, Proficient on to Expert, a developer can get sidetracked into believing that as an Advanced Beginner she is in fact an Expert. This prevents her, the argument goes, from ever graduating to the “intuitive” “grasp[ing] of the big picture” that marks an Expert:

As such, Advanced Beginners can break one of two ways: they can move to Competent and start to grasp the big picture and their place in it, or they can ‘graduate’ to Expert Beginner by assuming that they’ve graduated to Expert.

What’s the block? The premature self-congratulation of the Expert Beginner and his/her ignorance of the work that remains:

The Expert Beginner has nowhere to go because progression requires an understanding that he has a lot of work to do, and that is not a readily available conclusion. You’ll notice that the Expert Beginner is positioned slightly above Advanced Beginner but not on the level of Competence. This is because he is not competent enough to grasp the big picture and recognise the irony of his situation, but he is slightly more competent than the Advanced Beginner due mainly to, well, extensive practice being a Beginner.

Dietrich warns that this reasoning isn’t “as ridiculous as it sounds”, but the precaution is unnecessary, since the thesis is not ridiculous at all, even at first blush. In some sense, a large number of workers (in any field) are stuck at this Expert Beginner level and the above seems a very natural and sufficient explanation of their predicament. I can imagine a few thousand HTML+CSS+jQuery+PHP programmers nodding their head in dawning realisation and agreement. “He nailed it”, they (at least the honest ones) might be saying to themselves, “for here I am sitting on my false plateau of being able to churn out any web app by ad hoc hacking together the things I know well, but with a gnawing sense that my code is categorically different from that of celebrated experts”.

And what is the way out of this false peak?

But what happens when the Advanced Beginner doesn’t care enough to interact with the broader community and for whatever reason doesn’t have much interaction with peers? […] They fail even while convinced that the failure is everyone else’s fault, and the nature of the game is such that blaming others is easy and handy to relieve any cognitive dissonance.

And this above is my point of departure from Dietrich’s analysis. In fact, I am afraid, what I write below is going to sound very much like exactly the sort of “blaming others” in response to “cognitive dissonance” that Dietrich fingers. By the end of this essay though, I hope you will see, if I am right, that I am speaking of a very different kind of cognitive dissonance, and that my alternate version of the trials of a novice can sit alongside Dietrich’s and demand equal attention.

Cognitive Dissonance

Let us look at the landscape of the Advanced Beginner of Dietrich’s chart. Recall that per Dietrich this person can advance to being Competent (by working harder) or atrophy or plateau out at Expert Beginner (due to a false sense of confidence and sufficiency). The plateau I do not disagree with, but how a person gets there is a lot more complicated than mere overconfidence.

Often, what lies behind the Advanced Beginner is the tidy and simple world of beginner’s books and online guides. And the one thing that all such material have in common is that they eschew real world considerations and complexity in favour of describing language and tool primitives (simply put, “Hello World”++). No mention there of the performance implications of dynamic memory allocation or JavaScript event handling, the scattershot implementations of the HTTP protocol or the minefield of browser compatibility.

Stepping off this swift train the beginner lands knee deep in the vagaries of real world code — incomplete frameworks, inconsistent behaviours, poor or non-existent documentation, bugs, so on — and faces not only a path of hard work leading to possible expertise, but also a cacophony of quasi-religious positions, often garnished with a generous sprinkling of ridicule. This unfortunately is the “interact[ion] with the broader community” that Dietrich prescribes as an antidote to Expert Beginner hubris. It is not a pleasant terrain and there are few or no guides.

It is a world where the Advanced Beginner is upbraided for using tables rather than pure CSS for layout but also one where he finds the proposed alternatives are a set of hacks.

It is a world where he is browbeaten into speaking the language of MVC, and Factories, and Decorators, Patterns, Anti-Patterns, and what not, even as his suspicion grows that an enormous amount of rock solid code was written in the last 40 years that required none of these paradigms, of necessity, and that these ideas do not replace previous simpler, intuitive ones with the force (or beauty and promise of pleasure) with which an O(log(n)) algorithm can replace an O(n^2) algorithm.

It is a world of pointless language battles where the huge installed base and real world success of a C or a Perl or a PHP is pitted against or trumped by the abstractions and purity of preferred alternatives (most often though not always “functional programming languages”, but that is no fault of those languages themselves or the many fine men and women promoting them sans ridicule).

It is a world where he finds that graduating from jQuery (as prescribed) to AngularJS involves, inter alia, the questionable magic of dependency injection. Or a world where one must have separation of presentation and application logic (often revealingly called “business” logic), a pronouncement made simultaneous with promoting HTML templating engines that mix user interface specification (presentation or view) with bits of non-modular code almost always written in a custom DSL (domain-specific language).

This is merely a small taste of that world, or larger community that the beginner has to interact with (there is worse in fact: if the beginner happens to be a woman, she has to deal with condescension, discrimination and other varieties of sexism — but that is a larger issue better left for separate consideration).

Fight or Flight

Is there any surprise that cognitive dissonance and retreat to safe ground is the result for any Advanced Beginner who is incapable of self-preservingly consuming some flavour of Kool-Aid (in the hope of choosing an eventual winner of the propaganda wars)?

If she is lucky she will come across something like Dietrich’s post, or a good Steve Yegge rant or the measured thoughts of Reg Braithwaite, all of which might present a less dogmatic view into the world of Experts than the one held out in the grandiose promises of myriad languages, methodologies and frameworks (and I have not even touched on Agile, or TDD religionism, which she is bound to be bludgeoned with in the startup, and these days giant corporation, that just hired her). She might learn to filter the hype from value, to square the cognitive dissonance brought on by the gritty realities of actually using sanctioned tools, languages, frameworks, methodologies, style guides, edicts, and all else.

But more likely, she will find herself in the flame infested echo chambers of blogs and link “sharing” sites where the Competents and Proficients buff their egos by parroting in snide fashion the criticisms of Experts. Or poring over “Advanced Programming” and “Scalable Application Development with [X]” tomes (where X is one of the currently fashionable platforms) that aren’t going to fly off virtual shelves if they were to give due coverage to the flaws and limitations of the subject matter.

The Rise of the Sceptical Competent

If my suspicion is right, there exists a significant group of Advanced Beginners, who are good enough at what they do, who are smart enough to learn new ideas, and who are committed enough to work hard, but who are just unable to get on board some happy train such as this one:

  • JavaScript, that’s a scripting language! Meant for browser tricks. Not for real men.
  • JavaScript has really Good Parts, it is a functional language, really, wrapped in an enigma in a cipher. Lookee here, functions as first-class objects!
  • Wat, your language doesn’t do proper closures? Ha! Mine will have Namespaces any minute.
  • NodeJS is awesome (it is, actually) and async is the future (it was the past, really; check out your old copy of Stevens). You just don’t understand async, that’s all.
  • Oh noes, the pyramid of doom! Nested callback hell!
  • EventEmitters really. That’s the ticket.
  • Promises to the rescue!
  • NPM FTW! (Quit whining about multiple versions of modules. Oh also you don’t need to modify your include path at run-time so we removed that capability)

And it thunders on, that train, at a breathless and breakneck speed towards the next set of miracles that will expose and overcome the weaknesses of the current celebrated and flawless ones (disclaimer: this is not a rant against the authors of each of these ideas, without whose work we would have no progress at all).

What it leaves behind is not only Expert Beginners, but a class of Advanced Beginners turned Sceptical Competents. Developers who know what they are doing, who can solve problems often in elegant and simple ways, but who are wary of the possibility that while the train might be built by Experts, it is driven by Buzzword Proficients (you can tell this type by their “principled” refusal to answer a straightforward question on mailing lists and StackOverflow, because you just shouldn’t be trying to do that). These Sceptical Competents are the same people who sat out (or reluctantly participated in, for need of paycheck) CORBA as the one true solution, XML as the one true solution, Java as the one true solution, and other assorted ideologies of the past. And today they see big gaping holes in the NoSQL as one true data store solution, AWS as one true hosting solution, MVC as one true software structure solution, Web apps as the one true platform solution, A/B testing as the one true validation solution, Hadoop as the one true “big data” solution, and other contemporary stories.

The cognitive dissonance harboured by this class of developers can be productive, and even crucial, in every iteration of progress. Out of this group of unwillings will emerge a few who will be the next round of Experts — the next implementors of REST to counter the entrenched SOAP, the next Node+Express+etc developers to the entrenched WebSpheres (or whatever giant systems are in use for web services of which I am blissfully unaware), the next Agile (the current Waterfall) to (the old) Waterfall, and much more.

Leave a Reply

Leave a Reply

Your email address will not be published.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

If you need help with one of my software projects, please click: Using that link, you might even find a previous report of (and solution for) the issue!
»  Substance: WordPress  »  Style: Ahren Ahimsa