Category Archives: Ranting

All systems suck

I’ve been thinking a lot about this idea lately.  I’ve spent a lot of years as an engineer and consultant fixing other people’s systems that suck, writing my own systems that suck, and working on legacy systems, that, well, suck.

Don’t let anyone fool you.  All systems suck, to a greater or lesser extent.

If it’s an old system, there’s the part of the code that everybody is afraid to work on: the fragile code that is easier to replace than maintain or refactor.  Sometimes this seems hard, or nobody really understands it.  These parts of the code are almost always surrounded by an SEP field.  If you’re unfamiliar with the term, it means “Somebody Else’s Problem”.  Items with an SEP field are completely invisible to the average human.

New systems have the parts that haven’t been built yet, so you’ll hear things like “This will be so awesome once we build feature X”.  That sucks.

There’s also the prototype that made it into production, a common problem.  Something somebody knocked together over a weekend, whether it was because of lack of time, or because of their utter brilliance, is probably going to suck in ways you just haven’t worked out yet.

All systems, old and crufty or new and shiny, have bottlenecks, where a bottleneck is defined as the slow part, the part that will break first when the system is under excessive load.  This is also part of your system that sucks.

If someone claims their system has no bugs, I have news for you: their system sucks.  And they are overly optimistic (or naive).  (Possibly they just suck as an engineer, too.)

In our heads as engineers we have the Platonic Form of a system: the system that doesn’t suck, that never breaks, that runs perfectly and quietly without anyone looking at it.  We work tirelessly to make our systems approach that system.

Even if you produce this Platonically perfect system, it will begin to suck as soon as you release it.  As data grows and changes, there will start to be parts of the system that don’t work right, or that don’t work fast enough.  Users will find ways to make your system suck in ways you hadn’t even anticipated.  When you need to add features to your perfect system, they will detract from its perfection, and make it suck more.

Here’s the punchline: sucking is like scaling.  You just have to keep on top of it, keep fixing and refactoring and improving and rewriting as you go.  Sometimes you can manage the suck in a linear fashion with bug fixes and refactoring, and sometimes you need a phase change where you re-do parts or all of the system to recover from suckiness.

This mess is what makes engineering and ops different from pure mathematics.  Embrace the suck.  It’s what gets me up in the mornings.

Books change the world

Last week I read a tweet that really got my goat, so much so that I stewed on it all weekend. The author, who is someone from the tech/startup community, said, to Tim O’Reilly no less:

“No-one ever changed the world by writing books.”

This pushed my rant button.

I thought about mentioning some books that have changed the face of civilization. Religious books: the Bible, which defines the shape of many Western civilizations, and the equivalent books of religious law in other cultures. Science books: Copernicus’ “On the Revolutions of the Celestial Spheres”, which began the scientific revolution, and defined a heliocentric model of the universe. Newton’s “Philosophiæ Naturalis Principia Mathematica”, which outlines classical mechanics and gravitation. Einstein’s multitude of publications. Books on economics: Keynes, anyone? Feminism: Friedan’s “Feminine Mystique”. Political thought. Philosophy. Need I go on?

On a micro- level, think about a book you read as a child, as a teenager, last year, last week, that changed the way you felt, gave you hope, gave you relaxation that you needed, or an escape from an unpleasant reality.

I could go on about world-changing books all day. Instead, I’m going to tell you a story about a very unexciting book. This book happens to be one that I wrote, on a subject that I’m passionate about. It’s a book on web development.

Now, this book is only a technical book. It won’t start any revolutions or cause any epiphanies, neither will it make you laugh or cry. My family won’t ever read it, and when non-technical people who are excited to discover I’m a published author hear the topic, their faces fall. It will never be on the New York Times list, or on Oprah, or be banned in countries with oppressive governments. It is a humble technical book.

This technical book has, however, sold quite well over the years. Many people have bought it (thank you), some have liked it, and some have reviewed it. Copies are in libraries, used as the prescribed texts in colleges, sold secondhand, and pirated as PDFs on the internet.

Hundreds of thousands of people have read this book, and, I hope, learned a little something about coding for the web.

Some of those people have probably gotten jobs as a result. Some might have graduated college. Some have built a personal website. Some might have gotten a promotion.

Out of those people, I venture that there have to be a hundred, perhaps more, perhaps less, who have started a company that does some kind of web development, whether it’s a consulting company or a startup. Maybe some of those companies got funded, maybe some were bootstrapped, maybe some were successful.

I wonder if that benchmark is something that the author of the tweet might value.

I hope it’s not too arrogant as an author to hope these things: that the books you write change someone’s life for the better, and in doing so change the world. I continue to believe this, and that is why I continue to write.

Abysmal customer service

Recently I have been the recipient of a number of incidents of abysmal customer service and I really need to vent.

The worst of all is Bank of America.  They are as extraordinarily polite as they are incompetent.

On a number of occasions they have stopped my card.  Reasons for this have included:
  – You were spending a lot of money (it was Christmas Eve)
  – Someone used your card in another state (I live in Maryland, 20 minutes drive from Virginia malls).
  – You spent money in another country.

The last seems reasonable in some sense, so I have now taken to calling them before travel and advising of travel.   I have also on several occasions asked them to cease this practice as I travel a lot.  They always agree that this is fine and that they have made a note on my record, but it has never had any noticeable effect.  On my last trip home to Australia in September they stopped my card again.  After calling them – at least half an hour on the phone, at international rates – they assured me it had been unblocked.  I went out shopping and found it still blocked.  I repeated this process AGAIN and it was still blocked.  It was not until the third call that I managed to get the card unblocked.  On each occasion they assured me the problem was solved.  Overall I spent about three hours on the phone at international rates. 

I wrote an email to complain and was told they would send a physical letter to explain what had happened.  Such a letter never arrived.

Last Monday, 11/26, while en route to California, I misplaced my card.  Believing I had lost it in the airport, I rang to report it lost.   I spent 25 minutes on the phone and spoke to three different individuals.  The final one told me she could not suspend the card for some reason that she didn’t mention and that I would have to call back some other time.

I did not call back, hoping the card would turn up.  It did, on Friday 11/30, and then they promptly suspended it.  I emailed them to ask if it could be unsuspended, and was told that a new card had been issued and that I would receive it by mail 12/3.  Today 12/5 it had still not turned up, so I emailed again and they told me it was *mailed* 12/3.

On some level it is my own fault for continuing to bank with them, a problem which I will shortly remedy.

However, I am particularly grumpy with customer service assistants this week.

Number 2 was the gate agent at United in SJC on Friday afternoon.  After she twice stopped serving me to help someone else – and gave us each the wrong boarding passes – I politely asked if she could finish serving me before moving on.  She threw the boarding pass at me and it hit me in the face.  Charming.  I said nothing but took my pass and walked away.

Number 3, today I was on the phone to a doctor’s secretary.  She could not find me in the computer and then gave me a lecture on the fact that I was mispronouncing my own name.  If I would only pronounce it the way she did, instead of the way my entire family does, then people would be better able to assist me!

Vent over.  Will resume normal programming shortly.

Losing my religion

You’re over at a friend’s house – let’s call him Bob – and after opening a beer he invites you out to the garage to see something.  On the wall hangs a reciprocating saw.  You notice that there are no other tools visible, which seems kind of weird.  Bob says “You like it?  That reciprocating saw is the best tool ever.  When I got it, I really liked it, and I joined the local reciprocating saw users’ group.  They showed me all the cool things I could do with it, and I realized I didn’t need any other cutting tools.  I even use it to slice my cheese!”

What’s wrong with us? 

I am frequently irritated with the mindset that there’s One True Way of solving any kind of software problem, be it web platform, database choice, operating system, or methodological approach.  It’s been irritating me since I was an academic and I would present two different algorithms (let’s call them A and B) to solve problem X.  There would always be a student who wanted to know "which is better?"  Typically I would respond "In a situation such as […] A is better, but if you are looking at something more like […] B is better."  Most people would be happy with that but there are always people who insist that one must just be The Best Way.

So, reasons for choosing, or sticking with, a particular technology/methodology/whatever:
– It solves the problem you need to solve.
– You have a lot invested in it and there is not a sufficient business case to switch.
– It’s easier to hire people with the appropriate skills / your staff have this set of skills and you can’t just fire them all
– It matches your mindset, or, to go back to the hardware analogy, it fits your hand.
– It has good community / support .

There are very very few absolutes in choosing tools, save that whatever tool you choose should be of basically decent quality.  I have my biases like everybody else, but it’s important to realize when you have a bias and acknowledge the effect that has on your decision making.

In particular, telling another engineer that all their problems would be solved if they would just switch from technology A to technology B is often the worst kind of cultish thinking.  I’m talking about the kind of discussion you see on mailing lists, on IRC, or from some people in person where you say "I have this problem.  I’m using technology A and…" at which point someone butts in and says "Switch to B." when they know nothing about your problem.

(Having said this of course, there are some exceptions, in particular to do with upgrading versions and using basically dead technologies.  So sue me 😉 )

Do all frameworks really suck?

Ahem.  So anybody that’s known me for a while has likely heard me say something similar to what is quoted in Cal’s article about OSCON.  I should clarify because I hear jokes about "Tell us what you really think".

Choosing a framework to implement your web app is a trade off like any other design decision.  Let’s focus in on specifics and talk about what the trade off is that you make when you choose a framework.  Specifically, I’m talking about MVC frameworks in PHP.

Good things

  • Frameworks provide a common method of code organization, so developers can both a) get up to speed fast, and b) don’t have to think hard about how to architect an app.  They are effectively a standard for application architecture.  This is particularly useful when working with large dev teams or junior devs.
  • Frameworks encourage the separation of the presentation layer from the business logic, avoiding a frequent PHP worst practice where stuff is all mixed in together.
  • A framework can encourage secure coding through the use of dispatch architectures.
  • In general, frameworks avoid spaghetti coding.

Bad things

  • MVC is a design pattern.  As frequently mentioned by the good ol’ Gang of Four, every implementation of a design pattern is different, depending on the specific viewpoint of the implementor, and the specific application we are trying to produce.  This causes two problems.  One is that the implementor’s viewpoint is not necessarily the same as mine.  The second is that trying to shoehorn every app into the MVC structure isn’t always appropriate.
  • Most MVC frameworks have an intentionally flat design – models, views, controllers – and when codebases grow, you need to modularize for maintainability.  There are different ways to do this, but many frameworks don’t lend themselves well to this.
  • In the world of PHP, as with Perl, there is More Than One Way To Do It.  Specifically with frameworks, I believe Luke has been known to say there are 2.3 frameworks per PHP developer.  They are like content management systems or blogging systems.  We’ve all done it, sad to say.  The downside of this is that you lose a lot of the programmer speedup if programmers have to learn a new framework on every project.
  • Bloat is a problem in a lot of frameworks.  That’s what "makes the magic happen", but typically using a framework means lots of files getting opened (required/included) behind the scenes.  This slows down your app.  See for example Paul M. Jones’ (updated) benchmarks.  (That, by the way, is an excellent, excellent article that displays a good methodology for researching design decisions.)  I’ll also refer here to what I sometimes jokingly call Thomson’s first rule of software design: First, do the simplest thing that could possibly work.
  • It’s virtually impossible to retrofit an MVC framework on to existing code.  A lot of us spend most of our careers dealing with existing applications.

In summary:
Let’s be clear here: I am not recommending people write spaghetti code, or that they embed HTML willy- nilly in their PHP.  My recommendation in making any kind of architectural decision is to know what tradeoffs you are making and make an educated decision.  It’s important to remember that you can follow some of the basic rules of MVC and get a good number of the benefits without the bloat.  It’s equally important to remember that there is more than one way to architect a web app.

I’ll try and blog in future about a couple of other related topics: MVC in Rails compared to MVC frameworks in PHP, and templating systems.  (Unlike frameworks, all templating systems really do suck 😉 )

It had to happen.

I think I must be the only person in the PHP blogosphere who hasn’t said something about Ruby On Rails.  It seemed about time 🙂

We’ve been working for a while on a RoR project.  Originally it came in as a really high concept idea – the spec was more like a film treatment than a functional spec.  It came from the mind of an old friend of one of our biggest clients – he’s a high concept guy, who’s been amazingly successful with his out there ideas in the past.

So we set out to build a prototype in six weeks.  We are known as a PHP shop, although of our web dev, about half is in PHP and the other half in Perl.  At the time we had working for us a very talented Rails guru, so we thought we’d give it a try. 

Times change: the prototype turned into a production website, our Rails guru moved on to a pure Rails shop, and we now support this Rails app (served from lighttpd against PostgreSQL on Solaris).

Things I have learned about Rails:
– It makes getting started easy.  This is what it’s famous for.
– We had to do quite a bit of work to get everything working happily on Solaris (as part of our migration to doing most of our web dev in ZFS zones).  This included building custom packages of Ruby 1.8 on Solaris 10 resulting in numerous patches to Ruby’s build and configuration processes and building custom lighttpd for Linux and then Solaris 10.  We developed numerous patches addressing unexplained crashes, errors on large file uploads, and improper use of timezone localization.

– As things get more complicated, the ideal of convention over configuration starts to get in your way.  In several cases we have obtained huge performance improvements (up to twenty times) by hacking around the conventions.
– The documentation sucks.  This is however, improving.  I remember when I first started learning the language and platform there was an initiative on the Rails website to pay someone to write some, "because all the people that really know Rails are busy building apps or writing books about it".  This is indicative of another issue, namely:
– The community support is really immature.  Something I know as a long time user of lots of different Open Source technologies is that a good community can make or break a product.  For example, for a long time the MySQL community was much more supportive of new users than the PostgreSQL community.  I love Postgres and have been using it for years, long before I ever touched MySQL or wrote any books about it (although I have tech edited a Postgres book).  I am really pleased to say that the Postgres community has become seriously good in the last year or so.
– Learning to debug apps is really really hard to begin with.  All the magic that goes on makes it hard for the newbie to figure out what went wrong.  How I wish I had known more about script/console at the beginning. 
– Because of the state of the documentation, I bought a few Rails and Ruby books.  The first one I read was the Agile guide.  I found it interesting how this book is aimed at Java programmers.  All the arguments for switching to RoR are arguments for switching from Java.  It never occurred to me that people would be doing that.  This is a very readable book but I need more depth.
– What I want is a book on debugging, maintaining, and scaling RoR apps.  Recommendations welcome.  Worst comes to worst I guess we might have to write one.

The project is possibly the weirdest thing that’s ever been built with Rails – the functionality at its most basic is very very straightforward, but as my colleague put it "This site violates pretty much every rule of web UI design. The content
on the site is not information as much as it is a bunch of people
arguing about current events and the righteousness of public opinion….if you can survive the outright attack on the senses, it is clever and punchy on both the concept level and technical level".   That sums it up pretty well.

It’s been and continues to be an interesting journey.  The site is visible at

Footnote: Just as a note on the design – it’s entirely true to the client’s original vision and has been an interesting challenge for the UI designers (who have also been responsible for sites such as (our original designer, Amy Hoy) and (today’s designer, Dai Swengler).

The number one rule.

Warning: This post contains many occurrences of the word "arse".  ("ass" if you are American)  If you’ll be offended, you should stop reading now.

So, in my talk at kiwifoo I talked about hiring and keeping good staff.  I had a slide that read:
1.  Don’t be an arseclown.
2.  Don’t hire arseclowns.

I was really stealing from George, who said once that the best thing about OmniTI was that we didn’t employ any asshats.  I have always preferred the term arseclown myself, from first having heard it in Office Space to Roy and HG‘s telephone poll on The Dream ("Is Arseclown A Real Word?  Vote Yes or No") and their then nomination of the Arseclown of the Day.

I wandered into Borders and what should I see but this: "The No Asshole Rule: Building a Civilized Workplace and Surviving one that isn’t." by Bob Sutton.  You can even take the Am I an Asshole? quiz online via Guy Kawasaki‘s blog.

P.S. Don’t worry, one of the first things in the book is a note that different cultures use different words: asshole == arsehole == arseclown. 

more on indicators: Little Notes

If there are any repeat readers (hi! and welcome back!) you might remember my post on the T-shirt index.

I finally got around to reading Freakonomics, which was recommended to me by somebody who read that blog post.  It was damn entertaining, and had me nodding my head and grinning most of the way through, even though I was trapped on a 14 hour flight at the time.

I realized there are other indicators that I use as a mental barometer of Where Things Stand.  I’ll try and blog about these on a more regular basis.  The first of these, which I was discussing with Wez on Friday, is the Little Notes Indicator.

The Little Notes Indicator can be seen in action in any place where a largish group of people must share communal space.  I have seen it in the following places: shared houses and apartments, workplaces, community spaces, and agistment (livery for the English and boarding for the Americans) places for horses.

In all of these places you will find a note or two that one person has left for other people.  We have one in the kitchen at work.  It says, "Please put your dishes in the dishwasher."  This is completely reasonable and similar signs are found in virtually every shared kitchen I have ever used.    A single Little Note is normal.

When Little Notes proliferate, it is a sign that communication has broken down.  If each day is greeted with a new little note (and possibly a new rule for sharing space to go with it) it shows that people have stopped speaking and just entered a period of resentment.  In shared houses this is usually an indicator of the imminent demise of the sharing; in workplaces an indicator of low morale among staff.

When Little Notes start having phrases added to them such as "or else" or when they begin to appear in capitals ("OR YOU’RE OUT") you know you are in trouble.  Other signs of trouble are increase in frequency, the use of colored ink, bigger signs, signs on top of signs, or people writing addendums to printed signs.

One place I used to visit had new signs each day which were circulated so that each individual had their own copy.  They were written in ALL CAPS, and always peppered with phrases such as ‘OR YOU WILL BE EVICTED’ etc.  Each sign inevitably followed somebody doing something not even particularly annoying that was previously not against the rules.  Once things start to snowball, there’s no going back.

Please God let me never be the author of such a missive. </rant>

Argh. Apache vs Skype

So I’m relatively new to Skype.  Love it, great free service, blah yada blah.  Great for talking to the family at home.

It never occurred to me that when Apache started complaining that it couldn’t bind to port 80 that Skype had hijacked the port for its own foul purposes.  ("In Skype, we don’t share ours with anyone" for anyone that is familiar with that joke.)   Anyway, problem is now solved (quit Skype, apache -k install, start Skype).