Everyone uses Java/C# and Oracle, why choose something else?

If the tech guy (or girl) advises going for a less common technology, and something goes wrong, he’ll get blamed. “Why didn’t you use Java/Oracle/IBM like everyone else does?

And you can be sure that something WILL go wrong sooner or later… and this guy knows it.

He knows he has a mortgage, kids that have to go through college, a wife with a spending habit… why take any chances? What’s in it for him if it works out? And what if it doesn’t?

People avoid taking risks. Especially people who work a ‘safe’ job as an employee.

Is this the lizard brain at work?

For the record: There’s nothing wrong with Java/C# (or any other language for that matter). They all have their (dis)advantages on speed, memory usage, developer-friendliness, license/price…

Yet NONE of them fit all tasks and situations you can throw at them, while yes: you can use most of them to implement most tasks.

Do you have any idea how many programming languages exist?

Just look at this list. These languages weren’t developed because there’s hardly any difference between them, or because a ‘one size fits all’ language exists.

Now, if you are a developer/sysadmin, I probably needn’t explain. But as I’m hoping to enlighten the non-tech manager here, I think an analogy is in order.

Let’s say you’re building a house. Would you use nails or screws for all attaching situations?

No. Depending on the kind of materials you need to bond, the needed strength, the climate it’s in, the budget, how long it should last… you either use nails, screws, glue, cement, or god knows what else.

You use the right tool for the job. It’s the same with programming languages.

A project done in Java will cost 5 times as much, take twice as long, and be harder to maintain than a project done in a scripting language such as PHP or Perl. … But the programmers and managers using Java will feel good about themselves because they are using a tool that, in theory, has a lot of power for handling problems of tremendous complexity. Just like the suburbanite who drives his SUV to the 7-11 on a paved road but feels good because in theory he could climb a 45-degree dirt slope.

Philip Greenspun, Java is the SUV of programming tools

(I just used this quote to show that there can be some bias and strong opinions concerning certain languages. I also don’t like Java, but that’s beside the point.)

Yes, you can hammer a screw into wood (I tried this as a kid), but it will take a lot of effort and will provide less bonding than a nail. And you might hit your fingers in the process (I also did this as a kid).

Still not convinced? Do you know that some programming languages can be 100 times faster than others? And there are also other differences just as relevant as execution speed.

And though execution speed can be important, it shouldn’t be your guiding factor.

Often people, especially computer engineers, focus on the machines. They think, “By doing this, the machine will run fast. By doing this, the machine will run more effectively. By doing this, the machine will something something something.” They are focusing on machines. But in fact we need to focus on humans, on how humans care about doing programming or operating the application of the machines. We are the masters. They are the slaves.

Yukihiro Matsumoto, Ruby programming language on Wikipedia

People sometimes dismiss Ruby because Ruby is slow. Well it is, and I don’t care. The biggest performance bottlenecks that I have day to day is the database, the file system, the internet and bugs. These are not problems with Ruby, so speeding it up will have a negligible effect.

Additionally, hardware is cheap and developers are not. If you can use a programming language like Ruby to make your programmers more productive, then you can increase the performance of your code with better hardware. You can buy the better hardware with the money you saved by making your developers more productive.

Graham Jenson, What is Ruby? It is fun and makes you happy!

I once was part of a team that had to make an internal web application. We didn’t have any Java experience, but the company already had some customer-facing Java web apps developed by the ‘webdev’ team. So we agreed to use Java.

After evaluating several Java frameworks, one of us bumped into this new thing called ‘Ruby on Rails’ (V0.11 at that time iirc). ‘Rails’, as it’s often called, was like a breath of fresh air after being bombarded with Java that seemed more like configuration than programming at that time.

There was only 1 problem: Rails uses Ruby instead of Java as its programming language.

Even so, we decided to go for it. We finished the project in considerably less time than the original estimate. This also helped management accept that we had neglected to use Java.

Was this all because Rails somehow made us work faster?

Not really, but it was very suited for this job, it was a really fun language (it still is), it was new and exciting, and because we were supposed to use Java, our asses were on the line in a major way.

It was the right tool for the job AND the right tool for us. By the way, last time I heard, this application was still running (I no longer work there).

And when we showed the hardcore Java aficionados, one comment really struck: “it’s easy that way.” And this wasn’t the last time I heard this (often concerning Ruby). And shouldn’t that exactly be the whole point? Making things easier?

Unfortunately, when technology is concerned, the decisions aren’t always that clear and we rely a lot on other (less?) relevant factors to guide us.

Why your devs might not like a language that makes their lives easier

When I started out as a pro programmer, the company I worked for made client software in a not-so-easy language/framework. Why? Primary reason: they wanted the software to be as fast as possible, which is very valid.

Unfortunately, the database technology we were using was so damn slow that any advantage the faster language offered was totally obliterated. This was a huge bottleneck. The language was like a Ferrari, but it was waiting 98% of the time for the database’s traffic light. And that light won’t switch to green faster just because there’s a Ferrari waiting.

(For the tech guys: we were using C++/MFC and an Access database through ODBC… ok, stop laughing, we eventually switched to faster database tech.)

It also took several minutes to compile the project, meaning that we couldn’t just make changes and test them right away. No, we had to wait several minutes to test things, or get an error (and in the old days, that could have been several hours, but only graybeards remember this).

When I suggested that another language/technology (Delphi) was really more suited for the job, colleagues argued: “there’s no demand for Delphi programmers, this isn’t good for us.

They were right. For me, this was my first job and I wanted to make a good impression. However, they had already proven themselves, knew this wasn’t going to be their last job and didn’t want to lock themselves into a framework/language that wasn’t in demand.

And I have no idea if they would have been able to find Delphi programmers.

Besides that, we could be pretty sure that Microsoft (the manufacturer of the tool we used) would still be around in a couple of years… while Borland (the original developer of Delphi) always seemed to be hovering on the edge of bankruptcy.

Would we have been more productive using Delphi? I’m pretty sure we would have. It was just much more suited for what we were doing and more forgiving for the rather average developers we were.

Would it have been good for the company? Not if the devs ran away and couldn’t be replaced by others.

Would it have been good for me? In hindsight, no. Now I can put 8 years of ‘pro’ C++ development on my CV, which is a lot more impressive compared to Delphi/Pascal. A lot of people used Pascal in high school, so it’s often not taken seriously.

So how do you pick a suitable language/framework?

You should use what startups use!

That’s what I used to tell the CxO’s. And I was wrong, as most of them didn’t work for startups.

Tech startups are the hotbed of technology. They use cutting-edge tools and make amazing software that scales to millions of users. And you will have to look hard to find startups using Java/C#.

Why?

Technology is at the core of their business, and their goal is to grow fast or fail fast – which probably isn’t yours. You are probably looking for stability, and aren’t interested in failing, no matter how slow or fast it goes.

You’re part of a startup? You should have brighter guys on board than me who will tell you what language/framework to use. And pssst: don’t waste your time reading this, you’ve got a startup to grow!

So, should you just stick with classic, proven tech then?

Hell no!

The language and framework you use can make a real difference. I think that except for certain edge cases, the human impact far outweighs the technological impact.

Have your programmers experiment with alternatives. Even if they don’t get all excited and suggest ditching your current tools, they might just find them useful somewhere. Also, it will help to fight boreout and keep things interesting.

Depending on the kind of software you make, it might be worth investigating domain specific languages (DSL).

This is something that could motivate the seniors and benefit everyone.

I have worked in EDI in the past, and DSL’s would be perfect there. Of course, I’d do it in Ruby, but I’m biased.

But isn’t it hard to learn a new language?

The basic logical building blocks of all languages are more or less the same. To compare programmers to writers again: a good programmer can tell a solid, logical story, no matter what language or framework (s)he uses.

Regular learning will also make it easier to learn new things in the future. I like to think of this as keeping your ‘learning muscle’ in shape.

And it will certainly not hinder career opportunities to experiment with other tools.
Au contraire mon frère.

Finally, it will also help you attract new talent. A business that uses Java, but with some Ruby, PHP and Node.js where it makes sense, sounds a lot sexier to the kind of people you want (well, it does to me).

Some have even suggested using technology as a hiring filter.

“The programmers you’ll be able to hire to work on a Java project won’t be as smart as the ones you could get to work on a project written in Python. And the quality of your hackers probably matters more than the language you choose. Though, frankly, the fact that good hackers prefer Python to Java should tell you something about the relative merits of those languages.”

Paul Graham, Great Hackers

But don’t take my word (or Paul’s) for all this. Your mileage will vary. There are too many different situations to provide a ready-made answer. I hope I still gave you some food for thought and provided some insights though.

If there’s any meager advice I could give it would be this:

  • Talk to your developers
  • Are they using the right tools for the job?
  • Encourage experimentation. Without experimentation we’d still be using assembler (or gasp: Cobol!).
  • Have them investigate DSL’s.
  • Remember that this isn’t purely a tech/HR matter.

And know that something will always go wrong, and it often isn’t the technologies’ fault.

Had a good read? Sign up at here to get notified of future (dis)informative articles by email.

Agree? disagree? Let me know your thoughts in the comments.

Header image by Duncan Hull, used under CC BY 2.0.

Your Developers Aren’t Bricklayers, They’re Writers

If you have ten programmers, the best one is probably at least five times better compared to the worst one. No shit.

Define better: he works faster, produces less bugs and writes more readable, logical and maintainable code.

Programmers aren’t bricklayers or carpenters, but they are often treated as such. (not that there’s something wrong with these occupations)
“Why would I get a senior when I can get two juniors for the same price?”
“This feature takes three months for one programmer? Let’s just put two more on it so we can have it in a month.”

And why is that? Because there’s no decent, fool-proof way to measure the productivity of a programmer, I guess? Whatever can’t be measured, gets ignored. I’ve seen it too many times. [Tweet]

Also: would you rather have two juniors deliver your baby, fix your car, perform a lumbar puncture, or would you rather have one senior? Despite popular belief, the act of producing code is not a commodity. [Tweet]

The best programmers are up to 28 times better than the worst programmers, according to “individual differences” research. Given that their pay is never commensurate, they are the biggest bargains in the software field.

Robert Glass, Facts and Fallacies of Software Engineering

If you really want to compare, then I’d say programmers are more like writers.

Some writers can produce a page turner that goes on to sell millions, while another might produce something that is so boring it extinguishes the fire it’s thrown in!
Yet, they both produced a book and thus they’re both writers, and you wouldn’t know the difference unless you’d really start reading the book.

Programming managers have long recognized wide productivity variations between good programmers and poor ones. But the actual measured magnitudes have astounded all of us. In one of their studies, Sackman, Erickson, and Grant were measuring performance of a group of experienced programmers. Within just this group the ratios between the best and worst performances averaged about 10:1 on productivity measurements and an amazing 5:1 on program speed and space measurements!

Fred Brooks, The Mythical Man-Month

Let me give you an actual (yes, IRL) example

(Names have been changed to protect the innocent.)
The Company (no, not the CIA, just some software company) needs to implement a new module in their flag product. Mr. Lousy is free at the moment; great, let him start right away!

After four months of tinkering, Mr. Lousy is finally finished. The QA team finds loads of bugs and inconsistencies that need to be reproduced and reported back to Mr. Lousy. He then spends two weeks fixing those bugs and finally delivers a new version! Those are damn annoying bugs and Mr. Lousy often thinks about them while he stares into the distance.
This test/fix cycle is repeated two or three times.

The users have been promised this new module. Sales will be able to sign with some new customers that have been waiting for this module. There’s considerable pressure to just get it out there and it gets incorporated in the next release. Hurray! Champagne for everyone!

Unfortunately, the new module is still rather buggy. Customers are excellent at finding previously undetected bugs. So they contact support. The support team tries to find out what’s going wrong, whether the customer’s confused about the feature, whether they’re confused themselves, or whether it’s just a bug and can be reproduced. Yada, yada, yada. They decide to file a bug report. Unfortunately, Mr. Lousy is wreaking havoc—oh, I mean he is working on another project at this time, and he can’t just drop everything to fix them pestering bugs.

(And we haven’t even talked about whether the code is maintainable, logical and documented. Because sooner or later someone else will need to make enhancements, add features, etc.)

But, uh-oh…there are some bugs that Mr. Lousy can’t seem to fix. Also, new bugs turn up and no one is sure if they are indeed new, or just weren’t discovered before. Support gets pestered. Sales are getting annoyed, because those new customers find the module is too unreliable for day-to-day work and want to cancel their contract!

And, finally, the boss asks Mr. Rockstar Developer to take a look at the code.

Mr. Rockstar isn’t really able to make heads or tails of it and sees lots of senseless constructions. He advises to rewrite the code, thinking it’ll take about a month. He gets to work and hands over the finished module in just a few days longer than his original estimate (he’s Mr. Rockstar, not Mr. Perfect). Besides some minor bugs that are quickly detected by the QA team, everything works as expected. The QA team starts to worry: if they’d all be like him, they’d be out of a job. Mr. Lousy thinks that Mr. Rockstar is an arrogant twit, but this isn’t really relevant.

The revised module gets released. Everybody is happy because it finally works. Hurray!
Congrats! Rejoice! Champagne again!

By this time, it’s usually forgotten that Mr. Rockstar managed to do in about one month what Mr. Lousy couldn’t in 7-8 months.

The huge difference between these two developers only surfaced here because they performed the exact same task.

By writing less code that does more, and by writing maintainable code that has fewer bugs, a good developer takes pressure off of the QA staff, coworkers, and management, increasing productivity for everyone around. This is why numbers such as 28 times productivity are possible and might even seem low when you look at the big picture.

Phil Haack, 10 Developers For The Price Of One

Now, what is the worst that could happen here?

Mr. Rockstar eventually notices that he’s considerably more productive than Mr. Lousy. And, don’t be fooled, he doesn’t need to get the same task to find out. This is his job; he’s damn good at it and he easily recognizes bad code written by a bad programmer. Still, he’s pretty sure (or absolutely sure) that Mr. Lousy is earning more or less the same amount of pay. He gets disgruntled and/or he quits. And you’ve just lost a considerable amount of production power for your development team.

Even if he doesn’t quit, how do you think he’ll feel when a release/project is running late and the whole team is ‘punished’ with overtime? This is the inevitable waiting to happen.

If you’re really out of luck, he’ll be replaced by another Mr. Lousy.

So, why does this go unnoticed so often?

Because it’s easier to ignore this issue than to fix it—at least up to a certain point, and also because no one likes to rat on colleagues and be the cause of someone losing his job; because Mr. Lousy has three kids and just bought a new house; and, most importantly, because it’s very hard to measure a developer’s productivity, unless you’re really able to compare them both doing the same task as in the above story.

Are you a developer? Do you agree? Please let me know in the comments. Almost all non-tech people think ‘one developer day’ is an exact measurement. [Tweet]

Now, I’ve been thinking a lot about this: measuring a developer’s productivity. I’ve often been jealous of salesmen because their remuneration is based on their accomplishments. I’ve got some ideas (still ripening) on how to separate the wheat from the chaff. I’ve also been thinking about and will be posting about how to identify, attract and retain the Mr. Rockstar developers.

Sign up here to get notified about these and future posts via email.

Related articles:
10 Developers For The Price Of One
What Makes Bad Programmers Different?
What Makes Great Programmers Different?
Discussion on StackExchange: A good programmer can be as 10X times more productive than a mediocre one

Who am I and why am I telling you this

I’ve worked as a professional developer (that is, getting paid for it during a daytime job) for over 10 years. I made my first commercial product (a game) in 1989. Although this didn’t make me rich, it still felt great (I was 16 at the time). A couple of years later, I sold the idea of one of my games, and it eventually got released on the Nintendo Gameboy (and other formats) by Bandai! I can tell you from experience that it’s very satisfying to see something you thought up (including the name) actually being sold in a shop.

In 2008, I took a non-tech job at a very technology-driven company. I wanted to learn about running a business in general, about the sales process,… everything non-tech that happens in business. Well, my tech chops probably got me hired and are still very useful, but they are no longer an essential part of the job.

I don’t program for a living anymore (or rather: at the moment), but often tinker with various new technologies in after-hours projects. To me, it’s the same as reading a good book—exciting and relaxing at the same time.

In my current role, I often encounter people who express certain misconceptions or who lack basic knowledge of the development process. In their role (often CxO), this basic knowledge could make a huge difference.

Enlightening them isn’t on our agendas during these encounters and I usually just leave it at that. When appropriate, with the CxOs who I know on a more personal level, I’ve tried to convince them of their incorrect or incomplete view of the software development process (usually to no avail).

Now, because I like to think via writing, and I really think I can provide value here, I’m making a series of blog posts about the various clichés and misconceptions I’ve encountered in this industry.

Sign up here for more enjoyable, insightful and (dis)informative posts!

Header photo by Moresheth, used under CC BY 2.0 / Cropped from original

Generating a twitter OAuth access key – the semi-manual way

[UPDATE]
Apparently someone at Twitter was listening, or I’m going senile/blind. Let’s call it a combination of both.

Instead of following all the steps below, you could just login with the Twitter account you want to use on http://dev.twitter.com, register your application and then click ‘Edit Details’ on the application overview page at http://dev.twitter.com/apps. Next click the ‘Application detail’ button on the right, followed by the ‘My Access Token’ button in order to get your Access Token and Access Token Secret.

This makes the old post below rather obsolete. Clearly a case of me thinking everything is a nail and ruby is a hammer (don’t they usually say this about java coders?) 😉

[ORIGINAL POST]

OAuth is great! OAuth allows your application to use your user’s data without the need to ask for their password. So Twitter made the API much safer for their and your users. Hurray! Free pizza for everyone!

Unless of course you’re using the Twitter API for your own needs like running your own bot and don’t need access to other user’s data. In such cases a simple username/password combination is more than enough. I can understand however that the Twitter guys don’t really care that much about these exceptions(?). Most such uses for the API are probably rather spammy in nature.

!!! If you have a twitter app that uses the API to access external user’s data: look for another solution. This solution is ONLY meant when you ONLY need access to your own account(s) through the API.

Other Solutions

Mr Dallas Devries posted a solution here which involves requesting and scraping a one-time PIN.
But: I like to minimize the amount of calls I make to twitter’s API or pages to lessen my chances of meeting the fail whale. Also, as soon as the pin isn’t included in a div called anymore, this will fail.

However, mr Devries’ post was a starting point for my solution, so I’m much obliged to him posting his findings.

Authenticating with the Twitter API: old vs new

Acessing The Twitter API the old way:

require 'twitter'
httpauth = Twitter::HTTPAuth.new('my_account','my_secret_password')
client = Twitter::Base.new(httpauth)
client.update('Hurray!')

The OAuth way:

require 'twitter'
oauth = Twitter::OAuth.new('ve4whatafuzzksaMQKjoI', 'KliketyklikspQ6qYALcuNandsomemored8pQ6qYALIG7mbEQY')
oauth.authorize_from_access('123-owhfmeyAgfozdyt5hDeprSevsWmPo5rVeroGfsthis', 'fGiinCdqtehMeehiddenymDeAsasaawgGeryye8amh')
client = Twitter::Base.new(oauth)
client.update('Hurray!')

In the above case,
ve4whatafuzzksaMQKjoI is the ‘consumer key’ (sometimes also referred to as ‘consumer token’) and
KliketyklikspQ6qYALcuNandsomemored8pQ6qYALIG7mbEQY is the ‘consumer secret’. You’ll get these from Twitter when you register your app.

123-owhfmeyAgfozdyt5hDeprSevsWmPo5rVeroGfsthis is the ‘access token’ and
fGiinCdqtehMeehiddenymDeAsasaawgGeryye8amh is the ‘access secret’. This combination gives the registered application access to your account. I’ll show you how to obtain these by following the steps below.

(Basically you’ll need a bunch of keys and you’ll have to jump a bit through hoops to obtain them for your server/bot. )

How to get these keys

1. Surf to the twitter apps registration page

go to http://dev.twitter.com/apps to register your app. Login with your twitter account.

2. Register your application

Enter something for Application name, Description, website,… as I said: they make you jump through hoops.

If you plan on using the API to post tweets, Your application name and website will be used in the ‘5 minutes ago via…’ line below your tweet. You could use the this to point to a page with info about your bot, or maybe it’s useful for SEO purposes.

For application type I choose ‘browser’ and entered http://www.hadermann.be/callback as a ‘Callback URL’. This url returns a 404 error, which is ideal because after giving our account access to our ‘application’ (step 6), it will redirect to this url with an ‘oauth_token’ and ‘oauth_verifier’ in the url. We need to get these from the url. It doesn’t really matter what you enter here though, you could leave it blank because you need to explicitely specify it when generating a request token.

You probably want read&write access so set this at ‘Default Access type’.

3. Get your consumer key and consumer secret

On the next page, copy/paste your ‘consumer key’ and ‘consumer secret’. You’ll need these later on. You also need these as part of the authentication in your script later on:

oauth = Twitter::OAuth.new([consumer key], [consumer secret])
4. Obtain your request token

run the following in IRB to obtain your ‘request token’
Replace my fake consumer key and consumer secret with the one you obtained in step 3. And use something else instead http://www.hadermann.be/callback: although this will only give a 404, you shouldn’t trust me.

irb(main):001:0> require 'oauth'
irb(main):002:0> c = OAuth::Consumer.new('ve4whatafuzzksaMQKjoI',
'KliketyklikspQ6qYALcuNandsomemored8pQ6qYALIG7mbEQY',
{:site => 'http://twitter.com'})
irb(main):003:0> request_token = c.get_request_token(:oauth_callback => 'http://www.hadermann.be/callback')
irb(main):004:0> request_token.token
=> "UrperqaukeWsWt3IAlfbxzyBUFpwWIcWkHP94QH2C1"

This (UrperqaukeWsWt3IAlfbxzyBUFpwWIcWkHP94QH2C1) is the request token: Copy/paste this token, you will need this next.

5. Authorize your application

surf to https://api.twitter.com/oauth/authorize?oauth_token=[the above token], for example:

https://api.twitter.com/oauth/authorize?oauth_token=UrperqaukeWsWt3IAlfbxzyBUFpwWIcWkHP94QH2C1

This will bring you to the ‘An application would like to connect to your account’- screen on Twitter where you can grant access to the app you just registered. If you aren’t still logged in, you need to login first. Click ‘Allow’. Unless you don’t trust yourself.

6. Get your oauth_verifier from the redirected url

Your browser will be redirected to your callback url, with an oauth_token and oauth_verifier parameter appended. You’ll need the oauth_verifier.

In my case the browser redirected to:

http://www.hadermann.be/callback?oauth_token=UrperqaukeWsWt3IAlfbxzyBUFpwWIcWkHP94QH2C1&oauth_verifier=waoOhKo8orpaqvQe6rVi5fti4ejr8hPeZrTewyeag

Which returned a 404, giving me the chance to copy/paste my oauth_verifier: waoOhKo8orpaqvQe6rVi5fti4ejr8hPeZrTewyeag

7. Request an access token

Back to irb, use the oauth_verifier to request an access token, as follows:

irb(main):005:0> at = request_token.get_access_token(:oauth_verifier => 'waoOhKo8orpaqvQe6rVi5fti4ejr8hPeZrTewyeag')
irb(main):006:0> at.params[:oauth_token]
=> "123-owhfmeyAgfozdyt5hDeprSevsWmPo5rVeroGfsthis"
irb(main):007:0> at.params[:oauth_token_secret]
=> "fGiinCdqtehMeehiddenymDeAsasaawgGeryye8amh"

We’re there!

123-owhfmeyAgfozdyt5hDeprSevsWmPo5rVeroGfsthis is the access token.
fGiinCdqtehMeehiddenymDeAsasaawgGeryye8amh is the access secret.

Try it!

Try the following to post an update:

require 'twitter'
oauth = Twitter::OAuth.new('ve4whatafuzzksaMQKjoI', 'KliketyklikspQ6qYALcuNandsomemored8pQ6qYALIG7mbEQY')
oauth.authorize_from_access('123-owhfmeyAgfozdyt5hDeprSevsWmPo5rVeroGfsthis', 'fGiinCdqtehMeehiddenymDeAsasaawgGeryye8amh')
client = Twitter::Base.new(oauth)
client.update('Cowabunga!')

Now you can go to your twitter page and delete the tweet if you want to.

MODx character encoding

Ahhh character encodings. Don’t you just love them?

Having character issues in MODx? Then probably the MODx manager character encoding, the character encoding of the site itself, your database’s character encoding, or the encoding MODx/php uses to talk to MySQL isn’t correct.

The Website Encoding

Your MODx site’s character encoding can be configured in the manager under Tools/Configuration/Site/Character encoding. This is the encoding your website’s visitors will get.

The Manager’s Encoding

The manager’s encoding can be changed by setting $modx_manager_charset at manager/includes/lang/<language>.inc.php like this (for example):

$modx_manager_charset = 'iso-8859-1';

To find out what language you’re using (and thus was file you need to change), check Tools/Configuration/Site/Language (1 line above the character encoding setting).

This needs to be the same encoding as your site. You can’t have your manager in utf8 and your site in iso-8859-1.

Your Database’s Encoding

The charset MODx/php uses to talk to your database can be set by changing $database_connection_charset in manager/includes/config.inc.php. This needs to be the same as your database’s charset. Make sure you use the correct corresponding charset, for iso-8859-1 you need to use ‘latin1′. Utf8 is just utf8.

Example:

$database_connection_charset = 'latin1';

Now, if you check Reports/System info, the ‘Database Charset’ might say something else. This is because the mysql variable ‘character_set_database’ is displayed here, which contains the character set used by the default database and not the one for the current database/connection.

However, if you’d change this to display ‘character_set_connection’, it could still say something else because the ‘set character set’ statement used by MODx doesn’t change this value either. The ‘set names’ statement does, but since it turns out my MODx install works now as expected I’ll just leave it at this before I get a headache.

If I saved you a potential headache or you think I’m totally wrong or overlooked something, let me know in the comments.

btw: I want to be able to use a real editor with MODx. Somehow.

The hidden cost of interrupting knowledge workers

The November issue of pragpub has an interesting article on interruptions. The article is written by Brian Tarbox, who also mentions the article on his blog. I like the subtitle: ‘Simple Strategies for Avoiding Dumping Your Mental Stack’.

Brian talks about the effective cost of interrupting a ‘knowledge worker’, often with trivial questions or distractions. In the eyes of the interruptor, the interruption only costs the time the interrupted had to listen to the question and give an answer. However, depending on what the interrupted was doing at the time, getting fully immersed in their task again might take up to 15-20 minutes. Enough interruptions might even cause a knowledge worker to mentally call it a day.

According to this article interruptions can consume about 28% of a knowledge worker’s time, translating in a $588 billion loss for US companies each year.

Looking for a new developer to join your team? Ever thought about optimizing your team’s environment and the way they work instead?

Making non knowledge workers aware

You can’t. Well, I haven’t succeeded yet. And believe me: I’ve tried. When you’ve got a simple way to really increase your productivity (‘give me 2 hours of uninterrupted time a day’) it wouldn’t be right not to tell your boss or team-leader about it.

The problem is: only productive knowledge workers seem to understand this. People who don’t fall into this category just seem to think you’re joking, being arrogant or anti-social when you tell them the interruptions can really have an impact on your productivity.

Also, knowledge workers often work in a very concentrated mental state which is described here as:

It is the same mindfulness as ecstatic lovemaking, the merging of two into a fluidly harmonious one. The hallmark of flow is a feeling of spontaneous joy, even rapture, while performing a task.

Yes, coding can be addictive and if you’re interrupting a programmer at the wrong moment, you’re effectively bringing down a junkie from his high in just a few seconds. This can result in seemingly arrogant, almost aggressive reactions.

How to make people aware of the production-cost they’re inflicting: I’ve been often pondering that question myself. The article suggests that solutions based on that question never seem to work. To be honest: I’ve never even been able to find a half decent solution for this question. People who are not in this situations just don’t understand the issue, no matter how you try to explain it.

Fun (?) thing I’ve noticed: Programmers or IT people in general who don’t get this are often the kind of people who just don’t get anything done.

Interrupt handling (interruption management?) IRL

Have non-urgent questions handled in a non-interruptive way

It helps a bit to educate people into using non-interruptive ways to ask questions: “duh, I have no idea, but I’m a bit busy here now could you put it in an email so I don’t forget?”. Eventually, a considerable amount of people will skip interrupting you and just send an email right away. Some stubborn-headed people however will continue to just interrupt you, saying “you’re 10 meters from my desk, why can’t we just talk?”.

Just remember to disable your email notifications, it can be hard to resist opening your email client when you know a new email just arrived.

Use Do Not Disturb signals

When working in a group of programmers, often the unofficial sign you can only be interrupted for something important is to put on headphones. And when the environment is quiet enough, often people aren’t even listening to music. Otherwise music can help to block the indirect distractions (someone else talking on the phone or tapping their feet). You might get a “they’re all just surfing and listening to music”-reaction from outsiders though.

Peopleware talks about a team where the no-interruption sign was placing a shawl on the desk. If I remember correctly, I am unable to locate my copy of this really excellent must-read book.
If you have all standardized on the same IM tool, maybe that tool has a ‘do not disturb’ setting. Also some phone-systems have a ‘DND’ (do not disturb) setting.

Hide

Brian offers a number of good suggestions, some obvious like: hide away somewhere they can’t find you. Not sure how long it’ll be till someone thinks you’re just taking a nap somewhere though. Also, this often isn’t possible or your boss might not understand this. And if you really get caught taking a nap, make sure to explain that your were powernapping.

Counter-act interruptions

Another suggestion he offers is when you’re being interrupted to just hold up your hand, blocking the interruption, and at least giving you time to finish your sentence or your block/line of code. The last suggestion works more as a way to make it obvious to the interruptor that they really are interrupting your work and to offload some of the cost on the interruptor. In practice, this can also helps you cool down a bit so you don’t start saying nasty things to the interruptor.

Unfortunately I’ve sometimes been confronted with people who just ignore this signal and keep talking, as if they’re sure that whatever they’ve got to say is really worth listening to and without a doubt more important than anything you might be doing.
This behaviour usually leaves me speechless (not good when someone just asked a question). I’ve noticed that these people are usually also the first to complain when being interrupted themselves. They’re generally not very liked as colleagues, so try not to imitate their behaviour.

TDD as a way to minimize recovery time

I don’t like Test Driven Development. Mainly for only one reason: It interrupts flow. At least, that’s what it does for me, but maybe I’m just not grown used to TDD yet.

BUT a positive effect TDD has on me when I have to work in an interruptive environment and can’t really get into the ‘flow’ (also supposedly called ‘the zone’ by software developers, although I’ve never heard it 1st hand), TDD helps me to concentrate on the tasks at hand and helps me to get back at work after an interruption. I feel when using TDD, I can get by without the need for being totally ‘in’ the project and I can be reasonably productive without obtaining ‘flow’.


Do you have a suggestion on how to make people aware of the concept of ‘flow’ and the cost of interruptions? (without looking like an arrogant ass or a weirdo)

Twitter gem – undefined method `stringify_keys’

Have you been getting the following errors when running the Twitter gem lately ?


/usr/local/lib/ruby/gems/1.8/gems/httparty-0.4.3/lib/httparty/response.rb:15:in `send': undefined method `stringify_keys' for # (NoMethodError)
from /usr/local/lib/ruby/gems/1.8/gems/httparty-0.4.3/lib/httparty/response.rb:15:in `method_missing'
from /usr/local/lib/ruby/gems/1.8/gems/mash-0.0.3/lib/mash.rb:131:in `deep_update'
from /usr/local/lib/ruby/gems/1.8/gems/mash-0.0.3/lib/mash.rb:50:in `initialize'
from /usr/local/lib/ruby/gems/1.8/gems/twitter-0.6.13/lib/twitter/search.rb:101:in `new'
from /usr/local/lib/ruby/gems/1.8/gems/twitter-0.6.13/lib/twitter/search.rb:101:in `fetch'
from test.rb:26

It’s because Twitter has been sending back plain text errors that are treated as a string instead of json and can’t be properly ‘Mashed’ by the Twitter gem. Also check http://github.com/jnunemaker/twitter/issues#issue/6.

Without diving into the bowels of the Twitter gem or HTTParty, you could ‘begin…rescue’ this error and try again in 5 minutes. I fixed it by overriding the offending code to return nil and checking for a nil response as follows:

module Twitter
  class Search
    def fetch(force=false)

      if @fetch.nil? || force
        query = @query.dup
        query[:q] = query[:q].join(' ')
        query[:format] = 'json' #This line is the hack and whole reason we're monkey-patching at all.

        response = self.class.get('http://search.twitter.com/search', :query => query, :format => :json)

        #Our patch: response should be a Hash. If it isnt, return nil.
        return nil if response.class != Hash

        @fetch = Mash.new(response)
      end

      @fetch
    end

  end
end

(adapted from http://github.com/jnunemaker/twitter/issues#issue/9)

If you have a better solution: speak up!

WordPress hack might affect more than your WordPress installation!

Just a quick note…

If you’re managing WordPress installations, and haven’t heard about the recent WordPress hacks (http://wordpress.org/support/topic/307660) yet you probably just got back from holiday.

There’s plenty of advice on fixing a hacked installation if you follow some of the links in the above post, so I’m not going to recite that here.

However! While cleaning a WordPress installation that resides in a subdirectory of a modx cms I found that the modx index.php files were also infected. I searched for infected files using grep as follows:

grep -R 'function gpc_' *

If you’re on a shared hosting environment that doesn’t allow shell access: time to change your hosting provider. (mail me if you need recommendations)

SWFObject and IE6 causing hair-pulling agony

I recently used SWFObject to display a flash header on a website. I chose SWFObject because:

  • Instead of displaying an annoying ‘Install flash now’ message, it claims to be able to show alternate content. In this case: the original header image.
  • It claims to be compatible with more or less every browser out there.

Implementation went fine, until someone tested it on IE6 and got the following error:

Internet explorer cannot open the Internet site http://www…..
Operation aborted

Which basically means that the site just can’t be visited with IE6 (still used a lot in business environments), it even seems as if there’s something wrong with your internet connection.

Now, since about 10% of visitors to this site are still using IE6 (why does everyone still use Internet Explorer ???? Do YOU know that these days most people do NOT use Internet Explorer anymore ?)

Now after some googling, I found the suggestion to defer loading of the SWFObject.js as follows:

<script type="text/javascript" defer="defer" src="http://ajax.googleapis.com/ajax/libs/swfobject/2.2/swfobject.js">
</script>
<script type="text/javascript" defer="defer">
swfobject.registerObject("myId", "9", "");
</script>

What this does according to W3C: When set, this boolean attribute provides a hint to the user agent that the script is not going to generate any document content (e.g., no “document.write” in javascript) and thus, the user agent can continue parsing and rendering.
I don’t know exactly why, but: HURRAY! It works now!!! Only… IE6 and IE7 (didn’t try IE8) now gave the following error:


Line: 19
Char: 1
Error: ‘swfobject’ is undefined
Code: 0
URL: http://www…

But the flash was still running fine. Still, such an error isn’t clean, especially since almost half of the site’s visitors are using one of these Internet Explorer versions.

Now, wanting a quick fix I decided to do the following:

<script type="text/javascript" defer="defer">
if (typeof(swfobject) != "undefined")
swfobject.registerObject("myId", "9", "");
</script>

I admit this is a bit of a weird ‘fix’. You’d suspect the flash to stop working on IE6/IE7, which it doesn’t. Not planning on diving into it’s inner bowels, I regard this a ‘mission accomplished’ until someone somewhere posts a better solution (for which I setup some Google alerts).

Do you have a better solution?
What would be the impact on the webdev economy (or your life) if all browsers were compatible?

Addendum

Because the above turned out not to work with the new Firefox 3.5.3 (strangely, was OK with 3.5.2 when I tested it) I decided to cut the crap and use the ‘Dynamic Publishing’ way. Ok, so it won’t work for people who have javascript disabled, but who on earth would have flash installed AND javascript disabled?

To avoid the IE6 error with the ‘Dynamic Publishing’ way, I call swfobject.embedSWF right after the div that will be replaced with the flash content. Calling swfobject.embedSWF in the <head> would otherwise give me the above error in IE6 again.

Twitter traffic might not be what it seems

Are you using bit.ly stats to measure interest in the links you post on twitter?

I’ve been hearing for a while about people claiming to get the majority of their traffic originating from twitter these days.

Now, I’ve been playing with the twitter ruby gem recently, doing various experiments which I’ll not go into detail here because they could be regarded as spamming… if I’d conduct them on a large scale, that is.
It’s scary to see people actually engaging with @replies crafted with some regular expressions and eliza-like trickery on status updates found using the twitter api. I’m wondering how Twitter is going to contain the coming spam-flood.

When posting links I used bit.ly as url shortener, since this one seems to be the de-facto standard on twitter. A nice thing about bit.ly is that it shows some basic stats about the redirects it performs for your shortened links.

To my surprise, most links posted almost immediately resulted in several visitors. Now, seeing that I was posting the links together with some information concerning what the link is about, I concluded that the people who were actually clicking the links should be very targeted visitors.
This felt a bit like free adwords, and I suddenly started to understand why everyone was raving about getting traffic from twitter.

How wrong I was! (and I think several 1000 online marketers with me)

On the destination site I used a traffic logging solution that works by including a little javascript snippet in your pages. It seemed that somehow all visitors disappeared after the bit.ly redirect and before getting to the site, because I was hardly seeing any visitors there. So I started investigating what was happening: by looking at the logfiles of the destination site, and by making my own ‘shortened’ urls by doing redirects using a very short domain name I own. This way, I could check the apache access_log before the redirects.

Most user agents turned out to be bots without a doubt. Here’s an excerpt of user-agents awk’ed from apache’s access_log for a time period of about one hour, right after posting some links:

AideRSS 2.0 (postrank.com)
Java/1.6.0_13
Java/1.6.0_14
libwww-perl/5.816
MLBot (www.metadatalabs.com/mlbot)
Mozilla/4.0 (compatible;MSIE 5.01; Windows -NT 5.0 - real-url.org)
Mozilla/5.0 (compatible; Twitturls; +http://twitturls.com)
Mozilla/5.0 (compatible; Viralheat Bot/1.0; +http://www.viralheat.com/)
Mozilla/5.0 (Danger hiptop 4.6; U; rv:1.7.12) Gecko/20050920
Mozilla/5.0 (X11; U; Linux i686; en-us; rv:1.9.0.2) Gecko/2008092313 Ubuntu/9.04 (jaunty) Firefox/3.5
OpenCalaisSemanticProxy
PycURL/7.18.2
PycURL/7.19.3
Python-urllib/1.17
Twingly Recon
twitmatic
Twitturly / v0.6
Wget/1.10.2 (Red Hat modified)
Wget/1.11.1 (Red Hat modified)

Of the few user-agents that seem ‘real’ at first, half are originating from an ip-address used by Amazon EC2. And I doubt people are setting op proxies on there.

Oh yeah, Googlebot (the real deal, from a legit google owned address) is sucking up posted links like fresh oysters.
I guess google is trying to make sure in advance to never be beaten by twitter in the ‘realtime search’ department. Actually, I think it’d be almost stupid NOT to post any new pages/posts/websites on Twitter, it must be one of the fastest ways to get a Googlebot visit.

Same experiment with a real, established twitter account

Now, because I was posting the url’s either as ‘status’ messages or directed @people, on a test-account with hardly any (human) followers, I checked again using the twitter accounts from a commercial site I’m involved with. These accounts all have between 500 and 1000 targeted (I think) followers. I checked the destination access_logs and also added ‘my’ redirect after the bit.ly redirect: same results, although seemingly a bit higher real visitor/bot ratio.

Btw: one of these account was ‘punished’ with a 1 week lock recently because the same (1 one!) status update was sent that was sent right before using another account. They got an email explaining the lock because the account didn’t act according to their TOS. I can’t find anything in their TOS about it, can you?
I don’t think Twitter is on the right track punishing a legit account, knowing the trickery I had been doing with it’s api went totally unpunished. I might be wrong though, I often am.

On the other hand: this commercial site reported targeted traffic and actual signups from visitors coming from Twitter. The ones that are really real visitors are also very targeted. I’m just not sure if the amount of work involved could hold up against an adwords campaign.

Reposting the same link over and over again helps

On thing I noticed: It helps to keep on reposting the same links with regular intervals.
I guess most people only look at their first page when checking out recent posts of the ones they’re following, or don’t look too far back when performing a search.

Now, this probably isn’t according to the twitter TOS. Actually, it might be spamming but no-one is obligated to follow anyone else of course.

This way, I was getting more real visitors and less bots. To my surprise (when my programmer’s hat is on) there were still repeated visits from the same bots coming from the same ip-addresses. Did they expect to find something else when visiting for a 2nd or 3rd time? (actually,this gave me an idea: you can’t change a link once it’s posted, but you can change where it redirects to)
Most bots were smart enough not to follow the same link again though.

Are you successful in getting real visitors from Twitter?
Are you only relying on bit.ly to provide traffic stats?

What makes them click ?

The other day (well, actually some weeks ago while relaxing at the beach in Kos) I read ‘Neuro Web Design – What makes them click?’ by Susan Weinschenk. (http://neurowebbook.com)

The book is a fast and easy read (no unnecessary filler) and a good introduction on how your site’s visitors can be steered in the direction you want them to go.

The Obvious

The book handles some of the more known/proven techniques, like for example that ratings/testimonials of other people can help sell your product or service. Another well known technique it talks about is inducing a sense of scarcity/urgency in the visitor. Only 2 seats left! Buy now and get 33% off! It’s not because these are known techniques that they stop working.

Luckily 2/3rd of the book handles less obvious techniques, otherwise it wouldn’t be worth buying.

The Not So Obvious

A less known influencing technique is reciprocity. And then I’m not talking about swapping links with another website, but the fact that someone is more likely to do something for you after you did something for them first. The book cites some studies (I always love the facts and figures) and gives some actual examples of how to implement this in your site’s design, which is less obvious when you think about it. Want to know more ? Buy the book!

Other interesting sources

For a more general introduction to the same principles, I’d suggest ‘Yes! 50 Secrets from the Science of Persuasion’. ‘Yes!…’ cites some of the same studies (it seems there’s a rather limited pool of studies covering this subject), but of course doesn’t show how to implement these techniques in your site’s design. I read ‘Yes!…’ last year, making ‘Neuro Web Design’ just a little bit less interesting.

!!!Always make sure you’re able to measure your changes. If you haven’t yet, check out the advanced segmentation in Google Analytics (don’t be afraid because it says ‘beta’, it works just fine) and Google Website Optimizer.

Worth Buying?

Can I recommend it ? Sure, why not. I think it can be useful for anyone who ever had to think about the design or content of a site. You don’t have to be a marketing guy to want a site you’re involved with to be successful. The content/filler ratio is excellent too: you don’t need to wade through dozens of pages to filter out the interesting bits. (unlike ‘The Design of Sites’, which contains too much useless info and because it’s in dead-tree format, you can’t google it)

If you like it, you might also check out ‘Yes! 50 Secrets from the Science of Persuasion’.

Tip for people living in Europe: check Amazon UK for your book buying needs. Because of the low UK Pound exchange rate, it’s usually considerably cheaper and faster to get a book delivered to your doorstep by Amazon UK compared to having to order it at the local book store or web-shop.