Joomla.org Usability Testing Report
Drupal is doing it, WordPress is doing it, Amazon has been doing it for years, and now Joomla is doing it.
They’re all doing Usability Testing — and no wonder.
Usability Testing is probably the most underrated and overlooked method for improving the user experience of a site. It’s also the most powerful and the most cost-effective method. You can do it for very little cost or even for no cost at all.
What is Usability Testing?
Usability Testing asks users to complete a number of set tasks on a website. The ease with which they complete those tasks, the time it takes to complete them, and the user’s perception of the experience all help the publisher to improve the site.
So what are the goals in a Usability Test for Joomla.org? I wanted to find out:
- Whether the Tester understands what Joomla is;
- Whether the tester understands that Joomla is free;
- What the tester wants to click on the home page.
Joomla Day West was a perfect opportunity to do some Usability Testing on both Joomla.org and Joomla 1.6 because it was full of, well… Joomla users! The results were fascinating and eye-opening.
In this post I’ll focus on the Joomla.org home page. Future posts will look at the Joomla 1.6 usability testing that I’ve done.
To ensure the integrity of the test, it was important to test the Joomla.org home page with someone who had never even heard of Joomla —not easy to find at a Joomla conference. So I had to do this test when I got back home. I found Aaron, a 20-something English major student. When I asked him if he’d ever heard of Joomla, his reply was “Huh?” Perfect! In return for a bottle of beer, he agreed to sit with me for a short usability test.
Here’s the video test, watch it first and then read on:
I first asked Aaron to tell me his initial impression of the site. His reaction was positive — he thought it had a “clean and modern” design. I then asked him to tell me what is “Joomla”? Here he had more difficulty. He figured it was software, but he couldn’t tell me what it does. “I have no idea,” he said. It took a visit to the “About” page to figure it out.
That’s a problem. A site has less than eight seconds to grab someone’s attention before they move on. Visitors need to understand what Joomla is right on the home page, and as quickly as possible.
Once Aaron understood that Joomla allowed him to create websites, he was interested in downloading it. Expecting to find an executable file, he was surprised to receive a bunch of files in a folder. He didn’t know what to do with them, but he was willing to read the manual, although he described it as “a pain.” He also decided to try the demo but didn’t expect it to take up to an hour. Finally, he concluded that the software was not free because it came with a 30-day trial.
The second tester, Will, had been using Joomla but wasn’t very familiar with the Joomla.org website, since he knew what Joomla was, we didn’t focus on the home page as much as the other tester, we focused mainly on the “Demo” feature. He was expecting to find an actual demo site that he could browse without having to create a site. He said he didn’t want to commit to something he wasn’t sure he needed. We also had a few problems with FireFox 2, the browser we used to conduct the test.
Note: sometimes it’s hard to understand the tester on this video because of the recording the typing sounds, my apologies.
My recommendations based on these tests are:
- Include a clear explanation of Joomla right on the home page ;
- Offer a live demo, without having to register or create a site;
- Explain that Joomla is free and can be installed anywhere, and explain how to install it easily (with tools like Fantastico) ;
- Explain what the 30-day trial does;
- Explain what users are downloading and what they have to do with the downloaded file ;
- Test with Firefox 2.0.
Here’s a mockup for the home page, click on the image to see it in full size:
Joomla.org Home Page Mockup
As you can see, the top of the page clearly says: “A free and powerful way to create websites (1).” “CMS,” or “Content Management System” might be a better description but most people are not familiar with the term.
Three sections then follow a user’s logical sequence. First, users want to know what Joomla is and what it can do for them(2); second, they want to see a live demo to figure out if they like it (3); and third, once they have plenty of information, they are ready to set up their site (4).
Notice I didn’t use the word “demo” on the third section because what Joomla.org offers is not really a demo. It allows them to set up their site easily — which is really cool — but it’s not a demo.
The power of Usability Testing is that it’s a very accurate insight into the tester experience. Let’s try to make it a great one!
Tags: design, Joomla.org
Michael Pignataro
| #
Usability is very important in any web site. Big or small. We have used Merav to usability testing on our site for some time now. The results are pretty instant. What Merav did for the main hub of Joomla is excellent. This shows not only our sites we create have usability issues. But So does our parent site Joomla.org. This is where we all go to to get information, find out about great products, and read up on the latest. Some times it is hard. Plain and simple. It is out dated and has grown beyond it initial design. Here you can see simple stuff pointed out that will help for future growth of Joomla.
Thanks for a great read and looking forward to learning more.
Best,
Michael Pignataro
VP of Operations
http://www.corephp.com
Reply
Merav
| #
Thanks Mike, I enjoyed testing corePHP and I look forward to doing another round of testing when you’re ready!
Reply
merav
| #
Sorry, but the content you requested could not be found. Please visit our home page or click on any of the links above to access the different pages of our site.
Reply
John Coonen
| #
Making things dead-simple is a lot of work, but it pays off. The important advice of adding the text, “A free and powerful way to create websites.” = pure gold.
A new user needs to know what Joomla! is, and how it will benefit them in their work, volunteer and/or personal life. If they don’t know what Joomla! is, there’s no reason to stay. We know that few people will stick around long enough to figure it out on their own.
Inadvertently ignoring the new user is easy to do when you’re building a website. That singular piece of advice could be done in seconds, and the results would be hugely beneficial to the Joomla Community.
Reply
matt
| #
I agree Joomla.org does need the bold statement you propose. The first test with Arron was spot on accurate about how new users see Joomla. I have heard those similar comments many times from friends, family, or interested parties.
Hopefully this info will not go unused.
Reply
Jason Hull
| #
On every website, in the top, instantly-visible area of the homepage, a user should be able to answer their 3 questions:
1. What do you do (what is Joomla!)
2. Why should I use YOU (why use Joomla)?
3. What do I do next / What do you want me to do? (Download, Join, etc.)
I recommend a Tagline below the logo… A Leading Content Management System for Building Websites or something to that effect to answer #1. A module listing top 3-4 reasons to use Joomla (Nearly 6000 extensions available, unmatched versatility, FREE!) to answer #2. Then some call-to-actions (Download now, learn more, see demo, see example sites built with joomla) to answer #3
Reply
Merav
| #
Jason, I agree with your set of questions! However a tagline should not contain a phrase most people don’t understand “Content Management System”, ask around and see how many of your not-geeky friends know what it means.
Bullet point with the benefits is also something I agree with. I tried to keep the design as much as possible as it is, just so that it’s super easy to implement, but an overall re-design of the home page is something I would recommend.
Call to action should be something that makes sense. But asking them to download a file that they then don’t know what to do with – not a good idea. I think the main calls to action should be “Learn More” “View Demo” and “Get Started” in this order.
Reply
Web-JIVE
| #
This same test needs to be done with the Admin side of Joomla and for usage. We build lots of Joomla sites and all I can tell you is that Joomla confuses the heck out of users who are new to Joomla. We have invested a lot of time and money creating docs and customizing the APLite admin template to lower our support costs.
Some of the most immediate challenges with new users is the menu to content/component relationship. To start, the acronyms used in Joomla are very confusing to noobs. We have tried to simplify this as much as possible using comparisons to common terms. For instance, Articles vs Pages. Why articles were chosen vs pages is beyond me. I would say easily 95% of our clients would consider this a number 1 gripe.
Second, why stay with plugins, modules, etc. Some usability testing needs to be done to see what makes more sense to users vs. admins. For instance, why still a component when something like Feature or Application would suit better. Customers can relate to that. I can live with plugins since that’s behind the scenes. As for modules, this would need some community think-tank time to get sorted out. This is probably one of the most difficult concepts for people to grasp.
Don’t change the way this works, just rethink the way people would think when navigating around the backend and label it something more natural for people to comprehend. If we had the time and financial resources, we would create an internal mini-fork of Joomla just to go through and edit all the graphics and language files.
The downside to this approach is when updates are issued, things get overwritten and you have to spend a lot of time testing to make sure you didn’t miss something.
Just some thoughts on more usability testing.
Reply
Merav
| #
Eric,
Thanks for the great comment. I couldn’t agree with you more. We started doing usability testing to Joomla 1.6 because I think there is not a whole lot of point doing it for 1.5 at this time. I will be posting the testing as soon as I am done compiling them.
Reply
Kyle Ledbetter
| #
Great post Merav. It’s pretty eye-opening to see someone try to navigate j.org. We’re all so use to it that it’s easy to miss all the areas that could be improved.
Also, great comments everyone else, I totally agree.
Usability will certainly be a big part of the upcoming redesign of the site.
Usability will also be a huge part of the JUX (Joomla User eXperience) which will focus on the admin portion of J!
I’m also sold on more usability tests for my own admin templates 🙂
Reply
Merav
| #
Thanks Kyle, let’s talk about this, get it going. I have some more 1.6 admin testing that show some big problems with ACL and templates part, will post them ASAP.
Reply
Dan Joubert
| #
Excellent article. Seems that we take for granted, all too often that a given site is self-explanatory but usability testing very often proves us wrong.
You are absolutely correct Merav about the amount of time you have to impress a visitor before you are never to see them again.
Smart money has it that you take something that is inherently complex and make it as streamlined and functional as humanly possible. In other words, build a workhorse and not a temperamental thoroughbred. That should apply not only to the front-end but to the back as well.
Cheers!
Reply
Web-JIVE
| #
Merav, and others.. (great thread BTW)
It would be nice if the community could band together and somehow have more input on Joomla usability overall, instead of having so many fragmented ways of training and supporting it. Hands down Joomla beats Drupal and WordPress for expandability but, backend user friendly, it isn’t.
So far, Joomla is still a tool built by developers for advanced users. We need to find a way to dumb it down (backend) and simplify, not add to complexity. I for one, are not looking forward to 1.6.
Joomla 1.5 is complex enough to meet 99% of SMB (small to mid size businesses) and individual needs. Joomla 1.6 is geared more to the corporate market (20%) than the user/SMB (80%) market.
I wouldn’t be surprised to see a strong fork (not what’s out there now) of Joomla to keep 1.5 alive for those of us who don’t need ACL’s or nested categories. It would be OK if someone created a plugin that totally disabled the 1.6 ACL feature and stuck with the top level groups like 1.5 but, I don’t think that’s possible (to un-ACL Joomla).
One good approach to Joomla 1.5 nested categories would be to call Sections, Master Categories. Then users can create sub-categories. Granted, they can’t go 3 levels deep but, since 99% of the sites out there have less that 100 articles (err. pages), why do you need nested categories beyond 2?
Joomla would REALLY benefit IMHO by keeping both branches alive, 1.5 and 1.6x-1.xx. Let 1.5 meet the need of the larger market and 1.6 or later handle the big stuff. Addon purchases should be the true deciding factor of if, Joomla 1.5 should go. Not just because a bigger, better mouse trap was built.
Joomla 1.5 would continue to be refined to be simpler for users, not more difficult.
Reply
ssnobben
| #
Hi Merav and others.
Its great that Joomla.org start doing usability testing and it should of course be done with Joomla Framework-CMS as well!
But I believe it should be done in a more scientific professional way so its really show the, as much as possible, right implications for improvements on all levels.
Not exactly similar like post Joomla already ready Tobii solution but it should be done professionally to understand what the end user expect and need from a WCM system – not a core developers guessing.
I am surprised that Joomla development so far have not set this as before hand testing priority JUX as number one, before starting to recode again from J 1.5 –>J 1.6.
But Its still not to late but more work of changing things and get it right.
Now it have been more like somebody, or a few, in the core devs like it to be and not tested how it really could be and look like for the best for end users.
But its also never to late to make things better step by step of course.
One major thing I dont understand is why Joomla 1.6 (J 1.5.+, J 1.0.+) need its (sections)/categories instead of using menu(s) and menu module(s) instead? Do you ?
All the best!
Reply
Merav
| #
Just to clarify, I did the first test and not Joomla.org, the rest of the tests I’ve done on Joomla Day West and coordinated with some of the Joomla leadership team members. As for your suggestion, using eye tracking is very high end and really not necessary (certainly not as first step). As you can see, we found plenty of issues without any sophisticated equipment or any dollars spent 🙂 doing it this way will allow us to do a lot more without costing the project anything. I agree there is need to do more on the backend of Joomla itself.
Reply
info16
| #
The level of information was great, i am interested what the 30 day trial was trying to achieve?
Reply