Given Enough Eyeballs, Not All Bugs Are Shallow

I have been asked to write a chapter for a book about the experiences of people involved in Open Source with the idea of “If I knew what I know today”. I asked if I could re-print my contribution here. I hope it is interesting for people concerned about Open Source testing.

Dogfooding Is Not Enough

I have been involved with Open Source since my early days at University, in Granada. There, with some friends, we founded the local Linux User Group and organized several activities to promote Free Software. But, since I left university, and until I started working at Canonical, my professional career had been in the proprietary software industry, first as a developer and after that as a tester.

When working in a proprietary software project, testing resources are very limited. A small testing team continues the work that developers started with unit testing, using their expertise to find as many bugs as possible, to release the product in good shape for end user consumption. In the free software world, however, everything changes.

When I was hired at Canonical, apart from fulfilling the dream of having a paid job in a free software project, I was amazed by the possibilities that testing a free software project brought. The development of the product happens in the open, and users can access to the software in the early stages, test it and file bugs as they encounter them. For a person passioned about testing, this is a new world with lots of new possibilities. I wanted to make the most of it.

As many people do, I thought that dogfooding, or using the software that you are aiming to release, was the most important testing activity that we could do in open source. But, if “given enough eyeballs all the bugs are shallow”, (one of the key lessons of Raymond’s “The Cathedral & The Bazaar”), and Ubuntu had millions of users, why very important bugs were still slipping into the release?

First thing that I found when I started working at Canonical was that the organized testing activities were very few or nonexistent. The only testing activities that were somehow organized were in the form of emails sent to a particular mailing list calling for testing a package in the development version of Ubuntu. I don’t believe that this can be considered a proper testing activity, but just another form of dogfooding. This kind of testing generates a lot of duplicated bugs, as a really easy to spot bug will be filed by hundreds of people. Unfortunately, the really hard to spot but potentially critical bug, if someone files it, is likely to remain unnoticed, due to the noise created by the other hundreds of bugs.

Looking better

Is this situation improving? Are we getting better at testing in FLOSS projects? Yes, I really believe so.

During the latest Ubuntu development cycles we have started several organized testing activities. The range of topics for these activities is wide, including areas like new desktop features, regression testing, X.org drivers testing or laptop hardware testing. The results of these activities are always tracked, and they prove to be really useful for developers, as they are able to know if the new features are working correctly, instead of guessing that they work correctly because of the absence of bugs.

Regarding tools that help testing, many improvements have been made:

  • Apport has contributed to increase the level of detail of the bugs reported against Ubuntu: crashers include all the debugging information and their duplicates are found and marked as such; people can report bugs based on symptoms, etc.
  • Launchpad, with its upstream connections, has allowed having a full view of the bugs, knowing that bugs happening in Ubuntu are usually bugs in the upstream projects, and allowing developers to know if the bugs are being solved there.
  • Firefox, with its Test Pilot extension and program, drives the testing without having to leave the browser. This is, I believe, a much better way to reach testers than a mailing list or an IRC channel.
  • The Ubuntu QA team is testing the desktop in an automated fashion and reporting results every week, allowing developers to have a very quick way to check that there have not been any major regressions during the development.

Although testing FLOSS projects is getting better, there is still a lot to be done.

Looking ahead

Testing is a skilled activity that requires lots of expertise, but in the FLOSS community is still seen as an activity that doesn’t require much effort. One of the reasons could be that the way we do testing is still very old fashioned and does not reflect the increase of complexity in the free software world in the last decade. How can it be possible that with the amount of innovation that we are generating in open source communities, testing is still done like it was in the 80s? Let’s face it, fixed testcases are boring and get easily outdated. How are we going to grow a testing community, who is supposed to find meaningful bugs if their main required activity is updating testcases?

But, how do we improve testing? Of course, we cannot completely get rid of testcases, but we need to change the way we create and maintain them. Our testers and users are intelligent, so, why creating step-by-step scripts? Those could easily get replaced by an automated testing tool. Instead of that, let’s just have a list of activities you perform with the application and some properties it should have, for example, “Shortcuts in the launcher can be rearranged” or “Starting up LibreOffice is fast”. Testers will figure out how to do it, and will create their testcases as they test.

But this is not enough, we need better tools to help testers know what to test, when and how.  What about having an API to allow developers to send messages to testers about updates or new features that need testing? What about an application that tell us what part of our system needs testing based on testing activity? In the case of Ubuntu we have the data in Launchpad (we would need testing data as well, but at least we have bug data). If I want to start a testing session against a particular component I would love to have the areas that haven’t been tested yet and a list of the 5 bugs with more duplicates for that particular version, so I avoid filing those again. I would love to have all this information without leaving the same desktop that I am testing. This is something that Firefox has started with Test Pilot, although they are currently mainly gathering browser activity. Google is also doing some research in this area.

Communication between downstream and upstream and vice-versa also needs to get better. During the development of a distribution, many of the upstream versions are also under development, and they already have a list of known bugs. If I am a tester of Firefox through Ubuntu, I would love to have a list of known bugs as soon as the new package gets to the archive. This could be done by having an acknowledged syntax for release notes, that could then get easily parsed and bugs automatically filed and connected to the upstream bugs. Again, all of this information should be easily available to the tester, without leaving the desktop.

Testing, if done this way, would allow the tester to concentrate on the things that really matter and that make testing a skilled activity; concentrate on the hidden bugs that haven’t been found yet, on the special configurations and environments, on generating new ways to break the software. On having fun while testing.

Wrapping Up

From what I have seen in the latest three years, testing has improved a lot in Ubuntu and the rest of FLOSS projects that I am somehow involved with, but this is not enough. If we really want to increase the quality of open source software we need to start investing in testing and innovating the ways we do it, the same way we invest in development. We cannot test 21st century software, with 20th century testing techniques. We need to react. Open Source is good because is open source is not enough anymore. Open Source will be good because it is open source and has the best quality that we can offer.

 

Bye, bye Ubuntu Ready – Hello Ubuntu Friendly!

Discontinuing Ubuntu Ready

If a user goes to the Ubuntu certification site, what he or she will find, apart from a list of certified systems, two different types of certification Ubuntu Certified and Ubuntu Ready.

Having two commercial hardware validation programs confuses the customer, as it is quite difficult to understand the differences between the two. For this, an other problems with the programme, we have decided to discontinue the Ubuntu Ready programme in 11.10.

Presenting Ubuntu Friendy

Instead of just removing Ubuntu Ready, we would like to start a non-commercial new hardware validation programme, created by Canonical, and with co-ordination with the rest of the community. This new programme is called Ubuntu Friendly (although the name might change).

The great thing about this new Ubuntu programme is that it will be completely community driven. There won’t be any commercial requirements for systems to be Ubuntu Friendly.

Although all the specifics of the programme will be discussed at UDS Oneiric, the basic ideas of the programme are:

  • Anyone will be able to test their systems and provide test results.
  • Anyone will be able to review and provide feedback on the results (something like triaging bugs in the Bugsquad). All the test results will be public (except those made it private by the certifiers)
  • Only a small subset of people (certifiers) will be able issue the Ubuntu Friendly certificate, based on results (in the same way Bug Control is a subset of the Bugsquad that have more permissions to work with Ubuntu bugs). There will be a formal and specified way to apply to be a certifier.
  • Many positive results (and not just one) for a given model and hardware configuration will be needed to mark the system as Ubuntu Friendly.
  • All the client tools to test Ubuntu, and the tests themselves, will be open source.

UDS Oneiric

Most of the specifics of the programme will be discussed during UDS Oneiric, during two sessions, on Wednesday:

This blueprint will contain the work needed to be done in terms of the programme itself: description of the programme, governance, etc.

This blueprint will contain the work needed to be done in terms of the technical infrastructure that is required to make the programme possible: testing tools, backend infrastructure, etc.

If you are interested in hardware validation and would like to share your ideas and make this project possible, feel free to subscribe to the blueprint and attend the sessions at UDS. And remember that remote participation is also possible!

Moving to the Hardware Certification Team

After two years and a half in the Ubuntu QA team I have moved to the Hardware Certification team, on Platform Services. I was very happy to be given the opportunity to join the Hardware Certification team and I couldn’t reject. I am very excited to be able to help in the goal of making Canonical profitable.

How will that affect my collaboration with Ubuntu QA

It is been a great pleasure to work all this time with a great group of people  (both Canonical and non-Canonical). Obviously, as working directly on Ubuntu QA won’t be part of my paid job, I won’t be able to dedicate it as much as time as I did before. So, first of all, sorry if I am not getting back to you as quickly as I did in the past. But I love the Ubuntu QA community and I am still part of it. I will still be working on some of the things I did before, in my spare time. There are some projects, like the Desktop Testing Program or Desktop Testing Automation that are very close to my heart and I would like to stay somehow involved with them, I will try to attend the Ubuntu QA meeting at least twice per month and I will stay active in the ubuntu-qa mailing list.

The Hardware Certification Team

Having a successful certification program is both beneficial for vendors, ensuring their costumers that the hardware will work with Ubuntu, and the Ubuntu community, as they have access to the list of systems that work out of the box. We agree that sometimes we don’t expose the Hardware Certification program to our community as much as we should. There are people very active in the Ubuntu development that don’t know that Canonical has a public website where people can check what systems are certified to work with Ubuntu or even that the certification program exists. We don’t keep it as a secret, though; the Hardware Certification team has been attending UDS, our main testing tool is open sourced and installed by default in Ubuntu and, as I said, the systems that have been certified are published. I understand that we can do a better job explaining how the hardware certification program itself works and keep it more in the open, accepting suggestion and criticism.

To start with, Victor, the Hardware Certification team manager, has written a wiki page to explain a bit how the certification programs works and we have now a public project in Launchpad where you will be able to file bugs related to the project or ask questions. We would love to hear your thoughts and feedback about the program. In return, we will make our best to improve the program in ways that makes it more useful for our community.

Magomatic

It’s been a while since the last time I blogged about Mago, but Natty is going to be an exciting cycle for desktop testing automation (a lot is happening!) and I would like to present some of the work we have been doing.

Today I will write about Magomatic, a new side project related to Mago.

If you have tried to add a new testcases to an existing Mago wrapper, you can see that this is pretty straight forward. Most of the things that you need are already there, and you need to add only the code of the test, without thinking on the accessibility information of the application (OK, sometimes you have to, but it is quite simple to start the process). If you, however, have ever tried to add a new wrapper application to Mago I guess that you have found the process a bit difficult: you need to understand how the accessibility information is presented by LDTP, you need to create the application python file, you also will have to create a test suite python file, and a XML data file. This is time consuming and I though it could (and should) be automated.

So I created Magomatic. And how does it work?

Magomatic uses templates and accessibility information to create those files for you. Using it is pretty straight forward:

  1. Open the application you want to create the wrapper for.
  2. Run Magomatic:
    $ bzr branch lp:magomatic
    $ cd magomatic/bin
    $ ./magomatic
  3. When prompted, you will need to select the window you want to create the wrapper for with the mouse pointer.
  4. Done! Under the
    data/

    folder you will find a folder with the name of the application with the needed files to add to Mago and start coding your tests.

This is a work in progress, but the main and most important functionality is already there. We really hope that this will lower the entry barrier to Mago and more people will join us adding new tests in the Natty cycle.

HW Certification jobs at Canonical

The HW Certification team at Canonical is hiring three localized engineers: one in our offices in Lexington (MA), USA; one in Taipei, Taiwan; and one in Montréal, Canada.

The HW Certification team, part of Platform Services, provides certification as a service to partners. Basically, if a computer vendor wants to get one of their machines certified to work nicely with Ubuntu, they can buy this service.

The variety of work involved in our side is big and fun: from receiving the machines to be certified (some of them even brand new machines not yet public!), to commercial relationships, passing from a lot (a lot!) of technical work: writing testcases, maintaining our testing infrastructure, etc.

For me, working at Canonical, has been (and it is) the best professional experience I have had so far. It is great to be able to work in such a great environment, with very smart people and making Ubuntu better for everybody. I truly recommend Canonical as workplace. Please, ping me me on IRC if you would like to know more about how it is to work here.

So, if you live in any of these three locations or are willing to relocate (Canonical does not offer a relocation package), please, have a look to the following job descriptions and send your resume to victorp AT canonical DOT com.

On First Experiences

Everybody agrees how important the Out-Of-Box experience is for a product. If the users’ first experience with any kind of product is frustrating it is very likely that they’ll return it and never look back.

On Operating Systems, usually, this first experience is turning on the computer and reply to some basic questions. For all the major OS like Windows or Mac OS, the software comes preinstalled in the computer. For many users, therefore, that’s the computer itself: something that you turn on and it starts working.

For Linux is quite different. Despite the efforts that some companies (including Canonical) are doing in order to be easier and easier to buy a computer with a Linux distribution preinstalled, nowadays, the first experience a user has with Linux is, most of the times, a CD and an installation process. Well, if you then want to have one of the best first Linux experiences ever, wait until Sunday and install the brand new Ubuntu 10.10. Its installation process is, ladies and gentlemen, pure joy:

I would like to congratulate the Ubiquity team for the fantastic work they do every six months. They make the first experience of those people who start with Linux better every time. It is great to have the opportunity to work with you, guys.

My experience upgrading UNE 10.04 to Maverick (and yours!)

As you might know, Ubuntu version for netbooks, UNE, is having a major re-engineering work for Maverick Meerkat,  soon to be Ubuntu 10.10. The old interface, which included packages like netbook-launcher or maximus, is going to be replaced by Unity. If you don’t know what Unity is, the nice people behing OMGubuntu published some days ago a nice review of the brand new UNE interface.

I will give you a clue: it does look very different from what you’re used to. That’s why we want to collect as many reports as possible of people upgrading from UNE 10.04 to 10.10 (with Unity).

My experience upgrading

OK, I don’t want to ask people to test something if I haven’t (and I have the means for it). I took my Dell Mini 9 (well, technically is Canonical’s, but, anyway) and I installed Ubuntu Netbook Edition 10.04.1 in Spanish. The installation went very well and fast.

After reboot, I updated my 10.04 installation and started the upgrade to Unity. Although the upgrade itself did not have any major problems it took almost six hours! Of course, I reported this as bug 646638. I talked with Michael Vogt on IRC and he will investigate.

Once the upgrade finished everything worked as expected: the language was still Spanish and there were no major crashes. Nevertheless, the global menu stayed there with “File Editar” even when no applications were running. I filed that as bug 646890 in Unity.

Apart from that, I found a couple of bugs in the Dash, but unrelated with the upgrade itself. These are bugs 646758 and 646756.

Your experiences upgrading

My system is not a real system. I use the Dell Mini 9 with SSD 8GB for testing purposes. I don’t use it on a daily basis. I reinstall almost every flavour of Ubuntu on it every milestone. My upgrade experience was from a nice, cleaned, UNE 10.04 to 10.10. No PPAs or third party software installed.

That’s why we need real feedback from the people that often use UNE 10.04 on their upgrades to 10.10. If you want to participate in our testing effort, just follow the next steps:

  1. Create an account in our tracker
  2. Upgrade to Maverick
  3. Report back!

Use my testing report as an example. You see that you can add comments (like the system you used,or general impressions) and, of course, add bug numbers if you encounter any.

Thanks and happy testing!

So far, so good

It’s been more than two years since I started working at Canonical and, although I have been blogging about my daily job here, I have never talked about how this job is important to me.

I love testing software. Yes, I know it seems strange to love an activity that some other people find a bit tedious, but I do. I was a full time developer when I discovered that I liked testing the software. Testing software gets you the opportunity to see the product as a whole, but without losing the technical part of the job. So, when I got the opportunity to work at Canonical as a member of the Ubuntu QA team it was like a dream job. Not only I was going to be able to test free software during my daily job but, also, I was going to test ALL the free software that is included in Ubuntu.

When I was hired, one of my first missions was to create a way to to test the desktop in a repeatable way, easy to maintain. That’s how Mago project started, a couple of years ago. Working on this project has been great, as it has been working closely with LDTP upstream developers. I have contributed to LDTP through bug reports, patches and helping with the release of LDTP in Ubuntu. I always tried that the latest LDTP was successfully released and uploaded to Ubuntu.

I specially remember when we were trying to get the latest LDTP before Ubuntu 10.04 Feature Freeze. Nagappan, the main LDTP upstream developer and I worked closely on IRC to meet the deadline. Together, we fixes issued, verified them, got everything together and got it uploaded it to Ubuntu just in time. It was the perfect example of Open Source collaboration.

Mago, itself, is free software, released under the GPLv3. People inside and outside Canonical have contributed to it with bug reports, patches, new features and, of course, new tests to test the desktop applications, often GNOME applications.

But, apart from Mago and desktop testing automation, I am specially happy to be able to test all the open source bits that make Ubuntu: from the kernel to the desktop, from brand new topics as Multitouch, to all time classics as Firefox.

I work for Canonical, testing free software, trying to make it better for everybody. I need to remind myself everyday how privilege I am.

Multitouch testers in the Hall of Fame

Thanks to Daniel Holbach, the people testing uTouch and reporting back in the Multitouch Testing Tracker now appear in the Ubuntu Hall of Fame.

If you have been helping testing MT and you’re name does not appear there, is due to a bug in the ISO tracker, that prevents testers that haven’t set their Launchpad ID properly. This bug has been fixed, and it will be released in our next roll out of the testing tracker, but, in the mean while, there is a workaround that I explained in a previous post.

Please, if you are helping testing uTouch, check out how to appear in the Hall Of Fame.