Ubuntu Certification 101 – BIOS

Last week, at UDS, I was asked several questions about how Ubuntu Certification worked. I decided I would start a set of blog posts about several aspects of the certification programme, for people willing to know a bit more about how we test and what.

One of the most confusing bits about Ubuntu Certification are BIOS options. In several ocassions I was asked the questions whether UEFI (with and without Secure Boot) was going to be added as part of Ubuntu Certification or not.

The answer is both yes and no. I will explain it a bit more in detail.

In Ubuntu Certification we don’t test BIOS options, nor we change them. We try to install Ubuntu with what it comes with the system from factory, and it needs to work up to a certification level using those options. That means that if it is using Legacy BIOS, we will be testing Legacy BIOS, if it comes with UEFI without Secure Boot, that is what we will be testing. If it comes with Secure Boot enabled, Ubuntu will need to install out of the box with the keys already in that system.

From 12.10 we have started to parse the BIOS information as part of the testing process:

We will be updating the public certification site to show that information there as well.

Same thing for BIOS options. We test with factory default options, and only those. The goal of doing this is that Ubuntu Certified systems should provide and out-of-the-box Ubuntu experience. In some cases, tweaking BIOS options can improve the situation for non working systems, but that is something we don’t want to acept for Ubuntu Certification.

 

A new UI for System Testing

During the Precise cycle we have been working in building a new UI for System Testing, the desktop application to test your hardware with Ubuntu and report results to Launchpad and Ubuntu Friendly.

If you are running Precise, you can test the new UI searching the Dash for “System Testing”. The new UI is now feature complete and you can start using it to test your hardware and submit your results to Friendly.

Here you can find the differences between the old UI and the new one:

Old UI

 

New UI

I think it looks much better, doesn’t it? The most important thing is that it builds the grounds to be able to keep making improvements to the UI, which were blocked in many cases by the old one.

Now we need to a lot of testing to make sure it is fully working when 12.04 LTS gets released. So, if you are running Precise, I encourage you to give it a try and report as many bugs as possible.

To file bugs, run the following command in a terminal:

ubuntu-bug checkbox

The first week of Ubuntu Friendly Beta

Last week, on the same day Ubuntu 11.10 got released, we launched the Beta of Ubuntu Friendly website. This is a huge milestone for us, as we will be able to collect a lot of data coming from Ubuntu users in order to fix bugs and tweak the scoring system so it is better once Ubuntu Friendly finishes its beta period (once 12.04 LTS gets released).

In the first few days we got the following numbers:

  • More than 350 submissions
  • More than 100 submissions with more than 3 stars
  • 4 submissions with 5 stars
  • 7 models with more than 1 rater

I think this is a great achievement for the first week of a beta release. Let’s hope we get many more results, feedback and bugs as possible, so Ubuntu Friendly rocks when 12.04 LTS is released!

We are hiring!

The Ubuntu Hardware Certification team is growing!

I am happy to announce that the hardware certification team is growing and we are now looking for a Certification Engineer to work in our lab in Lexington (MA).

The job is great if you like to develop automated testcases, but you also want to get your hands dirty with hardware configuration, networking, etc. You will be able to help a team that helps Ubuntu to work better in a wide range of hardware, helping developers debugging weird kernel bugs. This is also the team behind Ubuntu Friendly, so you will be able to help this great project as well!

So, if you live around the Boston area and you think your the person for the job, please, apply in the Canonical job site. Looking forward to reading your resumes!

Ubuntu Friendly is almost there! (and help needed!)

As weeks go by, we are getting closer and closer to Ubuntu 11.10 release and, therefore, to Ubuntu Friendly beta release.

First of all, the website is taking shape nicely and you can already see (and use!) an alpha version of the final site.

As you can see from the screenshot, right now all systems have only 1 star. This is the minimum rate a system can get (we give 1 start because, at least, it was able to install Ubuntu on it and submit results). The reason for all the submissions so far to have only 1 star is that a bug in Checkbox was preventing for one test considered to be “core” to run properly.

Call for Testing

The mentioned bug in Checkbox is fixed in our PPA and it is now ready to be used in Oneiric. If you run Oneiric it will be really helpful if you could install Checkbox from our PPA and run the full Ubuntu Friendly suite (this is the suite that runs by default).

  • Add our PPA to your software sources
  • Install the latest version of checkbox
  • Run “System Testing” on your system and submit to Launchpad

This will help us on two sides:

  • First, you will run the latest code in trunk. We are planning to release version 0.12.8 to Oneiric in a week, so this is the last time to fix critical issues. Should you find any issues during your testing, please, file a bug against Checkbox project.
  • On the server side of things, having a lot of data will help us testing the website to make sure it works fine when 11.10 gets released. Should you find any issues on the website, please, file a bug against the Ubuntu Friendly project.

Thanks for the help and I hope you are as excited as I am about the project!

Ubuntu Friendly sprinting! (Wednesday)

Wednesday came and went! I will summarize our progresses as usual.

Ubuntu Friendly website

The website is coming together very nicely. Marc, who is working on the backend, worked closely with Mike, who is working on the frontend and they were able to communicate both, send a submission and show that information on the website. The website frontend still needs some polishing, but the basics are already there and working. One of the goals of the sprint was to have a full working solution, although not feature complete. I think we are going to make it.

Apart from this real progress we also worked a bit on new wireframes. Yesterday we got a comment on a previous post about having a mobile version of the site, as people will use their phones while shopping for a new system, to check how well they are reported to work with Ubuntu. We thought that it was a very good point, so we worked on some wireframes for the mobile version of the site:

Of course, we will focus on the desktop version first. But we will try to develop a mobile version based on these mock ups after that.

Checkbox

On the checkbox side we are almost done for what we expect to land in Oneiric. There is just one thing left: we are about to measure how long it takes to run the full suite for Ubuntu Friendly and we will remove the less useful tests to try to cut the running time to around 15 minutes.

Once we finish with that we will upload the new version to the archive.

Ubuntu Friendly sprinting! (Tuesday)

Yesterday was a great day in the UF sprint and we got a lot of stuff done. I will summarize the interesting bits.

Ubuntu Friendly Website

As I said on the previous post, we still needed a way to visualize what components are working and which are not (to a certain level). To achieve that we have created a new “details” page, once you click on “more” on the configuration page.

That page will give you two pieces of information. First, it will show the complete list of components that were reported for that configuration:

On that list you will see the core components on the top (the ones we use to distinguish a configuration from another one) and a list of the rest of components found in all the units that were reported.

The second piece of information are the results that were reported per component:

On this view you will see the number of people that reported that a particular component was working and a percentage. We will have some colour code for the percentage, to have a quick visual view of it.

The “Related Bugs” tab is just a way of saying that this view could be extended in the future, but currently this is out of the scope for this release.

Checkbox

On the client side we also made some important progress. We finished mapping tests to components, so we can know which components are failing and which ones are passing.

We just merged our changes into checkbox trunk and we hope to have it uploaded to Oneiric early next week.

Ubuntu Friendly sprinting! (Monday)

This week, myself and other 4 members of the Ubuntu Friendly Squad are working together, face to face, to try to have a working prototype of the Ubuntu Friendly programme as a whole. I will be blogging about it during the week to share the progress with the rest of the team.

Ubuntu Friendly Website

The main focus for Monday was to get an idea of how the Ubuntu Friendly site would look like. We think that the most difficult thing to solve is having right the idea of different configurations. As a lot of people pointed out while we were presenting the programme, a model name can contain many different hardware, and work much better or worse depending on those components.

We want to find a compromise between having a UI that’s easy to understand, and having it as complete as possible. For the Ubuntu homepage this is what we came up with:

Every different configuration will be in a different line, with some details of the components that are part of it. If the user wants to know more details about a particular component, hovering over it will do the trick.

You will be able to filter by release, rating or number of results to get you to the results that you are looking for. Once one of these systems is clicked on, the details page will appear that will look like this:

For that particular configuration we will be showing the full details of the hardware, the number of results and their rating it got in the different releases. Also, a box will show similar systems. For this cycle the similar systems will be other configurations of the same model.

There is still one thing that we need to work on for the details page. We need a way to say if a particular component is working or not. But hey! we still have the rest of the week to figure it out.

Checkbox

Another focus for today was to have a complete whitelist that include all the components that we have agreed as part of the UF scope. We added tests for components that were on the list, and removed unrelevant tests.

We are putting our progress in a public branch at Launchpad (https://code.launchpad.net/~ubuntu-friendly-control/checkbox/uf_sprint/). We are hoping to merge to trunk and to have a new version in Oneiric by Thursday, but in the mean time you can check the new whitelist using directly that branch.

While working on the whitelist we discovered a very nasty bug in Checkbox. If you deselect some tests when starting running checkbox, start testing, then close checkbox, and start it again, the whitelist won’t be used anymore, your previous selection will be used instead. This bug affects the Ubuntu Friendly experience, so we will have it fixed before release.

Ubuntu Friendly is not a certification programme (and other misconceptions)

There seems to be a little bit of confusion about what the Ubuntu Friendly programme is and what it is not. People tend to think that Ubuntu Friendly is a certification programme and, because of this, they obviously don’t understand the idea behind core and extra components.

Let’s try to clarify things a bit with a small FAQ:

Is Ubuntu Friendly a certification programme?

No. Ubuntu Friendly is not a certification programme. UF is a community driven hardware validation programme. Its goal is to have a list of systems that people have tested with a particular release of Ubuntu and an associated rating, based on the results of their testing.

Let’s say is the next generation HardwareSupport wiki pages (https://wiki.ubuntu.com/HardwareSupport/Machines/Laptops), but with a better structure, as the tests will be written upfront, and the process to get a rating for a system will be the same for every system.

What does it mean that a system is Ubuntu Friendly?

Nothing, as there is not such concept. Ubuntu Friendly is just the name of the programme and yes, it was chosen because of the concept of a system being Ubuntu friendly (working fine with Ubuntu), but there won’t be a classification of systems that gained the Ubuntu Friendly status or not.

Instead of that, any system tested with the testing tools that we provide will make it to the Ubuntu Friendly list, with a rating associated with it. The rating will be between 0 and 5, that will give the user of the Ubuntu Friendly list an overview of how well a system works with Ubuntu and how many people have tested it.

Will this substitute the Ubuntu Certification Programme?

No. Ubuntu Friendly is not a certification programme and it won’t substitute the current Ubuntu Certification Programme. The UCP is a commercial certification programme, run by Canonical, and it will be still available for commercial partners. Opposite to the Ubuntu Friendly programme, the UCP is a go/no-go decision. A system is certified with Ubuntu or it is not, there are no ratings. To be certified with Ubuntu the system must pass all the tests specified in the UCP coverage list.