Ubuntu Friendly Process

stars

Picture by stefanvds

Last week, current members of the Ubuntu Friendly Control gather together and we took the opportunity to discuss some of the Ubuntu Friendly processes. One of the biggest questions that we wanted to answer was what it meant for a system to be Ubuntu Friendly. How and where should we draw the line between an Ubuntu Friendly system and the rest of systems. After a very good discussion and brainstorming we came up with a different solution: we wouldn’t draw that line.

The basic idea from where we will define the way a system is tested and have an Ubuntu Friendly rating is the division between core components and extra components. What is a core component or an extra component we care about will be decided later in the process.

Testing

Ubuntu Friendly tests will be grouped by the component that they test. In order for a submission to be consider valid and accepted in our rating system, all the tests that cover the core components should have been tested (either pass or fail, but tested).

If a submission does not contain results for ALL tests that cover, the system will reject that submission and it won’t count for Ubuntu Friendly status.

Once that the core components have been covered, the submission will be accepted.

Ratings

Rating per submission

The rating for a particular accepted submission (at least all tests for core components) will be determined by the number of components that passed or failed. The rating will be between 0 and 5.

If any of the tests that cover the core components failed, the submission will get a value of 0. Even if any of the extra components worked.

If the all the tests that cover the core components passed, the submission gets a rating of 3.

The two remaining points to get to a rating of 5 are covered by the extra components with a simple rule of three. If a system does not have any extra components, and, again, all tests covering core components passed, the submission will receive a rating of 5.

Examples:

We have a list of 10 core components (UFC) and 7 extra components (UFE) that we care about. Any other component we will list it as additional component (AC).

System Components Pass Results Rating
Laptop1 6UFC, 3UFE, 0AC 5/6 UFC, 3/3 UFE 0
Laptop2 10UFC, 4UFE, 3AC 10/10 UFC, 1/4 UFE 3.5
Desktop1 7UFC, 5UFE, 1AC 7/7 UFC, 5/5 UFE 5
Netbook1 4UFC, 1UFE, 0AC 4/4 UFC, 0/1 UFE 3

Global rating per system

The global rating for a particular system will be the average of all the submission for that particular system.

Ubuntu Friendly Website

Each release of Ubuntu will get its own list. By default, the list shown will be for the latest release of Ubuntu.

Systems will be ordered by global rating first, and number of submission after.

Systems with very bad rating will also appear in the list. There will be no boarder line to call a system Friendly or not, it will be just a rating systems

Example:

****************************************************
Ubuntu 11.04 Friendly

Latitude 2120             4.3   (30 submissions)
Vostro                    4.3   (13 submissions)
Acer EeePC                3.2   (40 submissions)
Mac Book Pro              2.9   (3  submissions)
WinPro Laptop             0.3   (50 submissions)     
****************************************************

 

The representation of the ratings will be graphical (stars, bars, etc). Clicking on any of the results will give us the details of each of the submissions.

People will be able to filter by model, type of system, minimum rating, etc.

Giving Feedback on Results

Any user will be able to give feedback and comments for a particular system without needing to run the tests. The feedback will be things like “bluetooth is supposed to be working, but it is not working for me”. These comments will be showed on the details of a particular system, but they won’t affect the actual rating of the system.

Waiting on your comments!

As in previous processes and thoughts about Ubuntu Friendly, we are looking to get as much feedback as possible, and this part of Ubuntu Friendly is no different. Please, join us in the Ubuntu Friendly Squad and have your say!

13 comments

  1. Thank you, that will be very useful.
    One question: what about the version a system is tested on? especially over time, the compatibility can change with newer versions of ubuntu.
    Will there be a selection? will those be mixed or just the ratings with the current version?

    • The idea would be to have a page per Ubuntu release. By default, the website would show the latest release, but making it possible to choose any previous releases from a drop down.

  2. Great stuff! But two questions:

    1) would it be possible to have a different aggregation than “average” on the score? I think either the median (most common result) or a small barplot of results over score would be much more expressive.

    2) What would have to be done to make that (or a similar) process available for other distros (like Debian) too?

    • 1) would it be possible to have a different aggregation than “average” on the score? I think either the median (most common result) or a small barplot of results over score would be much more expressive.

      Thanks for the suggestion. I have added it to the page about the process (https://wiki.ubuntu.com/UbuntuFriendly/Process) for discussion with the team.

      2) What would have to be done to make that (or a similar) process available for other distros (like Debian) too?

      The tool that we are going to use is completely open source (http://launchpad.net/checkbox) and can be reused in Debian. The backend to host the data will be also opensource, so any distro could reuse those tools to implement a similar programme.

  3. 1. I had the same question as David about reusing, thanks for making it clear that the tool is open source. :)

    I suppose this is a Canonical project, so it requires copyright assignment?

    2. How will you handle “customizable” models?

    For example, a few years ago I bought an HP Pavilion “something” (I forgot the exact model). A friend of mine had the exact same model, bought at the same shop (hp.com), except I had paid a bit more to have an ATI graphics card (instead of the Intel one she had) and an extra 2GB of RAM.

    Again, we had the exact same model, an HP Pavilion “something”, so with what I can read on this blog post, they would appear as the same system. Obviously, the results might be very different (especially at that time when ATI cards were poorly supported).

    Is there something planned for those?

    • 1. Yes, this is a Canonical project and we will check with the legal department if it will be under the CA. I will post it under the project’s wiki page as soon as I find out.
      2. We talked about this during the meeting, and we decided that those with different core components will show as different models in the site, something like: “HP Pavilion something (ATI), HP Pavilion something (Intel)”. Although we haven’t yet decided which will be a core component, obviously, the GPU will be one of them. Thanks for pointing it out, I will add this information to the wiki page.

      • 2. The thing is, sometimes “HP Pavilion something (ATI)” would not be enough to describe it.

        For example, my current laptop is a “Clevo S3101″. But if you look at the page where I bought it [1], you will see that there are quite a lot of components that can be customized, and many of those could be what you will define as “Core Components”. :-/

        http://www.clevo.fr/shop/s3101pack.html

        There are 2 different choices of CPU, 5 choices of wireless card, there’s an optional 3G modem, etc… (when I bought it, there were 3 different CPUs, 3 different webcams, and more, I guess they streamlined their offer)

        So that seems like a very hard problem to solve, and I’m glad someone is tackling it, and releasing the software under a FOSS license. :)

        [1] I’m not sure it is appropriate to post a link to a shop here, it might be seen as advertisement. Feel free to remove the link from my comment if you are not comfortable with it.

  4. Pingback: What is a Core Component? | Ara Pulido

  5. A bit late, I know, but I wanted to thank you for presenting this in such a transparent way. I knew some of what to expect, since I heard about this while I was still at Canonical, but you did an excellent job of sharing it with the community clearly and comprehensively.

  6. Pingback: Amber Graner: The New Ubuntu Friendly Program Needs User Feedback | Ubuntu Forms

Leave a Reply

Required fields are marked *.