A long time ago, GNOME software used to show star ratings as popularity next to the application using the fedora-tagger application. This wasn’t a good idea for several reasons:
- People can’t agree on a scale. Is an otherwise flawless application with one translation issue 5 stars or 4? Is a useful computational fluid dynamics application that crashes on startup but can be run manually on the command line 1 star or 3 stars?
- It only worked on Fedora, and there was no real policy on how to share data, or the privacy implications of clicking a star
- People could “game” the ratings system, for example hardcore KDE users could go through all the GNOME apps and give then one star. We then limited this to only rate applications that you have installed, but it was really a cat and mouse thing.
So, lets go two steps back. What is the star rating trying to convey to the user? When I look at a star rating, I want to see a proportional number of stars to how awesome it is to me. The rest of this blog tries to define awesomeness.
As part of the AppStream generation process we explode various parts of the distro binary package and try to build metadata by merging various sources together, for example AppData, desktop files and icons. As part of this we also have access to the finished binary and libraries, and so can also run tools on them to get a metric of awesomeness. So far, the metrics of awesomeness (here-on known as “kudos”) are:
- AppMenu — has an application menu in line with the GNOME 3 HIG
- HiDpiIcon — installs a 128×128 or larger application icon
- HighContrast — installs hicontrast icons for visually impaired users
- ModernToolkit — uses a modern toolkit like Gtk-3 or QT-5
- Notifications — registers desktop notifications
- SearchProvider — provides a search provider for GNOME Shell or KDE Plasma
- UserDocs — provides user documentation
These attempt to define how tightly the application is integrated with the platform, which is usually a pretty good metric of awesomeness. Of course, some applications like Blender are an island in terms of integration, but of course are awesome. We still need new ideas for this, so ideas are very much welcome.
There are some other “run-time” kudos used as well. These are not encoded by the builder as they require some user information or are too specific to GNOME Software. These include:
- FeaturedRecommended — One of the GNOME Software design team chose to feature this
- HasKeywords — there are keywords in the desktop file used for searching
- HasScreenshots — more than one screenshot is supplied
- MyLanguage — has a populated translation in my locale, or a locale fallback
- PerfectScreenshots — screenshots are perfectly sized, in 16:9 aspect
- Popular — lots of people have downloaded this (only available on Fedora)
- RecentRelease — there been an upstream release in the last year
When added together, the number of stars will correspond roughtly to the number of kudos the application has.
You can verify the kudos your application is getting by doing something like:
killall gnome-software gnome-software --verbose
and then navigating to the details for an application you’ll see on the console:
id-kind: desktop state: available id: blender.desktop kudo: recent-release kudo: featured-recommended kudo: has-screenshots kudo: popular kudo-percentage: 60
Comments (as always) are welcome, as are new ideas on how to test for awesomeness.
And what about just using voting? Something like: “Do you like this app?” or “Do you find this app useful?”. Then you would get absolute numbers. And if someone sees that 100k users have found this app useful, then it’s probably really useful. There would be no negative voting. And KDE/GNOME/whatever fans could vote for their favourite apps as crazy. If the app has a lot of die hard fans, it also says something about its quality.
I don’t find the current solution very good. An app can suck ass, but will get a high rating just because it’s written in GTK 3 and someone wrote a nice AppData file about it.
Unfortunately Facebook has trademark and other protections on the [Like] button, so we’d have to use something that was sufficiently different. Good points otherwise, thanks.
They might have protections on [Like] button, but they certainly haven’t one on [Kudos] button
Well, Google has its +1 button. And others (https://en.wikipedia.org/wiki/Like_button). I don’t think there is a real issue here.
I also favor the idea of just having upvote/recommend/whatever you call it. As explained, the five-star system can be abused easily. Similar things happen with the +1/-1 vote (Vim, stackexchange, etc).
” a cat and mouse thing”
this is true for any voting / starring system. As soon as you ask any people to rank anything, you’re doomed to accept some people will cheat.
However, people do use ranking. For example, going out to restaurants. A restaurant review is highly subjective, it’s pretty impossible to say a restaurant is that good or this good, It depends on personal taste + on choice + on specific day and maybe “which people did serve me that day”. nevertheless trying new restaurants according to average people satisfaction remains the most effective.
But certainly leveraging stuff like “last release date” while exploring softs is extremly useful.
Many of the kudos reference .desktop files; can GNOME Software not install console applications as well?
To go along with ModernToolkit, how about MatchingEnvironment: for GNOME/XFCE/etc users, check if the app uses GTK+ (or provides a GNOME extension in the case of GNOME), and for KDE users, check if the app uses Qt.
ExtendsInstalledApp: extension for a piece of specific piece of software the user already has installed. Should count as a significant negative if they don’t have that software installed, so you don’t prominently see extensions to software you don’t have.
FileHandler: registered to handle a MIME type that the user has no handlers for, but has files of that type (counts double for file types that showed up in the last day). This will help a user find a program to work with a file they just got; for instance, if a user just grabbed an SVG file, show inkscape prominently.
Note that we’re removing HighContrast icons on most apps, as the symbolic one is used by the Shell now instead.
[WORDPRESS HASHCASH] The poster sent us ‘0 which is not a hashcash value.
Yes, I’m pretty sure we consider any app that ships symbolic as having this kudo too.
Same as Jiri, I also think that the best approach will be having one simple button to give possitive feedback.
I’ve been switching many blog sites from 5-star to 1-vote even before facebook came to the game, and used to get more votes overall after the switch.
I just called it “vote” and represented it with a heart symbol.
5-star system is confusing, and some people just pass over it.
You can combine the votes with the number of installs for better representation of popularity.
I really like that you implement automated compliance checks, that’s a new way of finding programs that I’m really looking forward to! I believe this will be very helpful and greatly improving the standardisation. However, I think users will find it important to able to browse the available programs based on userratings as well. I share the concern regarding “gaming” the numbers, but I think that is something that unfortunately apply to any ratingsystem – automated or not.
What if userratings are presented more like a “survey” where the user has to select scores in different category’s (usability, functions, general impression etc) and also justify their negative/positive opinions (unless neutral)? That i think would make it cumbersome regarding the concern, and could give both users and developers as well some helpful metrics.
I think the idea of multiple sources of “awesomeness” is interesting. In Fedora, I’d be interested in opening the “Featured” star to more community participation, possibly through a defined process in the Fedora Workstation Working Group.
I also would be interested in seeing non-anonymous rating as at least _a_ factor, possibly tied to accounts in the GOA system (and including a Fedora account there).
If Software goes the route of making it easy to enable repositories which contain non-free software, I’d very much love to see license being a highly-weighted factor — I think that’s in line with both the GNOME and Fedora missions.
> If Software goes the route of making it easy to enable repositories which contain non-free software, I’d very much love to see license being a highly-weighted factor — I think that’s in line with both the GNOME and Fedora missions.
This.
Putting road-blocks in users’ way, trying to prevent them from installing non-free software will just get them annoyed, and ensure that we remain an OS for a niche.
However, we should definitely guide them towards free software, and a lower rating for non-free software (or a higher rating for free software) is a nice way to do that.
Also, could Software show free software apps a user could install instead of a non-free one?
For example, if someone wants to install Chrome in Software, could we gently tell them that there is this great app called Firefox which works just as good as Chrome, and has the added benefits of being free software?
* The problem is that with this proposal, apps reali useful but that now follow the mark like Gimp (no resent tolkit, no keyword(?), no appmenu, no HIDPI and so) will get really low Kudos even if they are far more usefull than apps with more kudos like a basic calculator fullfilling those.
* Also who define how many kudos point a certain feature will grant to that app?
* Look all the names on all the languages, will not want you implement a word that ended be an insult in other language.
* Also the Popularity thing, distros could have they oun implementations with different rating, maybe define a standard to allow all distros have this field with a consistent puntuation or make it available from gnome side.
Extracting these features seems like a good idea, but perhaps you can just present them like small tags instead of combining them into an incomprehensible scor? Well, perhaps you can combine the integration related ones, but otherwise.
Or have them visible somewhere for those who care, hidden by default because most people really don’t/shouldn’t. :)
An off-by-default, gsettings-cli-only option would work, or even a tooltip when hovering the star rating, or…