[OT] plugins.intellij.net rating system is weird

Let's do some math:

SQL Query Plugin (http://plugins.intellij.net/plugin/?id=228)
- #1 for "10 Top Downloads"
- 20 ratings, avg. 4.5 stars
- NOT shown in "10 Top Rated"

but

Toggle Test Plugin (http://plugins.intellij.net/plugin/?id=1309)
(or actually just any of the plugins currently listed in "10 Top Rated")
- Position #1 for "10 Top Rated"
- 1 rating with 5 stars

I think this is quite misleading, as the number of votes is apparently not taken into account at all - still I wonder why nobody noticed this before?!

9 comments

i guess they take in consideration the number of downloads

0

There are 400+ plugins in repository, and I think following limits can be used for "10 Top Rated" plugins.
1. 1000+ downloads
2. 5+ guys post comment for the plugin

0

Yann Cébron wrote:

I think this is quite misleading, as the number of votes is
apparently not taken into account at all - still I wonder why nobody
noticed this before?!


I noticed it some time ago and remembered reading about a system for
avoiding these rating anomalies, but I just couldn't find a reference.
Now I finally did, and apparently it is called Bayesian rating.

In Bayesian rating there are no crisp thresholds such as a certain
number of votes or a certain number of downloads. Instead, the final
rating for an item (a plugin) is somewhere between its own average
rating and the average rating of all votes. The more votes a plugin
has, the closer the rating is to its own average; the fewer votes it
has, the closer the rating is to the global average.

See http://www.thebroth.com/blog/118/bayesian-rating for more information.

0

Jonas, I had something like this in mind. Thanks for the link to the nice article!

0

See http://www.thebroth.com/blog/118/bayesian-rating
for more information.


+1

Bayesian rating would be much better than the current system.
Any comment from JetBrains?

0

This would make a lot of sense. Noticed this plugin earlier - http://plugins.intellij.net/plugin/?id=51 - most of the votes are from the author, and at least one of the anonymous votes also appears to be from the author. The rating system should at least:
1. disregard any votes by the plugin's author
2. allow comments but no ratings from anonymous users

Otherwise, the system is meaningless.

0

Steve Chaloner wrote:

1. disregard any votes by the plugin's author
2. allow comments but no ratings from anonymous users


...and only consider one vote per logged-in user for any plugin.

Well, at least it is possible to comment without voting. Some systems
only seem to be set up for voting and commenting simultaneously, which
of course means that anyone who wants to respond to an earlier comment
is forced to vote again...

I completely agree that the current system is meaningless.

0

Half a year later, still the same situation.. no change in sight?

0

Half a year later, still the same situation.. no change in sight?

0

Please sign in to leave a comment.