[OT] plugins.intellij.net rating system is weird

Let's do some math:

SQL Query Plugin (http://plugins.intellij.net/plugin/?id=228)
- #1 for "10 Top Downloads"
- 20 ratings, avg. 4.5 stars
- NOT shown in "10 Top Rated"

but

Toggle Test Plugin (http://plugins.intellij.net/plugin/?id=1309)
(or actually just any of the plugins currently listed in "10 Top Rated")
- Position #1 for "10 Top Rated"
- 1 rating with 5 stars

I think this is quite misleading, as the number of votes is apparently not taken into account at all - still I wonder why nobody noticed this before?!

9 comments
Comment actions Permalink

i guess they take in consideration the number of downloads

0
Comment actions Permalink

There are 400+ plugins in repository, and I think following limits can be used for "10 Top Rated" plugins.
1. 1000+ downloads
2. 5+ guys post comment for the plugin

0
Comment actions Permalink

Yann Cébron wrote:

I think this is quite misleading, as the number of votes is
apparently not taken into account at all - still I wonder why nobody
noticed this before?!


I noticed it some time ago and remembered reading about a system for
avoiding these rating anomalies, but I just couldn't find a reference.
Now I finally did, and apparently it is called Bayesian rating.

In Bayesian rating there are no crisp thresholds such as a certain
number of votes or a certain number of downloads. Instead, the final
rating for an item (a plugin) is somewhere between its own average
rating and the average rating of all votes. The more votes a plugin
has, the closer the rating is to its own average; the fewer votes it
has, the closer the rating is to the global average.

See http://www.thebroth.com/blog/118/bayesian-rating for more information.

0
Comment actions Permalink

Jonas, I had something like this in mind. Thanks for the link to the nice article!

0
Comment actions Permalink

See http://www.thebroth.com/blog/118/bayesian-rating
for more information.


+1

Bayesian rating would be much better than the current system.
Any comment from JetBrains?

0
Comment actions Permalink

This would make a lot of sense. Noticed this plugin earlier - http://plugins.intellij.net/plugin/?id=51 - most of the votes are from the author, and at least one of the anonymous votes also appears to be from the author. The rating system should at least:
1. disregard any votes by the plugin's author
2. allow comments but no ratings from anonymous users

Otherwise, the system is meaningless.

0
Comment actions Permalink

Steve Chaloner wrote:

1. disregard any votes by the plugin's author
2. allow comments but no ratings from anonymous users


...and only consider one vote per logged-in user for any plugin.

Well, at least it is possible to comment without voting. Some systems
only seem to be set up for voting and commenting simultaneously, which
of course means that anyone who wants to respond to an earlier comment
is forced to vote again...

I completely agree that the current system is meaningless.

0
Comment actions Permalink

Half a year later, still the same situation.. no change in sight?

0
Comment actions Permalink

Half a year later, still the same situation.. no change in sight?

0

Please sign in to leave a comment.