• Knud Soerensen (30/30) Apr 17 2007 In the tread DSSS, Dsource and cpan it have been suggested that dsource ...
• Brad Anderson (5/44) Apr 17 2007 Interesting... How does this handle the differences in projects' goals? ...
• Knud Soerensen (3/6) Apr 17 2007 What you can do is select the rankings base on the projects category and
• Walter Bright (9/10) Apr 17 2007 Any ranking system is going to have misleading, unfair and inaccurate
• BCS (7/21) Apr 17 2007 How about whatever system is used, when the user asks for data, they are...
• Knud Soerensen (13/24) Apr 17 2007 Yes, correct.
• Walter Bright (3/6) Apr 17 2007 Sure, that's the big problem with democracy. On the other hand, every
• Knud Soerensen (20/27) Apr 17 2007 First I don't think that you should uncritical compare a comment moderat...
• Manfred Nowak (6/8) Apr 17 2007 Not quite because this implies that the first two projects cannot be
• Knud Soerensen (8/18) Apr 17 2007 Sorry, of cause you can also put the project at the top or at the bottom
• Dan (12/12) Apr 17 2007 I tend to think that both have obvious deficiencies.
• Manfred Nowak (19/21) Apr 18 2007 That is still overly restrictive.
• Carlos Santander (7/46) Apr 17 2007 What would be more important in such a system, the absolute position or ...
• Manfred Nowak (6/8) Apr 17 2007 There exists a bunch of characteristica that might be of interest:

Knud Soerensen <4tuu4k002 sneakemail.com> writes:

In the tread DSSS, Dsource and cpan it have been suggested that dsource should
implement a rating system like on the matlab site and the firefox extension
site. 

I would like to suggest a ranking system instead of the rating system.

In a ranking system every user have a list with rankings,
when ranking a new project the user add it into the ranking between 
two other projects.

Some of the problem i see with a rating system is:

  * Different users might use different bases.
  One user might examine code quality, user interface, documentation etc.
  and give 3 points on basis on this analyses.
  Another might just give 5 point to every project he likes.
  That a ranking system compare the project with each other help make a
  common basis for reference.  


  * It difficult for a user to change his base.
  Imagine a user discover a new brilliant project 
  that is much better than the project he already
  have rated with max points. 
  To rate the new project fairly he would have 
  to go back and re-rate all his rated projects. 
  With ranking he can just put it at the top.

  * Low differentiation between top projects.
  Given most users only rating the projects they like.
  As a result,an extremely good and a very good project will have almost the
same average score and 
  the total sum will tell more about the popularity than how good it is.   
   

A advantages of the ranking system is that it provide 
much more data for data mining.

Your can extract a value and make a simple average like with a ranking.
Your can run most voting system on the data, but you also have data which can 
tell you that n users think that project x is better than project y. etc.

Brad Anderson <brad dsource.org> writes:

Knud Soerensen wrote:
 In the tread DSSS, Dsource and cpan it have been suggested that dsource should
 implement a rating system like on the matlab site and the firefox extension
site. 
 
 I would like to suggest a ranking system instead of the rating system.
 
 In a ranking system every user have a list with rankings,
 when ranking a new project the user add it into the ranking between 
 two other projects.
 
 Some of the problem i see with a rating system is:
 
   * Different users might use different bases.
   One user might examine code quality, user interface, documentation etc.
   and give 3 points on basis on this analyses.
   Another might just give 5 point to every project he likes.
   That a ranking system compare the project with each other help make a
   common basis for reference.  
 
 
   * It difficult for a user to change his base.
   Imagine a user discover a new brilliant project 
   that is much better than the project he already
   have rated with max points. 
   To rate the new project fairly he would have 
   to go back and re-rate all his rated projects. 
   With ranking he can just put it at the top.
 
   * Low differentiation between top projects.
   Given most users only rating the projects they like.
   As a result,an extremely good and a very good project will have almost the
same average score and 
   the total sum will tell more about the popularity than how good it is.   
    
 
 A advantages of the ranking system is that it provide 
 much more data for data mining.
 
 Your can extract a value and make a simple average like with a ranking.
 Your can run most voting system on the data, but you also have data which can 
 tell you that n users think that project x is better than project y. etc.

Interesting... How does this handle the differences in projects' goals?   i.e.
rating Tango higher than Derelict - because they're two different categories,
or goals for end-user use?

BA

Knud Soerensen <4tuu4k002 sneakemail.com> writes:

On Tue, 17 Apr 2007 13:53:59 -0400, Brad Anderson wrote:
 Interesting... How does this handle the differences in projects' goals?   i.e.
 rating Tango higher than Derelict - because they're two different categories,
 or goals for end-user use?

What you can do is select the rankings base on the projects category and
thereby calculate the projects ranking within the category.

Walter Bright <newshound1 digitalmars.com> writes:

Knud Soerensen wrote:
 Some of the problem i see with a rating system is:

Any ranking system is going to have misleading, unfair and inaccurate 
results. So will any ranking system using annointed moderators. Any 
attempt to make fine distinctions based on a ranking system is doomed to 
failure.

But looking at it with a broad brush will be, I predict, fine. Look at 
sites with user or moderator ranking systems - slashdot, digg, imdb, 
amazon, download, youtube. By and large, they are successful at 
separating the ones worth a second look from the ones not.

BCS <BCS pathlink.com> writes:

Walter Bright wrote:
 Knud Soerensen wrote:
 
 Some of the problem i see with a rating system is:

 
 
 Any ranking system is going to have misleading, unfair and inaccurate 
 results. So will any ranking system using annointed moderators. Any 
 attempt to make fine distinctions based on a ranking system is doomed to 
 failure.
 
 But looking at it with a broad brush will be, I predict, fine. Look at 
 sites with user or moderator ranking systems - slashdot, digg, imdb, 
 amazon, download, youtube. By and large, they are successful at 
 separating the ones worth a second look from the ones not.

How about whatever system is used, when the user asks for data, they are 
never given actual numbers. In place of numbers the server converts the 
scores to percentile and then gives them all a random fuzz factor of 
about 5% and then orders them and spits out an order. That way if 2 
things are within 5-10% they will get shuffled back and forth somewhat 
randomly.

Knud Soerensen <4tuu4k002 sneakemail.com> writes:

On Tue, 17 Apr 2007 11:30:35 -0700, Walter Bright wrote:

 Knud Soerensen wrote:
 Some of the problem i see with a rating system is:

 
 Any ranking system is going to have misleading, unfair and inaccurate 
 results. So will any ranking system using annointed moderators. Any 
 attempt to make fine distinctions based on a ranking system is doomed to 
 failure.

Yes, correct.

But from the users ranking in my way, you can generate many types 
of ranking lists and here you also have data of how the project 
compared to each other something which you don't have in ordinary rating. 

A thing I use in eigenpolls.
http://all-technology.com/eigenpolls/

 But looking at it with a broad brush will be, I predict, fine. Look at 
 sites with user or moderator ranking systems - slashdot, digg, imdb, 
 amazon, download, youtube. By and large, they are successful at 
 separating the ones worth a second look from the ones not.

What I don't like with these type of moderation systems is
that a expert option very often is down moderated by what I call 
school kids.  (People which blindly believe what they learned in school.)
 
What i like best is amazons "the users also bought this
books" system but again I have discover better books with my eigenpolls.

Walter Bright <newshound1 digitalmars.com> writes:

Knud Soerensen wrote:
 What I don't like with these type of moderation systems is
 that a expert option very often is down moderated by what I call 
 school kids.  (People which blindly believe what they learned in school.)

Sure, that's the big problem with democracy. On the other hand, every 
other system tried is worse.

Knud Soerensen <4tuu4k002 sneakemail.com> writes:

On Tue, 17 Apr 2007 13:58:23 -0700, Walter Bright wrote:

 Knud Soerensen wrote:
 What I don't like with these type of moderation systems is
 that a expert option very often is down moderated by what I call 
 school kids.  (People which blindly believe what they learned in school.)

 
 Sure, that's the big problem with democracy. On the other hand, every 
 other system tried is worse.

First I don't think that you should uncritical compare a comment moderation
system with a system for electing government.

When I comments on slashdot.org i like to read good comments from expert
on the subject not the most comment option on the subject 
because that option I properly know already. The same way the
ranking/rating on dsource I see as tool to discover good software not just
popular software.

When making decisions for a whole group the option of the majority 
is the best way to go.

But to make a good decision about a problem it is important that 
the good solutions is discovered and analysed and it is this
discovery part we are trying to solve. 

And last if we should compare the ranking system to democracy it is not a
question of for or against but a question about what type of
voting system. http://en.wikipedia.org/wiki/Voting_system 
And my suggestion would be a systems that allow one 
to capture the voters option about each candidate 
and then allow one to calculate the result of a plurality, runoff, approval
or ranked vote.

Manfred Nowak <svv1999 hotmail.com> writes:

Knud Soerensen wrote

 when ranking a new project the user add it into the ranking
 between two other projects.

Not quite because this implies that the first two projects cannot be 
ranked without a third. Ranking ideally only compares two projects.

But ranking has the same vulnerabilities as rating in respect to "bad 
guys".

-manfred 

Knud Soerensen <4tuu4k002 sneakemail.com> writes:

On Tue, 17 Apr 2007 18:52:12 +0000, Manfred Nowak wrote:

 Knud Soerensen wrote
 
 when ranking a new project the user add it into the ranking
 between two other projects.

 
 Not quite because this implies that the first two projects cannot be 
 ranked without a third. Ranking ideally only compares two projects.

Sorry, of cause you can also put the project at the top or at the bottom
of you list.
You can thing of the list as a favorite list with ranking added.

 But ranking has the same vulnerabilities as rating in respect to "bad 
 guys".

I think that it might be best you compute different types of rating 
form the ranking lists. So, gaming might win on one type of rating 
but lose on another type. 

 
 -manfred

Dan <murpsoft hotmail.com> writes:

I tend to think that both have obvious deficiencies.

I would personally implement:

[] Did you use the project
[] Did you examine the source

[1 2 3 4 5] The maintainer is my best friend/family [ comments ]
[1 2 3 4 5] Project satisfied my needs [ comments ]
[1 2 3 4 5] Project was well designed [ comments ]
[1 2 3 4 5] Project was well documented [ comments ]
[1 2 3 4 5] Project performed well [ comments ]

This sort of system could then be fed into a user-defined weighted ranking
system.  If the user only cares about performance, they can ask for it.  If
they insist on documentation, they can ask for that.

That way we can search by our own priorities, rather than the priorities of the
people who voted.  We can also weed out people who vote without knowing shit,
or vote just because they like someone or hate someone else.

Developers can also get feedback from the comments and rankings so they know
what to work on rather than simply getting a "your project sucks" big fat zero.

Manfred Nowak <svv1999 hotmail.com> writes:

Knud Soerensen wrote

 Sorry, of cause you can also put the project at the top or at the
 bottom of you list.

That is still overly restrictive.

In essence you require voters to impose a total ordering on the set
of items they choose to vote on. 

But voters usually only have a partial knowledge of any such set and
therefore only a partial ordering available. Every imposed total
ordering then forces the voter to give out already distorted data. 

The evaluator of such a poll then goes off with this distorted data
and some algorithm that is believed to produce a reliable result on
undistorted data.

The evaluator then preents some pseudo acribic ranking/rating like 
"item1 has score 0.393, item2 has score 0.374, ..." and requires the 
audience to conclude that item1 should be their choice.

But Walter got it right when he wrote in this thread: 
| By and large, they are successful at separating the ones worth a
| second look from the ones not.

Everyone who is able to improve this outcome has found a philosopher's 
stone---and a future without qualms.

-manfred 

Carlos Santander <csantander619 gmail.com> writes:

Knud Soerensen escribió:
 In the tread DSSS, Dsource and cpan it have been suggested that dsource should
 implement a rating system like on the matlab site and the firefox extension
site. 
 
 I would like to suggest a ranking system instead of the rating system.
 
 In a ranking system every user have a list with rankings,
 when ranking a new project the user add it into the ranking between 
 two other projects.
 
 Some of the problem i see with a rating system is:
 
   * Different users might use different bases.
   One user might examine code quality, user interface, documentation etc.
   and give 3 points on basis on this analyses.
   Another might just give 5 point to every project he likes.
   That a ranking system compare the project with each other help make a
   common basis for reference.  
 
 
   * It difficult for a user to change his base.
   Imagine a user discover a new brilliant project 
   that is much better than the project he already
   have rated with max points. 
   To rate the new project fairly he would have 
   to go back and re-rate all his rated projects. 
   With ranking he can just put it at the top.
 
   * Low differentiation between top projects.
   Given most users only rating the projects they like.
   As a result,an extremely good and a very good project will have almost the
same average score and 
   the total sum will tell more about the popularity than how good it is.   
    
 
 A advantages of the ranking system is that it provide 
 much more data for data mining.
 
 Your can extract a value and make a simple average like with a ranking.
 Your can run most voting system on the data, but you also have data which can 
 tell you that n users think that project x is better than project y. etc.

What would be more important in such a system, the absolute position or the 
relative position?

Say I have rated 5 projects (A-E), and someone else has rated 10 (F-O). Has E 
been ranked the same as J or as O?

-- 
Carlos Santander Bernal

Manfred Nowak <svv1999 hotmail.com> writes:

Carlos Santander wrote
 What would be more important in such a system, the absolute
 position or the relative position?

There exists a bunch of characteristica that might be of interest:

http://en.wikipedia.org/wiki/Kemeny-Young_method#Characteristics

Sorrily I could not find any evaluations of these characteristica for 
Knud's ranking method "eigenpoll".

-manfred