Alex Rivera | Logout

how to calculate roc curves?

Asked 2012-10-19T18:26:28.440
8

I write a classifier (Gaussian Mixture Model) to classify five human actions. For every observation the classifier compute the posterior probability to belong to a cluster.

I want to valutate the performance of my system parameterized with a threshold, with values from 0 to 100. For every threshold values, for every observation, if the probability of belonging to one of cluster is greater than threshold I accept the result of the classifier otherwise I discard it.

For every threshold values I compute the number of true-positive, true-negative, false-positive, false-negative.

Than I compute the two function: sensitivity and specificity as

sensitivity = TP/(TP+FN);

specificity=TN/(TN+FP);

In matlab:

plot(1-specificity,sensitivity);

to have the ROC curve. But the result isn't what I expect.

This is the plot of the functions of discards, errors, corrects, sensitivity and specificity varying the threshold of one action.

This is the plot of the functions of discards, errors, corrects, sensitivity and specificity varying the threshold

This is the plot of ROC curve of one action This is the plot of ROC curve

This is the stem of ROC curve for the same action enter image description here

I am wrong, but i don't know where. Perhaps I do wrong the calculating of FP, FN, TP, TN especially when the result of the classifier is minor of the threshold, so I have a discard. What I have to incremente when there is a discard?

matlab classification threshold false-positive roc
Edit
Report

1 Answer

-1

You are trying to draw the curves of precision vs recall, depending on the classifier threshold parameter. The definition of precision and recall are:

Precision = TP/(TP+FP)

Recall = TP/(TP+FN)

You can check the definition of these parameters in: http://en.wikipedia.org/wiki/Precision_and_recall

There are some curves here: http://www.cs.cornell.edu/courses/cs578/2003fa/performance_measures.pdf

Are you dividing your dataset in training set, cross validation set and test set? (if you do not divide the data, it is normal that your precision-recall curve seems weird)

EDITED: I think that there are two possible sources for your problem:

  1. When you train a classifier for 5 classes, usually you have to train 5 distinctive classifiers. One classifier for (class A = class 1, class B = class 2, 3, 4 or 5), then a second classfier for (class A = class 2, class B = class 1, 3, 4 or 5), ... and the fifth for class A = class 5, class B = class 1, 2, 3 or 4).

As you said to select the output for your "compound" classifier, you have to pass your new (test) datapoint through the five classifiers, and you choose the one with the biggest probability.

Then, you should have 5 thresholds to define weighting values that my prioritize selecting one classifier over the others. You should check how the matlab implementations uses the thresholds, but their effect is that you don't choose the class with more probability, but the class with better weighted probability.

  1. As you say, maybe you are not calculating well TP, TN, FP, FN. Your test data should have datapoints belonging to all the classes. Then you have testdata(i,:) and classtestdata(i) are the feature vector and "ground truth" class
answered 2012-10-19T18:46:29.303

Your Answer