Trang chủ‎ > ‎IT‎ > ‎Data Science - Python‎ > ‎Tensorflow‎ > ‎

ROC Curve with tensorflow

QCross-entropy loss works better for binary outcomes (cf binomial likelihood).
 Sigmoid units get saturated almost immediately; try using ReLU variants for hidden layers. 
 There's a weak relationship between mini-batch size and learning rate; try playing with both. 
Dropout and other regularization can help improve generalization. And in general, 
there's lots of moving parts to ANNs so debugging is critically important

self.logits = tf.matmul(last_layer_outputs, weights) + bias
self.loss = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(
                                     self.logits,
                                     tf.argmax(self.y_placeholder,1)))
...
self.softmaxed_logits = tf.nn.softmax(self.logits)
(_, self.auc_update_op) = tf.contrib.metrics.streaming_auc(
                              predictions=self.softmaxed_logits,
                              labels=self.y_placeholder,
                              curve='ROC')
Then from asking sklearn.metrics and tensorflow for the AUC:
...
soft_logits = model_instance.sess.run(model_instance.softmaxed_logits,
                                      feed_dict=test_feed)
sklearn_auc = metrics.roc_auc_score(y_true=model_instance.T3SS_data.y_test,
                                    y_score=soft_logits)
tf_auc = model_instance.sess.run(model_instance.auc_update_op,
                                 feed_dict=test_feed)

The streaming_auc keeps accumulating the scores of repeated calls to it, so that you can use it 
for example to get the auc of several batch runs all accumulated. It does not just calculate the current auc.
In order to use it to get just the current auc, you can reset the local variables it uses (e.g. running tf.initialize_local_variables()) 
before running it's update operation.
I'm trying to plot the ROC curve from a modified version of the CIFAR-10 example provided by tensorflow. It's now for 2 classes instead of 10.

The output of the network are called logits and take the form:

[[-2.57313061 2.57966399] [ 0.04221377 -0.04033273] [-1.42880082 1.43337202] [-2.7692945 2.78173304] [-2.48195744 2.49331546] [ 2.0941515 -2.10268974] [-3.51670194 3.53267646] [-2.74760485 2.75617766] ...]
First of all, what do these logits actually represent? The final layer in the network is a "softmax linear" of form WX+b.

The model is able to calculate accuracy by calling
top_k_op = tf.nn.in_top_k(logits, labels, 1)
Then once the graph has been initialized:
predictions = sess.run([top_k_op])
predictions_int = np.array(predictions).astype(int)
true_count += np.sum(predictions) 
...
precision = true_count / total_sample_count
This works fine.
But now how can I plot a ROC curve from this?
I've been trying the "sklearn.metrics.roc_curve()" function (http://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_curve.html#sklearn.metrics.roc_curve) but I don't know what to use as my "y_score" parameter.
Any help would be appreciated!

A'y_score' here should be an array corresponding to the probability of each sample that will be classified as positive (if positive was labeled as 1 in your y_true array)
Actually, if your network use Softmax as the last layer, then the model should output the probability of each category for this instance. But the data you given here doesn't conform with this format. And I checked the example code : https://github.com/tensorflow/tensorflow/blob/r0.10/tensorflow/models/image/cifar10/cifar10.py it seems use the layer called softmax_linear, I know little for this Example but I guess you should process the output with something like Logistic Function to turn it into the probability.
Then just feed it along with your true label 'y_true' to the scikit-learn function:
y_score = np.array(output)[:,1]
roc_curve(y_true, y_score)
--------------------------------------------------------------------------------------------------------------------------------------


Comments