Ranter
Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Comments
-
NoMad136314yUmm, what?
You want people to report their false positives AND false negatives, instead of just giving you a total accuracy?
I mean, depending on application it may become necessary, l get that. But it's not always the important part. -
hitko31464y@NoMad It's mathematically impossible to prove A <=> B without measuring all intersections between A, B, Aᶜ, and Bᶜ (some of these measurements might be infinite, but only if their complement is measurably finite).
Even if you have high total accuracy, you can always have a significantly unspecific, unsensitive, or imprecise results. -
NoMad136314y@hitko 🤔 I think you need to read your last sentence again, and look up how "accuracy" is actually calculated. you can't have "significance" without having some measure of distance for this "significance". Ans what you're saying is not relevant to error type 2. Type 2 does not discuss significance at all as far as I know.
-
NoMad136314yTo not mistake that significance with the beta significance that we calculate on total errors aka on "number of cases that are type 2".
-
NoMad136314yActually, I just got super confused with the whole thing. Let's scrap this and start over. You're saying you want the type 2 error reported, which is false negatives. While accuracy is (tp+tn)/total. Which by default has the fn in total. But you don't want that. You want the some "distance" of these samples to be reported.
Am I getting that right so far? -
hitko31464y@NoMad (tp+tn)/total doesn't give the whole picture. Consider the following:
(5+100)/110 = 0.954, fp+fn = 5
Now, you might have fn = 5, fp = 0 in which case you only identified 50% positive cases, or you might have fn = 0, fp = 5 in which case there's 50% chance someone you identified as positive is actually negative, which can be equally worthless. And yet, in both cases you have 95.4% accuracy. -
Accuracy is a problem if your labels are unbalanced...which they generally are. That’s why it’s better to report recall or precision. F1 score can be used, but it’s a bit of a blunt tool. Better to be guided by your problem to recall or precision.
Related Rants
We don't talk enough about type 2 error! So many papers everywhere are just pure trash because they don't account for it, and people are so fucking oblivious about it, they don't even catch the obvious ones. Even researchers and publications which are supposed to properly review their articles simply fail to ask the obvious "Did you measure the segment which doesn't fit either of your variables?"
rant
error
type 2 error
research
review