I don't deny that. Although I don't think anyone uses surnames as a feature, and that would be pretty blatant. You also need to use census data to really make use of that for less common last names. Anyone going out of their way to do this might as well just discriminate directly instead of this convoluted method.
It may be true that a sufficiently complex machine learned model could learn race as a feature, but again, why would it? It has no prejudice against specific races. Unless you really believe that black people are inherently more likely to get into car accidents, even after controlling for income and education, etc. And even then it doesn't hate black people, it's just doing it's best to predict risk as accurately as possible. It's not charging blacks, as a group, a higher rate than they cost, as a group.
I fail to see why people get so upset at the mere possibility of this. I think they are anthropomorphizing AI as if it was a human bigot that has an irrational hatred for other races, and strongly discriminates against them for no reason. This is more like giving people who live in neighborhoods with slightly higher accident rates, slightly higher insurance rates, to make up for their increased risk. Maybe it correlates with race, maybe it doesn't, it doesn't really matter.
Based on a list of 137 questions[1] the Northpointe system predicts the risk of re-offending, and "blacks are almost twice as likely as whites to be labeled a higher risk but not actually re-offend." Meanwhile, whites are "much more likely than blacks to be labeled lower risk but go on to commit other crimes."
In other words:
- a white person labeled high risk will re-offend 66.5% of the time
- a black person labeled high risk will re-offend 55.1% of the time
- a white person labeled low risk will re-offend 47.7% of the time
- a black person labeled high risk will re-offend 28% of the time.
The model is specifically avoiding race as an input, but still overestimates the danger of black recidivism, while underestimating white recidivism.
It may be true that a sufficiently complex machine learned model could learn race as a feature, but again, why would it? It has no prejudice against specific races. Unless you really believe that black people are inherently more likely to get into car accidents, even after controlling for income and education, etc. And even then it doesn't hate black people, it's just doing it's best to predict risk as accurately as possible. It's not charging blacks, as a group, a higher rate than they cost, as a group.
I fail to see why people get so upset at the mere possibility of this. I think they are anthropomorphizing AI as if it was a human bigot that has an irrational hatred for other races, and strongly discriminates against them for no reason. This is more like giving people who live in neighborhoods with slightly higher accident rates, slightly higher insurance rates, to make up for their increased risk. Maybe it correlates with race, maybe it doesn't, it doesn't really matter.