I would argue `s/pulls the trigger/gives the go ahead/` is intuitively more appropriate. The sailor on the boat that fired off the tomahawk (iirc) was likely not in a position to make an in-the-moment determination
The one that pulls the trigger is often not the one doing any targeting, especially on a strategic strike weapon like a tomahawk. They are receiving a target data package with preprogrammed instructions. Their role is far more... mechanical. They're not seeing the targets at all.
Completely disagree. When we’re talking about something like air, missile, or artillery strikes, the operator of the weapon system may literally never even see the target, on a screen or otherwise. They may simply be entering targeting information provided to them by an operations center. These decisions are social and group decisions, with distributed responsibility, which is why it’s so important to have institutions like the Civilian Protection Center of Excellence which historically has identified and helped implement the policies which prevent these kinds of disasters, but which unfortunately the current administration decided to shut down without a replacement.
Sometimes, the difference between a successful strike on a lawful combatant and disaster is one analyst pointing out that the building the enemy is hiding in is next to a square that’s used for a market on Tuesdays and wouldn’t you know it, it’s Tuesday. That’s not something the person “pulling the trigger,” can always be expected to know, which is why every ops center needs to be designed to elevate those voices, when the time comes
While I think that it's important to do root cause analysis, we need to be very careful not to absolve blame from the people that decided to shift a significant weight of decision-making authority to an AI. An AI can't be tried in a court. There has to be a responsible human for everyone one of its actions, especially when it's killing (-people-) children.
> The Pentagon chief last year slashed offices that didn’t contribute to his goal of “lethality,” including the group that assists in limiting risk to civilians, known as the Civilian Protection Center of Excellence. Around 200 employees who worked on the issue, including at that office, have been reduced by about 90 percent, according to two current and former officials and a person familiar with the effort. The team that handles civilian casualties at Central Command, which oversees the Middle East, has dropped from 10 to one.
Negligent homicide in pursuit of their DEI goals, basically.
Taking AI out of the equation, surely these same kinds of mistakes have been made in the recent past. As in, not doing due diligence and mistakenly killing civilians
(setting aside when civilians are purposefully harmed which is a conversation about morals rather than the capability to make correct decisions)
What I am getting at is, has AI really made the problem worse or has the mechanism just changed?
We are increasingly seeing "Sorry out <Black box> made a mistake, the developers are fixing it" rather than any one person taking responsibility for the decision. This is a growing problem that is made worse by AI. Over-inflated trust in AI output is also documented issue; "The perfect billion dollar targeting system with a 99.9% success-rate (on its verification dataset)" just made a "fluke"; "Nothing could have stopped this tragedy"
When it was just called a "database query" or "algorithmic estimation", it was more implied that the actual decision is left to be made and that no analysis had actually been done yet. But when AI preemptively says "I've analyzed every source of data and come to the conclusion that this ship is carrying nuclear bomb material"; it really sounds urgent doesn't it? It probably doubled down when asked about it.
DOGE strikes again. Edit actually looks like it was Hegseth himself, king Testosterone.
> Staffing on civilian harm mitigation teams across the U.S. Department of Defense had also fallen by roughly 90% over recent years, shrinking to fewer than 20 people overall. Centcom’s group shrank from 10 people to one.
nice, Karp really reeled in a nice big fish, and ofcourse he is going to blame OpenAI/Anthropic for giving the targets, this explains his CNBC rants about the frontiers, his tool was the trigger in a killchain of hallucination.
> It found the U.S. “failed in its obligation to do everything feasible to verify” that the school was a military objective and that the failure “went beyond mere negligence.” The report said the United States “directed the strikes at the building of the school while being aware of a substantial risk of striking a civilian object and acting recklessly as regards the possibility that this would happen.”
Reading the details, "AI" doesn't really seem like the culprit -it's a scapegoat.
The intelligence that it was no longer a military target never entered the target database, the team that was responsible for vetting the target list was gutted, and said team was never even consulted.
The White House wanted 1000 targets and pulled from their database without any due diligence. Whether it was an AI call or an SQL query - this was from pure human maliciousness and incompetence.
> “No more politically correct and overbearing rules of engagement,” - Hegseth
...
> Hegseth and the commander in charge of the Iran operation, Adm. Brad Cooper, were questioned about the strike before Congress. Hegseth said the U.S. does not target civilians and would investigate
So the administration does not intentionally target civilians, they have just stopped caring about who a civilian is.
Well Israel has defined "combatant" as "male between 14 and 55", and the Western world more or less shrugged and moved on without missing a beat. I'm not optimistic.
Instead of worrying about woke things like "is this person actually a combatant actively fighting" they want the much simpler rule of "everyone we kill was actually a terrorist that deserves to die".
Basically they stopped caring those woke UN rules about war crimes.
Now, ask yourself if you want AI taking decisions around your life... it will pull numbers... someone will have a "goal" of 1000 "hits" and there you go, 2-3 months in jail working to check what went wrong...
But don't worry, the other P AI system will ensure that the first one doesn't do mistakes...
Important context is that the context wasn’t even in a connected system. Unclear if the human analysts would have even caught this, or if they would have known to check this unconnected system.
> Some US intelligence analysts took notice, the officials said. Bloomberg reported in June that one analyst detected changes to the site as early as 2019, and logged remarks about the modifications in a system that wasn’t connected to the primary military intelligence database, which informs targeting
"In the final days of February, the Trump administration demanded an overwhelming aerial assault. More than 1,000 Iranian targets were hit within the first 24 hours alone. That order compressed the time frame for finalizing targets, the officials said, instilling an urgency that — combined with gaps in underlying information about the site, cuts in civilian-protection personnel and an overreliance on artificial-intelligence technology — set the stage for the deadly error."
-------------------
Maven is an untested tool for rapidly selecting targets under time pressure during a war. It's reason for existing is speed. If you're in a shooting war, the ability to select good military targets faster than your enemy, but not necessarily with zero error, is an advantage.
Planning a first strike is a different situation. Time pressure should not be a significant factor. Humans should have been doing their due diligence on every single target. If an untested tool for rapid mid-battle target selection was relied upon completely because of a hasty executive order, then the responsibility falls entirely on the executive.
I think that’s the important distinction. Mistakes under pressure do happen. Unnecessary pressure is not a mistake, though, it’s a crime in this context. They could start planning earlier or commit to action later.
Except laundering our biases & discriminations, the unaccountability seems to be the purpose of a huge huge swarth of AI usage. To eliminate accountability. That sure seems like A Feature Not A Bug to the worlds worst people.
> The Minab site, which was cataloged as an Islamic Revolutionary Guard Corps facility due to outdated data, was fed into Maven with other candidates and came out as a recommended day-one target. Target-list work that once took hours was condensed into minutes.
I feel like we're optimizing the wrong metric here.
Even ignoring the risk of civilian casualties (which to be clear should be weighed heavily) it’s just bad ROI. Like each missile probably costs more than the extra targeting work they should have done.
Yeah, totally, I’m being cautious because I don’t know how large the team is that would spend a few more hours, but under most assumptions you’re right.
Is that really the case? Stories like this always have all sides muddying waters.
The terrorists are well known to use children and the rest of the public as hostage meatshields, and the US military is always looking for something/someone else to blame for the dirty work they have to do.
The broader public is so polarized by performative politics and botspam that they eat these stories up, but nothing I'm saying is secret. It's been this way since forever. Slow news today?
I'm not even defending AI at all and think these are crappy choices, but compared to humans gathering intelligence who can go rogue, what alternative does anyone really have?
What fucking terrorists? We're talking about a country bombing another country's school. A country with a population of 93 million. Are you trying to insinuate that all 93 million Iranians are terrorists?
Maybe you meant the American terrorists?
> Stories like this always have all sides muddying waters.
All sides agree that the US bombed a school full of children, actually.
> Officials said some users expected Maven to flag stale records or contradictions in the intelligence assembled for potential targets, though it is not clear why they thought the system would do that
This is what happens when people who don't understand AI and its limitations embrace it wholeheartedly, thinking it's going to behave like The ultimate analyst. Except it's not. Its got blind spots in more places than anyone knows. And not to mention the system itself has multiple areas of failure.
Chatbots in general can be quite opaque in how they operate, the biggest issue with agent products is communicating it's capabilities and limitations to users. If you have a bank app it's pretty clear whether or not it can send wire transfers, but a if that's a chatbot, users will have to think to ask it (and even then it could hallucinate)
No, this is what happens when software developers don't understand how users actually use their products.
This was a totally expected outcome, similar to how CV cameras arrest the wrong minorities in neighboring states 100s of miles away because it told the officer it was them; or how health insurance companies use ML algorithms to deny Americans care.
It's going to happen in other sectors of life and it will continue killing innocent civilians.
The pentagon says it was Palantir software’s fault, Palantir says it was the fault of bad data input to their software. Where’s the accountability by both organizations? How did we get to a world where military decisions leading to innocent deaths are responded to as if it was just another miscommunication implementing a B2B SaaS platform.
Small children often develop coping mechanisms for their lack of agency. It may be part of these people’s worldview since early childhood that anyone who opposes them should magically die.
you can also develop pro-social coping mechanisms like open-mindedly talking to someone who opposes you and trying to understand their worldview because you trust that it might expand your own
but, yeah, I doubt pro-social traits are common at the top of any hyperstrict hierarchical organization like the USAF
The new world is complex and AI technology has outpaced the regulation cycle, so it's inevitable that we come to a place where you don't have accountability and can never have it because 0) the courts can't handle the case, they don't have the frameworks for it, so it takes longer time 1) you can overload the courts with slop, 2) by the time there is a decision the world has moved on and technology has moved on
I disagree with this framing. Nothing in accountability has really changed other than our willingness to enforce it.
Replace "AI" with any other tool or software in your scenario, and the outcome is the same. The same people and organizations are accountable, it's just that we currently live in a world without accountability in the highest places where it's most needed, and that's unrelated to AI.
You disagree that there is no difference between the amount of material you can generate today compared to 10 years ago?
Or with that the EU AI Act has been redundant and postponed in 2023 because it was drafted on ResNets, and then redundant and postponed in 2025 because it was drafted on ChatGPT 3.5, and then this year probably again because it didn't have agents included?
Or with that other countries that getting their companies hacked by AI models are ignoring any kind of prosecution of said companies and delegating this to the US?
first, closer to 250 people were murdered by the $4 Million tomahawk
second, imagine if Iran even accidentally took out an elementary school in the US filled with 250 people
third, there was a human group that was supposed to review targets regardless of whomever picked them, the group was purposely disbanded, that's negligent homicide at best but more clearly a war crime on purpose
fourth, let's say it was 250 soldiers somehow among a civilian setting, IT CHANGED NOTHING, this war will change nothing, they cannot destroy pickaxe mountain, they will abandon the war in 2029 like the US abandoned Iraq, Afghanistan, Vietnam and Korea, WAR CHANGES NOTHING, you cannot "War to Peace"
Never so happy to see such an obvious scapegoat get thrown under the bus. I don't buy it for a second, but no tears will be shed if the stock market does.
They are also openly and proudly using this system along with Starlink to assist Ukraine, who a couple months ago killed a bunch of teens in a college in East Ukraine under very similar circumstances that a simple Google or Yandex maps search would have red-flagged plus on the ground real human intel. Unless the point was actually to kill kids.
Okay, that one is fake because Russia bad. Iran one is real because Orange man bad. Same ISR and targeting systems. Mental gymnastics is a hell of thing.
Thank you for reminding me about these posts, time has proved my statements true. Especially after the Iran war exposed these outdated systems via direct hits on US bases, which can't be narrative'd away like in Ukraine.
The S-400 is fairing well? F-35s apparently walked all over Iran's air defense network without much challenge.
> Especially after the Iran war exposed these outdated systems via direct hits on US bases…
Interception rates have been extremely high. The problem, of course, is the ammo shortage and drone cost asymmetry. Which has been widely documented, and presents an issue for the S-400s too.
Israel likes to argue that Hamas embeds fighters with first responders, which even if so, they're still responding to injured. But Israel considers this not even collateral damage but justified.
Mind you Israel is currently funding a lobbyist organization in the UK to flag Médecins Sans Frontières (Doctors Without Borders) as a terrorist group (because many years ago a couple of people who had affiliations with other anti-Israel groups were found to be also affiliated with MSF).
Yeah, I've heard about it, but presumable Americans soldiers still care about IHL at least to some degree, even though they've historically not bothered with the Geneva convention and have tortured people.
It's easy to blame the military and the higher-ups, after all they made this choice. But more relevant to HN, this couldn't have happened without technology professionals working for Palantir.
> A Palantir spokesperson told Bloomberg the company “is not responsible for the underlying data nor identifying intelligence deficiencies” and that there is no evidence its software was at fault.
If you work for Palantir, you are writing software that kills people. Software that, in this case, killed 123 children. What salary is worth that?
I encourage others who work in the public sector, whether directly or through a contractor, to read and internalize Don't Get Distracted by Caleb Hearth [0]. I think about it nearly every day.
>It's easy to blame the military and the higher-ups
My brother in Cthulu, they're the ones making the life or death decision that killed people, who else should be to blame?
>But more relevant to HN, this couldn't have happened without technology professionals working for Palantir.
If you wanna go the historical tech-tree scapegoat route then none of this would have happened if DARPA wouldn't have invented the internet, or if Thomas Edison wouldn't have invented the lightbulb, or if the fish haven't ventured onto land and evolve into us land mammal apes who invented sharpened sticks as murder objects.
I know, right? I'm not even sure if this is the biggest installation on Earth. If it's possible for us, it's possible for any number of private military contractors. I bet there's a similar capability in the hands of every advanced county on Earth. Anyway, shaddup, the focusing crystal's not calibrating right, and tomorrow I have present the targeting parameters to Dr. Evil.
Look, if Palantir had any redeeming qualities then I might agree with you. Unlike electricity which runs our computers, homes, cars, etc. What does Palantir do? It spies on people and helps governments to create "people we don't like" lists. In the absolute best case it's a "minority report" engine, but it's still just there to try and stop pre-crime (not crime).
This isn't the case where we are talking about a company doing anything socially good. It's simply there to help authoritarian governments.
Palantir only exists because government wants and funds these types of tools. If it Palantir wouldn't exist then another dozen companies with different names would exist that would create the same Torment Nexus for the government due to inexhaustible demand for such tools.
>It's simply there to help authoritarian governments.
The fact that you're so close yet so far away to figuring out who's to blame still boggles my mind.
Are you implying that the demand for these evil services absolves the suppliers from any moral responsibility? Simply because “if we don’t do this somebody else will”.
It’s an evil service and no matter who requests it or supplies it, doing so is an evil act.
>Are you implying that the demand for these evil services absolves the suppliers from any moral responsibility?
No, you are. I'm implying evil are (primarily) those who use it, not (just) those who make it. You can try to make whatever you want, if nobody will buy it, then you won't have money to make it. Palantir only exists because governments are evil and willing to buy it. If you want Palantir to not exist you need to hold your government accountable.
Are gun makers evil and responsible whenever someone gets killed by a murderer, or are those who pull the trigger responsible?
I absolutely knew you’d bring up guns and gun making but it’s not a valid comparison. If the gun companies were also involved in the actual shooting of people part that would work.
Palantir were in no way obliged or coerced into making an evil thing, they just decided being evil for money was ok.
Whether you may/should feel responsible depends on how the software works, I guess. Did it automatically recommend some drug, somewhat confidently, with gray text on gray background saying "AI sometimes makes mistakes"?
So I think we should behave responsibly when making software, even more now with AI
It depends. Did you write a bug that caused the wrong drug to get ordered? Did you design or implement an algorithm that recommends drugs, and the algorithm was a bad one? Did you design or implement a confusing UI flow that led to the doctor ordering the wrong drug?
If so, I'd say yes, you share some of the responsibility. Legally, I'm sure you'd be fine, but if that were me, I'd feel responsible.
The doctor of course is responsible too; they're the one who didn't double-check that it was an appropriate drug. But multiple people can share responsibility for a bad thing.
The govt is investing in the military whether you like it or not, there is a set amount of GDP going into the sector regardless of your moral choice.
Would you rather not have someone make good software that does not misidentify schools at sites? Or should we outsource this to worse engineers who don't care about morals and don't find this metric meaningful?
That's a reasonable argument, but I don't think I could make the decision to work on software that's designed to kill people. I consider myself a pretty good software developer, and yes, I could probably do a better job building software to help kill people than many others, but I don't think "if you don't do it, someone less capable will, and more innocent people will get killed" would sway me. I just don't want to be involved in that, and I'm lucky that I don't have to be.
Well you made a personal choice to stick your head in the sand to not have to deal with the hard emotions, and that's fine. But to then pass judgement on people who don't sounds a bit unfair given this.
Let's not forget the shareholders who invested in this company knowing what they do. That includes some of my colleagues. I told them that whoever created and named this company is clearly a psycho.
Of course it could have happened. You don't even need targeting for this to happen. You can drop bombs at random and kill whoever is unlucky enough to be in the wrong place. The US has a long history of doing it (Japan, Vietnam, ...).
So yeah, the military and whoever decided that a war was a good idea are to blame.
People working at Palantir are as much responsible for the killing of these 123 children than seatbelt manufacturers are responsible for the people who died because their product failed. That is, because they didn't do their job correctly, people died, definitely worth feeling bad about. However, should people stop working in seatbelt manufacturing because of it? Absolutely not, we need seatbelts, because without them, things would be worse.
Palantir, or whoever does target identification, is like the seatbelt of the reckless drivers of the militaries, who, for the lack of precise information, would just carpet bomb the place. It is not a hypothetical, it is how the US has worked before they had precision ammunition.
This is essentially a trolley problem. The war is the trolley, people who work for Palantir are the ones pulling levers. And if there is no one to pull the lever, the trolley is gong on a track with a lot of people on it.
> I encourage others who work in the public sector, whether directly or through a contractor, to read and internalize Don't Get Distracted by Caleb Hearth.
I know a guy who works for well-known hardware company. He isn't a software dev, but close enough, works with Verilog on hardware design and verification. One day he tells me how he is working on video processing module and he doesn't know if that module will be part of surveillance system, missile guidance or nanny cam. It's possible that even middle management doesn't know more than him, and very likely the module will find uses in multiple products. He is a small cogs in a large machine, maybe his work will save a baby's life, maybe it will kill a person, he cannot know.
ultimately someone or a group of people need to be held criminally accountable for this crime. it is on the magnitude of the my lai massacre and there should be justice for the victims
There is no justice. And Iran will probably blow up some innocent civillians in response the opportunity arrives, instead of targeting the actual culprits.
If true, there was no verification of the targets selected by AI, meaning AI is the final arbiter of the deadly use of force. Another Rubicon passed.
Trump pretends like it didn’t happen: “We might never know what really happened there” is a frustrating lack of responsibility.
Still missing, as it always is from all of these kinds of articles, is: How common is this error rate compared to pure human evaluation and decision-making? I’m not saying I support AI-assisted killing, but why is nobody comparing it to human failure rates? If the article is true, then there were 1,000 targets hit in the first 24 hours. If the majority of the legwork and selection was done by AI, how many schools, or equivalents, would one expect to be wrongly hit if all the targeting work were done by humans only?
Meanwhile, the Houthi's are trying to build missile technology with AI. We're only seeing the beginning of the consequences of an over-reliance on AI for military purposes.
I often see the argument "humans make mistakes too" when talking about AI, self driving cars, autonomous drones and such...
yes, they do, but most of the time they go to trial, there are consequences, there might even be justice.
when AI/self driving cars/drones make mistakes, which might very well be more rare in absolute [citation needed], there is (seems to me) a tendency to shrug it off, almost as if it were force majeure.
Did nobody here actually look at the school in question? The Iranians built a little kids school INSIDE a military base, a really important one at that. Same paint, style, literally ~20ft away.
A tragedy but it's hard to argue that it wasn't the intention of the Iranians for this exact senario to happen.
The USA has dozens of schools on military bases that house families of our soldiers. Is the intention that they be struck for propaganda? No. It's to educate their children just like in Iran.
No this is BS, they murdered these children on purpose to reenact Purim (when they started the war). Purim is a celebration of the murder of Persian civilians, including children by Jewish forces. Several Zionist venture capitalists such as Sam Lessin were literally celebrating this.
allears | 2 hours ago
ambicapter | 2 hours ago
From TFA.
esseph | 2 hours ago
vfclists | 2 hours ago
verdverm | 2 hours ago
esseph | 2 hours ago
derektank | 2 hours ago
Sometimes, the difference between a successful strike on a lawful combatant and disaster is one analyst pointing out that the building the enemy is hiding in is next to a square that’s used for a market on Tuesdays and wouldn’t you know it, it’s Tuesday. That’s not something the person “pulling the trigger,” can always be expected to know, which is why every ops center needs to be designed to elevate those voices, when the time comes
mito88 | 2 hours ago
jacquesm | 2 hours ago
colingauvin | 2 hours ago
I cannot believe this has come to pass.
jacquesm | 2 hours ago
teekert | 2 hours ago
Jtsummers | 2 hours ago
> An AI can be tried in a court.
An AI can't be tried in a court.
colingauvin | an hour ago
ceejayoz | 2 hours ago
> The Pentagon chief last year slashed offices that didn’t contribute to his goal of “lethality,” including the group that assists in limiting risk to civilians, known as the Civilian Protection Center of Excellence. Around 200 employees who worked on the issue, including at that office, have been reduced by about 90 percent, according to two current and former officials and a person familiar with the effort. The team that handles civilian casualties at Central Command, which oversees the Middle East, has dropped from 10 to one.
Negligent homicide in pursuit of their DEI goals, basically.
stuaxo | 2 hours ago
ceejayoz | 2 hours ago
Negligence was involved.
DEI for drunk white men who look good on TV got us Hegseth, who's now rather hilariously requiring gender-affirming treatment for the troops. (https://www.pbs.org/newshour/world/pentagon-moves-forward-wi...)
dmix | 2 hours ago
shepherdjerred | 21 minutes ago
(setting aside when civilians are purposefully harmed which is a conversation about morals rather than the capability to make correct decisions)
What I am getting at is, has AI really made the problem worse or has the mechanism just changed?
aDyslecticCrow | 9 minutes ago
When it was just called a "database query" or "algorithmic estimation", it was more implied that the actual decision is left to be made and that no analysis had actually been done yet. But when AI preemptively says "I've analyzed every source of data and come to the conclusion that this ship is carrying nuclear bomb material"; it really sounds urgent doesn't it? It probably doubled down when asked about it.
https://edition.cnn.com/2026/09/18/politics/us-military-ai-f...
feverzsj | 2 hours ago
throwaway198846 | 2 hours ago
I would like to know how you know this
lurk2 | 2 hours ago
bix6 | 2 hours ago
> Staffing on civilian harm mitigation teams across the U.S. Department of Defense had also fallen by roughly 90% over recent years, shrinking to fewer than 20 people overall. Centcom’s group shrank from 10 people to one.
https://www.propublica.org/article/hegseth-civilian-harm-mit...
Edit: hilarious downvotes. Does DOGE get Elon Stan bots or something?
krapp | 2 hours ago
verdverm | 2 hours ago
derektank | 2 hours ago
verdverm | 2 hours ago
https://en.wikipedia.org/wiki/Franz_Conrad_von_H%C3%B6tzendo...
macintux | 2 hours ago
https://gizmodo.com/almost-started-a-war-us-military-nearly-...
seizethecheese | an hour ago
chews | 2 hours ago
verdverm | 2 hours ago
France gave them the boot, you can too
vikas123456789 | 2 hours ago
jacquesm | 2 hours ago
andsoitis | 2 hours ago
ceroxylon | 2 hours ago
dfxm12 | an hour ago
sys32768 | 2 hours ago
angoragoats | 2 hours ago
elAhmo | 2 hours ago
igleria | 2 hours ago
1239971 | 2 hours ago
legitster | 2 hours ago
Reading the details, "AI" doesn't really seem like the culprit -it's a scapegoat.
The intelligence that it was no longer a military target never entered the target database, the team that was responsible for vetting the target list was gutted, and said team was never even consulted.
The White House wanted 1000 targets and pulled from their database without any due diligence. Whether it was an AI call or an SQL query - this was from pure human maliciousness and incompetence.
bix6 | 2 hours ago
legitster | 2 hours ago
...
> Hegseth and the commander in charge of the Iran operation, Adm. Brad Cooper, were questioned about the strike before Congress. Hegseth said the U.S. does not target civilians and would investigate
So the administration does not intentionally target civilians, they have just stopped caring about who a civilian is.
andrepd | an hour ago
ceejayoz | an hour ago
https://en.wikipedia.org/wiki/Military-age_male
noncoml | an hour ago
What a fucking moron
cogman10 | an hour ago
Basically they stopped caring those woke UN rules about war crimes.
xlayn | 2 hours ago
But don't worry, the other P AI system will ensure that the first one doesn't do mistakes...
toomuchtodo | 16 minutes ago
UnitedHealth uses AI model with 90% error rate to deny care, lawsuit alleges - https://news.ycombinator.com/item?id=38299921 - November 2023 (58 comments)
https://litigationtracker.law.georgetown.edu/wp-content/uplo...
8note | 6 minutes ago
jmathai | an hour ago
seizethecheese | an hour ago
> Some US intelligence analysts took notice, the officials said. Bloomberg reported in June that one analyst detected changes to the site as early as 2019, and logged remarks about the modifications in a system that wasn’t connected to the primary military intelligence database, which informs targeting
beloch | an hour ago
-------------------
Maven is an untested tool for rapidly selecting targets under time pressure during a war. It's reason for existing is speed. If you're in a shooting war, the ability to select good military targets faster than your enemy, but not necessarily with zero error, is an advantage.
Planning a first strike is a different situation. Time pressure should not be a significant factor. Humans should have been doing their due diligence on every single target. If an untested tool for rapid mid-battle target selection was relied upon completely because of a hasty executive order, then the responsibility falls entirely on the executive.
cgio | 8 minutes ago
jauntywundrkind | an hour ago
Except laundering our biases & discriminations, the unaccountability seems to be the purpose of a huge huge swarth of AI usage. To eliminate accountability. That sure seems like A Feature Not A Bug to the worlds worst people.
jmathai | 2 hours ago
I feel like we're optimizing the wrong metric here.
seizethecheese | an hour ago
ambicapter | an hour ago
These missiles cost a couple million each.
seizethecheese | an hour ago
jacquesm | 2 hours ago
bix6 | 2 hours ago
https://www.propublica.org/article/hegseth-civilian-harm-mit...
FireBeyond | 2 hours ago
sublinear | 2 hours ago
The terrorists are well known to use children and the rest of the public as hostage meatshields, and the US military is always looking for something/someone else to blame for the dirty work they have to do.
The broader public is so polarized by performative politics and botspam that they eat these stories up, but nothing I'm saying is secret. It's been this way since forever. Slow news today?
I'm not even defending AI at all and think these are crappy choices, but compared to humans gathering intelligence who can go rogue, what alternative does anyone really have?
23j423j423hj | 2 hours ago
ceejayoz | 2 hours ago
Which makes the administration's near complete silence on the strike rather suspicious, no?
bix6 | 2 hours ago
Yes it is really the case here.
applfanboysbgon | 2 hours ago
What fucking terrorists? We're talking about a country bombing another country's school. A country with a population of 93 million. Are you trying to insinuate that all 93 million Iranians are terrorists?
Maybe you meant the American terrorists?
> Stories like this always have all sides muddying waters.
All sides agree that the US bombed a school full of children, actually.
imranq | 2 hours ago
This is what happens when people who don't understand AI and its limitations embrace it wholeheartedly, thinking it's going to behave like The ultimate analyst. Except it's not. Its got blind spots in more places than anyone knows. And not to mention the system itself has multiple areas of failure.
pants2 | 2 hours ago
shimman | 2 hours ago
This was a totally expected outcome, similar to how CV cameras arrest the wrong minorities in neighboring states 100s of miles away because it told the officer it was them; or how health insurance companies use ML algorithms to deny Americans care.
It's going to happen in other sectors of life and it will continue killing innocent civilians.
banubis | 2 hours ago
hyperhello | 2 hours ago
paimapi | 55 minutes ago
you can also develop pro-social coping mechanisms like open-mindedly talking to someone who opposes you and trying to understand their worldview because you trust that it might expand your own
but, yeah, I doubt pro-social traits are common at the top of any hyperstrict hierarchical organization like the USAF
augment_me | 2 hours ago
Really good time to do various crime.
croon | an hour ago
Replace "AI" with any other tool or software in your scenario, and the outcome is the same. The same people and organizations are accountable, it's just that we currently live in a world without accountability in the highest places where it's most needed, and that's unrelated to AI.
augment_me | an hour ago
Or with that the EU AI Act has been redundant and postponed in 2023 because it was drafted on ResNets, and then redundant and postponed in 2025 because it was drafted on ChatGPT 3.5, and then this year probably again because it didn't have agents included?
Or with that other countries that getting their companies hacked by AI models are ignoring any kind of prosecution of said companies and delegating this to the US?
forshaper | 41 minutes ago
gambiting | 2 hours ago
testing22321 | 14 minutes ago
angoragoats | 2 hours ago
sedan_baklazhan | 2 hours ago
RIMR | 2 hours ago
Zigurd | 2 hours ago
onlyrealcuzzo | 2 hours ago
> That didn't happen.
> And if it did, it wasn't that bad.
> And if it was, it was AI's fault, not mine.
ceejayoz | 2 hours ago
It's gonna say something like "they were all military-aged males". https://en.wikipedia.org/wiki/Military-age_male
nilamo | an hour ago
moralestapia | 2 hours ago
https://en.wikipedia.org/wiki/2026_Minab_school_attack
This looks as much, if not more, like terrorism to me than other incidents the media endlessly portrays.
The US has done so much evil around the world. Enough is enough.
And yeah, Alex Karp is a complete idiot. I cannot imagine how vile one has to be to make a profit off this.
ck2 | 2 hours ago
second, imagine if Iran even accidentally took out an elementary school in the US filled with 250 people
third, there was a human group that was supposed to review targets regardless of whomever picked them, the group was purposely disbanded, that's negligent homicide at best but more clearly a war crime on purpose
fourth, let's say it was 250 soldiers somehow among a civilian setting, IT CHANGED NOTHING, this war will change nothing, they cannot destroy pickaxe mountain, they will abandon the war in 2029 like the US abandoned Iraq, Afghanistan, Vietnam and Korea, WAR CHANGES NOTHING, you cannot "War to Peace"
underlipton | 2 hours ago
cpursley | 2 hours ago
https://en.wikipedia.org/wiki/2026_Starobilsk_strike
allknowingfrog | 2 hours ago
cpursley | an hour ago
phainopepla2 | 16 minutes ago
FireBeyond | 2 hours ago
ceejayoz | an hour ago
https://news.ycombinator.com/item?id=47441817
https://news.ycombinator.com/item?id=36559017
etc.
cpursley | an hour ago
ceejayoz | an hour ago
The S-400 is fairing well? F-35s apparently walked all over Iran's air defense network without much challenge.
> Especially after the Iran war exposed these outdated systems via direct hits on US bases…
Interception rates have been extremely high. The problem, of course, is the ammo shortage and drone cost asymmetry. Which has been widely documented, and presents an issue for the S-400s too.
impossiblefork | 2 hours ago
There needs to be some protection for first responders and others helping the injured.
FireBeyond | 2 hours ago
Mind you Israel is currently funding a lobbyist organization in the UK to flag Médecins Sans Frontières (Doctors Without Borders) as a terrorist group (because many years ago a couple of people who had affiliations with other anti-Israel groups were found to be also affiliated with MSF).
impossiblefork | an hour ago
63 | 2 hours ago
> A Palantir spokesperson told Bloomberg the company “is not responsible for the underlying data nor identifying intelligence deficiencies” and that there is no evidence its software was at fault.
If you work for Palantir, you are writing software that kills people. Software that, in this case, killed 123 children. What salary is worth that?
I encourage others who work in the public sector, whether directly or through a contractor, to read and internalize Don't Get Distracted by Caleb Hearth [0]. I think about it nearly every day.
[0] https://calebhearth.com/dont-get-distracted
PorciiVorbesc | 2 hours ago
My brother in Cthulu, they're the ones making the life or death decision that killed people, who else should be to blame?
>But more relevant to HN, this couldn't have happened without technology professionals working for Palantir.
If you wanna go the historical tech-tree scapegoat route then none of this would have happened if DARPA wouldn't have invented the internet, or if Thomas Edison wouldn't have invented the lightbulb, or if the fish haven't ventured onto land and evolve into us land mammal apes who invented sharpened sticks as murder objects.
whatshisface | an hour ago
cogman10 | an hour ago
This isn't the case where we are talking about a company doing anything socially good. It's simply there to help authoritarian governments.
PorciiVorbesc | an hour ago
>It's simply there to help authoritarian governments.
The fact that you're so close yet so far away to figuring out who's to blame still boggles my mind.
smackeyacky | 15 minutes ago
It’s an evil service and no matter who requests it or supplies it, doing so is an evil act.
PorciiVorbesc | 10 minutes ago
No, you are. I'm implying evil are (primarily) those who use it, not (just) those who make it. You can try to make whatever you want, if nobody will buy it, then you won't have money to make it. Palantir only exists because governments are evil and willing to buy it. If you want Palantir to not exist you need to hold your government accountable.
Are gun makers evil and responsible whenever someone gets killed by a murderer, or are those who pull the trigger responsible?
smackeyacky | 4 minutes ago
Palantir were in no way obliged or coerced into making an evil thing, they just decided being evil for money was ok.
gameshot911 | an hour ago
ekzy | an hour ago
Whether you may/should feel responsible depends on how the software works, I guess. Did it automatically recommend some drug, somewhat confidently, with gray text on gray background saying "AI sometimes makes mistakes"?
So I think we should behave responsibly when making software, even more now with AI
kelnos | 58 minutes ago
If so, I'd say yes, you share some of the responsibility. Legally, I'm sure you'd be fine, but if that were me, I'd feel responsible.
The doctor of course is responsible too; they're the one who didn't double-check that it was an appropriate drug. But multiple people can share responsibility for a bad thing.
augment_me | an hour ago
Would you rather not have someone make good software that does not misidentify schools at sites? Or should we outsource this to worse engineers who don't care about morals and don't find this metric meaningful?
kelnos | 59 minutes ago
augment_me | 47 minutes ago
petre | an hour ago
GuB-42 | an hour ago
So yeah, the military and whoever decided that a war was a good idea are to blame.
People working at Palantir are as much responsible for the killing of these 123 children than seatbelt manufacturers are responsible for the people who died because their product failed. That is, because they didn't do their job correctly, people died, definitely worth feeling bad about. However, should people stop working in seatbelt manufacturing because of it? Absolutely not, we need seatbelts, because without them, things would be worse.
Palantir, or whoever does target identification, is like the seatbelt of the reckless drivers of the militaries, who, for the lack of precise information, would just carpet bomb the place. It is not a hypothetical, it is how the US has worked before they had precision ammunition.
This is essentially a trolley problem. The war is the trolley, people who work for Palantir are the ones pulling levers. And if there is no one to pull the lever, the trolley is gong on a track with a lot of people on it.
stanac | an hour ago
> I encourage others who work in the public sector, whether directly or through a contractor, to read and internalize Don't Get Distracted by Caleb Hearth.
I know a guy who works for well-known hardware company. He isn't a software dev, but close enough, works with Verilog on hardware design and verification. One day he tells me how he is working on video processing module and he doesn't know if that module will be part of surveillance system, missile guidance or nanny cam. It's possible that even middle management doesn't know more than him, and very likely the module will find uses in multiple products. He is a small cogs in a large machine, maybe his work will save a baby's life, maybe it will kill a person, he cannot know.
italiansolider | 2 hours ago
conn10mfan | an hour ago
petre | an hour ago
sashank_1509 | an hour ago
jtrn | an hour ago
If true, there was no verification of the targets selected by AI, meaning AI is the final arbiter of the deadly use of force. Another Rubicon passed.
Trump pretends like it didn’t happen: “We might never know what really happened there” is a frustrating lack of responsibility.
Still missing, as it always is from all of these kinds of articles, is: How common is this error rate compared to pure human evaluation and decision-making? I’m not saying I support AI-assisted killing, but why is nobody comparing it to human failure rates? If the article is true, then there were 1,000 targets hit in the first 24 hours. If the majority of the legwork and selection was done by AI, how many schools, or equivalents, would one expect to be wrongly hit if all the targeting work were done by humans only?
aliasxneo | an hour ago
eeem | an hour ago
blikdak | an hour ago
kelnos | 54 minutes ago
Dansvidania | an hour ago
I often see the argument "humans make mistakes too" when talking about AI, self driving cars, autonomous drones and such...
yes, they do, but most of the time they go to trial, there are consequences, there might even be justice.
when AI/self driving cars/drones make mistakes, which might very well be more rare in absolute [citation needed], there is (seems to me) a tendency to shrug it off, almost as if it were force majeure.
is it just me?
ChrisArchitect | 56 minutes ago
(https://news.ycombinator.com/item?id=49763739 https://news.ycombinator.com/item?id=49761106)
https://archive.ph/0V37g
ChrisArchitect | 52 minutes ago
Iran school bombing: grounds to believe US was behind atrocity, UN finds
https://news.ycombinator.com/item?id=49739018
chrisjj | 35 minutes ago
johnsmith1840 | 27 minutes ago
A tragedy but it's hard to argue that it wasn't the intention of the Iranians for this exact senario to happen.
elektronika | 12 minutes ago
LightBug1 | 19 minutes ago
Either this is true, and it's fucking bullshit.
Or it's an exaggeration/ false, and it's still fucking bullshit.
So, either way, something coming out of the Pentagon is fucking bullshit.
1832764323 | 8 minutes ago
https://x.com/lessin/status/2029663841181979097
https://en.wikipedia.org/wiki/Purim