I'm Jeff, one of the founders of this+that. I've worked on named entity detection, fact extraction, knowledge management and NLP throughout my career, starting in the NLP stone age of the 1990s. I wanted to see whether a model could now write a useful wiki straight from a company's email.
Enron's mail is an obvious test because it's public, so anyone can check the pages against the messages and the history. We used 45,047 emails from eight mailboxes. Every fact quotes the sentence it came from, and a fact expires on the date the mail shows it stopped being true.
The hardest part was that the model already knows Enron. It kept adding details from the public record that the email didn't contain. The methodology page shows how we caught those, what it still gets wrong and what it cost: about $11,000 at list prices (though we used a subscription), with no attempt to bring that down.
Your comment was dead, and many of your other comments are dead (some flagged.) I vouched for this one, but you may be shadow banned, or users may be picking up on LLM tendencies in your text and flagging them.
Maybe someone in Germany needs your help with the AFD Email leak; right now journalist are going through the data; your solution sounds like something that need
This is interesting and applicable to so many domains where people want to query a model about a dataset, and have it summarise or otherwise analyse that dataset.
One of the biggest problems about LLMs as machines for analysis is that they don't respect the boundaries of a problem and instead generate base on their corpus and training as well as the actual dataset in question in the context.
It would be wonderful if we could easily strip out facts about Enron somehow from a model or just instruct it to ignore anything outside the dataset (and have the instruction respected), it is quite frustrating that these machines are so fuzzy.
I had a look at the methodology page, well done for doing the hard work of going over the results a few times to find errors/inconsistencies. There are not too many inconsistencies according to that.
Of 62,205 candidate facts, 37 quoted a sentence we couldn't find in the message
The difficulty for a business considering using this (in law for example or analysing internal business documents), is that one incorrect fact could be very damaging and it's very hard to check them all.
I completely agree. It's hard to imagine this ever being perfect both because of AI and because people make mistakes in communications all the time. In our own usage of it we've found it essential to check things that seem suspicious once they're in the company brain. But, the more important gate, and one we expect may remain longer than people would like, is that we ask people to approve changes to a company brain that are proposed. Even this isn't perfect but it is a key defense from the brain turning into slop. And it's different than this demo, which was much closer to fully automated.
Any chance you'll be open-sourcing any part of the underlying model hierarchy? A lot of us could really use a good library for deriving knowledge graphs or expert systems with cool features (such as expiration dates, like your solution has, or weighted probabilities) rather than everyone re-inventing it on their own.
You're not wrong. These load-bearing air molecules are not too energized; they are quite tranquil―and the wavelengths emitted off plants' photosynthesis organs are equally pleasing.
I guess “something” meant a vetted web of trust. My children will be better off if they don’t need to learn to navigate the internet cesspool, so either you’re wrong and it sucks for them, or you’re right and it sucks for them. Either way, well-played! Very clever!
It's not impossible to solve, and it doesn't require any kind of onerous human verification either. We already saw this earlier this year: LLM-generated comments were everywhere on HN, rotting this place out. Then dang implemented a no-LLM-comments policy and a filter that does a remarkably good job of automatically detecting and flagging them. It is by choice that the submissions of blatantly obviously LLM-generated content are allowed to rot out ShowHN.
Ah, Enron. The good ole days when fraudulent execs still went to jail and corporations still saw consequences of fraud.
2002, Arthur Andersen gets indicted for deliberately aiding Enron in cooking the books. 2003, Bush DoJ's attorney general binds prosecutors to effectively always settle corporate cases without convictions. So nowadays you have companies like Boeing murdering 300+ without real consequences. It's a real shame.
Certainly what it feels like... but you've got: SBF, Holmes, Madoff who have done and are doing prison time. But yea, the Boeing case stands out for sure.
A company I had private stock in just sold. The email I got saying that they wanted to get me paid ASAP was electric. Finally, for just a day or two, I was in the elite of the elite: people who are owed money and that the enforcement arm of the US government will ensure are made whole. The payday was under $100, but the feeling was priceless.
> The good ole days when fraudulent execs still went to jail
Well, not one guy. Lou Pai cheated on his wife with stripper Melanie Fewell who was also married. He was known in the Enron offices to enjoy Houston area strip clubs. But in a first for him, he knocked Melanie up over the course of the affair. Pai's first wife and mother of his first two children Lana Lee learned about it and started divorce proceedings. To satisfy asset division as part of their divorce terms, Pai sold $250M worth of Enron stock just months before it crashed and Enron went bankrupt. Pai then married Fewell. Pai was briefly the largest(? maybe top two?) private land owners in Colorado, and his ranchland included Culebra Peak, one of CO's 14ers.
In 2008 he settled a civil suit alleging insider trading brought by the SEC. The settlement was $31.5 and does not include an admission of the SEC's claims. He was barred from serving as an officer or director of a public company for 5 years.
Lou landed on his feet. He and Melanie are still married. They and their daughter seem to be big figures in dressage, running Canaan Ranch and the Bred in the USA Syndicate, which seems to be a syndicate of 14 breeders. Maybe someone who knows horses can explain exactly how big a deal they are; they might not be much after all.
Lou Pai is an American role model, I love that he used divorce as a tool as the breadwinner. It is advertised as a failure mode, societally, and doubly so for the person that accumulated assets. Helped me realize to take a second look at the system at the level lawyers operate at
SEC only went after him because the movie came out
I think our current commissioner was working at the SEC at that time, I wonder what he thinks of it
He was having sex with strippers in the club's VIP, a feature that's often denied as existing, meeting more visually and sexually attractive, younger, more feminine women in that route, with a nice bankroll.
The utility of his prior lifestyle had diminished, he didn't need a dual income household and he certainly didn't want it.
He was bored of it, the university pipeline straight to running a corporate VC arm full of corporate climbers that want to prove how they fit in male dominated spaces with masculine behaviors. The representation of that was and is completely contrived - for any guy with options - and he went along with it because people told him to in the DC area, and that's who his peers were. Assortative mating that he realized was not serving him and he wasn't poor enough to need it.
I wouldn't say the conception was planned, he was just having fun with his favorite stripper, but pouring gas on himself at the gas station to mask her sent was.
Now, he retired his whole bloodline, after making a bloodline at all with his favorite stripper, giving her the dream of ponies, horses, and seemingly living happily ever after. I never followed up on his daughter being an adult now, but the privilege of pursuing dressage because of the father's successful indiscretions is fascinating.
I'll definitely aim to replicate, it looks far more appealing than upper middle class assortative mating does, with all the failure modes reappropriated as exception handling. I'll just avoid the first marriage, and refine the journey with trusts more effectively in such a wild wealth disparity and power dynamic, but also keep the divorce court ordered insider trading immunity in my back pocket.
I remember when Arthur Andersen Consulting (a software consulting firm that had split off from the accounting firm) renamed itself to "Accenture" in 2001 a few months before the Enron scandal. Although they already had split from Andersen legally, I wonder if somebody in the software firm knew what was going to happen with Enron and realized they needed to dissociate themselves with the name.
Yes, yes, I know, ctrl-f Credit Suisse and pretend it's 2005 and some crank with a tin foil ball cap is telling you about the NSA's wiretap program (with 2020 hindsight). No one ever talks about the 2019 bailouts.
> always settle corporate cases without convictions
I'm not sure that's a bad idea. You can't put a corporation in jail, but you can get their money and that also requires a lower standard of proof than a criminal conviction. In theory I could see it as a better way to recover restitution for those who suffered losses.
No, Bush's AG did not bind prosecutors to effectively always settle corporate cases without convictions. Following high-profile corporate scandals (e.g., Enron, WorldCom, Tyco), Bush established the Corporate Fraud Task Force in 2002. Over the next couple of years the DOJ obtained over 500 corporate fraud convictions or guilty pleas. They charged over 900 defendants and over 60 corporate CEOs and presidents with some type of corporate fraud crime in connection with over 400 filed cases.
[OP] jreynar | 20 hours ago
Enron's mail is an obvious test because it's public, so anyone can check the pages against the messages and the history. We used 45,047 emails from eight mailboxes. Every fact quotes the sentence it came from, and a fact expires on the date the mail shows it stopped being true.
The hardest part was that the model already knows Enron. It kept adding details from the public record that the email didn't contain. The methodology page shows how we caught those, what it still gets wrong and what it cost: about $11,000 at list prices (though we used a subscription), with no attempt to bring that down.
Happy to answer questions.
radlad | 16 hours ago
Vaushite | 16 hours ago
[OP] jreynar | 15 hours ago
BalinKing | 15 hours ago
number6 | 16 hours ago
fwip | 16 hours ago
grey-area | 15 hours ago
One of the biggest problems about LLMs as machines for analysis is that they don't respect the boundaries of a problem and instead generate base on their corpus and training as well as the actual dataset in question in the context.
It would be wonderful if we could easily strip out facts about Enron somehow from a model or just instruct it to ignore anything outside the dataset (and have the instruction respected), it is quite frustrating that these machines are so fuzzy.
I had a look at the methodology page, well done for doing the hard work of going over the results a few times to find errors/inconsistencies. There are not too many inconsistencies according to that.
Of 62,205 candidate facts, 37 quoted a sentence we couldn't find in the message
https://www.thisandthat.chat/enron/methodology.html
The difficulty for a business considering using this (in law for example or analysing internal business documents), is that one incorrect fact could be very damaging and it's very hard to check them all.
[OP] jreynar | 15 hours ago
jollyllama | 13 hours ago
anotherhue | 17 hours ago
Razengan | 16 hours ago
cwnyth | 16 hours ago
pixl97 | 16 hours ago
Unless we get some kind of verified human authentication it's nearly impossible to solve. And that in itself introduces all kinds of new problems.
Welcome to the dead internet.
irishcoffee | 16 hours ago
pixl97 | 16 hours ago
Well, the future sucks for your children.
Also, thank goodness I made thousands of HN accounts to sell to LLMs.
No, this really just shifts the shape of the problem around, but the problem is still the same size.
irishcoffee | 12 hours ago
anonymous908213 | 15 hours ago
pixl97 | 15 hours ago
At best we get this
https://xkcd.com/810/
chung8123 | 16 hours ago
jetbalsa | 16 hours ago
qclibre22 | 15 hours ago
pretty cool.
[OP] jreynar | 15 hours ago
Kwpolska | 15 hours ago
deaux | 17 hours ago
2002, Arthur Andersen gets indicted for deliberately aiding Enron in cooking the books. 2003, Bush DoJ's attorney general binds prosecutors to effectively always settle corporate cases without convictions. So nowadays you have companies like Boeing murdering 300+ without real consequences. It's a real shame.
scottrogers86 | 17 hours ago
jonathanlb | 16 hours ago
The difference here is that they defrauded the investor class.
stymaar | 16 hours ago
Too nice of a way to talk about the Epstein class.
scottrogers86 | 15 hours ago
59percentmore | 15 hours ago
1-more | 14 hours ago
1-more | 16 hours ago
Well, not one guy. Lou Pai cheated on his wife with stripper Melanie Fewell who was also married. He was known in the Enron offices to enjoy Houston area strip clubs. But in a first for him, he knocked Melanie up over the course of the affair. Pai's first wife and mother of his first two children Lana Lee learned about it and started divorce proceedings. To satisfy asset division as part of their divorce terms, Pai sold $250M worth of Enron stock just months before it crashed and Enron went bankrupt. Pai then married Fewell. Pai was briefly the largest(? maybe top two?) private land owners in Colorado, and his ranchland included Culebra Peak, one of CO's 14ers.
In 2008 he settled a civil suit alleging insider trading brought by the SEC. The settlement was $31.5 and does not include an admission of the SEC's claims. He was barred from serving as an officer or director of a public company for 5 years.
Lou landed on his feet. He and Melanie are still married. They and their daughter seem to be big figures in dressage, running Canaan Ranch and the Bred in the USA Syndicate, which seems to be a syndicate of 14 breeders. Maybe someone who knows horses can explain exactly how big a deal they are; they might not be much after all.
https://www.usef.org/us-equestrian-teams/athletes/natalie_pa...
yieldcrv | 16 hours ago
SEC only went after him because the movie came out
I think our current commissioner was working at the SEC at that time, I wonder what he thinks of it
1-more | 14 hours ago
yieldcrv | 13 hours ago
The utility of his prior lifestyle had diminished, he didn't need a dual income household and he certainly didn't want it.
He was bored of it, the university pipeline straight to running a corporate VC arm full of corporate climbers that want to prove how they fit in male dominated spaces with masculine behaviors. The representation of that was and is completely contrived - for any guy with options - and he went along with it because people told him to in the DC area, and that's who his peers were. Assortative mating that he realized was not serving him and he wasn't poor enough to need it.
I wouldn't say the conception was planned, he was just having fun with his favorite stripper, but pouring gas on himself at the gas station to mask her sent was.
Now, he retired his whole bloodline, after making a bloodline at all with his favorite stripper, giving her the dream of ponies, horses, and seemingly living happily ever after. I never followed up on his daughter being an adult now, but the privilege of pursuing dressage because of the father's successful indiscretions is fascinating.
I'll definitely aim to replicate, it looks far more appealing than upper middle class assortative mating does, with all the failure modes reappropriated as exception handling. I'll just avoid the first marriage, and refine the journey with trusts more effectively in such a wild wealth disparity and power dynamic, but also keep the divorce court ordered insider trading immunity in my back pocket.
jhbadger | 16 hours ago
59percentmore | 15 hours ago
https://www.reddit.com/r/Superstonk/comments/10ir695/nomura_...
Yes, yes, I know, ctrl-f Credit Suisse and pretend it's 2005 and some crank with a tin foil ball cap is telling you about the NSA's wiretap program (with 2020 hindsight). No one ever talks about the 2019 bailouts.
SoftTalker | 16 hours ago
I'm not sure that's a bad idea. You can't put a corporation in jail, but you can get their money and that also requires a lower standard of proof than a criminal conviction. In theory I could see it as a better way to recover restitution for those who suffered losses.
officeplant | 16 hours ago
I don't see why we can't just put an entire board of a company in jail. Might inspire them to do better.
opo | 15 hours ago
https://www.justice.gov/archive/opa/pr/2004/July/04_dag_495....
tonymet | 16 hours ago
contubernio | 16 hours ago
nonethewiser | 16 hours ago
Capricorn2481 | 12 hours ago
tonymet | 15 hours ago
happyash1 | 15 hours ago
xbar | 15 hours ago
[OP] jreynar | 15 hours ago
Kim_Bruning | 15 hours ago
[OP] jreynar | 15 hours ago
FLeXMurphy | 15 hours ago
enjoyyourlife | 15 hours ago
genxy | 11 hours ago