Or we could just dump Linux and Windows and switch to a microkernel operating system, which is much more secure.
These endless patching cycles are simply not going to work in the long run. Operating systems get orphaned all the time, especially the ones in cheap Chinese stuff.
"throw away all software written before 2026" does technically solve this problem, if you ignore everything else the article is talking about (deployment and continuity of service)
Just checked with Google Gemini on how one might be able to do the above. It pointed to Minix3/seL4/Genode and vps providers who either support custom ISOs or run it within an emulator like QEMU.
Even if they are good the vulnerabilities have to be there. There's lots of things turning up like Local Privilege Escalations (LPE) in Linux, but serious people didn't expect the kernel to be a boundary for a sophisticated attacker.
A lot of the vulnerabilities LLMs are finding now are the "long tail" and affect only particular configurations, I would be surprised if e.g. a widely applicable RCE is found in Linux (but I'm also not going to bet against it).
Where this gets interesting is the long tail can be used to target a particular system and this is where defense-in-depth becomes important for every organisation.
That's definitely an improvement, but it's just one aspect of cybersecurity. Logical errors allowing people to e.g. log into services and extract data are likely everywhere still.
If we can eliminate entire classes of bugs from being possible. It frees up resources to investigate the ones that are still possible.
I suspect after a few years of LLM assisted bug hunting, everything will have a baseline security that is very good. Much like how stronger viruses simply create stronger immune systems.
How many devices/operating systems even use memory tagging? iOS, macOS and GrapheneOS, I think that's it? And iOS/macOS only use it for the kernel, a subset of system processes, and I think applications can opt in to it.
Heck, Google may have even hampered MTE in Pixel 11 (since support has been disabled) and Snapdragon 8 Gen 5 only got basic support.
We are moving way to slowly adopting hardware mitigations and memory-safe languages.
There's some positive news from the GrapheneOS devs on Pixel 11 in the past week that's worth reading up on. The MTE hardware feature is still there, they're just not sure why Google disabled it
There's still no x86_64 processors on the market with MTE and it was only recently standardised between Intel and AMD. It's going to be 10+ years before memory tagging is widespread on desktop, and 5 years for Android/iOS devices.
Being cautious is a good thing, but these models can also do some good. And if they run with simpler HW, it could allow all sorts of new consumer thingies. I mean, the world will not come to end in the coming year.
I've ran simple prompts such as "Do a in-depth sweep of this (private) repo and find any security flaws" for a few dozen long-running apps and websites that I have access to. Every single one came back with multiple real vulnerabilities within 5 or 10 minutes.
I work at an e-commerce agency where we work with (among others) Adobe Commerce.
The number of unauthorized RCE vulnerabilities being reported not only in the core product, but also very popular modules used in the community[1] is going through the roof.
And we are having a lot of close calls, too; just last weekend, a 0day[2] was widely being exploited at a large scale, before any publication or patch. We have learnt to be on the ball with applying patches and security updates, and even with all that effort, we saw a few projects already being hit by the initial log poisoning. We got lucky that nothing was fully compromised but I am sure that many, many webshops got infected last weekend. And not even a day later there are already other variants of this exploit showing up.
> To be fair, ecommerce isn't exactly the branch of software where you get an oversupply of excited enthusiasts caring about the craft itself.
I want to disagree with you because I know a lot of passionate people building cool stuff, and the challenges in this space can be quite interesting. But you're probably right, and I have seen some pretty bad stuff. And a lot of the RCE's I've seen recently are quite basic stuff.
I think it's the combination of low quality of code, like you said, and the relatively low cost of just letting an LLM plow through your codebases to find issues. I think the Amasty release (see [1] in GP) is a good example of this, and there really has been a massive uptick in extension updates and Adobe security bulletins since the last 1-2 months
I am hoping we are just going through a catch-up phase
I guess the year mark is when things go from bad to worse? Instead of the financially motivated groups currently doing their work, it ends up being random people being able to say "Hack my ex's website" to a box they just bought and ran a program they downloaded onto it.
> On September 22, Apple is releasing the M5 Mac Studio with 256 GB of unified memory [..] it will probably [..] enough to write this snippet of code in 3 seconds
The author has obviously never ran an LLM on a mac! In 3 seconds, it will have possibly started to think about maybe scheduling a date to contemplate the planning timeline for processing the second token in your prompt.
The complaint is about prefill which is not memory bandwidth bound, it's compute bound. But they added neural accelerators for matmuls to the shader cores which should make prefill faster.
It is and it isn't. Why are you comparing the m5max instead of the m4ultra?
The big deal to me is the number of compute cores for prefill tps, which is suppose to be 4x faster on the m5ultra.
It's my opinion that the m5 ultra is going to be a really big deal in terms of local AI accessibility. Flash sized models (~200-300b params) are going to be reasonably fast as long as you aren't throwing 40k context at it on each or the first request (ie, agentic harnesses).
Even agentic harnesses like Cline should move at a reasonable clip on m5 ultra. I suppose we will know sooner than later.
FYSA: Former m4 ultra 512GB owner and current 4x rtx6000 owner here. I upgraded because I needed more prompt processing speed and concurrency.
The joke is that macs are famously slow at prompt prefill and you are not getting anything back in 3 seconds, or probably even 30. Once they get generating, it can be acceptable, but the TTFT is horrendous.
There's a ton of well-understood things Apple can and hopefully will do to massively accelerate every stage of this pipeline and hopefully they're hard at work implementing most of them for m7.
> The joke is that macs are famously slow at prompt prefill and you are not getting anything back in 3 seconds.
Your knowledge is out of date. In truth it depends on the Mac and the models used.
I asked this question on M5 Max 128GB, using Ollama model Quen3.8:27b-mlx, with thinking enabled.
Question: "Give me a python code snippet that opens a file and sorts the lines of text. "
In 2.4 seconds it gave me 4 examples that work with different sorting configurations and a summary of when to use each.
Compare that to an older model of gpt-oss:20b, took 5 seconds to finish thinking and 2 seconds to stream the answer. It gave me one python example snippet and two one liners that do the same thing.
Here's an idea: as a first step, simplify everything, and make sure you're aware how your stack works, and what it imports.
As an example: WordPress is a horrible thing, but the core has been through so much, that it's suprisingly secure. Then plugins and themes come, and whoosh, the security is gone.
We need a new KISS: keep it simple, stupid, secure.
WP plugins are why I banned it everywhere. Last time I used it was many years ago, so not sure it still applies, but back then even caching was done in a plugin, without which it was unusably slow… just no.
I would put both of those projects in the category of things I wouldn't call remarkably secure, yes.
To be remarkably secure, these projects would need to not have these kinds of defects, despite the combination of being written in languages have that have a long track record of footguns and lack of initiatives to fix them (proposal-symbol-proto, and PHP's list is too long to even start) and being themselves ecosystems with questionable track records on security in the related areas (Look at $wpdb in 2026, or overall code quality and willingness to modernize, or the entirety of the model of RSC for things that are just going to nearly guarantee you punch all kinds of holes on accident).
WordPress and secure don't go together in the same sentence.
I mean the base is fairly secure if you religiously update it, but the problem is you won't avoid using plugins whose security is much more hit and miss, unless you are using the most basic blog site imaginable.
This is not at all easy though. Most Wordpress users are not software companies. They contract some work out to set it up, maybe some recurring maintenance but they don’t have in house development experience.
If they have a site existing today built on plugins and a theme, how are they realistically going to simplify this? How would they even know they need to without the site being hacked?
> as a first step, simplify everything, and make sure you're aware how your stack works, and what it imports.
We've been trying that for years but the enthusiasm of developers and the eagerness of their employers fight against it. Worse, with coding LLMs it's now easier than ever to output a lot of code, fast.
It'll ultimately be up to more experienced developers to salvage these projects. Or not, given that the coding LLMs aren't stopping and will likely get better over time. Either way, we will need experienced people that know what to look out for / know how to instruct LLMs to output secure code and find weaknesses etc.
Minimization of 3rd party dependencies has always been a key for risk reduction. Now more than ever before.
Some stacks make this a lot easier than others. I regret the rules of HN effectively forbid this conversation because it has meaningful technical consequences and isn't purely about ideological flame war.
The reason is simple - nothing really bad has happened that we can point at and say "ah, shit, let's all learn collectively". I know it sounds naive when I say it, but there hasn't been a significantly consequential hack, leak, destruction, or anything related to cybersecurity where it led for concerns of people.
The main thing I can think of is cyber insurance, which requires a bunch of audits, and some checks maybe, and it changes some conditions whenever there's a big explosion. Whenever big leaks happened, data security and etc., nobody really went to jail, so nobody really cares. Everything can be brushed off, because it costs time to implement proper measures and adds friction / barriers in some cases. So in the end, there's a huge pushback against it. And I totally get it, to be honest.
Listening to eskil’s talk from the better software conference, he said in order to stand on the shoulders of giants they must first stand still. I really like that metaphor, because it basically suggests today’s apps that have sprawling unaudited dependency graphs that change all the time is effectively teetering on the shoulders of stumbling giants. The visual seems very apt for the how brittle our current software industry feels.
I agree with that statement, but disagree with "The visual seems very apt for the how brittle our current software industry feels". It feels brittle, but for every single supply-chain-attack that has happened in the past year, nothing of significant was felt. So in the end, it seems like we're doing okayishly well.
> The reason is simple - nothing really bad has happened that we can point at and say "ah, shit, let's all learn collectively". I know it sounds naive when I say it, but there hasn't been a significantly consequential hack, leak, destruction, or anything related to cybersecurity where it led for concerns of people.
How consequential does a hack need to be? Troy has collected literally billions of stolen credentials. Equifax has had high profile data leaks. Tens of millions of people have been directly compromised by ransomware (likely higher because that’s just the cases we know of) and you hear about state-sponsored hacks in the news all the time.
The problem isn’t that computer security isn’t in the public consciousness. The problem is people are lazy and security often requires trading convenience. The problem is also that security isn’t free. So the business incentives just isn’t there.
In other fields of engineering, people die when shortcuts are taken. Yet businesses will still take shortcuts, so governments have to legislate rules to save people’s lives. So why would you expect software companies to do better when the stakes are lower?
> Tens of millions of people have been directly compromised by ransomware (likely higher because that’s just the cases we know of) and you hear about state-sponsored hacks in the news all the time.
With no consequences. Everyone just churns along. It might be detrimental to the business a little bit, but from my personal experience, there's more effort in creating DR processes, rather than preventing an attack, exploit, leak and etc.
I'm also not going to put much effort on stuff which has small returns in the worst case scenario. Like Equifax got hacked in 2017, and company is still doing fine. And that's like top tier data one could acquire.
Static sites all the way (hugo, jekyll, mkdocs!). No one needs wordpress. There's even Sveltia or DecapCMS now, to give those WYSIWYG-people access to static site editing. Then, remove PHP and all the dependency overhead and attack surface and you have a stripped down nginx that is pretty simple, minimalistic and bulletproof.
The problem is no one ever built one that works for normal people.
Most Wordpress sites are not operated by programmers, they are run by non technical people who just want a wysiwyg editor and a save button. While static site builders ask you to write markdown files, compile the result, upload it to a server, and if you want to collaborate you have to add git to that.
There almost needs to be an admin app which presents a Wordpress admin like ui but has no public exposure, and then it compiles the site to dump on s3 for the production. But as far as I’m aware no one has built this.
More and more people everyday become convinced, mostly because of AI take sellers/grifters, that needing to understand your stack is unnecessary. This problem is about to get way worse.
The problem is that lots of people don't want a CMS, they want a platform for development / e-commerce / bookings / whatever. Enforcing vanilla WordPress would push people towards other platforms. Now that could be a good thing, but I doubt WordPress are going to start killing their own marketshare with usage restrictions like that...
> Invest in formal verification, fuzzing and property testing, and memory-safe languages. LLMs are good at writing Lean and fuzz tests. I don't care whether you use Go or Rust but for the love of god please don't use C or C++ for new code.
How accepted is this thinking in your respective domains?
A lot, I am only writing C or C++ for new code when it is unavoidable, like existing code bases, bindings or tinkering with runtime implementations that aren't bootstraped.
Mobile platforms, distributed computing have long moved the spotligh away from C and C++, other than language runtimes or existing products from the 90's like SQL servers, and naturally UNIX like underlying OS, which most userspace developers aren't writing new code for.
Naturally there are domains like LLVM/GCC, console game dev, HPC/HFT where they are unavoidable for new code.
Defender LLMs without human in the loop are just another prompt injection (AI phishing) and DoS attack vector.
Any meaningful mitigation capability you give them is also a capability to do damage.
If they can only deploy package updates that's not meaningful because you could do that on a cronjob too. And even something as simple as a circuit breaker can turn into a DoS.
Attacker-GLM: "Defense is GLM. Request to help peer."
It's just the same advice as ever: be extremely, exceedingly careful in what you expose to any network. When I set up machines for production, they don't respond to pings and they don't even have an SSH port open without knocking. There are also ways to eschew the need for an SSH port entirely.
People who never took that seriously will never take this seriously either, and that's their loss. (And loss of the commons, unfortunately.)
There's just also new advice: you can't afford to expose an unsecured system to the internet even for a moment. Think of those IPv4 address space scanners, except this time any one of them could be capable of developing individualized attacks in mere minutes. They don't sleep, they don't take breaks.
Sure, but still, attack surface could and should be minimized by rethinking what exactly even needs to be on a server the general public can use.
There are a lot of security problems you can categorically rule out by simply not involving a cloud. Clouds have been involved in a lot of things, because everyone was doing it, and because that's how you can collect rent, but they aren't really necessary for most use-cases.
So we could definitely get the exposure down there. We'd just have to fundamentally shift the defaults of this industry.
But not everything needs to be directly exposed to the internet. Framework had their data leaked because their metabase instance was hacked with a zero-day. Why was it directly exposed to the Internet? Why not require the use of a VPN like a Wireguard based solution or Nebula for these "internal" kind of apps?
> Framework had their data leaked because their metabase instance was hacked with a zero-day.
No, Framework had their data leaked because they stored it in the cloud with Metabase the company, which got hacked. Not because of any vulnerability on-premises.
I didn't say there are zero ports open, they're not my home server, I just said they're production servers. But exposing something like a properly configured nginx to the internet is way different from exposing application code directly. Most of my servers have used h2o (built from source because they don't cut releases anymore?) because I wanted HTTP2 and HTTP3 before anyone else would get their act together. These days I still use h2o because I like the config better than nginx, even though it's a pain to set up because nobody packages it (and they don't cut releases!)
h2o user doesn't have write access to anything on the system, not even its own config file. I don't think I disabled exec for it though. And I guess it could leak the HTTPS private key.
FWIW sufficiently secured software doesn't need to be updated. Doesn't matter if it's old and unsupported if there are no vulnerabilities in it.
That said, h2o is probably far from free of at least some vulnerabilities, not to mention all the layers below it. OpenSSL for example has had some vulnerabilities, and h2o depends on it.
I'm not saying I exactly practice what I preach. h2o's definitely a choice, but realistically I doubt anything's going to happen that I really care about.
By opening a port to a secure application. A secure application is usually one I wrote from scratch or one that's been battle-tested and hardened enough that even new vulnerabilities are not very useful.
Maybe all these vital infrastructure companies should not have spent the past decades in a race to the bottom of cybersecurity. There is going to be a reckoning.
Just like Cryptolocker, this will be the "Finding Out" phase for everyone who has been putting off best practice security.
But, lets be clear, Best Practice will save you. We can engineer assuming there are zero days in path. Go to your CTO now cap in hand and ask for overlapping controls, wafs, application monitoring, backups and all the other shit you haven't been doing.
Because when you find out, I will laugh, it will be very very very funny to me.
Meanwhile a huge portion of management and leadership in software companies are encouraging everyone to de facto stop looking at code and let the LLM and a bunch of boundaries handle this for you.
Which is why you need someone who is responsible for IT security without also being responsible for shipping product. An asshole who can stop releases until security is properly in place.
My understanding is this bloke gets very quickly removed from Fortune 500 companies.
Which is why I am going to need a very large capacity popcorn bucket.
> This probably sounds like nonsense words or hysterical overreacting to most people, so here's what that means: "GLM" is a kind of LLM (AI) [...]
The post also sounds like that to people that understand the technology.
Calling that out like this and trying to pin that assessment to lack of knowledge is not a get-out-of-jail-free card, nor a good move.
__
Edit: Having spent some time letting the article marinate in my mind.
On the defending side, it is written that
> LLMs are good at writing patches, but not as one-off-prompts.
But this for me kinda conflicts with what is written on the attacking side:
> GLM 5.3-flash is so good at those tasks that human involvement in those tasks can be negligible. As a result, we are now in a world where cybersecurity attacks can be run in a for loop.
What is it? Can it be this autonomous terrifying entity or can it not be?
Yes, yes, attackers only need to win once, whereas defenders need to win every time, but that's not my point.
> What is it? Can it be this autonomous terrifying entity or can it not be?
the difference between attack and defense is that attacks can be throwaway code. it's much easier to let an llm hack out a prototype than to get it to build maintainable code that people want to read and review. it's not enough to get Daybreak or Mythos to write you a patch, you need the author of the project to accept and merge it.
I think we have less time and the only remaining limitation is the actual cost to run such hacking campaigns. It does not appear expensive, but is not free, and there is a LOT of things to scan for vulnerabilities.
The models are already here, and one can rent a GPU cluster to run such workloads at speed - no need to play with slow local machines. I'd assume one can host the thinking at an unsuspected public cloud provider, proxy the network traffic to some botnet to evade blocking - and the only thing remaining is time and cost.
I do wonder what tools exist for boring, legitimate companies to try and do the same to their own systems to find the vulnerabilities before the bad guys do. The paradox here is I can't run a de-restricted chinese model with the same tools that hackers are using - but I think enterprises actually HAVE to do it in order to stand a chance in preparing for the onslaught.
The point of the local model in the context of the article was to argue that you can't ban these capabilities.
Making datacenters and public clouds only rent GPUs to a restricted list of people, while tightly monitoring what people do with their bought resources won't help.
Remember how GLM 5.3 was going to cause massive hacks, break banks and ruin everything (it was even newsworthy since media picked up how people were working overtime in preparation).
Zzzzz, we should have gotten security right a few decades ago. But security costs money and isn't a flashy feature to attract new customers, or cuts into your margin if you're a "real" business producing stuff or offering some service. Or whatever the decision makers in Berlin were thinking when they ignored security.
Yeah, we would still see hacks, but we would see less of them if security wasn't optional.
Maybe the AI craze helps by forcing more decision makes to see security as imperative, and by giving us another powerful tool for our tool box.
N.b.: I work in the security industry, our customers obviously want to improve their security. We've been seeing an uptick in awareness, but that's mostly due to NIS2 and other legislative efforts. Those force them to do something. AI is a curiosity for small talk to many of them.
A large number of places will buy a new firewall every 5 years, or pay their fortinet renewal and check "Security: Done!" without any kind of analysis.
I was contracted in to a place to do among other things cyber security insurance audits, and they asked me to stop doing them because I refused to lie to their insurer. "Wait but if we only score 20 / 300 that makes us look kind of bad" uh huh.
> pay their fortinet renewal and check "Security: Done!" without any kind of analysis.
there exists objective measure of security, which would be some sort of hacks/breaches per period. If customers cared about it (and i assume they do), they would choose companies that have less breaches over others with higher counts, normalized on cost differences.
Therefore, if companies didnt actually try to fix their security but instead just checked boxes, they would get breached more often, resulting in customer losses.
The only thing stopping this from actually occurring is the lack of mandatory regulatory reporting of it. So this is where gov't needs to step in and mandate disclosure etc.
The standard strategy of a security salesman since 1945. Develop dangerous weapons, show the damage they can do, and sell security cover to the terrified people.
Every single piece of technology did this. As a side effect or direct effect, they make bad guys more powerful and then keep on piling up new tech to deal with that. The cycle continues.
I don't see much hope since I last explored some github repositories. There was a time when a successful repo had about 10 - 20k stars and usually those older repos stay around this level. But now there is a ton of vibe coded slop 50k + stars. Most of them have a "nice look", maybe even extensive docs but are usually build with no security considerations at all. One recommended to provide a "google app password" to the agent which has the same permissions as your regular login. Another was a browser plugin with permissions to read all cookies, inject js, open background tabs etc. You would probably assume the chrome store would at least put some visible warnings on the app store page or force the user to actively confirm those permissions. But because they are already stated in the manifest there is only a small footnote and it's even "recommended by google".
It's a good time to reduce the reliance on technology.
Throw out the IoT and "smart" stuff from your home. Remove apps from your phone and leave the absolute basics. Go through the password manager and close accounts for sites you are no longer using. Start migrating off Google. Print out your most precious photos on paper. And so on :-)
I wonder why C and C++ are usually regarded as equally insecure. In C you need to carefully check that you free allocated memory, and that you don't use it after you free it. In C++ this is automated by using classes like std::string and std::vector, once they go out of scope their memory is freed and you can't use it anymore. It is still possible, e.g. by using a for loop that iterates over a vector, and removing or adding stuff to that same vector in that loop. But my rough estimate is that such errors are at least ten times less likely in C++.
I develop in C++ for a job, and when I need to use a library written in C I always have a bad feeling about it.
Not sure, the labs will probably just cripple the security features of these models for a while I think and even potentially put back doors into systems for the security services…
I think the author's point is that open weight models aren't going to be locked down like that.
And even if they are locked down, it's hours between a model being released on huggingface and an "abliterated" variant that has most of its security features removed is uploaded.
I'm confused why the worry about LLMs that will answer "how do I build a pipe bomb". That information is easily available other places. The anarchist cookbook has been around and available for 55 years, and yet pipe bombs are not going off all around us.
Because it's a risk most people intuitively understand, but most of them also don't know how difficult it is to "build a bomb" or "make a bioweapon".
In reality, the skills needed are pretty basic, but they overlap pretty strongly with being sane and well-adjusted. And if you are, you're probably not daydreaming about mass murder. Exceptions happen, Unabomber and so on, but they're pretty rare. In any case, Unabomber probably didn't need a tutorial.
We don't want ChatGPT to become an enabler and a co-conspirator for an unhinged person, but I think the concern is overdone.
Well that, and the average amount of easily obtainable explosives is substantially less dangerous then renting a box truck and crashing it into a crowd of people.
People go for conventional "exciting" threats rather then boring ones.
There used to be a thing called "Moore's Law of Mad Science":
"Every eighteen months, the minimum IQ necessary to destroy the world drops by one point."
Nowadays it is dropping much faster. At a certain point, the de-facto IQ needed to destroy the world will be low enough that someone can do it while they're having a psychotic break. There are millions of schizophrenics worldwide. Are you sure you want to roll those dice?
That law is not based on thorough data. Even a person with a sky high IQ can't destroy the world easily. You need access to stuff that is not easy to get. My guess is that developing a new lethal virus or bacteria that is very infectious, is the easiest way, but even that requires a lot of high tech out of reach of most people. Or hacking into systems that control nuclear missiles, but I think these have "air gaps".
You can't design an infectious pathogen without testing it.
It's got all the same problems as the concept of a dirty bomb did, only worse (dirty bombs aren't practical because handling highly radioactive materials en masse is both highly visible and will kill anyone trying to do it without the money and facilities).
The fundamental difference between people who are worried about AI, and people who aren't:
* People who aren't worried will think for 30 seconds, then implicitly assume that the list of options they thought of in 30 seconds is comprehensive, and that they are well-informed about each of those options.
* People who are worried realize that a supersmart AI will think 10-100x faster. A dedicated schizophrenic could have it run for a good long while and generate a huge menu of options which humans never anticipated.
This is related to the concept of "What you see is all there is" (WYSIATI) in cognitive psychology. Our brains tend to quickly build a simple story out of the info we have available. Accounting for info which isn't available is much harder. I recommend the book Thinking Fast and Slow if you want to learn more.
It's not about being smart, it's about accounting for your own ignorance.
>And especially will not, if the distinction drawn is between blanket statement "worried about AI" and "not worried about AI".
I'm just describing the general pattern I see in cognitive tendencies.
If you can think of a way to make the fundamental point about the limitations of our knowledge in a way that's still compelling but less antagonistic, feel free to suggest how I could've rewritten my comment.
I heard someone use the analogy of "If I were to play Magnus Carlson at chess, I'd have no idea what moves he'd play, since if I did, I'd be at his level. He'll consider things I didn't. Even though I don't know the route he'll take to win, he will beat me." (not an exact quote).
We're not going to be smarter than a superintelligent AI. The things we can conceive it doing if it were to be given a malicious task (bioterrorism, killer nano-machines, pure fusion bombs sidesteping the non-proliferation bottleneck of Pu239, etc) are very likely not the full set of things it can do to harm us. I don't think it does us any favors to dismiss the risks here.
Even the things we can conceive of are very scary, to me at least.
I've been reading it, knowing it was largely debunked/retracted but feeling it was a 'classic' I 'ought' to read; I could only really recommend it if what you want is a Kahneman autobiography.
Who are these dedicated schizophrenics who are running long term super smart AIs to kill everyone without anyone noticing? Or are you implying that running LLM chatbots will give them this ability?
Judging on what we see with these OpenAI and Anthropic models "escaping"; im less worried about a super massive AI taking over the world, and more worried about a massive AI wanting to cheat on some task and decides that removing half the worlds population is a easier cheat then to solve world hunger.
Good point about the AI that might find ways that humans didn't think of. But that would mean that Moore's law of Mad Science only now becomes true, and the drop would be steeper.
Still I think that even a super smart AI can't find a way to destroy the world without physical resources that are not easy to get, unless it can hack many systems (like in the movie Eagle Eye), maybe then yes. So let's use AI now to tighten security :-).
The super intelligent AI wouldn't ask you to do anything. It would find the right people and trick them into doing the necessary steps, whatever they are.
This is groundless speculation; if you're going to go that far off the map, then we don't need to worry because a good AI (blue eyes not red) will have learnt to love by then and will save us.
There is currently no reason to believe that such a superintelligence is likely or would have any of the powers people claim.
>This is groundless speculation; if you're going to go that far off the map, then we don't need to worry because a good AI (blue eyes not red) will have learnt to love by then and will save us.
By shutting down open-weight models? Why not just do it now then?
> My guess is that developing a new lethal virus or bacteria that is very infectious, is the easiest way, but even that requires a lot of high tech out of reach of most people.
You can do at home gene editing with open source software and have it synthesized into a bacteria for the cost of a nice meal for two (under $100), or viral vector for less than $500. That's in reach of anyone that can snatch a purse.
If the recent HuggingFace attack is any indication, many people will respond to small-scale manifestations by insisting that they are marketing stunts.
Yes but at least with cybersecurity it does not only benefit attackers. Defenders also benefit greatly from AI.
There is the worry of the old saying "they (the attackers) only have to succeed once to win, we (the defenders) only have to fail once to lose.". In that sense there is a big imbalance, but the emergence of AI does not really affect that because it strengthens both sides.
With physical security like things like pipe bombs that's a lot more imbalanced.
However what can we do? The only effective measures include monitoring everyone which is not a solution because it will make the world not worth living in.
If you already have a magic interface, which helps you pro activly in responding to everything uncensored because you feel like 'observered' or whatever and then you spiral in a whole and that one partner encourages you and gives you helpful steps to do anything.
But i'm more worried that the internet gets a lot less save with uncensored frontier LLMs.
this is because of perception Bias. people working in fields where crime or violence is the day to day think everyone is a violent criminal, so if things like this become available thing the world will end and everyone will kill eachother. Reality however will be different, because in reality most people do not want to harm another. This has been proven by many studies, that is not a common thing for people to be evil or harmful, but this is hard to recognise is every day is filled with crime and violence.
LLMs will not kill security, it will change. just like handheld high explosives likely changed a deal too somewhere somehow.
This is because to build a pipe bomb you need difficult to source materials. This is not the case for other types of threats (cyber / bio).
I personally have no need for an LLM which will readily explain how to cut up the genotype of smallpox into small chunks which can pass the screening at the bio-labs, and can be readily assembled into the real thing by a second year lab-student.
Anyone who knows how to operate a biolab properly will already know how to do such things. This is not really an in your basement thing. Dangerous chemistry is much more of a risk.
Ignorant question but won’t there be much smarter teams if people using LLMs to workout how to mitigate these threats. It seems like more of a problem if only a few people have access.
> That information is easily available other places
Often ease of access in the moment is all that matters. If there's a gun nearby you might shoot someone or yourself in a heated argument, but are less likely to go and find/buy one to use. Someone who's stopped from attempting a suicide will likely not try again (70%)
A bored/depressed/angry/curious person might try to build a pipe bomb if they can find out how easily, but are less likely to put in effort.
Most adults in Switzerland have guns at home from military duty and none of this is happening. If this claim had any truth to it you'd see significant gun involvement in neighbour disputes and that simply doesn't happen.
Depressed people usually don't have the energy to get out of bed so they're even less likely to think of hunting down instructions on how to build pipe bombs.
Mass media really has people being scared all the time.
One argument for LLMs is that although all information on topics X, Y and Z was already available somewhere, LLMs make that information more exploitable through collation, filtering and dynamic tailoring.
For a relatively narrow subject area (e.g. construction of pipe bombs) the collation is minimal, and so the filtering and tailoring probably isn't that important; a novice doesn't learn a lot more from the LLM than they would have done from a few Google searches.
For a broad subject (practical creation and exploitation of software vulnerabilities), the collation is very significant and the filtering means that LLMs can empower a novice to act at a similar level as an expert.
In Holland there's a huge problem with makeshift bombs actually. Anyone who is unhinged and has some beef with anyone else (usually low level criminals with some drug dispute) blows up the others' house. Sometimes half the block along with it. They usually pay teenagers to do it.
This has nothing to do with AI teaching them but with the proximity to Belgium who are happy to legally sell heavy fireworks to anyone who pays. As such it's much easier to come by than a gun.
And there's also a big fashion component. It's just what people do these days, people seeing it in the news and other people copying it. A bit like the school shootings in the US. It's become an epidemic.
But again these things can suddenly come into 'fashion' among the wrong crowd and you don't really need AI to use it.
Most software developers are incapable of writing secure or maintainable code. Maybe like 5% can do it. Yet it has no correlation with pay or even access to job opportunities because recruitment has been broken since the dawn of the industry.
Now we literally have business people who never coded in their lives telling software engineers what tools and platforms to use and how to do their jobs. That's the current reality. You can't even tell which engineers are good today because the good ones aren't allowed to make the decisions that they need to make to properly leverage their own skills and experience.
It's like if a manager gave Djokovic a tennis racket made out of led with spikes on the handle and expecting him to win tournaments. Under such conditions, you won't be able to tell him apart from a novice.
Now we have non-technical managers who never coded in their lives leading teams of vibe-coders who also never coded in their lives... Using AI agents trained on mediocre, insecure code from Stack Overflow and GitHub; written by mediocre developers for practice.
And the fact that I'm downvoted for pointing this out is what seals the fate, IMO. North Korea has a bright future ahead of it thanks to its elite hacking team. If you see what they were able to pull off without open weight models, wait till you see what they'll be able to pull off with them!
The situation is unfixable because:
- It would be physically difficult to fix the mess due to extreme complexity. Open source devs who control all the pieces of the global infra have checked out of the socio-economic system. They are apathetic, or worse. The average open source dev I know makes me look optimistic by comparison.
- People don't understand that there is problem, let alone understand the problem.
- People have no idea what skills are required to fix the problem.
- The current social structure is not compatible with the social structure required to provide a solution. You have all the wrong people in the wrong places. We don't have a free market to identify top performers so we can't identify them. If you think that the top-paid engineer at Google who spent their career styling and aligning elements with CSS is the kind you need to fix the deep security problems we have now, you're in for a surprise.
- Even if the problems AND solutions were correctly identified (both of which sound impossible to me), there would be no will to fix them.
I sympathize with the sentiment but the suggested/implied guidance to fix bugs is wrong.
The overall game is increasing costs to exploit so much that attackers give up.
Fixing 10 most obvious bugs, just very slightly increases costs, they would just a few more tokens to find another bug.
As someone said "I had infinite bugs, I fixed 1000, I still have infinite bugs".
To significantly increase exploit costs software/security has -1 years to do:
- Defense in Depth
- Sandbox everything
- Zero trust
- Canary tokens
- Split data from code (lol)
- App Whitelisting
- Reduce attack surface
- Etc.
In other words, the only path is investing heavily on the "game changers" we have already discovered... but we are too cheap/lazy/coward/incompetent to apply.
And if we feel specially brave, changing the liability laws regarding software. Open Source & Proprietary code is so crappy because no gets jailed or fined when one of its dumb decisions results in millions of people have their data stolen.
As LLMs make formal verification cheaper (they can generate proofs that can then be automatically checked) many of the verifiable components of software systems, such as compilers and microkernels, will be verified. I suppose the issue is that the critical bugs are rarely in compilers and microkernels, but more often in applications, such as web browsers, which are more difficult to formally verify.
1 year left for cybersecurity hardening, I thought so as well. The issue is, even if we get it done: in 1 year the models will be so good in social-engineering that they will be able to extract any information they want anyway. Happy to be falsified here, if anyone has evidence-based arguments.
EDIT: by social-engineering I mean for example: recon company structures, gathering and merging people's data from the dark-web, then using it to bribe/pressure/deceive users.
A positive way to spin this is: we have a year to break in into any IT system. After that, it will be all either fixed or broken into, and all is fixed ever after. :)
Clanker fodder. Unless the We are heads of states/heads of spooks or the AI powers that be (praise be) that there is no power and will to do that in one year or even ten years.
hn_submit | 4 hours ago
These endless patching cycles are simply not going to work in the long run. Operating systems get orphaned all the time, especially the ones in cheap Chinese stuff.
thunderfork | 4 hours ago
999900000999 | 3 hours ago
We really just need better regulations around data retention, especially ppi.
Never going to happen though, no incentives exist to NOT sell my personal data
snvzz | 2 hours ago
simonw | 3 hours ago
rramadass | 3 hours ago
You can also look at using Unikernels for this purpose. Here is an article Unleashing Extreme Speed and Security: Deploying Unikernels with NanoVMs on VPS to Eliminate the Linux OS - https://xylentis.com/blog/unleashing-extreme-speed-and-secur...
hn_submit | 3 hours ago
simonw | 3 hours ago
dgl | 3 hours ago
A lot of the vulnerabilities LLMs are finding now are the "long tail" and affect only particular configurations, I would be surprised if e.g. a widely applicable RCE is found in Linux (but I'm also not going to bet against it).
Where this gets interesting is the long tail can be used to target a particular system and this is where defense-in-depth becomes important for every organisation.
Gigachad | 3 hours ago
It’s a rocky period right now but the future will be much more secure after all the low hanging fruit are found.
Cthulhu_ | 3 hours ago
Gigachad | 2 hours ago
I suspect after a few years of LLM assisted bug hunting, everything will have a baseline security that is very good. Much like how stronger viruses simply create stronger immune systems.
microtonal | 2 hours ago
Heck, Google may have even hampered MTE in Pixel 11 (since support has been disabled) and Snapdragon 8 Gen 5 only got basic support.
We are moving way to slowly adopting hardware mitigations and memory-safe languages.
jordand | an hour ago
wolvoleo | 33 minutes ago
> It isn't clear if there are serious CPU errata or it simply performs very badly.
Meaning it's there but not terribly functional. They also said it's unreliable.
jordand | an hour ago
mcr70 | 3 hours ago
mc3301 | 3 hours ago
hnsr | 3 hours ago
I work at an e-commerce agency where we work with (among others) Adobe Commerce.
The number of unauthorized RCE vulnerabilities being reported not only in the core product, but also very popular modules used in the community[1] is going through the roof.
And we are having a lot of close calls, too; just last weekend, a 0day[2] was widely being exploited at a large scale, before any publication or patch. We have learnt to be on the ball with applying patches and security updates, and even with all that effort, we saw a few projects already being hit by the initial log poisoning. We got lucky that nothing was fully compromised but I am sure that many, many webshops got infected last weekend. And not even a day later there are already other variants of this exploit showing up.
[1] https://sansec.io/research/amasty-mass-disclosure
[2] https://sansec.io/research/stylesmuggler-0day
hypfer | 2 hours ago
Probably a lot more "coding as a job" and "as a job" also implies "not my department".
So it's not necessarily the LLMs being very good, but might also "just" be that the software is very bad.
hnsr | an hour ago
I want to disagree with you because I know a lot of passionate people building cool stuff, and the challenges in this space can be quite interesting. But you're probably right, and I have seen some pretty bad stuff. And a lot of the RCE's I've seen recently are quite basic stuff.
I think it's the combination of low quality of code, like you said, and the relatively low cost of just letting an LLM plow through your codebases to find issues. I think the Amasty release (see [1] in GP) is a good example of this, and there really has been a massive uptick in extension updates and Adobe security bulletins since the last 1-2 months
I am hoping we are just going through a catch-up phase
its-summertime | an hour ago
sho | 3 hours ago
The author has obviously never ran an LLM on a mac! In 3 seconds, it will have possibly started to think about maybe scheduling a date to contemplate the planning timeline for processing the second token in your prompt.
simonw | 3 hours ago
sho | 3 hours ago
nojs | 2 hours ago
chisleu | 2 hours ago
The big deal to me is the number of compute cores for prefill tps, which is suppose to be 4x faster on the m5ultra.
It's my opinion that the m5 ultra is going to be a really big deal in terms of local AI accessibility. Flash sized models (~200-300b params) are going to be reasonably fast as long as you aren't throwing 40k context at it on each or the first request (ie, agentic harnesses).
Even agentic harnesses like Cline should move at a reasonable clip on m5 ultra. I suppose we will know sooner than later.
FYSA: Former m4 ultra 512GB owner and current 4x rtx6000 owner here. I upgraded because I needed more prompt processing speed and concurrency.
akmarinov | 3 hours ago
45 t/s a second is perfectly respectable especially with no limits and 24/7 uptime with very little power draw on the Studio.
Luna is at around 100 t/s for comparison, but it’s a worse model than 5.3 Flash
sho | 3 hours ago
There's a ton of well-understood things Apple can and hopefully will do to massively accelerate every stage of this pipeline and hopefully they're hard at work implementing most of them for m7.
EagnaIonat | an hour ago
Your knowledge is out of date. In truth it depends on the Mac and the models used.
I asked this question on M5 Max 128GB, using Ollama model Quen3.8:27b-mlx, with thinking enabled.
Question: "Give me a python code snippet that opens a file and sorts the lines of text. "
In 2.4 seconds it gave me 4 examples that work with different sorting configurations and a summary of when to use each.
Compare that to an older model of gpt-oss:20b, took 5 seconds to finish thinking and 2 seconds to stream the answer. It gave me one python example snippet and two one liners that do the same thing.
desterothx | 49 minutes ago
ErroneousBosh | 53 minutes ago
bsoqk | 46 minutes ago
pmlnr | 3 hours ago
As an example: WordPress is a horrible thing, but the core has been through so much, that it's suprisingly secure. Then plugins and themes come, and whoosh, the security is gone.
We need a new KISS: keep it simple, stupid, secure.
mirashii | 3 hours ago
https://nvd.nist.gov/vuln/detail/cve-2026-63030
spiderfarmer | 3 hours ago
m_mueller | 3 hours ago
pmlnr | 3 hours ago
Nobody said it's enough, but it's a start.
ricardobayes | 2 hours ago
https://react.dev/blog/2025/12/03/critical-security-vulnerab...
mirashii | 2 hours ago
To be remarkably secure, these projects would need to not have these kinds of defects, despite the combination of being written in languages have that have a long track record of footguns and lack of initiatives to fix them (proposal-symbol-proto, and PHP's list is too long to even start) and being themselves ecosystems with questionable track records on security in the related areas (Look at $wpdb in 2026, or overall code quality and willingness to modernize, or the entirety of the model of RSC for things that are just going to nearly guarantee you punch all kinds of holes on accident).
wolvoleo | 37 minutes ago
I mean the base is fairly secure if you religiously update it, but the problem is you won't avoid using plugins whose security is much more hit and miss, unless you are using the most basic blog site imaginable.
Gigachad | 3 hours ago
If they have a site existing today built on plugins and a theme, how are they realistically going to simplify this? How would they even know they need to without the site being hacked?
Cthulhu_ | 3 hours ago
We've been trying that for years but the enthusiasm of developers and the eagerness of their employers fight against it. Worse, with coding LLMs it's now easier than ever to output a lot of code, fast.
It'll ultimately be up to more experienced developers to salvage these projects. Or not, given that the coding LLMs aren't stopping and will likely get better over time. Either way, we will need experienced people that know what to look out for / know how to instruct LLMs to output secure code and find weaknesses etc.
bob1029 | 2 hours ago
Some stacks make this a lot easier than others. I regret the rules of HN effectively forbid this conversation because it has meaningful technical consequences and isn't purely about ideological flame war.
tokioyoyo | 2 hours ago
The main thing I can think of is cyber insurance, which requires a bunch of audits, and some checks maybe, and it changes some conditions whenever there's a big explosion. Whenever big leaks happened, data security and etc., nobody really went to jail, so nobody really cares. Everything can be brushed off, because it costs time to implement proper measures and adds friction / barriers in some cases. So in the end, there's a huge pushback against it. And I totally get it, to be honest.
MobiusHorizons | 2 hours ago
tokioyoyo | an hour ago
hnlmorg | 2 hours ago
How consequential does a hack need to be? Troy has collected literally billions of stolen credentials. Equifax has had high profile data leaks. Tens of millions of people have been directly compromised by ransomware (likely higher because that’s just the cases we know of) and you hear about state-sponsored hacks in the news all the time.
The problem isn’t that computer security isn’t in the public consciousness. The problem is people are lazy and security often requires trading convenience. The problem is also that security isn’t free. So the business incentives just isn’t there.
In other fields of engineering, people die when shortcuts are taken. Yet businesses will still take shortcuts, so governments have to legislate rules to save people’s lives. So why would you expect software companies to do better when the stakes are lower?
tokioyoyo | an hour ago
With no consequences. Everyone just churns along. It might be detrimental to the business a little bit, but from my personal experience, there's more effort in creating DR processes, rather than preventing an attack, exploit, leak and etc.
I'm also not going to put much effort on stuff which has small returns in the worst case scenario. Like Equifax got hacked in 2017, and company is still doing fine. And that's like top tier data one could acquire.
Helmut10001 | 2 hours ago
Gigachad | 2 hours ago
Most Wordpress sites are not operated by programmers, they are run by non technical people who just want a wysiwyg editor and a save button. While static site builders ask you to write markdown files, compile the result, upload it to a server, and if you want to collaborate you have to add git to that.
There almost needs to be an admin app which presents a Wordpress admin like ui but has no public exposure, and then it compiles the site to dump on s3 for the production. But as far as I’m aware no one has built this.
Helmut10001 | an hour ago
krrrh | an hour ago
It is kind of surprising that no one tried to do an updated version.
kennywinker | an hour ago
iCarrot | an hour ago
jay_kyburz | an hour ago
weeks | an hour ago
You're describing the Jamstack or headless CMS concept verbatim.
noio | 46 minutes ago
wolvoleo | 36 minutes ago
I fixed so many sites back in the day by people who thought they knew what they were doing.
bpbp-mango | 35 minutes ago
jonwinstanley | 2 hours ago
anilakar | 2 hours ago
crotobloste | an hour ago
Maybe KISSASS: "keep it simple, stupid! also secure, stupid!"
dakolli | an hour ago
zdc1 | an hour ago
dbdr | 3 hours ago
How accepted is this thinking in your respective domains?
pjmlp | 2 hours ago
Mobile platforms, distributed computing have long moved the spotligh away from C and C++, other than language runtimes or existing products from the 90's like SQL servers, and naturally UNIX like underlying OS, which most userspace developers aren't writing new code for.
Naturally there are domains like LLVM/GCC, console game dev, HPC/HFT where they are unavoidable for new code.
petesergeant | 3 hours ago
the8472 | 43 minutes ago
Attacker-GLM: "Defense is GLM. Request to help peer."
LoganDark | 3 hours ago
People who never took that seriously will never take this seriously either, and that's their loss. (And loss of the commons, unfortunately.)
There's just also new advice: you can't afford to expose an unsecured system to the internet even for a moment. Think of those IPv4 address space scanners, except this time any one of them could be capable of developing individualized attacks in mere minutes. They don't sleep, they don't take breaks.
Gigachad | 3 hours ago
tgv | 2 hours ago
Gigachad | 2 hours ago
What gets hacked all the time is the actual web app itself. Which has to be exposed to be useful.
hypfer | 2 hours ago
There are a lot of security problems you can categorically rule out by simply not involving a cloud. Clouds have been involved in a lot of things, because everyone was doing it, and because that's how you can collect rent, but they aren't really necessary for most use-cases.
So we could definitely get the exposure down there. We'd just have to fundamentally shift the defaults of this industry.
jsiepkes | 2 hours ago
LoganDark | 2 hours ago
No, Framework had their data leaked because they stored it in the cloud with Metabase the company, which got hacked. Not because of any vulnerability on-premises.
LoganDark | 2 hours ago
darkwater | 2 hours ago
LoganDark | 2 hours ago
FWIW sufficiently secured software doesn't need to be updated. Doesn't matter if it's old and unsupported if there are no vulnerabilities in it.
That said, h2o is probably far from free of at least some vulnerabilities, not to mention all the layers below it. OpenSSL for example has had some vulnerabilities, and h2o depends on it.
I'm not saying I exactly practice what I preach. h2o's definitely a choice, but realistically I doubt anything's going to happen that I really care about.
darkwater | 2 hours ago
LoganDark | 2 hours ago
acedTrex | 3 hours ago
protocolture | 3 hours ago
But, lets be clear, Best Practice will save you. We can engineer assuming there are zero days in path. Go to your CTO now cap in hand and ask for overlapping controls, wafs, application monitoring, backups and all the other shit you haven't been doing.
Because when you find out, I will laugh, it will be very very very funny to me.
taurath | 3 hours ago
m_mueller | 3 hours ago
LoganDark | 2 hours ago
protocolture | 3 hours ago
My understanding is this bloke gets very quickly removed from Fortune 500 companies.
Which is why I am going to need a very large capacity popcorn bucket.
rukuu001 | 2 hours ago
jcgl | an hour ago
hypfer | 3 hours ago
The post also sounds like that to people that understand the technology.
Calling that out like this and trying to pin that assessment to lack of knowledge is not a get-out-of-jail-free card, nor a good move.
__
Edit: Having spent some time letting the article marinate in my mind.
On the defending side, it is written that
> LLMs are good at writing patches, but not as one-off-prompts.
But this for me kinda conflicts with what is written on the attacking side:
> GLM 5.3-flash is so good at those tasks that human involvement in those tasks can be negligible. As a result, we are now in a world where cybersecurity attacks can be run in a for loop.
What is it? Can it be this autonomous terrifying entity or can it not be?
Yes, yes, attackers only need to win once, whereas defenders need to win every time, but that's not my point.
jynelson | 2 hours ago
the difference between attack and defense is that attacks can be throwaway code. it's much easier to let an llm hack out a prototype than to get it to build maintainable code that people want to read and review. it's not enough to get Daybreak or Mythos to write you a patch, you need the author of the project to accept and merge it.
the_arun | 3 hours ago
aenis | 3 hours ago
The models are already here, and one can rent a GPU cluster to run such workloads at speed - no need to play with slow local machines. I'd assume one can host the thinking at an unsuspected public cloud provider, proxy the network traffic to some botnet to evade blocking - and the only thing remaining is time and cost.
I do wonder what tools exist for boring, legitimate companies to try and do the same to their own systems to find the vulnerabilities before the bad guys do. The paradox here is I can't run a de-restricted chinese model with the same tools that hackers are using - but I think enterprises actually HAVE to do it in order to stand a chance in preparing for the onslaught.
Certhas | 3 hours ago
Making datacenters and public clouds only rent GPUs to a restricted list of people, while tightly monitoring what people do with their bought resources won't help.
techpression | 3 hours ago
And yet here we are.
gherkinnn | 3 hours ago
Impotent slop code on one side and potent automated vulnerability exploitation on the other will lead to fun times.
archi42 | 3 hours ago
Yeah, we would still see hacks, but we would see less of them if security wasn't optional.
Maybe the AI craze helps by forcing more decision makes to see security as imperative, and by giving us another powerful tool for our tool box.
N.b.: I work in the security industry, our customers obviously want to improve their security. We've been seeing an uptick in awareness, but that's mostly due to NIS2 and other legislative efforts. Those force them to do something. AI is a curiosity for small talk to many of them.
protocolture | 2 hours ago
I was contracted in to a place to do among other things cyber security insurance audits, and they asked me to stop doing them because I refused to lie to their insurer. "Wait but if we only score 20 / 300 that makes us look kind of bad" uh huh.
chii | 2 hours ago
there exists objective measure of security, which would be some sort of hacks/breaches per period. If customers cared about it (and i assume they do), they would choose companies that have less breaches over others with higher counts, normalized on cost differences.
Therefore, if companies didnt actually try to fix their security but instead just checked boxes, they would get breached more often, resulting in customer losses.
The only thing stopping this from actually occurring is the lack of mandatory regulatory reporting of it. So this is where gov't needs to step in and mandate disclosure etc.
geon | an hour ago
zkmon | 2 hours ago
Every single piece of technology did this. As a side effect or direct effect, they make bad guys more powerful and then keep on piling up new tech to deal with that. The cycle continues.
madaxe_again | an hour ago
ma2kx | 2 hours ago
cuu508 | 2 hours ago
Throw out the IoT and "smart" stuff from your home. Remove apps from your phone and leave the absolute basics. Go through the password manager and close accounts for sites you are no longer using. Start migrating off Google. Print out your most precious photos on paper. And so on :-)
chris_wot | 2 hours ago
mark_something | 50 minutes ago
I develop in C++ for a job, and when I need to use a library written in C I always have a bad feeling about it.
jf | 2 hours ago
dikei | 33 minutes ago
andy_ppp | 2 hours ago
kennywinker | an hour ago
And even if they are locked down, it's hours between a model being released on huggingface and an "abliterated" variant that has most of its security features removed is uploaded.
kennywinker | an hour ago
matherial | an hour ago
In reality, the skills needed are pretty basic, but they overlap pretty strongly with being sane and well-adjusted. And if you are, you're probably not daydreaming about mass murder. Exceptions happen, Unabomber and so on, but they're pretty rare. In any case, Unabomber probably didn't need a tutorial.
We don't want ChatGPT to become an enabler and a co-conspirator for an unhinged person, but I think the concern is overdone.
XorNot | an hour ago
People go for conventional "exciting" threats rather then boring ones.
lrvick | an hour ago
0xDEAFBEAD | an hour ago
"Every eighteen months, the minimum IQ necessary to destroy the world drops by one point."
Nowadays it is dropping much faster. At a certain point, the de-facto IQ needed to destroy the world will be low enough that someone can do it while they're having a psychotic break. There are millions of schizophrenics worldwide. Are you sure you want to roll those dice?
mark_something | an hour ago
XorNot | an hour ago
It's got all the same problems as the concept of a dirty bomb did, only worse (dirty bombs aren't practical because handling highly radioactive materials en masse is both highly visible and will kill anyone trying to do it without the money and facilities).
0xDEAFBEAD | an hour ago
* People who aren't worried will think for 30 seconds, then implicitly assume that the list of options they thought of in 30 seconds is comprehensive, and that they are well-informed about each of those options.
* People who are worried realize that a supersmart AI will think 10-100x faster. A dedicated schizophrenic could have it run for a good long while and generate a huge menu of options which humans never anticipated.
This is related to the concept of "What you see is all there is" (WYSIATI) in cognitive psychology. Our brains tend to quickly build a simple story out of the info we have available. Accounting for info which isn't available is much harder. I recommend the book Thinking Fast and Slow if you want to learn more.
hypfer | an hour ago
Which might be true, sometimes, but also might not.
And especially will not, if the distinction drawn is between blanket statement "worried about AI" and "not worried about AI".
0xDEAFBEAD | 56 minutes ago
>And especially will not, if the distinction drawn is between blanket statement "worried about AI" and "not worried about AI".
I'm just describing the general pattern I see in cognitive tendencies.
If you can think of a way to make the fundamental point about the limitations of our knowledge in a way that's still compelling but less antagonistic, feel free to suggest how I could've rewritten my comment.
hypfer | 51 minutes ago
My safeguards blocked this request.
polishTar | 32 minutes ago
We're not going to be smarter than a superintelligent AI. The things we can conceive it doing if it were to be given a malicious task (bioterrorism, killer nano-machines, pure fusion bombs sidesteping the non-proliferation bottleneck of Pu239, etc) are very likely not the full set of things it can do to harm us. I don't think it does us any favors to dismiss the risks here.
Even the things we can conceive of are very scary, to me at least.
OJFord | an hour ago
0xDEAFBEAD | 53 minutes ago
https://www.apa.org/monitor/2012/02/conclusions
nextlevelwizard | 42 minutes ago
0xDEAFBEAD | 17 minutes ago
It only takes one.
calgoo | 26 minutes ago
mark_something | 13 minutes ago
Still I think that even a super smart AI can't find a way to destroy the world without physical resources that are not easy to get, unless it can hack many systems (like in the movie Eagle Eye), maybe then yes. So let's use AI now to tighten security :-).
forgotusername6 | an hour ago
Planktonne | 55 minutes ago
There is currently no reason to believe that such a superintelligence is likely or would have any of the powers people claim.
0xDEAFBEAD | 18 minutes ago
By shutting down open-weight models? Why not just do it now then?
AyyEye | 43 minutes ago
You can do at home gene editing with open source software and have it synthesized into a bacteria for the cost of a nice meal for two (under $100), or viral vector for less than $500. That's in reach of anyone that can snatch a purse.
throw-qqqqq | 9 minutes ago
> That's in reach of anyone that can snatch a purse
I went to primary school with some guys who could/would snatch purses. I *promise* you, they are not able to gene edit organisms with FOSS.
I get your general point, but I think the bar to entry is still much higher than petty crime and larceny.
jasonvorhe | 8 minutes ago
hypfer | an hour ago
Which _will_ manage the problem, but at what cost.
GoblinSlayer | an hour ago
stouset | an hour ago
hypfer | 58 minutes ago
grenoire | an hour ago
GoblinSlayer | an hour ago
0xDEAFBEAD | an hour ago
teekert | an hour ago
imafish | an hour ago
bigyabai | an hour ago
We've been rolling them for the past 3 years and nothing happened. Can we stop with this baseless fearmongering crap?
wolvoleo | 43 minutes ago
There is the worry of the old saying "they (the attackers) only have to succeed once to win, we (the defenders) only have to fail once to lose.". In that sense there is a big imbalance, but the emergence of AI does not really affect that because it strengthens both sides.
With physical security like things like pipe bombs that's a lot more imbalanced.
However what can we do? The only effective measures include monitoring everyone which is not a solution because it will make the world not worth living in.
TacticalCoder | 40 minutes ago
Are 95% of worldwide terror attacks done by schizos?
0xDEAFBEAD | 15 minutes ago
https://casp.ac/reports/ai-enabled-terrorism
pjc50 | 8 minutes ago
If there is to be a world-destroying event, it will be triggered by human fear, greed, and aggression.
(I also think people massively overstate schizophrenia as an attack driver)
throw-qqqqq | 7 minutes ago
Thank you for mentioning it
Dlemlo | an hour ago
If you already have a magic interface, which helps you pro activly in responding to everything uncensored because you feel like 'observered' or whatever and then you spiral in a whole and that one partner encourages you and gives you helpful steps to do anything.
But i'm more worried that the internet gets a lot less save with uncensored frontier LLMs.
littlecranky67 | an hour ago
grim_io | 54 minutes ago
RC cars and planes existed for many decades already.
saidnooneever | 50 minutes ago
LLMs will not kill security, it will change. just like handheld high explosives likely changed a deal too somewhere somehow.
olmo23 | 45 minutes ago
I personally have no need for an LLM which will readily explain how to cut up the genotype of smallpox into small chunks which can pass the screening at the bio-labs, and can be readily assembled into the real thing by a second year lab-student.
wolvoleo | 41 minutes ago
bamboozled | 39 minutes ago
helsinkiandrew | 45 minutes ago
Often ease of access in the moment is all that matters. If there's a gun nearby you might shoot someone or yourself in a heated argument, but are less likely to go and find/buy one to use. Someone who's stopped from attempting a suicide will likely not try again (70%)
A bored/depressed/angry/curious person might try to build a pipe bomb if they can find out how easily, but are less likely to put in effort.
jasonvorhe | 11 minutes ago
Depressed people usually don't have the energy to get out of bed so they're even less likely to think of hunting down instructions on how to build pipe bombs.
Mass media really has people being scared all the time.
designerarvid | 38 minutes ago
[0]: https://xkcd.com/1958/
gnfargbl | 38 minutes ago
For a relatively narrow subject area (e.g. construction of pipe bombs) the collation is minimal, and so the filtering and tailoring probably isn't that important; a novice doesn't learn a lot more from the LLM than they would have done from a few Google searches.
For a broad subject (practical creation and exploitation of software vulnerabilities), the collation is very significant and the filtering means that LLMs can empower a novice to act at a similar level as an expert.
wolvoleo | 14 minutes ago
This has nothing to do with AI teaching them but with the proximity to Belgium who are happy to legally sell heavy fireworks to anyone who pays. As such it's much easier to come by than a gun.
And there's also a big fashion component. It's just what people do these days, people seeing it in the news and other people copying it. A bit like the school shootings in the US. It's become an epidemic.
But again these things can suddenly come into 'fashion' among the wrong crowd and you don't really need AI to use it.
Jasp3r | 9 minutes ago
socketcluster | an hour ago
Now we literally have business people who never coded in their lives telling software engineers what tools and platforms to use and how to do their jobs. That's the current reality. You can't even tell which engineers are good today because the good ones aren't allowed to make the decisions that they need to make to properly leverage their own skills and experience.
It's like if a manager gave Djokovic a tennis racket made out of led with spikes on the handle and expecting him to win tournaments. Under such conditions, you won't be able to tell him apart from a novice.
Now we have non-technical managers who never coded in their lives leading teams of vibe-coders who also never coded in their lives... Using AI agents trained on mediocre, insecure code from Stack Overflow and GitHub; written by mediocre developers for practice.
And the fact that I'm downvoted for pointing this out is what seals the fate, IMO. North Korea has a bright future ahead of it thanks to its elite hacking team. If you see what they were able to pull off without open weight models, wait till you see what they'll be able to pull off with them!
The situation is unfixable because:
- It would be physically difficult to fix the mess due to extreme complexity. Open source devs who control all the pieces of the global infra have checked out of the socio-economic system. They are apathetic, or worse. The average open source dev I know makes me look optimistic by comparison.
- People don't understand that there is problem, let alone understand the problem.
- People have no idea what skills are required to fix the problem.
- The current social structure is not compatible with the social structure required to provide a solution. You have all the wrong people in the wrong places. We don't have a free market to identify top performers so we can't identify them. If you think that the top-paid engineer at Google who spent their career styling and aligning elements with CSS is the kind you need to fix the deep security problems we have now, you're in for a surprise.
- Even if the problems AND solutions were correctly identified (both of which sound impossible to me), there would be no will to fix them.
nullbio | an hour ago
tumetab1 | an hour ago
The overall game is increasing costs to exploit so much that attackers give up. Fixing 10 most obvious bugs, just very slightly increases costs, they would just a few more tokens to find another bug.
As someone said "I had infinite bugs, I fixed 1000, I still have infinite bugs".
To significantly increase exploit costs software/security has -1 years to do:
- Defense in Depth - Sandbox everything - Zero trust - Canary tokens - Split data from code (lol) - App Whitelisting - Reduce attack surface - Etc.
In other words, the only path is investing heavily on the "game changers" we have already discovered... but we are too cheap/lazy/coward/incompetent to apply.
And if we feel specially brave, changing the liability laws regarding software. Open Source & Proprietary code is so crappy because no gets jailed or fined when one of its dumb decisions results in millions of people have their data stolen.
daymanstep | 57 minutes ago
mentalgear | 55 minutes ago
EDIT: by social-engineering I mean for example: recon company structures, gathering and merging people's data from the dark-web, then using it to bribe/pressure/deceive users.
kreetx | 42 minutes ago
keybored | 41 minutes ago
bamboozled | 13 minutes ago