Congrats on the launch. "Successfully executing a trajectory doesn't mean the inspection worked" is a great frame, closing the loop on the measurement itself is the hard part.
> We use AI where semantic understanding and flexibility are useful, such as interpreting less structured information or understanding what in an unfamiliar scene is relevant to a procedure. Once the system knows what physical interaction it needs to perform, we prefer explicit geometry, planning, optimization, and control where possible...
> A few centimeters of error may not matter much when navigating down a hallway, but it matters if a sensor is supposed to remain normal to a curved surface.
This is a great specific example of where the creativity (read: randomness) of AI runs into a wall. Do you see this changing over time as models improve, or do you expect that safety-critical / highly specific tasks will always require a more explicit set of instructions?
That creativity/randomness presents itself in different ways depending on the application of the AI. In robotics, that randomness means variance in the motion of the robots physical body and arm(s). I think over time as labs get better at understanding and quantifying the uncertainty of model outputs there could absolutely be a place for full end-to-end AI in safety-critical environments. But today, we believe that is not the best approach for these applications.
Congrats on the launch. I've been in the robotics industry for a decade now, I've seen that boundary lie at many different places. I'm not sure there's a clear cut answer.
Self plug: I'm the founder of Clipper, a container registry that has 10x faster pulls and 7x faster builds over DockerHub, targeted at robotics. I got tired of robotics deploys taking all day and fixed it myself. If you're interested in that or just want to talk shop, let's chat.
Docker is built for web. The mental model is that the internet speed is relatively fast, the image is relatively small, and the device you're pulling onto is starting with essentially no data. All three of these assumptions are wrong for robotics:
- I've had someone in the Docker discord tell me at length that my container shouldn't be bigger than 1GB, which is impossible the moment you pull in CUDA or any ML libraries.
- Robots pull over 3G/Starlink/slow customer WiFi
- Robots are edge devices! They have previous related images, they aren't pulling onto a fresh disk like on web.
It's super common to have a layer in your Dockerfile that has several GB worth of dependencies. If you touch one of the dependencies, it thrashes that layer and anything below it. To quote the CPO from a company I left "Anytime anyone looks at the container the wrong way its a 13g dl over a shit cell connection or starlink". It's extra shit because most of the data the robot is pulling, the robot already has in another layer!
Clipper solves this by breaking apart Docker layers and indexing the files within. When you pull an image it will reuse related files in images you already have on your device. This has other side benefits as well, like being able to mount layers as networked filesystems and much more easily being able to run P2P updates.
It's less that robotics needs a different container registry and more that robotics needs a better container transport, and I've built the infrastructure around it for my registry.
Very interesting. We’ve for sure felt the pain of large robotics images and slow deployments, especially when CUDA and ML dependencies are involved, or when adding a seemingly small ROS package pulls a long chain of transitive dependencies. The file-level deduplication approach is clever. Curious how transparent it is to existing Docker/OCI workflows. Does it require any changes to the client, or it mostly sits underneath the existing tooling?
It mostly sits underneath existing tooling. I replace the push/pull client (run `clipper pull` instead of `docker pull`, and optionally the BuildKit driver (to get build outputs natively in my format). That pull command ingests the image into your chosen runtime, meaning your docker/podman/k8s run commands remain the same. If you're interested I can set up a demo/call/coffee.
> We use AI where semantic understanding and flexibility are useful, such as interpreting less structured information or understanding what in an unfamiliar scene is relevant to a procedure. Once the system knows what physical interaction it needs to perform, we prefer explicit geometry, planning, optimization, and control where possible. We're interested in the marriage between the two rather than trying to make every part of the robotics stack learned.
Congrats on the launch. I love this part, because it is also the working theory behind my robot clearing robot project (http://www.frost-e.com). I'm new to robotics, but it seems to me there's a gap between classic robotics tools and approaches (ROS2-based navigation, local YOLO-based object detection etc) and the video-input-to-actuator-output VLA models that robot labs are showcasing. In fact, I kinda think the big labs are trying to boil the ocean - long horizons, 6DoF actuator control, cross embodiment, generalization, etc etc.
Seems to me is that if you restrict the problem space, have well defined tasks, then marry the semantic understanding and reasoning of VLMs with the geometric tools of classical approaches, then you could have something practical and economical.
Yeah, you hit some great points! From a broader perspective not only are we seeing the sort of ai vs classical approaches camps at different conferences and companies, but industry wide we wonder about another shift.
A shift from a single company handling all software and hardware inhouse, to a sort of divergence of the two. Kind of cool to think about, given how that sort of divergence seems to have happened in a few other areas (a dell computer runs windows, an LG TV runs apple TV, your iphone runs airbnb, spotify, instagram)
Great point and realisation on many realities of robotics. Most successful and scaling projects have been traditionally in controlled environments. This always reminds me of William Shatner in "The Future is Now" ad in 1984: https://www.youtube.com/watch?v=pb_-BWHMQlc
When we can control the environment precisely, we should as it simplifies so much---we can clip the world to platonic geometries and apply common tricks and trigonometry.
But when we cannot control the environment, really really much harder. Self-driving cars are a good example how hard it is, and still mainly focusing on relatively structured environments like cities.
Arbitrary environment is really where many expect AI/ML/probabilistic robotics to bring some necessary flexibility. And still out of reach today, except some astonishing use cases like Roombas (well, more like a best effort approach, still).
Congrats on the launch! At Transitive Robotics we make full-stack modules so robotics companies like you can build their operational systems faster and with greater quality. We have a number of modules you might be interested in, e.g., for remote video-streaming and tele-op/-assist: https://transitiverobotics.com/caps/.
Let me know if you want to chat.
All the best! Very clear framing and definitely needed service. I remember in 2011 not too far from Fukushima after the problems with nuclear reactors. The company here faced very difficult inspection problems, including through rubbles. There were robots able to deal with some problems, but not in Japan, and anyway they had to act fast and so sent people. It apparently was safe enough, but able robots would have been so much safer.
A technical question: To scan a surface precisely, I'd rather put switches on the end effector aside the sensor, such that the arm places itself roughly above the surface to scan, pushes until the switches trigger, and scan keeping the switches on. A simple touch sense, basically. All trad robotics that smoothly follows many flat and curved surfaces. Not at all against AI here, it seems the whole thread is a constructive approach mixing the best tools to concrete targets. Not viable approach in your scenarios?
Thoughts on your concluding questions:
> where the abstraction boundary in robotics should sit. What should come from the robot manufacturer, what belongs in an application layer, and what will inevitably remain specific to the facility?
I really like when there is a "double SDK", with a low level one to target actuators and sensors individually (motor 2 of leg 4), and a higher level one with pre-defined scenarios (move forward, whatever the bot is a bipede, a quadcopter or a slug). These two API types often allow for easier work at the application layer. The SDK is also the main specific software that remains with the maker (facility?)
> industries where you've seen physical inspection tasks that look trivial to a person but are surprisingly difficult to automate
Too often a problem I have seen is... elevators. More generally unstructured environment, and environments explicitly tailored for humans. Proposals range from wireless interfaces (and so HW infrastructure dependent and hard to scale), to adding a manipulator to, say, push buttons on an elevator (and this means extra software to manipulate, recognise buttons, etc).
akshay_akula | 13 hours ago
[OP] Salem_robotics | 12 hours ago
akshay_akula | 10 hours ago
droidjj | 13 hours ago
If so, as someone who lives close to Salem, I like it :)
Good luck to you guys.
[OP] Salem_robotics | 12 hours ago
vegnus | 12 hours ago
janakpanthi | 12 hours ago
alexchantavy | 12 hours ago
sidcool | 12 hours ago
janakpanthi | 11 hours ago
lesiva | 12 hours ago
> A few centimeters of error may not matter much when navigating down a hallway, but it matters if a sensor is supposed to remain normal to a curved surface.
This is a great specific example of where the creativity (read: randomness) of AI runs into a wall. Do you see this changing over time as models improve, or do you expect that safety-critical / highly specific tasks will always require a more explicit set of instructions?
[OP] Salem_robotics | 12 hours ago
sidcool | 12 hours ago
janakpanthi | 11 hours ago
a_t48 | 11 hours ago
Self plug: I'm the founder of Clipper, a container registry that has 10x faster pulls and 7x faster builds over DockerHub, targeted at robotics. I got tired of robotics deploys taking all day and fixed it myself. If you're interested in that or just want to talk shop, let's chat.
ryanisnan | 11 hours ago
a_t48 | 11 hours ago
Clipper solves this by breaking apart Docker layers and indexing the files within. When you pull an image it will reuse related files in images you already have on your device. This has other side benefits as well, like being able to mount layers as networked filesystems and much more easily being able to run P2P updates.
It's less that robotics needs a different container registry and more that robotics needs a better container transport, and I've built the infrastructure around it for my registry.
tabith | 6 hours ago
a_t48 | 4 hours ago
janakpanthi | 4 hours ago
a_t48 | 4 hours ago
martythemaniak | 9 hours ago
Seems to me is that if you restrict the problem space, have well defined tasks, then marry the semantic understanding and reasoning of VLMs with the geometric tools of classical approaches, then you could have something practical and economical.
[OP] Salem_robotics | 7 hours ago
A shift from a single company handling all software and hardware inhouse, to a sort of divergence of the two. Kind of cool to think about, given how that sort of divergence seems to have happened in a few other areas (a dell computer runs windows, an LG TV runs apple TV, your iphone runs airbnb, spotify, instagram)
hic | 3 hours ago
When we can control the environment precisely, we should as it simplifies so much---we can clip the world to platonic geometries and apply common tricks and trigonometry.
But when we cannot control the environment, really really much harder. Self-driving cars are a good example how hard it is, and still mainly focusing on relatively structured environments like cities.
Arbitrary environment is really where many expect AI/ML/probabilistic robotics to bring some necessary flexibility. And still out of reach today, except some astonishing use cases like Roombas (well, more like a best effort approach, still).
chfritz | 7 hours ago
janakpanthi | 5 hours ago
hic | 2 hours ago
A technical question: To scan a surface precisely, I'd rather put switches on the end effector aside the sensor, such that the arm places itself roughly above the surface to scan, pushes until the switches trigger, and scan keeping the switches on. A simple touch sense, basically. All trad robotics that smoothly follows many flat and curved surfaces. Not at all against AI here, it seems the whole thread is a constructive approach mixing the best tools to concrete targets. Not viable approach in your scenarios?
Thoughts on your concluding questions:
> where the abstraction boundary in robotics should sit. What should come from the robot manufacturer, what belongs in an application layer, and what will inevitably remain specific to the facility?
I really like when there is a "double SDK", with a low level one to target actuators and sensors individually (motor 2 of leg 4), and a higher level one with pre-defined scenarios (move forward, whatever the bot is a bipede, a quadcopter or a slug). These two API types often allow for easier work at the application layer. The SDK is also the main specific software that remains with the maker (facility?)
> industries where you've seen physical inspection tasks that look trivial to a person but are surprisingly difficult to automate
Too often a problem I have seen is... elevators. More generally unstructured environment, and environments explicitly tailored for humans. Proposals range from wireless interfaces (and so HW infrastructure dependent and hard to scale), to adding a manipulator to, say, push buttons on an elevator (and this means extra software to manipulate, recognise buttons, etc).