fable 5.1 generated an interactive 3D union square, and the agent filmed its own tour guide vid inside it. you can walk Powell to Stockton, read the actual storefronts, cross a working intersection, watch a cable car go by. went inside Apple and the Nintendo store, lower level included
Not really. Whether you agree with LeCun's assertion that "LLMs Are a Dead End" or not, world modeling is the same domain. "World Modeling" in this post is confusing because it seems to reference the deep, specific machine learning that goes into these frontier world models and self-driving cars, where it's really just vibe coding turned at the problem of modeling the sort of 3D worlds you'd find in a video game.
I wish you would provide more information. About time, costs, reliability (how many duds did you have? did you need to do any babysitting or etc) and so on
But it is a very impressive looking demo, for sure.
this is a one-shot result but i have a really really lengthy prompt: https://github.com/PhiloLabs/fable51-worlds/blob/main/union-... with clear guidance in using subagents and self-QA loop.
~2 hour (extensive subagents usage), total ~8M tokens, ~$33 under API
Did you need to iterate on the prompt, or did you have a model help you author it? I frequently have problems with orchestration instructions in-prompt, and your is huge. Maybe this is just better with Fable? I honestly haven’t used it much.
this is really neat. I wonder if people will start building open world games or "AR" games in the vein of Pokemon Go and Ingress based on similar tech. I've had a similar idea for a long time but don't think it was feasible before now due to AI
yes we're working on it! trying to push a few open world rpg games with real economy and game design. it's also super interesting to benchmark the current models' capability in this direction, since this is a naturally hard and multimodal coding task
Cool. But the question remains, what is the point of this? I don't mean to pick on this project specifically, but it's indicative of AI usage in general. You end up with something that's like "wow an AI made this" but it's not really anything that interesting or useful in and of itself. I've yet to see a single agent generated piece of software above that bar.
Because it’s a very complex task to do especially with any level of quality and consistency.
Classical algorithmic approaches for data to CAD are quite unreliable heck commercial CAD software still struggles turning scanned 2D plans into sketches with any degree of reliability.
Something of this scale would normally require 1000’s and 1000’s of man hours in manual modeling, placement and quality control.
Some of the latest prompt to CAD agents and models I’ve seen are the most impressive use of AI I’ve seen in a long time.
At least for me this is rather exiting since it gives me the ability to turn many more ideas into reality as a hobbyist rather than spending the entire weekend in Fusion 360.
Why don't you try to do it yourself with Fable and compare it with this one? Even with AI there are still a lot of work, knowledge, and gotchas involved. What's the point of such comment?
the topology/texturing critique is fair for mesh generation, but code-generated worlds mostly sidestep it. when the model writes three.js or blender scripts, geometry comes from primitives, csg, and parametric construction, so topology is clean by construction rather than something you clean up after. texturing is still a gap, agreed, though procedural materials cover more than people expect.
Cool demo! That's funny. I literally sketched out something similar 30 minutes ago on paper. I wonder if there was some recent, semi-related article/link that subconsciously prompted the relevance.
we're working on a survey paper about world modeling via code, with folks from frontier labs (qwen omni, oai, etc) and academic institutions (e.g., oxford, stanford, etc) reach out to us about collab: team@philolabs.ai
I have experimented with this type of modeling for a RTS game I'm building: https://playwarpact.com
Few things:
1. Opus 5 is just as good for this tbh and cheaper.
2. They don't generate optimized 3d models. They have high poly count for simple geometries.
3. A better approach I've utilized for game ready assets is to use the model to create low poly silhouettes of the 3d model and then bake textures that include a lot more details like windows, doors etc with tools like Meshy. Can post a tutorial if there's interest.
The models need some more RL to be able to do this autonomously.
why would you force me to login to see what the game is, I have zero information and you're asking me for a password or my google information. hostile and creepy.
Sorry, the game is still in dev and recently opened up for early access. I am working on a marketing page for the game that would link to the gameplay.
I’m interested! I struggled to get goos detail on my Back to the Future Delorean 3D animation on my docs hero: https://contextify.sh/docs/
I posted this previously and people wanted more info so here is how I built it, with a deeper dive into some film scene recreation I worked on focusing on a scene from Apocalypto:
It is a 3D-ish model being animated by a procedural motion system, rather than a pre-rendered/model animation.
The car started from a low-poly DeLorean glTF. I had the model rebuild the geometry into inline Canvas code rather than just dropping a model into three.js.
The flight itself is procedural: position/rotation, hover, pitch, wobble, flames, camera, etc. are all driven by code and can be scrubbed/debugged.
That link has a link to an interactive page where you can adjust the car’s path and when it reaches 88mph if you want to see.
It is not mobile friendly atm, though.
I would like to apply textures to the people and scenery in the gauntlet scene. Just haven’t had the time or tokens, as getting motion and pathing and camera angle right seemed the mvp.
> Opus 5 is just as good for this tbh and cheaper.
It depends on the fidelity, there's a threshold slightly over "low poly" that Opus cannot get over. Once you get into creating foliage l-systems, or physical mob animation, or a house with realistic appliances, Opus is under the convergence threshold for a world model no matter how much time you give it, it will flail around and say it's done even though it's nowhere close to plausible. Fable takes forever but knows how to zoom in and out on the abstractions.
> They don't generate optimized 3d models. They have high poly count for simple geometries.
By default LLMs will do the quick prototype thing and slap some primitives into a THREE.js scene. Which is probably correct because most users don't know or care. But if prompted Fable will connect the manifolds/sculpt with a mesher, instance out the pieces, etc. and make you an efficient art pipeline. You can make it optimized. Just not in 10 minutes.
> A better approach I've utilized for game ready assets is to use the model to create low poly silhouettes
You can also let Meshy make the geo and then fit it back to the world with PCA + BB's. Diffusion models are still 10x better at making physically plausible geometries than LLMs, plus they are faster all things considered. One technique I've found that works well is socketing; let the LLM generate the high level structure with blockouts for sockets, then slot in the higher poly objects that fit. Which is closer to the professional approach for world design.
These things always look impressive but I fail to see how they are usable beyond demos or simple games. In my experience the topology is often a mess and texturing is difficult as well
i discovered doing this in claude design and i can tell you... it's useful in a professional domain. Although I am not going to share what I'm using it for, to keep an edge in my domain ;)
Woah amazing clips thanks for sharing! The GH repo is definitely in early stages and prob laying the ground work for using the real world as part of the pre production process for a game/movie, etc.
This reminds me of the experience VRML wanted to be 3 decades ago. The only difference is that links went to other worlds living on another server. But that seems trivial to implement with this.
yes experimenting with it actually, will update here! in fact we've generated most of the tourist spots in sf, should be reflected in the repo soon too
Fundamentally it’s an image model predicting out frames of a sequence.
It’s not enough to call it just an image model though. Not quite video either because it’s so specific to rendering images of the world.
Thus, “world model”?
The problem I have with “world” is it should imply so much more depth. To me this is actually a “POV image model” or “First-person perspective model” (maybe we call this an FPM).
There could also be TPMs (third-person models).
This maps better to modern game and 3D dev phrasing, where “world model” might imply 3D object collections spatially organized in a cohesive file format - complete with interactivity, audio, and all the baseline constituents of what one might call a “world”.
FPMs and TPMs, think about it. Reserve POVMs for photographic realism.
[OP] surreal_ | 9 hours ago
check source code + more worlds soon
giancarlostoro | 8 hours ago
> Geometry is derived from OpenStreetMap (ODbL) and USGS 3DEP (public domain).
This is probably how I would have tackled this, saves on time, especially if Fable just writes code to convert OSM data to a reasonable ThreeJS model.
andy99 | 6 hours ago
[OP] surreal_ | 6 hours ago
~2 hour (extensive subagents usage), total ~8M tokens, ~$33 under API
jaybuff | 8 hours ago
[OP] surreal_ | 6 hours ago
benatkin | 5 hours ago
hadlock | 8 hours ago
Fake edit: there is a longer video here 1 min long: https://github.com/PhiloLabs/fable51-worlds/blob/main/union-...
I would be especially curious to see the NPC person/car logic and if they're on rails or what, that's a pretty good NPC density for a demo.
Willish42 | 8 hours ago
Here ya go: - https://github.com/PhiloLabs/fable51-worlds/blob/main/union-... - https://github.com/PhiloLabs/fable51-worlds/blob/main/union-...
avdelazeri | 8 hours ago
[OP] surreal_ | 6 hours ago
threecheese | 3 hours ago
Willish42 | 8 hours ago
[OP] surreal_ | 6 hours ago
ramesh31 | 8 hours ago
dogma1138 | 7 hours ago
Classical algorithmic approaches for data to CAD are quite unreliable heck commercial CAD software still struggles turning scanned 2D plans into sketches with any degree of reliability.
Something of this scale would normally require 1000’s and 1000’s of man hours in manual modeling, placement and quality control.
Some of the latest prompt to CAD agents and models I’ve seen are the most impressive use of AI I’ve seen in a long time.
At least for me this is rather exiting since it gives me the ability to turn many more ideas into reality as a hobbyist rather than spending the entire weekend in Fusion 360.
trees101 | 4 hours ago
jryle70 | 7 hours ago
ChickeNES | 6 hours ago
Begs the question, an earnest one: what software would be above that bar?
[OP] surreal_ | 6 hours ago
vopi | 7 hours ago
[OP] surreal_ | 6 hours ago
T-A | 6 hours ago
https://news.ycombinator.com/item?id=49525160
kodefreeze | 7 hours ago
Few things:
1. Opus 5 is just as good for this tbh and cheaper.
2. They don't generate optimized 3d models. They have high poly count for simple geometries.
3. A better approach I've utilized for game ready assets is to use the model to create low poly silhouettes of the 3d model and then bake textures that include a lot more details like windows, doors etc with tools like Meshy. Can post a tutorial if there's interest.
The models need some more RL to be able to do this autonomously.
lukan | 7 hours ago
There is.
kodefreeze | 7 hours ago
dakolli | 6 hours ago
kodefreeze | 5 hours ago
bredren | 6 hours ago
I posted this previously and people wanted more info so here is how I built it, with a deeper dive into some film scene recreation I worked on focusing on a scene from Apocalypto:
https://banagale.com/cinematic-canvas-ai-film-animation.htm
I’d love to improve the look of these things.
kodefreeze | 5 hours ago
bredren | 5 hours ago
The car started from a low-poly DeLorean glTF. I had the model rebuild the geometry into inline Canvas code rather than just dropping a model into three.js.
The flight itself is procedural: position/rotation, hover, pitch, wobble, flames, camera, etc. are all driven by code and can be scrubbed/debugged.
That link has a link to an interactive page where you can adjust the car’s path and when it reaches 88mph if you want to see.
It is not mobile friendly atm, though.
I would like to apply textures to the people and scenery in the gauntlet scene. Just haven’t had the time or tokens, as getting motion and pathing and camera angle right seemed the mvp.
avaer | 6 hours ago
It depends on the fidelity, there's a threshold slightly over "low poly" that Opus cannot get over. Once you get into creating foliage l-systems, or physical mob animation, or a house with realistic appliances, Opus is under the convergence threshold for a world model no matter how much time you give it, it will flail around and say it's done even though it's nowhere close to plausible. Fable takes forever but knows how to zoom in and out on the abstractions.
> They don't generate optimized 3d models. They have high poly count for simple geometries.
By default LLMs will do the quick prototype thing and slap some primitives into a THREE.js scene. Which is probably correct because most users don't know or care. But if prompted Fable will connect the manifolds/sculpt with a mesher, instance out the pieces, etc. and make you an efficient art pipeline. You can make it optimized. Just not in 10 minutes.
> A better approach I've utilized for game ready assets is to use the model to create low poly silhouettes
You can also let Meshy make the geo and then fit it back to the world with PCA + BB's. Diffusion models are still 10x better at making physically plausible geometries than LLMs, plus they are faster all things considered. One technique I've found that works well is socketing; let the LLM generate the high level structure with blockouts for sockets, then slot in the higher poly objects that fit. Which is closer to the professional approach for world design.
maest | 3 hours ago
I read it as "play, warp, act"
hartator | 3 hours ago
jatora | an hour ago
vismit2000 | 28 minutes ago
julien_dev | 7 hours ago
eagleinparadise | 5 hours ago
ryan_n | 5 hours ago
nomel | 3 hours ago
Uncharted 4: https://youtu.be/NedDxIGQVs0&t=147
Coco: https://youtu.be/nl_JkjgHfFU&t=10
Up: https://www.youtube.com/watch?v=JHfLGgOs6gY
And, eventually you can just run it through Nvidia DLSS 7 for better hair physics and textures. ;)
breadsniffer01 | 3 hours ago
pelagicAustral | 7 hours ago
lanyard-textile | 7 hours ago
bb88 | 7 hours ago
bastawhiz | 6 hours ago
gimlids | 6 hours ago
minimaltom | 6 hours ago
mrinterweb | 6 hours ago
[OP] surreal_ | 6 hours ago
ameddserM | 5 hours ago
lauralin | 4 hours ago
hartator | 3 hours ago
heyl987 | 30 minutes ago
luciana1u | 3 hours ago
WordNotSee | 3 hours ago
It’s not enough to call it just an image model though. Not quite video either because it’s so specific to rendering images of the world.
Thus, “world model”?
The problem I have with “world” is it should imply so much more depth. To me this is actually a “POV image model” or “First-person perspective model” (maybe we call this an FPM).
There could also be TPMs (third-person models).
This maps better to modern game and 3D dev phrasing, where “world model” might imply 3D object collections spatially organized in a cohesive file format - complete with interactivity, audio, and all the baseline constituents of what one might call a “world”.
FPMs and TPMs, think about it. Reserve POVMs for photographic realism.
OldDrunkGuy | 2 hours ago
I am your biggest hater.
But god dammit if I’m not also your biggest respecter.
reaches out for a handshake
You guys are onto something with this one.
Keep going.
You crazy bastards!
You crazy fuckin bastards you hear me?! Hahahahah
WOO!
WOO!