Epic Games announced today they have created a new version control system that is now open-source as Lore. Given the proliferation and excellence of Git, you may be wondering why Epic Games is pursuing another VCS option... They are specifically catering Lore to games and entertainment purposes with large file sizes.
While there is Git LFS for large file storage with Git, Epic Games has crated Lore as a version control system designed entirely around the large file needs of modern game development as well as multimedia/entertainment purposes. Lore is designed to be fast and efficient for large files including binary files, and be easy-to-use including for 3D artists and more.
I was wondering if they built merging for binary files:
More advanced capabilities — such as partial binary merges exploiting Lore's fragment-level storage, and pluggable custom resolvers by content type — are part of the longer-term roadmap.
Not super interesting without that IMO. I'm imagining merging .psd files by layer/effect/etc. or meshes by disjoint segments (ex: you work on the stock of the gun I'll finish the barrel). They do however have a classic "check out" system where the server locks a file someone has made a temporary claim to.
Having used cvs and svn in a professional context, I absolutely love git. The cli took a little while to learn, but now all of my common workflow, troubleshooting, and branch/history hygiene commands are muscle memory. Throw in a few custom aliases for convenience and I’m good to go. The only time I leave the official git cli is to use IntelliJ’s conflict resolution GUI for complicated conflicts (the “magic” resolve gets me 90% of the way there, and I’m already using IntelliJ as my primary IDE, so it’s right there); I still commit after resolving and do everything else with the cli.
The UX is just bad. By that I mean the UX of the CLI primarily, if you throw enough tooling around it then sure it becomes usable. It was designed by Torvalds for his needs and it shows. Doing simple things is way too hard and pretty much unusable without Stack Overflow to show you the arcane commands you then repeat through bash history. Even basic things like branch tracking just weren't originally thought through holistically, let alone things that are more complex but necessary like "rewriting." And now there's the backwards compatibility issue ("don't break my workflow") so new enhancements that do things in better ways end up having to be jumbled up on top, turning it into an even more bloated and hard to use monstrosity.
I preferred hg, which wasn't perfect but a much better DVCS.
I think people just have the wrong mental model for git.
Git fundamentally is about graphs. A commit is a node in a graph that represents a snapshot of all of the tracked files in the repo. Labels are pointers that point at a node.
You can only do two things: add more nodes to the graph, or move labels to point at other nodes.
That’s it. Rewriting history? Editing commits? All impossible. No such thing in git. Adding nodes and moving label pointers is the only operations. Everything else is a syntactic wrapper around those two things.
Needing your users to have an unintuitive mental model and telling them that things they need to do are impossible because it violates some mathematical ideal is like the platonic ideal of bad UX.
If you want to solve complex problems, sometimes you need a complex mental model. SVN's mental model is pretty easy, but dealing with merge conflicts is basically why git relegated it to proportion obscurity.
If you wanna launch a satellite to Mars, you're not gonna get there with Newtonian Physics no matter how slick you make the calculator.
I can acknowledge git has a terrible CLI. They are slowly trying to improve it. But it’s the kind of thing I got the handle of after a year of being a dev full time. I don’t really get the volume of complaints I see online. It’s like if discussion about bikes online was dominated by people who hadn’t quite figured out the balance thing yet. Surely most git users have enough experience to have adapted to its quirks.
Well sure, I've got the handle of a million bad things because I'm forced to for one reason or another. But that's not a virtue. That's why it's good for there to be alternatives instead of monopolies.
Agreed -- so I was floored to see that they only have "rudimentary" (per the docs) locking, that apparently isn't scalable ... file locking is one of the major features that studios need, so it's shocking to see that not be available in a polished state on launch day.
The only thing bad about git is that people centralised in using GitHub as a common source repository hosting platform, antithetical to the idea of a decentralised VCS.
But that's just what people do: we trade some functionality for convenience.
(there's probably more bad things about git, but...)
I’ve switched almost completely to jujutsu, it’s been great. The best thing is that it can also use Git as its backend, so you can use it transparently on an existing git repo and benefit from its usability.
That looks really very cool! I like that they're pulling heavily from hg, and having a full log looks insanely useful.
I've been grudgingly using git even for my personal projects ever since bitbucket dropped hg support, as I do like to have the bells and whistles of a proper forge. This would solve that issue very well!
FWIW, with jj you can (and I think it’s recommended right now?) to use git as the backend for it so you can even use it with git as long as you don’t need lfs
This could actually be really useful to me… git lfs definitely is and definitely feels like a bolt on that doesn’t really solve the large files (and particularly large binary files) problem from the ground up. For a while (maybe still?) it even needed 2x the total stored data size free during a clone operation, which was particularly annoying for repos in the “small enough to fit on my laptop rather than the NAS, but big enough to stretch the available SSD space” kind of size range.
dvc is great if it fits your use case, but it has some “nuke the canonical copy of remote data with seemingly innocuous local-sounding commands” bear traps hidden in there that I don’t love, and even if you protect against those it utterly breaks down in a mixed use case where you might want to individually track hundreds of thousands of small but not tiny objects (e.g. images, hundreds of kB to few MB kind of size) that are too big and too binary individually for a non-lfs git repo, and too big in aggregate to want to use lfs either. Performance tanks completely, and the solution is to use directory-level tracking which then loses file-level granularity if you need to diff a list of what’s changed, so that’s a deal breaker in a lot of situations. I did talk to the lead dev about that a year or so ago and there was some appetite for working on it, but I don’t think it’s ever hit the top of the to do list - which is fair, it’s an open source project with limited resources and their own priorities and all - but was a blocker for me.
Perforce is proprietary and enterprisey as all hell, which isn’t something I want baked into the core of my projects, particularly ones I’m considering open sourcing.
Xet also caught my eye, but it does look like a kind of turbo-lfs tuned to extremely large files more than a general purpose solution. It solves some of my issues with lfs, but probably not in a significantly better way than dvc did.
So yeah, I’ll be interested to do some more reading and some playing around with this one!
After many years of exclusively using git (and before that, svn and cvs, so I concur with DrStone) I’m gonna need some time to wrap my head around the conceptual differences. I haven’t used Perforce or the other tools mentioned in this thread. In this comment I’m speculating naively about something I don’t understand, so forgive my ignorance…
Am I reading correctly that the CLI is just a client that communicates with the local loreserver, which has to be running? Maybe not, since there’s also a .lore/ directory created in the project root? I’m not clear on how/where the actual data is stored and mutated. If loreserver has read/write access to the whole filesystem that could be a concerning security risk but I assume it’s better engineered than that.
I think one advantage of a server-based design could be that it doesn’t need to draw a distinction between local repos and remotes. That might make the mental model simpler, as well as the underlying architecture, but I wonder if it complicates the ideas of “upstream” and “downstream” or of having canonical repos contributors push to.
I’ll need to read more about it, these are just my initial thoughts and they’re probably all nonsense.
[OP] kfwyre | 9 days ago
redwall_hp | 9 days ago
i.e. this is an alternative to Perforce. It's concerned with big binary blobs, not diffing plain text.
teaearlgraycold | 9 days ago
I was wondering if they built merging for binary files:
[source]
Not super interesting without that IMO. I'm imagining merging .psd files by layer/effect/etc. or meshes by disjoint segments (ex: you work on the stock of the gun I'll finish the barrel). They do however have a classic "check out" system where the server locks a file someone has made a temporary claim to.
1338 | 9 days ago
The more alternatives to the horrid git, the better,
DrStone | 9 days ago
Having used cvs and svn in a professional context, I absolutely love git. The cli took a little while to learn, but now all of my common workflow, troubleshooting, and branch/history hygiene commands are muscle memory. Throw in a few custom aliases for convenience and I’m good to go. The only time I leave the official git cli is to use IntelliJ’s conflict resolution GUI for complicated conflicts (the “magic” resolve gets me 90% of the way there, and I’m already using IntelliJ as my primary IDE, so it’s right there); I still commit after resolving and do everything else with the cli.
teaearlgraycold | 9 days ago
Why do you think git is horrid?
1338 | 9 days ago
The UX is just bad. By that I mean the UX of the CLI primarily, if you throw enough tooling around it then sure it becomes usable. It was designed by Torvalds for his needs and it shows. Doing simple things is way too hard and pretty much unusable without Stack Overflow to show you the arcane commands you then repeat through bash history. Even basic things like branch tracking just weren't originally thought through holistically, let alone things that are more complex but necessary like "rewriting." And now there's the backwards compatibility issue ("don't break my workflow") so new enhancements that do things in better ways end up having to be jumbled up on top, turning it into an even more bloated and hard to use monstrosity.
I preferred hg, which wasn't perfect but a much better DVCS.
stu2b50 | 9 days ago
I think people just have the wrong mental model for git.
Git fundamentally is about graphs. A commit is a node in a graph that represents a snapshot of all of the tracked files in the repo. Labels are pointers that point at a node.
You can only do two things: add more nodes to the graph, or move labels to point at other nodes.
That’s it. Rewriting history? Editing commits? All impossible. No such thing in git. Adding nodes and moving label pointers is the only operations. Everything else is a syntactic wrapper around those two things.
1338 | 8 days ago
Needing your users to have an unintuitive mental model and telling them that things they need to do are impossible because it violates some mathematical ideal is like the platonic ideal of bad UX.
vord | 8 days ago
If you want to solve complex problems, sometimes you need a complex mental model. SVN's mental model is pretty easy, but dealing with merge conflicts is basically why git relegated it to proportion obscurity.
If you wanna launch a satellite to Mars, you're not gonna get there with Newtonian Physics no matter how slick you make the calculator.
teaearlgraycold | 9 days ago
I can acknowledge git has a terrible CLI. They are slowly trying to improve it. But it’s the kind of thing I got the handle of after a year of being a dev full time. I don’t really get the volume of complaints I see online. It’s like if discussion about bikes online was dominated by people who hadn’t quite figured out the balance thing yet. Surely most git users have enough experience to have adapted to its quirks.
1338 | 8 days ago
Well sure, I've got the handle of a million bad things because I'm forced to for one reason or another. But that's not a virtue. That's why it's good for there to be alternatives instead of monopolies.
stu2b50 | 9 days ago
It's not really an alternative to git. Git is purpose built for repos of syntactically meaningful text; this is meant for source control of binaries.
If anything it's an alternative to perforce.
kacey | 9 days ago
Agreed -- so I was floored to see that they only have "rudimentary" (per the docs) locking, that apparently isn't scalable ... file locking is one of the major features that studios need, so it's shocking to see that not be available in a polished state on launch day.
json | 9 days ago
The only thing bad about git is that people centralised in using GitHub as a common source repository hosting platform, antithetical to the idea of a decentralised VCS.
But that's just what people do: we trade some functionality for convenience.
(there's probably more bad things about git, but...)
KawaiiCyborg | 9 days ago
I’ve switched almost completely to jujutsu, it’s been great. The best thing is that it can also use Git as its backend, so you can use it transparently on an existing git repo and benefit from its usability.
Barney | 9 days ago
I don't have much to add because one really needs to try it to see how brilliant it is.
JJ was the biggest quality of life improvement I have ever made to my coding flow.
It is actually incredible.
1338 | 9 days ago
That looks really very cool! I like that they're pulling heavily from hg, and having a full log looks insanely useful.
I've been grudgingly using git even for my personal projects ever since bitbucket dropped hg support, as I do like to have the bells and whistles of a proper forge. This would solve that issue very well!
kari | 9 days ago
FWIW, with jj you can (and I think it’s recommended right now?) to use git as the backend for it so you can even use it with git as long as you don’t need lfs
Greg | 9 days ago
This could actually be really useful to me…
git lfsdefinitely is and definitely feels like a bolt on that doesn’t really solve the large files (and particularly large binary files) problem from the ground up. For a while (maybe still?) it even needed 2x the total stored data size free during a clone operation, which was particularly annoying for repos in the “small enough to fit on my laptop rather than the NAS, but big enough to stretch the available SSD space” kind of size range.dvcis great if it fits your use case, but it has some “nuke the canonical copy of remote data with seemingly innocuous local-sounding commands” bear traps hidden in there that I don’t love, and even if you protect against those it utterly breaks down in a mixed use case where you might want to individually track hundreds of thousands of small but not tiny objects (e.g. images, hundreds of kB to few MB kind of size) that are too big and too binary individually for a non-lfs git repo, and too big in aggregate to want to use lfs either. Performance tanks completely, and the solution is to use directory-level tracking which then loses file-level granularity if you need to diff a list of what’s changed, so that’s a deal breaker in a lot of situations. I did talk to the lead dev about that a year or so ago and there was some appetite for working on it, but I don’t think it’s ever hit the top of the to do list - which is fair, it’s an open source project with limited resources and their own priorities and all - but was a blocker for me.Perforce is proprietary and enterprisey as all hell, which isn’t something I want baked into the core of my projects, particularly ones I’m considering open sourcing.
Xet also caught my eye, but it does look like a kind of turbo-lfs tuned to extremely large files more than a general purpose solution. It solves some of my issues with lfs, but probably not in a significantly better way than dvc did.
So yeah, I’ll be interested to do some more reading and some playing around with this one!
[Edit] Typos
balooga | 9 days ago
After many years of exclusively using git (and before that, svn and cvs, so I concur with DrStone) I’m gonna need some time to wrap my head around the conceptual differences. I haven’t used Perforce or the other tools mentioned in this thread. In this comment I’m speculating naively about something I don’t understand, so forgive my ignorance…
Am I reading correctly that the CLI is just a client that communicates with the local
loreserver, which has to be running? Maybe not, since there’s also a.lore/directory created in the project root? I’m not clear on how/where the actual data is stored and mutated. Ifloreserverhas read/write access to the whole filesystem that could be a concerning security risk but I assume it’s better engineered than that.I think one advantage of a server-based design could be that it doesn’t need to draw a distinction between local repos and remotes. That might make the mental model simpler, as well as the underlying architecture, but I wonder if it complicates the ideas of “upstream” and “downstream” or of having canonical repos contributors push to.
I’ll need to read more about it, these are just my initial thoughts and they’re probably all nonsense.