Good article on Rust's strengths with types. If you like this, you may be curious how you might build your own parse capabilities. I like the Rust crates Winnow and Nom, and also the Rust traits From and Into.
I was thinking about its implementation, too, and ended up deciding that it was probably a performance optimization.
`Vec<T>` stores all data on the heap, so getting anything out of it involves a pointer deref and possibly also an array bounds check. This `NonEmpty<T>` type keeps the first element of the list in a location that supports some low-level optimization that might make a significant difference in situations where accessing the first element is much more common than accessing subsequent elements.
This is great. Alexis King actually stated that, had she known how popular “Parse, Don’t Validate” had been, she would have written it in a language more widely used than Haskell.
With its crap type system Python might be even better, in a strange way.
Haskell's strong, static, non-reflective type system tends to make "parse, don't validate" produce code that also looks nicer. Which is great. So great that it steals a bit of the main message's valor.
In Python, though, it's really easy to just let your data be a dynamically typed list of dicts forever. So easy that parsing into something more strongly typed looks like a whole lot of extra effort. Upon looking at that sort of thing many a working Python programmer, myself included, hears the voice of GvR murmuring disparaging things about "academic" programmers down in the pit of their brain.
Which creates an opportunity to demonstrate all the ways the (arguably) more Pythonic way is actually a royal PITA when you try to make your code robust. Handling and reporting data validity errors gets scattered all over the code, which makes it annoying to maintain. Unit test suites get bloated because it's not obvious what inputs a function should be able to handle. Comments and docstrings to help keep track of this stuff begin to proliferate.
I write Python. While I read the original post and nodded along, nothing changed about how I worked until I started playing with Rust. Seeing a strongly typed language force the constraints and stop stupid errors was when the rationale finally clicked for me. Knowing that this function will never blow-up in an unexpected way changes how you think about problems.
You otherwise get accustomed to some wishy-washy blobs of data that get passed around and find it normal. Maybe you include some ad hoc guard rails here and there, which catches the egregious errors, but there is always some lingering uncertainty. Some string that should have been an int, missing key here, the object which never had the validation check, etc
It is like unit-testing - more-up front work, but I could never go back to a world where I did not get these automated assurances. Sadly, I am a grug-brain which could only feel the lesson from personal experience.
I’m a big fan of “Parse dont validate” and I feel like injecting that approach into python has so many benefits.
there’s all the code correctness stuff that we all love. But the biggest benefit is making it so much easier to maintain other parts of the stack.
Following a stack trace into somebody else’s functions and you see that the args are completely untyped or just a bunch of ‘dict[str, Any]’ is the worst. But if you see those inputs as more narrowly typed data classes it makes it so much easier to grok what the function is supposed to do
I had a very hard lesson in that once when I inherited a legacy Clojure application.
Previously I had been somewhat sympathetic to Rich Hickey’s impassioned monologues about making maps first-class and how everything gets easier when you let data just be itself. But when I had to learn my way around code that someone else had written according to that philosophy… hoo boy. Clojure quickly got demoted from being one of my favorite languages to merely being one of my favorite write-only languages.
I'm the author of typedload. For json-type of data that is loaded in dictionaries you can do A LOT of validation at runtime automatically if you define the types.
Another take, from the primary example: This is what `unwrap()` is for. I understand that the author is looking at this from a correctness and safety(?) perspective. For practical purposes, I would unwrap here. If it's less trivial than the example, unwrap with a comment explaining why it's fine.
Another angle: Unfortunately, the `first()` method being fallible here is just an issue of using an imperfect method/datatype here. This is where the author gets in to a non-empty-vec custom type. Then you are balancing using a more correct type that takes custom wiring vs a std lib thing everyone understands and takes no setup. I would lean towards this setup if I were using this non_empty_vec.first() unwrap pattern a number of times in the code base; then the setup would be worth it, at least for my own code bases. If I were exposing this in a lib others would use, I would keep the standard Vec so as to be more transparent for others.
In both views: "This is what unwrap is for" does it for me in all cases I've encountered to date. Maybe for aerospace or safety critical systems, I would have a different take.
A third take: I notice this trend in the rust community. It's not my cup of tea. Keep things simple, easy to maintain, and don't let "correctness" get in the way. In this example, I don't think it gets in the way, but I have seen this mindset lead to it getting in the way, especially in embedded, where mapping the Owernership model to hardware ends up in messy patterns and surprising assertions about embedded-101 concepts like DMA being "unsolved", "no good way", "difficult" etc.
Rust provides tools to make sure specific logic is correct if it passes the compiler. People sometimes go overboard and assume you have to type-maxx your code, regardless of complexity added by doing so.
For a untype language like js array, since it can be empty, you have to either always check the length, the item returned, or have a precondition to know the array is not empty.
All three of those cases is either code or context you’re holding in your head.
I lamented patterns of what I consider over-use of the type system in Rust (This applies to other languages as well). There is a balance: I prefer this over-use/strict use (e.g. implementing a custom non-empty Vec type) over not using typing at all, but a big margin!
You absolutely do have to deal with this in your untyped language otherwise you're just letting edge cases and runtime errors happen. The syntax will be different, it will have to happen at runtime instead of compile time, the chosen wording and language to represent the concepts and the work may even vary, but you are not exempt.
Everyone has to deal with this. The only question is whether you deal with it in the type system, through lots of checks at runtime, or buggy behavior when the user encounters cases you didn't envision.
You'd have to manually implement the traits to support the ergonomics of slices and iteration and costly reallocation if you need to pass ownership as a Vec:
I'd expect:
pub struct NonEmpty<T> {
v: Vec<T>,
}
The constructor would enforce the invariant and then you'd impl Deref and DerefMut for [T] to gain normal len/is_empty/indexing/iteration, passing as &[T] to other funcs and mutating values (which can't break the invariant).
To mutate length while preserving the invariant it's dealers choice e.g.
- add .into_vec() for unwrap/mutate/rewrap
- add invariant preserving mutators of your choice
Deref/DerefMut enables implicit type coercion rather than exposing an interface. You can choose the target type and immutable/mutable but not parts of the target type.
You can use all the slice reference methods (that do not require ownership) with:
If you DerefMut to a Vec then you won't be able to preserve the invariant.
If you want control over methods to expose then you need wrapper methods for those you want. If you want to expose some of the traits the inner type implements then there are likely derive macros available e.g. with derive_more you could expose just indexing as:
Not sure switching between languages makes for a compelling argument:
"Look how easy it is to accidentally bypass the invariant of a rust newtype by transliterating the data shape into Haskell and deriving a new type". Uh, ok.
If comparing the "risk of accident" between a newtype wrapper whose only role is enforcing the the invariant versus manually reimplementing vector and iterator semantics to use a different layout... I'd say the newtype wins that.
It would be good advice to keep a newtype that enforces an invariant as a single purpose primitive type. A building block and not a place to add other features.
There might be times I'd prefer structural enforcement e.g. something serialisation related. Converting into a non-rust format is what they are doing in their "accident"!
DerefMut would allow you to call `clear()` on `v` violating the invariant
I'd be great is there were a way to shadow methods but even then guarantees would be poor since Vec might add a new method in the future which isn't covered by invariant checks
I prefer a slightly more general rule: Make Illegal States Unrepresentable. The "Parse, don't validate" rule is a special case of MISU.
What's the difference? MISU applies even when there's no parsing-like transformation happening. For example, if you have a variable that represents the current state of a network connection, and let's say it can be Disconnected, or Connected to some IP address (this is an oversimplification).
Then one way to do it would be
struct {
connected: bool,
peer_ip: int32
}
The trouble is that this allows us to represent an illegal/meaningless state: we're disconnected but there's still some junk old peer_ip hanging in there. Even worse, we might have written
Now we could have connected = true but peer_ip = None.
The solution is to use a sum type:
type connection =
Disconnected
| Connected of int32
(sorry for using made-up syntax; I hope it's clear to anyone familiar with Rust.)
"Make Illegal States Unrepresentable" applies throughout your program, at every interface between modules or functions in the program, including but not limited to parsing input.
Yes, it'd probably be compiled into very machine code. However defining an enum for it puts the intention into the source code more explicitly, so that's considered the more idiomatic way. You'd probably have a `fn peer_id(&self) -> Option<Inet4Addr>` accessor that you'd use in the case where you'd want the Option.
Personally I don’t like using nil to represent specific states in my domain. I try to only judiciously use it to represent the lack of knowledge about some state, like if I need to reach out to some external service I don’t control, that may or may not give me an answer for my query (and conversely, I don’t like putting “unknown” cases into my enums).
jph | 7 hours ago
tjadfsaj | 7 hours ago
bunderbunder | 7 hours ago
`Vec<T>` stores all data on the heap, so getting anything out of it involves a pointer deref and possibly also an array bounds check. This `NonEmpty<T>` type keeps the first element of the list in a location that supports some low-level optimization that might make a significant difference in situations where accessing the first element is much more common than accessing subsequent elements.
andrepd | 7 hours ago
jelder | 7 hours ago
esafak | 7 hours ago
bunderbunder | 6 hours ago
Haskell's strong, static, non-reflective type system tends to make "parse, don't validate" produce code that also looks nicer. Which is great. So great that it steals a bit of the main message's valor.
In Python, though, it's really easy to just let your data be a dynamically typed list of dicts forever. So easy that parsing into something more strongly typed looks like a whole lot of extra effort. Upon looking at that sort of thing many a working Python programmer, myself included, hears the voice of GvR murmuring disparaging things about "academic" programmers down in the pit of their brain.
Which creates an opportunity to demonstrate all the ways the (arguably) more Pythonic way is actually a royal PITA when you try to make your code robust. Handling and reporting data validity errors gets scattered all over the code, which makes it annoying to maintain. Unit test suites get bloated because it's not obvious what inputs a function should be able to handle. Comments and docstrings to help keep track of this stuff begin to proliferate.
3eb7988a1663 | 4 hours ago
You otherwise get accustomed to some wishy-washy blobs of data that get passed around and find it normal. Maybe you include some ad hoc guard rails here and there, which catches the egregious errors, but there is always some lingering uncertainty. Some string that should have been an int, missing key here, the object which never had the validation check, etc
It is like unit-testing - more-up front work, but I could never go back to a world where I did not get these automated assurances. Sadly, I am a grug-brain which could only feel the lesson from personal experience.
parpfish | 4 hours ago
there’s all the code correctness stuff that we all love. But the biggest benefit is making it so much easier to maintain other parts of the stack.
Following a stack trace into somebody else’s functions and you see that the args are completely untyped or just a bunch of ‘dict[str, Any]’ is the worst. But if you see those inputs as more narrowly typed data classes it makes it so much easier to grok what the function is supposed to do
bunderbunder | 29 minutes ago
Previously I had been somewhat sympathetic to Rich Hickey’s impassioned monologues about making maps first-class and how everything gets easier when you let data just be itself. But when I had to learn my way around code that someone else had written according to that philosophy… hoo boy. Clojure quickly got demoted from being one of my favorite languages to merely being one of my favorite write-only languages.
LtWorf | 3 hours ago
the__alchemist | 6 hours ago
Another angle: Unfortunately, the `first()` method being fallible here is just an issue of using an imperfect method/datatype here. This is where the author gets in to a non-empty-vec custom type. Then you are balancing using a more correct type that takes custom wiring vs a std lib thing everyone understands and takes no setup. I would lean towards this setup if I were using this non_empty_vec.first() unwrap pattern a number of times in the code base; then the setup would be worth it, at least for my own code bases. If I were exposing this in a lib others would use, I would keep the standard Vec so as to be more transparent for others.
In both views: "This is what unwrap is for" does it for me in all cases I've encountered to date. Maybe for aerospace or safety critical systems, I would have a different take.
A third take: I notice this trend in the rust community. It's not my cup of tea. Keep things simple, easy to maintain, and don't let "correctness" get in the way. In this example, I don't think it gets in the way, but I have seen this mindset lead to it getting in the way, especially in embedded, where mapping the Owernership model to hardware ends up in messy patterns and surprising assertions about embedded-101 concepts like DMA being "unsolved", "no good way", "difficult" etc.
Rust provides tools to make sure specific logic is correct if it passes the compiler. People sometimes go overboard and assume you have to type-maxx your code, regardless of complexity added by doing so.
victorpudeyev | 6 hours ago
pyrolistical | 6 hours ago
For a untype language like js array, since it can be empty, you have to either always check the length, the item returned, or have a precondition to know the array is not empty.
All three of those cases is either code or context you’re holding in your head.
All that stuff is equivalent to a type system
well_ackshually | 5 hours ago
the__alchemist | 3 hours ago
Balance in all things?
evilduck | 2 hours ago
bigstrat2003 | 42 minutes ago
abc42 | 9 minutes ago
Fluorescence | 6 hours ago
I'd expect:
The constructor would enforce the invariant and then you'd impl Deref and DerefMut for [T] to gain normal len/is_empty/indexing/iteration, passing as &[T] to other funcs and mutating values (which can't break the invariant).To mutate length while preserving the invariant it's dealers choice e.g.
- add .into_vec() for unwrap/mutate/rewrap
- add invariant preserving mutators of your choice
eptcyka | 5 hours ago
Fluorescence | 4 hours ago
You can use all the slice reference methods (that do not require ownership) with:
https://doc.rust-lang.org/std/primitive.slice.htmlIf you DerefMut to a Vec then you won't be able to preserve the invariant.
If you want control over methods to expose then you need wrapper methods for those you want. If you want to expose some of the traits the inner type implements then there are likely derive macros available e.g. with derive_more you could expose just indexing as:
eptcyka | 2 hours ago
Rusky | 4 hours ago
Fluorescence | 4 hours ago
"Look how easy it is to accidentally bypass the invariant of a rust newtype by transliterating the data shape into Haskell and deriving a new type". Uh, ok.
If comparing the "risk of accident" between a newtype wrapper whose only role is enforcing the the invariant versus manually reimplementing vector and iterator semantics to use a different layout... I'd say the newtype wins that.
It would be good advice to keep a newtype that enforces an invariant as a single purpose primitive type. A building block and not a place to add other features.
There might be times I'd prefer structural enforcement e.g. something serialisation related. Converting into a non-rust format is what they are doing in their "accident"!
shim__ | 3 hours ago
I'd be great is there were a way to shadow methods but even then guarantees would be poor since Vec might add a new method in the future which isn't covered by invariant checks
Fluorescence | 2 hours ago
DerefMut to [T] not Vec<T>.
articulatepang | 4 hours ago
What's the difference? MISU applies even when there's no parsing-like transformation happening. For example, if you have a variable that represents the current state of a network connection, and let's say it can be Disconnected, or Connected to some IP address (this is an oversimplification).
Then one way to do it would be
The trouble is that this allows us to represent an illegal/meaningless state: we're disconnected but there's still some junk old peer_ip hanging in there. Even worse, we might have written Now we could have connected = true but peer_ip = None.The solution is to use a sum type:
(sorry for using made-up syntax; I hope it's clear to anyone familiar with Rust.)"Make Illegal States Unrepresentable" applies throughout your program, at every interface between modules or functions in the program, including but not limited to parsing input.
laszlokorte | 3 hours ago
slopinthebag | 3 hours ago
sendfoods | 2 hours ago
None = disconnected
treyd | 2 hours ago
mckn1ght | 2 hours ago
slopinthebag | an hour ago
ykonstant | 2 hours ago
Supermancho | 3 hours ago
Which is plainly moving the problem around, for types. The value validation is a much simpler problem, as a separate application-specific check.
reamaer | an hour ago
Crystal clear clarity is a nice thing to have. (As with everything, there are trade-offs)