I think Rust's choice to panic on OOM is the right choice for 95% of code people write. It would be absurdly verbose, and OOM can be very tricky to recover from without careful top down design because it's very easy for error handling code to itself try and allocate (print/format a message) leading to a process abort. The mistake was not having the fallible APIs on allocating containers from the start, for the users that do care and can tangibly recover.
Depending on the OS you likely won't even get the chance to handle the error because the allocation never fails, instead you just over commit and kill the process when physical memory gets exhausted.
I guess what I'm trying to say is designing your language's error handling primitives around OOM is probably not a good idea, even in a systems programming language, because it's a very rare class of generally unrecoverable error that only very carefully designed code can recover from.
Anyhow allocating isn't that absurd either. It keeps the size of Result<T, E> down by limiting the 'E' to be a pointer. Smaller Result<T, E> can help the compiler generate better code for passing the return value, and the expectation is the error path is rare so the cost of allocating might be outweighed by the lighter Result<T, E> benefits. Exceptions take a similar standpoint, throwing is very slow but the non-exceptional path can (theoretically) run without any performance cost.
Depending on the OS you likely won't even get the chance to handle the error because the allocation never fails, instead you just over commit and kill the process when physical memory gets exhausted.
I guess what I'm trying to say is designing your language's error handling primitives around OOM is probably not a good idea, even in a systems programming language, because it's a very rare class of generally unrecoverable error that only very carefully designed code can recover from.
Anyhow allocating isn't that absurd either. It keeps the size of Result<T, E> down by limiting the 'E' to be a pointer. Smaller Result<T, E> can help the compiler generate better code for passing the return value, and the expectation is the error path is rare so the cost of allocating might be outweighed by the lighter Result<T, E> benefits. Exceptions take a similar standpoint, throwing is very slow but the non-exceptional path can (theoretically) run without any performance cost.