Multiple return values and errors
I was pointed at a rather simplistic article on arstechnica about why most programming languages only return a single value from a function, and thought about my “first class tuples” post from a few weeks ago. Python can effectively support multiple return values due to its built-in tuple unpacking, and doesn’t need to resort to heavyweight or unnecessarily verbose custom types or dictionaries (see tuples post for my rationale). I use “multiple return values” in this way often for the internals of a package where the API can be more fluid and implicit.
This wouldn’t be interesting but I was rereading some articles by the most excellent Laurence Tratt, including this one: A Proposal for Error Handling, which is about his experience writing very fault-tolerant software in C with error codes and comparing that to languages with exceptions. His proposal basically comes down to language support for (return value, error code), and turning error codes not explicitly handled by the caller into exceptions that should only rarely be caught. I know that sounds *just* how exception handling works, but there are subtle yet vital differences he provides in his article (which is worth a read, as I rarely see people who “get” both error-code programming and exception-based programming so thoroughly).
The talk of multiple return values and Laurence’s proposal reminded me of GoLang, and its support for (return value, error code) and panic/recover (like exceptions that should only rarely be caught). The design has always intrigued me, though I have never written a large enough Go program (or feel I understand Go idioms enough) to have a strong opinion. But given Go’s primary use for service programming, in light of Laurence’s article it seems like a very solid design.
But back to Python. I’m curious if anyone ever implemented a system where (return value, error code) was used by convention instead of exceptions as the primary error-handling mechanism? Because we have this built-in tuple unpacking, we can use (return value, error) without explicit Go-style language support, but I’m curious if anyone has done it and what their experience has been.
I think a much better example of the pattern is Erlang rather than Go:
1. it meshes with Tratt’s goal as it’s first and foremost a tool for fault-tolerant systems
2. it does not use special syntax for MRV, just bog-standard tuples and tuple unpacking
3. in my opinion, pattern matching makes handling and rewrapping errors better than Go’s method
4. Erlang makes it trivial to transform an error value into a fault, perform a success match `{ok, Value} = some_call()` and it’ll automatically fault on errors
5. Erlang is dynamically typed, so tuples make perfect sense.
The latter point is by far my biggest issue with Go’s error handling through (special case) MRVs: it’s a statically typed language and statically typed languages can type-encode errors (thus being able to type check error handling) instead of delegating that to values. See option types (MLs, Haskell as “Maybe”), Either (Haskell, Scala) or more generally union types, of which option and either types are but very basic applications. Go also requires the existence of null references (for the invalid part of the pair), which I have a strong dislike for (again, in the context of a statically typed language).
As for Python, I’d expect a significant lack of fun as no API other than the internal one will use the pattern and you’ll find yourself with a mix of exceptions handling, return tuples and the odd C-style error code. It will also have the same (fairly bad I think) verbosity issue as Go’s error handling (it’ll actuality be slightly worse as Python does not support a pattern akin to `if value, err := call(); err != nil { etc…`
Hi! My scenario is not exactly what you describe, but I figured I’d mention it as a use case for multiple return values:
A colleague implemented something relatively similar to what you describe, in the context of a method enabling batch calls. We have a python client talking to a python backend, and round-trips are costly. So he created a ‘batch’ backend module containing methods like `batch_function_call`,
– … taking [(functionA, parametersA), (functionB, parametersB), …] as parameters
– … and returning [(returnvalueA, exceptionA), (returnvalueB, exceptionB), …]
→ So far it did the trick (i.e. when a client needs to do a bunch of similar backend queries, it can pack them into a single batch call, avoiding costly round-trips), then we handle exceptions client-side by looping over the (retval, exc) couples and raising the first potential exception we find, or processing the retvals if no exception is found.
A proper refactor enabling the backend to handle multiple inputs c/would have been cleaner, but this approach enabled us to achieve the same result very quickly, with no breaking backend API change and little performance loss, as opposed to multiple client→backend calls.
I’m curious to see what other kinds of use cases will pop up here…
Thanks for the post and happy new year!
Bruce Eckel gave a talk on this idea at PyCon 2013 last year, that you might be interested in watching—here is pyvideo’s link:
http://dev.pyvideo.org/video/1683/rethinking-errors-learning-from-scala-and-go
The first problem is the underlying assumption that writing highly fault-tolerant systems in software is a desirable task. That assumption is rarely true. Even when it is true, it’s typically only valid when you have highly fault-tolerant hardware as well (which may reduce the burden on the software considerably, depending on what you’re doing). So it’s pointless to talk about fault-tolerant software in a vacuum. We need to know what needs to be achieved and how it will be achieved.
Moreover, one could argue that if you’re treating OOM as fatal then you’re not really highly fault-tolerant And guess how many highly fault-tolerant systems avoid OOM conditions in the first place? They avoid dynamic memory allocation altogether. This is common in space-rated systems, for example. By analogy, we can quickly see that there are plenty of fault-tolerant techniques that have no purpose in mainstream programming. FAA requires the most critical avionics systems to have no dead code whatsoever and to be MC/DC coverage tested. When you’re doing that sort of testing, you give up all sorts of things.
He also picked a pretty poor example by using write(2), since you cannot handle errors from write(2) properly (if EINTR or EAGAIN can occur) via a mere switch statement! You must use a loop. Rarely is correct error-handling so simple in POSIX-compliant software. He, perhaps conveniently, ignores this rather dark reality. Error-handling is only simple if your responses are simple.
There’s also a lot of errors in his observations. Most people catch IOException or similar because they don’t care about the actual underlying cause–their error-handling response is the same regardless. And if he doesn’t think the idiom:
if (-1 == some_syscall(…))
err(“I failed”)
isn’t common all over UNIX then he needs to spend considerable more time surveying existing source. This is, of course, the exact same idiom with error-checking. Most software simply doesn’t care about the cause of an error, merely that an error occurred.
Most (all) languages don’t include exceptions in their type-checking because it provides no benefit and requires very complicated type checking in order to achieve. Some languages (e.g., Python) don’t even fully specify what exceptions the runtime can throw, so type checking exception specifications would be pointless. This is OK, because the CPU can and will do the same thing at the end of the day, so highly-fault tolerant systems must be prepared for any error at any time.
He’s crazy if he thinks UNIX man pages, especially error codes, are well-documented. That is historically not the case in the least. UNIX is filled with fun history lessons like NFS servers that returned errors based on their internal errno mappings. If your own system used different mappings because it was a different OS, then tough shit, your software got the wrong error code!
Most languages allow exceptions to be thrown at any time because the underlying runtime can and will do so out of necessity. This is OK, because correctly written software (highly fault-tolerant or not) is prepared for any exception to be thrown at any time. Your code is not exception-safe if it cannot cope with this, and is why techniques like RAII (C++), try/finally (Java, Python, C#, et al.) and with/using blocks (Python, C#, Java7+) exist.
While I agree that the structure of try/catch can make it difficult to carefully isolate exceptions without creating additional functions, it’s not a problem for the reason he suggests. His example is wrong in the first place.
He also misses the most important thing about exceptions: they provide the correct behavior 99% of the time with no programmer effort.
You can’t have strong typing of error conditions and let them be integers.
“Although it’s not directly related, an obvious problem with error checking is the difficulty with extending an API; any errors codes not explicitly checked for will be silently swallowed.” This statement is hugely problematic, as it makes any sort of higher-order programming impossible. It makes any sort of modularity impossible as well, as it means I could never fully replace a file-based interface with a socket-based interface, for example.
“Because there are fault-tolerant systems such as network servers – where staying alive is the number one priority” Most network servers would probably be much better off if they died at the first sign of trouble and let something external (e.g., a process manager) restart them. I’ve wasted more time due to hung J2EE app servers that tried to handle errors and failed (leaving them hung or broken in subtle ways) than due to software that exited at the first sign of trouble.
Ultimately, his proposal boils down to, “I don’t like the exception hierarchies of most languages, so here’s a half-assed attempt to fix them”. And while I won’t disagree that the exception hierarchies of Java, C#, and Python have some structural issues, I do not believe that restructuring the language is necessary to fix them. In particular, doing what he wants in Python is already pretty trivial since you have access to errno. If you’re happy in writing platform-specific code, then you can do it with try/catch today. Sure, it’s more verbose, but oh well.
Thanks for the comments everyone!
Adam: Very good insight.
Brandon: Thanks! We already watched that video at work but I think I was out that day. I’ll give it a go.
Ronan: Great use! I’ve certainly used return codes of some sort in the same way, when batching stuff for systems that may not otherwise support it.
masklinn: Really good points about Erlang. Right now I’m doing some experiments with Go and Scala, I’ll check out Erlang next. I have even less practical use with it than the other two. And certainly pattern matching is a good way to handle this, as Tratt mentions.