What's the idiomatic way to exit a program with some error code?
The documentation for Exit says "The program terminates immediately; deferred functions are not run.", and log.Fatal just calls Exit. For things that aren't heinous errors, terminating the program without running deferred functions seems extreme.
Am I supposed to pass around some state that indicate that there's been an error, and then call Exit(1) at some point where I know that I can exit safely, with all deferred functions having been run?
I do something along these lines in most of my real main packages, so that the return err convention is adopted as soon as possible, and has a proper termination:
func main() {
if err := run(); err != nil {
fmt.Fprintf(os.Stderr, "error: %v\n", err)
os.Exit(1)
}
}
func run() error {
err := something()
if err != nil {
return err
}
// etc
}
In Python I commonly use a pattern, which being converted to Go looks like this:
func run() int {
// here goes
// the code
return 1
}
func main() {
os.Exit(run())
}
I think the most clear way to do it is to set the exitCode at the top of main, then defer closing as the next step. That lets you change exitCode anywhere in main, and it's last value will be exited with:
package main
import (
"fmt"
"os"
)
func main() {
exitCode := 0
defer func() { os.Exit(exitCode) }()
// Do whatever, including deferring more functions
defer func() {
fmt.Printf("Do some cleanup\n")
}()
func() {
fmt.Printf("Do some work\n")
}()
// But let's say something went wrong
exitCode = 1
// Do even more work/cleanup if you want
// At the end, os.Exit will be called with the last value of exitCode
}
Output:
Do some work
Do some cleanup
Program exited: status 1.
Go Playgroundhttps://play.golang.org/p/AMUR4m_A9Dw
Note that an important disadvantage of this is that you don't exit the process as soon as you set the error code.
As mentioned by fas, you have func Exit(exitcode int) from the os package.
However, if you need the defered function to be applied, you always can use the defer keyword like this:
http://play.golang.org/p/U-hAS88Ug4
You perform all your operation, affect a error variable and at the very end, when everything is cleaned up, you can exit safely.
Otherwise, you could also use panic/recover:
http://play.golang.org/p/903e76GnQ-
When you have an error, you panic, end you cleanup where you catch (recover) it.
Yes, actually. The os package provides this.
package main
import "os"
func main() {
os.Exit(1)
}
http://golang.org/pkg/os/#Exit
Edit: so it looks like you know of Exit. This article gives an overview of Panic which will let deferred functions run before returning. Using this in conjunction with an exit may be what you're looking for. http://blog.golang.org/defer-panic-and-recover
Another good way I follow is:
if err != nil {
// log.Fatal will print the error message and will internally call System.exit(1) so the program will terminate
log.Fatal("fatal error message")
}
Related
I am trying to upload bunch of files using the company's api to the storage service they provide. (basically to my account). I have got lots of files like 40-50 or something.
I got the full path of the files and utilize the os.Open, so that, I can pass the io.Reader. I did try to use client.Files.Upload() without goroutines but it took so much time to upload them and decided to use goroutines. Here the implementation that I tried. When I run the program it just uploads one file which is the one that has the lowest size or something that it waits for a long time. What is wrong with it? Is it not like every time for loops run it creates a goroutine continue its cycle and creates for every file? How to make it as fast as possible with goroutines?
var filePaths []string
var wg sync.WaitGroup
// fills the string of slice with fullpath of files.
func fill() {
filepath.Walk(rootpath, func(path string, info os.FileInfo, err error) error {
if !info.IsDir() {
filePaths = append(filePaths, path)
}
if err != nil {
fmt.Println("ERROR:", err)
}
return nil
})
}
func main() {
fill()
tokenSource := oauth2.StaticTokenSource(&oauth2.Token{AccessToken: token})
oauthClient := oauth2.NewClient(context.TODO(), tokenSource)
client := putio.NewClient(oauthClient)
for _, path := range filePaths {
wg.Add(1)
go func() {
defer wg.Done()
f, err := os.Open(path)
if err != nil {
log.Println("err:OPEN", err)
}
upload, err := client.Files.Upload(context.TODO(), f, path, 0)
if err != nil {
log.Println("error uploading file:", err)
}
fmt.Println(upload)
}()
}
wg.Wait()
}
Consider a worker pool pattern like this: https://go.dev/play/p/p6SErj3L6Yc
In this example application, I've taken out the API call and just list the file names. That makes it work on the playground.
A fixed number of worker goroutines are started. We'll use a channel to distribute their work and we'll close the channel to communicate the end of the work. This number could be 1 or 1000 routines, or more. The number should be chosen based on how many concurrent API operations your putio API can reasonably be expected to support.
paths is a chan string we'll use for this purpose.
workers range over paths channel to receive new file paths to upload
package main
import (
"fmt"
"os"
"path/filepath"
"sync"
)
func main() {
paths := make(chan string)
var wg = new(sync.WaitGroup)
for i := 0; i < 10; i++ {
wg.Add(1)
go worker(paths, wg)
}
if err := filepath.Walk("/usr", func(path string, info os.FileInfo, err error) error {
if err != nil {
return fmt.Errorf("Failed to walk directory: %T %w", err, err)
}
if info.IsDir() {
return nil
}
paths <- path
return nil
}); err != nil {
panic(fmt.Errorf("failed Walk: %w", err))
}
close(paths)
wg.Wait()
}
func worker(paths <-chan string, wg *sync.WaitGroup) {
defer wg.Done()
for path := range paths {
// do upload.
fmt.Println(path)
}
}
This pattern can handle an indefinitely large amount of files without having to load the entire list in memory before processing it. As you can see, this doesn't make the code more complicated - actually, it's simpler.
When I run the program it just uploads one file which is the one
Function literals inherit the scope in which they were defined. This is why our code only listed one path - the path variable scope in the for loop was shared to each go routine, so when that variable changed, all routines picked up the change.
Avoid function literals unless you actually want to inherit scope. Functions defined at the global scope don't inherit any scope, and you must pass all relevant variables to those functions instead. This is a good thing - it makes the functions more straightforward to understand and makes variable "ownership" transitions more explicit.
An appropriate case to use a function literal could be for the os.Walk parameter; its arguments are defined by os.Walk so definition scope is one way to access other values - such as paths channel, in our case.
Speaking of scope, global variables should be avoided unless their scope of usage is truly global. Prefer passing variables between functions to sharing global variables. Again, this makes variable ownership explicit and makes it easy to understand which functions do and don't access which variables. Neither your wait group nor your filePaths have any cause to be global.
f, err := os.Open(path)
Don't forget to close any files you open. When you're dealing with 40 or 50 files, letting all those open file handles pile up until the program ends isn't so bad, but it's a time bomb in your program that will go off when the number of files exceeds the ulimit of allowed open files. Because the function execution greatly exceeds the part where the file needs to be open, defer doesn't make sense in this case. I would use an explicit f.Close() after uploading the file.
I'd like to achieve 100% test coverage in go code. I am not able to cover the following example - can anyone help me with that?
package example
import (
"io/ioutil"
"log"
)
func checkIfReadable(filename string) (string, error) {
_, err := ioutil.ReadFile(filename)
if err != nil {
log.Fatalf("Cannot read the file... how to add coverage test for this line ?!?")
}
return "", nil
}
func main() {
checkIfReadable("dummy.txt")
}
Some dumy test for that:
package example
import (
"fmt"
"testing"
)
func TestCheckIfReadable(t *testing.T) {
someResult, err := checkIfReadable("dummy.txt")
if len(someResult) > 0 {
fmt.Println("this will not print")
t.Fail()
}
if err != nil {
fmt.Println("this will not print")
t.Fail()
}
}
func TestMain(t *testing.T) {
...
}
The issue is that log.Fatalf calls os.Exit and go engine dies.
I could modify the code and replace built-in library with my own - what makes the tests less reliable.
I could modify the code and create a proxy and a wrapper and a .... in other words very complex mechanism to change all calls to log.Fatalf
I could stop using built-in log package... what is equal to asking "how much is go built-in worth?"
I could live with not having 100% coverage
I could replace log.Fataf with something else - but then what is the point for built-in log.Fatalf?
I can try to mangle with system memory and depending on my OS replace memory address for the function (...) so do something obscure and dirty
Any other ideas?
Use log.Print instead of log.Fatal and return the error value that you declared in signature of function checkIfReadable. Or don't the error it and return it to some place that knows better how to handle it.
The function log.Fatal is strictly for reporting your program's final breath.
Calling log.Fatal is a bit worse than calling panic (there is also log.panic), because it does not execute deferred calls. Remember, that overusing panic in Go is considered a bad style.
A good way to get 100% test coverage and not fail at the same time is to use recover() to catch the panic that is thrown by log.Fatalf().
Here are the docs for recover. I think it fits your use case nicely.
I am developing some tests for my code (using the testing package), and I am wondering what's the best way to mock functions inside the tested function:
Should I pass the function as parameter?
In that case, what if that function calls another function? Should I pass both the first and second function as parameters in the tested one?
Note: some of the functions are called on objects (i.e. someObj.Create()) and use HTTP API calls.
UPDATE for clarification:
Example: functions
func f1() error {
... //some API call
}
func (s *SomeStruct) f2() error {
return f1
}
func f3() error {
return nil
}
func f4() error {
...
err = obj.f2()
...
err = f3()
...
}
For the above: if I want to test f4, what's the best way to mock f2 and f3?
If I pass f2 and f3 to f4 as parameters it would work, but then what for the f2 test? Should I pass f1 to f2 as parameter?
And if that's it, should then f4 have f1 as well in the parameters?
As a general guideline, functions aren't very mockable so its in our best interests to mock structs that implement a certain interface that may be passed into functions to test the different branches of code. See below for a basic example.
package a
type DoSomethingInterface interface {
DoSomething() error
}
func DoSomething(a DoSomethingInterface) {
if err := a.DoSomething(); err != nil {
fmt.Println("error occurred")
return
}
fmt.Println("no error occurred")
return
}
package a_test
import (
"testing"
"<path to a>/a"
)
type simpleMock struct {
err error
}
func (m *simpleMock) DoSomething() error {
return m.err
}
func TestDoSomething(t *testing.T) {
errorMock := &simpleMock{errors.New("some error")}
a.DoSomething(errorMock)
// test that "an error occurred" is logged
regularMock := &simpleMock{}
a.DoSomething(regularMock)
// test "no error occurred" is logged
}
In the above example, you would test the DoSomething function and the branches that happens eg. you would create an instance of the mock with an error for one test case and create another instance of the mock without the error to test the other case. The respective cases are to test a certain string has been logged to standard out; in this case it would be "error occurred" when simpleMock is instantiated with an error and "no error occurred" when there simpleMock is not instantiated with an error.
This can of course be expanded to other cases eg. the DoSomething function actually returns some kind of value and you want to make an assertion on the value.
Edit:
I updated the code with the concern that the interface lives in another package. Note that the new updated code has a package a that contains the interface and the function under test and a package a_test that is merely a template of how to approach testing a.DoSomething.
I'm not sure what you're trying to do here but I'll explain how testing should be done in Go.
Lets say we have an application with the following directory hierarchy:
root/
pack1/
pack1.go
pack1_test.go
pack2/
pack2.go
pack2_test.go
main.go
main_test.go
We'll assume that pack2.go has the functions you want to test:
package pack2
func f1() error {
... //some API call
}
func (s *SomeStruct) f2() error {
return f1
}
func f3() error {
return nil
}
func f4() error {
...
err = obj.f2()
...
err = f3()
...
}
Looks good so far. Now if you want to test the functions in pack2, you would create a file called pack2_test.go. All test files in go are named similarly (packagename_test.go). Now lets see the inside of a typical test for a package (pack2_test.go in this example):
package pack2
import (
"testing"
"fmt"
)
TestF1(*testing.T) {
x := "something for testing"
f1() // This tests f1 from the package "pact2.go"
}
TestF2(*testing.T) {
y := new(somestruct)
y.f2() // tests f2 from package "pact2.go"
}
TestF3(*testing.T) {
/// some code
f3() // tests f3
}
TestF4(*testing.T) {
/// code
f3() // you get the gist
}
Let me explain. Notice how in pack2_test.go, the first line says that the package is pack2. In a nutshell, this means that we're in the "scope" of the package pack2 and thus all the functions found in pack2 can be called as if you're within pack2. Thats why, within the Testf* functions, we could've called the functions from pack2. Another thing to note is the imported package "testing". This helps with two things:
First, it provides some functionality for running tests. I won't go into that.
Second, it helps identify the functions that go test should run.
Now to the functions. Any function within a test package that has the prefix "Test" and the parameters "t *testing.T" (you can use "*testing.T" when you don't need to use the testing functionality) will be executed when you run go test. You use the variable t to reference the testing functionality I mentioned. You can also declare functions without the prefix and call them within the prefixed functions.
So, if I go to my terminal and run go test, it will execute the functions you want to test, specified in pack2_test.go
You can learn more about testing here and here
I was trying to throw errors in my Golang program with log.Fatal but, log.Fatal does not also print the line where the log.Fatal was ran. Is there no way of getting access to the line number that called log.Fatal? i.e. is there a way to get the line number when throwing an error?
I was trying to google this but was unsure how. The best thing I could get was printing the stack trace, which I guess is good but might be a little too much. I also don't want to write debug.PrintStack() every time I need the line number, I am just surprised there isn't any built in function for this like log.FatalStackTrace() or something that isn't costume.
Also, the reason I do not want to make my own debugging/error handling stuff is because I don't want people to have to learn how to use my special costume handling code. I just want something standard where people can read my code later and be like
"ah ok, so its throwing an error and doing X..."
The less people have to learn about my code the better :)
You can set the Flags on either a custom Logger, or the default to include Llongfile or Lshortfile
// to change the flags on the default logger
log.SetFlags(log.LstdFlags | log.Lshortfile)
Short version, there's nothing directly built in, however you can implement it with a minimal learning curve using runtime.Caller
func HandleError(err error) (b bool) {
if err != nil {
// notice that we're using 1, so it will actually log where
// the error happened, 0 = this function, we don't want that.
_, filename, line, _ := runtime.Caller(1)
log.Printf("[error] %s:%d %v", filename, line, err)
b = true
}
return
}
//this logs the function name as well.
func FancyHandleError(err error) (b bool) {
if err != nil {
// notice that we're using 1, so it will actually log the where
// the error happened, 0 = this function, we don't want that.
pc, filename, line, _ := runtime.Caller(1)
log.Printf("[error] in %s[%s:%d] %v", runtime.FuncForPC(pc).Name(), filename, line, err)
b = true
}
return
}
func main() {
if FancyHandleError(fmt.Errorf("it's the end of the world")) {
log.Print("stuff")
}
}
playground
If you need exactly a stack trace, take a look at https://github.com/ztrue/tracerr
I created this package in order to have both stack trace and source fragments to be able to debug faster and log errors with much more details.
Here is a code example:
package main
import (
"io/ioutil"
"github.com/ztrue/tracerr"
)
func main() {
if err := read(); err != nil {
tracerr.PrintSourceColor(err)
}
}
func read() error {
return readNonExistent()
}
func readNonExistent() error {
_, err := ioutil.ReadFile("/tmp/non_existent_file")
// Add stack trace to existing error, no matter if it's nil.
return tracerr.Wrap(err)
}
And here is the output:
On a plain Go HTTP handler, if I disconnect a client while still writting to the response, http.ResponseWritter.Write will return an error with a message like write tcp 127.0.0.1:60702: connection reset by peer.
Now from the syscall package, I have sysca.ECONNRESET, which has the message connection reset by peer, so they're not exactly the same.
How can I match them, so I know not to panic if it occurs ? On other ocasions I have been doing
if err == syscall.EAGAIN {
/* handle error differently */
}
for instance, and that worked fine, but I can't do it with syscall.ECONNRESET.
Update:
Because I'm desperate for a solution, for the time being I'll be doing this very dirty hack:
if strings.Contains(err.Error(), syscall.ECONNRESET.Error()) {
println("it's a connection reset by peer!")
return
}
The error you get has the underlying type *net.OpError, built here, for example.
You should be able to type-assert the error to its concrete type like this:
operr, ok := err.(*net.OpError)
And then access its Err field, which should correspond to the syscall error you need:
operr.Err.Error() == syscall.ECONNRESET.Error()
The answer by #zian is more useful than the accepted answer, but now on Go 1.13+ it is preferable to avoid manually unwrapping the errors:
if errors.Is(opErr,syscall.ECONNRESET) {
fmt.Println("Found a ECONNRESET")
}
This has the benefit that you can also use it more generally, such as after:
resp, err := http.Get("http://127.0.0.1:4444")
Here this err would otherwise have an extra layer of wrapping (*url.Error) and would be missed by the condition #zian used without explicitly unwrapping it a third time.
I came across this issue and the accepted answer was sufficient to point me in the right direction. However, the code it provides to check if the Error embedded inside *net.OpError is ECONNRESET is not complete, at least not for Golang 1.9.
The error embedded at OpError.Err is actually of type *os.SyscallError (https://golang.org/pkg/os/#SyscallError). The Write() function implemented by struct *net.netFD (which is what's being written to when sending a response over the network) looks like this:
func (fd *netFD) Write(p []byte) (nn int, err error) {
nn, err = fd.pfd.Write(p)
runtime.KeepAlive(fd)
return nn, wrapSyscallError("write", err)
}
And wrapSyscallError:
func wrapSyscallError(name string, err error) error {
if _, ok := err.(syscall.Errno); ok {
err = os.NewSyscallError(name, err)
}
return err
}
The error inside the *os.SyscallError struct can be directly compared against syscall.ECONNRESET.
So, given an error returned from a network write (e.g. a call to http.ResponseWritter.Write), the full code block to determine if that error is ECONNRESET is:
if opErr, ok := err.(*net.OpError); ok {
if syscallErr, ok := opErr.Err.(*os.SyscallError); ok {
if syscallErr.Err == syscall.ECONNRESET {
fmt.Println("Found a ECONNRESET")
}
}
}
#zian - thanks for your good solution to João Pinto's (and my) question : How can I match them, so I know not to panic if it occurs ?
As at go version 1.13, an improvement is to use the errors.Is function which does error unwrapping and testing sequentially 'under the hood'. For example :
if errors.Is(opErr,syscall.ECONNRESET) {
fmt.Println("Found a ECONNRESET")
}
#SteveCoffman - adding to your good answer, cheers!
Working with Errors in Go 1.13 - The Go Blog - Golang