Go makes it easy to run code in parallel. Just put go in front of a function call and the function runs concurrently.
go fetchProfile()
go fetchOrders()
go fetchInventory()
A lot of the time, you want to wait for all goroutines to finish before proceeding. This is the classic fan-out / fan-in pattern — you fan work out to multiple goroutines, then wait for them to finish and collect results (fan in).
Waiting with a channel ¶
A simple way to wait for goroutines is to signal completion over a channel:
done := make(chan struct{}, len(tasks))
for _, currentTask := range tasks {
go func() {
err := runTask(currentTask)
if err != nil {
log.Printf("task %s failed with error: %v", currentTask.name, err)
}
done <- struct{}{}
}()
}
for range tasks {
<-done
}
demo_channel_naive_no_result.go
This example creates a buffered channel with capacity equal to the number of tasks. Each goroutine sends a signal on the channel when done, and the main goroutine waits by receiving from the channel as many times as there are tasks. struct{} is used as a zero-size type since we only care about the signal, not any data.
Because this pattern is so common, Go provides the sync.WaitGroup type in the standard library to make it easier.
Waiting with sync.WaitGroup ¶
sync.WaitGroup provides a simple API to wait for a collection of goroutines to finish. All you need to do is call Go to launch the goroutines and call Wait to block the calling code until all goroutines have completed.
var wg sync.WaitGroup
for _, currentTask := range tasks {
wg.Go(func() {
err := runTask(currentTask)
if err != nil {
log.Printf("task %s failed with error: %v", currentTask.name, err)
}
})
}
wg.Wait()
demo_waitgroup_naive_no_result.go
This makes the code more concise and less error-prone than the channel-based approach. You don't have to worry about accidentally forgetting to send a signal.
But when moving away from simple examples, you'll find that sync.WaitGroup doesn't solve some common problems.
Errors are lost ¶
sync.WaitGroup doesn't know about errors. The goroutine signature is func(), not func() error. You can log errors inside, but the caller of Wait() has no idea whether anything failed.
wg.Wait()
// Did it succeed? Did 2 out of 3 tasks fail?
No limit on concurrent goroutines ¶
Every call to wg.Go launches a goroutine immediately. If you're calling an external service in a loop of 10,000 items, you'll fire 10,000 concurrent requests — and likely overwhelm the downstream.
for _, item := range tenThousandItems {
wg.Go(func() {
callExternalService(item) // 10,000 goroutines at once
})
}
No fail-fast cancellation ¶
If one goroutine hits a fatal error, the others keep running. There's no built-in way to say "something failed, cancel the rest."
wg.Go(func() {
if err := runCriticalTask(); err != nil {
log.Printf("critical task failed: %v", err)
// other goroutines keep running
}
})
All these issues can be solved manually with channels and cancellable contexts, but it requires some boilerplate code and careful wiring.
Fortunately, there is a better way; errgroup.Group helps you solve these problems with a minimal API.
How errgroup.Group solves these problems ¶
errgroup.Group is a higher-level alternative to sync.WaitGroup that adds error propagation, concurrency limits, and fail-fast cancellation.
errgroup.Group is not part of the standard library, you have to import it from golang.org/x/sync/errgroup.
The x packages are maintained by the Go team and are considered stable, but they are not part of the core library.
Return error ¶
errgroup.Group.Go takes a func() error and Wait() returns the first error encountered by any goroutine:
var g errgroup.Group
for _, currentTask := range tasks {
g.Go(func() error {
err := runTask(currentTask)
return err
})
}
if err := g.Wait(); err != nil {
log.Printf("failed with first error: %v", err)
}
Note that only the first error is returned. If multiple goroutines fail, you won't see all errors, just the first one that occurred.
Limit concurrent goroutines ¶
errgroup.Group has a built-in concurrency limit. Just call SetLimit(n) to specify how many goroutines can run at the same time:
ctx := context.Background()
var g errgroup.Group
g.SetLimit(2)
for _, currentTask := range tasks {
g.Go(func() error {
return runTaskWithCtx(ctx, currentTask)
})
}
if err := g.Wait(); err != nil {
log.Printf("failed with first error: %v", err)
}
SetLimit turns the Go method into a blocking call when the limit is reached.
If you don't call SetLimit, there is no limit and all goroutines will start immediately, just like with sync.WaitGroup.
Cancel siblings on first failure ¶
This feature does not come for free when using errgroup.Group, it needs a bit of setup, but the API makes it easy to wire up correctly.
This feature is not enabled when instantiating errgroup.Group with var g errgroup.Group as shown above. Instead, you need to create the
errgroup with errgroup.WithContext and pass it a context. This method returns the errgroup.Group and a derived context
that is canceled when any goroutine returns a non-nil error.
With this setup errgroup.Group will automatically cancel the context when the first goroutine returns a non-nil error.
baseCtx := context.Background()
g, ctx := errgroup.WithContext(baseCtx)
for _, currentTask := range tasks {
g.Go(func() error {
return runTaskWithCtx(ctx, currentTask)
})
}
if err := g.Wait(); err != nil {
log.Printf("failed with first error: %v", err)
}
It's important to note that errgroup.Group does not automatically stop all other running goroutines. It only cancels the context, and it's up to each
method that runs as a goroutine in the group to observe the context and exit early (cooperative cancellation).
Each method must check ctx.Done() (e.g., via select or by passing ctx to downstream calls that support cancellation) for fail-fast to actually work. If a method doesn't check ctx.Done() or pass ctx to downstream calls that support cancellation, it will keep running even after the first error is returned and the context is canceled.
In general if a method takes a context.Context parameter, you can assume that it supports cancellation. But it's important to check the documentation
of the specific method you're calling to confirm that it does support cancellation and that it will return early when the context is canceled.
Methods in the standard library that take a context.Context parameter will observe cancellation. For example with the HTTP client, you can use http.NewRequestWithContext to create a request that observes the cancellation of the context.
req, _ := http.NewRequestWithContext(ctx, "GET", url, nil)
Similarly, for database/sql, you can use db.QueryContext(ctx, query) to execute a query that observes the cancellation of the context.
rows, err := db.QueryContext(ctx, "SELECT * FROM users")
In your own code, if you have a select statement, you can add a case for <-ctx.Done() to check for cancellation and return early.
func runTaskWithCtx(ctx context.Context, t task) error {
fmt.Printf("running %s...\n", t.name)
select {
case <-time.After(t.d):
if t.fail {
fmt.Printf("%s failed\n", t.name)
return fmt.Errorf("%s failed", t.name)
}
fmt.Printf("%s succeeded\n", t.name)
return nil
case <-ctx.Done():
return ctx.Err()
}
}
If you have CPU-intensive code that runs for a long time without any blocking calls, you can check ctx.Err() periodically to ensure that it can exit early when cancellation is signaled. When a context is canceled, ctx.Err() will return a non-nil error, so you can use that as a signal to stop work and return early.
for i := range 1_000_000_000 {
if i%10_000 == 0 {
if ctx.Err() != nil {
return ctx.Err()
}
}
}
return nil
I would recommend not to check on every iteration, but rather at some reasonable interval (e.g., every 10,000 iterations) to avoid the overhead of checking the context too frequently. The optimal interval depends on the expected runtime of the task and how quickly you want it to respond to cancellation.
Limitation: no return values ¶
Both sync.WaitGroup and errgroup.Group share one limitation — goroutines can't return results through the API. sync.WaitGroup takes func(), errgroup.Group.Go takes func() error. Neither has a way to say func() (T, error).
Additionally errgroup.Group only returns the first non-nil error. This is fine if you use the fail-fast cancellation feature, but if you don't use that feature and you want to run all tasks to either succeed or fail, you won't be able to see all errors, just the first one that occurred.
If you need to collect results or see all errors, you need to wire that up yourself. An idiomatic way to do this is to create a channel and send the result of each task back to the main goroutine via this channel. The main goroutine can then collect all results and errors, and process them after all goroutines have completed.
The following example uses errgroup.Group only for the concurrency limit and to wait for all goroutines to finish, but it doesn't use the error propagation or fail-fast cancellation features.
type fetchResult struct {
taskName string
data string
err error
}
baseCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
var g errgroup.Group
g.SetLimit(2)
resultCh := make(chan fetchResult, len(tasks))
for _, currentTask := range tasks {
g.Go(func() error {
data, err := runTaskWithResultCtx(baseCtx, currentTask)
resultCh <- fetchResult{
taskName: currentTask.name,
data: data,
err: err,
}
return nil
})
}
_ = g.Wait()
close(resultCh)
for r := range resultCh {
if r.err != nil {
log.Printf("task %s failed with error: %v", r.taskName, r.err)
} else {
log.Printf("task %s succeeded with data: %s", r.taskName, r.data)
}
}
Wrapping up ¶
Go makes it easy to run concurrent code. sync.WaitGroup, a primitive in the standard library, gives you a way to run multiple goroutines and wait for them to finish. errgroup.Group from the golang.org/x/sync/errgroup package is a higher-level alternative that adds error handling, concurrency limits, and fail-fast cancellation.
Choose based on your requirements. For simple cases, WaitGroup may be sufficient. For scenarios that need error propagation and cancellation, errgroup.Group can remove boilerplate and make orchestration easier to reason about.
It's important that when using errgroup.Group with fail-fast cancellation, that you create the group with errgroup.WithContext and ensure that all goroutines in the group are designed to observe the cancellation of the context that WithContext returns.