Skip to content
Santekno.com | Level Up Your Engineering Skills
EN
📖 0%
23 Apr 2021 · 3 min read ·Article 96 / 119
Go

Concurrency Worker Pool Implementation on Golang

IH
Ihsan Arif
Writer at Santekno · Backend Engineer

Basic Definition

A Worker pool is a goroutine management technique in concurrent programming in Go-Lang with powerfull processes. A number of workers are executed and each has the same task of completing a number of jobs. With this worker pool method, the use of memory and program performance will be optimized.

Program Implementation

implement-worker-pool.go
 1package main
 2
 3import (
 4	"fmt"
 5	"math/rand"
 6	"time"
 7)
 8
 9func echoWorker(in, out chan int) {
10	for {
11		n := <-in
12		time.Sleep(time.Duration(rand.Intn(3000)) * time.Millisecond)
13		out <- n
14	}
15}
16
17func producer(ch chan<- int) {
18	i := 0
19	for {
20		fmt.Printf("-> Send Job: %d\n", i)
21		ch <- i
22		i++
23	}
24}
25
26func main() {
27	in := make(chan int)
28	out := make(chan int)
29
30	for i := 0; i < 4; i++ {
31		go echoWorker(in, out)
32	}
33
34	go producer(in)
35
36	for n := range out {
37		fmt.Printf("<- receive job : %d\n", n)
38	}
39}

output: `

Danger
Send Job: 0
-> Send Job: 1 -> Send Job: 2 -> Send Job: 3 -> Send Job: 4 -> Send Job: 5 <- receive job : 0 <- receive job : 4 -> Send Job: 6 -> Send Job: 7 <- receive job : 5 -> Send Job: 8 <- receive job : 2 <- receive job : 3 -> Send Job: 9 -> Send Job: 10 <- receive job : 1 <- receive job : 9 ^Csignal: interrupt

text
 1## Discussion
 2In the example program above we will print a value that is * * input *. Method `echoWorker` is a function that will take the value of *input* then we make *sleep* a few *miliseconds*, suppose in the *sleep* time we seem to do another process. While the method `producer` provides input values into `channel` which is sequential.
 3
 4When we run the program we must first initialize the `channel` *input* and *output*. why do we use 2 variables? So `in` is used to set the input that will be processed in the worker while `out` is used to receive the results of the *worker* process to be processed at the next stage.
 5
 6The first step is to prepare the * worker* using * goroutine*, the number of workers adjusts to the needs, in the program above we use 5 workers that will run. When we execute `go producer(in)` then the `in` channel is constantly filled with continuous iterations if we do not stop the program.
 7
 8We can see the output of the program, because we first set up 5 workers, then when `in` is set to data, the 5 workers run simultaneously with different processes (times). So why are the first ones received or printed are **job 0** and **job 4**? It is possible that the **Job 0** and **job 4** processes have a smaller waiting time than the others. Likewise, other workers, whoever finishes first will send the value and display the value directly.
 9
10## Conclusion
11**Worker pool** is a type of *concurrent* that is easy to use to process a lot of data with a fast process. For example, if we want to move data from CSV into a *database*, and the *records* can be up to millions, this **worker pool** can be the right choice to process such data. 
12
13With such a fast process, keep in mind that we also need to consider the number of **workers** that are running adjusted to the capacity of the computer/server where the **worker pool** will run. Do not let when we run the program on our computer/server there is a `hang` because the CPU reaches 100%.

Related Articles

💬 Comments