Creating a Simple Distibuted Search Engine Using Worker Pool on Golang
Introduction
You know Google is a place where we search for several keywords that can produce various millions of information that we can get just by typing a few keywords and in less than 1 second we can already get various information related to the keywords we are looking for. This is an example of a miniature or simple search engine version of how we can implement a worker pool into an engine that is useful for searching.
Do you understand how Worker Pool works? If you don’t know or already know but want to deepen it again, you can read first in the previous article that Santekno discussed in how to implement concurrency worker pool golang .
Basic Data Preparation
Before we create a search program, we must first prepare the data to be searched. Santekno here will provide the simplest data sample so that it is easy for everyone to digest.
We will create data in the form of User information which contains name and email attributes. This name and email will be the data that we will search.
1type User struct {
2 Name string
3 Email string
4}
5
6var Database = []User{
7 { Name: "ihsan arif", Email: "ihsan.arif@example.com"},
8 {Name: "ihsan nugraha", Email: "ihsa@example.com"},
9 {Name: "arif susanto", Email: "arif.susanto@example.com"},
10 {Name: "ines nur", Email: "ines.nur@example.com"},
11 {Name: "inez hendriani", Email: "inez.hendriani@example.com"},
12 {Name: "deni siahaan", Email: "deni.siahaan@example.com"},
13 {Name: "sumarno treggono", Email: "sumarno.trenggono@example.com"},
14 {Name: "nida fauziah", Email: "nida.fauziah@example.com"},
15 {Name: "nurul kamilah", Email: "nurul.kamilah@example.com"},
16 {Name: "muhammad rafi", Email: "muhammad.rafi@example.com"},
17 {Name: "irfan maulana", Email: "irfan.maulana@example.com"},
18 {Name: "irsyad aprilianto", Email: "irsyad.aprilianto@example.com"},
19 {Name: "iqbal fauzi", Email: "iqbal.fauzi@example.com"},
20 {Name: "rani zahroh", Email: "rani.zahroh@example.com"},
21 {Name: "ajeng syifa", Email: "ajeng.syifa@example.com"},
22 {Name: "aldi arikandi", Email: "aldi.arikandi@example.com"},
23 {Name: "firman shah", Email: "firman.syah@example.com"},
24 {Name: "irwan setiawan", Email: "irwan.setiawan@example.com"},
25}Worker Pool Initialization
Why do we use worker pool? The simple answer is so that the search process based on keywords into the database (data source) is faster and shared. We can see below the initialization of Worker struct into an object that we will use for searching in the main function. If you are still confused, just look at the program snippet for the Woerker initialization.
1type Worker struct {
2 Users []User
3 ch chan *User
4 Name string
5}
6
7func NewWorker(users []User, ch chan *User, name string) *Worker {
8 return &Worker{Users: users, ch: ch, Name: name}
9}Search Function
We will create the main function of the worker under the worker object to make searching easier.
1func (w *Worker) Find(email string) *User {
2 for i := range w.Users {
3 user := &w.Users[i]
4 if strings.Contains(user.Email, email) {
5 w.ch <- user
6 }
7 }
8 return nil
9}Main Program Function
This function runs the process of initializing the worker, retrieving data and reading the input to be the keyword to be searched in the database.
1func main() {
2 // to retrieve input arguments
3 email := os.Args[1]
4 // initialize the user object channel
5 ch := make(chan *User)
6 // initialize worker pool with 3 pools
7 for i := 0; i < 3; i++ {
8 go NewWorker(Database[(len(Database)/3*i):len(Database)/3*(i+1)], ch, fmt.Sprintf("#%s", i)).Find(email)
9 }
10
11 // print the result of the search
12 select {
13 case user := <-ch:
14 log.Printf("the email %s is owned by %s", email, user.Email)
15 case < time.After(100 * time.Millisecond):
16 log.Printf("the email %s was not found", email)
17 }
18}Program Output
1-> go run content/post/main.go ihsan.arif
22021/05/10 19:38:00 the email ihsan.arif is owned by ihsan.arif@example.com
3-> go run content/post/main.go ines
42021/05/10 19:38:04 the email ines is owned by ines.nur@example.com
5-> go run content/post/main.go unyil
62021/05/10 19:40:49 the email unyil was not foundOverall program
1package main
2
3import (
4 "fmt"
5 "log"
6 "os"
7 "strings"
8 "time"
9)
10
11type User struct {
12 Name string
13 Email string
14}
15
16var Database = []User{
17 {Name: "ihsan arif", Email: "ihsan.arif@example.com"},
18 {Name: "ihsan nugraha", Email: "ihsa@example.com"},
19 {Name: "arif susanto", Email: "arif.susanto@example.com"},
20 {Name: "ines nur", Email: "ines.nur@example.com"},
21 {Name: "inez hendriani", Email: "inez.hendriani@example.com"},
22 {Name: "deni siahaan", Email: "deni.siahaan@example.com"},
23 {Name: "sumarno treggono", Email: "sumarno.trenggono@example.com"},
24 {Name: "nida fauziah", Email: "nida.fauziah@example.com"},
25 {Name: "nurul kamilah", Email: "nurul.kamilah@example.com"},
26 {Name: "muhammad rafi", Email: "muhammad.rafi@example.com"},
27 {Name: "irfan maulana", Email: "irfan.maulana@example.com"},
28 {Name: "irsyad aprilianto", Email: "irsyad.aprilianto@example.com"},
29 {Name: "iqbal fauzi", Email: "iqbal.fauzi@example.com"},
30 {Name: "rani zahroh", Email: "rani.zahroh@example.com"},
31 {Name: "ajeng syifa", Email: "ajeng.syifa@example.com"},
32 {Name: "aldi arikandi", Email: "aldi.arikandi@example.com"},
33 {Name: "firman syah", Email: "firman.syah@example.com"},
34 {Name: "irwan setiawan", Email: "irwan.setiawan@example.com"},
35}
36
37type Worker struct {
38 Users []User
39 ch chan *User
40 Name string
41}
42
43func NewWorker(users []User, ch chan *User, name string) *Worker {
44 return &Worker{Users: users, ch: ch, Name: name}
45}
46
47func (w *Worker) Find(email string) *User {
48 for i := range w.Users {
49 user := &w.Users[i]
50 if strings.Contains(user.Email, email) {
51 w.ch <- user
52 }
53 }
54 return nil
55}
56
57func main() {
58 email := os.Args[1]
59
60 ch := make(chan *User)
61
62 for i := 0; i < 2; i++ {
63 go NewWorker(Database[(len(Database)/3*i):len(Database)/3*(i+1)], ch, fmt.Sprintf("#%s", i)).Find(email)
64 }
65
66 select {
67 case user := <-ch:
68 log.Printf("the email %s is owned by %s", email, user.Email)
69 case <-time.After(100 * time.Millisecond):
70 log.Printf("the email %s was not found", email)
71 }
72}