Creating a Web Crawler using Golang

Web Crawlers are often used to retrieve something on a website so that we get the content we need. This is usually used for content needs. In this case we will try to use Golang to create a simple Web Crawler and will retrieve some content such as URLs on a website page.

Project Preparation

Now we will create a new project by creating the learn-golang-web-crawler folder. After that, initialize the project module with this command.

1go mod init github.com/santekno/learn-golang-web-crawler

Creating a Web Crawler using sequential

First we will try first using the sequential method where we just do a regular loop to do a crawler to the website.

Create a main.go file then fill the file with the code below.

 1package main
 2
 3import (
 4	"fmt"
 5	"net/http"
 6	"time"
 7
 8	"golang.org/x/net/html"
 9)
10
11var fetched map[string]bool
12
13func Crawl(url string, depth int) {
14	if depth < 0 {
15		return
16	}
17	urls, err := findLinks(url)
18	if err != nil {
19		fmt.Println(err)
20		return
21	}
22	fmt.Printf("found: %s\n", url)
23	fetched[url] = true
24	for _, u := range urls {
25		if !fetched[u] {
26			Crawl(u, depth-1)
27		}
28	}
29}
30
31func findLinks(url string) ([]string, error) {
32	resp, err := http.Get(url)
33	if err != nil {
34		return nil, err
35	}
36	if resp.StatusCode != http.StatusOK {
37		resp.Body.Close()
38		return nil, fmt.Errorf("getting %s: %s", url, resp.Status)
39	}
40	doc, err := html.Parse(resp.Body)
41	resp.Body.Close()
42	if err != nil {
43		return nil, fmt.Errorf("parsing %s as HTML: %v", url, err)
44	}
45	return visit(nil, doc), nil
46}
47
48func visit(links []string, n *html.Node) []string {
49	if n.Type == html.ElementNode && n.Data == "a" {
50		for _, a := range n.Attr {
51			if a.Key == "href" {
52				links = append(links, a.Val)
53			}
54		}
55	}
56	for c := n.FirstChild; c != nil; c = c.NextSibling {
57		links = visit(links, c)
58	}
59	return links
60}
61
62func main() {
63	fetched = make(map[string]bool)
64	now := time.Now()
65	Crawl("http://santekno.com", 2)
66	fmt.Println("time taken:", time.Since(now))
67}

Some explanations so that friends understand what each function created is used for the following explanation.

• The function visit(links []string, n *html.Node) []string is used to browse on one web page there are any URLs and if the URL has been accessed it will re-access to a different URL later all URLs on the first URL website will be returned as a result.
• The findLinks(url string)([]string, error) function is used to find the URL to be crawled by checking whether the website is available or not and retrieving all its HTML pages to be sent to the visit function.
• The last function func Crawl(url string, depth int) is used to detect the same URL found so that it does not need to be crawled repeatedly.
• The main function is used to define the URL to be crawled and is the main function of this program.

Do you understand the functions one by one? If so, we will try to run this program directly with the command below.

1go run main.go

The program will run and access the URL that we have defined in the main function. Don’t forget Make sure the internet on your computer or laptop is running smoothly so that the process won’t take too long.

If it has finished running, it will exit the terminal as below.

1found: https://www.santekno.com/jenis-jenis-name-server/
2found: https://www.santekno.com/tutorial/hardware/
3time taken: 3m7.149923291s

We can see that it means that it takes about 3 minutes 7 seconds to search or crawl this santekno.com website. Not too long and this is also conditioned by the internet on your laptop.

If it’s only 1 URL, maybe this is faster and what if for example we want to crawl 100 URLs / websites then if sequential we need to need at least 100 times from the first one, which is 300 minutes and this really takes a long time.

Then how can we make the process even faster to do a Web Crawler? In the next process we will try to change the Crawler process using Concurrent which we have learned before.

Then how can we make the process even faster to do a Web Crawler? In the next process we will try to change the Crawler process using Concurrent which we have learned before.

Changing the Web Crawler using Concurrent

We will modify the previous Web Crawler by adding some improvements, namely by using channels. Create a struct first like this.

1type result struct {
2	url string
3	urls []string
4	err error
5	depth int
6}

This struct is used to store the URL that we will crawl. Add a channel to the Crawler function at the beginning of the function.

1results := make(chan *result)

We will add a channel to this crawler so that it can use goroutines and modify the Crawler function to be as below.

 1func Crawl(url string, depth int) {
 2	results := make(chan *result)
 3
 4	fetch := func(url string, depth int) {
 5		urls, err := findLinks(url)
 6		results <- &result{url, urls, err, depth}
 7	}
 8
 9	go fetch(url, depth)
10	fetched[url] = true
11
12	for fetching := 1; fetching > 0; fetching-- {
13		res := <-results
14		if res.err != nil {
15			fmt.Println(res.err)
16			continue
17		}
18
19		fmt.Printf("found: %s\n", res.url)
20		if res.depth > 0 {
21			for _, u := range res.urls {
22				if !fetched[u] {
23					fetching++
24					go fetch(u, res.depth-1)
25					fetched[u] = true
26				}
27			}
28		}
29	}
30	close(results)
31}

We can see that we created a fetch function which will call the findLinks function and store the results into the results channel. Please note that after that we will run the fetch function using goroutine as mentioned earlier.

See the next code which is looping. In this code we will retrieve all the URL data in the results channel. When will the looping code finish? This loop will finish if the fetching value has become 0.

Alright, let’s run this last modification with the same command as above.

1go run main.go

After it is finished running, it will be seen how long the process execution takes to do this Crawler.

1found: https://www.santekno.com/tags/encoder
2found: https://www.santekno.com/categories/tutorial/page/2/
3time taken: 11.673643875s

It’s amazing how much faster the initial process takes about 3 minutes but after we modify it using concurrent we summarize the time and process only 11 seconds.

Conclusion

We often use this Web Crawler for certain needs, especially if we want to analyze data that already exists on a particular website. So if we want to make a Web Crawler using Golang, try to pay attention and use concurrent so that it can be more efficient in doing it so that the process is shorter and does not need to take longer especially if we do not only one Web Crawler.