Creating a Thumbnail Image Generator Using Pipeline Pattern
We also use Generate Image using Golang to make it easier for editors so they don’t need to edit using other applications so that we can easily put the desired image. Now santekno will try to create a Thumbnail Image generator that already exists in this tutorial. Suppose we want to make this thumbnail image more concise than the original image. Then we need to convert it into a lighter file with a small size. What if there are many images, then if we use the usual Golang sequencial, it will be long when we execute it. So, we will try to compare how the process of generating this Thumbnail image with sequential golang using concurrent Pipeline Patter.
If you haven’t learned what a Pipeline Pattern is, you can check out this tutorial first.
Project Preparation
Now we will create a new project by creating the learn-golang-generator-image-thumbnail folder. After that, initialize the project module with this command.
1go mod init github.com/santekno/learn-golang-generator-image-thumbnailPrepare the required image or photo or can take a photo in the santekno repository here
1https://github.com/santekno/learn-golang-generator-image-thumbnail/tree/main/imagesGenerate Image Thumbnail Using Sequential
Before going into the code we need to understand the big point process that will be processed in this Thumbnail Image generator. Here are the stages of the process that we must understand and later we will divide it into several functions as follows.
- The function reads the image file from the
images/folder by validating the file must have an image extension. - The function manipulates the image file by using the library package
github.com/disintegration/imagingwith a size of 100 x 100 pixels. - The function saves the resulting thumbnail image into the
thumbnail/folder.
Have you imagined what the process will be like? Hopefully friends can understand the process that we will make in this Golang.
More clearly we describe the process illustration below.
LR flowchart
subgraph subGraph1 ["func walkFiles()"]
C("func\n getFileContentType()") --> D("func\n processImage()")
D --> E("func\n saveThumbnail()")
end
id1((start)) --> d("main func") --> subGraph1 --> e("print\nprocess time") --> id2((finish))
Create a main.go file where we will create all the functions in this file. First we create this generator process with a normal sequential process.
Function Retrieve Image from Folder
The function that we will create will read a folder that contains several image files while checking whether this extension is an image or not. We see below the function.
1func walkFiles(root string) error {
2 err := filepath.Walk(root, func(path string, info os.FileInfo, err error) error {
3
4 // filter out errors
5 if err != nil {
6 return err
7 }
8
9 // check if it is a file
10 if !info.Mode().IsRegular() {
11 return nil
12 }
13
14 // check if it is image/jpeg
15 contentType, _ := getFileContentType(path)
16 if contentType != "image/jpeg" {
17 return nil
18 }
19
20 return nil
21 })
22
23 if err != nil {
24 return err
25 }
26 return nil
27}
28
29// getFileContentType - return content type and error status
30func getFileContentType(file string) (string, error) {
31
32 out, err := os.Open(file)
33 if err != nil {
34 return "", err
35 }
36 defer out.Close()
37
38 // Only the first 512 bytes are used to sniff the content type.
39 buffer := make([]bytes, 512)
40
41 _, err = out.Read(buffer)
42 if err != nil {
43 return "", err
44 }
45
46 // Use the net/http package's handy DectectContentType function. Always returns a valid
47 // content-type by returning "application/octet-stream" if no others seem to match.
48 contentType := http.DetectContentType(buffer)
49
50 return contentType, nil
51}walkFiles function which is useful for reading files in one folder sent from the parameter, then the second function, namely getFileContentType, is useful for checking whether the file has a content type in the sense that its type is image or not so that when we want to make a thumbnail later when generating not all files that support only images so that it has been filtered from the beginning only images can be generated by our program.Image File Manipulation Function
This function is a process to change the image that will be compressed into a thumbnail image type where the size will be 100x100 pixels. In this function we have help using an additional library, namely the library
github.com/disintegration/imaging library. Then we need to add the library first with this execution
1go get -u github.com/disintegration/imagingNext add the main.go file with the function below it like this.
1// processImage - takes image file as input
2// return pointer to thumbnail image in memory.
3func processImage(path string) (*image.NRGBA, error) {
4
5 // load the image from file
6 srcImage, err := imaging.Open(path)
7 if err != nil {
8 return nil, err
9 }
10
11 // scale the image to 100px * 100px
12 thumbnailImage := imaging.Thumbnail(srcImage, 100, 100, imaging.Lanczos)
13
14 return thumbnailImage, nil
15}And don’t forget to update and add to the walkFiles() function to access this processImage function after checking the image.
1func walkFiles(root string) error {
2 err := filepath.Walk(root, func(path string, info os.FileInfo, err error) error {
3
4 ...
5
6 // process the image
7 thumbnailImage, err := processImage(path)
8 if err != nil {
9 return err
10 }
11
12 ...
13
14 return nil
15 })
16
17 if err != nil {
18 return err
19 }
20 return nil
21}Function Save Thumbnail Image Result
The process when we will save the results of this thumbnail image into a folder with the folder name thumbnail/. Later the result of the generate image function processImage in the form of a thumbnailImage file, so we will save the result file from the generator image function into one folder. The following is more complete as below.
1// saveThumbnail - save the thumnail image to folder
2func saveThumbnail(srcImagePath string, thumbnailImage *image.NRGBA) error {
3 filename := filepath.Base(srcImagePath)
4 dstImagePath := "thumbnails/" + filename
5
6 // save the image in the thumbnails folder.
7 err := imaging.Save(thumbnailImage, dstImagePath)
8 if err != nil {
9 return err
10 }
11 fmt.Printf("%s -> %s\n", srcImagePath, dstImagePath)
12 return nil
13}That means also prepare the folder of the saved thumbnail image in this folder thumbnails/. and we will also access the function walFiles() after calling the function processImage().
1func walkFiles(root string) error {
2 err := filepath.Walk(root, func(path string, info os.FileInfo, err error) error {
3
4 ..
5
6 // process the image
7 thumbnailImage, err := processImage(path)
8 if err != nil {
9 return err
10 }
11
12 // save the thumbnail image to disk
13 err = saveThumbnail(path, thumbnailImage)
14 if err != nil {
15 return err
16 }
17 return nil
18 })
19
20 ...
21
22 return nil
23} 1➜ learn-golang-generator-image-thumbnail git:(main) ✗ ./learn-golang-generator-image-thumbnail images
2images/sample-1.jpg -> thumbnails/sample-1.jpg
3images/sample-10.jpg -> thumbnails/sample-10.jpg
4images/sample-11.jpg -> thumbnails/sample-11.jpg
5images/sample-12.jpg -> thumbnails/sample-12.jpg
6images/sample-13.jpg -> thumbnails/sample-13.jpg
7images/sample-14.jpg -> thumbnails/sample-14.jpg
8images/sample-2.jpg -> thumbnails/sample-2.jpg
9images/sample-3.jpg -> thumbnails/sample-3.jpg
10images/sample-4.jpg -> thumbnails/sample-4.jpg
11images/sample-5.jpg -> thumbnails/sample-5.jpg
12images/sample-6.jpg -> thumbnails/sample-6.jpg
13images/sample-7.jpg -> thumbnails/sample-7.jpg
14images/sample-8.jpg -> thumbnails/sample-8.jpg
15images/sample-9.jpg -> thumbnails/sample-9.jpg
16Time taken: 145.78275msChanging the Process Mechanism using Pipeline Pattern Concurrent Golang
We have seen above when using a sequential process to generate a thumbnail image of 14 images takes about 145ms. If we calculate one, divided by 14, it becomes 14ms for every one image processed. So if we have 1 million images the time required is about 14ms x 1 million = 14,000,000ms or 3.89 hours. This is quite long if you want to process that much data. So we will try to implement this Pipeline Pattern whether it can reduce the process to be faster or not.
First, we need to make some code changes. So that our previous program code is not deleted, we create a sequential folder to move the code we previously created into the folder. Then we create another new folder called pipeline-pattern so that the folder structure in the project will be like this.
1.
2├── README.md
3├── learn-golang-generator-image-thumbnail
4├── go.mod
5├── go.sum
6├── images
7│ ├── sample-1.jpg
8│ ├── sample-10.jpg
9│ ├── sample-11.jpg
10│ ├── sample-12.jpg
11│ ├── sample-13.jpg
12│ ├── sample-14.jpg
13│ ├── sample-2.jpg
14│ ├── sample-3.jpg
15│ ├── sample-4.jpg
16│ ├── sample-5.jpg
17│ ├── sample-6.jpg
18│ ├── sample-7.jpg
19│ ├── sample-8.jpg
20│ └── sample-9.jpg
21├── main.go
22├── pipeline-pattern
23└── pipeline.go
24├── sequential
25│ └── sequential.go
26└── thumbnailsIn accordance with the folder structure that we have created, the functions related to sequential are in the sequential folder while for what we will create now is the pipeline pattern in the pipeline-pattern folder. Let’s try to create it directly in the pipeline.go file.
First we need struct to help deliver standardized pipeline data so that each process will receive the same struct data like this.
1type result struct {
2 srcImagePath string
3 thumbnailImage *image.NRGBA
4 err error
5}Changing the Function to Retrieve Image from Folder
In the pipeline.go file we create the same function, walkFiles() but there are some things that we have to change including the parameters changed to channel type which can be asynchronous when the program is run.
1func walkFiles(done <-chan struct{}, root string) (<-chan string, <-chan error) {
2 // create output channels
3 paths := make(chan string)
4 errc := make(chan error, 1)
5
6 go func() {
7 defer close(paths)
8 errc <- filepath.Walk(root, func(path string, info os.FileInfo, err error) error {
9 // filter out errors
10 if err != nil {
11 return err
12 }
13
14 // check if it is a file
15 if !info.Mode().IsRegular() {
16 return nil
17 }
18
19 // check if it is image/jpeg
20 contentType,_ := sequential.GetFileContentType(path)
21 if contentType != "image/jpeg" {
22 return nil
23 }
24
25 // send file path to next stage
26 select {
27 case paths <- path:
28 case <-done:
29 return fmt.Errorf("walk canceled")
30 }
31 return nil
32 })
33 }()
34 return paths, errc
35}The above process will run using a goroutine that will send the files read and sent the file path so that it can be processed to the next function. Then the function call sequential.GetFileContentType as validation we take from the previous package in the sequential folder. Then there is a need to update the function to a global function by changing it so that it can be accessed in various packages from
1func getFileContentType(file string) (string, error)1func GetFileContentType(file string) (string, error)Changing the Image File Manipulation Function
In the manipulation function the process is the same but we will apply channeling where the function can be processed in parallel. Here are more details below.
1func processImage(done <-chan struct{}, paths <-chan string) <-chan result {
2 results := make(chan result)
3 var wg sync.WaitGroup
4
5 thumbnailer := func() {
6 for srcImagePath := range paths {
7 srcImage, err := imaging.Open(srcImagePath)
8 if err != nil {
9 select {
10 case results <- result{srcImagePath, nil, err}:
11 case <-done:
12 return
13 }
14 }
15 thumbnailImage := imaging.Thumbnail(srcImage, 100, 100, imaging.Lanczos)
16
17 select {
18 case results <- result{srcImagePath, thumbnailImage, err}:
19 case <-done:
20 return
21 }
22 }
23 }
24
25 const numThumbnailer = 5
26 for i := 0; i < numThumbnailer; i++ {
27 wg.Add(1)
28 go func() {
29 thumbnailer()
30 wg.Done()
31 }()
32 }
33
34 go func() {
35 wg.Wait()
36 close(results)
37 }()
38
39 return results
40}processImage() process is more complicated because we are implementing channels and goroutines so that processes do not need to wait for each other because the process we do is based on the process sent by the paths channel. As long as the paths channel still has data being sent, this function will continue to work.Changing to Global in the Save Thumbnail Image Result Function
In the save thumbnail function, we will also change the parameter to channel as below.
1func saveThumbnail(done < chan struct{}, thumbs < chan result) < chan result {
2 results := make(chan result)
3 var wg sync.WaitGroup
4
5 saveThumbnailer := func() {
6 for img := range thumbs {
7 filename := filepath.Base(img.srcImagePath)
8 dstImagePath := "thumbnails/" + filename
9
10 // save the image in the thumbnails folder.
11 err := imaging.Save(img.thumbnailImage, dstImagePath)
12 if err != nil {
13 select {
14 case results <- result{img.srcImagePath, dstImagePath, img.thumbnailImage, err}:
15 case <-done:
16 return
17 }
18 }
19 select {
20 case results <- result{img.srcImagePath, dstImagePath, img.thumbnailImage, err}:
21 case <-done:
22 return
23 }
24 }
25 }
26
27 const numGoroutine = 5
28 for i := 0; i < numGoroutine; i++ {
29 wg.Add(1)
30 go func() {
31 saveThumbnailer()
32 wg.Done()
33 }()
34 }
35
36 go func() {
37 wg.Wait()
38 close(results)
39 }()
40
41 return results
42}Create SetupPipeline Function
This SetupPipeline function is used to collect all running goroutine processes into one function that can later be accessed by the main function more easily.
1func SetupPipeLine(root string) error {
2 done := make(chan struct{})
3 defer close(done)
4
5 // do the file walk
6 paths, errc := walkFiles(done, root)
7
8 // process the images
9 resultImages := processImage(done, paths)
10
11 // save thumbnail images
12 results := saveThumbnail(done, resultImages)
13
14 // save thumbnail images
15 for r := range results {
16 if r.err != nil {
17 return r.err
18 }
19 fmt.Printf("%s -> %s\n", r.srcImagePath, r.destImagePath)
20 }
21
22 // check for errors on the channel, from walkfiles stage.
23 if err := <-errc; err != nil {
24 return err
25 }
26
27 return nil
28}We have created all the functions for the needs of this pipeline pattern generate image, then we just have to try to run the program by first changing the main.go file because previously we used sequential functions now we use the pipeline pattern that we have created.
1// Image processing - sequential
2// Input - directory with images.
3// output - thumbnail images
4func main() {
5 if len(os.Args) < 2 {
6 log.Fatal("need to send directory path of images")
7 }
8 start := time.Now()
9
10 // using sequential
11 // err := sequential.WalkFiles(os.Args[1])
12
13 // using pipeline pattern
14 err := pipelinepattern.SetupPipeLine(os.Args[1])
15
16 if err != nil {
17 log.Fatal(err)
18 }
19 fmt.Printf("Time taken: %s\n", time.Since(start))
20}Seen above the use of sequential functions we comment first so that it is not executed when the program runs. Run the program with the same command as above, namely
1go run main.go imagesThe results of the process will be seen 2 times faster, which is approximately 64ms
1➜ learn-golang-generator-image-thumbnail git:(main) ✗ go run main.go images
2images/sample-11.jpg -> thumbnails/sample-11.jpg
3images/sample-10.jpg -> thumbnails/sample-10.jpg
4images/sample-1.jpg -> thumbnails/sample-1.jpg
5images/sample-12.jpg -> thumbnails/sample-12.jpg
6images/sample-13.jpg -> thumbnails/sample-13.jpg
7images/sample-14.jpg -> thumbnails/sample-14.jpg
8images/sample-4.jpg -> thumbnails/sample-4.jpg
9images/sample-2.jpg -> thumbnails/sample-2.jpg
10images/sample-3.jpg -> thumbnails/sample-3.jpg
11images/sample-5.jpg -> thumbnails/sample-5.jpg
12images/sample-6.jpg -> thumbnails/sample-6.jpg
13images/sample-7.jpg -> thumbnails/sample-7.jpg
14images/sample-8.jpg -> thumbnails/sample-8.jpg
15images/sample-9.jpg -> thumbnails/sample-9.jpg
16Time taken: 64.981125msExperiment Results
Here is a table of experimental results with a larger amount of image data so that we can see the difference in processing time between the two flows that we have used.
| Amount of Data | Sequential | Pipeline Pattern |
|---|---|---|
| 14 | 145.78ms | 64.98ms |
| 1792 | 17.38s | 6.46s |
| 3584 | 33.51s | 12.09s |
| 14.336 | 2m18.07s | 50.83s |
Conclusion
Pipeline Pattern is very useful when we have processes that are interrelated but the data is a lot and each data to the other data does not need to wait so that we can parallelize the process. It is very useful when we implement a process like this to make the process more efficient so that each data that will be processed sequentially does not need to wait for the previous data process to finish.
This can be seen from the experiments that we do by comparing the first two processes, namely using a sequential process where each data waits for the process to finish, while the second process uses a pipeline pattern where the first data, second data and so on do not need to wait for the previous process to finish, as long as each data has the same process sequence so that it provides faster data processing that can be up to 2 times faster than the process using ordinary sequential.
This experiment does not have large data, only 14 images, but if you want to try further exploration, you can add more images so that you can see whether the process is more efficient or not.