49 Manual Serialization and Deserialization

Serialization and deserialization are fundamental processes in modern software development. Both form the backbone of exchanging data across systems, persisting complex objects to files, and even communication between services using specific protocols. However, the automatic tools or libraries we use—such as JSON, Protocol Buffer, or YAML—often “hide” this complexity. But what happens when we have to perform serialization and deserialization manually?

This article explores manual serialization and deserialization—the explicit effort of converting a data structure into a specific format (and back again) without relying on a dedicated library. We will look at why this technique matters, the challenges that come with it, along with code examples in Python and Java together with a simulation of the process.

What Are Serialization and Deserialization?

Put simply:

• Serialization is the process of converting an object/data structure into a format that can be stored or transmitted (a string, a byte stream, and so on).
• Deserialization is the reverse process: turning the serialized data representation back into an object/data structure.

If you have ever used:

1import json
2json.dumps({'name': 'Budi', 'age': 30})

then you have already used automatic serialization.

Why Manual Serialization?

1. Full Control: Sometimes the output or input format must meet a very specific standard (for example, fixed-length ID formats or unconventional field naming).
2. Data Size: Standard serialization libraries can produce large output; a manual approach can be designed to be far more minimal.
3. Performance Needs: For high-volume data and time-critical (low-latency) communication, manual serialization allows for low-level optimization.
4. Security: Automatic serialization can sometimes open up injection or deserialization attack vectors.

Manual Serialization: A Case Study

Let’s use a simple example—serializing a user object with the attributes: id: int, username: string, active: bool.

1. Serialization Format Specification

We define a custom format:

• id is a fixed 4 digits with leading zeros (for example, 42 => “0042”)
• username is a maximum of 8 characters, padded with spaces if shorter (for example, “budi” => “budi “)
• active is 1 (True) or 0 (False)

An example serialized result, for a user with id=7, username=“lina”, active=True:

10007lina    1

2. Manual Serialization Code

a) Python Implementation

 1class User:
 2    def __init__(self, id, username, active):
 3        self.id = id
 4        self.username = username
 5        self.active = active
 6
 7    def serialize(self):
 8        id_str = str(self.id).zfill(4)
 9        username_str = self.username.ljust(8)
10        active_str = '1' if self.active else '0'
11        return f"{id_str}{username_str}{active_str}"
12
13    @staticmethod
14    def deserialize(serialized_str):
15        id = int(serialized_str[:4])
16        username = serialized_str[4:12].rstrip()
17        active = serialized_str[12] == '1'
18        return User(id, username, active)
19
20# Process simulation
21u = User(42, "neo", True)
22ser = u.serialize()
23print('Serialized:', ser)   # Output: "0042neo     1"
24
25u2 = User.deserialize(ser)
26print('Deserialized:', (u2.id, u2.username, u2.active))  # Output: (42, 'neo', True)

b) Java Implementation

 1class User {
 2    int id;
 3    String username;
 4    boolean active;
 5
 6    User(int id, String username, boolean active) {
 7        this.id = id;
 8        this.username = username;
 9        this.active = active;
10    }
11    String serialize() {
12        return String.format("%04d%-8s%d", id, username, active ? 1 : 0);
13    }
14    static User deserialize(String str) {
15        int id = Integer.parseInt(str.substring(0, 4));
16        String username = str.substring(4, 12).trim();
17        boolean active = str.charAt(12) == '1';
18        return new User(id, username, active);
19    }
20}

c) Go Implementation

 1package main
 2
 3import (
 4	"fmt"
 5	"strconv"
 6	"strings"
 7)
 8
 9type User struct {
10	ID       int
11	Username string
12	Active   bool
13}
14
15// Serialize converts a User into a string in the format: 4-digit ID, 8-character username (padded), 1-digit active (0/1)
16func (u User) Serialize() string {
17	activeInt := 0
18	if u.Active {
19		activeInt = 1
20	}
21	return fmt.Sprintf("%04d%-8s%d", u.ID, u.Username, activeInt)
22}
23
24// Deserialize converts a string back into a User struct
25func Deserialize(data string) (User, error) {
26	if len(data) < 13 {
27		return User{}, fmt.Errorf("data too short")
28	}
29	id, err := strconv.Atoi(data[0:4])
30	if err != nil {
31		return User{}, fmt.Errorf("invalid ID: %w", err)
32	}
33	username := strings.TrimSpace(data[4:12])
34	active := data[12] == '1'
35
36	return User{
37		ID:       id,
38		Username: username,
39		Active:   active,
40	}, nil
41}
42
43func main() {
44	u := User{ID: 42, Username: "ihsan", Active: true}
45	serialized := u.Serialize()
46	fmt.Println("Serialized:", serialized)
47
48	parsed, err := Deserialize(serialized)
49	if err != nil {
50		panic(err)
51	}
52	fmt.Printf("Deserialized: %+v\n", parsed)
53}

Output:

1Serialized: 0042ihsan   1
2Deserialized: {ID:42 Username:ihsan Active:true}

Notes:

• The username is guaranteed to always be 8 characters long, in line with the Java format.
• The Deserialize function handles input with a minimum-length validation.
• The struct mirrors the fixed-width serialization approach.

Manual Serialization and Deserialization Flow Diagram

flowchart LR
    A[Objek User] -- serialize() --> B[Serialized String]
    B -- deserialize() --> A

flowchart LR
    A[Objek User] -- serialize() --> B[Serialized String]
    B -- deserialize() --> A

The illustration above makes it clearer: serialization turns an object into a string that is stored, transmitted, or processed further; deserialization restores it to its original form.

Simulation: Manual vs Automatic Serialization

Potential Risks and Considerations

1. Human Error: An offset mistake (for example, incorrect index slicing) will corrupt the data or cause it to be misinterpreted.
2. Compatibility: If the format changes, the entire serialization/deserialization process must be updated consistently.
3. Scalability: For nested/complex objects, manual code quickly becomes hard to maintain.
4. Validation: There is no automatic validation unless we build it ourselves (for example, the username must be 8 characters or fewer).

Manual Serialization Best Practices

• Define the Format Specification clearly and document it.
• Include unit tests for all edge cases (numbers of different lengths, short and long names, boolean flags).
• Separate the serialization code from the core business logic to make refactoring easier.
• Use exception handling so that errors are easy to track down (for example, an incomplete string).
• If the structure evolves (for example, a new field is added), introduce versioning into the serialized string.

Conclusion

Manual serialization and deserialization remain an important weapon in an engineer’s toolkit, even in an era of fully automated frameworks. They guarantee maximum flexibility and control whenever business or system requirements push toward specific standards and optimal performance. That said, this should always be accompanied by solid documentation and testing discipline so it doesn’t become a source of latent problems down the road.

So, amid the freedom and power of automatic serialization, understand when to “roll up your sleeves” and serialize manually. It is often precisely there that the true art of programming shines through: turning structures into code, and code into structures, with the precision of a classic watchmaker.

*Note: From a security standpoint, both manual and automatic approaches still require protection against deserialization exploit attacks on untrusted data.

References:

• Python Documentation: struct module
• Java String Formatting
• OWASP Deserialization Cheat Sheet