98. Case Study: Distributed Point of Sale (POS) System

Point of Sale (POS) systems have become the backbone of modern retail operations. But as a business grows and expands into many branches, the demands on the POS architecture level up too: the POS system needs to be distributed, available across many locations yet still integrated. What are the challenges? What are the design choices and solutions? In this case study, I’ll break down the process of building a distributed POS system based on real-world project experience — from architecture design and synchronization scenarios to sample code.

The Business Problem

A national retail chain wants to build a new POS system that is reliable and scalable. They have 50+ branches spread across multiple cities. Their main requirements:

• Transactions must still be able to run even when a branch temporarily loses its internet connection.
• Data from each branch must be centralized at the central server, in real time or as quickly as possible.
• Integration with the ERP and reporting must be accurate.

The challenges: offline/online synchronization, data integrity, conflict resolution, and operational ease.

Architecture Overview

Let’s start with the high-level architecture of a distributed POS system:

flowchart LR
    A[Cashier/Local Branch POS] -->|Synchronization| B(Branch POS Server)
    B --> |Periodic Sync| C[Central Server]
    B <-->|API Access| D[Backoffice Operator]
    C <-->|Integration| E[ERP/BI]

flowchart LR
    A[Cashier/Local Branch POS] -->|Synchronization| B(Branch POS Server)
    B --> |Periodic Sync| C[Central Server]
    B <-->|API Access| D[Backoffice Operator]
    C <-->|Integration| E[ERP/BI]

Explanation:

• Local POS Cashier: A desktop/mobile application running on the store’s cashier machine.
• Branch POS Server: A local database & service in the store, which could take the form of a Raspberry Pi/PC/small server.
• Central Server: The main data warehouse, receiving data from all branches and providing reporting/ERP APIs.
• Backoffice: User management, inventory, and so on.
• ERP: Consumes central data.

Distributed POS Requirements

• Availability: The cashier can keep operating locally (without internet).
• Reliability: Data is safe — nothing is lost or missed.
• Consistency: No double transactions or messy stock.
• Scalability: Easy to add branches/cashiers.

The Synchronization and Conflict Process

The key to a successful distributed POS system is synchronization between branches and the center. Don’t imagine database replication across locations, because of bandwidth and the intermittent nature of internet connections. The common approaches are:

1. Transactional Queues
2. Delta Sync: Only changed data is synchronized.
3. Conflict Resolution: Timestamps and unique transaction IDs.
4. Async Queue: Sends data asynchronously, retrying on failure.

Synchronization Flow:

sequenceDiagram
    participant POS-Kasir as Local POS
    participant POS-Server as Branch Server
    participant Core as Central Server

    POS-Kasir->>POS-Server: Create transaction (offline / online)
    loop Periodic Sync
        POS-Server->>Core: Push batch of new data
        Core->>POS-Server: Ack success/failure status
        opt Failure
          POS-Server-->>POS-Server: Retry/Delay
        end
    end

sequenceDiagram
    participant POS-Kasir as Local POS
    participant POS-Server as Branch Server
    participant Core as Central Server

    POS-Kasir->>POS-Server: Create transaction (offline / online)
    loop Periodic Sync
        POS-Server->>Core: Push batch of new data
        Core->>POS-Server: Ack success/failure status
        opt Failure
          POS-Server-->>POS-Server: Retry/Delay
        end
    end

Database Schema: Accommodating Distribution

The key to the distributed POS database design: Every transaction has a Global Unique ID (UUID), a timestamp, and a marker of which branch it originated from.

Transaction Table (Simplified):

• sync_status indicates whether the data has been sent to the center yet or not.

Example Transaction Entity Model (Python SQLAlchemy)

 1import uuid
 2from sqlalchemy import Column, String, DateTime, Numeric, Boolean
 3from sqlalchemy.ext.declarative import declarative_base
 4
 5Base = declarative_base()
 6
 7class Transaksi(Base):
 8    __tablename__ = 'transaksi'
 9    id_transaksi = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
10    id_cabang = Column(String, nullable=False)
11    waktu_transaksi = Column(DateTime, nullable=False)
12    total = Column(Numeric, nullable=False)
13    status_sync = Column(Boolean, default=False)

Batch Synchronization Code Example

Each branch server has a scheduler to push new transactions to the center.

 1import requests
 2from models import session, Transaksi
 3
 4def sync_to_central():
 5    unsynced = session.query(Transaksi).filter_by(status_sync=False).all()
 6    data = [trans.__dict__ for trans in unsynced]
 7    res = requests.post("https://central-pos.company.com/sync", json=data)
 8    if res.ok:
 9        for trans in unsynced:
10            trans.status_sync = True
11        session.commit()
12    else:
13        print("Sync failed, will retry later.")
14
15# scheduler every 5 minutes

Simulation: From Offline to Online Transactions

Suppose a cashier makes 2 sales while the internet is down.

1. The transactions are still saved to the local database (sync_status=False).
2. As soon as the branch server regains connectivity, the batch sync runs automatically.
3. If a transaction ID has already been sent to the center (due to a retry), the data is ignored (idempotent).

Flow Simulation Table:

Conflict Resolution

What happens if two cashiers at two different branches create transactions that happen to look similar (e.g., the same time, the same amount)?
Because each one generates a unique UUID and the branch ID is always recorded, there is no data conflict.

If an out-of-order situation occurs (a transaction syncs late), the central system only accepts it if the transaction_id does not already exist.

1# Pseudocode endpoint on the central server
2@app.route('/sync', methods=['POST'])
3def receive_transactions():
4    for trx in request.json:
5        exists = db.session.query(Transaksi).get(trx['id_transaksi'])
6        if not exists:
7            db.session.add(Transaksi(**trx))
8    db.session.commit()
9    return "OK"

Failure & Recovery Scenarios

• Prolonged network outage: Data remains stored locally until the connection is restored.
• Branch server failure: Automatic backup or manual restore from the center (two-way synchronization/fallback database).
• Cashier mistypes price/quantity: Safe, because cash drawer consistency is handled on the local side.

Monitoring & Logging

Monitoring matters: the number of unsynced transactions, sync delay times, and error rates. You can use Prometheus + Grafana.

graph LR
    A[Local POS] -->|log sync_status| S[Monitoring Stack]
    S --> G[Grafana Alerts]

graph LR
    A[Local POS] -->|log sync_status| S[Monitoring Stack]
    S --> G[Grafana Alerts]

Lessons Learned & Best Practices

1. Design for Disconnection: Don’t rely on 100% connectivity.
2. Eventual Consistency: It’s important to understand the trade-off; real time doesn’t mean always instantly consistent.
3. Idempotency: The sync API must be resilient to double submits.
4. Monitoring is mandatory, not optional.
5. Test your failure scenarios more often!

Conclusion

Building a distributed POS system isn’t merely a technology problem — it’s about maturity in designing systems that are resilient to reality: the internet isn’t always available, data can arrive late, and recovery matters. With a batch synchronization scheme, a UUID per transaction, a sync_status table, and adequate monitoring, a modern POS system can be reliable — even across hundreds of branches. Hopefully this experience can serve as a reference for engineers stepping into the realm of mid-cost distributed systems.

Have experiences, questions, or other solutions? Drop them in the comments!

Happy coding and building resilient systems! 🚀