39 Filter & Pagination Schemes: Best Practices

Filtering and pagination are two crucial topics in modern web application development. Lately, the need to process large amounts of data keeps growing—whether it’s product data in e-commerce, articles, users, activity logs, or other API resources. Efficient API connections, an optimal user experience, and even your back-end resources all depend on the filter and pagination scheme you put in place. That’s why it’s so important to understand the best practices and the right implementation patterns.

In this article, I’ll walk through the 39 most common filter & pagination schemes, review the best practices you need to understand, and show you code examples, simulated API responses, and even flow diagrams (using mermaid). While there are literally dozens of variations, we’ll focus on the most relevant ones you’ll often run into in the real world.

Why Are Filtering & Pagination Important?

In both REST and GraphQL applications, without pagination or filtering, an endpoint will return the entire dataset—which is clearly inefficient (and can even take down your system!). Users only want a subset of results, with specific criteria—this is exactly the role of filtering and pagination.

Benefits:

• Faster Responses: The query only fetches a subset of the data.
• Bandwidth Savings: You transfer only the data that’s actually needed.
• Better UX: Navigating the data becomes more comfortable.
• Avoiding Database Overload: Queries become lighter.

39 Filter & Pagination Schemes

The schemes below are categorized by filtering, pagination, and combinations of the two.

I. Filtering Schemes (14)

1. Basic Query String
   Example: GET /products?category=shoes&color=black
   As simple as adding parameters to the URL.

2. Advanced Filter Operators
   For example: GET /users?age[gt]=18&age[lt]=45

3. Multiple Values (Array Query)
   GET /posts?tags=tech,devops,cloud

4. Exclude Filter
   GET /books?exclude_status=archived

5. Range (Between)
   GET /sales?date_from=2024-01-01&date_to=2024-01-31

6. Partial Match
   GET /contacts?name_like=adam

7. IN/NOT IN
   GET /id_list?user_id[in]=1,2,3

8. Nested Object Filter
   GET /orders?customer.address.city=Jakarta

9. Boolean Filter
   GET /items?active=true

10. Case Sensitive/Insensitive
    GET /customers?email__iexact=gmail.com

11. Regex Filtering
    GET /files?name_regex=^foo

12. Null/Not Null
    GET /payment?receipt_number[null]=true

13. Sorting
    GET /products?sort=price,-name

14. Aggregation Filtering
    GET /teams?member_count[gt]=5

II. Pagination Schemes (13)

15. Offset-Limit

    • GET /posts?offset=0&limit=10
      Classic and easy, but problematic with large or changing datasets.

16. Page-Size

    • GET /items?page=1&size=25

17. Cursor-based Pagination

    • GET /products?after=YXNkMTIz&limit=10
      More reliable when data changes frequently.

18. Seek Method

    • GET /comments?last_id=2024&limit=10

19. Keyset Based

    • Similar to seek, using a unique field (createdAt, id, etc.).

20. Time-based Keyset

    • Pagination based on a timestamp.

21. Backward Pagination (previous)

    • Supports navigating backward.
    • GET /orders?before=...

22. Range Pagination

    • Use a range: GET /logs?from=567&to=577

23. Bookmarked/Custom Cursor

    • Supports bookmarks to jump to a specific position.

24. Numbered Link Response

    • The API provides numbered links: prev, next, first, last.

25. Infinite Scroll Support

    • For mobile/SPA apps with endless scrolling.

26. Page Token

    • Google APIs: GET /videos?pageToken=abc123

27. Relational Pagination

    • Pagination within a subresource.

III. Combinations & Extended Schemes (12)

28. Filter + Pagination

    • Common: GET /products?category=shoes&limit=10&offset=0

29. Filter + Sort + Paginate

    • Everything combined.

30. Aggregated Pagination

    • While computing summary data at the same time.

31. Stateless Cursor

    • The cursor doesn’t store any user state.

32. Stateful Cursor

    • State specific to each user/role.

33. Caching-aware Pagination

    • Pagination stays consistent even when cached data changes.

34. Pre-Fetch Pagination

    • Processes 2 pages at once for a better user experience.

35. Chunked Pagination

    • Includes a “hint” with the total number of chunks/pages.

36. Meta-based Pagination

    • The response contains a meta object:
      json

      Copy

      1{
      2  "data": [...],
      3  "meta": { "pagination": { "page": 1, "total": 99 } }
      4}

37. GraphQL Connection Pattern

    • edges, nodes, pageInfo.

38. Custom Encrypted Tokens

    • Cursors in the form of an encrypted blob.

39. Slicing, Windowing, Partitioning

    • Data is fetched per slice (window), typically in OLAP/BI APIs.

Filter & Pagination Best Practices

Let’s go over the best practices that, in my opinion, are a must-adopt:

1. Combine Filtering, Sorting, and Pagination — Almost every real-world requirement needs all three at once.
2. Reasonable Default Limit — Always set a default limit and a maximum limit per page (for example, max 100).
3. Use Cursors for Fast-Changing Data — Cursor-based pagination is far more reliable on rapidly changing tables.
4. Document Your Parameters — List the valid filter/sort fields in your API documentation, including operators/fields.
5. Meta Pagination Info — Include total records/pages if performance allows, to make life easier for the frontend.
6. Consistent Responses — The response schema (fields, meta, structure) must be consistent.
7. Don’t Support Hidden Filters — Only filter on fields that are already queryable/indexed in the database.
8. Handle Input Errors — Validate input (for example, limit must not exceed the max, filter fields must be valid).
9. Allow Multi-value Filter Combinations — Array filters and OR/AND operators are important for advanced search.
10. Paginate the API Response, Not the DB Query — Get into the habit of paginating the query result, not the entire table.

Pagination Flow Diagram (Offset vs Cursor)

Let’s compare offset vs cursor:

flowchart TD
    Start([Start Request]) --> CheckType{Pagination Type?}
    CheckType -- Offset-Based --> OffsetDB[Query DB dengan OFFSET dan LIMIT]
    CheckType -- Cursor-Based --> WhereCursor[WHERE id > last_id ORDER BY id LIMIT N]
    OffsetDB --> RespOffset[Build & Send Response]
    WhereCursor --> RespCursor[Build & Send Response (with next cursor)]
    RespOffset --> End
    RespCursor --> End

flowchart TD
    Start([Start Request]) --> CheckType{Pagination Type?}
    CheckType -- Offset-Based --> OffsetDB[Query DB dengan OFFSET dan LIMIT]
    CheckType -- Cursor-Based --> WhereCursor[WHERE id > last_id ORDER BY id LIMIT N]
    OffsetDB --> RespOffset[Build & Send Response]
    WhereCursor --> RespCursor[Build & Send Response (with next cursor)]
    RespOffset --> End
    RespCursor --> End

Quick Notes:

• Offset:
  Simple, but slow for very large datasets (because the DB has to skip N rows).
• Cursor:
  More efficient when data changes (insert/delete), and a better fit for infinite scroll.

Simulated Pagination Responses

Offset-based

1{
2  "data": [{ "id": 101, "title": "Item 1" }, ...],
3  "meta": {
4    "page": 1,
5    "limit": 10,
6    "total": 99
7  }
8}

Cursor-based

1{
2  "data": [{ "id": 121, "title": "Item 21" }, ...],
3  "next_cursor": "e2lkOjEyMSxjcmVhdGVkQXQ6dGltZXN0YW1w",
4  "has_next_page": true
5}

Code Example: Implementing Pagination & Multi-Filter (Node.js + Express + Prisma)

 1app.get("/api/users", async (req, res) => {
 2  const { page = 1, size = 20, sort = "id", order = "asc", name, age_gt, age_lt } = req.query;
 3  const where = {};
 4  if (name) where.name = { contains: name, mode: 'insensitive' };
 5  if (age_gt) where.age = { ...where.age, gt: Number(age_gt) };
 6  if (age_lt) where.age = { ...where.age, lt: Number(age_lt) };
 7  
 8  const [data, total] = await Promise.all([
 9    prisma.user.findMany({
10      where,
11      skip: (page - 1) * size,
12      take: Number(size),
13      orderBy: { [sort]: order }
14    }),
15    prisma.user.count({ where })
16  ]);
17
18  return res.json({
19    data,
20    meta: {
21      page: Number(page),
22      size: Number(size),
23      total
24    }
25  });
26});

Table: 39 Schemes (Summary)

Tips & Antipatterns

• Antipattern: limit=1000 without validation → it will hammer your DB resources.
• Don’t leak internal data: Don’t allow filtering on sensitive fields, such as hashed_password.
• Use an Index on the fields most frequently used for filtering & sorting.
• Check your filter results: Empty data should still be a success (return [], not an error).

Closing

Choosing and implementing the right filter & pagination scheme can be a major factor in your application’s scalability, performance, and ease of development. Master the 39 scheme variations above, pick the one that best fits your business context, and make sure user experience stays a priority.

Share the filter & pagination scheme you find most effective in the comments! 🚀