Page the results

The Output Module also supports paging the results for the root node of the query. You can specify an offset and a length parameter to limit the results to a subset of the complete list.

JSON key Default value Description
displayStart 0 The offset in the list of results. This offset is a zero-based index value.
displayLength -1 The maximum total number of results to return. A negative value means unlimited.
maxCountLimit -1

The maximum count value.  A count of all records can lead to performance problems. When paging, you can limit the max count to this value. Passing 0 means no count is done.
  • JSON
  • YAML
{
  "ViewConfig": {
    "displayStart": 10,
    "displayLength": 5,
    "maxCountLimit": 10000,
    "Resources": {
      "Community": {
        "Id": { "name": "communityId" },
        "Name": { "name": "communityName" },
        "Description": { "name": "communityDescription" },
        "Order": [ { "Field": { "name": "communityName", "order": "ASC" } } ]
      }
    }
  }
}
---
ViewConfig:
  displayStart: 10
  displayLength: 5
  maxCountLimit: 10000
  Resources:
    Community:
      Id:
        name: "communityId"
      Name:
        name: "communityName"
      Description:
        name: "communityDescription"
      Order:
      -
        Field:
          name: "communityName"
          order: "ASC"

The example query above selects page 3 of all communities, with five results per page.

  • Paged results should always be sorted, otherwise the results might seem inconsistent from page to page.
  • The paged results list is recalculated upon each request.
  • All entities that have been added or removed will appear/disappear from the list, modifying the indexes of the elements in the results list.
  • The Collibra Console allows limiting the number of results returned by queries. The values range from 10,000 to 100,000. If enabled, and the limit is set, then:
    • The default displayLength value (-1) is overwritten by the limit set through the console.
    • If the displayLength set  in the ViewConfig/TableViewConfig is larger than the limit value set in the Collibra Console, an exception is thrown.

Cursor-based pagination

Offset-based pagination does not scale for a big data set. The bigger the offset, the slower the response time. For a bulk export of many assets, it's better to use cursor-based pagination:

  • Run the initial query sorting by ID. This is the recommended option to achieve maximum performance. Use a medium page size, such as 10,000.
  • Read the results and keep the last ID value.
  • Pass this ID as a filter to the next query.
  • Repeat the process until the number of returned results is less than the page size. This means you retrieved all data.
{
  "ViewConfig": {
    "displayLength": 10,
    "Resources": {
      "Asset": {
        "Id": { "name": "assetId" },
        "Signifier": { "name": "assetName" },
        "Order": [
          {
            "Field": {
              "name": "assetId",
              "order": "ASC"
            }
          }
        ],
        "Filter": {
          "Field": {
            "name": "assetId",
            "operator": "GREATER",
            "value": "00000000-0000-0000-0000-000000005032"
          }
        }
      }
    }
  }
}