Pagination

PreviousPost Sync Scripts NextBatch Data Sync Example

Last updated 1 year ago

Was this helpful?

Pagination

1. Pagination

When syncing a Data Source, you may have the option to add in additional configuration sections, such as an Pagination, under the "Add a Section" drop down tab in the Connection Experience (Image 1).

There are two types of pagination available (Image 2):

Cursor: The cursor is considered to be a key parameter in this type of pagination. You receive a variable named Cursor along with the response. It is a pointer that points at a particular item that needs to be sent with a request. The server then uses the cursor to seek the other set of items. Cursor-based pagination is more complex and is preferred when dealing with a real-time data set.
Offset: Offset-based pagination is for parameters with a specific limit (the number of results) and offset (the number of records that need to be skipped). Offset-based pagination is easy to use and is preferred for static data.

Cursor Pagination

To set up cursor pagination, fill in the following parameters (Image 3):

Type: Select Cursor
Next Page URL JSON Path: This is the JSON Path within the response to the URL for the next page
Cursor Key: This is the key used in the query string to specify the cursor value. This is only required if the cursor returned is not a fully qualified URL.

Offset Pagination

To set up offset pagination, fill in the following parameters (Image 4):

Type: Select Offset
Limit Key: The key used in the query string to specify the limit
Limit: The desired page size
Offset By: The type of offset that the API does pagination by. This will be either Record Number or Page Number.
Offset Key: The key used in the query string to specify the offset
Initial Offset: The starting offset

Note that a pagination block is mandatory even if the API does not return results from multiple pages. You can use the following as the placeholder:

<Pagination type="OFFSET" limitField="" offsetField="" limit="0" initialOffset="0" />

<RestApiDataSource
    method="GET or POST"
    format="JSON"
    rootPath="$.result"
    endpoint="https://data.com/api/v1/records/@recordId/details?apiKey=@apiKey">
	<!-- optional -->
  <RequestHeaders>
		<Header name="" isEncrypted="true or false">value goes here</Header>
	</RequestHeaders>
    <!-- optional -->
	<Body>body goes here, only relevant for POST operations</Body>
	<!-- one of the following pagination blocks is required. If there's no pagination you still need a placeholder -->
	<Pagination type="OFFSET" 
	    limitField="name of the field in the querystring to use when passing the limit" 
	    offsetField="name of the field in the querystring to use when passing the offset"
	    limit="20"
	    initialOffset="0">
	<Pagination type="CURSOR"
	    pathInResponse="JSON path to the field in the response containing the next cursor value"
	    cursorPropertyName="name of the field in the querystring to use when passing the cursor">
  <Schema>
			<Column name="$.id" dataType="Text" />
			<Column name="$.variants.field1" dataType="Text" />
            <!-- you need to use the "item" placeholder if you're looking for fields in an array -->
			<Column name="$.colors.item[1]" dataType="Text" />
      <Column name="$.colors.item[2]" dataType="Text" />
	</Schema>
</RestApiDataSource>

PreviousPost Sync Scripts NextBatch Data Sync Example

Last updated 1 year ago

Was this helpful?

1. Pagination

When syncing a Data Source, you may have the option to add in additional configuration sections, such as an Pagination, under the "Add a Section" drop down tab in the Connection Experience (Image 1).

There are two types of pagination available (Image 2):

Cursor: The cursor is considered to be a key parameter in this type of pagination. You receive a variable named Cursor along with the response. It is a pointer that points at a particular item that needs to be sent with a request. The server then uses the cursor to seek the other set of items. Cursor-based pagination is more complex and is preferred when dealing with a real-time data set.
Offset: Offset-based pagination is for parameters with a specific limit (the number of results) and offset (the number of records that need to be skipped). Offset-based pagination is easy to use and is preferred for static data.

Cursor Pagination

To set up cursor pagination, fill in the following parameters (Image 3):

Type: Select Cursor
Next Page URL JSON Path: This is the JSON Path within the response to the URL for the next page
Cursor Key: This is the key used in the query string to specify the cursor value. This is only required if the cursor returned is not a fully qualified URL.

Offset Pagination

To set up offset pagination, fill in the following parameters (Image 4):

Type: Select Offset
Limit Key: The key used in the query string to specify the limit
Limit: The desired page size
Offset By: The type of offset that the API does pagination by. This will be either Record Number or Page Number.
Offset Key: The key used in the query string to specify the offset
Initial Offset: The starting offset

Note that a pagination block is mandatory even if the API does not return results from multiple pages. You can use the following as the placeholder:

<Pagination type="OFFSET" limitField="" offsetField="" limit="0" initialOffset="0" />

<RestApiDataSource
    method="GET or POST"
    format="JSON"
    rootPath="$.result"
    endpoint="https://data.com/api/v1/records/@recordId/details?apiKey=@apiKey">
	<!-- optional -->
  <RequestHeaders>
		<Header name="" isEncrypted="true or false">value goes here</Header>
	</RequestHeaders>
    <!-- optional -->
	<Body>body goes here, only relevant for POST operations</Body>
	<!-- one of the following pagination blocks is required. If there's no pagination you still need a placeholder -->
	<Pagination type="OFFSET" 
	    limitField="name of the field in the querystring to use when passing the limit" 
	    offsetField="name of the field in the querystring to use when passing the offset"
	    limit="20"
	    initialOffset="0">
	<Pagination type="CURSOR"
	    pathInResponse="JSON path to the field in the response containing the next cursor value"
	    cursorPropertyName="name of the field in the querystring to use when passing the cursor">
  <Schema>
			<Column name="$.id" dataType="Text" />
			<Column name="$.variants.field1" dataType="Text" />
            <!-- you need to use the "item" placeholder if you're looking for fields in an array -->
			<Column name="$.colors.item[1]" dataType="Text" />
      <Column name="$.colors.item[2]" dataType="Text" />
	</Schema>
</RestApiDataSource>