How to get next page from Okta api call with CURL - api

Question

Im currently querying Okta for a list of events. The result set is tens of thousands. Currently the requests limits the return to 1000 results. Per the Okta API. http://developer.okta.com/docs/getting_started/design_principles.html#pagination I can use the Link header with the value "next".
How do I use CURL to capture this value and issue that CURL command on that url to get the rest of the values and loop till the end?

Answer 1

In cURL, if you include the include '-i' option, the headers are returned to the console. Within these headers, you'll find a link called rel="next". If you replace the resource in the first GET call with this filter, you'll get the next set of results.

Answer 2

The pagination code is based on https://michaelheap.com/follow-github-link-header-bash
#!/usr/bin/env bash
# Set these:
url="https://COMPANY.okta.com/api/v1/users"
token="..."
# Pagination code based on https://michaelheap.com/follow-github-link-header-bash
while [ "$url" ]; do
r=$(curl --compressed -Ss -i -H "authorization: SSWS $token" "$url" | tr -d '\r')
echo "$r" | sed '1,/^$/d' | jq -r '.[].profile.login'
url=$(echo "$r" | sed -n -E 's/link: <(.*)>; rel="next"/\1/pi')
done
Note that if you're making multiple requests (which you probably are, since you're paginating), using a language like Python with the requests package is much faster than curl. A requests session will keep the connection open, whereas curl closes it and reopens it every time.
#!/usr/bin/env python
import requests
# Set these:
url = 'https://COMPANY.okta.com/api/v1/users'
token = '...'
# If you're making multiple API calls, using a session is much faster.
session = requests.Session()
session.headers['authorization'] = 'SSWS ' + token
def get_objects(url):
while url:
r = session.get(url)
for o in r.json():
yield o
url = r.links.get('next', {}).get('url')
for user in get_objects(url):
print(user['profile']['login'])

Answer 3

This version of the shell script has better variable names to make it easier to understand.
As commented above, running curl with the -i or --include option includes the response headers.
#!/usr/bin/env bash
# Set these:
url='https://COMPANY.okta.com/api/v1/users'
token='...'
# Pagination code based on https://michaelheap.com/follow-github-link-header-bash
while [ "$url" ]; do
r=$(curl -i --compressed -Ss -H "authorization: SSWS $token" "$url" | tr -d '\r')
headers=$(echo "$r" | sed '/^$/q')
body=$(echo "$r" | sed '1,/^$/d')
echo "$body" | jq -r '.[].profile.login'
url=$(echo "$headers" | sed -n -E 's/link: <(.*)>; rel="next"/\1/pi')
done