Articles in this section

See more

Extracts: View & Download

Avatar
Adverity Pre-Sales
  • Updated
Follow

This endpoint is used to retrieve metadata about extracts that have been generated by Datatap, as well as to download extracts.

View and Filter

Perform a GET request to get a list of all extracts.

curl -X GET \
 https://YOUR_STACK.datatap.adverity.com/api/extracts/ \
 -H 'Authorization: Token TOKEN_VALUE' \

This will return a count of all extracts, as well as meta information about each one, e.g. the EXTRACT_ID, CREATE_DATE, EXTRACT_NAME, number of rows, number of columns, and state.

As in the UI, the list of extracts can be filtered and sorted:

  • DATASTREAM_ID: Filter by Datastream ID
  • STATE: Filter by extract state*
    • 1: collected
    • 3: processing
    • 4: import error
    • 6: transform error
    • 12: imported 
    • 13: failed
  • ORDERING: Sort by extract created date or extract scheduled date. Prefix ordering with a dash to reverse the sort order, e.g. -created.
  • CREATED_FROM / CREATED_TO: Filter by EXTRACT_CREATED timestamp.  Date format (YYYY-mm-dd) and timestamps (YYYY-mm-dd HH:MM:SS) are supported.
  • TAGS: Filter to limit the result set to extracts with the specified tag.
  • OFFSET / ROWS: Use parameters offset and rows to iterate (page) through results. By default, all rows will be retrieved.
*Note: In total, there are 14 extract states available, some of which are not commonly used. Visit https://YOUR_STACK.datatap.adverity.com/core/datastreamextract/ and click on State: All to see the complete list of available states and their IDs.

Example: 

curl -X GET \
 'https://YOUR_STACK.datatap.adverity.com/api/extracts/?datastream_id=31&ordering=-created&state=2&created_from=2019-01-01' \
 -H 'Authorization: Token TOKEN_VALUE' \

 

Downloading Extracts

1. Downloading a single extract by EXTRACT_ID

Use the download_url, (i.e. /api/extracts/EXTRACT_ID/download/). There is also the optional parameter 'fileformat' and options 'json', 'csv', 'xlsx'.

Input the EXTRACT_ID of the extract required.

curl -X GET \
 https://YOUR_STACK.datatap.adverity.com/api/extracts/EXTRACT_ID/download?fileformat=json \
 -H 'Authorization: Token TOKEN_VALUE' \u

The result will be the requested extract. Unless you specify a file format, like in the example above, the default will always be csv.

2. Downloading a Datastream's most recent extract

Use /api/datastreams/DATASTREAM_ID/extracts/most-recent/download/.

Input the DATASTREAM_ID of the Datastream required.

curl -X GET \
 https://YOUR_STACK.datatap.adverity.com/api/datastreams/DATASTREAM_ID/extracts/most-recent/download \
 -H 'Authorization: Token TOKEN_VALUE' \

This endpoint also supports the 'fileformat' parameter.

3. Bulk downloading (streaming) extracts from a specific scheduled date

Use /api/datastreams/DATASTREAM_ID/extracts/streaming/download/?scheduled_from=%Y-%m-%d %H:%M:%S. 

curl -X GET \
 'https://YOUR_STACK.datatap.adverity.com/api/datastreams/DATASTREAM_ID/extracts/streaming/download?scheduled_from=YYYY-MM-DD%20HH:MM:SS' \
 -H 'Authorization: Token TOKEN_VALUE' \

When streaming large result sets, they are automatically chunked.

Related articles

Comments

0 comments

Article is closed for comments.