Skip to main content

Scroll logs

POST 
/v1/scroll

This endpoint can take 2 types of call requests. The first type runs a search query that returns a scrollID and the first batch of paginated results. The second request type passes only the scroll_id (The variation in the field name is intentional) to fetch the next batches of paginated results. This endpoint always returns results as a stringified JSON.

How it works:

First, send a request to establish the scrollID. This initial request contains the query object and additional parameters, similar to the v1/search endpoint, with the exception that dayOffset and accountIds are not supported. The request will return the field scrollId and the number of hits, representing the number of matching results. For example, the scroll_Id string may have a value *************80Y1JVcldDaVEAAAAAjeoh8hZYNkVkXzNhWVJRaUIwcWF5TEVnU2ZR.

Next, send the scroll_id in the request body to retrieve the log results as a stringified JSON. Each call returns the next page, where each page can return a maximum of 1000 results. Every time you resend the same scroll_id in the request body, it returns the next page until it reaches the end of the results. Note that 'scrollID' expires after 20 minutes.

Every time you send the request with the same scroll_id, the next batch of results is returned. Keep sending the same scroll ID as many times as needed to retrieve all of the available results. The results are paginated, and every request returns the next page, one at a time.

When the call returns an empty array, you'll know you've reached the end of your results.

Note:

  • Send the field scroll_id in requests (snake_case).
  • Receive the field scrollID in your responses (camelCase). It expires after 20 minutes. Please ensure to change the region in the URL to match your account's region.

Request​

Body

    query object

    Add a search query to receive the scrollID in the result.

    The query can take any of the parameters described in the Elasticsearch Search API DSL documentation with the exceptions stated below.

    You can only add the query parameters if you are not passing the scroll_id in the request.

    Limitations

    • The query can only run on 2 consecutive indexes. By default, the query runs on data sent today and yesterday. You can also add a filter on timestamp to search a smaller time frame.
    • When using query_string, allow_leading_wildcard must be set to false to disable leading wildcards. In other words, the query can't start with * or ?
    • Can't use fuzzy_max_expansions, max_expansions, or max_determinized_states
    size int32

    Possible values: <= 1,000

    Default value: 10

    Number of results to return

    from int32

    Of the results found, the first result to return.

    sort object[]

    Limitations

    • Can't sort on analyzed fields, such as the message field
    _source string[]

    The object includes specifies an array of strings specifying an array of fields to return.

    • If you omit _source from the request, all fields are returned.
    • If you pass '_source': false, it will exclude the _source field from the results.
    includes undefined[]

    Array of fields to return

  • Array [

    • string

  • ]
    • post_filter object
    scroll string

    These time units are supported:

    UnitDescription
    mminutes
    sseconds
    msmilliseconds
    microsmicroseconds
    nanosnanoseconds

    Limitations

    • Time search must be ≤ 5 minutes. If no time is specified, default is 1m (1 minute).
    aggregations object

    Apply field aggregations. See the Elasticsearch guide for details.

    Limitations

    • When using the size element, the value must be ≤ 1000
    • Can't nest 2 or more bucket aggregations of these types: date_histogram, geohash_grid, histogram, ip_ranges, significant_terms, terms
    • Can't sort or aggregate on analyzed fields, such as the message field
    • Aggregation type significant_terms and multi_terms can't be used
    • If the request specifies aggregations, only the initial search response will contain the aggregations results

    Note: You can use aggs or aggregations as the field name

Responses​

successful operation. hits are the total number of logs that match the query, which will always be in the 0-2 day range. total are the actual logs that are returned when using the query, which are not limited by the selected time range.

Schema
    code int32
    scrollId string

    Keep passing this ID in the request until you've retrieved all of the results. Copy this ID and pass it as the field scroll_id in a request to the same endpoint to retrieve the next page of results. (Remember to first clear the request body of all other parameters. The scrollId is valid for 20 minutes.)

    hits string

    Query results in stringified JSON format. 'hits' are the total number of logs that match the query.

Loading...