API endpoint

Base URL

Search API of the Data Lake is served using from a URL composed by three elements:

1 - Platform base URL

Your platform has a base URL composed by the HTTP schema (HTTPS), the hostname you choose and your domain, in the next form:

Example: https://soc.yourdomain.com

2 - Path: /datalakeapi added to the base URL of your request

Example: https://soc.yourdomain.com/datalakeapi

3 - Target index name or indices wildcard

This is the only element that will be different depending on your search (previous two elements will remain the same in all your search requests).

This element is composed of four different elements concatenated and separated by a hyphen:

customerlog-socname-customername-indextype-YYYY.MM.DD

Each of the elements and their possible values ​​are described below.

Element	Possibe values	Description	Example
customerlog	customerlog	Fixed string. Cannot be changed	customerlog
socname	Your company name	This is the name of your company as Cybersecurity Servics Provider	acme
customername	Name of the customer An asterisk (*) can be used to search all clients	Name of the client on which you want to execute the search statement	demobank
Index type	cef ad cloud *	Index type that defines the scope of the type of data sources where the search is to be performed	cef ad cloud *
YYYY.MM.DD	Year.Month.Day Year.Month.* Year.* *	Time range in which to execute the search. This time window will be covered by the configuration specified in the request payload.	2024.09.10 *

IMPORTANT NOTE Using an asterisk instead of customer name, or instead of the index type, to execute a global search in all customers information, can saturate the data lake and generate specific performance problems that can affect the user experience, such as the speed of loading dashboards and even availability problems.

It is recommended to perform searches in the context of a single client or index type to avoid potential performance issues.

Example of a full URL where a search request can be sent:

https://soc.yourdomain.com/datalakeapi/customerlog-socname-customername-cef-2024.08.24

Allowed methods

All requests sent to this API must always be using the GET method once. Requests sent using any other method as POST or PUT, will be ignored.

Payload

The payload of the request must include the search criteria that will be performed on the indexes specified in the request URL.

A search involves submitting one or more queries to the Data Lake, which processes them and retrieves matching documents. These matching documents are included in the response in the form of JSON objects inside a list, representing the search results. Searches can also include additional parameters to refine how queries are handled. For instance, a search can target a specific time window or limit the number of returned results. The search API enables you to query and aggregate data stored within data lake indices. The API accepts queries written in Query DSL, provided through the request body.

Example of a simple search:

GET /datalakeapi/customerlog-socname-customername-cef-*
{
    "query": {
    "match": {
    "sourceAddress": "8.8.8.8"
    }
  }
}

Example of search API response:

[
    {
        '_index': 'customerlog-oneseq-aecc-ad-2024.11.13', 
        '_type': '_doc', 
        '_id': 'lStKJpaB9XraJO3XUcx5', 
        '_score': 0.0, 
        '_source': {
            '@timestamp': '2024-08-24T18: 30: 09.301Z', 
            'winlog': {
                'opcode': 'Info', 
                'task': '', 
                'api': 'wineventlog', 
                'activity_id': '{
                    530c8cf8-00cf-4924-aa99-39aa94278684
                }', 
                'user': {
                    'identifier': 'S-1-5-21-316130601-3633804318-1287380094-17420', 
                    'name': 'adfs', 
                    'domain': 'PORTALCORP', 
                    'type': 'User'
                }, 
                'channel': 'Security', 
                'provider_name': 'AD FS Auditing', 
                'computer_name': 'ACEXADFS2.portalcorp.com', 
                'process': {
                    'thread': {
                        'id': 15472
                    }, 
                    'pid': 736
                }, 'event_data': {'param17': 'S-1-5-11'}}
]

Structure of a result

The result of a search request will contain a list of documents that match the search criteria. The documents will have a JSON format and each of them will have the following structure:

Field	Description
_index	Name of the index that contains the events that matched with the specified criteria
_type	This field will have always the value _doc is not useful for searching
_id	Unique identifier for the document in the whole Data Lake
_score	This field will have always the value _0.0 is not useful for searching
_source	This field contains the contents of the document that matched with the specified criteria