Basic concepts
The Integrations → Transformation Pipelines section allows you to list and manage all Transformation Pipelines across all clients and their integrated sources.
The listing view allows filtering Transformation Pipelines by the following parameters:
Pipeline or Integration Identifier
Client
Vendor
Status
Transformation
A Transformation is an action executed on every event in a log stream that modifies its content—by adding or removing fields, performing calculations on existing values, enriching with external data, or applying any custom logic written in source code.
Transformations are chained together in Pipelines to build complete processing workflows on a given log stream, before being sent to the user-defined outputs.
Transformations are agnostic to the source generating the events and to the final destination where the processed events are delivered.
Transformation pipeline
A Transformation Pipeline is a sequence of transformations (or steps) that defines how events are processed. The order of these actions determines their execution sequence. Events enter the Pipeline through an integration and are processed step by step, with each action passing the event to the next transformation in the chain.
A Transformation Pipeline has different states:
Enabled / Disabled
Enabled: All steps in the Transformation Pipeline are executed on the incoming log stream. Events are fully processed and transformed before being sent to the selected outputs.
Disabled: None of the transformation steps are applied. The log stream is forwarded in its raw format, meaning events are sent exactly as generated by the original data source.
Assigned / Unassigned
Assigned: The Transformation Pipeline is linked to a single integration. All events received through that integration will be processed by this TP.
Unassigned: The TP is not linked to any integration. This mode is useful for creating TP templates, which can later be cloned and assigned to new integrations.
Step
A step is a transformation configured in a pipeline, where the steps order is critical to achieve the expected results. Steps can be reordered within the same transformation pipeline
Output
Once events have been processed through all steps of a Transformation Pipeline, they are sent to the configured outputs. A single pipeline can forward the transformed events to multiple outputs simultaneously, depending on the outputs selected during configuration.
Processor
It is the component responsible for executing the Transformation Pipeline, and therefore for processing the events of a given integration.
The Processor includes a console that allows analysts to view, in real time, the results of the configured transformations applied to incoming events.
Last updated