Layer 1: Local collection (Forwarder)
Logs and telemetry are collected in the local network where the data sources are located, to guarantee that in the event of any communication issue in the the source environment, no logs will be lost since they have been collected by an element that has been deployed within the environment: the Forwarder.
The Forwarder is specifically designed to be distributed in as many locations as necessary, collect the logs locally at each location, and ensure that the logs will be sent to The Platform, either directly, or through a chain Forwarders architecture or another kind of topology.
The Forwarder can be deployed in two ways:
Forwarder as a container cluster service
The extraction of metrics and logs from containerized environments is a challenge for log collection and processing systems that need to process massive amounts of events. Containerized systems pose the following three challenges to logging systems:
1. There is no permanent storage; events are lost as fast as are generated.
2. There is no fixed network addressing at the network and physical layers.
3. There is no rigid association between servers and roles.
The flexibility in execution capabilities and resource allocation provided by the containerized execution paradigm also brings challenges that impact on the possibilities and performance of traditional log and metric collection systems. Below are these three challenges, and how they should be addressed by ASIP Platform to support systems based on cloud native technologies:
No permanent storage
Send container events & metrics immediately when they are generated
No fixed physical & network addresses
Push logs directly from containers, not pull from them or generate intermediate log files
No fixed relationship between servers & roles
Tag events with your business logic, depending on how you want to identify and process them in the last stages of events processing
This deployment method allows a system based on containerisation to be provided with native logging capabilities for its containers, thus using a specific driver to extract the container logs. This model allows you to extract logs and metrics from containers natively, without any fingerprint on cluster performance.
This method is recommended when you intend to extract logs from fully containerized environments, such as edge computing used in 5G cores, since it guarantees that all events will be extracted from the containers and sent to the next in the chain one as soon as they are generated, guaranteeing that there will be no loss of events in case a container is destroyed by the orchestration system, either because it has a performance problem, or it is automatically replaced by another container as part of a rollout process.
Forwarder as a Virtual Machine
Forwarder can be deployed as a virtual machine in any environment that has a virtualization system capable of running Linux guests, so it is compatible with any virtualization platform, whether it is based on well-known vendors such as VMWare or open source such as Proxmox, as well as with private cloud creation systems, such as OpenStack or OpenShift, among others.
It’s agnostic of the virtualization system. Instead of deploying through an OVA-type artifact, the solution is based on ISO images that will auto-install and configure the forwarder within the platform itself, guaranteeing that the Forwarder will be completely native with the host system, as well as to its operating system and versions of its ecosystem of libraries.
The logs collected by the Forwarder, regardless of whether they come from an active or passive source, follow the following flow of actions:
Logs reception
The Forwarder allows the collection of logs through any transport protocol and is agnostic from the format of the logs, guaranteeing that there will be no limitation in the type of logs that it can collect or in the type of sources to integrate.
The Forwarder allows you to collect logs passively and actively:
• Passive: this is the usual way, in which other elements of the environment such as perimeter devices, applications, operating systems or any other source of logs or telemetry, configure the address of the Forwarder as the destination of their data, and push them events to it.
• Active: allows the deployment of processes in charge of actively connecting with the information sources to be integrated, and that will pull their events, such as by data sources that have an API that gives access to their information, but do not have the functionality to do sending logs via push.
Pre-processing
The Forwarder has pre-processing capabilities for received events, before being sent to the next layer, allowing filtering, classification or enrichment logic to be applied when necessary, for example to apply a business logic necessary for the subsequent processing of events.
Compression
Logs are compressed with a ratio of 8:1, being reduced by eight times than when they were generated by the original source. This allows a great decrease in the necessary bandwidth in the transmission phase of the events to The Platform, which allows the sending of massive amounts of logs in real time even in communications links with low bandwidth.
Persistence
Once collected, tagged and compressed, events persisted in Forwarder's local storage before being sent to the next layer, outside the local environment where the Forwarder resides. This guarantees that events will not be lost in the event of a communications problem in any of the intermediate networks between the Forwarder and the The Platform, either due to a networking problem, a misconfiguration or a cyber-attack.
Forwarder will continue to receive and persist events until the next layer -a Collector or another Forwarder- is available. Once the communication is recovered, the sending process will restart, and only when the sending destination confirms the reception, the events will be deleted from the local storage. This mechanism avoids events lost in case of availability problems at the following layers of the infrastructure.
Encryption & shipping
Communication between the Forwarder and the next element in the chain (another Forwarder or a cloud Collector) is done using a binary protocol, designed for logs shipping, into a tunnel with asymmetric encryption that guarantees the confidentiality of the events in their transmission to The Platform, as well as end-to-end authorization to ensure that Man-In-The-Middle attack techniques cannot be performed, and event transmissions cannot be intercepted.
Last updated