> For the complete documentation index, see [llms.txt](https://docs.invisiblebits.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.invisiblebits.com/integration-of-data-sources/ingestion-architecture.md). # Ingestion architecture Data source integration is a core functionality of the platform, enabling seamless integration of any data source regardless of its location—whether situated within a client's private network, hosted publicly on the Internet, sending events directly from the cloud, running on a public cloud provider, or residing within a 5G telecommunications core. The platform is designed with a 5 + 1 layer architecture, where each layer is responsible for providing services to its immediate upper and lower layers. This approach facilitates the construction of robust log shipping pipelines that include comprehensive error handling, layer-specific specialised mechanisms, and abstraction between layers.

Multi-tenant log collection & shipping Ingestion architecture

Below are detailed descriptions of the roles and mechanisms associated with each layer within the ingestion architecture:

Layer #	Role	Functionalities
1	Local collection	· Local log collection in customer premises · Local persistency · Encryption & compression
2	Shipping	· Shipping using one single TCP for all customer data sources · Asymmetric encryption. Ensures confidentiality · Compression. Reduces bandwidth
3	Processing	· Processing of source formats · Normalizing to standardized field names · Enrichment & events contextualization
4	Enrichment	· Matching against Threat Intelligence · Contextualization · Frameworks
5	Data Lake	· Storage in Data Lake · Second layer of detection use cases based in AI models · Correlation & business analytics · Visualizations & dashboarding
6	Archiving	· Long term retention · Restoration to live if required

Data source integration is managed from the **Integrations → Data Sources** section. This section provides the tools for integrating data sources, including: * **Stream log ingestion** from sources located within clients' on-premises networks * **Cloud services** located in the Internet, that send it's streams logs directly to the plateform * **Public cloud platforms** such as AWS, Azure, and Google Cloud Platform

--- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.invisiblebits.com/integration-of-data-sources/ingestion-architecture.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.