In particular, the parsing pipeline actually consists of three pipelines: parsing, merging, and typing, which together handle the parsing function.

It provides a functional view of the architecture and does not fully describe Splunk Enterprise internals. Note: This diagram represents a simplified view of the indexing architecture. This diagram shows the main processes inherent in indexing: See Extract data from files with headers in Getting Data In.įor more information about events and how the indexer transforms data into events, see the chapter Configure event processing in Getting Data In. Note that both types of forwarders do perform a type of parsing on certain structured data. Instead, universal forwarders forward the raw data to the indexer, which then processes it through both pipelines. Universal forwarders do not parse data in this way. Heavy forwarders can run raw data through the parsing pipeline and then forward the parsed data on to indexers for final indexing. The differentation between the parsing and indexing pipelines is of relevance mainly when deploying heavy forwarders. Note: The term 'indexing' is also used in a more general sense to refer to the entirety of event processing, encompassing both the parsing pipeline and the indexing pipeline. Writing the raw data and index files to disk, where post-indexing compression occurs.You can determine the level of segmentation, which affects indexing and searching speed, search capability, and efficiency of disk compression. Breaking all events into segments that can then be searched upon.In the indexing pipeline, Splunk Enterprise performs additional processing, including: It can also be configured to apply custom metadata to incoming events. Splunk can be set up to mask sensitive event data (such as credit card or social security numbers) at this stage.

At the same time that it processes timestamps, Splunk identifies event boundaries. Identifying timestamps or creating them if they don't exist.While many events are short and only take up a line or two, others can be long. Identifying line termination using linebreaking rules.Extracting a set of default fields for each event, including host, source, and sourcetype.While parsing, Splunk Enterprise performs a number of actions, including: During parsing, Splunk Enterprise breaks these chunks into events which it hands off to the indexing pipeline, where final processing occurs. All data that comes into Splunk Enterprise enters through the parsing pipeline as large (10,000 bytes) chunks. A single instance of this end-to-end data pipeline is called a pipeline set.Įvent processing occurs in two main stages, parsing and indexing. This pipeline consists of several shorter pipelines that are strung together. Finally, the processed data it is written to disk. Event processing and the data pipelineĭata enters the indexer and proceeds through a pipeline where event processing occurs. The indexing process follows the same sequence of steps for both events indexes and metrics indexes. When Splunk Enterprise indexes data, it breaks it into events, based on the timestamps. Splunk Enterprise can index any type of time-series data (data with timestamps).

Splunk phantom list buckets