Please see article (KB0821631) on Data Ingestion in Sophie.
At this stage, most of the logs are structured for you. You may fine-tune it to customise it even further- to best fit your team needs.
This guide will provide you with manual steps that can be taken to fine-tune your log structure.
Release or Environment
Sophie standalone versions 3.4.x-3.7.x
Some examples may include:
● There is no source tag
● The source tag recognized by Loom does not represent the source type
● You wish to apply a more sophisticated logical partition of your data (e.g. separate same sources coming from different hosts that representing different customers)
You can view the structure of your logs, and interact/ manipulate your data in a few levels:
- Transport Header (Configured on the shipper side) - Syslog header, Syslog 3164, Syslog 5425, JSON header, Filebeat, winlogbeat, logstash.
- Preprocessor Screen
- Mapping Screen
- Structure Screen
Note: Based on manipulated function in the Pre-processor/transportHeader mapping, Sophie will change the data before it is read by the system.
Loom Event Flow
The following screens are accessible in Data Input Options menu (the three lines located to the right of each Data Input)
View any transported header metadata & map your data. 'Application', 'Service' & 'Source Type' are tagged here. Mapping your Data Input to a Source Type should be done here.
Header Detection | TransportHeader Mapping
Configured on the shipper side, metadata tagged as key:value properties. Common headers are Syslog & Filebeat.
For common header transports, Sophie automatically extracts the header properties.
Most common properties that are mapped using header detection, is 'Application', Source Type' & 'Service'
If header detection exists for you logs, you see the detected header in the Data Input Mapping Function screen.
Note: It is not recommend to disable header detection in Sophie's UI. Rather, best to define mapping in your shipping/Data Input configuration file.
Source Type Structure Screen
Last stage of the Data Event flow. Once preprocessor & mapping of service and application, source type allows you to parse manipulated fields within the log.
Data Input Structure Screen
At this point of the data eventflow, Sophie has already automatically separated the transport-header(metadata) from the inner message.