Summary
Please see article (KB0821631) on Data Ingestion in Sophie.
At this stage, most of the logs are structured for you. You may fine-tune it to customise it even further- to best fit your team needs.
This guide will provide you with manual steps that can be taken to fine-tune your log structure.
Release
Sophie standalone versions 3.4.x-3.7.x
Instructions
When streaming data into the system, you may encounter some cases where you’ll need to edit manually, using JavaScript, the mapping that the system does to route to specific source types, application and services.
Some examples may include:
● There is no source tag
● The source tag recognized by Loom does not represent the source type
● You wish to apply a more sophisticated logical partition of your data (e.g. separate same sources coming from different hosts that representing different customers)
You can manually manipulate (map, drop, split, edit) your data as per your needs by using JavaScript functions.
Please see KB0832515 for Log Mapping & Parsing Help (JavaScript)
You can view the structure of your logs, and interact/ manipulate your data in a few levels:
- Transport Header (Configured on the shipper side) - Syslog header, Syslog 3164, Syslog 5425, JSON header, Filebeat, winlogbeat, logstash.
- Preprocessor Screen
- Mapping Screen
- Structure Screen
Note: Based on manipulated function in the Pre-processor/transportHeader mapping, Sophie will change the data before it is read by the system.
Loom Event Flow
The following screens are accessible in Data Input Options menu (the three lines located to the right of each Data Input)
Pre-Processor Screen
At this stage you can add JavaScript relating to manipulating your data before it reaches Sophie. Removing unnecessary properties & added needed properties relating to all of the logs. At this level it is beneficial to drop unnecessary data, allowing for the system to ingest only what really matters to you!
Mapping Screen
View any transported header metadata & map your data. 'Application', 'Service' & 'Source Type' are tagged here. Mapping your Data Input to a Source Type should be done here.
Header Detection | TransportHeader Mapping
Configured on the shipper side, metadata tagged as key:value properties. Common headers are Syslog & Filebeat.
For common header transports, Sophie automatically extracts the header properties.
Most common properties that are mapped using header detection, is 'Application', Source Type' & 'Service'
If header detection exists for you logs, you see the detected header in the Data Input Mapping Function screen.
Note: It is not recommend to disable header detection in Sophie's UI. Rather, best to define mapping in your shipping/Data Input configuration file.
Source Type Structure Screen
Last stage of the Data Event flow. Once preprocessor & mapping of service and application, source type allows you to parse manipulated fields within the log.
Data Input Structure Screen
View the final structure of your data & add/remove additional Key/Value Labels. You can add JavaScript at this last stage of the structuring level. (at this stage, the logs are post preprocessor, mapping & source type structure functions).
At this point of the data eventflow, Sophie has already automatically separated the transport-header(metadata) from the inner message.
