Use Logstash to ship logs to Loom - Loom Predictive Intelligence

Summary

Logstash is a popular log shipping tool, which can be used to forward data from almost every source to Loom.

Release

Sophie standalone versions 3.4.x - 3.7.x

Instructions

Step 1 - Installing Logstash

Locate a server where LogStash will run. If forwarding files, this should be the server where the files are located. If forwarding data from a database, It is recommended to install Logstash on a dedicated server / VM.

On the designated server, install Java 11, then download and install Logstash.

Step 2 - Preparing for encryption

To encrypt the traffic, download the Loom CA certificate and place under the Logstash folder in a sub-folder named cert .
Next, generate a client-side certificate:

download openssl
run the following commands:

openssl genrsa -out selfsigned.key 2048
openssl req -new -key selfsigned.key -batch -out selfsigned.csr
openssl x509 -req -days 3650 -in selfsigned.csr -signkey selfsigned.key -out selfsigned.crt
rm selfsigned.csr

Copy the key and crt files to the cert folder as well.

Step 3 - Create the configuration

Under the Logstash folder, create a directory called conf and inside create a file named logstash.conf . Paste the following in the file:

input {
    # debug input
    # stdin {}
}

output {
#    tcp {
#        host => "{{customer}}-data.loomsystems.com"
#        port => 5044
#        codec => json_lines
#        ssl_enable => true
#        ssl_cert => "cert/selfsigned.crt"
#        ssl_key => "cert/selfsigned.key"
#        ssl_cacert => "cert/loom.cer"
#    }

#    syslog {
#        host => "{{customer}}-data.loomsystems.com"
#        port => 6514
#        appname => "{{application-name}}"
#        protocol => "ssl-tcp"
#        rfc => "rfc5424"
#        ssl_cert => "cert/selfsigned.crt"
#        ssl_key => "cert/selfsigned.key"
#        ssl_cacert => "cert/loom.cer"
#    }

     # debug output
     # stdout { codec => rubydebug }
}

The configuration template includes two outputs.

To use the TCP output (recommended)

First, make sure that your firewall is not blocking outbound traffic over port 5044. To test this, follow this article.

To use the TCP output with encryption you must have the TCP output plugin version 3.2 or later. To check your version, run:

bin/logstash-plugin list --verbose | grep tcp

To update the plugin, run:

bin/logstash-plugin update logstash-output-tcp

Uncomment the tcp clause in the configuration.
Then, replace the {{customer}} handle with your customer name, which can be found in the domain you use to browse to Loom.

To use the Syslog output

First, make sure that your Firewall isn't blocking traffic over port 6514. To test this, follow this article.
Next, install the required output plugin by running from the Logstash directory:

bin/logstash-plugin install logstash-output-syslog

Finally, uncomment the syslog clause in the configuration file, and replace the {{customer}} and {{application-name}} handles.
If shipping more than one input, it's best to populate the appname field based on some property (or tag) of the input - contact us if you need help with doing that!

Step 4 - Adding inputs

Add inputs as appropriate. If you intend to read data from a database and forward to Loom, use this guide.

Step 5 - Test the configuration and run LogStash

By now, your folder structure should look as follows:

logstash/
├── bin/
├── certs/
│   ├── loom.cer
│   ├── selfsigned.crt
│   └── selfsigned.key
└── config/
    └── logstash.conf

Test your configuration file by running:

bin/logstash --configtest -f config/logstash.conf

Start LogStash by running the following command:

bin/logstash -f config/logstash.conf

Step 6 - (optional) Run Logstash as a service

If you installed Logstash using a package manager, you should already be able to simply start the service.
If running on Windows, we recommend using the life-saving NSSM.