Logstash is a popular log shipping tool, which can be used to forward data from almost every source to Loom.

Release or Environment

Sophie standalone versions 3.4.x - 3.7.x


Step 1 - Installing Logstash

Locate a server where LogStash will run. If forwarding files, this should be the server where the files are located. If forwarding data from a database, It is recommended to install Logstash on a dedicated server / VM.

On the designated server, install Java 11, then download and install Logstash.

Step 2 - Preparing for encryption

To encrypt the traffic, download the Loom CA certificate and place under the Logstash folder in a sub-folder named cert .
Next, generate a client-side certificate:

openssl genrsa -out selfsigned.key 2048
openssl req -new -key selfsigned.key -batch -out selfsigned.csr
openssl x509 -req -days 3650 -in selfsigned.csr -signkey selfsigned.key -out selfsigned.crt
rm selfsigned.csr

Copy the key  and crt  files to the cert  folder as well. 

Step 3 - Create the configuration

Under the Logstash folder, create a directory called conf  and inside create a file named logstash.conf . Paste the following in the file:

input {
    # debug input
    # stdin {}

output {
#    tcp {
#        host => "{{customer}}"
#        port => 443
#        codec => json_lines
#        ssl_enable => true
#        ssl_cert => "cert/selfsigned.crt"
#        ssl_key => "cert/selfsigned.key"
#        ssl_cacert => "cert/loom.cer"
#    }

#    syslog {
#        host => "{{customer}}"
#        port => 6514
#        appname => "{{application-name}}"
#        protocol => "ssl-tcp"
#        rfc => "rfc5424"
#        ssl_cert => "cert/selfsigned.crt"
#        ssl_key => "cert/selfsigned.key"
#        ssl_cacert => "cert/loom.cer"
#    }

     # debug output
     # stdout { codec => rubydebug }

The configuration template includes two outputs.

To use the Syslog output (recommended) 

First, make sure that your Firewall isn't blocking traffic over port 6514. To test this, follow this article.
Next, install the required output plugin by running from the Logstash directory:

bin/logstash-plugin install logstash-output-syslog

Finally, uncomment the syslog  clause in the configuration file, and replace the {{customer}} and {{application-name}} handles.
If shipping more than one input, it's best to populate the appname  field based on some property (or tag) of the input - contact us if you need help with doing that!

To use the TCP output

To use the TCP output with encryption you must have the TCP output plugin version 3.2 or later. To check your version, run:

bin/logstash-plugin list --verbose | grep tcp

To update the plugin, run:

bin/logstash-plugin update logstash-output-tcp

Uncomment the tcp  clause in the configuration.
Then,  replace the {{customer}} handle with your customer name, which can be found in the domain you use to browse to Loom.

Step 4 - Adding inputs

Add inputs as appropriate. If you intend to read data from a database and forward to Loom, use this guide

Step 5 - Test the configuration and run LogStash

By now, your folder structure should look as follows:

├── bin/
├── certs/
│   ├── loom.cer
│   ├── selfsigned.crt
│   └── selfsigned.key
└── config/
    └── logstash.conf

Test your configuration file by running:

bin/logstash --configtest -f config/logstash.conf

Start LogStash by running  the following command:

bin/logstash -f config/logstash.conf

Step 6 - (optional) Run Logstash as a service

If you installed Logstash using a package manager, you should already be able to simply start the service.
If running on Windows, we recommend using the life-saving NSSM.

Article Information

Last Updated:2020-05-19 11:40:09