Release or Environment
Sophie standalone versions 3.4.x - 3.7.x
When writing high-volume of logs network might become a bottleneck. Here are some guidelines to reduce or limit the network bandwidth your log-shipper produces.
Option 1 - encode
All modern log shippers (we know of) support efficient encoding (examples are Lumberjack, Protocol-buffers, Message-pack). Encoding the data efficiently, v.s. sending using e.g. JSON, are significant. Loom supports many modern encodings - let us know if you'd like to use one!
Option 2 - compress
Compression can be CPU-costly but by changing the compression-level you can get to your sweet point for this trade-off.
Rsyslog, for example, supports 9 levels of compression.
Option 3 - filter what you send
Option 4 - Place a cap on bandwidth usage
If you're concerned about sudden peaks in network usage by the log shipper that might impact other systems, we recommend using IPTABLES (available in all modern Linux distributions) to cap the bandwidth the process is allowed to use. Here's a good StackExchange article on how to do this.
That's it! If you have any additional ideas we'd love to hear from you so that we can expand this article.