Logging outbound TCP connections on Linux servers

In my experience, you can never have enough logging information. If you’re trying to piece together the causes of system failure, or attempting to trace the origins of a cyber-attack, you’re logs are often crucial in either case. In this post I’ll show how you can enhance a Linux installation to log all outbound TCP connections for future reference.

Regardless of the operating system, most default configurations leave a lot to be desired when it comes to logging security-relevant events.

If you’re server gets compromised, it may be very difficult to detect the presence of a threat actor on the system if you have insufficient logging.

Once an attacker achieves success in the weaponization, delivery, exploitation and installation phases of the Intrusion Kill Chain, they will typically seek “hands on keyboard” access in the command and control phase by communicating with outside servers. Detecting these communications channels is an important security objective.

Logging TCP connections is also useful if the Linux server is compromised and used for further reconnaissance, for example if a scanner is downloaded onto the server and used for network and port scans.

It would be handy if we could not only analyse properties of packets, but also analyse the TCP state to detect new outbound connections (and related packets). It would also be handy if we could carry out some logging based on the state of a connection.

Assuming logging services remain operating, it turns out iptables can do both, using the state module for the former, and the LOG jump target of iptables, linking into rsyslog.

You can integrate logging into iptables scripts easily, so lets dive in to some code.

In the Bash code below, I’m creating a new chain (LOGNC) and using this to host the logging behaviour.

# Create the LOGNC chain for logging
iptables -N LOGNC
# Define the logging behaviour and the custom log prefix string
# Only log packets for the NEW TCP state and use log level set above
iptables -A LOGNC -m state --state NEW -j LOG --log-prefix ' New TCP Connection ' --log-level ${LOGLVL}
# After logging, adopt the default outbound behaviour we expect (ACCEPT)
iptables -A LOGNC -j ACCEPT
# Append a jump to the LOGNC chain on OUTPUT, to process all outbound packets
iptables -A OUTPUT -p tcp -j LOGNC

What is important to remember with the above is to remove any earlier lines where a jump to ACCEPT is defined on OUTPUT. This is commonly done as a housekeeping step in custom firewall scripts. Otherwise you’re newly appended LOGNC jump will never get processed, and the log entries will not be created.

What kind of logging output do we get?

New TCP Connection IN= OUT=ethX SRC=[srcip] DST=[dstip] LEN=[len] TOS=[tos] PREC=[prec] TTL=[ttl] ID=[id] PROTO=TCP SPT=[source port] DPT=[dest port] WINDOW=[window] RES=[res] SYN URGP=[urgp]

We can see the SYN is processed by the logging rule, as we’d expect. The prefix string can be customised using the corresponding iptables command, which could be used in grepping.

The optimal configuration will use rsyslog to send log output to a log server, as we’d have to assume our box would be compromised by any attacker. The protective monitoring architecture should also branch away from the I/O flows as soon as possible, e.g. using a dedicated NIC and network.

What could we do with logged data? One interesting idea would be to run target IP addresses against IP address Indicators of Compromise (IOCs) periodically. It would certainly add a whole extra level of detective capability.

There are some obvious limitations here. It’s only capturing outbound TCP connections, which limits the scope of effectiveness. In addition there will inevitably be an overhead.