Application Logging "Worst Practices"

Outline

1. Why Logging ? Why not logging ?
2. Developpers and logging 
3. Bad log - good log ? 
4. Conclusions ans action items

Log Data Overview

• What logs?
• Audit logs
• Transaction logs
• Intrusion logs
• Connection logs
• System performance records
• User activity logs
• Various alerts and other messages

• From Where ?
• Firewalls/intrusion prevention
• Routers/switches
• Intrusion detection
• Servers, desktops, mainframes
• Business applications
• Databases
• Anti-virus
• VPNs

“ Standard” Messages

• 10/09/200317:42:57,10.10.94.13,48352,10.10.97.14,909,,,accept,tcp,,,,909,10.10.93.29,,0,,4,3,,' 9Oct2003 17:42:57,accept,labcpngfp3,inbound,eth2c0,0,VPN-1 & FireWall-1,product=VPN-1 & FireWall-1[db_tag={0DE0E532-EEA0-11D7-BDFC-927F5D1DECEC};mgmt= labcpngfp3;date=1064415722;policy_name= Standard],labdragon,48352,10.10.97.14,909, tcp,10.10.93.145,',eth2c0,inbound
• Oct 9 16:29:49 [10.10.94.4] Oct 09 2003 16:44:50: %PIX-6-302013: Built outbound TCP connection 2245701 for outside:10.10.98.67/1487 (10.10.98.67/1487) to inside:10.10.94.13/42562 (10.10.93.145/42562) PIX
• 2003-10-20|15:25:52|dragonapp-nids|TCP-SCAN|10.10.94.10|10.10.94.13|0|0|X|------S-|0|total=484,min=1,max=1024,up=246,down=237,flags=------S-,Oct20-15:25:34,Oct20-15:25:52|

What Commonly “Gets Logged”?

• System or software startup, shutdown, restart, and abnormal termination (crash)
• Various thresholds being exceeded or reaching dangerous levels such as disk space full, memory exhausted, or processor load too high
• Hardware health messages that the system can troubleshoot or at least detect and log
• Access to resources and authentication decisions
• Network connections , failed and successful
• User access privilege changes such as the su command—both failed and successful
• User credentials and access right changes , such as account updates, creation, and deletion—both failed and successful
• System configuration changes and software updates—both failed and successful

Why Logs? What Are They Good For?

• Security teams
• Monitor, detect, investigate, track, analyze
• Auditors
• Well, audit 
• The rest of IT
• Troubleshoot, measure, verify, monitor
• Finally, developers
• “ Hmmmm… maybe for debugging… I dunno  ”

Why NOT Log? Why Logs Are Baaaaaaad! 

• “ Who needs logs? I sure don’t!” - “ Debugger is better! Printf is fine too…”
• Isn’t it what system (network) infrastructure does?
• It slows down the systems, uses up disk and memory
• “ It was not in the spec ”
• We don’t know what to log – we will log something (and then hate it)
• We don’t know how to do it – we will do it somehow (and then hate it more…)

Performance Tweak

Before assembling your greppable log statement you may want to check log.isInfoEnabled() or log.isDebugEnabled(). This way you save on some cpu cycles:

// Uses many cpu cycles:
String fancyLogString = buildFancyLogString();
    
// Does nothing because "debug" is currently disabled in log4j.properties:
log.debug( fancyLogString );

Better:

if ( log.isDebugEnabled() )
{
  // Uses many cpu cycles:
  String fancyLogString = buildFancyLogString();

  // Does nothing because "debug" is currently disabled in log4j.properties:
  log.debug( fancyLogString );
}

Separate files for different areas of interest

For each file, you'll need a separate Logger.

private static Logger log = Logger.getLogger( MyClass.class )
private static Logger connectionsLog = Logger.getLogger( "connections." + MyClass.class.getName() )
private static Logger stacktracesLog = Logger.getLogger( "stacktraces." + MyClass.class.getName() )
private static Logger httpLog = Logger.getLogger( "http." + MyClass.class.getName() )