Masking confidential information in log files with logback framework

Recently I came across one logging related issue where in I was supposed to mask some confidential information before logging. I was using logback framework. The simplest approach would be to programmatically mask the information before log statement.

E.g: Suppose we have a method logCCDetails(), which logs credit card details. Below is the pseudo code to do logging with masking.

logCCDetails(){
       LoggerFactory logger = LoggerFactory.getLogger(MyClass.class);
       logger.info(mask(accountNumber));
       logger.info(accountType);
}


//method to mask the confidential number
String mask(String acctNum){
//logic to replace all the digits of account number to 'X'
//then return the masked account number
}


So, basically we are using an utility method mask() to mask the confidential information before logger.info() statement. This approach works well if logger.info() statements for confidential information is in our application code. In those cases where we are using some external jars and that jar's code is having these logging statement, we can't add mask() method call before log statement, so this approach will not work.
In my case I was using Spring framework RestTemplate(available in spring-web jar) and apache HttpClient(available in commons-httpclient jar) to invoke web service calls using Json request/response.  Apache HttpClient internally logs every json request/response and headers before and after invoking web service call. There were some web service calls which had some confidential information in request/response. My task was to mask those confidential information in logs.
Below is the entry defined in logback.xml for web service calls:

<appender name="RESTSERVICE"
class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>restservice.log</file>
<append>true</append>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>restservice-%d{yyyy-MM-dd}.log</fileNamePattern>
<MaxHistory>1</MaxHistory>
</rollingPolicy>
<layout class="ch.qos.logback.classic.PatternLayout">
<Pattern>%d %-5p [%X{IPAddress}] [%X{SessionId}] %c - %m%n</Pattern>
</layout>
</appender>


As per the logback documentation, there is a conversion specifier called "replace", that can be used to replace strings in log statement. The format of this specifier is as follows:
replace(p){r, t} : Replaces occurrences of 'r', a regex, with its replacement 't' in the string produces by the sub-pattern 'p'. For example, "%replace(%msg){'\s', ''}" will remove all spaces contained in the event message. This can be used in above configuration inside <Pattern> tag.
<Pattern>%d %-5p [%X{IPAddress}] [%X{SessionId}] %c - %replace(%msg){'\s', ''}%n</Pattern>


I tried this in multiple ways, but it didn't work as expected. Probably I might be doing something wrong.

Then the solution I tried was to add my own custom Pattern Layout class. In above configuration the configured pattern layout class is ch.qos.logback.classic.PatternLayout provided by logback. I extended this class and overriden the doLayout() method as follows:


public class RestServicePatternLayout extends PatternLayout{


@Override
public String doLayout(LoggingEvent event) {
   String message=super.doLayout(event);
   if(event.getLoggerRemoteView().getName().equalsIgnoreCase("httpclient.wire.content")){
    message = MaskingUtil.maskConfidentialInfo( message );
   }
   return message;
 }
} 


public class  MaskingUtil {
     public static String maskConfidentialInformations(String message) {
      //  first match the pattern of confidential information
     // for e.g. - the pattern inside json for confidential info could be "confInfo":"1234".
    // Once match is found, replace all the matching strings with "X"
    }
}


This method returns the String back to the calling method, that will be printed in log files. In above code, I am comparing the logger name with string "httpclient.wire.content". This is the name which apache httpclient jar uses to initialize LogFactory for logging web service requests/responses. You can find this string in org.apache.commons.httpclient.Wire.java file.
Once the pattern matches, we mask the confidential information with X's. In above code I have written, a separate utility method to do this masking task.

After this, I configured this custom PatternLayout class in logback.xml as follows:

<appender name="RESTSERVICE"
class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>restservice.log</file>
<append>true</append>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>restservice-%d{yyyy-MM-dd}.log</fileNamePattern>
<MaxHistory>1</MaxHistory>
</rollingPolicy>
<layout class="com.my.logging.RestServicePatternLayout">
<Pattern>%d %-5p [%X{IPAddress}] [%X{SessionId}] %c - %m%n</Pattern>
</layout>
</appender>

One of the advantage what I could see is that, in logback.xml we can configure our custom layout to run for some specific appenders to improve performance. The appender should have logger name configured in such a way that it runs only for the log statements where we can expect the confidential information. We don't need to run it for all the log statements. Since we are doing String comparison every time before actual logging, it may impact performance if we run it for every log statement. So, to avoid this, I configured the "RESTSERVICE" appender only for below logger:

<logger name="httpclient.wire" additivity="false">

<level value="debug" />

<appender-ref ref="RESTSERVICE" />

</logger>

The logger name "httpclient.wire" is used only for web service requests/responses, so our custom layout will run only for those log statements. This way we can avoid unnecessary String comparison for other log statements.