One week after Westpac .. NAB's payroll processing system fails

Only one week after Westpac's computerised processing resulted in double pay or non-payments to over a million customers; it was reported on Wednesday 30 July that, as of 1.40pm, NAB's bulk payment systems had not cleared the processing backlog from the failure to process that occurred on Monday night.

Some of the NAB's customers were ensuring that their staff received their pay by resorting to old tried and true methods: including the withdrawal of $60,000.00 in cash and placing the cash in envelopes to pay staff; other customers entered the individual salary payments and bank details of a whole payroll via the online banking – one transaction at a time.

While it may not be proof positive that accountants are part of the creative class, it does show commitment to employees and is evidence that some organisations do truly put their staff first. Hopefully their heroic efforts will pay off when their employees consider other employment options in this talent-short employment market.

When a small process failure can affect a million customers it really highlights the incredible job the banks do each day keeping so many complex processes working seamlessly.

The processes that failed for NAB would have been repeated every night, hundreds of times with 100% precision. This time a small change would have connected with a few latent conditions and the 100% precision, over a whole complex processing procedure, fell to zero - in a very public way.

When the IT Problem Solving Team are able to move away from just fixing the problem (it was reported that many of their 2000 technology workers were prioritising clearing the processing backlog) they will have time to analyse what went wrong and set about changing their business process to ensure that it just cant happen again.

When we look at process management, we ask our clients to consider their core business process in a matrix of two dimensions, "Criticality" and "Frequency".

When considering the type of incident that Westpac and NAB faced, it is clear that the transaction automation is a high-criticality and high-frequency business process - every near miss and certainly any process failure demand the highest level of rigour be applied to analysing the causal structure that allowed even the smallest of near-miss to occur.

NAB and Westpac are very mature organisations with well established quality processes. What ever their internal process for problem solving, part of the analysis process will include a documentation of the causal structure; and a search of their Lessons Learned Database to see if there was any near-miss or other incidents that should have been resulted in a warning being raised.

For our clients they would use the REASON® methodology to guide us through the analysis process and to document the full causal structure of the incident and any early warning signs.

We would ask our clients to pay particular attention to any latent conditions contributing to a major adverse outcome; if these conditions played a significant role in this process failure then they are just sitting there waiting for some other small change to interact with them, in an undesirable way, so they can play a role in the next process failure.



REFERENCE:
http://www.australianit.news.com.au/story/0,25197,24101547-15306,00.html
http://www.news.com.au/adelaidenow/story/0,22606,24100923-5006301,00.html