Breaking the Rules | By Martin Baldock, Operations Director, Kroll OnTrack Legal Technologies. Reprinted from issue of Internal Auditing & Business Risk.
For many auditors, testing of controls forms a major part of the audit year. Controls are designed and put in place to, perhaps, segregate duties; rules are agreed between the businesses and audit to limit risk, but also to permit commercial transactions to take place. A typical rule might be that for purchases above £2000 a director’s signature is required. One might, therefore, assume that data mining techniques, such as rule induction, would form an important part of the internal auditor’s CAATs toolkit. But as I said in my first article, “if you can think of the question, then you don’t need data mining” so you may well not need this type of technique. However, I think data mining is vital if you want to do much wider testing of controls across large data sets, rather than simply taking a sample.
Rule induction software comes in many guises, often as part of a suite of tools. Although not a recommendation, a product I have used in the past is XpertMiner from XpertRule Software, and I will be using this for examples.
At the most basic level, rule induction software, when run against a data set with default settings, simply sorts and filters a database on criteria set by the user. Most systems attempt to divide the data into branches with yes/no tests on certain attributes contained in the data. These branches or divisions of the data result in a tree-like structure; software that uses decision trees attempts to classify or group the transactions together into sets.
Also as I have said before, simply having a great tool is not enough. Around 60% to 70% of your time and effort will be spent getting the data you want to examine imported and in the correct form. Most tools in this space have great data import screens, XpertMiner is no exception. It has a number of filters and routines that you can apply to the data almost as simply as drawing a diagram as a process flow (see screenshot 1).
Here you can see that by a simple drag and drop mechanism you can build a sophisticated import mechanism including bespoke filters and operations. The import into these tools is also very fast – this product took only seconds to load a million records onto my standard laptop.
Having got the data into the tool there are many things you can do with it. For me the most useful in an audit sense is the tree profiling technique. The “Tree Miner” in XpertMiner has automatic induction, making it very easy to discover rules and patterns in data. Conversely, there are many instances where the auditor will have great knowledge of the data set, perhaps from a previous audit. In these cases, tools like this often allow the user to interact with the rule induction process to force the knowledge extraction based on statistical information.
It has to be remembered that we often do not have the time or resource to investigate every minor anomaly and it is therefore necessary to focus on the high-risk areas. Using the interactive capability the auditor can add splits on each node of the tree to get a better understanding of the rule discovery process, or use a process of guided queries to explore how the frequencies divide into each path.
The thing I really like about this sort of product is the graphical display (see screenshot 2). The information in the screenshot is simple loan application data and shows, based on historical information, whether a loan application should be accepted or not. The very same technique could be used on purchase ledger data, stock control or even call centre data. If this technique was applied, without any preconceived ideas or expected rules imposed, the data itself would show whether controls were being bypassed. For me, this type of technique makes computer audit an exciting place to be and really helps remove the drudgery of tick and bash audit techniques of the past.
The way I like to look at testing is that if you can think of the test then so can the bad guys: use these types of understandable data mining techniques and you may well be ahead of the game.
Data mining uses
As auditors you may have to check areas of the business where you have little knowledge of the underlying technical decision-making process. For example, using rule induction techniques you could look at maintenance and repair schedules, allowing the software and data content to show what rules are in place. These actual rules – i.e. what really happens – could then be audited against procedures. Payments of any kind will have some form of process surrounding them, whether that be a sign-off level or a segregation of duties; these types of scenario are ideal for rule-based techniques and quickly and graphically show any anomalies.
If the same analysis is run every year and the rules have not changed, then it may be possible to compare the resulting tree diagrams, which would highlight any issues quickly. Likewise, branch comparison may be possible simply by comparing the pattern produced by the tree mining process.