Recommended Reading | How to Ensure the Objectivity and Reliability of Audit Evidence Obtained Through Application of Information Technology

The COVID-19 pandemic at the beginning of 2020 led to "closed" management in many parts of China, which had a significant impact on the certification industry. Most enterprises were in a production halt state after the Spring Festival, and after gradually resuming work, there were situations where it was impossible or inconvenient to receive external audits. Certification bodies also faced difficulties or inconveniences in arranging audit teams to conduct on-site audits at client locations. In response, the National Certification and Accreditation Administration, the China National Accreditation Service for Conformity Assessment, and the China Certification and Accreditation Association successively introduced policies to cope with the pandemic period. Certification bodies are also planning response policies, and using information technology to implement remote audits has become an important agenda in the industry. This article discusses some theoretical and practical aspects of how to properly use information technology to conduct remote audits of management systems, ensuring the objectivity and credibility of audit evidence obtained through information technology.

I. Audit and Objective Evidence

1. Standard Requirements

The definitions of audit and objective evidence in "GB/T 19000-2016 Quality Management Systems — Fundamentals and Vocabulary" are:

(1) Audit: A systematic, independent, and documented process for obtaining audit evidence and objectively evaluating it to determine the extent to which audit criteria are fulfilled.

(2) Objective Evidence: Data supporting the existence or truth of something.

Note 1. Objective evidence can be obtained through observation, measurement, testing, or other methods.

Note 2. Usually, objective evidence used for audit purposes consists of records, statements of fact, or other information related to the audit criteria and can be verified.

From this, it can be seen that the fundamental work of audit activities is to obtain verifiable objective evidence.

2. Objective Evidence

A complete audit activity (including certification audit, surveillance audit, and recertification audit) is a systematic activity to determine whether the management system related to the certification scope of the auditee meets standards, laws, regulations, other requirements, and the auditee's management system requirements. The management system is a set of interrelated or interacting elements established by the auditee to set policies and objectives and achieve these objectives. Therefore, audit activities need to obtain objective evidence of these interrelated or interacting elements. The objective evidence of interrelated or interacting elements is not a collection of independent pieces of evidence but a chain of evidence with interrelations or interactions. The evidence chain must meet at least the following three requirements: first, there is an adequate amount of evidence; second, the evidence can prove the sampled object; third, the evidence can corroborate each other, excluding reasonable doubt about the sampled facts.

Objective evidence usually includes documentary evidence, physical evidence, audiovisual materials, statements from parties involved, witness testimony, test reports, and inspection records. Methods to obtain objective evidence typically include inquiry, identification, and inspection.

Methods to confirm the authenticity or credibility of objective evidence mainly include:

(1) Reviewing the reason for evidence formation: Evidence formation includes the reason, process, and result of evidence formation. The reason for evidence formation refers to the process of how the evidence itself was formed, which can reflect the credibility of the evidence source. The formation of evidence is often influenced by subjective and objective factors, such as motivation and interests. The reason for the formation of objective evidence is also one of the methods to judge the credibility of obtained or provided evidence.

(2) Considering the objective environment when the evidence was discovered: The acquisition of evidence is inevitably closely related to the environment in which it is located. Reviewing the objective environment of the evidence is also an important method to determine the authenticity of the evidence. For example, in managing the work environment of a position, factors such as lighting, distance, noise, equipment, and workstation layout at the production site must be considered for their comprehensive impact on the operator. Therefore, to judge the authenticity of evidence, the objective environment at the time the evidence was discovered must be fully considered to prevent one-sidedness or insufficient verification.

(3) Reviewing whether the evidence is an original: Originals come directly from the facts of the case, have strong reliability, and greater evidential value. If it is a copy or email, it is often difficult to verify and determine its source and authenticity, making its credibility hard to confirm.

(4) Reviewing whether the provider of the evidence has an interest relationship with the auditee: If the provider of objective evidence has an interest relationship with the auditee, the authenticity of the evidence provided may be doubted and lack persuasiveness. For example, if the evidence provider is not the actual party sampled but has a consulting relationship with the auditee (consultant), the provider may be influenced by subjective factors and provide false or non-objective evidence.

Systematic audit evidence provides the basis for audit findings and conclusions. The sufficiency, objectivity, accuracy, or credibility of audit evidence determines the correctness of audit findings and conclusions, directly affecting the sufficiency, appropriateness, and correctness of the evaluation results of the auditee's management system effectiveness.

II. The Impact of Information Technology Application on Obtaining Objective Evidence and Audit Effectiveness

1. The Impact of Evidence Collection Methods on Audits

Currently, auditors usually obtain evidence through document and record review, on-site observation, and physical measurement. Extracting the auditee's documents and records can help understand their management and operational requirements and characteristics, and trace their management and operational trajectories; on-site observation obtains objective evidence by focusing on the presentation of physical objects and the actual operation of activities (process behavior), and understanding the environmental status of the audit sampling object; physical measurement verifies the authenticity or feasibility of evidence by personally conducting or participating in measurements (including qualitative and quantitative measurements) of physical objects.

For audits conducted entirely through document review plus on-site audit, this "seeing is believing" approach can ensure obtaining one or more sets of interrelated audit evidence and a relatively complete chain of audit evidence. It is also the common method of traditional audits, providing assurance for the sufficiency, appropriateness, objectivity, and credibility of audit evidence, ensuring the integrity and credibility of the audit process, and thereby ensuring the accuracy of audit findings and conclusions.

With the development of information technology and the impact of the COVID-19 pandemic, the certification industry has also begun to apply information technology to carry out certification activities and management business, using information technology means to optimize audit effectiveness and efficiency, and providing support and assurance for the integrity and credibility of the audit process.

Currently, in certification activities of certification bodies, information technology is mainly applied to remote audits. After preliminary audit scheme planning and audit schedule arrangement, objective evidence is collected through email requests and video review of documents and records, including video or audio communication and video spot checks of the auditee's on-site physical objects and activities, providing a basis for audit findings and conclusions. Remote audits can be realized through information technologies such as video, audio, and email exchanges, which can address audits that cannot be conducted on-site or where on-site audits are uneconomical. This provides opportunities for remote audit locations and personnel, shortens distance, travel time, and costs, and reduces environmental impacts related to audit travel. However, remote audits are usually planned (or purposefully) designated audit objects, obtaining audit evidence by understanding the designated audit objects, so there are the following limitations and audit risks:

(1) Since the materials and records obtained through email requests and video reviews are pre-planned or requested during off-site remote audits, they are highly specific, with a significant possibility of subjective assumptions in sampling and insufficient sample size. They may also lack verification of physical objects and on-site situations. For example, the content of rules, regulations, and work documents may not be fully supported and confirmed by behavioral evidence observed on-site (sometimes it is difficult to accurately determine whether the on-site video provided by the auditee is "staged"). Additionally, due to the lack of on-site contact, it is impossible to accurately perceive the emotional changes of the auditee personnel and their surrounding environment, making it difficult to grasp changes in their psychological state. It is also impossible to avoid situations where the providers of materials and records have no vested interest with the auditee, and sometimes it is difficult to determine or verify whether the auditee personnel are the actual parties involved or consultants, which may affect the authenticity (credibility) of the materials and records and make verification difficult, impacting the completeness and credibility of the audit process.

(2) Issues described in the ISO 9001 APG "Remote Audit Guide" appendix "Examples of Using Remote Audit Technology to Identify Risks and Opportunities," such as "monitoring remote or high-risk work; on-site visit guidance; inherent risks in equipment use and presence," including drone crashes, equipment use, adverse weather conditions, image quality, insufficient understanding of the site, equipment and conditions, and data accuracy; as well as issues related to "monitoring cameras and videos specifically recorded for audits," such as activities not conducted during remote audits, processed videos, recordings made or edited by call centers, and specially recorded training webinars.

(3) Remote audits can observe the required site and request behavioral evidence through video means, but the on-site videos provided by the auditee are often subjectively planned by the auditor or controlled by the auditee, with strong purposefulness and specificity, and are highly prominent and independent, possibly unrelated to scenes outside the video frame. For example, during a remote audit, the auditee’s video focuses on a close-up of an exhaust gas treatment device, not showing a simple workshop 15 meters away, causing the auditor to miss assessing the impact of the device’s high-power fan noise on the workshop employees. In contrast, during on-site audits, besides observing the required site and requesting behavioral evidence, auditors can also understand the closely related environment (a form of confirmation for the authenticity or credibility of objective evidence) and obtain a set of evidence through real-time observation of physical objects and process behaviors, forming a more complete evidence chain. On-site observation can also allow auditors to "accidentally" or "non-sample" observe certain scenes or phenomena, triggering emotional responses and associations that help obtain objective evidence or conduct follow-ups to acquire sufficient audit evidence to verify the correctness and effectiveness of the auditee’s planning and implementation, which is often unattainable in remote audits. For example, during an on-site audit, the author once noticed a 68-year-old man working alone outdoors (at about 40°C) nailing wooden boards on equipment packaging boxes. He was an outsourced worker for a small private company. The company considered that signing an exemption contract with the outsourcing party was sufficient to neglect safety management, failing to identify potential injuries from heatstroke or overwork accidents unnoticed by anyone, which could lead to the company’s joint liability. The company has supervisory responsibility for the safety of outsourced personnel. This is a defect and nonconformity in outsourcing management, but during remote audits, due to the limitations of the auditee’s filming or video, and the auditor’s visual, sensory, or attention limitations, such scenarios may be overlooked (as shown in Figure 1).

(4) Remote audits cannot fully utilize or may not utilize at all the organs involved in perceiving external stimuli (means of obtaining objective evidence), such as the nose, body, and ears, unlike on-site audits. For example, chemical plant audits often rely on smell, hearing, and tactile sensations to judge the presence of material leaks, volatile poisoning hazards, excessive exhaust emissions, abnormal reactions, noise emissions exceeding limits or workplace hazards, equipment vibration, ground subsidence, high-temperature hazards, and heat energy waste. These are difficult to detect in remote video audits, representing a measurement deficiency or defect that affects the sufficiency and authenticity of on-site evidence.

(5) Remote audits face difficulties in obtaining on-site video evidence in certain industries, such as: confidential sites where photography or filming is prohibited; locations or parts inaccessible to mobile phones or non-explosion-proof cameras; oil depots, hazardous chemical warehouses, high-gas mines, high-temperature work points, high-altitude work locations, and some high-risk sites (auditors on-site can access these areas if they wear personal protective equipment and do not carry mobile phones). These represent observation deficiencies or defects that affect the sufficiency and authenticity of on-site evidence.

(6) Auditors conducting remote audits may lack the ability and wisdom to understand and utilize the information and communication technologies employed to achieve the desired audit results. They may not be aware of the risks and opportunities of applying such technologies, nor the impact of these technologies on the effectiveness and objectivity of information collection. This leads to insufficient, inappropriate, or inaccurate audit planning, sampling, evidence collection, and objective judgment, directly affecting the completeness, credibility, and effectiveness of the audit process.

2. Impact of Remote Audits on Auditing

By applying information technology, certification bodies use remote audits to complete audit work without visiting the auditee’s site. During the COVID-19 pandemic, this helped many certification bodies overcome difficulties in arranging on-site audits and addressed urgent needs of enterprises requiring system certification or maintaining certification registration qualifications. Remote audits offer some flexibility under certain conditions; for example, before the scheduled remote audit period, materials and records can be requested irregularly by email to conduct preliminary document reviews. During the remote audit period, information technology can be fully utilized for targeted key audits or verification work. Proper remote audits can optimize audit effectiveness and efficiency. Additionally, remote audits may reduce travel fatigue and risks for audit teams and avoid travel and reception expenses or burdens associated with on-site audits.

However, currently, certification bodies lack guiding standards and reference templates for applying certification information technology to remote audits, and auditors lack accumulated experience or summaries. Therefore, there is a lack of systematic and standardized guidance for remote audits, resulting in insufficient guidance requirements for audit implementation. This causes some remote audits to become mere document and material reviews or formalities, with significant loss of on-site objective evidence, seriously affecting audit effectiveness.

3. Reasonable Use of Information Technology to Ensure Sufficiency of Objective Evidence Acquisition and Audit Effectiveness

As can be seen, applying information technology as a means for certification bodies to conduct remote audits has certain advantages and will become a routine audit mode increasingly used in certification activities. However, since remote audits have certain limitations (information technology is like a double-edged sword), it is necessary to leverage strengths and avoid weaknesses and use it reasonably to ensure the accuracy of information technology application.

When arranging projects that require remote audits, it is first necessary to ensure the auditor's capability, including having received training and passed assessments on "CNAS-CC14 Information and Communication Technology (ICT) Application in Audits", relevant remote audit operational requirements, and ICT-related knowledge; the audit team and the auditee should have software and hardware such as smartphones, handheld devices, laptops, desktop computers, drones, cameras, wearable technology, artificial intelligence, etc., to ensure the collection, storage, retrieval, processing, analysis, transmission of information, and two-way communication, retaining corresponding audit evidence; the main considerations for project arrangement planning are as follows:

1. Full Remote Audit

For low-risk surveillance and recertification audit projects, if there are no significant changes in the organizational structure and certification scope, audit evidence can basically be obtained through collecting documents and records, video and audio, and other certification information technology communications. The objectivity and reliability of the obtained audit evidence can be verified through video and audio certification information technology communication. It is possible to arrange the entire project to be conducted as an information technology-based off-site audit. Whenever possible, auditors who have previously participated in the audit of the project should be assigned to remote audit work, as their familiarity with the auditee can improve the rationality and accuracy of audit sampling and reduce the risks of off-site audits. Additionally, it is not advisable to consecutively arrange full remote audits for the auditee; that is, the next audit after a full remote audit should be arranged as an on-site or partial remote audit to ensure the adequacy of direct and on-site evidence collection.

For stage two initial audit projects, and surveillance and recertification audit projects with significant changes in organizational structure and certification scope (including related products and activity scopes), due to unfamiliarity with these projects or significant changes, risks such as insufficient and inappropriate sampling, lack of integrity of the auditee, lack of cooperation and understanding (lack of "common language") between the auditee and certification body auditors, inaccurate audit evidence collection and weak verification, inaccurate audit findings and conclusions, usually full remote or fully off-site audits are not advisable.

According to "CNAS-CC105 Determining Management System Audit Time (QMS, EMS, OHSMS)", regardless of the remote audit method used by the certification body, at least one on-site visit to the physical locations of the client organization should be conducted annually. On-site visits can verify to some extent the evidence and audit findings from remote audits, summarize lessons learned from planning and implementing full remote audits, provide a basis for future audit scheme planning for the project, and offer reference for the certification body's remote audit scheme planning.

2. Partial Remote Audit

Partial remote audit of a project means using information technology to conduct part of the audit remotely, while the other part is conducted on-site. Partial remote audits include project partial remote audits and multi-site partial remote audits.

(1) Project Partial Remote Audit ("Remote + On-site")

For high-risk, stage two initial audits, and surveillance and recertification audits with significant changes in organizational structure and certification scope (including related products and activity scopes), a fully on-site audit approach can be adopted. Through evaluation, for areas and parts with risks such as insufficient and inappropriate sampling, lack of integrity of the auditee, lack of cooperation and understanding (lack of "common language") between the auditee and certification body auditors, inaccurate audit evidence collection and weak verification, inaccurate audit findings and conclusions, on-site audits should be implemented. For areas or parts of the project that are familiar and without significant changes or the above risks, remote audits can be conducted. For these remote audit parts, on-site confirmation of audit evidence should be arranged during the partial on-site audit. During the COVID-19 pandemic, if professional auditors from certain regions (e.g., Wuhan during epidemic control) cannot go to the project site, but other auditors in the audit team can, the project partial remote audit ("Remote + On-site") approach can be adopted.

Attention should be paid when adopting this "Remote + On-site" audit approach:

For high-risk, stage two initial audits, and surveillance and recertification audits with significant changes in organizational structure and certification scope (including related products and activity scopes), generally only processes unrelated or less related to high-risk control should be selected for remote audit on-site. If necessary, some evidence collected through remote audit can be verified on-site during subsequent on-site audits. For example, during the COVID-19 pandemic, some non-critical processes and some documents can be audited off-site, while planning on-site audits and verification for critical processes and other documents after the pandemic. The ratio of remote audit to on-site audit man-days depends on the complexity of the auditee's business, the risk and complexity of objects requiring on-site observation for evidence, the risk and complexity of projects requiring on-site verification, and the maturity of the auditee's management system, as well as the relevant requirements of "CNAS-CC105 Determining Management System Audit Time (QMS, EMS, OHSMS)".

Evaluate the professional categories involved in the auditee's audit scope, arrange the audit team leader and main professional auditors to participate in on-site audits; if secondary professional auditors cannot attend on-site audits, they can be arranged for remote audits and, if necessary, guide on-site auditors to verify or confirm relevant professional evidence. In principle, the number of professional auditors on-site should not be less than 50% of the total professional auditors in the audit team. Practice shows that this "Remote + On-site" hybrid audit can reduce the risks of remote audits.

Planning for remote audits should involve sufficient communication with the auditee in the early stages of the audit plan to understand whether the auditee is capable of accepting ICT audits, so that evidence to be verified in the planned off-site audit ("expected collected evidence") and evidence found or felt to require on-site verification during the remote audit process ("unexpected collected evidence") can be arranged in the later stage of the audit plan, i.e., on-site audit, ensuring the sufficiency and authenticity of audit evidence collection in the remote audit part.

(2) Multi-site Partial Remote Audit

For some multi-site audits of auditees where on-site audits are inconvenient, such as some remote work sites, drone topographic survey operations with irregular and short durations (based on the client's temporary requirements), or sites considered stable in management (no significant changes) and highly credible based on past audit experience, remote audit methods can be used to obtain audit evidence. For such cases, corresponding audit scheme planning is required, for example:

Multi-site remote audits should be arranged in the early stages of the audit phase and can be conducted before the main headquarters audit, so that supplementary on-site audits or verifications can be planned if necessary, and to provide evidence for the overall effectiveness judgment of the headquarters' management system operation.

For audits of some remote work sites, risk assessment should be conducted. For high-risk projects, on-site audit should be the preferred method in certification audit scheme planning to obtain accurate audit evidence and system operation status. If certification audits or other systems have previously been certified by this certification body, and the certification body has sufficient basis to believe that the multi-site system management and risk control performance and stability are good, providing certain credibility and corresponding ICT means, remote audits can be considered for such multi-sites during surveillance and recertification audits.

For situations where on-site working hours are irregular and brief, and pre-planning for on-site audits is inconvenient, the preferred approach during certification audit planning should be on-site audits. For projects already scheduled with multiple on-site audits at various locations, other similar multiple locations may consider remote audits. For such projects, remote audit plans and auditor arrangements can be prepared in advance, maintaining communication with the auditee. Once the auditee's on-site working time is confirmed, the designated auditor should be contacted promptly to conduct the remote audit in a timely manner, allowing real-time understanding of the on-site work situation and obtaining "online" objective evidence of the "process presentation."

Based on past audit experience, remote audits can be planned for sampled multiple locations where management is relatively stable (no significant changes) and the credibility is high. However, before arranging remote audits, full communication with the auditee is necessary to understand whether there have been significant changes in the audit scope, organizational structure, personnel, or management that affect management stability and the credibility of past audit experience. If such significant changes exist, the risks and feasibility of remote audits should be evaluated.

It should be made clear that the application of information technology is a means or tool for certification audits. Whether auditors are physically present on-site or use information technology, the methods differ but the objectives are the same. Whether it is a full on-site audit, partial remote audit, or full remote audit, objective evidence must be obtained to support audit findings and conclusions. Therefore, when conducting partial or full remote audits, the focus should not be limited to the type of information technology used or the form of evidence retained. Whether sampling "static" documented information or "dynamic" video or audio materials, the initiative and relevance, planning, and on-site adaptability of sampling must be fully grasped. Attention should be paid to the relevance of sampling to the audit scope, audit objectives, and sampling targets. Only audit evidence that is targeted, sufficient, appropriate, reproducible (easy to verify), reliable, or credible can be called objective evidence.

The sufficiency and credibility of systematically obtained objective evidence during audits are prerequisites to ensuring the sufficiency and accuracy of systematic audit findings. Only sufficient and accurate audit findings can serve as the basis for correct audit conclusion judgments, and only the accuracy of audit conclusions can ensure the integrity and effectiveness of the audit as a whole.