Project Report
Title: Evaluation Of Software Development Project
Overview of the Project
The board of the directors of XYZ, a government-owned power, supply and distribution (whole cycle) enterprise decided to re-define the customer process (customer-facing connection, billing, etc.) and to renew the underlying information systems. A one-year programme was planned, and first programme results were delivered with a two-year delay, while still suffering from quality issues and a lack of interoperability with other enterprise systems (connection of new customers, measurement of client’s energy consumption, etc.).
An external provider was hired to support the change of customer processes and the underlying technology, which was new for the enterprise. The enterprise staff was not convinced of the new system’s adequacy, particularly because the legacy system provided specific functionalities to the business users that were not considered in the initial programme planning and had to be developed in parallel.
Purpose
The IT assets delivered by the programme need to be corrected/amended to meet the full functionality. Functional specifications were created, but developers deviated from those without appropriate approval or feedback. The additional work and inefficiencies in service development caused delays on the deliveries, exceeding costs on IT and on the provider’s services, and lower service quality to the customers, e.g., from incomplete information for customer service and support staff. The delay of 200 percent and the excess of 100 percent of the project costs summarize the performance of the programme delivery. The management was concerned with the delay and impact on the assessment and thought there was dire need of an independent IS Audit of the software development process to identify current areas of control.
Executive Summary
The Power Sector has been beset by significant challenges for a number of years. These include limited availability of reliable and affordable electric power, aging and inadequate transmission and distribution networks, and utility policies and practices that badly lag behind those of modern utilities elsewhere in the world. Moreover a current-day, technology infrastructure that can enable efficient, back-office operations such as handling customer service requests is not in evidence.
For a major electric distribution utility like the XYZ enterprise, the deficiencies may translate into a level of financial performance that cannot be considered self-sustaining. XYZ enterprise has undergone sweeping changes, re-defining the customer process and renewing the underlying
information systems. The under mentioned findings as well as chronically low levels of utilization of resources pose serious barriers to a successful transition.
The IT department charged with responsibility of IT projects does not have structured approach and most of the processes are not documented and are ad-hoc. Based on review of the Software development process documentation and interaction with key members of the IT department, we have identified following key issues in project implementation. These are:
- Planned improvement on efficiency was not achieved and
- Other initiatives had to be postponed because of the delays, and the corresponding information systems could not be planned accordingly.
- Delayed delivery of project results
- Overrun of
- Incomplete functionality of the applications delivered and undetected errors in the systems due to weak testing.
- Incomplete/inaccurate information that is provided to customer service, support and
- Delays on the service provision to the end customers (e.g., connecting new customers) due to incomplete/inaccurate
- Information security problems that are caused by giving access to critical customer (individuals and enterprises) information due to inadequate security in application
Based on these findings, we have put forward certain recommendations in front of the management and it has agreed with the same. An action plan has been developed which will serve to further strengthen the enterprise’s systems.
Introduction
XYZ Enterprise is a government-owned power, supply and distribution (whole cycle) enterprise based in India. It is a company incorporated under the Companies Act 1956 and a “Government Company” within the meaning of the act. The headquarters of the company is situated at New Delhi. XYZ’s core business is generation and sale of electricity to state-owned power distribution companies and State Electricity Boards in India. XYZ Company is responsible for supply of power to six (6) districts. XYZ has 4 Circles, 23 Operation Divisions and 116 Subdivisions to manage its operations. Circles are headed by Superintending Engineers (SEs), Divisions are managed by Executive Engineers (XENs), and Subdivisions are run by Sub Divisional Officers (SDOs). Each division has a Customer Services Officer (CSO).
TABLE: XYZ CHARACTERISTICS
| No | Description | Value |
| 1 | Administrative Districts Served | 6 |
| 2 | Service Area (km2) | 17,207 |
| 3. | Operation Circles | 4 |
| 4 | Operation Divisions | 23 |
| 5 | Operation Sub-divisions | 116 |
As per recent data available, XYZ reported over 2.4 million registered customers. Approximately 86% of its total customers are domestic. The other predominant category is commercial, comprising more than 11% of customers. The industrial and agricultural customers account for 2% and 1.4% of all customers served, respectively.
The board of the directors decided to re-define the customer process (customer-facing connection, billing, etc.) and to renew the underlying information systems. A one-year programme was planned, and first programme results were delivered with a two-year delay, while still suffering from quality issues and a lack of interoperability with other enterprise systems (connection of new customers, measurement of client’s energy consumption, etc.).
An external provider was hired to support the change of customer processes and the underlying technology, which was new for the enterprise. The enterprise staff was not convinced of the new system’s adequacy, particularly because the legacy system provided specific functionalities to the business users that were not considered in the initial programme planning and had to be
The IT assets delivered by the programme need to be corrected/amended to meet the full functionality. Functional specifications were created, but developers deviated from those without appropriate approval or feedback. The additional work and inefficiencies in service development caused delays on the deliveries, exceeding costs on IT and on the provider’s services, and lower service quality to the customers, e.g., from incomplete information for customer service and support staff. The delay of 200 percent and the excess of 100 percent of the project costs summarize the performance of the programme delivery. The management was concerned with the delay and impact on the assessment and thought there was dire need of an independent IS Audit of the software development process to identify current areas of control.
Our firm, A G S & Co., Chartered Accountants, has been appointed for conducting IS Audit on Evaluation Of Software Development Process. The IS Audit Team consist of three DISA qualified Chartered Accountants (Ms. S, Mr. A & Mr. G) having immense experience in the field
of IS Audit. Ms. S will be the team leader. It also includes five assistants who have already been involved in Evaluation of Software Development Process.
Challenges
XYZ Power Sector is beset by a number of significant challenges. These include availability of reliable and affordable electric power; modernization of the aging and inadequate transmission and distribution networks; and focusing on effective, efficient system planning, construction, operation and maintenance to achieve business objectives and customer satisfaction. With respect to retailing electric service to consumers, the challenges include control of an increase in unauthorized connections, elimination of collusion between employees and customers to reduce unauthorized use of electricity, and rationalization of retail electricity tariffs; promotion of a cultural change under a more conducive work environment and compensation packages to the employees; and introduction and enforcement of merit based selection of employees. With respect to regulation of service, the challenges include achieving a balance between business and social objectives, improvement of quality of service and technical performance standards, and introduction of advanced technologies.
Background
System Development Life Cycle, is a software lifecycle management methodology that typically consists of nine phases viz.:
Phase 1: Feasibility Study
The feasibility study is based on technical, economical and social aspects and this helps in determining strategic benefits of using system. These benefits can be either in productivity gains or in future cost avoidance. The study has to also identify and quantify the cost savings and estimate the probable return on investment. This information is used to building a business case covering both tangible as well as intangible factors such as readiness of the business users and maturity of the business processes. The business case provides justification for proceeding to the next phase and is also used for reviewing progress or evaluating success of SDLC project.
Phase 2: Requirements Definition
This phase involves preparing the statement of intent explaining the problem or the need for new application to provide functional, service and quality requirements of the solution system. The user needs to be actively involved in requirements definition. This involves:
- Studying needs of the users
- Obtaining inputs from employees and managers on their expectations
- Determining information requirements of the users
Phase 3: System Analysis
This refers to the process of gathering and interpreting facts, diagnosing problems, and using the information to recommend improvements to the system. Before arriving at new design, one must thoroughly understand existing process/system and map them against new requirements to understand changes and rationale for changes. Analysis is also important to decide upon system design approach. Traditional system development generally adopt a data oriented approach since it had been focused on processing and presenting of business data.
Phase 4: Design
This phase takes primary inputs from phase 1 requirement definition. Based on the requirements identified, the team may need to finalize requirements by multiple user interactions and establishes a specifications baseline for development of system and sub system. These specifications describe:
- Parts of the system
- How they interface
- How the system need to be implemented
- Type of hardware, operating system and other software
- Network facilities
- Programme and database specifications
- Security considerations
Phase 5: Development
Use the design specifications to begin programming and formalizing supporting operational processes of the system. After the system design details are resolved, the resources needs such as specific type of hardware, software, and other services are determined. The choices depend on many factors such as time, cost and availability of skilled resources programmers and testers. The analyst works closely with the programmers. During this phase, the analyst also works with users to develop required documentation for software, including various procedure manuals. In the development phase, the design specifications are converted into a functional system that will work in planned system environment. Application programmes are written, tested and documented.
Phase 6: Testing
Before the information system can be used, it must be tested. Systems testing are done at various stages during development till implementation. There are primarily two types of testing:
- Quality assurance testing includes unit testing, interface testing, integration testing and peer
- User acceptance testing (UAT) also known as final acceptance
Testing primarily focuses on ensuring that the software does not fail i.e. it will run according to its specifications and in the way users expect. Special test data are input for processing and the results are examined against predetermined output. If it is found satisfactory, it is eventually tested with actual data from the current system.
Phase 7: User acceptance testing (UAT) or final testing
Establishes the actual operation of the new information system, with the final iteration of user acceptance testing and user sign-off. Organization may consider going for a certification and accreditation process to assess the effectiveness of the business application. This provides assurance to the management about:
- Mitigating risks to an appropriate
- Providing accountability over the effectiveness of the system in meeting
- Establishing an appropriate level of internal
UAT supports the process of ensuring that the system is production-ready and satisfies all documented requirements. The methods include:
- Definition of test strategies and
- Design of test cases and
- Execution of the
- Utilization of the results to verify system
Acceptance criteria are defined so that a deliverable satisfies the predefined needs of the user. A UAT plan must be documented for the final test of the completed system. The tests are written from a user perspective and should test the system in a manner as close to production as possible. For example, tests may be based around typical pre-defined, business process scenarios. If new business processes have been developed to accommodate the new or modified system they should also be tested at this point. A key aspect of testing should also include testers seeking to
verify that supporting processes integrate into the application in an acceptable manner. Successful completion would generally enable a project team to hand over a complete integrated
package of application and supporting procedures. Ideally, UAT should be performed in a secure testing or staging environment. A secure testing environment where both source and executable code are protected helps to ensure that unauthorized or last-minute changes are not made to the system without going through the standard system maintenance process. The nature and extent of the tests will depend on the magnitude and complexity of the system change. Test plan, data and results are to be maintained for subsequent audits.
Phase 8: Implementation
This involves roll out of the application which has been developed or acquired for the business function based on the current state. The approach for implementation will be decided based on this state. Following approaches may be adopted:
- Cut-off: Where old system/process is discontinued and new application is made live (operational).
- Phased implementation: Where new application is started in logical phases for different
- Pilot: Where a part function is implemented using new application and based on result either phased or cut-off approach is followed.
- Parallel: Where both the old and new system run simultaneously and based on problem resolution and reliability of processing by the new system, the old system is
Phase 9: Support and operations
This is the post-implementation stage following the successful implementation of a new or extensively modified system. This requires, implementation of a formal process that:
- Provides support and assistance to users in smooth operations and end-user
- There is a mechanism to record, review and implement deficiencies and future changes
- Assess adequacy of the system and projected ROI measurements as per business
- Update project management process based on lessons learned and recommendations for future projects regarding system development.
The objectives of System development process include:
- To reduce the risk of project failure
- To consider system and data requirements throughout the entire life of the system
- To identify technical and management issues early
- To disclose all life cycle costs to guide business decisions
- To foster realistic expectations of what the systems will and will not provide
- To provide information to better balance programmatic, technical, management, and cost
aspects of proposed system development or modification
- To encourage periodic evaluations to identify systems that are no longer effective
In systems development, the temptation to skip certain prescribed tasks associated with documentation, combined with the fast paced life of IT professionals, can create an environment that is not able to properly employ the best practices of systems development. However, the employment of best practices has proven over the years to provide returns in both efficiencies and effectiveness.
In all types of audit, the employment of any set of “best practices” is generally seen by auditors as a positive impact on the quality of the information, systems or operations being audited. In the case of the systems development life cycle (SDLC), some practices provide additional benefits in terms of IT audits. Specifically, throughout the steps in the SDLC, documentation is being created that provides valuable potential sources of evidence for IT auditors. In other words, employing SDLC as it is prescribed in the industry is a control.
WHAT’S THE PURPOSE OF SDLC? The purpose of SDLC is to institute a standard practice, or methodology, that is repeatable, leading high software quality. This translates into lower total cost of ownership and greater return on investment, resulting in direct savings to the corporate “bottom line”. This is the real business case for SDLC. Cost effectiveness is the desired outcome, more favorably the end result if done properly. SDLC focuses on the steps required to ensure software quality (i.e., software that is engineered and developed in such a way as to be reliable, maintainable, and supportable in a cost effective manner for the life of the system). Robust software engineering and project management practices act in concert with a good SDLC model to make this happen.
Audit Objectives
The overall objectives of the audit / review are to:
- Provide management with an independent assessment of the progress, quality and attainment of project/program objectives at defined milestones within the project/program
- Provide management with an evaluation of the internal controls of business processes at a point where enhancements can be easily implemented and processes adapted
- Satisfy process audit/assurance objectives in reviewing the process, place future reliance on the process based upon the assurance work performed and implement integrated
computer-assisted audit techniques (CAATs) as part of the design of the application
Scope
The scope of the assignment includes:
- Review the system and end-user documentation to determine its completeness and
- Review whether relevant users have participated during testing
- Verify cyclical processes for correctness (example: year-end process, quarter-end process)
- Interview end-users of the system for their understanding of new methods, procedures and operating
- Review whether reconciliation of control totals and converted data has been performed to verify the integrity of the data after
- Review all parallel testing
- Test the system randomly for
- Review unit test plans and system test plans to determine that tests for internal control are
- Verify that the system security is functioning as designed by developing and executing access
- Ensure test plans and rest results are maintained for
Audit Programme
The audit programme for the said assignment has been developed considering the control objectives that will help in the analysis of the project implementation effectiveness and assessing the control weaknesses at various points.
Control Objective 1 – SDLC Methodology
Determine the extent of the responsibilities of management, internal audit, users, quality assurance, and data processing during the system design, development, and maintenance.
Review SDLC work papers to determine if the appropriate levels of authorization were obtained for each phase.
Obtain and review requests for DP services. Determine if the University’s procedures are being followed.
Control Objective 2 – Needs Analysis
Review and evaluate the procedures for performing a needs analysis.
Review a needs analysis for a recent project and determine if it conforms to standards.
Control Objective 3 – Systems Design and Development
Review and evaluate the procedures for systems design and development.
Review design specifications schedules, look for written evidence of approval, and determine if the design specifications comply with the standards.
Determine if an audit trail and programmed controls are incorporated in the design specifications of a recent project.
Review samples of source documents used for data entry which are included in SDLC work papers of a recently developed application. Determine if they are designed to facilitate accurate gathering and entry of information.
Obtain and review programs to determine if they comply with the University’s programming standards.
Control Objective 4 – Testing Procedures
Review and evaluate the procedures for system and program testing.
Review documented testing procedures, test data, and resulting output to determine if they appear to be comprehensive and if they follow University standards.
Review the adequacy of testing performed on the manual phases of an application.
Control Objective 5 – Implementation Procedures
Review and evaluate procedures for program promotion and implementation.
Review documentation of the program promotion procedure. Determine if the standards are followed and if documentation of compliance with the standards is available. Trace selected program and system software changes to the appropriate supporting records to determine if the changes have been properly approved.
Review documentation of the conversion/implementation of a newly developed application. Determine if the University’s implementation procedures were followed.
Control Objective 6 – Post-implementation Review
Review and evaluate the procedures for performing post-implementation reviews.
Review program modifications, testing procedures, and the preparation of supporting documentation to determine if the University’s standards are being followed.
Control Objective 7 – Maintenance of Applications
Review and evaluate the procedures for the maintenance of existing applications.
Review program modifications, testing procedures, and the preparation of supporting documentation to determine if the University’s standards are being followed.
Control Objective 8 – Control over Systems Software
Review and evaluate the procedures for modifying systems software.
Review systems software modifications, testing procedures, and the preparation of supporting documentation to determine if the University’s standards are being followed.
Review and evaluate documentation of in-house developed systems software and the features/options of proprietary systems software in use.
Control Objective 9 – Documentation Standards
Obtain and review the documentation standards to determine if they are complete.
Regulations & Frameworks that have been referred to for the purpose of the said assignment include IT Assurance Framework (ITAF), IT Act 2000, IT Act 2008 (amendment) act, ISACA’s Guidelines, COBIT 5.
Tools & Techniques that have been used for the purpose of the said assignment include Functional Testing, Performance Testing, Stress testing, Peer Reviews, Interviews.
Logistic arrangements required
In order to assess the overall system structure access to the organization’s IS facilities will be a pre-requisite. Access to the following, including the hardware setup throughout, sensitive system software utilities, usage log, activity log. Overall documentations relating to the organizational structure, system development process, user rights, contract with the developers etc. need to be provided for evaluation.
Audit Checklist
A checklist has been formulated to The checklist covers the various areas identified as essential for the smooth execution of the plan as and when required to be performed. This checklist will be put to use for recording observations and for collecting evidences that become the basis of the opinion to be given, upon the overall evaluation.
Checklist – Level Controls
| S. No. | Checkpoints | Status |
| 1. | Whether a review is done of the overall IT organization structure to ensure that it provides for clear assignment of authority and
responsibility over IT operations and that it provides for adequate segregation of duties? |
|
| 2. | Whether a review is done for the IT strategic planning process to ensure that it aligns with business strategies. Evaluate the IT organization’s
processes for monitoring progress against the strategic plan? |
|
| 3. | Is it determined whether technology and application strategies and
roadmaps exit, and evaluate processes for long-range technical planning? |
|
| 4. | Whether a review is done for the performance indicators and measurements for IT. Ensure that processed and metrics are in place (and approved by key stakeholders) for measuring performance of day-
to-day activities and for tracking performance against SLAs, budgets, and other operational requirements? |
|
| 5. | Whether a review is done for the IT organization’s process for approving and prioritizing new projects? Determine whether this process is adequate for ensuring that system acquisition and development projects cannot commence without approval. Ensure that management and key stakeholders review project status, schedule, and
budget periodically throughout the life of significant projects. |
|
| 6. | Is there any evaluation of standards for governing the execution of IT projects and for ensuring the quality of products developed or acquired by the IT organization? Determine how these standards are communicated and enforced. | |
| 7. | Is there any assurance that IT security policies exist and provide adequate requirements for the security of the environment? Determine
how those policies are communicated and how compliance is monitored and enforced. |
|
| 8. | Is there any review and evaluation of risk-assessment process in place for the IT organization? | |
| 9. | Is there any review and evaluation of the process for ensuring that IT employees at the company have the skills and knowledge necessary for performing their jobs? | |
| 10. | Is there any review and evaluation of the policies and processes for assigning ownership of company data, classifying the data, protecting the data in accordance with their classification and defining the data’s life cycle? | |
| 11. | Whether it is ensured that effective process exist for complying with applicable laws and regulations that affect IT (e.g. HIPAA, Sarbanes- Oxley) and for maintaining awareness of changes in the regulatory environment. | |
| 12. | Is there any review and evaluation of the process for ensuring that end users of the IT environment have the ability to report problems, have appropriate involvement in IT decisions, and are satisfied with the services provided by IT.? | |
| 13. | Is there any review and evaluation of the processes for managing third-party services, ensuring that their roles and responsibilities are clearly defined and monitoring their performance? | |
| 14. | Is there any review and evaluation of the processes for controlling non employee logical access? | |
| 15. | Is there any review and evaluation of the processes for ensuring that the company is in compliance with applicable software licenses? | |
| 16. | Is there any review and evaluation of the controls over remote access into the company’s network? | |
| 17. | Is there any assurance that hiring and termination procedures are clear and comprehensive? | |
| 18. | Is there any review and evaluation of the policies and procedures for controlling the procurement and movement of hardware? | |
| 19. | Is there any assurance that system configurations are controlled with change management to avoid unnecessary system outages? | |
| 20. | Is there any assurance that media transportation, storage, reuse, and disposal are addressed adequately by company-wide policies and procedures? | |
| 21. | Is there any verification that capacity monitoring and planning are addressed adequately by company policies and procedures? | |
| 22. | Based on the structure of your company’s IT organization and processes, is there any identification and audit for other entity-level IT processes? |
The process objectives are:
To ensure an appropriate acquisition and / or development of information systems including software, to maintain the information systems in an appropriate manner.
The following checklist is to be used for this purpose:
| S. No. | Checkpoints | Status |
| 1. | Whether information system acquisition and / or development policy and procedure documented? | |
| 2. | Whether system acquisition and / or development policy and procedure approved by the management? | |
| 3. | Whether the policy and procedure cover the following: Problems faced in the existing system and need for replacement Functionality of new IS Security needs Regulatory compliance Acceptance Criteria Proposed roles and responsibilities Transition/ Migration to new IS Interfaces with legacy systems Post implementation review Maintenance arrangements. | |
| 4. | Whether policy and procedure documents are communicated / available to the respective users? | |
| 5. | Whether policy and procedure documents are reviewed and updated at regular intervals? | |
| 6. | Whether the organization has evaluated requirement and functionalities of proposed IS?
(Verify the requirement analysis conducted at three levels viz. process level, application level and organization level. Verify the site visit reports and other customer references obtained with respect to functionalities of proposed IS). |
|
| 7. | Whether the organization carried out feasibility study in respect of the following Financial feasibility Operational feasibility Technical feasibility | |
| 8. | Whether the selection of vendor and acquisition terms considers the following:
Evaluation of alternative vendors Specification on service levels and deliverables Penalty for delays Escrow mechanism for Source codes Customization Upgrades Regulatory Compliance Support and maintenance. |
|
| 9. | Whether the organization has identified and assigned roles in development activities to appropriate stakeholders?
(Verify the assigned roles should be on “need to know” and “need to basis”. and duties of developers and operators are segregated). |
|
| 10. | Whether the organization has a separate development, test and production environments? | |
| 11. | Whether the IS developed plan is prepared and approved by the management?
(Verify that IS development plan to include: Input data elements, Validations controls viz. Field/ Transactions/ File with appropriate error reporting Process workflow data classifications with security are in place, viz. Read only for users, Read/ Write for authorized persons Output). |
|
| 12. | Whether the testing of IS includes:
Confirms the compliance to functional requirements Confirms the compatibility with IS infrastructure Identifies bugs and errors and addresses them by analyzing root causes Escalating functionality issues at appropriate levels. |
|
| 13. | Whether the adequate documentation for: Preserving test results for future referencePreparation of manuals like systems manual, installation manual, user manual
Obtaining user sign off / acceptance |
|
| 14. | Whether the implementation covers the following? User Departments’ involvement and their role User Training
Acceptance Testing Role of Vendor and period of Support Required IS Infrastructure plan Risk involved and actions required to mitigate the risks Migration plan |
|
| 15. | If the development activities are outsourced, are the outsourcing activities evaluated based on the following practices:
What is the objective behind Outsourcing? What are the in-house capabilities in performing the job? What is the economic viability? What are the in-house infrastructure deficiencies and the time factor involved? What are the Risks and security concerns? What are the outsourcing arrangement and fall back method? What are arrangements for obtaining the source code for the software? Reviewing the capability and quality of software development activities by visit to vendor’s premises? Review of progress of IS development at periodic intervals. |
|
| 16. | Whether the organization carried out a post implementation review of new IS? | |
| 17. | Whether a process exists for measuring vendors’ performance against the agreed service levels? | |
| 18. | Whether the post implementation review results are documented? |
Findings, Observation, Recommendations, Management Response
- Audit Finding: Planned improvement on efficiency was not achieved
- Observation: Presently, XYZ‘s distribution system practices are characterized by manual and cumbersome processes, inadequate controls, insufficient commercial focus, limited transparency, little accountability and lack of reliable information. As a result, commercial operations are highly inefficient, with substantial revenue leakages and poor customer orientation. The use of information technology to improve efficiency and effectiveness is inadequate. Several standalone applications limiting the ability to effectively interface & integrate either with other applications or with potential applications to be deployed in the future.
- Recommendation: It is recommended that the overall structures be reviewed, the shortcomings be analyzed and same be rectified in a phased manner.
Management Response
Management agrees with the recommendation. Action plan will be set out to recognize the lacunas and remove them.
- Audit Finding: Delayed delivery of
- Observation: For some reason your team is following a traditional project management approach or, worse, it is not following any kind of explicit process (“let’s just start programming”). In this case, you are probably not yet aware that your project is delayed because of lack of visibility .
- Recommendation : Go agile by doing some basics:
- Identify what’s not really done, missing features and any other activity that remains to complete your deliverable.
- Split your organization into small, cross functional, self organizing teams.
- Split your work into a list of small, concrete deliverables, sorting this list by priority.
- Split time into short fixed-length iterations (from one to six weeks) where you can fit a small, coherent set of functionality.
- At the end of every iteration, demonstrate the progress your team has made and perform a retrospective to optimize your process.
- By doing this, you will have small teams using short time iterations to build a small set of features. Your product functionality will grow steadily and predictably after every iteration. As a side-effect, tracking progress and identifying and removing obstacle will become easier.
Management Response
Management agrees with the recommendation. However, measures have been taken to remove this deficiency and it has been vouched that proper steps will be strictly taken in future.
- Audit Finding: Overrun of Budget
- Observation: We found that one major factor that is the reason for cost overrun is design errors. Proper representation of client’s requirement and the blue print to achieving good technical input to project execution are not mapped out base on project designs and designs are done without extensive investigation of site leading to potential errors.
Recommendation : In controlling project delay and cost overrun due to design errors, the basic thing to be considered is the involvement of professional skills and application of competent tools throughout the project. Achieving error free design entails good communication with the entire design team and integrating a design process that is properly planned, giving enough time for corrections, extensive investigation and reviews. Similarly, an effective project planning, controlling and monitoring should be established to enhance project performance throughout the project life cycle.
Management Response
The Management has accepted the suggestions and it will further enhance the effectiveness of the process.
- Audit Finding: Incomplete functionality of the applications delivered and undetected errors in the systems
- Observation: The Enterprise has not focused upon the most important Phase of SDLC i.e. Testing Phase. Testing has been done in an inadequately-informed informal non-explicit manner based largely on presumptions which is illusory and misleading. Not focusing on Testing lead to incomplete functionality and the errors in the system remained undetected.
- Recommendation: We’ve found during testing that a certain module or area of the application is particularly error-prone, we recommend the management to not only focus on fixing the bugs. Instead, take the time to look into the development process and work with team members to understand why that area has more bugs. That way they can find the underlying causes and address those by improving their process. Some ways to improvise testing are:
Code Freeze
Assuming that you have a team of people writing code, shortly before you begin a testing cycle, you’ll want to have a code freeze where nobody can add any new code. It helps give you a consistent codebase to test and release and prevents any rushed code from breaking things right before testing.
Release to QA and Smoke test
Once the code freeze starts, and the build passes, you’ll want to get the application into your QA environment so that everyone is testing the exact same codebase. This consistency is key. If one person is testing and finds a bug in the wrong version of the codebase, it can be impossible to track down the source of the problem. So it’s always best to have everyone testing on the exact same version of your application.
Management Response
The Management regards it as an important aspect to introspect and rectify.
- Audit Finding: Information security problems that are caused by giving access to critical customer (individuals and enterprises) information
- Observation: We found that XYZ Enterprise has not maintained proper security on its information assets. Computers are physically accessible to all users. The people responsible for system administration and data security at the site are not reliable. Enterprise has not performed background checks on DBAs before making hiring decisions. Storage, access, manipulation, and
transmission of data is not safeguarded by technology that enforces their particular information control policies. No access control mechanism is in place for critical information.
Recommendation:
Access control is generally considered in three steps: Identification, Authentication, and Authorization.
Identification is an assertion of who someone is or what something is.
Strong authentication requires providing more than one type of authentication information (two- factor authentication). The username is the most common form of identification on computer systems today and the password is the most common form of authentication.
Authorization is After a person, program or computer has successfully been identified and authenticated then it must be determined what informational resources they are permitted to access and what actions they will be allowed to perform.
XYZ Enterprise can adopt Privileged Account Management products which can help mitigate the risks associated with elevated access. These products can help close out audit findings, assist in meeting compliance mandates, and increasingly enable enterprise to pass its SAS 70 reviews. Privileged account management products control access to accounts via two mechanisms. The first mechanism forces the administrator or program to check out the account password and the second mechanism changes the account’s password frequently on the target platform. These products also provide some workflow capabilities for approval and follow-up after giving emergency access to a privileged account.
Management Response
The Management will follow the recommendation and pave way for a more secured environment.
- Audit Finding: Incomplete / inaccurate information that is provided to customer service, support and customers.
- Observation: Customer service is primarily complaint resolution. At the local levels, there are no dedicated customer service representatives. Personnel are assigned to the windows for a few hours and then resume their regular duties; hence there is little or no continuity in resolving customer issues. Policies and procedures are not applied consistently.
- Recommendation: Customer service personnel need to be made aware of the existing XYZ procedures, which are not consistently applied. One customer service representative will adjust a wrong reading bill for any amount, stating there were no limits or need to verify the reading. Installments can be made if the consumer pays at least half of the bill; others will provide an installment only if the entire current balance is paid. Not all centers have new connection information and they direct the customer to the subdivision offices. The majority of complaints relate to supply failures. However, the record keeping methodology is to be made such that it is easy to determine the number and cause of complaints.
Management Response
The Management will be considering the recommendation so as to focus upon the ultimate object of customer satisfaction.
- Audit Finding: Delays on the service provision to the end customers due to incomplete/inaccurate
Observation:
- New service connections– Although new connection application forms are available in all front offices and at the utility website, many consumers seek the aid of staff in completing the application and throughout the remainder of the process to obtain a connection. Several factors, designed to minimize mistakes in the data entry into the CIS, unfortunately contribute to significant delays in consumer billing – sometimes for several billing cycles.
- Meter reading— Numerous problems were found in the area of meter reading. Commercial management and employees indicate there is insufficient time to perform the randomized evaluations of meter reading accuracy supposed to occur. Many meters are not read every month; estimates are not identified on the meter reading list or the bill. Meter readers do not consistently identify and record problems with meters.
- Bill preparation— The billing process involves manual data transfers and data entry, which often cause delays. The revenue office staff enters all the readings, bill adjustments, and connection/disconnection data for general consumers, and transfers the information electronically to the computer centers. Other readings and all payments are entered at the computer center. The billing program is outdated and batch driven; therefore one division can delay processing for all the circles served by the computer center. Although the computer center prepares exception reports that warn of possible incorrect readings or inconsistencies, as noted generally no action is taken.
XYZ‘s new connection policy is similar for general and large consumers. The difference is in the documentation required and who has the authority to approve the application.
Table: Sanctioning Authority
| Load | Sanction Authority |
| Up to 5KW | Asst. Manager Operations (SDO) |
| 5-70KW | Deputy Manager Operations (XEN) |
| 70-500KW | Manager Operations (SE) |
| > 500KW | CEO |
The delay in billing the consumer is the result of the process. First, subdivision operations staff is to prepare a summary of new connections and send it to the revenue office, usually once a month. This office uses the summary to prepare a master data addition/change form to send the customer information to the billing center for data entry with the next meter reading batch.
After data entry, the billing center prints a posting list showing new connection details and sends it to the revenue office for verification. If the data is not correct, the corrections are sent back to
the computer center by the revenue office through master data addition/change form and/or consumer meter data change form. Once the data is verified and accepted as correct, the first billing cycle may have passed. Because the meter reading list is prepared days in advance, the new consumer may have missed the first billing cycle by a matter of a few days.
- Recommendation: A detailed documentation is required to be prepared specifying the various areas, practices, policies, procedures, etc. for effectively looking after the customer services. The document must covers all the aspects and awareness for the same be made so that the functions are carried out efficiently.
Management Response
The Management has agreed to set out a specific document for the above purpose.
Conclusion
Overall the major lacunas that were identified were:
First, the development and implementation strategy used was parallel changeover, which apart from increasing the costs also results in duplication of work. However, the phased or pilot changeover would have been more convenient and cost-effective.
Second, there was not a specific model used for the development of the system (neither waterfall, prototyping, spiral nor incremental) and everything was carried out haphazardly. However, the XYZ enterprise should have used the incremental model for the development of the process so as to aid in phasing changeover to take place throughout the organization.
Third, the methodology used was also not specific. It has drawn out that the methodology adopted should have been otherwise agile software development methodology so as to cater to the changing environmental needs or should have been component based development so as to deliver the best at each phase.
It is concluded, the enterprise needs to recognize the flaws in the system, analyze them properly, and set out stringent procedures for rectifying the deviations so that it is at least able to reach at break-even and thereafter focus upon the overall profitability. As the world today is characterized the concept of survival of the fittest and that can only be achieved if the resources are optimally utilized and the process are carried out in a disciplined way rather than moving at your own will.
References:
ISA Module V www.isaca.org www.iso.org
CISA Review Manual 2015