Research Article: 2021 Vol: 24 Issue: 5
Fian William, Bina Nusantara University
Viany Utami Tjhin, Bina Nusantara University
Citation Information: William, F., & Tjhin, V. U. (2021). The evaluation of enterprise resource planning application using information systems success model. Journal of Management Information and Decision Sciences, 24(5), 1-13.
SAP ERP application is an application designed to provide useful information to support the strategy, operations, and decision-making at PT XYZ. However, when researchers made observations at the head office of PT XYZ, the ERP applications at PT XYZ were very manynonstandard programs. So it becomes a question of whether the ERP application still providesNet Benefit to PT XYZ. This study used the Information System Success Model as the basis for the evaluation. As a result, the significant variable is System Quality towards User Satisfaction,User Satisfaction towards Net Benefits. Meanwhile, the variables that did not significantly affect Information Quality towards Intention to Use, Information Quality towards User Satisfaction, System Quality towards Intention to Use, Service Quality towards Intention to Use, Service Quality towards User Satisfaction, and Intention to Use towards Net Benefits.
ISSM; ERP; SAP ECC6; Smart PLS; Variable.
The rapid development of Information Technology (IT) today provides many conveniences in various aspects of business activities (Oesterle et al., 2019) technology becomes an important part of the business to integrate business processes and data in various companies. Companies use information systems to collect data, process data, and produce valuable information for the company. One of the IT systems that large companies widely use to integrate business processes, data, and applications in various departments is the Enterprise Resource Planning (ERP) system. ERP systems allow companies to integrate and improve data quality for decision making, increase efficiency in business processes, improve coordination between different units, increase business growth and innovation (Mahendrawathi et al., 2017).
Many medium-scale to large-scale companies worldwide are implementing ERP systems to integrate business processes and data so that they can produce precise and accurate data for making decisions. Henceforth in this study, the ERP (SAP) application will be called an ERP application only. However, an ERP implementation project's cost is very high, which is approx $5-6.5 million USD Kalaimani (2016). Then 66-70% of ERP implementation projects in developed countries fail to achieve all project implementation objectives Benlamri and Sparer (2016). ERP implementation in Indonesian companies still faces tremendous challenges, with failure rates reaching more than 80% Mishra (2016). ERP implementation is not an end goal but a critical milestone to initiate continuous improvements in a company that can provide tangible and intangible benefits. Hence, companies need to conduct evaluations to measure the success of the ERP system. Evaluations can determine failure factors and provide recommendations for improvements to strengthen areas for improvement in the application Zare and Ravasan (2014). Therefore, application success is needed for the management to measure whether the company's investment can provide added value to the company.
PT XYZ is a company engaged in information technology services. PT XYZ has been using the ERP application since 2017; the modules implemented include Financial Accounting, Controlling, Material Management, Human Resource. These modules help employees carry out end- to-end business process operational tasks and automate business processes Junnarkar and Verma (2017). However, there has never been an application evaluation since PT XYZ used an ERP (SAP) application. Researchers also made observations at the head office of PT XYZ, which is located in Jakarta, Indonesia; these observations were made to understand the condition of the ERP (SAP) application at PT XYZ. The observation results show that the ERP (SAP) application at PT XYZ has many Z programs or additional programs that are not standard programs from SAP. One of the SAP internal consultants who was developing the Z program did not know the direction and purpose of developing the Z program and did not know the root cause of the factors that needed to be fixed from the SAP application. It raised a question, does the ERP (SAP) application at PT XYZ still provide Net Benefit to PT XYZ?
With the current condition of PT XYZ, the problem formulation or research question for this study is:
1. What variables affect the success of the ERP (SAP) application at PT XYZ?
2. How do the variables of Information Quality, Systems Quality, Service Quality, User Satisfaction, and System use and Intention affect Net Benefits?
3. What variables need to be improved to increase the success of the ERP (SAP) application in the company?
Data
Data can be defined as raw, unprocessed facts and figures with no context or meaning (Al-Mamary et al., 2014).
Information
Information is data that has been converted into a meaningful and useful context for specific users (Al-Mamary et al., 2014)
Information is very important because it can be used to improve business processes and performance. Information can become a more comprehensive knowledge of the needs of consumers and other stakeholders Žemgulienė and Valukonis (2018).
System
The system is a set of interrelated components with clear boundaries, work together to achieve common goals by receiving input and producing output in an organized manner by the transformation process (Al-Mamary et al., 2014)
System Information
Information systems involve a variety of information technologies (IT) such as computers, software, databases, communication systems, the Internet, mobile devices, and so on, to perform specific tasks, interact with and inform various actors in different organizational or social contexts (Boell & Cecez-Kecmanovic 2015).
Enterprise Resource Planning
Enterprise Resource Planning (ERP) is user-interfaced and designed to provide useful information to support strategy, operations, management analysis, and decision making in an organization. ERP system implementation affects users at various levels of the organization as it crosses all functional units (Matende et al., 2015).
The positive consequences of using ERP are efficiency, effectiveness, and reduction of operational costs. In contrast, the negative consequences of using ERP are constant system maintenance and upgrades, constant employee supervision or control, and dependence on ERP vendors Pavin and Klein (2015).
SAP System Application and Product (SAP)
SAP stands for System Application and Product; SAP is widely used and is an ERP application. SAP uses ERP software applications to improve organizational resource performance planning, management, and operational control. This SAP software can integrate functional departments' activities, one of which is quality control, human resources and financial control Junnarkar and Verma (2017). Enterprise Resource Planning (ERP) is user-interfaced and designed to provide useful information to support strategy, operations, management analysis, and decision making in an organization. ERP system implementation affects users at various levels of the organization as it crosses all functional units (Matende et al., 2015).
Information Systems Success Model (ISSM)
The evaluation theory used is the Information Systems (IS) Success Model developed by William H. DeLone and Ephraim R. McLean. The IS Success Model consists of several variables: Systems Quality, Information Quality, Service Quality, User Satisfaction, System use and Intention to use, and Net Benefits Jaafreh (2017). These variables are interrelated and have a dependence on measuring the success of information systems. The working model IS Success framework has been considered the most influential model for assessing information systems Zare and Ravasan (2014). There are other models such as the Technology Acceptance Model (TAM), Diffusion of Innovation Theory (DOI), Theory of Reasoned Action (TRA), Theory of Planned Behavior (TPB), Model of PC Utilization (MPCU), Unified Theory of Acceptance and Use of Technology (UTAUT). However, according to Islam (2013); Montesdioca & Maçada (2015) in Aldholay et al. (2018) (Figure 1).
"These models have largely ignored evaluations of information technology use, except for the IS Success Model which evaluates IT use by examining the effect of overall quality (system, information, and service quality) on user satisfaction and actual usage that affects performance impacts and the IS Success Model has been widely used to measure the success of information systems".
The success variable measured using the IS Success Model, namely Jaafreh (2017).
System Quality
The system quality variable is a desired characteristic of an IS. System quality variables focus on aspects of usability, ease of use, system flexibility, system reliability, and ease of learning, as well as intuitiveness, sophistication, flexibility, and system response time.
Information Quality
The information quality variable is the desired output characteristic in IS. Information that the system can generate is relevant, easy to understand, accurate, concise, complete, fast, and usable.
Service Quality
The service quality success variable represents the quality of support users receive from the IS department and IT support personnel. For example, training, helpdesk, and so on.
System use and intention to use
The success variable of intention to use IS represents its users' use the degree and manner in which IS. Measuring IS use is a broad concept that can be considered from several perspectives. In that case, using IS voluntarily and the actual use of an IS is an appropriate measure of success.
User satisfaction
The success variable of user satisfaction is the level of user satisfaction when using IS. The user satisfaction variable is considered as one of the most important variables for IS success.
Net benefits
The success variable of net benefits is the extent to which IS contributes to the success of individuals, groups, organizations, or various stakeholders. This variable summarizes the separate dimensions of individual impact, and organizational impact and additional IS impact measures from other researchers such as workgroup impact and social impact into a single success dimension. The choice of what impact to measure depends on the system being evaluated, the study's objectives, and the level of analysis.
Data Collection Methods
This study will collect primary data obtained through the head office of PT XYZ, which is located in Central Jakarta, and secondary data obtained from various sources on the Internet. The data collection methods used in the study is as follows:
Literature review
The literature study is carried out by collecting data and information from journals or articles on the Internet, books, and other print media related to research.
Observation
The researcher made direct observations to the head office of PT XYZ, which is located in Indonesia to observe the ERP system that PT XYZ is using.
Interview
Make a list of questions to determine the current condition of the need for evaluating ERP applications at PT XYZ. Then after conducting the interview the researcher can decide whether the ERP application at PT XYZ needs to be evaluated or not. This interview was conducted at the end of March 2020.
Questionnaire
Data collection is carried out by using questionnaires containing close questions to ERP application users to find out whether the application is running according to the needs and desires of the user. The samples taken for this study were all ERP application users at PT XYZ, totaling at least 30 users. Then 34 respondents filled out an online questionnaire which was conducted from the end of December 2020 to the end of January 2021.
Population and Sample
Population
The population in this evaluation is all users who use the ERP application (SAP ECC6) which has been used by PT XYZ, the total of all users who use this ERP application is 54 users.
Sample
The sample in this evaluation is a minimum of 30 users who use the ERP (SAP ECC6) application that has been used by PT XYZ. According to Hogg & Tanis in Wan Yunus et al. (2020). A sample size of at least 30 is recommended for research. A sample size greater than 25 or 30 is sufficient.
Questionnaire Analysis Method
At this stage, data from respondents will be grouped and processed through 3 activities in quantitative data processing techniques as follows:
Coding
Activities carried out by giving a sign or symbol in the form of a number to the respondent's answer are received to simplify the respondent's answer, making it easier to analyze and interpret data. For closed questions, the code can be determined easily, such as 1 for the answer strongly disagree, 2 for the answer to disagree, 3 for the answer quite disagree, 4 for the neutral answer, 5 for the answer fairly agree, 6 for the answer for agreeing, and 7 for the answer strongly agree. This activity will produce a score for each question obtained from the respondent's answer.
Editing
Activities are carried out by examining the entire list of questions that the respondent has filled in by considering the suitability of the respondent's answer with the questions asked. This activity will produce valid and consistent data using the SmartPLS 3 software.
Tabulation: Activities to compile and calculate data from the results of respondents to be grouped into a table so that it is easy to process using the SmartPLS 3 software. This activity will produce a table of the results of the average calculation of all questions obtained from each respondent's answers.
Data Test Methods
The following is a model testing data from respondents used in the research to evaluate the success of ERP (SAP) applications at PT XYZ:
Measurement Model (Outer Model)
A measurement model relates indicators to their latent variables. The measurement model needs two tests, namely the validity test and the reliability test. The validity test can be done by looking at the convergent validity and discriminant validity indicators. In contrast, the reliability test can be done by looking at the indicators based on composite reliability and Cronbach's alpha (Sarstedt et al., 2020).
Structural Model (Inner Model)
After testing the validity and reliability then forming a measurement model, the next step is to analyze the influence between latent variables called the structural model (inner model). Evaluation of the structural model (inner model) can be done by using the statistical T-test and the significance of the structural path coefficients through bootstrapping, which is intended to minimize the problem of abnormal research data (Sarstedt et al., 2020).
Based on the research model above, several research hypotheses can be formulated, namely as follows:
H1: Information Quality does not have a significant effect on Intention to Use H1: Information Quality has a significant effect on Intention to Use
H2: Information Quality does not have a significant effect on User Satisfaction H1: Information Quality has a significant influence on User Satisfaction
H3: System Quality does not have a significant effect on Intention to Use H1: System Quality has a significant effect on Intention to Use
H4: System Quality does not have a significant effect on User Satisfaction H1: System Quality has a significant influence on User Satisfaction
H5: Service Quality does not have a significant effect on Intention to Use H1: Service Quality has a significant effect on Intention to Use
H6: Service Quality does not have a significant effect on User Satisfaction H1: Service Quality has a significant influence on User Satisfaction
H7: Use does not have a significant effect on Net Benefits H1: Use has a significant effect on Net Benefits
H8: User Satisfaction has no significant effect on Net Benefits H1: User Satisfaction has a significant effect on Net Benefits
Demographics of Respondents
Respondent data analysis was conducted to determine the characteristics of respondents who are users of the SAP application at PT XYZ through distributing questionnaires given to respondents.
Figure 2 shows that the data for 34 respondents based on gender were 14 people (44.1%) male respondents and 19 female respondents (55.9%).
Figure 3 showed that the data obtained from 34 respondents are 1 person (2.9%) with a working length of less than 1 year, 10 respondents (29.4%) with a length of work 0 people (0%) worked 3 to 5 years, 2 respondents with 5 to 7 years of work (5.9%), 7 to 9 years of work respondents (20.6%), 9 respondents 5 people (14.7%) of up to 11 years, 1 person (2.9%) with 11 to 13 years of work, 0 respondents (0%) with 13 to 15 years of work (0%), and > 15 respondents years as many as 8 people (23.5%).
Data Analysis Methods
The measurement model (Outer Model) is used to test the construct validity and instrument reliability. It can be said that an instrument is measured by looking at the Average Variance Extracted (AVE) value, where if the AVE value is greater than 0.5, then the statement in the instrument is considered valid (Sarstedt et al., 2020).
Validity test
The validity test is intended to measure whether a questionnaire is valid or not. An instrument is said to be valid if it can measure what is desired and reveal data from the variables studied appropriately. The instrument's level of validity indicates the extent to which the collected data did not deviate (Suroso et al., 2018).
Based on Table 1, Validity Test Results, two indicators have an AVE value below 0.5, while the minimum indicator value is declared valid if the AVE value is above 0.5. Therefore, the System Quality 3 and User Satisfaction 2 indicators are invalid and excluded from further testing.
Table 1 Validity Test Results | |||
Variable | AVE | AVE Minimum | Information |
Information Quality 1 | 0,933 | 0,5 | Valid |
Information Quality 2 | 0,931 | 0,5 | Valid |
Information Quality 3 | 0,804 | 0,5 | Valid |
Information Quality 4 | 0,702 | 0,5 | Valid |
System Quality 1 | 0,924 | 0,5 | Valid |
System Quality 2 | 0,683 | 0,5 | Valid |
System Quality 3 | 0,455 | 0,5 | Invalid |
System Quality 4 | 0,947 | 0,5 | Valid |
System Quality 5 | 0,909 | 0,5 | Valid |
System Quality 6 | 0,865 | 0,5 | Valid |
Service Quality 1 | 0,777 | 0,5 | Valid |
Service Quality 2 | 0,752 | 0,5 | Valid |
Service Quality 3 | 0,884 | 0,5 | Valid |
Service Quality 4 | 0,870 | 0,5 | Valid |
Intention to Use 1 | 0,914 | 0,5 | Valid |
Intention to Use 2 | 0,918 | 0,5 | Valid |
Intention to Use 3 | 0,659 | 0,5 | Valid |
Intention to Use 4 | 0,812 | 0,5 | Valid |
User Satisfaction 1 | 0,661 | 0,5 | Valid |
User Satisfaction 2 | 0,477 | 0,5 | Invalid |
User Satisfaction 3 | 0,889 | 0,5 | Valid |
User Satisfaction 4 | 0,900 | 0,5 | Valid |
Net Benefits 1 | 0,938 | 0,5 | Valid |
Net Benefits 2 | 0,882 | 0,5 | Valid |
Net Benefits 3 | 0,963 | 0,5 | Valid |
Net Benefits 4 | 0,810 | 0,5 | Valid |
Net Benefits 6 | 0,944 | 0,5 | Valid |
Based on Table 2, all indicators have an AVE value above 0.5; the minimum indicator value is valid if the AVE value is above 0.5. Therefore, all indicators are declared valid and can be processed to the next test.
Table 2 2nd Validity Test Results | |||
Variable | AVE | AVE Minimum | Information |
Information Quality 1 | 0,934 | 0,5 | Valid |
Information Quality 2 | 0,932 | 0,5 | Valid |
Information Quality 3 | 0,808 | 0,5 | Valid |
Information Quality 4 | 0,692 | 0,5 | Valid |
System Quality 1 | 0,923 | 0,5 | Valid |
System Quality 2 | 0,673 | 0,5 | Valid |
System Quality 4 | 0,950 | 0,5 | Valid |
System Quality 5 | 0,910 | 0,5 | Valid |
System Quality 6 | 0,870 | 0,5 | Valid |
Service Quality 1 | 0,786 | 0,5 | Valid |
Service Quality 2 | 0,740 | 0,5 | Valid |
Service Quality 3 | 0,886 | 0,5 | Valid |
Service Quality 4 | 0,863 | 0,5 | Valid |
Intention to Use 1 | 0,914 | 0,5 | Valid |
Intention to Use 2 | 0,917 | 0,5 | Valid |
Intention to Use 3 | 0,660 | 0,5 | Valid |
Intention to Use 4 | 0,813 | 0,5 | Valid |
User Satisfaction 1 | 0,672 | 0,5 | Valid |
User Satisfaction 3 | 0,887 | 0,5 | Valid |
User Satisfaction 4 | 0,928 | 0,5 | Valid |
Net Benefits 1 | 0,938 | 0,5 | Valid |
Net Benefits 2 | 0,882 | 0,5 | Valid |
Net Benefits 3 | 0,962 | 0,5 | Valid |
Net Benefits 4 | 0,809 | 0,5 | Valid |
Net Benefits 5 | 0,803 | 0,5 | Valid |
Net Benefits 6 | 0,944 | 0,5 | Valid |
Table 3 explained that all items of the questionnaire statement used to measure all variables are declared valid. Obtained the AVE value of all statement items of the information quality variable is greater than the minimum AVE value, which means that all items of the information quality statement are valid.
Table 3 Validity Test Results of all Variables | |||
Variable | AVE | AVE Minimum | Information |
Information Quality |
0.718 | 0.5 | Valid |
System Quality | 0.758 | 0.5 | Valid |
Service Quality | 0.674 | 0.5 | Valid |
Intention to Use | 0.693 | 0.5 | Valid |
User Satisfaction | 0.700 | 0.5 | Valid |
Reliability Test
The instrument is reliable if the instrument can reveal reliable data and the actual reality. Reliability shows that the instrument is reliable enough to be used as a data collection tool because the instrument is good (Suroso et al., 2018).
Table 4 explained that all questionnaire statement variables used to measure Information Quality, System Quality, Service Quality, Intention to Use, User Satisfaction, and Net Benefits have Cronbach's alpha greater than 0.600. Reliability less than 0.6 are not good, while 0.7 is acceptable and above 0.8 is good (Suroso et al., 2018).
Table 4 Reliability Test Results | |||
Variable | Cronbach’s Alpha | Minimum Score | Information |
Information Quality | 0.865 | 0.600 | Reliable |
System Quality | 0.918 | 0.600 | Reliable |
Service Quality | 0.844 | 0.600 | Reliable |
Intention to Use | 0.853 | 0.600 | Reliable |
User Satisfaction | 0.789 | 0.600 | Reliable |
Net Benefits | 0.948 | 0.600 | Reliable |
All values or scores of each variable are above 0.7, which is between 0.736-0.948. This means that from the results of this reliability test, each questionnaire statement variable has a fairly good reliability level and is declared reliable.
Structural Model (Inner Model)
After the measurement of the model (Outer Model) is met, the next step is to measure the structural model (Inner Model). The structural model (Inner Model) is carried out to see the structural path's coefficient obtained by means of bootstrapping. The measurement model (Outer Model) is used to test the construct validity and instrument reliability. It can be said that an instrument is measured by looking at the Average Variance Extracted (AVE) value, where if the AVE value is greater than 0.5, then the statement in the instrument is considered valid (Sarstedt et al., 2020).
Regression analysis is a method to determine the effect of one variable on another. In regression analysis, the variable that affects is called the independent variable, and the variable that is affected is called the dependent variable.
Table 5 explained that there is a hypothesis H0 is rejected and H1 is accepted. However, there is also a hypothesis H0 is accepted, and H1 is rejected because the overall T-statistics is greater and less than 1.96 at the 5% significance level. So it can be concluded that:
Table 5 T Statistics Results | ||
T Statistics | Information | |
Information Quality -> Intention to Use | 0,110 | H1 rejected, and H0 accepted |
Information Quality -> User Satisfaction | 1,336 | H1 rejected, and H0 accepted |
System Quality -> Intention to Use | 1,570 | H1 rejected, and H0 accepted |
System Quality -> User Satisfaction | 2,002 | H0 rejected, and H1 accepted |
Service Quality -> Intention to Use | 0,361 | H1 rejected, and H0 accepted |
Service Quality -> User Satisfaction | 0,625 | H1 rejected, and H0 accepted |
Intention to Use -> Net Benefits | 0,192 | H1 rejected, and H0 accepted |
User Satisfaction -> Net Benefits | 5,458 | H0 rejected and H1 accepted |
Hypothesis 1 stated that the Information Quality variable does not significantly affect the Intention to Use. Hypothesis 1 has a statistical t-test result of 0.110, which is smaller than 1.96 At the 5% significance level. Thus it can be concluded that H0 is accepted where Information Quality does not have a significant effect on Intention to Use and H1 is rejected.
Hypothesis 2 states that the Information Quality variable does not significantly affect the User Satisfaction variable. Hypothesis 2 has a statistical t-test result of 1.336, which is smaller than 1.96 at the 5% significance level. Thus it can be concluded that H0 is accepted where Information Quality has no significant effect on User Satisfaction, and H1 is rejected.
Hypothesis 3 stated that the System Quality variable does not significantly affect the Intention to Use. Hypothesis 3 has a statistical t-test result of 1.570, which is smaller than 1.96 at the 5% significance level. Thus it can be concluded that H0 is accepted where the Quality System does not have a significant effect on Intention to Use and H1 is rejected.
Hypothesis 4 states that the System Quality variable has a significant influence on the User Satisfaction variable. Hypothesis 4 has a statistical t-test result of 2.002, which is greater than 1.96 at the 5% significance level. Thus it can be concluded that H1 is accepted where System Quality has a significant effect on User Satisfaction, and H0 is rejected.
Hypothesis 5 stated that the Service Quality variable does not significantly affect the Intention to Use. Hypothesis 5 has a statistical t-test result of 0.361, which is smaller than 1.96 at the 5% significance level. Thus it can be concluded that H0 is accepted where Service Quality does not have a significant effect on Intention to Use and H1 is rejected.
Hypothesis 6 stated that the Service Quality variable does not significantly affect the User Satisfaction variable. Hypothesis 6 has a statistical t-test result of 0.625, which is smaller than 1.96 at the 5% significance level. Thus it can be concluded that H0 is accepted where Service Quality has a significant effect on User Satisfaction and H1 is rejected.
Hypothesis 7 stated that the Intention to Use variable does not significantly affect the Net Benefits. Hypothesis 7 has a statistical t-test result of 0.192, which is smaller than 1.96 at the 5% significance level. Thus it can be concluded that H0 is accepted where Intention to Use has no significant effect on Net Benefits, and H1 is rejected.
Hypothesis 8 states that the User Satisfaction variable has a significant effect on the Net Benefits variable. Hypothesis 8 has a statistical t-test result of 5.458, which is greater than 1.96 at the 5% significance level. Thus it can be concluded that H1 is accepted, where User Satisfaction has a significant effect on Net Benefits and H0 is rejected.
Based on the review of the research results, the writer intends to provide suggestions that can be useful for the company and further researchers, namely as follows:
1. Optimizing the net benefits of ERP applications by increasing user satisfaction generated by the ERP applications.
2. Optimizing user satisfaction (User Satisfaction) of ERP applications by improving the system quality (System Quality) of ERP applications.
3. Researchers suggest evaluating this ERP application routinely under PT XYZ management decisions; it could be evaluations every six months or per 1 year so that the company always knows what variables need to be improved to optimize the ERP application's net benefits.
4. Further similar research is also suggested to expand the research object to other applications such as Employee Self Service, service desk applications, and other applications used by PT XYZ.