Keywords

Research Evaluation, National Research Institute, Local Organization, Fuzzy Analytic Hierarchy Process, Science, Technology, Research Facility, Research Equipment.

Introduction

The National Research Institute is the country’s organization for national research. It can conduct a nationwide research for a national project when the government, academe and industry cannot do so. The government has supported the stable operation of the institute and has provided a budget for it, so that it could fulfil the government’s research and development (R&D) requirements. The local organization of the National Research Institute is very important for the establishment of a local innovation system (Kromrey, 2003; Levin-Rozalis, 2003; Fetterman, 2008). It is the network hub for local innovation. The local innovation system is based on the national innovation system (NIS). The NIS has carried out government R&D for the creation, introduction and improvement of the country’s technologies. Also, it is the network for non-government organizations (NGOs) and public institutions (Freeman, 1987). The NIS is defined as the gathering of innovation factors for the technology innovation process. The interaction and role of the innovation factors are recognized as significant (Nelson & Rosenberg, 1993). Edquist (1997) suggested issues for the general economic indicator, such as economic growth and employment. The general local development strategy (public infrastructure investment, enterprise attraction, etc.) imposes a limit on the country’s balanced development and local innovation. The core factor of local development is local technology innovation. Local development can enhance the country’s local growth potential and long-term development through science and technology. The absolute gap is rectified and local policymaking for science and technology innovation is needed for the country’s balanced development. Also, a local research base is needed for science and technology innovation. The local organization of the National Research Institute will serve as the essential research base. The local innovation system model for the local organization of the National Research Institute is shown in Figure 1.

The research evaluation of the National Research Institute is defined as the comprehensive evaluation of the institute’s main task (Blomberg & Waldo, 2002). The research evaluation results can be reflected in the government’s R&D policy because the production, advantage, effectiveness and value of the research results have been specifically analysed (DeLeon, 1997; Lafond et al., 2000; Wollmann, 2007). Also, the performance of and outcomes obtained by the National Research Institute are evaluated. The research evaluation involves monitoring the research effectiveness of the National Research Institute. It is needed for the optimal resource allocation and competitiveness enhancement of the institute (Hemlin, 1996; Brown, 2006; Molas-Gallart, 2015). The research evaluation has become the most important factor in public and R&D management since 1980. Technology innovation has been recognized as the most effective means of enhancing the country’s competitiveness and national welfare promotion all over the world and it needs research evaluation on the government level to determine the input relevance of R&D resources, the R&D effectiveness, the qualitative level of the research results and the R&D ripple effect (society, economy, culture, etc.) (Ormala, 1989). The past research evaluations of the National Research Institute focused on the whole institute or on its headquarters, but the demand for the local organization of the National Research Institute has increased of late due to the rising concern of the local government for regional R&D (Halachmi, 2004; Kobal, 2005; Sample et al., 2012). Also, diverse opinion shave been presented to the central government with regard to the establishment and operation of the local organization of the National Research Institute (Toda, 1999; Shi, 2012). This requires the objective determination of the role and identity of the institute (Midorikawa, 2003; Darema, 2007). The research evaluation objectives for the local organization of the National Research Institute are to determine the role and function of the institute for the establishment objective and to establish an improvement plan. In this paper, the research evaluation model and the analysis results for local organizations of 25 National Research Institutes in Ministry of Science and ICT (MSIT) of South Korea are discussed and presented. We chose local organizations of 25 National Research Institutes in Ministry of Science and ICT (MSIT) of South Korea for the research evaluation model study. Ministry of Science and ICT (MSIT) of South Korea has managed 25 National Research Institutes through National Research Council of Science & Technology (NST) of South Korea. Until recently, the research evaluation has been performed for the headquarters of 25 National Research Institutes. Ministry of Science and ICT (MSIT) and National Research Council of Science & Technology (NST) of South Korea are measured the research result and distributed the government R&D budget through the research evaluation. In recent years, the research evaluation is needed for the research result measurement and the government R&D budget distribution of the local organization of the National Research Institute because local organizations of 25 National Research Institute is greatly increased. The fuzzy analytic hierarchy process (AHP) method was utilized for the research evaluation model study because it is the most appropriate method for the research evaluation of the local organization of the National Research Institute. The fuzzy AHP is a method of resolving the uncertainty in the decision making process. In this paper, a research evaluation model for the local organization of the National Research Institute is suggested for the first time. It is expected that the results of this study will be very useful for the review and subsequent improvement of the country’s research evaluation policy.

Global Research Evaluation Trend

United States

The science and technology system of the United States consists of the assembly and the administration. The science and technology policy has been implemented through the central government and the public institute (Hansson, 2006; Tremper et al., 2010). The science and technology organizations of the White House have performed the roles of offering consultation, mediation and assistance with regard to the country’s science and technology policy. The National Science and Technology Council (NSTC) plays an important role in making decisions related to the country’s science and technology policy and R&D investments. The research evaluation of the U.S. consists of the Government Performance and Results Act (GPRA) and R&D management evaluation (DOE, NSF, NIH, NIST, NASA, etc.). GPRA is the research evaluation method of the federal government R&D; it involves objective measurement and systematic analysis. The program assessment rating tool (PART) was developed by the Office of Management and Budget (OMB) of the White House. It effectively links the research results and budget.

The representative research evaluation methods of the U.S. are the research evaluation of the National Science Foundation (NSF) and the Advanced Technology Program (ATP) of the National Institute of Standards and Technology (NIST). The research evaluation of NSF measures the said foundation’s production, efficiency and management activity (Ruegg, 2007). The internal activities of NSF have been evaluated through the staff, the high-ranking officials and the advisory committee. Especially, the R&D evaluation of NSF has been conducted every three years, through the Committee of Visitors (COVs). The COVs evaluates the fairness, effectiveness and qualitative level of the examination process for the proposal. The advice of the COVs has been utilized for the management improvement of NSF and for R&D planning. NIST’s ATP supports the R&D budget in the precompetitive stage for the enhancement of the country’s technology competitiveness. ATP has been managed by the Economic Assessment Office (EAO) for R&D effect evaluation (society, economy, etc.). The research evaluation of ATP focuses on research.

Japan

The Ministry of Education, Culture, Sports, Science and Technology (MEXT) has managed the National Research Institute for Japan’s administrative reform and R&D management system construction (Imura & Maeda, 2002; Nakamura, 2008). The National Research institutes were merged and the resulting institute became an independent administrative institution. RIKEN Institute (RIKEN), the National Institute of Advanced Industrial Science and Technology (AIST) and the Japan Research Institute (JRI) are the representative independent administrative institutions. The General Science and Technology Council have adjusted the government’s R&D direction and science and technology policy. The Science and Technology Basic Plan of Japan suggest the direction for excellent research performance, a competitive R&D environment and efficient resource distribution. Also, it suggests a new research evaluation system to promote fairness, transparency and evaluation system reformation. JRI carries out internal and external evaluation. The internal evaluation consists of a preliminary evaluation, an interim evaluation and a final evaluation (Gonda & Kakizaki, 1995). The institute has also performed research unit evaluation and research support department evaluation. The external evaluation, on the other hand, consists of business performance evaluation and intermediate-term target evaluation.

EU

EU (European Union) has collaborated on R&D and the European Committee has evaluated the plan, support modalities, activities etc. (Hills, 1997; Ohnuma et al., 2000). The Framework Programme (FP) is being implemented for the effective utilization of the research resources, R&D investment expansion and industrial competitiveness enhancement (Saunders et al., 2011). EU had formulated research evaluation guidelines for public accountability and efficiency improvement. The research evaluation objective is industrial-competitiveness enhancement. Since recently, however, the contribution of the society and the economy (employment, environment, health, etc.) has been emphasized in the evaluation. FP is evolving through knowledge spread, technology transfer, product development and commercialization, technology innovation support service etc. The research evaluation principles of the program for research, technological development & demonstration (RTD) are excellence, transparency, fairness, neutrality, effectiveness and durability.

The Research Evaluation

Research evaluation is an evaluation process that is done to determine the appropriateness, efficiency and effectiveness of R&D activity. Appropriateness refers to whether the R&D performances suitable for the science and technology policy, the R&D strategy of the enterprise and the demands of the market and the society. Efficiency refers to the research results through the input of resources and the resource input saving level (Barker, 2007). Effectiveness refers to the research result level for the achievement of the goal.

The evaluation of the appropriateness, efficiency and effectiveness of R&D activity can be performed through the relation review of the R&D system factors for R&D activity. The appropriateness of the R&D activity can be evaluated through the demands of the market, the government policy, the enterprise strategy and the R&D program. The effectiveness can be confirmed through the comparison of the input of resources and the research results (Peter, 2015).

The general and institution evaluation are performed at the same time in research evaluation (Rons et al., 2008). It is necessary to consider the evaluation keynote (evaluation objective and target), evaluation factor (evaluation period, agent, standard, index and method), evaluation support (evaluation manpower, organization and information) and evaluation utilization (evaluation report and results utilization).

For evaluation system design, it is necessary to consider the internal-external evaluation balance, R&D performance evaluation, objective organization evaluation, evaluation system simplification and evaluation system objectivity for effective R&D system development. The analysis frame for research evaluation is described in Figure 2. The evaluation objective needs intangible-value measurement. The evaluation standard plays a key role in setting the evaluation objective (Rogers & Jordan, 2010). The evaluation method measures the evaluation standard and index. The evaluation utilization decides the utilization form for the evaluator and the evaluation group.

The standard deviation and the mean are utilized for the National Research Institute’s research evaluation. The standard deviation is a measure that is used to quantify the amount of variation and dispersion of a set of data values for the National Research Institute’s research evaluation data. The mean is the average and is computed as the sum of all the observed outcomes from the total number of the National Research Institute’s research evaluation data. The detailed formula is as follows:

The analysis index score is used for the Z-value, through data normalization. The linear equation is utilized for the scoring. A linear equation is an algebraic equation in which each term is either a constant or a single variable. The analysis index score is being utilized by the National Research Institute through a linear equation. The analysis index score for the National Research Institute is described in Figure 3. The detailed formula for the linear equation is as follows:

Where, y: Analysis Index Score; x: Z – value

Employment promotion and added-value creation are needed and are realized through the new industry and the new technology for regional development and economic growth (Prins, 2016). Therefore, technology innovation capacity enhancement is essential for regional development and economic growth. The local governments need the R&D activity of the university, enterprise and National Research Institute for their technology innovation capacity enhancement and endogenous growth. The central government can expand its R&D investment and can establish the local organization of the National Research Institute for balanced regional development and technology innovation capacity enhancement. The evaluation standards for the correlation of the local organization and the region are decided through the research field, the local industry and the local organization establishment for the National Research Institute.

In this paper, the detailed formula that is used for the correlation of the local organization and the region is shown below, where W_ij is the correlation of the region (R_i) and the National Research Institute (L_j).

W_ij: correlation of region R and national research institute L_j

R_i: region i

L_j: local organization j of national research institute

C_k: weighted value k of evaluation

The Fuzzy Analytic Hierarchy Process (Ahp) Method

The fuzzy AHP is the combination of the fuzzy theory and the AHP for the alternative suggestion and the problem solution. The fuzzy AHP is a way of resolving the uncertainty in the decision making process. It has been researched on by Laarhven & Pedrycz (1983), with the two attempting to combine the Saaty theory & the fuzzy theory. Chang (1996) suggested the fuzzy AHP utilizing the triangular fuzzy number through the extent analysis method. The triangular fuzzy number application shown in Figure 4 consists of l, m and u (l,m,u).The detailed formula is as follows:

Satty & fuzzified Satty’s for the triangular fuzzy number are shown in Table 1. If δ=1 is fixed, the triangular fuzzy number is (1, 1, 2), (1, 2, 3), (2, 3, 4), (3, 4, 5), (4, 5, 6), (5, 6, 7), (6, 7, 8), (7, 8, 9), (8, 9, 9).

Table1 Satty and Fuzzified Satty’s for the Triangular Fuzzy Number
Definition	Scale of Satty	Fuzzyfied Satty scale
Of equal importance (M)	1	(1, 1, 1+d)
Somewhat more important (H3)	3	(3-d, 3, 3+d)
Much more important (H5)	5	(5-d, 5, 5+d)
Very much more important (H7)	7	(7-d, 7, 7+d)
Absolutely more important (H9)	9	(9-d, 9, 9)
Intermediate value	2, 4, 6, 8	(x-1, x, x+1), x=2, 4, 6, 7

The detailed formula for the triangular fuzzy number of the fuzzy synthetic extent is as follows:

S_i is the fuzzy synthetic extent for i, and M_ij is the triangular fuzzy number. It is presented as . The detailed formula is as follows:

The weighted-value vector is calculated through the relative importance of the fuzzy synthetic extent. The relative importance is the highest probability in two triangular fuzzy numbers. M₂≥M₁ is fixed in and . The detailed formula is as follows:

hgt is the highest intersection point of the two triangular fuzzy numbers and d is the coordinate value of the x axis for the intersection point of and . The detailed formula is as follows:

The detailed formula for the probability calculation is as follows:

If are assumed, the detailed formula is as follows:

W' is the weighted-value vector and W is the normalized weighted-value vector.

In this study, the research evaluation model was researched on for the local organization of the National Research Institute. The criteria and indicators for the research evaluation of the local organization of the National Research Institute are shown in Figure 5.

Organization of the National Research Institute

In this study, 30 research evaluation specialists were surveyed for the analysis of the research evaluation for the local organization of the National Research Institute. The research evaluation specialists consisted of the staff in charge of the National Research Institute and of the central government and research evaluation specialists. This study was based on the survey results. The resulting judgment matrix using triangular fuzzy number (a) judgment matrix for main criteria is described in Table 2.

Table 2 Resulting Judgment Matrix Using the Triangular Fuzzy Number (A) Judgment Matrix for the Main Criteria
(a) Judgment matrix for the main criteria
Criteria	C1	C2	C3
C1	(1, 1, 1)	(4, 5, 6)	(0.167, 0.393, 0.500)
C2	(3,4,5)	(1, 1, 1)	(1, 1, 2)
C3	(2, 3, 4)	(0.501, 1, 1)	(1, 1, 1)
(b) Index of judgment matrix for the suitability of the establishment objective
Criteria	C11	C12	C13
C11	(1, 1, 1)	(0.500, 1, 1)	(1, 1, 2)
C12	(0.267, 0.341, 0.500)	(1, 1, 1)	(2, 3, 4)
C13	(4, 5, 6)	(3,4,5)	(1, 1, 1)
(c) Index of judgment matrix for local connectivity
Criteria	C21	C22	C23
C21	(1, 1, 1)	(1, 1, 2)	(0.213, 0.276, 0.331)
C22	(3,4,5)	(1, 1, 1)	(0.267, 0.341, 0.500)
C23	(0.500, 1, 1)	(2, 3, 4)	(1, 1, 1)
(d) Index of judgment matrix for performance excellence
Criteria	C31	C32	C33
C31	(1, 1, 1)	(0.167, 0.393, 0.500)	(0.500, 1, 1)
C32	(2, 3, 4)	(1, 1, 1)	(3,4,5)
C33	(0.267, 0.341, 0.500)	(4, 5, 6)	(1, 1, 1)

In Table 2(a), the same column addition was made for the suitability of the establishment objective (C1), local connectivity (C2) and performance excellence (C3) [(5.167, 6.393 and 7.500), (5.000, 6.000 and 8.000) and (2.501, 5.000 and 6.000), respectively]. The added whole matrix was (12.668, 17.393 and 21.500). In this result, the fuzzy synthetic extent (S₁, S₂, S₃) can be calculated.

The calculated triangular fuzzy numbers for the fuzzy synthetic extent (S₁, S₂, S₃) is shown in Figure 6. The detailed formula for the probability of the fuzzy number is as follows:

In this result, the detailed formula for the weighted value of the relative importance is as follows:

Performance excellence (C3) had a high weighted value (0.595). By order of weighted value, local connectivity (C2) had a medium weighted value (0.305) and suitability of the establishment objective (C1) had a low weighted value (0.100). The resulting weighted value for AHP is shown in Table 3.

Table 3 Resulting Weighted Value for Ahp
C1			C2			C3
0.100			0.305			0.595
C11	C12	C13	C21	C22	C23	C31	C32	C33
0.031	0.045	0.024	0.167	0.037	0.101	0.217	0.139	0.239

The analysis results of the research evaluation model showed that the research evaluation needs to be performed according to the weighted value and the order of priority for research evaluation development and efficiency improvement.

Conclusion

In this paper, the research evaluation model and the analysis results are discussed and presented for the local organization of the National Research Institute. The research evaluation of the National Research Institute focuses on the management and research performance. The National Research Institute’s research evaluation is continually evolving. The Institute’s past research evaluation focused on the whole institute or its headquarters. The demand for the local organization of the National Research Institute as increased of late due to the rising concern of the local government for regional R&D. Also, diverse opinions have been presented to the central government on the establishment and operation of the local organization of the National Research Institute. This requires the objective determination of the role and identity of the National Research Institute. The research evaluation objectives for the local organization of the National Research Institute are to determine the role and function of the institute for the establishment objective and to establish an improvement plan. The fuzzy AHP method was utilized for the research evaluation model study. The fuzzy AHP is a way of resolving the uncertainty in the decision making process. The research evaluation items were performance excellence, local connectivity and the suitability of the establishment objective. Also, the detailed research evaluation items were derived from the results of this study. The research evaluation needs to be performed according to the weighted value and the order of priority for the research evaluation development and efficiency improvement. In this paper, the research evaluation model for the local organization of the National Research Institute is suggested for the first time. It is expected that the results of this study will be very useful for the review and subsequent improvement of the country’s research evaluation policy. The future studies should develop a research evaluation model for the reflection of the characteristics of the local organization of the National Research Institute. This research evaluation model should be categorized into a research evaluation model for local connectivity or that for non-local connectivity. Also, internal and external communication should be reflected as a research evaluation item. The research evaluation model for the local organization of the National Research Institute will be further discussed and analysed in detail in the future.

References

1. Barker, K. (2007). The UK research assessment exercise: The evolution of a national research evaluation system. Research Evaluation, 16(1), 3-12.
2. Blomberg, T.G. & Waldo, G.P. (2002). Evaluation research, policy and politics. Evaluation Review, 26(3), 40-351.
3. Brown, J.H. (2006). Interviewer as instrument: Accounting for human factors in evaluation research. Evaluation Review, 30(2), 188-208.
4. Chang, D.Y. (1996). Applications of the extent analysis method on fuzzy AHP. European Journal of Operational Research, 95(3), 649-655.
5. Darema, F. (2007). Panel: Future directions in performance evaluation research. Performance Evaluation Review, 35(1), 251-260.
6. DeLeon, P.H. (1997). Evaluation research and public policy formation. The American Psychologist, 52(5), 551-561.
7. Edquist, C. (1997). Systems of Innovation: Technologies, Institutions and Organizations. London: Pinter/Cassell.
8. Fetterman, D. (2008). Evaluation practice: How to do good evaluation research in work settings by E. DePoy & G.F. Gilson. American Journal of Evaluation, 29(4), 583-601.
9. Freeman, C. (1987). Technology policy and economic performance: Lessons from Japan. London: Pinter Publishers.
10. Gonda, K. & Kakizaki, F. (1995). Research, technology and development evaluation: Developments in Japan. Akademiai Kiado, 34(3), 375-390.
11. Halachmi, A. (2004). Evaluation research and performance measurement: Dealing with issues of purpose and perspective. Public Administration and Public Policy, 107(1), 247-262.
12. Hansson, F. (2006). Organizational use of evaluations: Governance and control in research evaluation. Evaluation, 12(2), 159-178.
13. Hemlin, S. (1996). Research on research evaluation. Social Epistemology, 10(2), 209-250.
14. Hills, P.V. (1997). Research evaluation and management in the UK government. NATO Science Series. Series 5: Science and Technology Policy, 28, 127-137.
15. Imura, H. & Maeda, K. (2002). Organizing and communicating science and technology. Science and Technology in Japan, 21(81), 1-10.
16. Kobal, E. (2005). Elements of national science and technology policy. NATO Science Series. Series 5: Science and Technology Policy, 48, 13-18.
17. Kromrey, H. (2003). Evaluating and evaluation research: Concepts, models and methods. Psychologie in Erziehung und Unterricht, 50(1), 11-26.
18. Lafond, C., Toomey, T.L., Rothstein, C., Manning, W. & Wagenaar, A.C. (2000). Policy evaluation research: Measuring the independent variables. Evaluation Review, 24(1), 92-101.
19. Levin-Rozalis, M. (2003). Evaluation and research: Differences and similarities. Canadian Journal of Program Evaluation, 18(2), 1-32.
20. Midorikawa, K. (2003). Evaluation of research. Review of Laser Engineering: Laser Society of Japan, 31(9), 581-596.
21. Molas-Gallart, J. (2015). Research evaluation and the assessment of public value. Arts and Humanities in Higher Education, 14(1), 111-126.
22. Nakamura, O. (2008). Strategic evaluation of research and development in Japans public research institutes. New Directions for Evaluation, 118, 5-36.
23. Nelson, R. & Rosenberg, N. (1993). Technical innovation and national systems. In R. Nelson (Eds.), National innovation systems: A comparative analysis. New York, Oxford: Oxford University.
24. Ohnuma, T., Kawamura, M. & Iwasaki, H. (2000). Introductory article: Research evaluation in the US and Europe: Status of research evaluation system and utilization of quantitative methods. Journal of Information Processing and Management, 43(2), 138-145.
25. Ormala, E. (1989). Nordic experiences of the evaluation of technical research and development. Research Policy, 18(4), 333-342.
26. Peter, W. (2015). The ‘logic of escalation’ in performance measurement: An analysis of the dynamics of a research evaluation system. Policy and Society, 34(1), 75-85.
27. Prins, A.A. (2016). Using google scholar in research evaluation of humanities and social science programs: A comparison with web of science data. Research Evaluation, 25(3), 264-270.
28. Rogers, J.D. & Jordan, G. (2010). Introduction to a special section on new research evaluation frameworks and methods for systems level learning. Research Evaluation, 19(4), 235-238.
29. Rons, N., Bruyn, A.D. & Cornelis, J. (2008). Research evaluation per discipline: A peer-review method and its outcomes. Research Evaluation, 17(1), 45-58.
30. Ruegg, R.T. (2007). Quantitative portfolio evaluation of US federal research and development programs. Science and Public Policy, 34(10) 723-730.
31. Sample, M.L.B., Basile, C. & Cobb, R.B. (2012). An organizational model to distinguish between and integrate research and evaluation activities in a theory based evaluation. Evaluation and Program Planning, 35(4), 508-516.
32. Saunders, J. Wong, V. & Saunders, C. (2011). The research evaluation and globalization of business research. British Journal of Management, 22(3), 401-419.
33. Shi, Y.G. (2012). Research on evaluation model of enterprise network resource coordination. Advanced Materials Research, 481(3), 95-100.
34. Toda, I. (1999). Research and development evaluation at corporations. Information Science and Technology Association, 49(11), 557-562.
35. Tremper, C., Thomas, S. & Wagenaar, A.C. (2010). Measuring law for evaluation research. Evaluation Review, 34(3), 242-266.
36. Van Laarhoven, P. & Pedrycz, W. (1983). A fuzzy extension of Saaty's priority theory. Fuzzy Sets and Systems, 11, 199-227.
37. Wollmann, H. (2007). Policy evaluation and evaluation research. Public Administration and Public Policy, 16(12), 393-404.