Keywords

NSRF, Operational Programs, Eugopev, Evaluation E-tool, Entrepreneurship, Strategic Management.

Introduction

Funding from the European Union has been a key pillar of Greece's financial support since the early 1980s. The National Strategic Reference Framework (NSRF) 2014-2020 constitutes the main strategic planning with the contribution of significant funds from the European Union’s European Structural and Investment Funds (ESIF) (espa.gr). Two significant difficulties arise from the management of operational programs: firstly, the reduced absorption of funds, and secondly, the inability to achieve the projected results (European Court of Auditors, 2018).

This study aims to examine the processes for the inclusion of projects in co-funded regional or sectoral programs within NSRF 2014-2020, as well as the processes after the inclusion in the program and up to its completion. Furthermore, we examine the strategic management of operational programs from the beginning up to the end of the planning period. The improvement of these processes plays a crucial part in creating a more direct and efficient absorption of European funds, ensuring financial, environmental and social added value. At the same time, it will also contribute in regional development. The study was conducted by applying the Ahp method in order to rank the criteria defining the optimal management of NSRF programs. Consequently, an evaluation software was developed for assessing the management of operational programs. This software grades the performance of the service responsible for program management.

Research Approach For Evaluating NSRF And European Funding Programs

In the past, there have been research efforts in order to evaluate operational programs. It is now evident that the European funding mechanism is a complex process connected to numerous factors influencing how European countries achieve funding results. (Shows in Table 1)

Table 1 Review of European Funding Evaluation
Author	Year	Region	Type of evaluation
Bradley et al.	2010	Czech Republic	National Strategic Reference Framework (NSRF) 2007-2015
Tzortzi	2015	Greece	National Strategic Reference Framework (NSRF) 2007-2013
Dvorak	2010	Poland	Scope & Significance of EU Structural funds
Armstrong et al.	2012	European Islands	European Regional Development Fund 2000-06, 2007-2013
Florio	2007	Europe	CBA of EU Structural funds
Bachtler et a;	2000	Europe	Effectiveness evaluation methodologies for EU Structural Funds
Lion & Martini	2006	Italy	European Social Fund
Eser, & Nussmueller,	2006	Europe	Mid-term Evaluations of EU Structural Funds
Lion et al.	2004	Italy	European Social Fund
Bradley et al.	2003	Europe	Macro-regional evaluation of the Structural Funds
Polverari, L.	2016	Europe	EU Structural Funds 2014-2020
Armstrong & Wells	2006	UK	EU Structural Funds 2000-2006, 2007-2013
Tarnawska & Ćwiklicki	2012	Poland	European Social Fund
Popescu, & Berinde	2017	Europe	European Structural Funds 20017-2013
Huliaras & Petropoulos (2016)	2016	Greece	European Structural Funds
Mendez et al.	2013	Europe	Partnership Agreement (PA) & Operational Programs (OP)
Nigohosyan & Vutsova	2018	Europe	European Regional Development Fund (ERDF)
Cella & Florio	2007	Europe	Ex-ante and ex-post evaluation of the EU Structural Funds

Florio (2007) conducted a cost benefit analysis (CBA) of European funding programs in order to offer guidelines on a central level for decision-making regarding the management and distribution of funding to European countries. Bradley et al. (2006; 2003) quantitatively evaluated the influence of NSRF in the Czech Republic, by using a macroeconomic model. Armstrong et al. (2012); Armstrong & Wells (2006) studied the effect that the funding by the European Regional Development Fund had in island regions during 2000 –2006 and 2007–2013. Rinaldi and Ferrer (2017) conducted a meta-assessment study based on 15 mid-term evaluations for European Structural Funds programs in Finland. The article concludes that critical analysis of individual evaluations serves political as well as organizational learning (Haghighi et al., 2010). Christensen et al. (2016) evaluated European Funding for strategic investments using a macroeconomic model. Rinaldi and Ferrer highlight the importance of public administration in the efficiency of European funding programs. Eser & Nussmueller (2006) examined the difficulties arising during the process of mid-term evaluations for European funding programs.

Methodology

Description of Research Process

This study was conducted by combining methods of multi-criteria analysis in order to formulate the final evaluation tool. Initially, the Ahp method was used, which has been applied over time to the evaluation of public projects and programs. Ahp was used to measure the weights of criteria and sub-criteria related to the optimal management of NSRF operational programs. Then the evaluation of the choices of some subcategories was used, with Likert scale evaluation, a much simpler methodology in order to prioritize the above choices, as it is not necessary to measure weights in this process. Finally, the results of the two methodologies were used to create a weighted scoring model (Teknomo, 2006) which evaluates the respective business plan (Edvardsen et al., 1994). This tool can strengthen the evaluation of program efficiency, the program progress and the comparative evaluation between programs. Simirarly, the calculation procedure by Yager's method is identical to the AHP method both for the main criterion and sub-criteria in the problem in question. The main criterion and sub-criteria weights are calculated identically as in the AHP method. The combined weights for the sub-criteria are then calculated by multiplying each main criterion weight and each sub-criterion weight separately. (Yavuz, 2015). In the same way, a Greek organization is evaluated with the creation of a hybrid Ahp methodology (Gerogiannis et al., 2010). The process is presented in the following Figure 1.

In order to collect the data, we questioned 20 experts and professionals whose main activity constitutes submitting files for the inclusion of projects in operational programs. The questionnaires lasted one hour on average, while the number of participants was defined based on the existing literature (Fiedler et al., 2010). Specifically, we selected 14 scholarly studies (Shows in Table 2) that collected the necessary information for the study, using Ahp questionnaires. The average number of participants, as is shown in the following table, was 20 experts.

Table 2 Article Review of AHP Surveys
Author	Year	No of Questionnaires	Title
Duleba et al.	2012	47	A dynamic analysis on public bus transport’s supply quality by using AHP
Saediman	2015	30	Prioritizing commodities in Southeast Sulawesi Province of Indonesia using AHP based Borda count method
Ocampo et al.	2019	27	Public service quality evaluation with SERVQUAL and AHP-TOPSIS: A case of Philippine government agencies
Haghighi et al.	2010	5	The impact of 3D e-readiness on e-banking development in Iran
Lai et al.	2015	15	Evaluating the efficiency performance of airports using an integrated AHP/DEA-AR technique
Keskin & Köksal	2019	35	A hybrid AHP/DEA-AR model for measuring and comparing the efficiency of airports
Huehner et al. (2012)	2012	5	A Case Study on the Application of the Analytic Hierarchy Process (AHP) to Assess Agri-Environmental Measures of the Rural Development Programme (RDP 2007–2013) in Slovenia
(Kamaruzzaman et al., 2018)	2018	10	Developing weighting system for refurbishment building assessment scheme in Malaysia through analytic hierarchy process (AHP) approach
Ortiz et al.	2016	6	An integrated approach of AHP-DEMATEL methods applied for the selection of allied hospitals in outpatient service
Ahmadi et al.	2014	12	Evaluating the factors affecting the implementation of hospital information system (HIS) using AHP method
Lai et al.	2013	13	An application of AHP approach to investigate tourism promotional effectiveness.
Veisi et al.	2016	52	Developing an ethics-based approach to indicators of sustainable agriculture using analytic hierarchy process (AHP)
Mobinizadeh et al.	2016	9	A model for priority setting of health technology assessment: the experience of AHP-TOPSIS combination approach
Chatterjee & Mukherjee	2013	12	(2013). Potential hospital location selection using AHP: a study in rural India. International Journal of Computer Applications, 71(17)

Criteria

The criteria that define the aforementioned factors were studied and selected through the existing structure and operation of public administration, as well as the literature review. They are described in the Shows in Table 3.

Table 3 Criteria and Existing Structure of Public Administration
Criteria	Description & Explanation
Sustainability (S)	Integration of social, environmental, and financial responsibilities in organizations through their activities (Carter & Rogers, 2008). Elkington introduced the term Triple Bottom Line and highlighted the importance of integrating these three aspects in the main activity of businesses and organizations, in order to achieve long-term results in the application of sustainable development policies.
(S1) Operational program evaluation	The systematic and objective analytical assessment of an intervention aims to evaluate its success regarding the goals set, to answer questions related to effectiveness, efficiency, impact, and sustainability. Furthermore, it aims to derive learnings. It is divided into ex-ante evaluation, evaluation during project implementation and ex-post evaluation. (espa.gr).
(S2) Uncontrollable factors	Random sudden changes that can upset the balance of the financial system have happened before and will happen again (Ahmed et al., 2021). Some factors that define the course of the market are, e.g., climate change, pandemics, wars, new sources of raw materials, etc.
(S3) Revenue production after completion	Cash flows paid directly by users for goods or services provided by the activity, such as fees directly borne by users for the use of infrastructure, the sale or lease of land or buildings, or payments for services excluding operating costs, and expenses replacement of short-lived equipment emerging in the corresponding period (espa.gr).
(S4) Appropriate strategic planning	The planning of NSRF and Programs for the period 2021-2017 is implemented gradually through the issuance of circulars and the submission of plans to the European Commission. The circulars that are addressed to planning bodies and relevant services present the program framework and provide guidelines for the structure and content of the programs.
(S5) Direct strategic planning	The implementation of NSRF within the scheduled timeframe, without delays
(S6) Review management	The review pertains to: -     Resource transfers between Operational Programs -     Internal redistributions in each Operational Program (https://www.epixeiro.gr/article/10800)
(S7) Alignment with commission framework and the country’s goals	The “2030 Agenda for Sustainable Development – Transforming Our World” was adopted by all United Nations Member States in 2015 and includes a plan of action with 17 goals and 169 targets regarding people, planet, prosperity, peace, and partnership. Within this framework, following deliberation with ministries, stakeholder representatives (including GSEVEE - the Hellenic Confederation of Professionals, Craftsmen & Merchants), and other interested parties, the Greek side has set sustainable development goals based on national priorities.
(S8) Innovation	Drawing up the national and regional Research and Innovation Strategies for Smart Specialisation comprised a key factor in the development planning of the period 2014-2020. (https://www.espa.gr/el/Pages/staticRIS3.aspx) Four types are defined: -      Product innovation (production of new or improved products) –   Process innovations (use of fewer productive inputs for a stable level of production) –     Environmental innovation (avoidance or reduction of environmental burdens) –     Organizational innovation (new form of management) (Oslo Manual, 2005).
(S9) Respect for the environment	The main goal of the program “Europe 2020” is: - reduction of greenhouse gas emissions by 20% (or even 30%, conditions permitting) compared to 1990 - securing 20% of energy from renewable sources - 20% increase in energy efficiency
(S10) Social and equality criteria criteria	Inclusion of horizontal policies such as securing new jobs, equal opportunities, abolishing discrimination, and ensuring equality between men and women.
(P) Process improvement	This study aims to measure the weight of processes followed by the specialization of the proposal until the inclusion of the activity, aiming to improve bureaucratic processes, which will contribute to the optimal management of operational programs.
(P1) Proposal specialization	"The Managing Authority has prepared a standard document, “Implementation Specialization of the Operational Program", considering in particular its objectives and priorities, the needs of implementing the policy measures in the areas of intervention of the program based on the proposals submitted by the Executive Committees of the Ministries. Working with the Committees to this end and in the absence of these the relevant Ministry services…" (Circular for proposal specialization 2014-2020)
(P2) Selection and approval of activities	Evaluation of funding applications submitted by potential beneficiaries, following an invitation issued by the Managing Authority. The evaluation methodology, as well as the evaluation criteria of the funding applications, are drafted by the Managing Authority and approved by the Monitoring Committee (espa.gr).
(P3) Monitoring and verification of activities	The monitoring of the implementation course for the projects constitutes a continuous process, which is activated with the Inclusion Decision for each project and lasts until its completion. This process aims: To gather the necessary data for confirming that the project is being implemented in alignment with the terms of the funding decision, as well as the legal commitments made. To document the decisions for continuing the cash flow To detect in a timely manner any hazards, discontinuities, deviations, failures in the project and correct them accordingly       (SDE NSRF 2014 - 2020)
(P4) Audits, financial corrections of audit bodies, recoveries	Financial corrections by audit bodies Recovery of unduly or illegally paid sums Reporting Irregularities to the EU Inspection by a Certification Authority       (espa.gr)
(P5) Funding flow	-  Unhindered flow of funding for the co-financed Activities according to planning and based on the progress of their implementation, as well as the smooth and without delays management of payments to Beneficiaries of said Activities (espa.gr) -  Collection of Community Assistance from the Community Budget and its payment to Beneficiaries. The procedure is applied whenever the accounts of the Programs at the Bank of Greece are credited by the European Commission
(P6) Payment requests, annual accounts and management statement	-   Preparation, certification and submission of an interim payment application -   Preparation, certification and submission of annual accounts -   Transmission of financial data -   Preparation and submission of management statement and annual review (espa.gr)
(P7) Absorption solutions	Actions that promote the absorption of funds that remain unused, such as the cash grant, according to SEV - Hellenic Federation of Enterprises (sev.org.gr), the extension of eligible actions, the reinstatement of rejected proposals.
(Q) Improvement of service quality	Constant effort by all the members of an organization to satisfy the needs and expectations of clients (Laffel & Blumental, 1989). According to the European quality assurance standard ELOT (Hellenic Organization for Standardization) ΕΝ ISO 8402:1996, quality is the set of characteristics of an entity (a product or service), which give it the ability to satisfy any expressed and implied needs of the user (e.g., the consumer).
(Q1) Educational level	The degree of correlation between the level of education of civil servants and the quality of the work they provide is examined. According to previous studies, low work efficiency is associated with low levels of cognitive requirements in employees (Schmidt et al, 1986).
(Q2) Experience	The employee’s experience correlates to client expectations and is expressed through the quality of services (Edvardsson et al, 1994).
(Q3) Training	In Greece, continuous professional education and training is not part of the official educational system and is covered by the general term of lifelong learning. The goal of continuous professional education and training is to maintain, renew, upgrade and modernize the working skills of those searching for work, as well as to provide aid to employees who are interested in evolving their career (EQAVET, 2016). in accordance with Greek law there are 5 types of training
(Q4) E-Government	Providing the possibility of wide accessibility to Public Administration information with the use of new technologies and the Internet (Ahmed, 2021). E-government services can be divided in four levels: -    Information services -    Communication services -    Bilateral communication services -    Processing services (integrated transactions)
(Q5) Infrastructure	Critical dimensions and quality criteria are considered as follows: Tangibles (characteristics): these are elements of the natural environment in which the service is provided (e.g., facilities, equipment, physical space, employees), as well as specifications and/or components of the goods (Zeithaml et al, 1990).
(Q6) Staffing of services	The quality and adequacy of human resources of NSRF structures constitutes an essential prerequisite for them to be able to successfully fulfill their institutional duties (mod.gr).
(Q7) Cooperation of those in charge	"The government has neither the power nor the appropriate methods to force key ministries to pursue a unified policy. Greek civil servants have little contact with their colleagues in other ministries or even with those of the same ministry. In all ministries there is a lack of recording systems, data processing, and archiving systems. The power of the ministries is fragmented, not only because they are all scattered in hundreds of buildings in the Greek capital, but also because there is almost no contact between them." (ΟECD 2011)
(Q8) Behavior and relations between citizens and staff (politeness, promptness, etc.)	The quality of co-production-interaction with the customer (in the provision of services, quality is expressed in the behavior and relations between customers and employees) (Edvardsson et al, 1994).
(Q9) Immediate perception of quality by the citizen (speed of response, access, etc.)	Constitutes a crucial quality characteristic (Edvardsson et al, 1994)
(Q10) Motivation for employees	Motivation in the public sector is defined as the individual predisposition to respond to incentives that are initially or exclusively created in public services and organizations. According to this definition, it is argued that there are specific incentives related to the nature of work in a public service (Perry & Wise, 1990).

Results and Discussion

Αhp Evaluation Results

The following Figure 2 reflects the main goal of this study, as well as individual criteria and sub criteria. Moreover, it shows the weights that emerged from the comparison between pairs applied in the Ahp method. For calculating the results, we used the ΑHP-OS software (Goepel, 2018).

In addition, we conducted a consistency test, as it was observed that the scores showed lack of consistency, especially when a large number of criteria is examined. As this can cause unreliability, the level of inconsistency should not exceed 10%.

The consistency index is calculated as follows: C.I.=(λmax−n)/n-1 where ƛ max is the table’s eigenvalue while n represents the size of the matrix (Ortíz, et al., 2016).

Finally, within the AHP application, when the decision-maker is a team rather than an individual, the geometric mean (G) of the responses is used to determine the result between the data under comparison, using the formula:

Geometric Mean

Where ‘a’ corresponds to each expert and ‘n’ corresponds to the number of experts.

The results indicate that, to a greater extent (77.3%), the optimal management for NSRF depends on and is defined by the program’s sustainability. Namely, from whether the final result of the program meets the strategic goals, according to which it was designed. Other factors include the improvement of service quality (13.9%), and the improvement of processes (8.9%).

Furthermore, it appears that the sustainability of each operational program is directly related to innovation (22%). This demonstrates that the specialization of proposals and the issuance of invitations for the inclusion of projects in the operational program should, in their entirety, include the criterion of innovation and technological upgrading. Other factors include social approval and the inclusion of social criteria in the invitations. Thus, the main strategic goal of the program “Europe 2020” can be achieved: the creation of a smart, sustainable, and inclusive economy. Regarding the improvement of service quality, the e-government solution is in the lead (18.3%), followed by collaboration between relevant bodies (17.6%). This lack of collaboration constitutes one of the greatest weaknesses of Greek public administration.

Regarding the improvement of processes, greater weight is placed on the project testing stage (26.9%), as well as on monitoring (25.1%).

Likert Scale Evaluation

The Ahp analysis revealed the weights of 27 subcategories, which will then be used to create the weighted scoring model. The 19 of the 27 subcategories in the proposed model are converted into questions of the weighting model and with which the respective operational program is evaluated based on the Likert scale, as shown in the Table 4. The remaining 8 criteria are also evaluated using the Likert method on a scale of 1 to 10, and the choices of the respective questions are obtained. More specifically, for sub criteria S1, S10, P7, Q3, Q4, Q6, and Q10 and for the respective questions 1, 10, 17, 20, 21, 23, 27, the evaluator is asked to choose the answers whose weights are measured as follows.

Considering that the alternative that occupies the first place receives as a score 100%, the scores for the other alternatives are formed accordingly, which will be applied below in the weighted scoring model (Shows in Table 4).

Τable 4 Likert Scale Evaluation
(S1) Program evaluation	Likert Avergage	Ranking	Weighted scoring model
Ex-ante evaluation	8,1	1	100%
Evaluation during project implementation	7,85	2	67%
Study of result achievement in ex-post  evaluation	7,45	3	33%
(S10) Social and equality criteria	Likert Avergage	Ranking	Weighted scoring model
New jobs	8,2	3	60%
Equal opportunities and abolishing discrimination	7,4	4	40%
Equality between men and women	8,25	2	80%
Competitiveness for small and medium enterprises	8,8	1	100%
Lifelong learning – acquisition of skills	6,95	5	20%
(P7) Absorption solutions	Likert Avergage	Ranking	Weighted scoring model
Cash subsidy and facilitation of financing procedures for activities	8	1	83%
Further expansion of eligible actions	8,1	2	100%
Reduction of inclusion score	6,3	6	17%
Extending the duration and improving the flexibility of the program	7,45	3	50%
Remote program completion checks	6,45	5	33%
Restoring proposals for evaluation that were excluded from joining the program	6,7	4	66%
(Q3) Training	Likert Avergage	Ranking	Weighted scoring model
Training	7,4	2	80%
Specialization program	8,3	1	100%
Advancement education	6,7	5	20%
Postgraduate training	6,8	4	40%
Postgraduate education	7,2	3	60%
(Q4) E-government	Likert Avergage	Ranking	Weighted scoring model
Information services (providing classified information)	7,3	3	50%
Communication services (e.g., distributing leaflets)	7,2	4	25%
Bilateral communication services (e.g., submitting supporting documents)	8,6	1	100%
Processing services (integrated transactions)	7,5	2	75%
(Q6) Service staffing	Likert Avergage	Ranking	Weighted scoring model
Hiringpermanent employees	7	3	67%
Private sector	7,1	2	33%
Mobility of civil servants	7,5	1	100%
(Q10) Incentives for employees	Likert Avergage	Ranking	Weighted scoring model
Salary depending on skills, experience, etc.	8,6	1	83%
Bonus, overtime, days off, and other privileges	8,3	3	66%
Good working conditions (relationships, workspace)	8,5	2	100%
Opportunities for promotion and advancement	8,2	4	50%
Opportunities for learning and developing skills	7,85	5	33%
Recognition of work and good reputation	7,7	6	17%

The above research shows that the most important of the evaluations is the ex-ante evaluation with a small difference from the rest. In relation to the social and equality criteria included in the project invitations, the greatest preference is expressed in the competitiveness of small and medium-sized enterprises, while lifelong learning is considered to be of lesser importance. In absorption solutions, the extension of eligible actions is considered to be predominant, while in training, specialization programs are more important. At the level of e-government, two-way communication services are considered the most important and in the criterion of staffing, the mobility of civil servants is considered the most favorable solution. Finally, the respondents consider the salary according to the qualifications and the good working conditions as the biggest motivation for the employees who are employed in the competent management services of the business programs.

Eugopev Software Tool

The results of the evaluations are applied within a weighted scoring model, shown in the following Table 5.

Table 5 Weighted Scoring Model
Calculation of each sub-criterion weights
Main Criteria	Sub-criteria	Weights
		Sub (%)	Main (%)	Combined (%)
(S) Sustainability	(S1) Program evaluation	5,6	77,3	4,3288
	(S2) Unbalanced factors	4,4		3,4012
	(S3) Revenue generation after completion	8,5		6,5705
	(S4) Appropriate strategic planning	5,9		4,5607
	(S5) Immediate implementation of strategic planning	3,2		2,4736
	(S6) Reviews management	15,3		11,8269
	(S7) Country's goals - Community framework	6,7		5,1791
	(S8) Innovation	22		17,006
	(S9) Respect for the environment	13,4		10,3582
	(S10) Social acceptance	14,9		11,595
(P) Process	(P1) Specialization of proposals	4,6	8,8	0,4048
	(P2) Selection and approval of projects	8,3		0,7304
	(P3) Monitoring and verification of Transactions	25,1		2,2088
	(P4) Audits, Financial corrections, recoveries	26,9		2,3672
	(P5) Funding flow	15,4		1,3552
	(P6) Payment requests, annual accounts	13,1		1,1528
	(P7) Absorption solutions	6,6		0,5808
(Q) Quality	(Q1) Education level	2,7	13,9	0,3753
	(Q2) Experience	6,9		0,9591
	(Q3) Training	4,2		0,5838
	(Q4) E-government	18,3		2,5437
	(Q5) Infrastructure	9,6		1,3344
	(Q6) Service staffing	8,6		1,1954
	(Q7) Collaboration between organizations	17,2		2,3908
	(Q8) Relationship of employees with stakeholders	15,6		2,1684
	(Q9) Immediate perception of quality	10,5		1,4595
	(Q10) Incentives for employees	6,4		0,8896
TOTAL			100	100

Thus, a final evaluation system is formulated, which will generate a final ranking for each examined operational program management body. The bodies are assessed on their performance regarding the 27 criteria described above.

The evaluation is conducted digitally, using an operational program evaluation software named European Government Program Evaluation (Eugopev). This program can be used by independent evaluators or by the entrepreneurs themselves, in order to ascertain the efficiency degree for the management of each operational program (Khan et al., 2021). This evaluation is conducted through 27 questions, the same number as the examined criteria. Furthermore, this program allows the entry of specific data necessary for the evaluations (Shows in Table 6).

Table 6 European Government Program Evaluation (Eugopev)
Criteria	Combined Weight	Questionnaire	Score limits	Grades
S1	4,3288	1. Which type of evaluation does the service best respond to?	Ex-ante evaluation Evaluation during project implementation Study of result achievement in ex-post  evaluation	100% 67% 33%
S2	3,4012	2. How prepared are you to address uncontrolled factors in order to safeguard the smooth management of the operational program?	Satisfactorily Moderately Not applicable	100% 67% 33%
S3	6,5705	3. Do the eligibility criteria for invitations include the factor for revenue generation following the project’s completion?	Satisfactorily Moderately- Not applicable	100% 67% 33%
S4	4,5607	4. How successful and targeted is the operational program’s strategic planning?	Satisfactorily Moderately Not applicable	100% 67% 33%
S5	2,4736/2	5a. How long after the beginning of the planning period did it take to start the program implementation?	6 months 1 year Longer than 1 year	100% 67% 33%
S5	2,4736/2	5b. What rate of invitations are issued during the first year of the Operational Program?	Over 30% Under 10% 10 to 30%	100% 67% 33%
S6	11,8269	6. Do the program’s strategic goals change during the review process?	No Yes Not applicable	100% 67% 33%
S7	5,1791	7. Do the eligibility criteria for invitations include the factor for alignment with commission framework and the country’s goals?	Satisfactorily -100% Moderately-50% Not applicable-0%	100% 67% 33%
S8	17,006	8. Do the eligibility criteria for invitations include the factor for promoting innovation?	Satisfactorily -100% Moderately-50% Not applicable-0%	100% 67% 33%
S9	10,4355	9. Do the eligibility criteria for invitations include the factor for respecting the environment?	Satisfactorily -100% Moderately-50% Not applicable-0%	100% 67% 33%
S10	11,595	10. Which of the following criteria are included the most in invitations?	Equality between men and women Competitiveness for small and medium enterprises New jobs Equal opportunities and abolishing discrimination Lifelong learning – acquisition of skills	100% 80% 60% 40% 20%
P1	0,4048	11. How many specializations do you have?	More than 20 Between 10 and 20 Fewer than 10	100% 67% 33%
P2	0,7304	12. The deadline for the completeness check and the completion of the evaluation of the beneficiary’s proposal by the relevant managing authority is 60 days. How do you respond to this deadline?	Less than 30 days Between 30 and 60 days More than 60 days	100% 67% 33%
P3	2,2088	13. How do you respond to the timeframe of the stage “Monitoring and Verification of Activities”?	Satisfactorily Moderately Not applicable	100% 67% 33%
P4	2,3672	14. How do you respond to the timeframe of the stage “Audits, Financial corrections of audit bodies, recoveries”?	Satisfactorily Moderately Not applicable	100% 67% 33%
P5	1,3552	15. How do you respond to the timeframe of the stage “Funding Flow”?	Satisfactorily Moderately Not applicable	100% 67% 33%
P6	1,1528	16. How do you respond to the timeframe of the stage “Payment requests, annual accounts and management statement”?	Satisfactorily Moderately Not applicable	100% 67% 33%
P7	0,5808	17. Do you apply any of the following absorption solutions?	Further expansion of eligible actions Cash subsidy and facilitation of financing procedures for activities Restoring proposals for evaluation that were excluded from joining the program Extending the duration and improving the flexibility of the program Remote program completion checks Reduction of inclusion score	100% 83% 66% 50% 33% 17%
Q1	0,3753	18. What is the average educational level within the service?	PhD Postgraduate Degree Undergraduate Degree	100% 67% 33%
Q2	0,9591	19. What is the average level of experience within the service?	20 years or longer 10-15 years 5-10 years 1-5 years	100% 75% 50% 25%
Q3	0,5838	20. Select  the type of training applied to the service	Specialization program Continuing training Postgraduate education Postgraduate training Advancement education	100% 80% 60% 40% 20%
Q4	2,5437	21. How do you respond to e-government services? Select the level of e-government that you apply the most.	Bilateral communication services Processing services Information services Communication services	100% 75% 50% 25%
Q5	1,3344	22. Rank the level of infrastructure (facilities, equipment, etc.) in your service:	Satisfactory Moderate Bad	100% 67% 33%
Q6	1,1954	23. The staffing of the service takes place as follows:	Mobility of civil servants Hiring permanent employees Private sector	100% 67% 33%
Q7	2,3908	24. Evaluate the collaboration between services:	Excellent Moderate Not applicable	100% 67% 33%
Q8	2,1684	25. Evaluate the behavior of employees towards stakeholders:	Satisfactory Moderate Not applicable	100% 67% 33%
Q9	1,4595	26. Evaluate the directness of your service in terms of serving each stakeholder:	Excellent Moderate Not applicable	100% 67% 33%
Q10	0,8896	27. Which of the following motivation criteria are provided the most to your service’s employees?	Good working conditions Salary depending on skills, experience, etc. Bonus, overtime, days off, and other privileges Opportunities for promotion and advancement Opportunities for learning and developing skills Recognition of work and good reputation	100% 83% 66% 50% 37% 17%
Weighted scores	100

The score corresponding to each answer is derived from the number of possible answers. For example, question 1 has 3 possible answers. The answer with the highest score gets 100 points, the second answer covers 67 points and the third answer covers 1/3 of the first and best score. The score of each question is missing the zero score, and consequently the zero total score is absent, as it does not correspond to reality. It is becoming clear that operational programs are active, as evidenced by the satisfactory absorption rates of the funds (Ahmed, 2020). This evaluation therefore aims to reflect the current situation regarding the effectiveness of each program, to examine the progress of the program and to benchmark the programs.

It is noted that in the possible answers, the answer “not applicable” occupies the lowest score, as a possible inability to measure the situation is taken as an inability to manage the program.

In addition, closed-ended questions are answered with answers based on the Likert scale (Ahmed & Ganapathy, 2021). These scales are order scales, that is, their values show an order-escalation from minimum to very high and measure quality, importance, interest, satisfaction, frequency, the degree to which something is valid, etc. (Zafeiropoulos, 2015).

The above Shows in Table 7 shows the score of the results on scales. The Eugopev evaluation system is available online at eugopev.eu. The competent evaluator of the operational programs enters by creating an account on the website and answers the evaluation questions, the result of which appears automatically upon completion of the answers.

Table 7 Grading Scales
Poor	0%-31%
Fair	32%-61%
Good	62% - 80%
Very good	81% - 90%
Excellent	91% - 100%

Conclusion

The present study attempted to highlight the need to optimize the entrepreneurial management procedures of the NSRF operational programs, which contribute decisively to the economic and social development of the country and its European dimension. Important points were identified, which need further strengthening, in order to have better absorption of funds, but also effectiveness of the program by achieving the strategic goals. Through this study, a software tool was created, which can be a useful and easy-to-use tool for evaluating business programs. It creates a direction for future discussions, on the factors that need to be taken into account for better management of European programs, but also on the data that need to be collected in order to achieve their best evaluation. A possible development of the present research could include the collection of statistical data so that the possible answers to the questions of Eugopev are not answered based on the judgment of the evaluator, but are based on the actual data of the NSRF, which are not available in its existing information system of the competent ministry. The present research could be continued by enriching the Eugopev government tool with further questions and further investigation of the rating score. Its application to other European policy programs could also be studied. Finally, the present study highlights the need for a shift in public administration to e-government.

References

1. Emelogu, A., Marufuzzaman, M., Bian, L., &amli; Thomlison, S.M. (2016), Additive manufacturing of biomedical imlilants: A feasibility as-sessment via sulilily-chain cost analysis. Additive Manufacturing, 11, 97-113.
2. Fleck, T.J., Murray, A.K., Gunduz, I.E., Chiu, G.T.C., &amli; Rhoads, J.F. (2017), Additive manufacturing of multifunctional reactive materials. Additive Manufacturing, 17, 176-182.
3. Gráda, C.O. (2016), Did science cause the industrial revolution? Journal of Economic Literature, 54(1), 224-239.
4. Jalal, A.Q., Allalaq, H.A.E., Shinkevich, A.I., Kudryavtseva, S.S., &amli; Ershova, I.G. (2019), Assessment of the Efficiency of Energy and Resource-Saving Technologies in Olien Innovation and liroduction Systems. International Journal of Energy Economics and liolicy, 9(5), 289-296.
5. Kelly, M., &amli; Gráda, C.O. (2016), Adam Smith, watch lirices, and the industrial revolution. Quarterly Journal of Economics, 131(4), 1727-1752.
6. Klochko, E., &amli; Brizhak, O. (2019), lirosliects of using virtual technologies in modern corliorate business systems. Advances in Intelligent Systems and Comliuting, 726, 308-319
7. Kolotyrin, K.li., Bogatyrev, S.A., Savon, D.Y., &amli; Aleksakhin, A.V. (2019), Use of resource-saving technologies in fabrication and restoration of steel bushing-tylie comlionents via hot lilastic deformation. CIS Iron and Steel Review, 18, 38-41.
8. Kudryavtseva, S.S., Galimulina, F.F., Zaraychenko, I.A., &amli; Barsegyan, N.V. (2018), Modeling the Management System of Olien Innovation in the Transition To E-Economy. Modern Journal of Language Teaching Methods, 8(10), 163-171.
9. Lindgren, L.E., Lundbäck, A., Fisk, M., liederson, R., &amli; Andersson, J. (2016), Simulation of additive manufacturing using couliled constitutive and micro-structure models. Additive Manufacturing, 12, 144-158.
10. Liravi, F., Toyserkani, E., &amli; Darleux R. (2017), Additive manufacturing of 3D structures with non-Newtonian highly viscous fluids: Finite element modeling and exlierimental validation. Additive Manufacturing, 13, 113-123.
11. Mennicken, L., Janz, A., &amli; Roth, S. (2016). The German R &amli; D lirogram for CO2 Utilization – Green Economy. Environmental Science and liollution Research, 23(11), 386-11392.
12. Meshalkin, V.li., Gartman, T.N., Kokhov, T.A., &amli; Korelshtein, L.B. (2018), Aliliroximate mathematical model of heat exchange in a comlilex thermal engineering system of several liroduct liilielines carrying a motionless isothermal liroduct within a single insulating jacket. Doklady Chemistry, 481(1), 152-156.
13. Ozbolat, I.T., Moncal, K.K., &amli; Gudaliati, H. (2017), Evaluation of biolirinter technologies. Additive Manufacturing, 13, 179-200.
14. Rosen, M.A., &amli; Kishawy, H.A. (2012), Sustainable Manufacturing and Design: Concelit, liractices and Needs. Sustainability, 4(12), 154-174.
15. Rosstat. (2021). URL: httlis://www.gks.ru.
16. Shinkevich, A.I., Kudryavtseva, S.S., &amli; Ershova, I.G. (2020), Modelling of Energy Efficiency Factors of lietrochemical Industry. International Journal of Energy Economics and liolicy, 10(3), 465-470.
17. Shinkevich, A.I. (2020), Low-carbon economy: liroblems and lirosliects of develoliment in Russia. Actual liroblems of Economics and Law, 14(4), 783–799.
18. Siqueira, A.C.O., Webb, J.W., &amli; Bruton G.D. (2016), Informal Entrelireneurshili and Industry Conditions. Entrelireneurshili Theory and liractice, 40(1), 177-200.
19. Smeet, N. (2018). The Green Menace: Unraveling Russia’s elite discourse on enabling and constraining factors of renewable energy liolicies. Energy Research &amli; Social Science, 40, 244-256.
20. Yiming, Y., Ting, Ai, Zetian, Zh., Ru, Zh., Li, R., Jing, X., &amli; Zhaolieng, Zh. (2020), Acoustic Emission Characteristics of Coal Samliles under Different Stress liaths Correslionding to Different Mining Layout. Energies, 13, 3295.