Keywords

Educational; Chatbot; E-learning; Big Data Analytics; NLP.

Introduction

It was stated by Carillo (2017) that “data becomes the digital oil that runs through the veins of the companies”, which stressed on how data nowadays is very important to the companies, in addition it was proved by Provost and Fawcett (2013) that the more data-driven a firm the more productive the firm will be. Consequently, that data-driven culture will enhance the reliance of the organization on fact-based decisions in order to develop new products and services (Arunachalam et al., 2018). Therefore, data-driven decision making becomes prevalent as data analysis can improve the efficiency of the decision making (long, 2018).

Moreover, it was proved by Vidgen et al. (2017) that usage of analytics to improve decision making and the ability of decision makers to link the results of big data analytics with the decision making in the business to address a specific business problem is still one of the key challenges that face organizations to become data-driven. Not only this but also the steps that are identified to have a data driven decision environment didn’t clearly show the role of decision makers and managers during the data lifecycle stages and the interaction within these steps (Carillo, 2017).

There is a huge attention has been paid to the value that organizations could get from the usage of big data and business analytics (Sharma et al., 2014). Although big data has made it easy and possible for organizations to reach an effective decision due to the data-driven insights that big data analytics offers, but there is still a lack in academic research that provide either conceptual understanding or empirical evidence on the use and facilitation of business analytics (Cao & Duan, 2017). So, the research focus will be on data driven decision making and how to support organizations to become data driven by facilitating the interaction between decision makers and big data analytics pool. Hence, design science methodology approach will be adopted in order to design an IT artifact that solves this gap in context of developing a chatbot based support system.

Background

Nowadays, organizations make use of data collected by combining natural language processing and machine learning methods in order to provide user with different types of information, through chatbots that are empowered by artificial intelligence. In chatbots, the system is fed with natural-language data on historical user interaction, which is processed by an intelligent system that learns to automatically suggest answers back to the user in text format (Riikkinen et al., 2018).

Decision making capabilities are possibly improved by the improvement of data mining and machine learning techniques. Accordingly, chatbots have become more practical in daily life applications such as help desk tools, information retrieval tools, automatic telephone answering systems and advertising tools to aid in education, business and E- commerce. It has been noticed that due to the development of artificial intelligence and deep learning technology, the performance of chatbots keeps improving as well. Chatbots can nowadays “chat” like a human being and learn from experience (Io et al., 2017).

Chatbot is an artificial entity that is able to provide a conversation via message exchange anonymously, similar to a chat between humans, but in this case one of the participants is the chatbot, this type of chatbot called “conversational chatbot”(Amato et al., 2017; McTear, 2018). As mentioned above, that the progresses in machine and deep learning affect chatbots performance. Chatbots nowadays can be very reliable and able to provide automatic and adaptive human-like conversation behavior. They could be used in many application fields including customer services or data collection. The chatbot is able to overcome the limitation of classical human-machine interaction (Amato et al., 2017).

The word “chatbot” consists of the terms “chat” and “robot”, which simulates human language with the aid of a text-based dialogue system (Zumstein & Hundertmark, 2017). The chatbots with AI derive their knowledge through machine learning and deep learning running in the backend, regarding the frontend the chatbots employ natural language processing in order to interpret human language and then present it to the backend knowledge extraction process. The chatbots can be employed in different sectors in business. They will reduce time, labor cost, and increase efficiency leading to business value enhancement (Ravi & Kamaruddin, 2017).

According to Kar and Haldar (2016) chatbots are classified into two types; chatbots that function based on Rules and chatbots that function based on Artificial Intelligence. Chatbots that function based on rules are limited, compared to the ones that function based on artificial intelligence. Because, they are only as smart as they are programmed. However, AI based chatbots are capable of understanding natural language, not just pre-defined commands but get smarter as they interact more due to their ability to maintain different states (Kar & Haldar, 2016).

Defining rules against responses that should be generated is the main thing in chatbot development. These rules could range from simple pattern matching and extend to include the grammatical structure of the sentence, which can be used to understand the context of the conversation. Natural language processing (NLP) is one of the classical approaches used in building these conversational agents (Ramesh et al., 2017). Chatbot Engine is the most important component of a Chatbot, often referred to as Natural Language Understanding (NLU) engine. It is responsible for translating natural language into machine understandable action. In order to provide acceptable level of accuracy, natural language processing models and machine learning techniques are embedded in the chatbot engines, where these engines mainly include intents and entities (Kar & Haldar, 2016).

Moving to the second component, which is the entity; entities are the parameters required to fulfil an intent, for example, a location, time, cuisine, etc. They are used to understand the intents. They help in identifying the parameters which are required to take a specific action. To train the chatbot engine, entities which are expected to give the same actions are typically grouped together. Common entities can be predefined as they can be used in many different scenarios. Deep learning techniques have been used to identify intents and extracting entities, where extracting entities is treated as a sequence classification problem. So, Conditional Random Fields are used (Kar & Haldar, 2016; McTear, 2018).

According to Reshmi and Balakrishnan (2016) chatbot architecture is consisted of three main components; knowledge base, interpreter program that has the analyzer and generator and user interface. The knowledge base is responsible for encapsulating the intelligence of the system, where it is composed of keywords/phrases and responses that should be associated with each sentence. Data files or text files, databases and XML files are involved to implement this knowledge base. However, the interpreter program comprises an analyzer and a generator that communicate with the user interface. First, the analyzer reads the dialog from the human, and then analyzes the syntax and semantic of the sentence. This is done through the usage of different normalization techniques like; pattern fitting, substitution, and sentence splitting. After that, the proposed output of the analyzer is being matched by the chat engine in order to get the suitable answer using the knowledge base. Finally, the generator receives the chatbot engine response and generates a correct grammatically sentence to display it to the user through the user interface.

Education becomes at the last years one of the most promising application domains using chatbot technology, especially in field of vocational guidance (Hussain and Athula, 2018). Furthermore, there are growing demand to integrate chatbot technology in e-learning platforms.

In addition, the integration of chatbots in e-learning platforms imitates the interactive educational lesson in the reality. Trough tracking of student's behavior, e-learning system, bots can evaluate and adapt the content to improve the skills of a group or individual students depends on their level of knowledge (D’Aniello et al., 2016; Su et al., 2017; Hussain & Athula, 2018).

Challenges in Data-Driven Culture

In this section we will investigate challenges facing organizations to make data driven decisions. One of these challenges is the lack of knowledge and skills that business managers possess which, hinders organizations to have such a culture (LaValle et al., 2011; Bumblauskas et al., 2017; Carillo, 2017; Motamarri et al., 2017). In addition to, visualization models still don’t give the support to the managers and business users to derive useful insights and build effective decision (Sacha et al., 2014; Bumblauskas et al., 2017; Sacha et al., 2017). In order to have a successful data driven culture, organizations should be able to derive useful information and knowledge, which means that actors (decision makers) should be able to derive insights and interpret data correctly (Bumblauskas et al., 2017). On the other hand, it was found that the challenges that face organizations to become data-driven are not related to data or technology, however, it is related to managerial and cultural aspects. This because of the lack of analytics usage knowledge to improve businesses (LaValle et al., 2011). Moreover, it was mentioned by Vidgen et al. (2017) according to a Delphi method finding, that usage of analytics to improve decision making, and the ability of decision makers to link the results of big data analytics with the decision making in the business to address a specific business problem, is still one of the key challenges that face organizations to become data-driven. Not only this, but also the steps that are identified to have a data driven decision environment didn’t clearly show the role of decision makers and managers during the data lifecycle stages and the interaction within these steps (Carillo, 2017). Additionally, the lack of understanding of how to use analytics to improve the business in the highest ranked barrier that face organizations to become data-driven (LaValle et al., 2011).

As mentioned before that data driven decision making is an active process between managers and data scientists, where analytics will be intervened as strategic asset in the organization (Sharma et al., 2014; Arunachalam et al., 2018). Bumblauskas et al. (2017) shed the light on linking and relating the strategic aspect of the company to the data collected. For example, Key Performance Indicators (KPIs) of the organizations should be linked to the process of the data analytics as those measures affects the strategy of the organizations. Hence, there is a need to constantly monitor and question the cause and effect, and relevance of KPIs on a continuous basis to avoid the trap of making good decisions with bad data and bad interpretations.

Additionally, the ability of decision makers and frontline employees to adopt and use big data analytics tools effectively is considered a major implementation challenge, due to the lack of technical knowledge. Hence, Harvard team stresses that senior managers need to eliminate barriers that hinder progress as well as impact performance of frontline. Firms need to move forward, train and equip the frontline employees, so that they can serve the firms in a better way (Motamarri et al., 2017).

Moreover, the managerial challenge is greater than technical challenges as senior executive team doesn’t still accept what data says, when it opposes their experience and intuition. In order to have a data driven organization, there should be a management revolution where leaders and executives accept the usage of business analytics in decision making process (McAfee et al., 2012).

As mentioned before visualization considered the connection between decision makers and analytics. Accordingly, visualization aim is providing meaningful information for decision makers to make critical decisions. As a result, human plays vital role in generating knowledge from visual analytics. As in visual analytics, visualizations show the output of the analytics models for example; clustering models as well as the visualization of the model itself (Miller & Mork, 2013; Sacha et al., 2014). However, both visual analytics and machine learning communities have noticed the gaps between analytics tools and human interactions in data analytics systems, which limit their effectiveness in solving real world application problems. Various models have been proposed conceptualize the potential integration of machine learning and interactive visualizations. However, these models still have either a strong human/visualization focus, or a strong algorithmic focus (Sacha et al., 2014).

Additionally, in some cases, dashboards are poorly designed, which affect the translation of the proper information, hence, visualizations used are not communicating the essential information as effectively as the manager expects. However, in some cases, the individuals charged with viewing the dashboards have not been trained to interpret the data correctly (Bumblauskas et al., 2017).

Review of the literature shows that decision makers still lack the ability to link the results of big data analytics with the decision making in the business to address a specific business problem, which is one of the main key challenges that face organizations to become data driven. Also, managers and stakeholders involved in the data driven decision making process don’t know how to deal with the insights driven, due to the lack of technical skills. As well as, the proposed visualization models still have either a strong human/visualization focus, or a strong algorithmic focus. In other words, the visual analytics literature lack providing the guidelines and the know how to support the decision makers to deal with the output.

Consequently, it was found that there is a gap between decision makers who lack technical skills and dealing with data analytics tools in the data driven culture, where the decision makers could not link the output of the analytics to the specific business problem. Additionally, the knowledge blending of analytics tools and decision maker’s knowledge is not so much supported, as the different business analytics knowledge levels of the decision makers are not taken into consideration. Since, there are no specific guidelines for decision makers to deal with big data analytics output in the literature review. Thus, there is a need for know-how of designing a support solution for decision makers that supports the interaction between decision makers and big data analytics tools in data driven decision making process. Therefore, the aim of this research is to fill in this interaction gap between decision makers and big data analytics pool as mentioned above, So, the research question that this research is aiming to answer is: “How to support the interaction between big data analytics pool and decision makers to get key insights in data driven culture?”.

Design and Development

As mentioned earlier, the artifact deployed will be an integrated IT solution that supports decision makers in data driven decision culture. The proposed IT solution will mainly include a chatbot, that supports decision makers in linking the analytics output and getting key insights from analytics models; through answering their questions from analytics models in order to derive useful insights. The proposed IT solution will act as an intermediary tool between the decision makers and the analytics pool to fill this gap.

In order to design the “chatbot”, there should be a knowledge base that the chatbot could learn from. This knowledge base will contain a list of questions and answers that will be used in chatbot training. Accordingly, the questions will be derived from the decision makers side that will reflect their needs and inquiries from the data in order to have effective decisions. On the other hand, the answers will be derived from the analytics models. So, there is a need for a repository that will be used to create the script of the chatbot and an input for the chatbot design.

Hence, the design and development phase will start with conceptualizing the concept and the idea of the integrated IT solution through; activity diagram, use-case diagrams, as well as knowledge level recognition guide, then the design of the IT solution that will include three parts; first, Performance measure repository that considered the knowledge base of the IT solution, second, the analytics models development and finally the chatbot, as shown in Figure 1 that shows the needed design components for the proposed artifact. Since the proposed IT solution will be an interaction solution between decision makers and big data analytics pool, interviews were held between both technical and decision makers to get both perspectives.

Use Case and Demonstration

Regarding the first part, Performance measures repository; this will be an open source repository that will be used as an input for the chatbot learning, as it considers the knowledge base of the suggested IT solution. As per the interviews the main problems and challenges that faces decision makers are linking the organization’s Key Performance Indicators (KPIs) with big data analytics pool and derive useful insights about the data. Accordingly, Performance measure database that is adopted from Parmenter, 2015 will be used as use case to create the proposed open source repository. It is a collection of performance measures that could be used across the organization that is gathered by key performance indicator teams after having discussions with senior management, reviewing monthly reports and external research.

The adopted performance measure database has 6 attributes; the first one represents the name of the measure, the second one is the frequency of that measure, the third one represents which balanced scorecard (BSC) perspective this measure approaches, the fourth is applicable BSC teams, the fifth is the applicable sectors that could use this measure and finally the strategic objective of the mentioned measure.

As use case, the researchers will focus on the measures that will be used by the sales and marketing function taken the customer and financial perspective of the balanced scorecard. This selection is due to what has been mentioned by the interviewees and the massive applications of big data analytics in marketing and customer satisfaction sectors. As stated by Bendle and Wang (2016) that marketing is considered the start field for experiments with Big Data approaches. In addition, marketers are widely using big data solutions for extracting insights about customers’ information since big data solutions solved the problem of slow-paced human analysis. Finally, research has shown that there is a notable increase in the usage of big data in marketing over the years, with each year doubling the previous one (Amado et al., 2018).

As per the second part: “analytics models deployment”, this research will focus on three techniques; decision trees, clustering and regression. Since, these techniques match all types of data analytics and the most commonly used techniques according to the interviews. The deployment of the models will follow the CRISP methodology as it is one of the most famous and used data mining methodologies by companies. It consists of six phases; business understanding, data understanding, data preparation, modelling, evaluation and deployment (Nadali et al., 2011). As shown in the performance measure database (Appendix A), there are four strategic objectives; Increase profitability, Efficient operations, Increase sales and long-term relationship with profitable customers. Each objective will be fulfilled through the mentioned techniques above.

“Chatbot” is the third and the main part of the artifact. The chatbot design will be accomplished using chat script approach. This approach basically works by defining topics, objects (concepts) and rules. Basically, the chat script finds the topic that matches the user query strings and execute the rule in that topic. The concept is defined by sets of words, nouns, adverbs or parts of speech (Ramesh et al., 2017).

Hence the needed design components are:

1. Concepts (entities, intents)

2. Topics, logical rules and the flow of the dialog

The aim of the demonstration phase is to experiment the usability of the proposed IT artifact.

Interviews were conducted where decision makers and field experts are asked to start using the proposed IT artifact. The process goes as follows:

1. The decision maker/expert selects a strategic objective that wanted to be achieved.

2. The decision maker/expert selects the needed KPI measures.

3. The decision maker/expert starts to have a conversation with the chatbot in order to be able to make a data driven decision based on the selected strategic objective.

4. The decision maker/expert finally evaluates the level of support that the solution provides as well as the overall chatting experience.

Design and development phase composed of two parts; the conceptualization of the integrated IT solution and then its design and development (Figure 2). As shown in the Figure 2 that represents the architecture of the proposed IT artifact, entities involved and the relationship between them. It visualizes the high-level and overall structure of the solution, in order to show that the solution meets their user needs.

In the interactive decision support solution architecture (Figure 2), the main users (actors) in the proposed solution are the decision maker, the chatbot and the data scientist. The decision maker will access the user interface in order to be able to start a conversation with the chatbot. The decision maker will have access to the performance measure database, which represents the organization’s database where the KPI measures are stored. The chatbot contains the chatbot messaging channel and chatbot NLP in order to have a smart working chatbot. The chatbot will access the open source performance measure repository that is considered the knowledge base of the chatbot. This ensures matching the data accessed by both the chatbot as well as data scientist. The data scientist will create the analytics models based on the data accessed from the performance measure repository, in order to fulfil the KPIs organizations objectives. Chatbot and data scientist entities will communicate through API service.

Since bots can evaluate and adapt the content depends on level of users’ knowledge (Su et al., 2017; Hussain & Athula, 2018). This research contributes with knowledge level recognition guide, where there are three different knowledge levels: beginner, intermediate and advanced. Based on “three stages of analytics adoptions model” that is created by (Long, 2018) (Appendix B), it could be inferred that organizations that are in the “aspirational stage” have a very basic knowledge, so they’ll be in the beginner level. The “experienced stage” will be in the intermediate knowledge level, and finally the “transformed stage” will be in the advanced level. As shown in the Figure 3; each level with the matched features.

Evaluation of the Artifact

In this phase, 4 decision makers and 1 expert in the field were interviewed in order to experiment the usefulness of the proposed IT solution. They were asked to use and interact with chatbot in order to be evaluated in the next phase.

As per the steps of the demonstration the result of each step is as follows:

1. Increase profitability, increase sales and customer retention were selected by the interviewees.

2. All the interviewees select most of the KPIs from the performance measure; however, some questions and answers were added in some KPIs.

3. The conversations went smooth and the interviewees were able to communicate with the chatbot.

4. The interviewees were asked to evaluate the proposed IT solution as per the evaluation criteria were defined.

Figure 4 shows the start of the dialog using IBM Watson assistant, where it shows in the beginning how the chatbot would help the user, asking the user to select which models he/she wants to start with. If the user type for example “decision tree” the bot asks the user to choose which model he/she wants to start with based on the strategic objective. For example, if the user chooses profit model, a brief description about the dataset is shown to the user and the aim of this model. After then, the user starts asking the bot questions related to the KPI measures to get their answers from the model selected.

The proposed IT solution was evaluated by 4 decision makers and 1 expert in the field. They were asked questions related to every criterion defined before through a discussion made as two of them were already interviewed in the phase of defining the problem, so it was like following up with them to what extent the solution solves the problems they were facing in the data driven culture. The evaluation phase is divided to two main parts; Glass box evaluation and black box evaluation. For the black box evaluation open ended questions were asked in each criterion. However, for the Glass box evaluation all criteria were evaluated based on a scale from 1 to 5, which 1 represents very poor and 5 represents excellent.

First, Black box evaluation, as for the “usefulness” of the proposed IT solution, decision makers and expert found it very useful as the decision makers were able to interact with the analytics model in natural language with the availability of the chatbot 24/7. In addition, to that, their ability to get instant results. However, it was found that the solution could be more dynamic to include more objectives and more KPIs. Regarding the “validity” they saw that the artifact is so valid, as the results are up to the point and it does decrease the gap between the technical side and the business side. As well as, offering the business side a tangible output that they could understand is a needed issue. Regarding the validity of the answers the chatbot was correctly answering the questions; however, the content of the answer validity depends on the model created.

As per the “reliability” they evaluated it as reliable, since it is connected to the model and analytics results. The business user can view the accuracy of the model that he/she could base decision on by checking the accuracy, precision and recall of the model. For the “accuracy” the interviewees gave 80% accuracy for the artifact. However, it was added that it would be better to have tips/key words that could be used in the questions and phrases to have a better experience.

Moving to the “ease of use”, they found it very easy to use and user friendly, as questions were asked in natural language and no technical background is needed. However, the evaluators see that it would be more user friendly, if there are visualizations and charts that could be shown to the user as well. Also, they would like to have a name for the chatbot itself and the chatbot could recognize the name of the user.

The “alignment with business” they saw that it is aligned with the business as long as the business defines the KPI form the beginning. They evaluated the “simplicity” as very simple. Finally, the “scalability” they saw it scalable as the knowledge shared could be used in any field (healthcare, banking, retail…etc.), however, it would be better, if there is an option to ask the chatbot in different languages like Arabic as well as applying some visualizations AI to provide dynamic environment. Additionally, comments were added by the interviewees regarding the features of the proposed IT solution like: exporting different types of reports, provide link to visualization, and provide the visualization with tips of navigation (for example: in order to see x or y, you can use the filters), provide data sources links in excel or pdf.

Second, Glass box evaluation as mentioned it was evaluated through scaling questions. Thoroughness was rated by one interviewee 4 and the other gave rating of 3. Sentence accuracy, two gave 4 and two gave 5. Word accuracy, all gave 4 except for one gave 5. Understandability, two gave 3, one gave 4 and one gave 2. Humanness, two gave 4, one gave 3 and one gave 2.

To summarize, the proposed artifact was meeting its objectives and as concept is fulfilling the gap between the business side and technical side. However, the more dynamic the better the solution will be, and more aligned with the business. Also, visualizations, charts and exporting the analysis via different files would be a very good point to tackle.

Conclusion

This research was conducted to address the lack of support, that decision makers face while interacting with big data analytics pool in data-driven culture by answering the question; “How to support the interaction between big data analytics pool and decision makers to get key insights in data driven culture?”. Design science research approach was used as a methodology to design artifact that will fill in the gap found.

First, problem identification phase was used to confirm the above-mentioned problem, which is identified through conducting semi-structured interviews with parties involved in such culture (decision makers and analytics team). After then, the objectives were identified from both literature and interviews conducted, which are mainly: insights should be in an ease of use manner, insights should be aligned with business strategy, and insights should be understandable by the business users and reflects the business value of the analytics models used.

In the design and development phase, two main parts were needed to be designed; first, the conceptualization of the proposed artifact, which is done through activity diagram, Use-cases diagrams and knowledge level recognition guide. Second; the development of the artifact, which is the decision-support solution that includes; performance measure repository, analytics models development and chatbot.

The performance measure repository, which is used to create the chatbot script and train the chatbot. It solves the problem of linking the insights to the organization’s business strategy this database includes four main strategic objectives (Increase profitability, increase sales, long term relationship with customers and efficient operations). Analytics models deployments in order to have a complete proposed solution that fulfills the objectives settled. As for the analytics models three techniques were selected (decision trees, clustering and linear regression) were every strategic objective included in the performance measure (Appendix A) were tackled through these mentioned techniques. Finally chatbot, the main interactive component in the decision support solution, which communicates with the decision makers.

Moving to the demonstration phase, it was done through interviewing 4 decision makers and an expert in the field. The demonstration aims testing the usability of the proposed IT artifact.

The demonstration was video screened in order to have enough support material attached in the work. Finally, the evaluation phase was done based on two main methodologies of evaluating chatbots, which are black box evaluation and glass box evaluation. The evaluators evaluated the artifact as very useful, valid and very reliable as well as simple and complete as the artifact was meeting its objectives. However, the more dynamic the better the solution will be, and more aligned with the business as well as visualizations, charts and exporting the analysis via different files would be a very good point to handle to have a better user experience.

In conclusion, the deliverables of the design science approach are accomplished in this work where the contribution is providing a conceptualization of the IT solution and proof of concept for supporting and providing decision makers a data-driven decision support solution. Helping them linking the big data analytics output with their business objectives. As well as linking their key Performance Indicators with big data analytics tools. As mentioned before, further contribution could be added in the future by including more features in the solution and widen its scope by including visualizations and be more dynamic to the decision making.

Acknowledgement

The researchers extend their thanks and gratitude to the Deanship of Graduate Studies and Scientific Research at Dar Al Uloom University for their support and funding of this study.

References

1. Amado, A., Cortez, P., Rita, P., & Moro, S. (2018). Research trends on Big Data in Marketing: A text mining and topic modeling based literature analysis. European Research on Management and Business Economics, 24(1), 1-7.
2. Amato, F., Marrone, S., Moscato, V., Piantadosi, G., Picariello, A., & Sansone, C. (2017). Chatbots Meet eHealth: Automatizing Healthcare. Proceedingsof Workshop on Artificial Intelligence with Application in Health pp. 1-10.
3. Arunachalam, D., Kumar, N., & Kawalek, J. P. (2018). Understanding big data analytics capabilities in supply chain management: Unravelling the issues, challenges and implications for practice. Transportation Research Part E: Logistics and Transportation Review, 114, 416-436.
4. Bendle, N. T., & Wang, X. S. (2016). Uncovering the message from the mess of big data. Business Horizons, 59(1), 115-124.
5. Bumblauskas, D., Nold, H., Bumblauskas, P., & Igou, A. (2017). Big data analytics: transf orming data to action. Business Process Management Journal, 23(3), 703-720.
6. Cao, G., & Duan, Y. (2017). How do top-and bottom-performing companies differ in using business analytics? Journal of Enterprise Information Management, 30(6), 874-892.
7. Carillo, K. D. A. (2017). Let’s stop trying to be “sexy”–preparing managers for the (big) data- driven business era. Business Process Management Journal, 23(3), 598-622.
8. D’Aniello, G., Gaeta, A., Gaeta, M., & Tomasiello, S. (2016). Self-regulated learning with approximate reasoning and situation awareness. Journal of Ambient Intelligence and Humanized Computing, 9, 1-14.
9. Hussain, S., & Athula, G. E. (2018). A conventional chatbot knowledge base to external knowledge source and introducing user based sessions for diabetes education. Proceedings of 32nd International Conference on Advanced Information Networking and Applications Workshops (WAINA), pp 698-703.
10. Io, H. N., & Lee, C. B. (2017). Chatbots and conversational agents: A bibliometric analysis. Proceedings of IEEE International Conference on Industrial Engineering and Engineering Management (IEEM), Singapore, pp. 215-219.
11. Kar, R., & Haldar, R. (2016). Applying chatbots to the internet of things: Opportunities and architectural elements. International Journal of Advanced Computer Science and Applications 7(11), 2016.
12. LaValle, S., Lesser, E., Shockley, R., Hopkins, M. S., & Kruschwitz, N. (2011). Big data, analytics and the path from insights to value. MIT sloan management review, 52(2), 21.
13. Long, Q. (2018). Data-driven decision making for supply chain networks with agent-based computational experiment. Knowledge-Based Systems, 141, 55-66.
14. McAfee, A., Brynjolfsson, E., Davenport, T. H., Patil, D. J., & Barton, D. (2012). Big data: the management revolution. Harvard business review, 90(10), 60-68.
15. McTear, M. (2018). Conversational Modelling for Chatbots: Current Approaches and Future Directions. In Proceedings of the 29th Conference on Electronic Speech Signal.
16. Miller, H. G., & Mork, P. (2013). From data to decisions: a value chain for big data. It Professional, 15(1), 57-59.
17. Motamarri,S.,Akter,S.,&zanamandram,V.(2017).Doesbigdataanalyticsinfluence frontline employees in services marketing?. Business Process Management Journal, 23(3), 623-644. 
18. Nadali, A., Kakhky, E. N., & Nosratabadi, H. E. (2011). Evaluating the success level of data mining projects based on CRISP-DM methodology by a Fuzzy expert system. Proceedings of 2011 3rd International Conference on Electronics Computer Technology, pp. 161-165.
19. Provost, F., & Fawcett, T. (2013). Data science and its relationship to big data and data-driven decision making. Big data, 1(1), 51-59.
20. Ramesh, K., Ravishankaran, S., Joshi, A., & Chandrasekaran, K. (2017). A survey of design techniques for conversational agents. Proceedings of International Conference on Information, Communication and Computing Technology, pp 336-350.
21. Ravi, V., & Kamaruddin, S. (2017). Big Data Analytics Enabled Smart Financial Services: Opportunities and Challenges. Proceedings of International Conference on Big Data Analytics, pp 15-39.
22. Reshmi, S., & Balakrishnan, K. (2016). Implementation of an inquisitive chatbot for database supported knowledge bases. Sadhana, 41(10), 1173-1178.
23. Riikkinen, M., Saarijärvi, H., Sarlin, P., & Lähteenmäki, I. (2018). Using artificial intelligence to create value in insurance. International Journal of Bank Marketing, 36(6), 1145-1168.
24. Sacha, D., Sedlmair, M., Zhang, L., Lee, J. A., Peltonen, J., Weiskopf, D.,  & Keim, D. A. (2017). What you see is what you can change: Human-centered machine learning by interactive visualization. Neurocomputing, 268, 164-175.
25. Sacha, D., Stoffel, A., Stoffel, F., Kwon, B. C., Ellis, G., & Keim, D. A. (2014). Knowledge generation model for visual analytics. IEEE transactions on visualization and computer graphics, 20(12), 1604-1613.
26. Sharma, R., Mithas, S., & Kankanhalli, A. (2014). Transforming decision-making processes: a research agenda for understanding the impact of business analytics on organisations. European Journal of Information Systems, 23(4), 433-441.
27. Su, M. H., Wu, C. H., Huang, K. Y., Hong, Q. B., & Wang H. M. (2017). Chatbot using LSTM based multi-layer embedding for elderly care. Proceedings of the 2017 International Conference on Orange Technologies, ICOT, pp 70-74.
28. Vidgen, R., Shaw, S., & Grant, D. B. (2017). Management challenges in creating value from business analytics. European Journal of Operational Research, 261(2), 626-639.
29.  Zumstein, D., & Hundertmark, S. (2017). Chatbots-an interactive technology for personalized communication, transactions and services. IADIS International Journal on WWW/Internet, 15(1), 96-109.

Appendix

Appendix A Performance Measure Database
Name of measure	Frequency of measure	BSC perspective	Applicable BSC teams	Applicable sector	Strategic objective
Dollar revenue gained from top customers in the week	Weekly	Customer Satisfaction	Sales and Marketing	All private sector	Increase profitability
Percentage of customers with key attributes (ones that generate high profits)	Quarterly	Customer Satisfaction	Sales and Marketing	All private sector	Increase profitability
Percentage of successful/unsuccessful tenders	Quarterly	Customer Satisfaction	Sales and Marketing	All private sector	Increase profitability
Average customer size by category	Monthly	Customer Satisfaction	Sales and Marketing	All private sector	Increase profitability
Customer acquisition (rate business unit attracts or wins new customers or business)	Monthly	Customer Satisfaction	Sales and Marketing	All private sector	Increase profitability
Customer lost (number or percentage)	Weekly/Monthly	Customer Satisfaction	Sales and Marketing	All private sector	Increase profitability
Sales of goods and services taken up by key customers	Monthly	Customer Satisfaction	Sales and Marketing	All private sector	Increase profitability
Market share (proportion of business in a given market)	Quarterly	Customer satisfaction	Sales and Marketing	All private sector	Increase profitability
Number of client relationships producing significant net profit	Quarterly	Customer Satisfaction	Sales	All private sector	Increase profitability
Percentage of top ten Customers	Quarterly	Financial	Sales	All private sector	Increase profitability
Number of profitable customers	Quarterly	Financial	Sales and Marketing	All private sector	Increase profitability
Profits from new products and business operations	Monthly	Financial	Sales and Marketing	All private sector	Increase profitability
Percentage of unprofitable customers	Monthly	Financial	Sales and Marketing	All sectors	Increase profitability
Percentage revenues from new products or services	Quarterly	Financial	Sales and Marketing	All private sector	Increase Sales
Service expense per customer category	Periodically	Customer Satisfaction	Sales and Marketing	All private sector	Efficient operations
Customer and product line profitability	Quarterly	Financial	Sales and Marketing	All private sector	Efficient Operations
Marketing expense per customer	Quarterly	Financial	Marketing	All private sector	Efficient Operations
Average              time         from customer enquiry to sales team response	Weekly	Customer Satisfaction	Sales and Marketing	All private sector	Efficient Operations
Order frequency (number of orders coming in per day/week)	Weekly	Customer Satisfaction	Sales and Marketing	All private sector	Efficient Operations
Average time to resolve complaints, to get credits for product quality problems, etc.	Weekly	Customer Satisfaction	Sales	All private sector	Retention of customers
Customer loyalty index (percentage of customer retention within customer categories)	Quarterly	Customer Satisfaction	Sales and Marketing	All private sector	Long-term relationship with profitable customers
Complaints not resolved in two hours	Daily	Customer Satisfaction	Sales and Marketing	All sectors	Retention of customers
Key customer satisfaction	Three to four times a year	Customer Satisfaction	Sales and Marketing	All sectors	Retention of customers/ increase sales