Iot data analytics architecture free download pdf

Fourthly, we apply our proposed framework on a case study to demonstrate development of an IoT-DA healthcare application. The results of this research can facilitate researchers and practitioners to engineer emerging and next-generation of IoT-DA software applications.

Typical example of such contex- as a disruptive technology and an enabling platform that tualised data includes crowd-sensed traffic congestions or interconnects heterogeneous things such as humans, systems, environmental pollution that can be captured by embedded services, and devices in a smart environment [1]—[3]. A recent sensors of mobile devices, manipulated by mobile applica- survey by Gartner predicts that by the year , the world tions, and transmitted over wireless networks [3].

IoT systems will become home to 20 billion internet-connected things in general and IoT-DA applications in particular act as the with data produced, consumed, and processed by IoTs will backbone for data-driven smart cities and infrastructure ini- funnel through IoT-DA platforms [1]. A rapid proliferation tiatives across the globe such as in United States [4], Europe, of IoT systems is primarily due to portable devices that and Asia [5], [6].

Software or hardware novelties for IoTs unify hardware embedded sensors software applications are vital, however; true potential and business value of IoTs that manipulate sensors and network protocols that connect lies with IoT-DA that derives useful intelligence from data captured by sensors and things for strategic decision making. The associate editor coordinating the review of this manuscript and Due to an inherent complexity and heterogeneity of IoTs, one approving it for publication was Mehdi Hosseinzadeh.

Ahmad et al. Researchers and practitioners argue A. SE FOR IoTs about the application of traditional software development Software engineering principle and practices support the life-cycles SDLCs such as agile and iterative approaches design, development, evaluation, and maintenance phases of to develop IoT systems and applications [10].

However, complex and large scale software-intensive systems effec- some recent surveys indicate that IoT systems represent a tively and efficiently [8]. In recent years, SE research and complex combination of hardware and software components development have mainly focused on addressing challenges that need customised methods and tools for their develop- that relate to the development of IoTs in the context of smart ment [10], [11].

In addition to the technical challenges, IoT systems and cities [3], [7]. A recently published roadmap for systems such as smart transportation system involves multiple the adoption of emerging technologies streamlines engineer- domains and a diverse set of stakeholders including pub- ing efforts — processes and frameworks with tool support — lic organisations, policy makers, and citizens with varying for sustained development and economic viability of systems requirements [14].

SE for B. The International Data Corporation forecasts methods, technologies, and processes that enable or enhance that by the year the revenue generated by big data engineering lifecycle [15]—[18]. To date, there does not exist analytics will reach approximately This research is a pioneering effort to stream- consuming to maintain due complexities of exponentially line the process and engineering lifecycle - based on academic growing data and escalating costs of scaling these solutions research and industrial solutions - for software applications to real problems [7].

SE for IoT-DA can support strategic that derive key intelligence from data ingested from IoT capabilities of enterprises by exploiting software tools, algo- sensors. We propose that by unifying i software engineering rithms, and applications that collect data from IoT-driven processes, ii internet of things development, and iii data sensors and deliver intelligent analytics to stakeholders.

For analytics methods, an engineering lifecycle can be adopted example, an application that logs availability and perfor- that supports design, development, operationalisation, and mance of interconnected robots in industrial automation evolution of emerging and next generation of IoT-DA appli- can algorithmically mine those logs to discover patterns as cations. The objective of this research is to empirically iden- reusable knowledge, best strategies, and key insights for opti- tify and systematically evaluate the existing processes and mising process automation [10].

In the context of engineer- their underlying practices based on academic research and ing IoTs [11], IoT application implementation differs from industrial solutions to develop IoT-driven data analytics mainstream mobile or web application development due to applications. From system development and operation perspective, C. A complex blend of hardware and software As in Figure 1, the research phases include: artefacts poses challenges for engineering and development - Phase 1: Develop a framework to evaluate SE processes of dynamic IoT applications that require reusable knowl- and their underlying practices that enable SE for IoT-DA edge and best practices of software and system develop- applications.

Engineering life-cycle to develop IoT systems - Phase 2: Apply systematic mapping study to qualitatively and IoT-DA applications must take precedence over ad-hoc select and document 16 processes 08 each from academic and once-off efforts that lead to increased efforts, decreased research and industrial solutions that support SE for IoT-DA quality, and ultimately inferior product delivery [2], [3].

One applications. Standardised processes and IoT-DA applications. Millions of physical and virtual sensors generate invaluable data, i. However, we will see in this section that limited research exists on how IoT-DA applications are designed, implemented, operationalised, and evolved in the context of software and system engineering life-cycle.

In this section, first we review related research and development in the context of SE for IoT-DA applications that helps us to justify the scope and contribution s of pro- posed work Section 2. Review of the existing research and development also enable us to derive a reference model that conceptualises software engineering process for IoT-driven data analytics Section 2.

Overview of research method and proposed approach. The discussion about related work applications. Figure 2 synergises The proposed research complements one of the recent already established principle and practices from different community-wide initiatives on exploiting engineering prac- research domains to pinpoint the needs for SE methods and tices to develop IoT-driven systems [2]. Primary research techniques that can be applied to data analytics systems that contributions include: gather data from IoTs sensors.

Case study along with recommended practices and In software engineering, a number of frameworks have lessons learnt can help: been used for criteria-based evaluation of software pro- - Researchers to understand state-of-the-art, analyse its cesses and their underlying activities such as software archi- strengths and limitations, and derive new theories or hypothe- tecture, software evolution, and software evaluation. Other notable examples include prototypes etc. The evaluation Organisation of Paper: Section 2 presents related research framework in [23] appraises existing cloud migration pro- and reference model for IoT-DAs.

Section 3 presents research cesses to objectively assess their similarities, weaknesses, and methodology and evaluation framework. Section 4 presents strengths. The framework and results of evaluation enable the results of process evaluation. Section 5 presents a case both the researchers and practitioners of cloud computing to study on IoT-DA application.

Section 6 presents recom- assess the suitability of processes for the cloudification of mendations, lessons learnt, and threats to research validity. Existing frameworks are limited to evalu- Section 7 concludes the paper. Smart cities and societies provide state-of-the-art approaches In recent years, these frameworks have also been used to for urbanisation built on smart applications, systems, evaluate the systems such as cloud [24] and IoTs [25]. The and other infrastructures [19]. Comparative analysis of existing vs.

An example is ThingML systems. Specifically, the analytical framework in [26] eval- approach that is inspired by UML based system modelling to uates and provides comparative analysis of nine well-known addresses the challenges of distribution and heterogeneity in IoT architectures. The results of architectural evaluation pro- the Internet of Things [12]. ThingML supports model-driven vide insights to city leaders, architects, and developers to engineering approach to develop high-level design models select the most appropriate architecture for implementing that drive coding, testing and evolution of IoT-based e-health smart city systems.

We summarise comparative analysis of solution. ThingML [12] as IoT specific modelling notation existing evaluation frameworks and taxonomies detailed extends the traditional architecture description languages [15] above with the proposed framework in Table 1 at the end of for domain specific modelling of IoTs [12]. Aca- and devices producing, consuming, and exchanging large demic research and industrial solutions are in a continuous sets of data volume, velocity, and variety of it give rise pursuit to empirically derive and validate processes, meth- to big data [19], [21].

IoT supported big data is the key ods, and tools of software engineering, architectural patterns, enabler for data-driven intelligence that provide foundations design patterns and development methods to develop solu- for sustainable computing environments in the form of intel- tions for IoT-DA [3]. Some of the key industrial players for ligent systems and smart cities [3].

In recent years, a number IoT infrastructures and solutions e. The backbone for these smart systems systems [2], [3], [27]. A recently published road map to is interconnected sensors that continuously produce or con- software engineering in IoT era suggests that SE research sume mission critical data to support the operations of the and development have mainly focused on addressing issues system [2].

Mission critical data refers to data and key infor- that relate to the development of IoTs in the context of smart mation that supports the mission of a particular system. For systems and cities [11]. However, SE for IoT approaches example, in smart transportation system, data such as route primarily focus on modelling and implementation efforts planning and traffic congestion aims to support mission of only that overlooks process-centric approach and engineer- smart transportation based on efficient route planning and ing life-cycle of IoTs systems [7].

The key to successful execution [28]. Existing research is mainly focused on archi- application of SE approaches to engineer IoT systems lies tectural abstractions [2], design patterns [12], and deployment with abstractions on which IoT system and application engi- models for IoT-DA.

There is more focus on implementation neering could be developed [7], [8]. These abstractions can but less focus on engineering cycle of the IoT-DA solutions. Reference model of software engineering for IoT-driven data analytic. For example, [26] published in provides a process automation, human roles and decision support to cus- framework for classification and comparison of architectural tomise and supervise the development process [7], [29], [14].

The list of evaluation frameworks in Table 1 is of existing process models to analyse, design, implement, and not exhaustive but most relevant in terms evaluating SE spe- deploy IoTs for smart systems [10].

A recent survey on IoT cific processes and artefacts. Table 1 compares the state- development highlight that IoT driven applications represent of-the-art of evaluation frameworks based on i artefacts a unique class of systems with blended hardware components of evaluation, ii number of solutions evaluated, iii type and software services [25]. In IoT-DA applications, the chal- of system under investigation and iv focus of evaluation.

In Figure 2, building blocks of IoT IoT-driven data analytics applications that lacks in existing system are things such as a vehicle, digitise home, and service work.

Criteria-based evaluation of the SE processes can help robot that are equipped with sensors and control software to researchers and practitioners to understand the key chal- coordinate with each other [7]. For example, in smart home lenges, recommended practices, methods, tools, algorithms, settings the vehicle can coordinate the arrival time of resident needs for process automation and human decision to develop to signal the service robot to finish up with cleaning and start emerging and new generation of IoT-DA applications.

Also, the vehicle can commu- nicate with electronic appliances for maintaining appropri- B. The reference model in Figure 2 acts by robot, and power usage patterns by home appliances for as a system-level blueprint to unify software engineering pro- energy efficiency. By following a from IoT devices and sensors [2], [7].

Specifically, real-time process-centric approach, software engineering methods can data consumed and produced by IoT devices such as traffic be applied in terms of architectural models, pattern templates, flows, health monitoring, and environmental conditions can and algorithmic specifications to optimise resource manage- be analysed to gather key intelligence for operationalising ment, computation demands, and operational efficiency of smart cities, systems, and infrastructures [3], [7].

As in Fig- IoT systems. In the context of SDLCs [8], [23], SE process ure 2, software engineering process and its underlying activ- provides a blue-print of underlying activities and practices, ities lay the foundation for analysis, design, development, supporting engineering life-cycle of software-intensive sys- testing, deployment, and evolution a.

SDLC of soft- tems, that includes analysis, modelling, implementation, test- ware intensive applications and systems [8]. The generic ing, and deployment of software applications under consid- process in Figure 2 is adopted from [8] and customised to eration. In this context, process as higher-level abstraction represent the de facto activities for SE both from academic outlines: what needs to be done? For exam- for an incremental engineering with possible iterations for ple, software modelling as a sub-process of the SE process process refinements.

Tool support and human roles are com- contains an activity named requirements specifications that plementary part of the process in order to support automa- focuses on representing high-level design and specifications tion, customisation, and, human decision support to execute of the systems in the context of functional requirements and and supervise the process for engineering software-intensive quality attributes that must be satisfied in the implemented systems.

For example, at the system design phase, design patterns A typical argument is that instead of any re- inventions as reusable knowledge and best practices can be exploited to of development processes and engineering lifecycles for IoT address the recurring challenges of developing IoTs. Specif- software, traditional development cycles e. However, recent surveys and empirical studies sug- nected devices and sensors [3], [7], as in Figure 2.

Moreover, gest that development of IoT-intensive software differ from the layered deployment model helps with modularising data traditional development practices for mobile or web appli- analytics application based on deployed sensors and devices cation [11], [10].

Industrial solutions [13] also suggest that as front-end layer to collect data that is processed and stored IoT systems exhibit specific requirements that need tailored by cloud-based servers as backend layer [7]. The example also conduct systematic mapping studies and thematic analysis demonstrates that well established principle and practices of of topic under investigation [32].

Moreover, some existing SE i. Based on the engineering process and features of software-intensive sys- smart transportation scenario in [28], some of the typical tems are being used to derive the framework.

Figure 3 is activities of SE process and their impact on engineering used for illustrative purposes to discuss different steps of the IoT-DA applications include: research methodology. Step A includes 1 investigating existing evalu- ics. In this scenario, the layered architecture model ation frameworks and taxonomies of SE, and 2 analysing supports separation of functional concerns.

Specifi- the reference models and architectures for IoT systems. After cally, the front-end layer consists of portable and conceptualising the framework, we engaged 7 domain experts context-sensitive mobile devices that capture contex- i. The location, traffic bottleneck on a given route.

The back- feedback from domain experts helped us to refine and finalise end layer supports storage and computation of traffic the framework. This means that a backend system must illustrated in the middle-top part of Figure 3. We conducted a be integrated as an additional architectural layer to mapping study to identify SE processes that have emerged which mobile devices can off-load crowd-sensed data from published academic research and documentation of for analytics and storage.

Furthermore, quality attributes industry specific solutions. We selected 08 processes and applicability of the system. The distribution pattern tively evaluate the process in terms of process planning, pro- deploys and operationalises crowd-sensing transporta- cess execution, and process support for software engineering. The process repos- itory accumulates all the process specific information e. The list of In this section, we overview the research methodology and selected SE processes for IoT-DA applications is presented evaluation framework.

First, we discuss the research method in Appendix A. The next step in the methodology relates to documenting the results of evaluation that is illustrated in the middle-bottom A. Illustrative overview of the different steps of research method.

Case study also helps to present some cution: Engineering and Development, iii Process Support: recommendations and lessons learnt from SE process and Roles, Tools, and Technologies along with iv Domain of practices for developing IoT-DA applications. The framework in Figure 4 is based on generic process for SE as in [8], [7], [15].

Moreover, two distinct views of the Engineering and Development supports the sub-process of framework help to distinguish between abstraction and Modelling and Design. However, the activity Modelling instantiation of the framework detailed below. In the soft- Notation shows that in order to model IoT systems, specific ware engineering context, an evaluation framework is modelling notations such as ThingsML [12] are required that referred to as a hierarchical abstraction that organises and can support features such as i abstraction of hardware and interrelates i artefacts being evaluated e.

Figure 4 shows to backend cloud nodes processing and storage servers. For exam- 1 In the context of evaluation frameworks, the terms features and activities ple, Figure 4 illustrates that feasibility analysis helps to create are virtually synonymous and often used interchangeably.

Overview of the criteria-based evaluation framework process activities and process illustrations. In software engineering context, processes and Figure 4 presents a fine-grained hierarchical composition of activities are complementary to each other, i. Table 2 complements while practices implements process es to achieve the engi- Figure 4 to detail the criteria for evaluation in terms of a neering objectives how to do?

For example, the sub-process comprehensive catalogue of process activities. We reviewed a total of 16 processes and example. In addition, Table 2 also shows activities in the practices, 8 each from academia and industry. For example, process that depend on each other. In comparison, SeeboIoT guages being used to create system design? Typical example [P12] as an industrial solution support sensor data from indus- of such modelling could be UML based notation and profiles trial machines and processes for industrial automation and to specify system design and architecture [12], [31].

The smart manufacturing. Complementary view to process life cycle is domain of appli- IV. For example, the domain of application for CLOTHO We now present the artefacts and results of evaluation for SE [P3] is disaster management by analysing emergency scenar- processes and their underlying activities to develop IoT-DA ios and their impact in smart city context.

The pro- - Evaluation Criteria represents individual element to cess matrix in Table 2 is derived from evaluation framework objectively evaluate the processes and practices.

In Table 3, Figure 4 and process catalogue Table 2 that comprises of there are a total of 17 criterion to evaluate the process lifecycle four primary elements, referred to as the artefacts of evalua- that are distributed as A. Feasibility Analysis 03 criterion , tion. In order to support process evaluation: B. Engineering and Development 10 criterion , C. Roles, - Evaluation framework in Figure 4 conceptualises differ- Tools, and Technologies 03 criterion , and D.

Domain of ent phases and activities of the SE process. Application 01 criteria. For example, the criterion Quality - Process catalogue in Table 2 complements the framework Attributes SR2 evaluates the engineering and development to provide fine-grained description of each process, its sub- phase of IoT-DA system based on specification of quality process es , and activities along with collaboration or depen- attributes or non-functional requirements that must be satis- dencies between the activities.

Each structured representation of information for fine-grained level is assigned with a numerical value ranging from 4 to 0 evaluation of individual process. The numerical values provide a quantified average of the evaluation score. The A. Matrix-based eval- all the activities in a single solution. For example, the total uation driven by artefacts of evaluation reflects a collec- quantified value i.

In comparison, the quantified fine-grained interpretation of process information, detailed value based on all activities for an individual process named below. As in Table 2, - Sources of SE processes. The processes, sub-processes, a framework named CityPulse [P1] has satisfactory represen- activities, and methods to engineer IoT-DA software emerge tation of the functional aspects in terms of a efficient route from two main sources namely Academic Research planning, b vehicular coordination to develop a smart trans- portation system.

However, there are poor trade-off analysis and Industrial Solutions. Academic Research repre- for solution in terms of performance and availability of the sents well documented, peer-reviewed, published solutions system. In comparison, Industrial Solutions are commer- cially adopted operational systems and practices that are 3 The notation [Pn] n is a number represent references to processes that has been evaluated and presented in Appendix A.

The notation also 2 The term Table 3 and Process Matrix can be used interchangeably both maintains a distinction between the referencing for bibliography and list of referring to same element of evaluation. Catalogue of process activities assessing processes, activities, sub-processes. The examples below demonstrate how to interpret implementation s? In the following we use examples to demonstrate how ality being collected and analysed in ITS solution? Amazon IoT Analytics ular process or its underlying activities to objectively assess solution [P14] uses Amazon Platform such as Amazon the strengths and limitations of process-centric approaches Web Services for data analytics from IoT devices.

The for engineering and development of IoT-DA applications. In general, the overall process support for IoT-DA In comparison to the structured analysis, un-structured applications is weak with lack of tools and human decision results can be interpreted based on specific information that support.

Based on the evaluation results, we present a case needs to be extracted. For example, anyone interested to know study and outline some recommendations that highlight the about the level of tool support can identify Google IoT [P15] lessons learnt to develop emerging and futuristic solution for and ThingsSpeak [P10] as existing solutions that exploit IoT-DA applications.

The following is further exemplifi- V. We also discuss in the development life-cycle for IoT-DA applications? Figure 5 provides a the process matrix indicates that during IoT-DA appli- visual illustration of the process and its decomposition into cation lifecycle, engineering activities that are most various activities.

Table 4 complements Figure 5 to provide focused are business needs and platform analysis as structured analysis of each process, its sub-process and under- part of the feasibility analysis. During the engineering lying activities that support system development. Modelling In software engineering context, process-centric development notations are focused less and level of tool support is not refers to an incremental approach to support the modelling, much supported. Health-Connect case study is part of smart city initiative Prominent tools and technologies refer to tool chains that enables individuals to exploit on-body portable and and enabling technologies that are being used more fre- connected sensors that frequently monitor health signals quently to support process automation, as per findings originating from human body.

These signals related to body of reviewing and evaluating the processes. Specifically, temperature, pulse rate, and blood pressure etc.

For example, the tool named Eclipse IoT Health analytic server as cloud-base storage and process- is among the most frequently used tool for creating the ing infrastructure generates health analytics in terms of a design and implementing source code for IoT-DA sys- Health Profile.

Health Profile is medically critical document tems. In addition to the analytics, the health profile s stored tions using a particular technology. For example, Ama- at server could be shared with other medical profession- zon Web Services AWS as a collection of on-demand als for consultation.

The ultimate goal of Health-Connect cloud computing technologies to operationalise, deploy system is to exploit sensor-based data to analyse, profile, and host IoT-DA applications, managed via pay-per-use and manage connected and smart health care.

In the follow- services. For example, the solutions [P14] used the AWS ing, Health-Connect case study and its development process, technologies to perform analytics based on data storage decomposed into three sub-processes with seven underlying with Amazon Simple Storage Service S3. Overview of process-centric development of the Health-Connect case study.

SE process cf. Figure 2. The various advantages of IoT has been visualized in fig 1. IoT comprises of not only the computing devices but also humans who can sense, communicate and compute. Thus along with the advantages IoT comes with it inherent complexities and challenges.

The major concerns related with IoT are complexity of the system, space, size, security and privacy. Due to huge number of interconnections there is a great possibility of increase in complexity of the system. The Internet of objects would encode 50 to trillion objects and would be able to follow the movement of those objects [9].

The size of IoT would be a major concern. Direct collection of sensitive personal information, such as precise geo location, financial account numbers, or health information may create privacy risk. The data intensive nature of IoT can be channeled with Big Data as a part of the solution to the challenges faced by IoT. IoT are one of the major sources for Big Data. With the count of interconnected devices increasing the data associated with them is mounting to a humongous one.

IoT intersects with Big Data and it is evident that the two trends would fit one another. This is not being complimented by the actionable data [9]. The Big Data analytics would provide a platform to enhance and obtain actionable data for the humongous data being collected. This paper deals with the relationship between IoT and Big Data and its significance. But IoT will be creating streams of data similar to social networking. The concept of IoT can be visualized in a smart warehouse where data is being stored regarding the opening of the door of warehouse like duration, temperature, time and date, frequency per hour, per day, per week so on.

This is a continuous stream of data which is being captured by various sensors deployed. Similarly in a use case of smart home issues like roof damage, water and gas leakage, power consumption can be effectively handled with the help of the sensors and computing devices which would be streaming data at a very regular interval of time. These scenarios indicate huge growth of data in implementation of IoT.

Big Data and IoT infuse at this junction. IoT and Big Data analytics Data though collected by the devices need to be filtered to make it relevant and useful. The redundancy in the data being collected is predominant due to the sheer nature of the framework of IoT.

The data is continuous hence the extraction of valuable information is not simple. This requires a good mechanism of protocols and software to ensure that the data is secured and also significant. Similarly the data is distributed back to the devices also.

These activities require performance efficiency of the network to be optimum. These networks would help in transmission of data and also involve various types of quality issues ranging from performance to energy efficiency. Big Data and IoT are complimentary to each other and are two dimensions of a perception.

Managing the data and extracting information from it is a very vital task associated with IoT. An appropriate analytical platform is required to enable to derive knowledge from IoT data. IoT devices generate continuous streams of data in a scalable way. It is essential to handle the high volume of stream data and exploit the data.

In a normal scenario Big Data, the data might not be stream data, but the actions are. While in IoT data, it is continuous flow. Applying real time analytics is the need in IoT environment. The advantages of IoT can be seen only when real time analytics is applied on the data stored.

Real time Big Data analytics and IoT equates to value creation which is depicted in fig 2. To create models to forecast future outcomes and to optimize the same [18]. Collect information to estimate factors that would not be directly measured by sensors, by determining the relationship between different system parameters, and their impact on each other.

Technologies and analytical techniques employed, attacking a big data project being analyzing and researching. Several software technology products are available. Hadoop is key technology used to handle big data.

Apache Hadoop is an open source frame work that deals with distributed computing of large data sets across clusters of computers using simple programming models. Major advantages that Hadoop offers are we can use inexpensive hardware. Hadoop distributed file system provides high-throughput access to application data stores large amounts of data.

HBase is a scalable and distributed database supports structured data storage for large tables provides for transactional kind of capabilities by allowing updates inserts deletions etc; HBase allows for random check. Pig a high level data flow language and execution frame work parallel computations. Apache Pig is a scripting language, Map reduce transformations including summarizing. Hive a data warehouse infrastructure that provides data summarization software tool used for managing and analyzing large datasets.

SQL is traditional languages. Sqoop software tool designed to transfer bulk data. Zookeepers a high performance co-ordination service for distributed applications. It is a centralized service used for maintaining configuration information named registry. Avro is a data serializations system.

Cassandra a scalable data base with no points of failure.