Assessment Methodology for a Maturity Model for Interorganizational Systems – The Search for an Assessment Procedure

Assessment Methodology for a Maturity Model for Interorganizational Systems – The Search for an Assessment Procedure

Please download to get full document.

View again

of 10
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.


Publish on:

Views: 0 | Pages: 10

Extension: PDF | Download: 0

  Assessment Methodology for a Maturity Model for Interorganizational Systems – The Search for an Assessment Procedure    Norbert Frick University of Koblenz-Landau Tim Florian Küttner University of Koblenz-Landau Petra Schubert University of Koblenz-Landau   Abstract   The benefits of interorganizational systems (IOS) can often not be realized due to unknown or undiscovered impediments, technical deficiencies or lacking organizational capabilities. Maturity models are prominent examples for the assessment of organizations and their capabilities for interoperability. Nevertheless, they are often subject to critiques with respect to their purpose of use, model evaluation, general structure and assessment methodology. This research paper analyses 43 existing maturity models in the domain of IOS with a special  focus on the assessment methodology. The goal is to  find existing assessment methodologies that can be adapted to a new maturity model for IOS. The findings indicate that there is no common or established assessment methodology present but several assessment types. We propose an assessment method matrix for classification and allocation of assessment methods and types according to Model Complexity and Subject Complexity to determine an assessment methodology. 1. Introduction The second most significant IT priority according to a recent survey of 1,500 CIOs in 30 industries and 48 countries are enterprise applications [33]. Still, the  benefits contribution of IT investments to the organization proves to be uncertain [29]. The  juxtaposition of both statements reveals a common conflict in everyday business management which especially reflects in interorganizational IT implementation projects [47]: well-known and identified beneficial effects of interorganizational activities (e.g. [24]; [20]; [12]) cannot be realized due to impediments hindering organizations engaging in a  business-to-business (B2B)-relationship (e.g. [26], [30]). The struggle with the implementation of an interorganizational system that is optimized for an organization’s needs srcinates, for example, from inherent conflicts between the integration partners [36], a lack of management support and technical infeasibility [21] or information transparency problems [47]. Academia and practitioners alike have tried to address this problem by developing different kinds of theoretical assumptions and models for the classification and assessment of interorganizational systems [42]; [51]. Prominent examples of these attempts are so-called maturity models. They typically “[…] consist of a sequence of maturity levels for a class of objects” [4] which outline an evolutionary path from a bottom stage of maturity to the highest level of maturity. Despite their differing purpose, for example as a tool for continuous improvement [1] or as means for the assessment respectively benchmarking [16], they classify and assess institutional, organizational and/or technical capabilities of an organization or information system that provide certain beneficial effects according to the corresponding maturity level. By definition, these model types should be sufficient to support organizations in the assessment of their maturity level and capabilities to conduct interorganizational integration by a) the identification of beneficial effects corresponding to each maturity level and b) the enactment of necessary measures to overcome existing impediments preventing interorganizational activities. However, the literature criticizes that many model approaches lack a sound underlying theoretical basis and a holistic view of all relevant maturity issues of a domain [35]; [5]. This ultimately leads to a collection of IOS-related maturity models that differ in many aspects e.g. scope, granularity or applicability. Our overall research attempts to overcome these deficiencies by developing a maturity model for interorganizational integration that is based on a rigorous development strategy (cf. chapter 3) and a solid empirical database. The results presented in this paper merely reflect the overall design process in our research project but focus on a very important part within it, namely the 2013 46th Hawaii International Conference on System Sciences 1530-1605/12 $26.00 © 2012 IEEEDOI 10.1109/HICSS.2013.106273   2013 46th Hawaii International Conference on System Sciences 1530-1605/12 $26.00 © 2012 IEEEDOI 10.1109/HICSS.2013.106274  assessment methodology  of the maturity model. Our  preliminary analysis of a small set of existing maturity models within the IOS-domain showed different assessment strategies depending on the model design and the model use [17]. Further research suggests that there seems to be no overall accepted assessment methodology for a certain type of maturity model [35].  Nevertheless, the assessment methodology itself is a critical component within the model use as it is responsible for the collection of data, on which basis the maturity level is determined. This contradiction  between the call for an intersubjectively verifiable assessment methodology [41] and a supposedly inconsistent use of assessment methodologies in  practice motivated us to investigate 43 existing maturity models within the research domain of interorganizational systems to answer the following research question: What assessment methodologies are used within IOS maturity models in relation to their model design? The remainder of this paper is structured as follows: Chapter 2 introduces the state of current research. Chapter 3 relates our research steps for this  particular design phase with the overall research design of our longitudinal research project. The analysis in chapter 4 discusses the results from the model comparison. Chapter 5 draws important conclusions, limitations and outlines further research. 2. Critical appraisal of maturity models in IS The term maturity has been discussed in the field of information systems (IS) for a long time. Nolan’s stage model for enterprise data processing (EDP) was one of the first attempts where the evolution of an initial stage to a more mature stage for EDP was theorized [38]. Several other model approaches followed for different domains of IS like quality management [14], organizational maturity [6], use of ERP systems [25] or service operations [32] to name a few. The Capability Maturity Model (CMM) is likely to  be the most prominent example of a maturity model description and assessment [40]. It was developed by the Carnegie Mellon Software Engineering Institute (SEI) and comprises five stages that organizations go through as they move from an immature to a mature understanding of business processes. Its key assumption is that more mature organizations perform more consistently and are thus more successful. The levels are:  Initial  ,  Repeatable ,  Defined  , Managed   and Optimizing  . Quantitative feedback from the process and from piloting innovative ideas and technologies enables continuous process improvement. Probably stimulated by its success, a lot of maturity model approaches for different domains were developed consecutively (cf. [16]; [37] for a broader overview). With more than 150 maturity models [10] in the domain of IS this hybrid of a model description (in terms of defining maturity levels and their evolutionary structure) and method application (in terms of assessment methodology and improvement measures) [34] has become quite popular for academia and practitioners alike. In recent years, the perceived ease-of-use of these models was complemented by a more critical appraisal of the theoretical foundation of maturity models and their applicability (e.g. [10]; [34]; [41]). Typical, but not exhaustive, areas of critiques are: purpose of use, general structure, model evaluation and assessment methodology. 2.1. Purpose of use The purpose of a maturity model is the structured and formalized guidance through an evolutionary  progress by evaluative and comparative measures [10]. To include a broad range of organizations within the  potential domain-specific user group of a maturity model, model designers typically provide an abstract description of the maturity levels and their respective assessment criteria. Consequently, many maturity models open themselves to the critique of simplicity.  Nolan’s stage model was one of the first models to be subject to this kind of critical scientific discussion [7], [27] that pointed to the pure sequential step-by-step view lacking any aspect of evolutionary change. Ironically, more than 25 years of maturity model development did not quite lead to a less critical appraisal of existing maturity models in retrospective [41]. Another aspect of the purpose of use is emphasized by de Bruin et al. [10]: To meet the intended audience’s needs, model designers have to focus on why  the audience wants to apply the model, how  it can be applied (given different organizational structures), who  will apply the model and what   is the possible outcome of the model application. This consideration is important for the later design process, as the design result (the model itself) will always have to find a balance between necessary model granularity and model applicability and understandability. A model that is too simple may leave out necessary information or aspects for the overall maturity assessment, a model that is too complex may limit interest or even produce misleading assessment results. 274   275  2.2. Model evaluation   Maturity models are typical design artifacts [34] that have to undergo some kind of evaluation to show their utility and applicability according to Hevner et al.’s [23] design guidelines. Subject to evaluation can  be: the  process  of model design, the quality  and/or the components  of the maturity model design product [41]. Looking at many published maturity models in IS, the  process of model design  is often unclear and its distinct design steps are not mentioned by the authors [4]. That makes it almost impossible to follow the research steps in terms of repeatability, verifiability or completeness. Furthermore, the contribution to the  body of knowledge with regard to the design process itself is questionable. Authors like de Bruin et al. [10] and Becker et al. [4] set out to overcome this problem by providing future authors of maturity models with generic  procedure models to conduct their model design. De Bruin et al.’s [10] model focuses more on the overall model use stating the phases Scope, Design, Populate, Test, Deploy and  Maintain . Becker et al. [4] follow Hevner et al.’s [23] design guidelines to formulate specific design requirements reflecting in the phases  Problem definition, Comparison of existing maturity models, Determination of development strategy,  Iterative maturity model development, Conception of transfer and evaluation, Implementation of transfer media, Evaluation  and  Rejection of maturity model.  The quality of the design product   is even more subject to criticism than the design process. In most cases quality criteria like verifiability, validity, reliability, generalizability, or cost efficiency [13]; [10]; [45] are dealt with in a non-conclusive manner. That does not mean that there is no empirical evaluation of the proposed model but (e.g. depending on the chosen benchmark variables) there can be differing conclusions about a proven validity (e.g. [28]) or a failed attempt to do so (e.g. [15]). This problem  becomes more apparent when the overall model design and quality criteria are more abstract. The components of a maturity model   are least indicative for a proper evaluation as current research still struggles for a common composition of a “good” maturity model. Suggestions range from a general divide into a domain reference model and assessment model [39] to a more detailed component description consisting of maturity levels, descriptors, level descriptions, dimensions, process areas, activities for process areas and activity descriptions [16]. 2.3. General structure The successful impact of the CMM tempted authors to adapt its structural build-up to their own maturity models [5]. Consequently, many maturity models in the domain of IS were classified as CMM-like  [16]. However, without a proper derivation of their decision for their model structure during the design process the choice for a CMM-like build-up seems arbitrary [41]. Fraser et al. [16] identified further model types as  Maturity grids  (array-like structure with level- and aspect-related statements),  Likert-like  (assessment of maturity aspects according to questionnaires) and  Hybrids (combination of Maturity grids and Likert-like model structure). Yet, an overall classification attempt of maturity models is rare. Many researchers use implicit structural definitions like Solli-Sæther and Gottschalk [46]: Maturity models typically consist of several stages that are (1) sequential within their evolutionary progress, (2) represent a hierarchical structure that cannot be reversed easily and (3) encompass a broad collection of organizational activities and structures. Although these kinds of definitions include a typical model setup and its generic characteristics, they fail to suffice as a classification scheme covering all types of maturity models and possible domain-related specifications. One of the first classification attempts for the IS domain was conducted by Mettler et al. [35] and based on an analysis of 117 maturity models in IS literature. They identified the following three dimensions that maturity models should include covering all relevant aspects and fulfilling their initial purpose: The first dimension, (1) General model attributes  serves mainly as a descriptive part for the model’s assessment. The second dimension, (2)  Maturity model design  deals with conceptual issues like construction and organization of the model. (3)  Maturity model use , as the third dimension, covers the deployment, the suggested assessment and practicality. Each dimension is given a distinct set of attributes that stand for a specific requirement or property of the maturity model. Special emphasis is laid on the  Maturity model use  as Mettler et al. [35] propose three complementary requirements: Practicality of evidence  (how can suggestions for improvement be made), Support of application  (how can users be supported during the model assessment) and  Method of application  (how can the model assessment take place). 275   276  2.4. Assessment methodology Several authors made the observation of missing or insufficient assessment methodologies in existing maturity models (e.g. [16]; [35]; [41]). The main concern is not about the mislead use of the assessment methods themselves but emphasizes the orchestration of these methods, so that an assessment may yield verifiable and valid results. Methods for an assessment can be briefly categorized into human-machine and human-human interaction. The simple self-information process based on available documentation of maturity-related information within the organization can help assessors to gain a preliminary assessment. This can be enriched by questionnaires that are answered e.g. via an online survey tool [10]. These results can be complemented by a face-to-face inquiry (e.g. interviews) or by face-to-face (moderated) workshops respectively focus groups. Another aspect leading to a methodology utilizing the aforementioned methods is the distinction between three types of assessments [16]; [35]: Self-assessment   (gathering information about the own organization by internal members), Third-party assessment   (extension of the self-assessment by including external assessor specialists supporting the internal staff) and Certified assessment   (outsourcing of assessment to certified practitioners). In particular, Self-assessment   is subject to critique as it is biased by definition due to the fact that the assessors stem from the internal staff of the assessed organization. Nevertheless, none of these types gives any hint to an assessment procedure that deploys specific methods for a pre-defined purpose to yield verifiable or valid results [41]. 3. Adoption of critical appraisal to maturity models for IOS As stated earlier, our overall research attempts to design a maturity model for interorganizational systems that is based on a rigorous development strategy and on a solid empirical database for later evaluation. We adopted Becker et al.’s [4] and de Bruin et al.’s [10] procedure models for maturity model development to follow a design-oriented research approach. Our phases are in line with Becker et al.’s [4] proposed development phases:  Problem definition, Comparison of existing maturity models,  Determination of development strategy, Iterative maturity model development, Conception of transfer and evaluation, Implementation of transfer media,  Evaluation  and  Rejection of maturity model.  With respect to the intended long-term use of our model we extended the procedure model with one further phase from de Bruin et al.’s [10] procedure model: Maintain.  Reflecting on de Bruin et al.’s [10] four questions that have to be considered during the design of such a model (cf. chapter 2.1), it is especially the what  -question referring to the outcome of the model ( Purpose of use, cf. chapter 2.1) that is currently in question at this point in our design process. The assessment methodology and the corresponding assessment methods are critical for any model evaluation (  Model evaluation, cf. chapter 2.3). Therefore, we conducted an in-depth analysis of 43 existing maturity models in the domain of IOS to examine applied assessment methodologies/ assessment methods and their applicability for our model design. This research step was performed within our overall design process adapted by Becker et al. [4] and de Bruin et al. [10], namely: Comparison of existing maturity models . Within this design phase it is necessary to reflect on already existing work in the domain of IOS to incorporate already sound and valid research results in the overall design and thereby laying the foundation for the consecutive design phase: determination of development strategy. To conduct a comprehensive analysis of existing assessment methodologies in IOS maturity research we decided to perform a preliminary literature research. Existing guides for a thorough and rigorous literature search are rare and differ in size and scope [50]. Therefore, we decided to base our literature research on a recently developed framework for literature search and analysis by vom Brocke et al. [9]. They distinguish five different phases: (1) Definition of review scope, (2) Conceptualization of topic, (3) Literature search, (4) Literature analysis and synthesis and (5) Research agenda. We chose journal databases as preliminary scope of our search (ACM Digital Library, AIS Electronic Library, EBSCOhost, Emerald, IEEE Xplore, INFORMS, ScienceDirect and SpringerLink) limiting the time period from 1993-2010 (ad 1). Our database search was initiated with chosen keywords from the domain of interest, namely: maturity , maturity model   and interoperability  (ad 2). The search itself proved to be a bit more difficult as the term maturity was used in many publications but seldom hinted towards a complete maturity model. Additionally, the database research was extended to cover maturity models published in conference proceedings and for practitioners use. Overall, we found 43 maturity models that described and assessed different topics within the domain of IOS (supply chain management, enterprise interoperability in general, data exchange, service-oriented architectures and Web services) (ad 276   277  3). For purposes of the analysis, we decided to use the existing classification framework for maturity models in IS by Mettler et al. [35] as analytical lens with a special emphasis on the model use ( General structure ) (ad 4). This allowed us to create comparability between all analyzed models as far as a classification was possible given the published content in the papers. For a more detailed description we refer to the following chapter 4. Finally, the proposed research agenda (cf. chapter 5) lines out several important issues concerning assessment methodologies/assessment methods for future research (  Assessment methodology ) (ad 5). 4. Analysis Overall, 43 1  maturity models could be identified during our literature research and were analyzed according to Mettler et al.’s [35] classification framework for maturity models in IS. During our analysis we found that a mere comparison of all available models could cause a distorted result as the focus differed significantly (e.g. supply chain management vs. Web services). Accordingly, we decided to perform a pre-classification of the model base. From the literature we derived three main perspectives on interorganizational integration that seemed reasonable: (1) technical, (2) organizational, and (3) institutional (e.g. [30]; [2]; [11]) (cf. Table 1 for an excerpt of maturity models assigned to the three perspectives). Technical integration  describes how information is processed and shared electronically within and across organizations. Electronically supported communication eliminates manual workload, prevents false data capture and data redundancies and is thus deployed to save costs and transaction time. Organizational integration  refers to the organizational structures and processes, which are put in place to improve the efficiency and effectiveness of the supply chain. Business processes are a key element of organizational design and are considered the core of IT-based value creation.  Institutional integration  describes the formal and informal agreements, which govern inter-organizational relationships, thereby reflecting the concepts that have been developed by transaction cost theory and the resource-based view. This area describes the governance structures among the players. The following analysis according to Mettler et al.’s [35] classification scheme comprises only an 1  Please note: Due to page limitations we were not able to include all references for the 43 maturity models. The complete reference information is available upon request via e-mail. excerpt of the   total number of maturity models due to page limitations. 4.1. General model attributes We examined a total of 43 maturity models, 29 with academic srcin and 14 from business initiatives. The period of publication ranges from 1993 to 2009. The literature review confirms the findings of Mettler et al. [35] and de Bruin et al. [10], describing the lack of documentation and classification of existing maturity models. Some publications only provide the model itself (e.g. Types of Supply Chain Evolution), others focus on a mere illustration of the model (e.g. Supply Chain Redesign Capability Maturity Model) or give a brief description (e.g. Levels of Information Sharing between Organizations). Fourteen models could be assigned to the technical integration perspective, 27 to the organizational and 2 to the institutional integration perspective. The overall distribution shows a significantly lower number in institutional maturity research whereas technical and organizational aspects are both well represented. The relevance of these models shows a more interesting aspect, as it does not mainly reflect on the models’ same target audience (business users) but on the fairly high number of business related srcins (18) in comparison to models developed in academia (25). From an epistemological point of view the analyzed models represent a broad bandwidth of different theoretical and even philosophical stances ranging from a subjectivist [48] to a positivist worldview [18]. Still, our investigation did not reveal a direct link between the epistemological model background and the use of an assessment procedure. 4.2. Model design The majority of maturity models follows the  Maturity grid   model type (27) while 16 models use the CMM-like  build-up. The  Hybrid   model type could not be identified in any analyzed model design. Based on this analysis of more than forty models, however, no apparent pattern can be discerned in terms of model design adoption. It remains unclear, why and in what context the developers of the models chose e.g. the Maturity grid structure and discarded the CMM-like structure. It seems almost arbitrary what type of model structure is applied to a given problem domain. See Table 1, below, for the classified maturity models. 277   278
Related Search
Similar documents
View more...
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks