Enterprise Architecture Demystified

Sethuraj Nair’s Enterprise Architecture Thoughts

Posts Tagged ‘MDM’

EAI, EDI, Convergence (and Convenience ! ).

Posted by snair007 on July 27, 2008

Integration Methodologies: Melting Frontiers Or Age of Divergence ? – Part II

 

In the first part of this discussion, we have seen quite a hatful of interesting jargons and acronyms ‘basking’ under both EAI and EDI, the niche of each is apparently unique and delimited. There are overlaps – pardonable ones. But our core discussion is not on these subsections anyway; we are here to know how appropriate it is to address data and application integration problems separately, where they converge – if at all- , and where down the road the industry and integration solution providers want to see each of them.

In IT architecture, the greatest rule of thumb says that if a problem baffles you, state it in business terms before addressing it. Our problem thus reads as to (1) what the users want to see after integration and (2) how far data and application integration methods can work together or in isolation to bring about what they want to see. So, yet again, it all boils down to the obvious center of enterprise universe – requirements.

Here too, the classical classification of requirements is invariably applicable – ‘business’ and ‘technical’. (I would always choose to call them more broadly and sensibly as operational requirements and enabling requirements – but that’s another discussion, anyway). Assuming that the business requirements heavily depend on the pertinent domain, and that it is not all that worthwhile to delineate some across-the-board pattern in business requirements that drive the integration, we’d better focus on the technical ones. But it will be even better if we combine the requirement in a need-and-fulfillment model. Let’s do that.

Hitting The Object(ives)

So, what is it, in simplest terms, a typical enterprise ‘E’ expects out of enterprise integration methodologies in general and how these methodologies meet the expectations?

·          To integrate business processes by facilitating enhanced interaction among ‘relevant’ enterprise components in IT spectrum (data, departments (business units, sub- organizations), actors( human resources), applications, interfaces, locations, services, sub-processes, functions, standards, rules  and infrastructure).

·          To broaden the business perspective across the enterprise by collating/ aggregating/re-organizing/re-structuring/re- representing relevant enterprise data segments.

·          To widen the ambit of enterprise’s functional standards, business policies, operational rules and governance models by unifying/centralizing/federating the hosting and management of business rules.

·          To assist effective and seamless co-ordination/mergers/amalgamation processes among business-units by improving connectivity, inter-operability and communication across disparate platforms and technologies.

·          To improve the enterprise’s adaptability to changes in line with business goals and objectives by establishing enough  ‘channels’  in terms of configurable mappings between business logic and technology for efficient change- propagation.

·          To optimize the efforts involved in altering(adding/removing/updating) an ‘enterprise component’ by optimizing degree of coupling among components

·          To reduce the cost of maintaining and scaling the enterprise IT framework by standardizing and reusing enterprise-wide interfaces or other middle-ware infrastructure and by reducing the number of “point-to-point” connections.

·          To ensure functional and informational consistency in the enterprise by unifying/standardizing transferring protocols in tandem with an increased usage of business semantics while defining interfaces.

Now that we have seen the expectations of integration, we are in a very good position to sum up the “KSAs( Key Solution Areas)” that could be used to appraise the role and effectiveness of integration methodologies to position them accurately :

·          Process Orchestration  : Components’ Coordination (broadly : Functional Co- ordination)

·          Enterprise Perspective : Data Organization

·          Governance : Business Rules Management

·          Business Scalability and Technology Portability : Platform Independence

·          Agility/Change Management -: Change-Propagation

·          Agility/Flexibility  : Coupling Optimization

·          Cost Control: Interface and Components Reusability

·          Consistency :Message Standardization

 

EAI to Bear The Torch

The distribution of applicability of EAI and EDI methods (that we have seen in Part-1 of this discussion) across each of these KSAs would look somewhat like this:

·          Components’ Coordination  – EAI , EDI

·          Data Organization – EDI

·          Business Rules Management -EAI , EDI

·          Platform Independence EAI , EDI

·          Change-Propagation-  EAI

·          Coupling Optimization – EAI

·          Interface Reusability – EAI,EDI

·          Message Standardization  – EAI , EDI

 

This is as much an interesting result as an important one. EAI by its definitions and for all its accepted forms/types can be found to be ‘wider and deeper’ in its scope. In other words, EAI turns out to be a definitely broader area, or at least to be more significance to the enterprise in terms of meeting its expectations out of  consolidating its  IT ‘ingredients’.

EAI also has its firm footprints on many layers of integration where data integration methodologies lack obvious and explicit applicability – be it Presentation Layer and Network Layer ( No, not OSI reference – just the implementation layers). If someone wants to adulterate the scopes and definitions to argue otherwise, well – s/he should be an awfully loyal EDI fan. Recommended 3-step algorithm for him/her  is –  start with that assumption , use Reductio ad absurdum and get disappointed.

Shared Pains

 Sarcasm apart,  there are a lot of common integration problems where there is enough room for solid give-and-take between these methodologies. Some of them are,

·          Need for Independence/transparency of platform heterogeneity

·          Data Exchange Requirements and formalized Data Formats.

·          Need for message integration and workflow management

·          Need for Control and coupling

·          Usage of common business integration middleware.

·          Repository-driven Meta Data management requirements

·          Need of Federated Models (in some scenarios)

·          Common security challenges.

·          Distributed data manipulation and concurrency control.

·          Usage of integration assistants like APIs, XML,RMI Service Interface standards such as WSDL and HTTP/SOAP.

·          Desired common traits of integrated framework include Scalability, reliability, consistency and availability.

Role Convergence  

To hone our discussion, we need to have a look at two more distributions – this time to affirm the activity-overlaps across the integration methodologies :

(1) Distribution of EDI presence across EAI categories : The amount of Data Integration activities typically performed as part of the majot EAI types we have seen in Part-1 of this discussion (Functional Integration , User Interface Integration and Data-level integration)

 

 

(2)  Distribution of EAI presence across EDI categories : The amount of Application Integration activities typically performed as part of the major EDI types we have seen in Part-1 of this discussion (( EDR,ETL,ECM,EII,MDM,CDI,BPI and EAI itself))

 

Conclusion

Presence of several monolithic integration approaches may slim down over the years as evolving paradigms would either supersede them or make them completely redundant. Still at lower strata of activities, these changes would prove to be nothing more than surges of ‘terminological feats’ , for, from an Enterprise Architecture perspective, both Application and Data Integration methodologies will forever be considered as key enabling techniques for all broader enterprise technical programs such as mergers and acquisitions, strategic architecture reorientations, SOA implementations, service clustering, business process integration and enterprise activity monitoring.

A good example is what the enterprise integration arena has seen with the advent of ESB or SOA – many schools of thought preferred to see them in mutual exclusion with EAI.  EAI, despite its terminological advantage in terms of ‘interpretive flexibility’ that should have qualified it to be considered as a functional superset of even SOA, was demeaned to deal with sheer point-to-point integration (hub-and- spoke, at best). Fact of the matter is that SOA as a conceptual framework and ESB as a middleware infrastructure ( or as a common information model) could all be a part of EAI union that should help a grand classification of like-minded models. But then,  the proponents of each new model want to see them in isolation, properly insulated from the ‘adulterating  influence’ of their less sophisticated technological ancestors. So things never so happened that SOA had been added to EAI concepts, but rather, SOA has been ‘introduced in place of’ EAI concepts. Of course, only then it’s convenient to present the world with a roadmap so strikingly titled as ‘From EAI-to-SOA’!

My feeling is that EAI is the only term with enough historical and semantic potential to accommodate ESB-based approaches, most of data integration methods, Composite/Federated Applications, Saas, Service Oriented approaches as well as those “inter-silo mashups” ( for which EAI is unfairly synonymous with). I am hardly a name junkie, but I do believe that consistent categorization and evolutionary traceability are important – very important.

Posted in Architecture Process, Data Integration, EA Requirements, Enterprise Application Integration, Enterprise Architecture, IT Architecture | Tagged: , , , , , , , , , , | Leave a Comment »

Integration Methodologies: Melting Frontiers Or Age of Divergence ?

Posted by snair007 on July 18, 2008

Part1 : The Integration Big League

Integration is pretty much what IT architecture is all about. Once we get rid of the ‘fat’  from this area and get to the core of affairs, all that matters is the way components ( data,application,technological ) are fastened together. No wonder Enterprise IT Architecture and Systems Integration were used interchangeably up until a close past.

But then there are a huge lot of methods out there, the purpose of which would rather seem to confuse than to assist. But I am sure there can be no argument as to which of them are most significant – Enterprise Application Integration (EAI) and Enterprise Data Integration (EDI). As a term, EAI has proven to be monumentally more significant than EDI, but when it comes to the activities that either of them deals with, the disparity vaporizes.

In an Enterprise Architecture evolution context, where would they stand tomorrow as two approaches of integration? In a time and age when ‘components’ are made with integration in mind, how relevant (and possible) is it to retain their distinct identity? Are they heading for convergence? If so, which of the two would potentially supercede another (if at all)? Which other integration methods would get encapsulated in either of these two?

Loads of great questions, and let’s not wait to hash them out.

Egg is not the mother of Chicken, or is it?

All serious Enterprise Architecture practicians might’ve noticed how EAI and EDI are often used in affiliation with one another, if not in place of one another. But if we try to ignore this phenomenon as yet another industry misnomer, we need to be careful. I have come across many research papers and books making contradicting remarks when EDI and EAI are referred in context with each other. Some say EDI activities and tools are a part of EAI framework; others speak something quite the opposite. Still others argue that there is not much difference anyway.

And they can all be right, too!

For the time, though, let me jot down my understanding of the most accepted distinctionsbetween EAI and EDI, before dissecting each of them too much:

·          EDI focuses on unifying enterprise data views; EAI enables unifying business processes by blending applications’ capabilities.

·          EDI  is data-centric; EAI is ‘message’ centric.

·          EDI deals with high-volume data movements; data movements in EAI is normally limited to real- time/near real-time data ‘propagation’

·          The technologies used for typical EDI operations are different from those of EAI

·          EAI is normally defined to be integration of data with application or application with application; EDI is to do with data to data integration.

·          In general, data extraction is a ‘pull’ activity in EDI and in EAI; it is a ‘push’ one.

·          Data Integration may not radically alter the systems architecture of an enterprise, but Applications Integration does.

·          At one level down, the frequently talked about sub- types of EDI (EII,MDM,ETL etc) are significantly different from typical activities surrounding EAI

·          EAI can start right from the component development (by ensuring compatibility), whereas EDI is almost always an implementation/interfacing activity.

·          There are more standards, guidelines and protocols to be followed to realize EAI than in the case of EDI. Many EDI efforts can be ad-hoc and tactical.

Sigh! There are quite a few, right?

But as I posited, these ‘differences’ haven’t seemed to restrain many experts from bracketing either of them under the other. Some believe EAI is a part of EDI just like EII,ETL and Data Federation. There are other schools of though that sees no reason why EDI is not a part of EAI framework, which according to them, is immense and flexible enough to accommodate that and much more.

These chickens and eggs!

Let the argument go on; we shall go back to tackle some more important questions by taking a nigher look at each of them.

Implicitly implicit

Interestingly (and quite thankfully), there’s been a clear pattern in the way EAI technologies have evolved overtime. It is fair to say that the definition of EAI has also evolved with them. But then, arguments in favor of placing EDI under the EAI umbralla have been there in all elementary definitions of EAI. Here is how Gartner group defines EAI:

            unrestricted sharing of data and business processes among any connected application or data sources in the enterprise.” (  a clear indication that we can regard           EDI as a subset of operations and framework that EAI offers.)

But the moment we see something like “business process” in its definition, EAI gets much more than a set of application or data components technically welded together. Linking business processes demands a well- orchestrated, business-centric approach.In other words, EAI is the technical consolidation of application silos in accordance with a set of business rules,to satisfy a set of business requirements.

An interesting aspect of EAI is that it can give rise to a set of new functionalities that the component applications wouldn’t provide in isolation. But mind you – these ‘generated functionalities’ are not‘technical accidents’ of EAI, but a result of well thought-out and estimated efforts.But both the expectations as well as the methods around this area have been varying over the years.

Wing and Venky (Enterprise Architecture and Integration,2007) give an interesting picture of how the focus of applications integration efforts have shifted over time – from one of technical to that of business.

They accurately points out that

            “The perception of integration has therefore shifted over time from tactical, low-business-value technical plumbing to a strategic and high-value enabler for     achieving business competitiveness.”

And the roadmap they present has a picture of EAI transmuting from “technical plumbing” to business-enabling frameworks generation.The focus areas making their appearances in the chronological order are systems integration, real-time integration, ERP Integration, Web Integration, Supply chain management, e-business integration,CRM, B2B Commerce and finally – collaborative commerce.

So, the future of integration is decided by business trends and paradigms rather than technological advances in data/message propagation. What does it mean to the question of addressing EAI and EDI as distinct integration streams? Answer seems quite obvious at first glance, now that we have seen the diminishing importance of integration methodologies in a rather business-centric integration environment. But is that truly the case?

To analyse that further, we must have to take a look at the types of integrations falling under each of EAI and EDI.

Let me encapsulate major EDI and EAI categories in two tables :

           

Table-1 : EDI Categories
  EDI Type  What is it ?
 
1 EII

Enterprise Information Integration

  • Multiple DBs, single virtual view
  • Fedarated approach
  • Holistic  Business View
  • Need-driven query redirection
  • “Divide-and-conquer” approach to execute queries. 

 

 

2 MDM

Master Data Management    

  • Multiple business components, single virtual ‘view’.
  • Focus on conformance with “Reference Data”
  • Focus on business consistency
  • Similar to CDI in many ways
  • Depends on a predefined / accepted “business semantics”  
  •  

3 ETL

Extract, Transform and Load

  • Synonymous with Data Warehouses
  • Normally batch- oriented
  • Differed data transfer
  • Normally based on “Pull” extracts
  • Normally uses an intermediate “staging area” for preparation and transformation
  • asynchronous
  • centralized hosting
4 RT-ETL

Real Time ETL

  • ETL process done in real-time or near real time
  • Data Propagation rather than bulk- transfer; similar to EAI
  • Normally based on “push” methods
  • “semi-synchronous”
  • centralized hosting
5 EDR 

Enterprise Data Replication

  • Replication of partial or full database
  • Rarely used as an integration solution; normally used for back-up and duplication
  • Normally use DB- triggers / transaction logs
  • CDC and Data propagation are used   
  •  

6 ECM

Enterprise Content Management

  •  Best suited for integrating data “as-is” / unstructured
  • Used for integration/consolidation of documents, web contents and rich media.
  •  ECM Products normally use a Content Management engine on top of  Data Store
7 EAI (We will see this in detail below)
8 CDI Customer Data Integration
9 CDC

Change Data Capture

  • Not so much of a type of EDI as a technique.
  • Publish-Subscribe mechanism
  •  Normally used for near real-time propagation purposes  

     

  

Table-2 : EAI Categories
  EAI Type  What is it ?
 
1 EDI

(We have seen this in detail)

  •  Integration at data level
  • Whitebox or Blackbox approach.
  • All EDI methods come under this category ( as seen above )
2

Functional Integration (FI)

-AND-

Process Integration (BPI+)

  • Integration at processes and data manipulation / interpretation rules level.
  • “Tight” application-to-application integration
  • Approach can be White / Blackbox
  • Used when presentation integration is inadequate
  • Normally wses any of the following technologies :
Message Oriented Middleware (MOM): integration through messages exchanges     

 Configurations : message queuing and message passing.

Examples : IBM’s MQSeries and Talarian’s Smart Sockets.

Distributed object technology:  Integration through application of object- oriented concepts t o middleware.

Configuration : Software components are made into objects which is then can be accessed by other applications across a network through the object interfaces.

Example : CORBA, MS COM+, J2EE,DCOM,RMI

 Transaction processing monitors (TPMs):   Integration through preserving integrity of distributed information resources (databases, files, MQs).

Configuration :To impliment distributed transaction processing support concepts like two-phase commit.  

 Example : BEA Tuxedo,web services 

 

 

3

Presentation Integration (PI)

-OR-

IUI (User Interface Integration)

  • Integration at applications’ prsentation /UI level
  • Whitebox appoach
  • Improved , Integrated access
  • Integrated user interface management(e.g validation, error handling)
4

Application Interfacing

  • Out-of-the-box Integration
  • Implemented with specializedAPIs, connectors or other forms of interfaces to access application components.
5 BPI

 Improved Functional Integration

  • Blackbox Appoach
  • Normally uses advanced middleware solutions like message brokers
  • Standardized and controlled the flow of information with a bus / hub-and- spoke framework.  

 *black box integration approach hides internal complexities of the application or database. White box approach does it the other way around.

 Next : Part2 : Convergence and Convenience

 

Posted in Data Integration, Enterprise Application Integration, Enterprise Architecture, IT Architecture | Tagged: , , , , , , , , , , , , , , , , | 1 Comment »