Integration Methodologies: Melting Frontiers Or Age of Divergence ?

Part1 : The Integration Big League

Integration is pretty much what IT architecture is all about. Once we get rid of the ‘fat’  from this area and get to the core of affairs, all that matters is the way components ( data,application,technological ) are fastened together. No wonder Enterprise IT Architecture and Systems Integration were used interchangeably up until a close past.

But then there are a huge lot of methods out there, the purpose of which would rather seem to confuse than to assist. But I am sure there can be no argument as to which of them are most significant – Enterprise Application Integration (EAI) and Enterprise Data Integration (EDI). As a term, EAI has proven to be monumentally more significant than EDI, but when it comes to the activities that either of them deals with, the disparity vaporizes.

In an Enterprise Architecture evolution context, where would they stand tomorrow as two approaches of integration? In a time and age when ‘components’ are made with integration in mind, how relevant (and possible) is it to retain their distinct identity? Are they heading for convergence? If so, which of the two would potentially supercede another (if at all)? Which other integration methods would get encapsulated in either of these two?

Loads of great questions, and let’s not wait to hash them out.

Egg is not the mother of Chicken, or is it?

All serious Enterprise Architecture practicians might’ve noticed how EAI and EDI are often used in affiliation with one another, if not in place of one another. But if we try to ignore this phenomenon as yet another industry misnomer, we need to be careful. I have come across many research papers and books making contradicting remarks when EDI and EAI are referred in context with each other. Some say EDI activities and tools are a part of EAI framework; others speak something quite the opposite. Still others argue that there is not much difference anyway.

And they can all be right, too!

For the time, though, let me jot down my understanding of the most accepted distinctionsbetween EAI and EDI, before dissecting each of them too much:

·          EDI focuses on unifying enterprise data views; EAI enables unifying business processes by blending applications’ capabilities.

·          EDI  is data-centric; EAI is ‘message’ centric.

·          EDI deals with high-volume data movements; data movements in EAI is normally limited to real- time/near real-time data ‘propagation’

·          The technologies used for typical EDI operations are different from those of EAI

·          EAI is normally defined to be integration of data with application or application with application; EDI is to do with data to data integration.

·          In general, data extraction is a ‘pull’ activity in EDI and in EAI; it is a ‘push’ one.

·          Data Integration may not radically alter the systems architecture of an enterprise, but Applications Integration does.

·          At one level down, the frequently talked about sub- types of EDI (EII,MDM,ETL etc) are significantly different from typical activities surrounding EAI

·          EAI can start right from the component development (by ensuring compatibility), whereas EDI is almost always an implementation/interfacing activity.

·          There are more standards, guidelines and protocols to be followed to realize EAI than in the case of EDI. Many EDI efforts can be ad-hoc and tactical.

Sigh! There are quite a few, right?

But as I posited, these ‘differences’ haven’t seemed to restrain many experts from bracketing either of them under the other. Some believe EAI is a part of EDI just like EII,ETL and Data Federation. There are other schools of though that sees no reason why EDI is not a part of EAI framework, which according to them, is immense and flexible enough to accommodate that and much more.

These chickens and eggs!

Let the argument go on; we shall go back to tackle some more important questions by taking a nigher look at each of them.

Implicitly implicit

Interestingly (and quite thankfully), there’s been a clear pattern in the way EAI technologies have evolved overtime. It is fair to say that the definition of EAI has also evolved with them. But then, arguments in favor of placing EDI under the EAI umbralla have been there in all elementary definitions of EAI. Here is how Gartner group defines EAI:

            unrestricted sharing of data and business processes among any connected application or data sources in the enterprise.” (  a clear indication that we can regard           EDI as a subset of operations and framework that EAI offers.)

But the moment we see something like “business process” in its definition, EAI gets much more than a set of application or data components technically welded together. Linking business processes demands a well- orchestrated, business-centric approach.In other words, EAI is the technical consolidation of application silos in accordance with a set of business rules,to satisfy a set of business requirements.

An interesting aspect of EAI is that it can give rise to a set of new functionalities that the component applications wouldn’t provide in isolation. But mind you – these ‘generated functionalities’ are not‘technical accidents’ of EAI, but a result of well thought-out and estimated efforts.But both the expectations as well as the methods around this area have been varying over the years.

Wing and Venky (Enterprise Architecture and Integration,2007) give an interesting picture of how the focus of applications integration efforts have shifted over time – from one of technical to that of business.

They accurately points out that

            “The perception of integration has therefore shifted over time from tactical, low-business-value technical plumbing to a strategic and high-value enabler for     achieving business competitiveness.”

And the roadmap they present has a picture of EAI transmuting from “technical plumbing” to business-enabling frameworks generation.The focus areas making their appearances in the chronological order are systems integration, real-time integration, ERP Integration, Web Integration, Supply chain management, e-business integration,CRM, B2B Commerce and finally – collaborative commerce.

So, the future of integration is decided by business trends and paradigms rather than technological advances in data/message propagation. What does it mean to the question of addressing EAI and EDI as distinct integration streams? Answer seems quite obvious at first glance, now that we have seen the diminishing importance of integration methodologies in a rather business-centric integration environment. But is that truly the case?

To analyse that further, we must have to take a look at the types of integrations falling under each of EAI and EDI.

Let me encapsulate major EDI and EAI categories in two tables :

           

Table-1 : EDI Categories

	EDI Type	What is it ?
1	EII	Enterprise Information Integration Multiple DBs, single virtual view Fedarated approach Holistic  Business View Need-driven query redirection “Divide-and-conquer” approach to execute queries.
2	MDM	Master Data Management     Multiple business components, single virtual ‘view’. Focus on conformance with “Reference Data” Focus on business consistency Similar to CDI in many ways Depends on a predefined / accepted “business semantics”
3	ETL	Extract, Transform and Load Synonymous with Data Warehouses Normally batch- oriented Differed data transfer Normally based on “Pull” extracts Normally uses an intermediate “staging area” for preparation and transformation asynchronous centralized hosting
4	RT-ETL	Real Time ETL ETL process done in real-time or near real time Data Propagation rather than bulk- transfer; similar to EAI Normally based on “push” methods “semi-synchronous” centralized hosting
5	EDR	Enterprise Data Replication Replication of partial or full database Rarely used as an integration solution; normally used for back-up and duplication Normally use DB- triggers / transaction logs CDC and Data propagation are used
6	ECM	Enterprise Content Management  Best suited for integrating data “as-is” / unstructured Used for integration/consolidation of documents, web contents and rich media.  ECM Products normally use a Content Management engine on top of  Data Store
7	EAI	(We will see this in detail below)
8	CDI	Customer Data Integration
9	CDC	Change Data Capture Not so much of a type of EDI as a technique. Publish-Subscribe mechanism  Normally used for near real-time propagation purposes

     

  

Table-2 : EAI Categories

	EAI Type	What is it ?
1	EDI	(We have seen this in detail)  Integration at data level Whitebox or Blackbox approach. All EDI methods come under this category ( as seen above )
2	Functional Integration (FI) -AND- Process Integration (BPI+)	Integration at processes and data manipulation / interpretation rules level. “Tight” application-to-application integration Approach can be White / Blackbox Used when presentation integration is inadequate Normally wses any of the following technologies : Message Oriented Middleware (MOM): integration through messages exchanges       Configurations : message queuing and message passing. Examples : IBM’s MQSeries and Talarian’s Smart Sockets. Distributed object technology:  Integration through application of object- oriented concepts t o middleware. Configuration : Software components are made into objects which is then can be accessed by other applications across a network through the object interfaces. Example : CORBA, MS COM+, J2EE,DCOM,RMI  Transaction processing monitors (TPMs):   Integration through preserving integrity of distributed information resources (databases, files, MQs). Configuration :To impliment distributed transaction processing support concepts like two-phase commit.    Example : BEA Tuxedo,web services
3	Presentation Integration (PI) -OR- IUI (User Interface Integration)	Integration at applications’ prsentation /UI level Whitebox appoach Improved , Integrated access Integrated user interface management(e.g validation, error handling)
4	Application Interfacing	Out-of-the-box Integration Implemented with specializedAPIs, connectors or other forms of interfaces to access application components.
5	BPI	Improved Functional Integration Blackbox Appoach Normally uses advanced middleware solutions like message brokers Standardized and controlled the flow of information with a bus / hub-and- spoke framework.

 *black box integration approach hides internal complexities of the application or database. White box approach does it the other way around.

 Next : Part2 : Convergence and Convenience