Enterprise Architecture Demystified

Sethuraj Nair’s Enterprise Architecture Thoughts

Posts Tagged ‘Federated Architecture’

Integration Methodologies: Melting Frontiers Or Age of Divergence ?

Posted by snair007 on July 18, 2008

Part1 : The Integration Big League

Integration is pretty much what IT architecture is all about. Once we get rid of the ‘fat’  from this area and get to the core of affairs, all that matters is the way components ( data,application,technological ) are fastened together. No wonder Enterprise IT Architecture and Systems Integration were used interchangeably up until a close past.

But then there are a huge lot of methods out there, the purpose of which would rather seem to confuse than to assist. But I am sure there can be no argument as to which of them are most significant – Enterprise Application Integration (EAI) and Enterprise Data Integration (EDI). As a term, EAI has proven to be monumentally more significant than EDI, but when it comes to the activities that either of them deals with, the disparity vaporizes.

In an Enterprise Architecture evolution context, where would they stand tomorrow as two approaches of integration? In a time and age when ‘components’ are made with integration in mind, how relevant (and possible) is it to retain their distinct identity? Are they heading for convergence? If so, which of the two would potentially supercede another (if at all)? Which other integration methods would get encapsulated in either of these two?

Loads of great questions, and let’s not wait to hash them out.

Egg is not the mother of Chicken, or is it?

All serious Enterprise Architecture practicians might’ve noticed how EAI and EDI are often used in affiliation with one another, if not in place of one another. But if we try to ignore this phenomenon as yet another industry misnomer, we need to be careful. I have come across many research papers and books making contradicting remarks when EDI and EAI are referred in context with each other. Some say EDI activities and tools are a part of EAI framework; others speak something quite the opposite. Still others argue that there is not much difference anyway.

And they can all be right, too!

For the time, though, let me jot down my understanding of the most accepted distinctionsbetween EAI and EDI, before dissecting each of them too much:

·          EDI focuses on unifying enterprise data views; EAI enables unifying business processes by blending applications’ capabilities.

·          EDI  is data-centric; EAI is ‘message’ centric.

·          EDI deals with high-volume data movements; data movements in EAI is normally limited to real- time/near real-time data ‘propagation’

·          The technologies used for typical EDI operations are different from those of EAI

·          EAI is normally defined to be integration of data with application or application with application; EDI is to do with data to data integration.

·          In general, data extraction is a ‘pull’ activity in EDI and in EAI; it is a ‘push’ one.

·          Data Integration may not radically alter the systems architecture of an enterprise, but Applications Integration does.

·          At one level down, the frequently talked about sub- types of EDI (EII,MDM,ETL etc) are significantly different from typical activities surrounding EAI

·          EAI can start right from the component development (by ensuring compatibility), whereas EDI is almost always an implementation/interfacing activity.

·          There are more standards, guidelines and protocols to be followed to realize EAI than in the case of EDI. Many EDI efforts can be ad-hoc and tactical.

Sigh! There are quite a few, right?

But as I posited, these ‘differences’ haven’t seemed to restrain many experts from bracketing either of them under the other. Some believe EAI is a part of EDI just like EII,ETL and Data Federation. There are other schools of though that sees no reason why EDI is not a part of EAI framework, which according to them, is immense and flexible enough to accommodate that and much more.

These chickens and eggs!

Let the argument go on; we shall go back to tackle some more important questions by taking a nigher look at each of them.

Implicitly implicit

Interestingly (and quite thankfully), there’s been a clear pattern in the way EAI technologies have evolved overtime. It is fair to say that the definition of EAI has also evolved with them. But then, arguments in favor of placing EDI under the EAI umbralla have been there in all elementary definitions of EAI. Here is how Gartner group defines EAI:

            unrestricted sharing of data and business processes among any connected application or data sources in the enterprise.” (  a clear indication that we can regard           EDI as a subset of operations and framework that EAI offers.)

But the moment we see something like “business process” in its definition, EAI gets much more than a set of application or data components technically welded together. Linking business processes demands a well- orchestrated, business-centric approach.In other words, EAI is the technical consolidation of application silos in accordance with a set of business rules,to satisfy a set of business requirements.

An interesting aspect of EAI is that it can give rise to a set of new functionalities that the component applications wouldn’t provide in isolation. But mind you – these ‘generated functionalities’ are not‘technical accidents’ of EAI, but a result of well thought-out and estimated efforts.But both the expectations as well as the methods around this area have been varying over the years.

Wing and Venky (Enterprise Architecture and Integration,2007) give an interesting picture of how the focus of applications integration efforts have shifted over time – from one of technical to that of business.

They accurately points out that

            “The perception of integration has therefore shifted over time from tactical, low-business-value technical plumbing to a strategic and high-value enabler for     achieving business competitiveness.”

And the roadmap they present has a picture of EAI transmuting from “technical plumbing” to business-enabling frameworks generation.The focus areas making their appearances in the chronological order are systems integration, real-time integration, ERP Integration, Web Integration, Supply chain management, e-business integration,CRM, B2B Commerce and finally – collaborative commerce.

So, the future of integration is decided by business trends and paradigms rather than technological advances in data/message propagation. What does it mean to the question of addressing EAI and EDI as distinct integration streams? Answer seems quite obvious at first glance, now that we have seen the diminishing importance of integration methodologies in a rather business-centric integration environment. But is that truly the case?

To analyse that further, we must have to take a look at the types of integrations falling under each of EAI and EDI.

Let me encapsulate major EDI and EAI categories in two tables :

           

Table-1 : EDI Categories
  EDI Type  What is it ?
 
1 EII

Enterprise Information Integration

  • Multiple DBs, single virtual view
  • Fedarated approach
  • Holistic  Business View
  • Need-driven query redirection
  • “Divide-and-conquer” approach to execute queries. 

 

 

2 MDM

Master Data Management    

  • Multiple business components, single virtual ‘view’.
  • Focus on conformance with “Reference Data”
  • Focus on business consistency
  • Similar to CDI in many ways
  • Depends on a predefined / accepted “business semantics”  
  •  

3 ETL

Extract, Transform and Load

  • Synonymous with Data Warehouses
  • Normally batch- oriented
  • Differed data transfer
  • Normally based on “Pull” extracts
  • Normally uses an intermediate “staging area” for preparation and transformation
  • asynchronous
  • centralized hosting
4 RT-ETL

Real Time ETL

  • ETL process done in real-time or near real time
  • Data Propagation rather than bulk- transfer; similar to EAI
  • Normally based on “push” methods
  • “semi-synchronous”
  • centralized hosting
5 EDR 

Enterprise Data Replication

  • Replication of partial or full database
  • Rarely used as an integration solution; normally used for back-up and duplication
  • Normally use DB- triggers / transaction logs
  • CDC and Data propagation are used   
  •  

6 ECM

Enterprise Content Management

  •  Best suited for integrating data “as-is” / unstructured
  • Used for integration/consolidation of documents, web contents and rich media.
  •  ECM Products normally use a Content Management engine on top of  Data Store
7 EAI (We will see this in detail below)
8 CDI Customer Data Integration
9 CDC

Change Data Capture

  • Not so much of a type of EDI as a technique.
  • Publish-Subscribe mechanism
  •  Normally used for near real-time propagation purposes  

     

  

Table-2 : EAI Categories
  EAI Type  What is it ?
 
1 EDI

(We have seen this in detail)

  •  Integration at data level
  • Whitebox or Blackbox approach.
  • All EDI methods come under this category ( as seen above )
2

Functional Integration (FI)

-AND-

Process Integration (BPI+)

  • Integration at processes and data manipulation / interpretation rules level.
  • “Tight” application-to-application integration
  • Approach can be White / Blackbox
  • Used when presentation integration is inadequate
  • Normally wses any of the following technologies :
Message Oriented Middleware (MOM): integration through messages exchanges     

 Configurations : message queuing and message passing.

Examples : IBM’s MQSeries and Talarian’s Smart Sockets.

Distributed object technology:  Integration through application of object- oriented concepts t o middleware.

Configuration : Software components are made into objects which is then can be accessed by other applications across a network through the object interfaces.

Example : CORBA, MS COM+, J2EE,DCOM,RMI

 Transaction processing monitors (TPMs):   Integration through preserving integrity of distributed information resources (databases, files, MQs).

Configuration :To impliment distributed transaction processing support concepts like two-phase commit.  

 Example : BEA Tuxedo,web services 

 

 

3

Presentation Integration (PI)

-OR-

IUI (User Interface Integration)

  • Integration at applications’ prsentation /UI level
  • Whitebox appoach
  • Improved , Integrated access
  • Integrated user interface management(e.g validation, error handling)
4

Application Interfacing

  • Out-of-the-box Integration
  • Implemented with specializedAPIs, connectors or other forms of interfaces to access application components.
5 BPI

 Improved Functional Integration

  • Blackbox Appoach
  • Normally uses advanced middleware solutions like message brokers
  • Standardized and controlled the flow of information with a bus / hub-and- spoke framework.  

 *black box integration approach hides internal complexities of the application or database. White box approach does it the other way around.

 Next : Part2 : Convergence and Convenience

 

Posted in Data Integration, Enterprise Application Integration, Enterprise Architecture, IT Architecture | Tagged: , , , , , , , , , , , , , , , , | 1 Comment »

Enterprise Architecture : How Not To Miss Anything ? -Part II

Posted by snair007 on June 21, 2008

“Getting Meticulous About It…..”

EA and EA Projects – Federated Scoping

Sorry, I can’t help harping on (I tried, though) the great ‘problem of rendition’ in EA. Right from the definition through the details, there is enough room for ‘convenient interpretation’ of many terms and ideas we find in countless Enterprise Architecture standards, frameworks and guidelines. Granted – the end-to-end roadmaps like the ones offered by TOGAF ADM are jobs very well done, but confusion lingers on in some less cheerful corners of every framework. A good example would be the difference in approaches to EA and to what can be termed as EA projects.

Wait a minute – did I say EA project? What can be termed as an EA Project? Well – everything you actually do as a part of ‘conducting’ Enterprise Architecture (You heard it right – “Conducting Enterprise Architecture” are the exact words used in the definition of MODAF. Not Conducting EA project, nor Conducting EA Activity – just conducting EA. They say MODAF is a “standardized way of conducting Enterprise Architecture and…”). So an EA Project can be any activity with enough significance to alter the EA status-quo as much as much as  being any component activity of building up Enterprise Architecture or being one that is formally aligned with Enterprise Architecture governance mode,.

The last statement, however, adds an improper degree of flexibility and thus sets in motion the wheels of convenient interpretation.

Out of Governance = Out of Scope?

Can we really take into account something that is not governed by Architecture Board as a part of Enterprise Architecture? Practically – yes , because practically in all organizations, there could be activities significant enough to disturb the EA status-quo without getting adequately controlled by governing council. The process of ‘adoption’ to EA framework would only be done at a later stage as a separate activity in such cases – mostly due to pressing deadlines, cash-flow reasons or simply sheer holes in the framework.

That leads us to having a federated approach to the scoping activity too – no matter at which rung of evolution we are at in terms of EA Framework compliance. What it means to us is this: apart from having a firm traction on governed EA Activities, there is a need for considering all substantial undertakings with definite inputs and outputs happening within the enterprise to be an EA Project; the substantiality can ideally be evaluated by governing body with minimal upfront involvement.

An EA Project can thus have a management and control outside Architecture Board. As shown in the graphic, Adoption and integration of the project to EA Framework are controlled by a pre-defined adoption strategy.

EA and EA Projects

 

But how are all these important in defining scope? They are very important, as, for one, we just saw that a project can be conceived, planned, scoped, executed and managed independently and can still be considered as an EA effort. Setting up right boundaries for such an isolated effort would be the key to all right things to follow.

 

The long and short of it: Reductionism in Scoping

Think of a sprint event. Which one would you pick as of more consequence- the distance to be covered in the race or each step that is to take you till the finishing line? Like most of you, I too would say : both. Every step is as much important as the distance to be covered. Though done inadvertently and being mastered by practice, every breath taken in, every muscle movement and every hop has its role in deciding the winner.

No holistic framework can be sound enough when it is frail in parts. Assembly may be fantastic but a week link in the chain can spoil the show, and that is where the importance of federation lies. In both the phases of scoping and planning an EA Project, the architect (an individual or a body) should be conscious of the Enterprise Significance of its outcome.

This gives a whole new importance to the scoping activity. Here we are not only concerned with scoping of the implementation of EA as a framework, but also of individual project scope in context with EA. So, the scope of the project should be defined as the scope of the business, application and technology areas of the enterprise as a whole that will be impacted by the project as well as those that constitute the project in isolation. This activity can prove to be providing great inputs to planning phase.

 

There are more to “TOGAF dimensions”

Back to the question of ‘meta’ scoping. What should we expect out of scoping exercise? TOGAF talks about 4 different dimensions ( details to be captured/ defined as a part of scoping exercise) : Scope of the Enterprise, Scope of Domains, Scope of Details, Scope of Time (not in the exact words used by The Open Group- refer TOGAF 8.1 for details ).

But are they enough?

Most of us who have ever even tried our hands on an Enterprise-Wide projects would not probably defer when I venture to say they are not all. There are some 4 more dimensions that I can think of:

Scope of Inputs:  A finite line should be drawn to mark the boundaries of information (required + available) concerning the effort. Identifying how much information would be open and/or available as inputs to the architecture effort is vital for successful execution of an EA Project . There can be numerous constraints for the Architecture Body in gathering information related to the project – like security, availability, time considerations and accessibility. On top of it, this effort should address the need of distinguish between available information and required information.

Scope of Compliance: Many Enterprise Architecture efforts are “cross-standard”. For example, while the actual development process of the architecture is being carried out in alignment with TOGAF, the security aspects of the enterprise may need to comply with SOX even as the implementation is to be carried out in an environment that follows ITIL-way of functioning. A ‘matrix of accord’ needs to be in place at the scoping phase of the project in order to meet the standards set by a multitude of standards that may or may not have requirements at variance with one another ( there should not be any, ideally).  

Scope of Usage: This is a highly overlooked factor, especially in Federated Architecture environments. As I mentioned in the previous part of my Thoughts, the effort here is to systematically capture answers to the questions like who is going to take up the architecture from the point where you leave, what will they be doing with the deliverable and whether it is development-facing or the business-facing or both.

Scope of Approval: Not everything is to be approved by everybody. Review and approval of the Enterprise Business Model should be limited to the business reps, whereas Application and Technology Models should follow a different/other review stream. I don’t very much buy in to the idea that this should be a part of defining roles and responsibilities (under planning) and not scoping, for this activity is more to do with defining boundaries of ‘jurisdiction’ rather than assigning work.

Posted in Architecture Process, Enterprise Architecture, IT Architecture | Tagged: , , , , , , , , , , | 1 Comment »