Enterprise Architecture Demystified

Sethuraj Nair’s Enterprise Architecture Thoughts

Posts Tagged ‘EDI’

Integration Methodologies: Melting Frontiers Or Age of Divergence ?

Posted by snair007 on July 18, 2008

Part1 : The Integration Big League

Integration is pretty much what IT architecture is all about. Once we get rid of the ‘fat’  from this area and get to the core of affairs, all that matters is the way components ( data,application,technological ) are fastened together. No wonder Enterprise IT Architecture and Systems Integration were used interchangeably up until a close past.

But then there are a huge lot of methods out there, the purpose of which would rather seem to confuse than to assist. But I am sure there can be no argument as to which of them are most significant – Enterprise Application Integration (EAI) and Enterprise Data Integration (EDI). As a term, EAI has proven to be monumentally more significant than EDI, but when it comes to the activities that either of them deals with, the disparity vaporizes.

In an Enterprise Architecture evolution context, where would they stand tomorrow as two approaches of integration? In a time and age when ‘components’ are made with integration in mind, how relevant (and possible) is it to retain their distinct identity? Are they heading for convergence? If so, which of the two would potentially supercede another (if at all)? Which other integration methods would get encapsulated in either of these two?

Loads of great questions, and let’s not wait to hash them out.

Egg is not the mother of Chicken, or is it?

All serious Enterprise Architecture practicians might’ve noticed how EAI and EDI are often used in affiliation with one another, if not in place of one another. But if we try to ignore this phenomenon as yet another industry misnomer, we need to be careful. I have come across many research papers and books making contradicting remarks when EDI and EAI are referred in context with each other. Some say EDI activities and tools are a part of EAI framework; others speak something quite the opposite. Still others argue that there is not much difference anyway.

And they can all be right, too!

For the time, though, let me jot down my understanding of the most accepted distinctionsbetween EAI and EDI, before dissecting each of them too much:

·          EDI focuses on unifying enterprise data views; EAI enables unifying business processes by blending applications’ capabilities.

·          EDI  is data-centric; EAI is ‘message’ centric.

·          EDI deals with high-volume data movements; data movements in EAI is normally limited to real- time/near real-time data ‘propagation’

·          The technologies used for typical EDI operations are different from those of EAI

·          EAI is normally defined to be integration of data with application or application with application; EDI is to do with data to data integration.

·          In general, data extraction is a ‘pull’ activity in EDI and in EAI; it is a ‘push’ one.

·          Data Integration may not radically alter the systems architecture of an enterprise, but Applications Integration does.

·          At one level down, the frequently talked about sub- types of EDI (EII,MDM,ETL etc) are significantly different from typical activities surrounding EAI

·          EAI can start right from the component development (by ensuring compatibility), whereas EDI is almost always an implementation/interfacing activity.

·          There are more standards, guidelines and protocols to be followed to realize EAI than in the case of EDI. Many EDI efforts can be ad-hoc and tactical.

Sigh! There are quite a few, right?

But as I posited, these ‘differences’ haven’t seemed to restrain many experts from bracketing either of them under the other. Some believe EAI is a part of EDI just like EII,ETL and Data Federation. There are other schools of though that sees no reason why EDI is not a part of EAI framework, which according to them, is immense and flexible enough to accommodate that and much more.

These chickens and eggs!

Let the argument go on; we shall go back to tackle some more important questions by taking a nigher look at each of them.

Implicitly implicit

Interestingly (and quite thankfully), there’s been a clear pattern in the way EAI technologies have evolved overtime. It is fair to say that the definition of EAI has also evolved with them. But then, arguments in favor of placing EDI under the EAI umbralla have been there in all elementary definitions of EAI. Here is how Gartner group defines EAI:

            unrestricted sharing of data and business processes among any connected application or data sources in the enterprise.” (  a clear indication that we can regard           EDI as a subset of operations and framework that EAI offers.)

But the moment we see something like “business process” in its definition, EAI gets much more than a set of application or data components technically welded together. Linking business processes demands a well- orchestrated, business-centric approach.In other words, EAI is the technical consolidation of application silos in accordance with a set of business rules,to satisfy a set of business requirements.

An interesting aspect of EAI is that it can give rise to a set of new functionalities that the component applications wouldn’t provide in isolation. But mind you – these ‘generated functionalities’ are not‘technical accidents’ of EAI, but a result of well thought-out and estimated efforts.But both the expectations as well as the methods around this area have been varying over the years.

Wing and Venky (Enterprise Architecture and Integration,2007) give an interesting picture of how the focus of applications integration efforts have shifted over time – from one of technical to that of business.

They accurately points out that

            “The perception of integration has therefore shifted over time from tactical, low-business-value technical plumbing to a strategic and high-value enabler for     achieving business competitiveness.”

And the roadmap they present has a picture of EAI transmuting from “technical plumbing” to business-enabling frameworks generation.The focus areas making their appearances in the chronological order are systems integration, real-time integration, ERP Integration, Web Integration, Supply chain management, e-business integration,CRM, B2B Commerce and finally – collaborative commerce.

So, the future of integration is decided by business trends and paradigms rather than technological advances in data/message propagation. What does it mean to the question of addressing EAI and EDI as distinct integration streams? Answer seems quite obvious at first glance, now that we have seen the diminishing importance of integration methodologies in a rather business-centric integration environment. But is that truly the case?

To analyse that further, we must have to take a look at the types of integrations falling under each of EAI and EDI.

Let me encapsulate major EDI and EAI categories in two tables :

           

Table-1 : EDI Categories
  EDI Type  What is it ?
 
1 EII

Enterprise Information Integration

  • Multiple DBs, single virtual view
  • Fedarated approach
  • Holistic  Business View
  • Need-driven query redirection
  • “Divide-and-conquer” approach to execute queries. 

 

 

2 MDM

Master Data Management    

  • Multiple business components, single virtual ‘view’.
  • Focus on conformance with “Reference Data”
  • Focus on business consistency
  • Similar to CDI in many ways
  • Depends on a predefined / accepted “business semantics”  
  •  

3 ETL

Extract, Transform and Load

  • Synonymous with Data Warehouses
  • Normally batch- oriented
  • Differed data transfer
  • Normally based on “Pull” extracts
  • Normally uses an intermediate “staging area” for preparation and transformation
  • asynchronous
  • centralized hosting
4 RT-ETL

Real Time ETL

  • ETL process done in real-time or near real time
  • Data Propagation rather than bulk- transfer; similar to EAI
  • Normally based on “push” methods
  • “semi-synchronous”
  • centralized hosting
5 EDR 

Enterprise Data Replication

  • Replication of partial or full database
  • Rarely used as an integration solution; normally used for back-up and duplication
  • Normally use DB- triggers / transaction logs
  • CDC and Data propagation are used   
  •  

6 ECM

Enterprise Content Management

  •  Best suited for integrating data “as-is” / unstructured
  • Used for integration/consolidation of documents, web contents and rich media.
  •  ECM Products normally use a Content Management engine on top of  Data Store
7 EAI (We will see this in detail below)
8 CDI Customer Data Integration
9 CDC

Change Data Capture

  • Not so much of a type of EDI as a technique.
  • Publish-Subscribe mechanism
  •  Normally used for near real-time propagation purposes  

     

  

Table-2 : EAI Categories
  EAI Type  What is it ?
 
1 EDI

(We have seen this in detail)

  •  Integration at data level
  • Whitebox or Blackbox approach.
  • All EDI methods come under this category ( as seen above )
2

Functional Integration (FI)

-AND-

Process Integration (BPI+)

  • Integration at processes and data manipulation / interpretation rules level.
  • “Tight” application-to-application integration
  • Approach can be White / Blackbox
  • Used when presentation integration is inadequate
  • Normally wses any of the following technologies :
Message Oriented Middleware (MOM): integration through messages exchanges     

 Configurations : message queuing and message passing.

Examples : IBM’s MQSeries and Talarian’s Smart Sockets.

Distributed object technology:  Integration through application of object- oriented concepts t o middleware.

Configuration : Software components are made into objects which is then can be accessed by other applications across a network through the object interfaces.

Example : CORBA, MS COM+, J2EE,DCOM,RMI

 Transaction processing monitors (TPMs):   Integration through preserving integrity of distributed information resources (databases, files, MQs).

Configuration :To impliment distributed transaction processing support concepts like two-phase commit.  

 Example : BEA Tuxedo,web services 

 

 

3

Presentation Integration (PI)

-OR-

IUI (User Interface Integration)

  • Integration at applications’ prsentation /UI level
  • Whitebox appoach
  • Improved , Integrated access
  • Integrated user interface management(e.g validation, error handling)
4

Application Interfacing

  • Out-of-the-box Integration
  • Implemented with specializedAPIs, connectors or other forms of interfaces to access application components.
5 BPI

 Improved Functional Integration

  • Blackbox Appoach
  • Normally uses advanced middleware solutions like message brokers
  • Standardized and controlled the flow of information with a bus / hub-and- spoke framework.  

 *black box integration approach hides internal complexities of the application or database. White box approach does it the other way around.

 Next : Part2 : Convergence and Convenience

 

Posted in Data Integration, Enterprise Application Integration, Enterprise Architecture, IT Architecture | Tagged: , , , , , , , , , , , , , , , , | 1 Comment »

Agility Requirements: The Groundwork Of Agile Architecture – Part I

Posted by snair007 on July 6, 2008

Agility, Architecture and Agile Architecture

What a good word Agility is!  You are agile only when you are alert and adaptable, both. Hats off to the wise soul – whoever it was – that discovered its utility in ever-wavering frames of Information Technology practices. In any case, it was just a matter of time since its possible coinage before we started seeing its decisive presence in a gamut of IT methodologies and standards – Agile Programming, Agile Design, Agile Application Development and of course Agile architecture.

Why should architecture be agile? Is it advisable to see something that is to be rightly regarded as an authentic reference/blueprint of action to be as a variable in itself? Is it just about architecture change management?

Or to ask more directly- how important is the concept of agility in architecture?

One plain (and proper) definition of agility is as the ability to react to upcoming changes effectively and efficiently (Ambrose, Morello, 2004). In this sense, each architecture effort, if it is not done as a part of a fresh full life-cycle project, is meant to deliver exactly this purpose. It is so important that one can even argue we do architecture for agility rather than with agility.

Implicit and Explicit Agility

We need to know what we have and how it all add up in order to be able to assess the impact of a change so much as to know what is going to change, when, where, how and why it changes. We also need to know the architecture of our systems to know what each change will cost us in terms of money, time and effort. So to have a good architecture is the undoubtedly the first step of being agile. In other words, the greatest objective of Enterprise Architecture is to assist agility. Going by this logic, one may tend to think that Agile Enterprise Architecture is badly phrased; if not that it is an absolute oxymoron. And believe me – this is not entirely wrong. Agility is implicit in IT architecture.

But intention of this being regarded as a major EA topic is good. We normally say Agile Architecture to stress the importance of an architecture that is optimized for changes. So, while capturing the “As-Is” architecture is implicitly and arguably agility-driven, defining the ‘To-Be’ state is much more than that. In the latter case, we strive to devise an architecture that is to respond to changes ‘effectively and efficiently’, rather than to pull the current architecture together to fine-tune our understanding of the system(s) to handle potential changes ’effectively and efficiently’. We should therefore rather focus on this future-state modeling when we talk on Agile Architecture.

Focus On Agility: Agility Requirements – Foresee & Be Agile

Agility is the conceptual parent of many best-loved architecture features like dynamism, vibrancy and some most popular paradigms like SOA. They all deal with agility in their own ways. Agility is everywhere; though it is important to know how much is present where.

We have just seen how architecture can be implicitly and/or explicitly agile. But what do we normally do for making architecture agile? A good enterprise future-state architect definitely keeps the big agility factor in mind, but how often do we address agility factor upfront? Many architecture frameworks overlook considering agility in scoping and planning phases. We need to aim directly at the bull’s eye to increase the probability of striking it – we need to treat agility as the foremost success-criteria upfront for an architecture undertaking to make the implemented system agile. The first and most important step in this direction is to understand the organization’s ‘agility requirements’. When we gather architecture requirements, we normally tend to neglect the need of capturing a set of requirements under the head ‘agility requirements’. But the challenging question is how we can identify our ‘agility requirements’. How can we put down something that is uncertain and fluid? And most important of all, who is going to provide agility requirements?

We will see this in detail in later sections soon, and for the time, we only need to understand that agility as an architectural characteristic that is worth getting prime attention – expressly and systematically.

If we can actually expect changes, we would respond to them more effectively. And this simple logic makes a powerful case for gathering, organizing and representing agility requirements as one separate part of requirements collection phase of the architecture effort. Not only that it saves time, money and efforts, it also places the change in a politically and strategically favorable zone now that the change was an anticipated one. But is it about predicting changes, really? And if so it is, is it practical?

Science and Methods of Agility Requirements

Deductive Method

One way of identifying agility requirements is to understand the actual pattern of your business and technology changes. In most enterprises, there could be a significant degree of commonality across ‘System Change Events’ (SCE, if you like) happening over a period of time, due to business, ‘political’, geographic, economic and other reasons. It is possible to get considerable insights on the nature of ‘expectable’ changes by analysing changes that have been happening for a certain period of time. Presenting a full- fledged methodology for identifying the SCEs and analysing them is beyond the scope of our present discussion, though I would wish to rake up the applicability of some of the classic methods of pattern identification in this area – such as statistical approaches and data mining.

It is quite possible for a good enterprise to keep a history of their key business, application, data and technology changes, which I prefer to classify as System Changes (yes – I understand ‘system’ is a subjective term). In some cases though, the changes would be captured at a very granular level and without any proper plan to place the changes in context with each other to make it architecturally significant. The changes in Network topologies , changes in CI configurations and resources, applications patching and upgrades, adding new types of users to an application or configuration items are all System Changes but not with much use until they are rolled up to a proper higher level. Either way, with a focused and scientific approach, it is possible not only to conform these changes to a certain desired level of abstraction, but also to ‘make sense’ of this data.

So the ‘deductive method’ of agile requirement capture include mainly 2 steps :

      (1) To Prepare: To collect, organize, level-up the System Change Events in order to dress them       up for analysis.

      (2) To Analyze: To apply mathematical-statistical data mining techniques to draw    inferences that are direct inputs to Agile Requirements capture process.

 

What we look out for in an organized set of historical change records is a pattern of occurrence of various categories of changes, so as to be able to make sensible prediction and projections..

Let’s see some examples

Result of SCE analysis:

      There is a pattern in which a particular technology component ‘x’  is rolled out in a particular  geographical entity ‘y’.

The agility requirement corresponding to this finding would be something like:

      ‘To anticipate the implementation of upcoming releases or versions of ‘x’ in ‘y’.

Another One:

      ‘There are cases of recurring failures of implementing a certain strategic directive /standard ‘s’ in     certain client account ‘y’ ‘.

This translates into an Agility requirement statement that goes like:

      ‘To anticipate implementing an alternative to ‘s’ in account ‘y’ ; to keep an alternate blueprint of       implementation just for this account.’

In some cases, the ROI can be found significantly higher with specific changes, which would make way to repeat similar changes in future. Cases like this enable us to make fair, rather direct predictions.

Analysis and consultation

The most obvious method of gathering agility requirements is to do that in a requirement gather session from business representatives and stake-holders as ‘parking lot’ buy-ins. It is not recommended to get very formal with the process, though – as it may sound to be a rather speculative business. There should be enough transparency while asking for expectations. The ‘donor’ of a strategic tip-off may not be ready to own it or vouch for it , and rightly so. But it may prove to be valuable when you do agility requirements analysis.

Another important option is to seek internal or external professional assistance to do keep a tab on industry trends and developments. Agility of the enterprise is proportional to its foresight substantiated by quality trend-analysis, and the architecture responsive to the industry developments always identifies and assimilates trend data in consultation with business experts and analysts who can make quantified predictions as to which business and technology components can undergo changes in a certain period of time.

Next: Agility Requirements: The Groundwork of Agile Architecture Part-2 : Real Actions, Range of Change, Cost , resources and risk

Posted in Agile Architecture, Architecture Process, Enterprise Architecture, IT Architecture | Tagged: , , , , , , , , , , , | Leave a Comment »