How to deal the most severe Technical issue in developing Enterprise Scale Software

The background for this post is my experience relate to Software Component Integration for enterprise scale projects. After a component Integration stage in a Software development project, for me the most severe technical issue observed is Performance. The end to end performance realities can not be often tested and measured till the final phases of a enterprise software development project. Once you actually start integration the components only, you will start to realize the planned technical architectures and implementation/deployment activities not achieving the planned performance and consistency target as expected. Always planned situations are very much deviated from the final results, when plans are over judged and planned work is not correctly tracked and measured for control.

For years I have been thinking why many software projects exceeds their budgets, and schedules specially during this last phases of Software development life cycle at large scale (over 100 developers) software projects when we start to do component integrations to produce the final product. Perhaps my observations related such projects are very pessimistic, when it comes to the final stages - close to delivery.

Generally - until few month before delivery, the enterprise applications will be developped as separate components. They will either test separately. However during the Integration phase all those components will be connected together, so that the work collectively appear as an enterprise solution. This is where the most Severe Technical Issues come in to surface.  The issues emerging up may be various and may result undesired outcomes in various terms, bugs, cycle time, look and feel, user friendlyness, etc etc. We would call them performance issues in General. When it is a bug, when it is a change in requirement, when it is an error - integration issue, at some cost as developers we have the confidence,  the things could sorted out some how within few days. But when actual end to end cycle time start to call for the challenge, we know that it is not easy as any thing else.  Even a well planned, and properly managed, Project goes smoothly till the last end. The projects will run – as planned, well with in buffers, according to the critical path or other triple constrain base lines we all agreed. But ultimately once the project is near to be delivered, reached this critical phase, during integration you will see crippled spikes of effort and cost variances are quite deviating than planned. Finally such integration issues generally contradict the performance expectations of client and would result very unpleasant experience.  

The project management knowledge and expertise on integration of software components are found to be very scarce perhaps. This knowledge is different from team to team – project to project, place to place, culture to culture, Technology to Technology. In Project Management perspective both Internal and External forces matter here. There is no wander, why that modern software is developed as components. To meet the deadlines, for future scalability and extensibility, easy to understand and maintain, help to deliver software fast, from the design – the Architects, and other various technical experts, will like to plan the software as distinguished components – so that they could be developed parallel and independently, may be which are scattered distributed in different computers for actual deployment. However once integrate only you will find the software is not performing up to the level of extend you have been thinking.What is the reason, how should you avoid this – in order to make sure a smooth integration.

The thousand probable technical reasons for this may be various. There can be communication deficiencies between different layers (Load balancing layer,  web layer app layer, etc), Middleware used generally need to be optimized considering various perspectives. There may be mechanical deficiencies, hardware deficiencies, scaling issues during the coupling of components. There may be batch processing issues. There may be streaming issues. There may be messaging issues. There may be other technologywise incompatibilities never identified at previous design phases.Various these issues, anomalies are hardly understood till the final phase of the project. Seeing various technical specs, following various standards only – doesn't actually ensure achieving the performance preferred as notes declared. The performance of an integrated system is always unique to the system and environment it has actually deployed on.

Therefore at the first place, during planning adequate buffer need to be in place for the actual integration and testing. Then;  for performance tuning. Even though if you find all the components are working fine, well tested before integration, after the integration only you will  see the integrated components are not performing  to the level of expectation, and giving bugs which never were observed before. Some very critical defects may be pop up during onsite deployment. Seeing these bugs, customer may too feel retarded and un-confident. Some times they may even reject the project and will proceed towards legal steps. This may loosen the future businesses and credibility. Greedy to make fast money, expecting a quick delivery - many project managers, shrink the deadlines, whilst creep the scope.

Therefore, understanding the real severity of proper integration, the project leadership specially need to be keen to avoid conflicts between different technical teams at integration. It is natural if they will start to blame each other team –delaying, arguing and worsening the situation. The technical leadership is criticized and blamed for improper design and planning, then their moral, may be dying at the most wanted time. Actually these issues are hard to tackle sometimes from the design only. Issues are realized only during the integration.  May be even only at deployment. Therefore leadership at the time of integration have to be very sensitive and well understood regarding the turbulent time arise unexpectedly at the integration phases. I have even observed, many lead profile sometimes tend to leave the projects, falling the issue into a further utter crisis. The replacement teams will therefore have to learn the premature software, code – which is not well documented, not adequately tested, not adequately maintained – hence perishing the situation, irritating the technical crowd.

Technically – at enterprise scale developments, we are observing that different vendor technologies are used at various levels of component communication. This happen during the middle of the development phase. As the vendors always want to change the requirement and bring additional functionalities. Since the business change fast, they would want to mount new features fast which were not in the scope during planning and designing stages. Such need of quick alterations during implementation will intrude unwanted heterogeneity and complexity to the systems. It is always best, well understood – if we can maintain a kind of homogeneous stature by default among the components.  For example you may find web services are both Synchronous, Asynchronous,  you may find different middle ware technologies are used between diverse components, you may find various technologies are used for XML parsing at different ends of the some product, you may find various versions of software 3^rd party libraries are used by different teams, etc. These silly and deliberate avoidable technical complexities and heterogeneous must be well identified during planning.  A proper communication workout plan needed to be enacted between the diverse teams. Otherwise it would be tricky and very time consuming to debug the multiple complexities and performance drops hence finding  the exact root cause at the integrated times.

The project management best practice in business should be to avoid such critical failures by tailoring existing process for software development and establish a mature Integration process to the Project Governance. This must always account the experience of previous projects and matured best practices with in the organization. Using this experience, existing resources, at the very initial stages of the project planning – the leadership team must tailor a well-defined integration process, along with the integration plan. It could be altered during the various stages of the software development life cycle. However – these process and plans need to be well communicated and sharply accommodated to all levels of technical team members at its very early stages. A good integration plan may include considerations, guidelines definitions and timelines considering Integration Sequence, integration environment, integration procedures, integration criteria, defining the build series etc. 


Above all, there is a huge responsibility for the marketing or business teams, who do talk the business, and finally making the agreements. There you must not give false promises to the client - specially in terms of cycle time of single operation. Generally the business discussions ends, with out quantifiable agreement on performance - per a single transaction. People forget performance during planning. But once the project start, developed and test, the performance come in to play. Performance must be given a priority, and performance expectation should be documented, once before the contracts are done. Such agreements should also incorporate fail in which terms. Other wise both customers, and develpers, going to pay for the hard life. The business relationships can be irritated and ultimately may end up courts - wasting time and good will.