How We Develop Software – Creating a New System/Application Platform

As follow up on my last post on “SDLC Methodology Styles”; in this article we are going to discuss my method of creating a Foundation Platform for the development of a new System or Application.

It is my opinion that for a Software Development Manager to be successful, they need to be intimately involved in the creation of the foundation and frameworks that their development team will use going forward to create and expand the system. I believe in Leading through Example, and that means that your Team Leads and even yourself need to not only be involved in the architecture and design, but the implementation itself (Yes, that means I believe a good manager of a successful system needs to at least at the very beginning write code).

The key to building a successful system lays in the foundation of the code base and it’s overall organization. A successful Development Manager will require their team to adhere to specific design principals, third party tools, and an implementation strategy that the Development Manager needs to set at the very start of a new project. A project starts to crack once developers start injecting into the code base and approved third-party library set new frameworks or large scale third party mechanisms using 5% or less of what the library provides to build a new feature for the application. The reason a develop would normally do this is just to gain experience with a product, so that they can add it on to there resume. A successful Development Manager needs a trusted set of team leads to have a watchful eye over what the developers check in to the code base, to avoid this Resume Building Code Base Pollution.

My own answer to developers that want to learn a new product is to do it on their own time, I’ll even allow developers to “evaluate” new products and libraries during work hours so long as it’s during their down-time. My belief is that even in a high active development project there’s always periods of down-time for each develop as it’s the nature of large teams and OOP in general.

Setting Design Strategy

At the very start of a new System or Application, there MUST be a design strategy set for each of the following areas:

• Commons
• Batch Processes
• Standalone Daemon Processes
• Middleware APIs
• Messaging
  • Publishers
  • Listeners
• User Interfaces (Depending on the project one or more of the following)
  • Web Applications
  • Mobile Platforms
  • Desktop Clients

But before we start talking about each of these development areas, I want to focus on what I feel is the MOST important aspect of designing an architecture. That is what I refer to as “Resource Management”.

Resource Management

What I consider “resources” that are critical to control and deal with across all aspects of an application are: Configurations, Database Connections, and Out-of-Band Communications.

Configurations can be anything from simple Name-Value Pair Properties, or complex XML documents. The problem is that everyone usually needs to store configuration data for a process or function to run correctly outside of the binary itself. The issue I feel that needs to be solved by a robust architecture is how Configurations are loaded by any process or component, and ensuring that this method is reused easily across all tiers of an application.

I think Database Connections are self-explainatory, however obtaining the connections is the critical point, which I feel must be consistent throughout an architecture. For example in a multi-tier application, connections to databases can be obtained via Connection Pools in Enterprise Application Servers (Container Servers such as WebLogic, WebSphere, JBoss), or when working outside of Containers, working directly with Drivers or Driver Managers to obtain a direct connection to a database. I usually create an abstraction layer, so that a developer working on a new Middleware API or a Batch Process doesn’t know if their are working with Connection Pools or Direct Connections. In the passed a lot of developers create ODBC, ADO, ADO.net or JDBC wrappers that everyone uses in a particular project. Because this was such a common process, a lot of open source solutions have popped up, such as iBatis/MyBatis, Hibernate, and other ORM tools.

Personally I’m not a fan of ORM tools, and I think applications with large or complex data models are better off writing direct SQL or Stored Procedures and interacting directly with the database via JDBC or ODBC, or DBI for Perl, etc… Usually one of my biggest requirements when I write a Job Description for a new hire for my own teams is that they know SQL and Direct JDBC (without and ORM frameworks). But this is a topic for another article.

Finally, I believe a robust architecture that provides services for Resource Management, include a method for functions and/or components to send data between each other in an Out-of-Band manner. A lot of current scalable architectures call for Stateless designs, which means usually sending database from one component to another has to rely on method arguments and return parameters. However sometimes to simplify the passing of data, we as developers want to naturally fall back on Class or Object Fields or “Global Variables”. However sometimes this can cause scalability issues, or if not designed carefully multi-threading issues, specially when a developer is creating a component that will be used in an Application Server and threads are implied. A robust architecture can allow for “global” like data to be transient in regards to having a lifetime relating to the call stack, but still shared safely between all different levels of the call stack and other components.

In my Architecture called the Data Services Framework, which we will cover in a future article, I combine all three of these key resources, Property Management, Database Connection Abstraction, and Out-of-Band data transport, into a single structure called a “Resource Bundle” which a new instance of one is passed to each business logic component when a new invocation on a middleware API occurs. I have also created something known as the “Standalone Resource Helper” which allows processes running outside of a Container Server, such as Batch Processes or Standalone Daemons to be able to obtain an instance of a resource bundle, so both Middleware, and Standalone processes can both deal directly with resource bundles, instead of trying out how to read and store properties, and obtain database connections.

By setting a design strategy at the beginning of a system may take more time to get developers implementing the targeted Application at full speed, but it will ensure that the code produced by many difference developers each having their own unique approach and conventions at writing code, all contribute to a code base that is easily maintainable, extended, and worked on by all individuals, including new team members in the future. It creates a code based that is Manageable.

Commons

Most developers when they hear the word “Commons”, think Apache Commons or something similar. However when I use the word and concept of “Commons” in my projects, I mean a separate module or directory in the Source Code Control Repository (SVN, GIT, CVS, etc), that contains common utilities and frameworks used by all other modules within the Application’s Source Tree. It can contains simple things like a custom “StringUtils” class which contains commonly reused String Manipulation functions, to larger scale mechanisms such as SQL Result Set Paging Systems, or Socket Wrapper Libraries. The goal of the Commons is to encourage the creation of reusable components both large and small by the entire development team so that we have consistent implementations of varied business functions using a robust and maintained common component set that may be highly customized for a particular organization or project. I normally encourage my own developers to constantly look for opportunities to contribute to our Commons; if they see a function or component that is likely going to be written again for a separate business requirement, I ask that they try to create an abstract reusable component and then customize it for their use case and add it to the Common’s source tree. The easiest example of this is String Utility functions, or SQL Utility functions, I always ask that if you create an interesting utility method dealing with Strings or SQL Result Sets, SQL Statements, etc, to add it to the commons instead of directly embedding it in your code.

Your own implementations of Resource Management, for me specifically my Resource Bundle Framework is probably the first component that needs to be build and is the most important component of the Commons module of any project following my design strategy. When I first come onboard as an Architect, Head Development Lead or Development Manager, having this framework build is the very first thing I do when the development phase of a project begins. Usually I take the time between meetings with the Business Analysts, Users, and Project Management Office teams during the phases before the actual development phase of a project to develop this component. BA’s, Users, and sometimes even management, won’t see direct value in developing a robust Resource Management implementation, so it is up to you as a Development Manager or Architect to ensure this component gets built; trust me, getting something like this on the project plan, will save you a lot of future grief.

A robust Resource Management framework is the key to creating Stable, Scalable, Flexible, Extendible, and easily Maintainable Systems and Applications!

Batch Processes

Batch Processes are usually deployed on a backend Linux or Unix Server (although it can be on Windows as well), which are executed via a Scheduler. A simple one that every Unix programmer knows is Cron. There are also commercial and open source Schedulers such as Computer Associates‘ Autosys that are much more robust, and allows for small scripting languages (such as Autosys’s JIL), that enable developers not only to run jobs on a time based schedule, but also using some logic, such as detecting the failure or success of other jobs running from the scheduler, and therefore taking appropriate actions.

A Development Manager must design an approach to handling Batch Processes. In my mind, the first thing that must be done is creating an easy to follow startup procedure for each Process the developers will write. This may sound simple, but the worst thing that I have seen in my professional career is when a medium to large development team has a different startup procedure for each of their individual team members. Usually half the problem with having a developer debug or maintain another developer’s batch process is trying to figure out how to start the thing. If you can’t get it running for a couple of days you really can’t start the debugging process, delaying a potentially critical release.

Enforcing that all batch processes use your Resource Management framework helps to ensure that processes will have similar startup processes, as most process’s startup procedures involves bootstrapping the process with configurations, database connections, etc.

As a positive side affect of using something like Resource Bundles to pass around connections and configuration data, you will soon find that components such as Data Access Objects can be easily reused between both Batch Processes and Middleware Components.

Standalone Daemon Processes

The only real difference between a Batch Process and a Standalone Daemon Process in my mind is that a Batch process usually runs on a schedule, it starts at a specific time or a combination of a specific time and an event occurring, and it stops once it finishes processing a finite set of data.

In the case of a Standalone Daemon Process, the idea is it start up at some point, usually say on a Sunday morning, and it runs continually, processing data a random times, depending if events occur such as a messaging arriving in a queue, or a file arriving in a public FTP/SFTP directory, for which the process is watching. And this process doesn’t stop unless the system owners choose to manually stop it or invoke some programmatic shutdown method intended to bring the process down for weekly or monthly server maintenance.

I’m really not going to spend too much time on this section, because a Standalone Daemon should follow the same design strategy as a Batch Process, especially making use of the Resource Management and Startup procedure, however the one addition which I think a robust architecture must have is how a process “becomes” a daemon.

Usually, it’s done via some type of event loop which never ends until some signal for shutdown occurs. This loop can have certain conventions set, as well as the shutdown procedure, so that all Daemon Processes within a System work the same way. Like the startup procedure, we spoke about in the Batch Process section, it’s all about maintenance. You don’t want to waste a lot of developer cycles trying to figure out how the daemon process remains running. Having a common convention and set of utilities, such as perhaps even abstracting the event loop itself will ensure any developer on your team once familiar with a single Daemon Process, can work on any other daemon process in your system.

Middleware APIs

In the Java world, it’s always easier to find good Core Java developers then JavaEE/J2EE developers. And in my opinion you have to be a good Core Java developer to be a JavaEE developer anyway. It always amuses me when a candidate on a technical interview prefixes an answer to a question about a core concept such as Collection as they re “rusty” because they are a JavaEE developer… What does that even mean? Business Logic is always in Core Java! It makes no sense to call yourself a JavaEE developer. In fact if you apply for a Java developer position are don’t consider yourself a Core Java developer, you need not apply (at least that’s my opinion)!

Ok, we got a little off topic, but what I stated above leads into my core design strategy for Middleware APIs. I like to implement an architect that abstracts the developers from having to deal with any of the EJB, SOAP, or other RPC concepts of JavaEE. I do this again in my Data Services Framework architecture, but right now all you need to know is that I believe in creating an architecture that allows developers to focus 99% of their development time on implementing the business logic or the objective of the business requirements, not worrying about the plumbing.

Over the years I have refined a design over the cause of 10 years which actually allows developers to create and run Middleware APIs from unit test classes right out of an IDE such as Eclipse without having to build and deploy the middleware to a container server such as WebLogic and without remote debugging! Their code is automatically included in the build process which will deploy it to the contain application server without a single line of code change! This is what my Data Services Framework does, and is exactly why I’m saving it for it’s own article.

A successful Development Manager MUST create an architecture or at least a design convention for each of their Middleware APIs to follow. This will simplify maintaining these APIs over time, and if done in a certain way, such as leveraging the Resource Management / Resource Bundle design I have mentioned in this article, a lot of code can be reused by non-middleware components.

Messaging – Publishing / Listening

There are two method of creating publishers and listeners. One method which I am strongly against is writing publishers or listeners that are deployed as components within an Application Server. Instead I mandate that all publishers and listeners (except Messaging Driven Beans) must be written as standalone daemon processes. This usually means that there has to be some mechanism for transferring data from middleware APIs to publishers running as separate processes.

Most times this is done via an event table in the database, and the publisher process includes some type of database table poller, which constantly reads the event table looking for new events to send out as messages.

In the case of Listeners, it really depends on if you are using a listener as a device to update your database from upstream or source systems automatically in real-time without user intervention, or if you are using listeners as a RPC (Remote Procedure Call) mechanism, for external systems to interact with your system components programmatically in real-time via messaging instead of an API approach like SOAP or RESTful Web Services. But in either case I keep these listeners as external standalone daemon processes. In the case of the real-time database loader, there’s no question about how this works, it just executes SQL, a Stored Procedure, or a DAO method each time a message arrives. In terms of the RPC usage of a Listener, I treat these as proxies to Middleware APIs, basically my listener will call the API on behalf of the publishing client, each time a new method arrives.

The benefit of keeping publishers and listeners outside of the Application Server is that they are more stable and scalable in my experience. Especially in the case of persistent or reliable messaging, these types of publishers and listeners have things such as Ledger files or some type of non-volitile storage backing the in memory queues, so that messages are not lost, and occasionally these storage mechanisms either get overloaded or otherwise get corrupted, and it’s usually a lot easier to deal with if you went through the pain of creating event tables in order to republish outgoing messages or reprocess incoming messages when production support issues arise. Also there are special considerations you have to deal with when your Application Contain Severs are running in a multi-node clustered environment. Sometimes you have to bind your listener to a single node, and the fail-over procedure in that type of environment becomes much more complex. Same is true for publishers in a multi-node clustered environment. Usually to ensure the ordering of data you need to only have a single sender publishing at any one time; so which node in the cluster publishes?

All this is removed by creating Publishers and Listeners as standalone processes. It’s sometimes a little more work upfront, but it’s worth it in the end.

finally, since all Publishers and Listeners are forms of Daemon Processes, the event loops, etc, which I mentioned in the section on Standalone Daemon Processes should be adhered to when developing these types of processes.

User Interfaces – Web Apps, Mobile Apps, Desktop Clients

I consider myself more of a Server Side Developer than a Client or UI Developer. However you can not discount User Interfaces when designing your system architecture. This is a fatal flaw I have seen in a lot of projects when the Managers start out on the Server or Batch side and look at User Interfaces as the nice add on of their system for the users to use. However leaving the User Interface as an afterthought like this can cause you to mis-design other aspects of your application such as the Middleware APIs and even the Data Model.

How I like to split the team is a Server Side development branch, which can build Middleware, Batch, Database, etc, and a separate branch of the team for User Interfaces. The reason for this is that it take a special set of skills to develop good User Interfaces. It’s somewhat of an Art rather than a science. And based on my professional and personal experience, you usually need to hire specific UI developers if you want your system to be a success. Also if the budget allows for it, I also feel you should hire Designers separate from the actual UI developers to design the templates and screen layouts used in the UI.

From an architectural standpoint one of the most important aspects of the User Interface on Day One, is the Client Library of the Middleware. I believe the middleware development team should wrap the middleware APIs in a client library in the native language which the client uses. This usually is a thin Facade (or wrapper) around SOAP Stubs or RESTful Web Service.

As most modern front-end architectures I believe in the N-Tier architecture, where you minimally have a Front-End, a Middleware, and a Database. All business logic, data access, even validation logic (other than simple syntax validation) should be embedded in the Middleware, I call this being UI-Agnostic.

Being UI-Agnostic allows you to build multiple front-end such as a Web Application, a Desktop Client, and Mobile Apps for different mobile platforms all leveraging the same middleware without much if any at all code duplication for the business logic, data access, and validation logic layer.

Also, although this is becoming less the common case and more of the exception, since Server and Front-end environments are becoming more heterogeneous then ever before, especially with the mobile platform, if your front-end is written in the same language as your Middleware and Batch, I would enforce that the User Interface developers use the same Commons as the server side developers. This is more easy say with Traditional Web Apps in the Enterprise, where you might have a Java middleware and Java based web front-ends.

Creating Robust Enterprise Systems

What is a Robust Enterprise System? It is any system which is designed to be Stable, Scalable, Flexible, Extendible, and easily Maintainable (SSFEM). By creating an architecture and a common set of utilities at the very onset of your projects, you will help to ensure that you have a robust enterprise system. In future articles we will discuss specific architectural design I believe enable Systems to be SSFEM. If you can do this in your career you will not only be a successful Development Manager or Architect or Developer, but you will also have pride in your systems, which will be in use for many years to come, perhaps even decades. The goal I always have is to design systems that have the capability of lasting between 10 and 20 years. People may think that in these times where technology is changing faster than any of us even in the industry can keep up, that talking about systems that last this long is absurd, but if the systems you build are SSFEM, you will find that it is cheaper to extend the system to meet the needs of the business than for the business to just replace them system.

Just Another Stream of Random Bits…

– Robert C. Ilardi