Software artists

Development 1 Comment »

On the way into work this morning I was catching up on some podcasts and listening to the software engineering radio’s interview with Dick Gabriel.

The interview is predominantly about the origins and future of lisp. An engaging subject considering the current resurgence of functional languages, but what hooked me and had me reading his site was his assertion that we need to move away from purely scientific and engineering approaches to educating the next generation of software engineers to one more like that used to educate artists. Where a large part of the focus is on studying great works of past masters. Learning to critique your own and others work striving to gain a greater depth of understanding.

Software education today is embodied in Computer Science and Software Engineering programs, supplemented by informal mentoring on the job. I find this approach unsatisfactory. Software development is a performance exhibiting skills developed by an individual—often in groups of teams in order to achieve the scale of software required. In this way, software development is like putting on a play, which requires the skills and performances of a number of people working in tandem on stage and behind the scenes. Such skills can be developed in isolation through practice with other amateurs or even by putting on plays in public without any training at all. But how much faster could talent be developed in a educational program that recognized that writing software has enough of an arts-like performance component that the program was tailored to it? Master of Fine Arts in Software - Richard P. Gabriel

For me software development is a highly creative activity. I have no formal education in software engineering. I’ve spent a large part of my professional life studying others work, trying to see the form behind the function. I’ve striven to create my own blend of form and function. I’ve watched my development patterns change over the years as my influences, languages and environments have shifted. Each change adding a little more to the depth knowledge and opinions I carry around with me. I’ve always felt fee to experiment, to make mistakes. I’ve never really fallen into the single pattern of development trap. I wonder if this trait is more prevalent in people with informal educations?

Hudson Git Clone

Development No Comments »

My SCM of choice at present is git. So naturally when I want to do a little tinkering on an opensource project I want to use it to track my changes. Unfortunately not everyone has seen the light and most of the projects I’m interested manage their code base with either CVS or Subversion. Fortunately the git team foresaw my need and support incremental importing of legacy repositories via git-cvsimport and git-svnimport.

So with the tools available (and a little help from Mike) I now have a public clone of Hudson.

You can view the change log at http://git.codecritters.com/projects/hudson.git and clone it via http from http://git.codecritters.com/hudson.git.

Mike has clones of a few useful projects like Django and ReviewBoard.

Liskov vs Hibernate

Development, Java 2 Comments »

In Data Abstraction and Hierarchy, SIGPLAN Notices 23,5 (May, 1988) Barbara Liskov wrote;

What is wanted here is something like the following substitution property: If for each object o1 of type S there is an object o2 of type T such that for all programs P defined in terms of T, the behavior of P is unchanged when o1 is substituted for o2 then S is a subtype of T.

and the Liskov substitution principle (LSP) was Born.

Applied diligently the LSP is one of the more powerful modeling techniques in an application designers bag of tricks. Unfortunately it’s also one of the most frustrating and difficult to grok because it seems so simple and intuitive.

When applying the principle it should be seen from the interface clients point of view. So “the behavior of P is unchanged”, doesn’t mean that the program has the same effect on the world, it means that the caller doesn’t have to condition its interaction with the subject based on whether they are o1 or o2. To the client both objects fulfill the contract and the client can ignore the differences between them.

Lets say I’m asked to provide a list interface (let’s pretend that java.util.List doesn’t exist for now). The client requires three operations; get an item at a specified index; put and item in a specified index and size that returns the greatest value for index. I create my interface and an implementation or two and deliver it to the world to great fanfare.

    +------------------------------+
    | List<T>                      |
    +------------------------------+
    | <T> get(int index)           |
    | <T> put(int index, <T> item) |
    | int size()                   |
    +------------------------------+

Another group requires a similar interface, the only difference being that they don’t want the list to be modifiable. Seeing as the original List interface was so successful I’m asked to create the new ImmutableList without the put operation.

    +------------------------------+
    | ImmutableList<T>             |
    +------------------------------+
    | <T> get(int index)           |
    | int size()                   |
    +------------------------------+

Several months later I’m reviewing my library and I notice that the two list interfaces share a common set of methods and I decide to redefine List to inherit from ImmutableList and unthinkingly violate the LSP.

    +------------------------------+
    | ImmutableList<T>             |
    +------------------------------+
    | <T> get(int index)           |
    | int size()                   |
    +------------------------------+
                   ^
                   |
    +------------------------------+
    | List<T>                      |
    +------------------------------+
    | <T> put(int index, <T> item) |
    +------------------------------+

You see from the standpoint of an ImmutableList client a List isn’t an ImmutableList as it can be modified. The inheritance relationship allows a List to be used wherever an ImmutableList is specified. Thus a client expecting that the list will not change may function incorrectly if when a List instance is supplied.

So how does this relate to Hibernate? Hibernate allows you to specify persist your java model with minimal impact on your code (this is grossly simplified view but will allow me to illustrate my point). It manages the relationships between objects and can be configured to either load an object graph completely, lazily or a combination of both. This is achieved by substituting the referenced classes with Hibernate proxies. These proxies will then load an objects details as required. Hibernate also allows you to detach and reattach and object from it’s data source. When an object is detached attempting to access parts of the object graph that haven’t been loaded results in an exception being generated.

Not so long ago I was involved with a project that used Hibernate to persist it’s state. The model was relatively small but the data set was very large. The data was organized into hierarchies. A node could have many children, these children could also have children (think massive XML document). A single node could literally have millions of descendants. Eager loading the complete graph for any node simply wasn’t an option and as the objects needed to be sent over the wire lazy loading was out. The team had developed a number of queries that would load part of the tree to a specified depth to suite the callers purpose. The problem was that the caller had to know which query had been used to load the object and adjust their behavior accordingly, violating the LSP. The team had effectively created derivative classes that behaved differently. This manifested itself in subtle (and not so subtle) defects, the solution to which was often to add additional queries to service the specific client needs.

To be fair the problem wasn’t really hibernates fault, but by the way it was being used (give someone enough rope …). Developers must be constantly vigilant to ensure their im implementation choices don’t introduce subtle design flaws.

The Lost Art of Separating Concerns - Noise & Misinformation

Development 2 Comments »

I feel somewhat frustrated when I read articles like Mark Barkers ‘The Lost Art of Separating Concerns‘. SOA is one of the most over used and under appreciated acronyms in our industry today. It’s unfortunate, because underneath the fluff, hype and vendor promises are solid architectural principles. Worst than the hype is the plethora of noise and misinformation, Marks article falls into this group. The intent of his article is to show how REST separates interface from implementation, while SOA/WS doesn’t. What it does is make the false assumption that in the SOA world services equal application interfaces.

While I was at ThoughtWorks I was ‘fortunate’ enough to work with Jim Webber, SOA proponent and all round nice guy. Jim is a fountain of knowledge and opinions on all things SOA. Until I met Jim I really wasn’t interested in the whole SOA phenomenon. There were just too many vendors offering it up as some form of silver bullet. Jim convinced me otherwise. Through conversations, careful guidance and the judicious use of his backhand (never suggest that SOAP and Web Services are CORBA wanna-bes) I came to an understanding of where an SOA fits in the development landscape.

A service-oriented architecture is not about using tools to create WDSL definitions from application interfaces defined in java or c#. It’s not about using tools to create application interfaces from WSDL. In fact it’s not about application interfaces at all. It’s about exchanging messages. These messages may be logically combined to form protocols and services.

In the current corporate world, for better or worst, the accepted message form is SOAP and WSDL is most often used to describe services. Unfortunately there is also a vendor supported tendency to blur the distinction between service definitions and application interfaces. Vendors (open source and commercial) tout the ability to ‘transparently’ enable web services from application interfaces or to generate application interfaces from WSDL definitions. In both cases we end up with service architectures that are tightly coupled to application implementation. An alternative approach is described by the proponents of MEST. MEST attempts to describe service-oriented architectures in terms of services, messages and architectural principles. It moves the focus from application interfaces to protocols built on patterns of message exchange. This focus allows you to design message protocols in line with business services. Because the focus in on the messages the services naturally exist independently of application interfaces. This disconnection allows application and service interfaces evolve independently. There is a small price to be paid in the form of adaptors but frameworks like Spring Web Services make this task quiet a bit simpler by focusing on message processing rather than service generation.

Retrospective

Development, Random thoughts 1 Comment »

I bumped into an old friend in the pub the other day. We started talking and she told me about a retrospective she’d recently been involved in after the project she was working on had been suspended due to yet another management restructure (there have been several in the last twelve months). A large number of people from the business and the development group were in attendance. Her team of four developers was there along with the architect who had created the solution architecture and the non-functional requirements and the project manager. There where one or two people in the room she didn’t recognize and she assumed they were from the business side.

Introductions were made and she was surprised to find that “John” had worked on the project reviewing the architecture. She had never encountered him and had worked closely with the solution architect, who’s eyes seemed to roll when John introduced himself. Still, she knew that the architecture had been through a number of reviews. She’d bounced a couple of architectural decisions that had sprung from these as non-functional requirements for business approval as they had had the potential to significantly increase the amount of effort required to produce the system without adding any direct business benefit.

As in most retrospectives the meeting was an attempt to extract lessons learned while delivering the project in order to improve the process in the future. The meeting was going well. Many of the positive items where directed directly at her team and she was proud of the work they had done. She was surprised when one of the people she didn’t recognize began to raise issues around the clarity of the projects purpose. She’d been working with the business on a daily basis and, although the requirements had been subject to some change and clarification, she had never felt that the projects final goals were unclear. “How can you even begin to develop a piece of software without a vision statement?” True she’d never seen one, but the project wasn’t really rocket science and she thought the objectives of the project were clear.

True, they were running behind scedule. This was due, in the majority, to the restructure and a freeze on resources. She had a project team half the size she’d been promised with a higher proportion of junior developers than had been agreed. Another contributing factor had been the toolset mandated by the architecture team. When early on in the project she’d queried the reasons that they needed to use a particularly painful tool. She was told that it was vendor “A”s product in the space and we only use vendor “A”s products.

John continued to insist that the project didn’t have enough governance. He insisted that the project needed more rigid processes. He failed to understand that the project delivery team hadn’t failed and my friend took offense at the assertion they had because there wasn’t a huge paper trail of ill-informed sign offs.

The business side of the project had been very happy with aspects of the system they had been able to preview. That the project had been put on hold due to a restructure was not in any way a failing of the delivery team, but John continued to prattle on about roles, responsibilities, more detailed task reporting on time-sheets and an interesting observation that if you have half the number of development resources the spend rate would be 50%. “We’re not even CMU level 0” he exclaimed, confusing everyone until the project manager corrected the term.

We had a couple of drinks and laughed and wondered how far they would progress down the CMM track before they realized that people interacting was more important than process they employ. “No matter how rigid a process you have, a monkey will still play with its shit.”

Misplaced wallet

Random thoughts No Comments »

Earlier this week I managed to misplace my wallet. After a search around my desk and through my various bags and draws I concluded that I must have either left it at the store I’d purchased lunch from or dropped it sometime after. I headed home knowing I would have the great pleasure of getting all of my credit cards and identification reissued. A couple of minutes after walking in the door I received a call from a sergeant at Richmond police station. It turns out that I had in-fact dropped it in Lonsdale street on my way back from lunch. An anonymous individual found it and handed it in at Richmond police station, cash and cards intact. Thank-you doesn’t adequately express my gratitude for your simple act of honesty.

Interface Assertions

Java 2 Comments »

For some time I’ve wanted to be able to define basic contracts for interface clients in a stricter manner than simply specifying the client constraints in javadoc. The problem is that interfaces can’t provide implementation so the constraints can’t be applied in the interface but instead must be applied in the implementations. This leads to duplication of the constraint logic and make it easy to omit these checks when implementing the service. Good test design and coverage will detect and expose the problems but it would be better if we could prevent the problem in the first place.

Below is a simple service that manages a collection of users. There are a number of simple client constraints, such as not passing null or empty strings to findUser, that are captured in the javadoc. What we want to do is enforce these across all implementations.

/**
 * A service to manage user accounts
 */
public interface UserService {
	/**
	 * Locate the user.
	 *
	 * @param id the id of the user to locate. Must not be null or empty
	 * @return the located user or null if a user with the supplied id is not found.
	 */
	User findUser(String id);

	/**
	 * Remove the user from the system. If the user does not exits the request is ignored.
	 *
	 * @param id the id of the user to remove. Must not be null or empty.
	 */
	void removeUser(String id);

	/**
	 * Store a user instance. If the user already exists the record is updated, otherwise a
	 * new record is created.
	 *
	 * @param user the user to store. Must not be null.
	 */
	void storeUser(User user);
}

In-line assertions

When it comes time to implement the interface you need to be aware of the client contract constraints and implement them via assertions or run-time exceptions. I like assertions so I’ll use them to add the rules around my implementation TransientUserService. The first version will simply add these constraints directly to the implementation.

public class TransientUserService implements UserService {

	private Map _users = new HashMap();

	public User findUser(String id) {
		assert null != id && id.length() > 0 : “An id must be supplied”;
		return _users.get(id);
	}

	public void storeUser(User user) {
		assert null != user : “A user must be supplied”;
		_users.put(user.getId(), user);
	}

	public void removeUser(String id) {
		assert null != id && id.length() > 0 : “An id must be supplied”;
		_users.remove(id);
	}
}

Nothing surprising here, the assertions are diligently applied in a simple and consistent manner. When a second implementation or subclass is created the assertions are once again added, again in the third and fourth implementations and so on. I find this duplication of code undesirable so what other options exist.

Wrapper Assertions

The next implementation wraps a user service instance and enforces the assertions before delegating to a concrete implementation.

public class UserServiceAssertionWrapper implements UserService {

	private final UserService service;

	public UserServiceAssertionWrapper(UserService service) {
		assert null != service : "the UserService must be supplied";

		this.service = service;
	}

	public User findUser(String id) {
		assert null != id && id.length() > 0 : "An id must be supplied";
		return service.findUser(id);
	}

	public void removeUser(String id) {
		assert null != id && id.length() > 0 : "An id must be supplied";
		service.removeUser(id);
	}

	public void storeUser(User user) {
		assert null != user : "A user must be supplied";
		service.storeUser(user);
	}
}

This approach reduces the amount of duplication as the rules are captured in a single class that applies the assertions before proceeding. It does however mean that we must remember to wrap all of our instances in the wrapper before providing them to clients. The can be made easier with using a dependency injection framework or factories but at some point it will be forgotten and the contract wont be enforced.

Injection via Aspects

Aspect orientated programming provides us with an additional solution. Until java has support for design by contract (DBC) I believe that inserting contract constraints via aspects is the cleanest and most robust approach. The idea is simple. As part of the interface create an aspected that provides pointcuts that pick out all implementations of the interface methods and advice that injects the contract constraints.

/**
 * A service to manage user accounts
 */
public interface UserService {
	/**
	 * Locate the user.
	 *
	 * @param id the id of the user to locate. Must not be null or empty
	 * @return the located user or null if a user with the supplied id is not found.
	 */
	User findUser(String id);

	/**
	 * Remove the user from the system. If the user does not exits the request is ignored.
	 *
	 * @param id the id of the user to remove. Must not be null or empty.
	 */
	void removeUser(String id);

	/**
	 * Store a user instance. If the user already exists the record is updated, otherwise a
	 * new record is created.
	 *
	 * @param user the user to store. Must not be null.
	 */
	void storeUser(User user);

	static aspect UserServiceConstraints {
		pointcut findUser(UserService service, String id): execution(User UserService.findUser(String))
			&& target(service)
			&& args(id);

		pointcut removeUser(UserService service, String id): execution(void UserService.removeUser(String))
			&& target(service)
			&& args(id);

		pointcut storeUser(UserService service, User user): execution(void UserService.storeUser(User))
			&& target(service)
			&& args(user);

		before(UserService service, String id) : findUser(service, id) {
			assert null != id && id.length() > 0 : “An id must be supplied”;
		}

		before(UserService service, String id) : removeUser(service, id) {
			assert null != id && id.length() > 0 : “An id must be supplied”;
		}

		before(UserService service, User user) : storeUser(service, user) {
			assert null != user : “A user must be supplied”;
		}
	}
}

In the above code the pre-conditions for calling UserService methods have been added via an inner static aspect. This keeps the contract and interface nice and close to each other. Implementations have the pre-conditions applied via the before advice consistently without any need to manually wrap the service instances. The service definition and constraints are both documented and coded in the same file making it easier to keep them in sync as the interface evolves. The interface and aspect can be contained in a separate library and you can use aspectj’s binary weaving at either compile or run-time (though I prefer compile time application of advice in most circumstances) to apply the advice. As a bonus if you use eclipse and the AJDT the editor will mark each point that the aspect is applied and you can use the cross references view to see where an advice is applied or what advice applies to the block of code you’re looking at.

Other Contract Assertions

Although the example above only enforces a few pre conditions, it’s not hard to add simple post conditions and invariants. For example after successfully adding a user we should be able to find that user via the service. The advice below implements the rule.

		after(UserService service, User user) returning : storeUser(service, user) {
			assert null != service.findUser(user.getId()): "User must be able to found after being added";
		}

Final Note

With this new approach it’s easy to get carried away. There is a fine line to be carefully walked between what rules you add via contract advice and the rules checked and asserted for in unit tests. Whenever I create an interface I create an abstract unit test that validates implementations. Contract advice doesn’t remove the need for these tests.

Should application architects be separated from developers?

Development No Comments »

I’ve recently encountered a project where the application architects are very detached from the development team. The Architects sit on high making proclamations for the developers to follow. They draw diagrams, proclaim standards and generally ignore pragmatic needs, instead focusing on white papers, rambling meetings, and modelling tools.

Now there is a place for white papers and modelling tools. My concern is that the architects, while not really being bad people, just aren’t really responsible for delivery. They seem to make arbitrary decisions about the way the system should be partitioned and constructed with little concern for the practical downstream effects. The effect, in my opinion, on the project concerned has been disastrous.

The system seems incredibly complex for the functionality it delivers. Tasks that would normally take a competent developer a few hours take days, sometimes weeks. Subtle side effects leave people scratching their heads looking for answers. What would normally be done in a few lines of code in a couple of methods is done with a dozen classes and as many xml configuration files. Worst of all developer moral is incredibly low and belief in what they are doing is almost non-existent.

My solution is simple; make the architects lead the delivery teams. That way they share their wisdom and reasoning in a collaborative manner. They also become responsible to the project for the decisions at a very practical level. They are the ones responsible for delivery of the component. When it takes weeks to deliver something that would normally be estimated in days because flexibility never used is designed in, they are the ones explaining it to the project.

Good architecture needs to be pragmatic and responsible. I’ve worked with some good architects in the past. These architects focused on the practical outcomes of their decisions as well as the latest white paper.

Subversion update script

Development 1 Comment »

I keep most of my projects in a directory off my home ~/project. Most days I want to ensure that everything on my machine is up to date. Rather than cd into each directory and calling svn update by hand, I use the following.

for project in `find . -maxdepth 2 -type d -name ".svn" | cut -d / -f 2`; do svn update ./$project; done

The same could be done for CVS if that’s your poison. Not sure if this is really any use to anyone else, but I’m sticking it up anyhow.

Subversive, great subversion handling for eclipse

Eclipse No Comments »

As a regular subversion and eclipse user I’ve been longing for an eclipse feature that provided the functionality of the CVS feature but for subversion. I battled along using subclipse but it never quite seemed to feel stable and I often ended up reverting to using the command line. Now there is a better solution Subversive

I’ve been using it for a couple of weeks now and I’m really impressed. It works most of the time and if it errors an error report is generated. With a click of a button you can send it to the developers. When this happened to me the issue was fixed in the next release, bloody impressive.

WP Theme & Icons by N.Design Studio
Entries RSS Comments RSS Log in