Managed Mayhem

I frequently run into the request for a ‘ball-park’ estimate for software development projects. I loathe to give these type of estimates. They are inherently dishonest from the very start because they are a ‘promise’ based on incomplete and sometimes incorrect information.

I love to give a free one hour consultation for my clients. It gives us a chance to get to know each other, introduces me to their product or idea … and begins the process of figuring out ways that I am able to help them. I can usually tell rather quickly within this one hour period if I’m going to be an asset to their team or not. In most cases, clients have a very basic idea of what they are really wanting to accomplish. A loose set of ideas, sometimes scribbled on a notepad or napkin while you watch. Other times, clients will actually have some prior education and will have something a little more organized… perhaps some storyboards or a diagram showing what they want to do. This is better, but honestly… still not enough to give a proper estimate.

The problem that I have with ball-park estimates is that they are too subjective and emotional. I might be hungry that day, or I might not be hungry at all. I might be really excited about working with this particular project or client, or perhaps I’m not really interested in what they are doing but still want to help. Maybe I just want the client to like me. Maybe I’m without work at the moment, or I might be swamped. None of these things should really matter when building project estimates because they are subjective, emotional, and variable. We have all been in that position where we were without work, underbid a project (usually as a ball-park estimate) and then half-way through the gig we’re late, overbudget, swamped with new work feeling taken advantage of, and no longer really motivated to do a good job. Why do we do this?

When I have given them, my ball-park estimates for custom software development swing manic depressively between polly-anna optimism and grinch like pessimism. This is never good because either way, someone is going to be unhappy with the end result. If I am pessimistic and shoot high, clients get skittish or I automatically lose to moderates and optimists without the opportunity to close. If I am optimistic and shoot low, inevitably the client is disappointed anyhow because:

• Time lines are missed
• Market opportunities are missed
• Budgets are missed
• Rush to release results in mistakes

Some clients have been thinking about and developing their product / idea for years… it’s only reasonable to expect that we are given the time needed to study their product / idea thouroughly in order to do our jobs properly. There are a lot of software development firms out there who  with respect to setting budgets…they hedge on margin / volume, or occasionally getting lucky. They figure ’some we win, some we lose’.

I don’t like to lose… I like both the client and me to win every time.

The analysis portion of our jobs as software developers is primarily about reducing risk. I see gambling with incomplete / inaccurate project estimates as a risk that is deadly to projects, and one of the primary reasons why they fail. Get as much information about the project up-front. Spend the time that it takes to get as accurate an estimate as possible. Don’t worry about being perfect, but at least make the attempt.

I promise to write an article about how I estimate software projects soon. For now, feel free to comment about some of the project estimation tools you’ve used that have worked for you!

I use Firebug for Firefox normally, but was always irritated when I had issues in Safari only, and did not have the same sort of inspector available to me. IE8 now has one included as well, which should make debugging IE much easier now. freaking cool… you can turn this on from terminal by running this:

defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1

I have dealt with every single one of these types of clients before, and have been amazed with how frequently people will expect creative industry folks to do things that they would not ask their mechanic or doctor to do. Why is there a different level of expectation and total lack of respect for the time and talent of designers and developers vs. other professions? Why does it always seem to be a negotiation whenever giving prices? Would you expect your doctor to work on spec?

For more information on spec work, and why it is bad for the creative industry as a whole, go here:

http://www.no-spec.com/

From the browsershots.org website:

“Browsershots makes screenshots of your web design in different browsers. It is a free open-source online service created by Johann C. Rocholl. When you submit your web address, it will be added to the job queue. A number of distributed computers will open your website in their browser. Then they will make screenshots and upload them to the central server here.”

4 Platforms (Linux, Windows XP, Mac, Debian).
Almost 80 browser / version / OS combinations.

What I love most about the site… is not just what it can do, though it is pretty fantastic. What I really love is it’s use of distributed computing. The ‘distributed computers’ that they are using are actually computers that have been volunteered for use by folks on the web. Browsershots.org makes available a downloadable installation (written in Python) that is responsible for receiving screenshot requests and then fulfilling those requests, uploading them to the mother ship after it is completed. Genius.

Since the early-90’s and the RSA Challenge, I’ve been fascinated by the concept of distributed computing, particularly volunteer / open source distributed computing that makes use of available cycles / resources on people’s otherwise unused computers. Seti@Home uses this method to ‘listen’ to the results of mountains of data coming from space telescopes around the globe in a search for extra terrestrial life. FightAIDS@Home helps to identify candidate drugs that have the right shape and chemical characteristics to block HIV protease. Many of these projects have latched onto IBM’s ‘World Community Grid‘ program, running on the ‘BOINC‘ platform at about 2 Petaflops, and over 500,000 volunteered computers.

Service Oriented Architecture (SOA) with web services provides a great method for accomplishing some of these things. Now I want to build a distributed computing application.

Factory methods are responsible for manufacturing objects.

From ‘Gang of Four’:

‘Frameworks use abstract classes to define and maintain relationships between objects. A framework is often responsible for creating these objects as well’

Coldspring as a framework has an excellent implementation of the Factory Method pattern.

Example:

Let’s say I am building an application that has ‘CDs’ as objects.

If I am using a Service / Gateway / DAO pattern in my application… (I know that many folks rail against the 5:1 approach, but let’s just say for sake of argument that transfer is not a good fit for this application.), I have quite a bit of code to write just to get to a point where all of my required objects are instantiated:

<cfset datasource="library">
 
<cfset cdBean = createObject("component", "extensions.com.jimrising.cd.cd").init(title: "Joshua Tree", artist: "U2")>
<cfset cdDAO = createObject("component", "extensions.com.jimrising.cd.cdDAO").init(dsn: #datasource#)>
<cfset cdGateway = createObject("component", "extensions.com.jimrising.cd.cdGateway").init(dsn: #datasource#)>
<cfset cdService = createObject("component", "extensions.com.jimrising.cd.cdService").init(cdDAO: cdDAO, cdGateway: cdGateway)>

Note that the ‘cdService’ depends on the injection of the cdGateway and cdDAO. Coldspring handles this dependency injection nicely as well.

In order for me to create a new ‘cd’, and then save it to the database:

<cfset cdService.save(cdBean)>

The issue is one of persistence. In order for me to use these objects throughout my application, without the Factory Method, I would need to instantiate them each time I needed them. Using the Factory Method and a little setup within Coldspring, these objects persist and I’m able to access them through the factory, without needing to know all of the dependencies or subclasses:

<cfset application.serviceFactory.cdBean.init(title: "Joshua Tree", artist: "U2")>
<cfset application.serviceFactory.cdService.save(cdBean)>

Note that the DAO, Service, and Bean have already been instantiated, and so saving a new ‘cd’ object to the database (from anywhere within your application) is a simple matter of building the bean and saving it through the service within the serviceFactory.

Here is what the XML configuration (services.xml) looks like for ColdSpring:

<beans>
    <bean id="cdGateway" class="extensions.com.jimrising.cd.cdGateway">
        <constructor-arg name="dsn"><value>${dsn}</value></constructor-arg>
    </bean>
    <bean id="cdDAO" class="extensions.com.jimrising.cd.cdDAO"> 
        <constructor-arg name="dsn"><value>${dsn}</value></constructor-arg> 
       </bean>
    <bean id="cdService" class="extensions.com.jimrising.cd.cdService">
        <constructor-arg name="cdDAO"><ref bean="cdDAO" /></constructor-arg>
        <constructor-arg name="cdGateway"><ref bean="cdGateway" /></constructor-arg>
    </bean>
 
    <bean id="cdBean" class="extensions.com.jimrising.cd.cd" />
</beans>

And the simple one time instantiation of the serviceFactory within your application.cfc onRequestStart() method:

        <cfset application.defaultProperties.dsn = "myDsn"/>
        <cfset application.serviceFactory = createObject('component','coldspring.beans.DefaultXmlBeanFactory').init(structnew(),application.defaultProperties)/>
        <cfset application.serviceDefinitionLocation = expandPath('/config/services.xml')>
        <cfset application.serviceFactory.loadBeansFromXmlFile(application.serviceDefinitionLocation)/>

I was working last night on an application built on the following platform:

Railo 3.0
Centos 5.1
Apache2
MySql 5

The application isn’t doing anything crazy… just some queries and some file uploads. I kept getting this though:

Communications link failure due to underlying exception: ** BEGIN NESTED EXCEPTION ** java.io.EOFException STACKTRACE: java.io.EOFException at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1956) at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2368) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2867) at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1616) at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1708) at com.mysql.jdbc.Connection.execSQL(Connection.java:3249) at com.mysql.jdbc.Connection.execSQL(Connection.java:3178) at com.mysql.jdbc.Statement.execute(Statement.java:711) at railo.runtime.type.QueryImpl.<init>(QueryImpl.java:175) at railo.runtime.tag.Query.execute(Query.java:406) at railo.runtime.tag.Query.doEndTag(Query.java:340) at …

It took me a bit, but I finally figured out how to handle it…

In /etc/my.cnf
Adjust the ‘wait_timeout’ directive to a reasonably high number. It is in seconds… and default is set to 30 seconds.

wait_timeout = 1200

The second thing you want to do is make sure that within the Railo Web Administrator’s datasource dialog, your ‘Connection timeout (in minutes)’ is set to a number that is significantly lower than MySql’s ‘wait_timeout’. What this does is it makes sure that Railo properly closes inactive database connections before MySql does.

[this is actually an older article from my previous blog, just moving it to my mm blog] …

I started looking at <cfthread> today, and realized that it really does not have any sort of management facility built into it. while you can view / manage threads from within the server monitor…

1) you can’t get a status for a thread except from the calling page, or from another thread that was called from that page. this presents a problem if you have long running threads, or if you have other parts of your application that rely on thread status.

2) if there is an error that was generated during a thread process, again … you can access this at the page level, or from another thread, but there really isn’t anything that will help you to manage this within your application logic, say if you wanted to fire a thread again later.

Matt and I discussed an approach that could work well for managing these two things that are missing natively in cfthread, and act as a good foundation for further thread management. Keep in mind that this article really only consists of some raw ideas with pseudocode at this point. I’ll get around to writing the actual components later, and i’m sure it will develop into something much different from the ideas on this page.

The concept is based on two components that manipulate a server struct and act as a wrapper for cfthread:

the server struct consists of the following:

server.instance.threadName
server.instance.initTimeStamp
server.instance.startTimeStamp
server.instance.endTimeStamp
server.instance.elapsedTime
server.instance.function (join, run, sleep, terminate)
server.instance.priority
server.instance.object
server.instance.method
server.instance.argList
server.instance.status
server.instance.error

threadManager.cfc
This is the thread manager responsible for creating new threads, cleaning up old threads, and contains the getters / setters for thread status and errors.

thread.cfc
This is the thread object that acts as a container for thread processes. It builds a thread using <cfthread>, and creates a structure in the server scope containing the name of the thread, timestamp when it was created, current thread status, and any errors that occur.

myThread = thread.new(run, object.method, argumentList);

what this would do is create a new ‘run thread’ using the object.method(), given a name | value list of arguments. it would return the name of the thread as a UUID, and would add the thread instance to the server.instance struct. note that we’re passing in a list of arguments and values not for thread.new, but for the object.method.

newThread.exec(myThread);

this would execute the thread created previously, updating server.instance.startTimeStamp and server.instance.status in the server.instance struct. if the thread runs into any try/catch … this would be updated to server.instance.error. once the thread is completed, server.instance.status would be updated, as well as server.instance.endTimeStamp. all we’re doing here is passing the myThread object into newThread.exec, and then wrapping the object.method(argumentList); within <cfthread>