Mid-iteration Session Recap

Since we have now decided to switch to 2 week iterations, we had talked about now posting a thorough session recap every 2 weeks, and just a brief summary after each mid-iteration session. This week was a mid-iteration session, but after typing everything I thought should be shared, I see that my summary was not that brief. Anyway, here we go…

The session started with just Kristie and me for the planning session, which was interesting because this was only the second session for Kristie. We took our time with this because Kristie had many good questions and topics for discussion, and this is, after all, a learning group. The planning session went very smoothly though. Of the data we have so far (our previous two 2-week iterations), we had finished 2 stories in one session, and none the second, so we did our planning based on a velocity of one story per iteration, with 5 iterations until the next release. Putting on our customer hats, we were able to pick 5 stories that looked like they would add up to a very useful release.

About an hour in, Tom arrived for his first session working with us, so we had a brief Q&A session, then got in (as I recall) just a bit of development before lunch. Over lunch, we brought the cards, and covered a bit about the stories in our release plan and how the planning process works. After lunch, we first had Tom watching and asking questions while Kristie and I worked on the "Simple Preview" story, then Sam arrived in the afternoon, and paired with Tom on the VB Line Reader as part of the "Context Parameters" story.

The work day went really well, especially considering the time spent working with new and newish participants, etc. We actually finished the Simple Preview story, and made significant headway on the "Context Parameters" story. This time, we remembered to stop coding in time to do the integration while most of us were still around to participate, and Sam stayed later, so he and I kept working a bit after the afterward.

We had an issue with the new version of the VB Lite Unit DLL on one of the machines, and had to muck with it to get the registration right. Finally, it seemed to work and pass all tests, but was giving repeated memory violation dialogs, which we automatically attributed to still having some problem with VB Lite Unit. We worked around this by having Sam and Tom work on Tom’s laptop, which worked fine.

After everyone else left, I did some debugging, and determined the problem to be an interaction with the code in our project and Access 2000 (we’re using Access 2002 on the other machines). There’s a place where we were not setting an automation object’s reference to Nothing before quitting the Access instance we got the variable from. Explicitly setting the reference to Nothing first solved the problem.

Note that for one of the cases of the memory error described above, the problem actually occurred with an implicit reference to the VBComponents collection looped through in a For Each loop. To solve this, I had to create a variable for an explicit reference to VBComponents, loop through that, then set the explicit variable’s reference to Nothing before quitting the application. It would seem that in this case, the implicit reference is not actually going out of scope automatically on exiting the For Each loop, but hangs around for some unknown period of time thereafter. In Access 2002, I don’t know if this is fixed because VBA drops the reference sooner, or because the reference to VBComponents no longer generates an error when its application is closed with a dangling reference. The same problem does not occur with the variable used to iterate, which was found to always be Nothing after the loop completes.

I guess this makes a case for being extra cautious in VB/VBA to explicitly drop object references used in reverse-dependency order, even if that means making the code messier by using some explicit reference variables where they appear to be unnecessary.

New version of VB Lite Unit

In case no one has checked in the last few days, I put up a new revision of VB Lite Unit (1.2.3) on Sourceforge - http://sourceforge.net/projects/vb-lite-unit.

This revision has a lot of the fixes and features y'all have been asking for including a progress indicator.

I've recently learned some things the hard way about working with these DLLs that you might want to be aware of.
1. It's possible to have 2 copies of the same library registered, and this -will- cause problems when you try to use it.
2. It doesn't work to try to unregister a DLL after it has been deleted or replaced with a different version. Instead, you have to spend a 1/2 hour deleting all traces of it from the registry.
3. When you build an ActiveX DLL in Visual Basic, it is automatically registered in the path in which you built it - along with any version previously registered.

So, this leads me to recommond the following practices:
1. Before replacing a registered ActiveX DLL or building a new copy in VB, unregister the old one. Do this using regsvr32.exe /u <full-path>.
2. After building an ActiveX DLL in VB, and before moving it to a permanent location on a hard drive, unregister it in its build location, then re-register it in its permanent location after moving it.

On Error Resume Next considered dangerous

I was debugging some of the problems we came up with in VB Lite Unit, and I tracked one down to a wierd interaction between the test runner and the AssertKeyInCollection procedure I recently copied into VB Lite Unit from our project.

It seems that AssertKeyInCollection was passing, but not clearing the error number generated in the On Error Resume Next block. Also, the On Error Resume next block encompassed most of the procedure, when it only really needed to encompass one line to try to get the item from the collection, and another line to copy the error number to a variable, though that was not part of the problem.

Anyway, the test runner itself (until now anyway) used On Error Resume Next around the call to each test procedure, and examined the Err object afterward for exceptions. This turns out to be a bad idea since we now see that code in the tests can ignore a VB Error on purpose, but still leave the extraneous info in the Err object. In other words, after executing a procedure with On Error Resume Next in effect, it's entirely possible to have a value in Err.Number even though the procedure exited normally.

From now on, here's a couple of rules I'm going to follow...
1. If I ever use On Error Resume Next, I'll put Err.Clear at the end of the block.
2. I will not use On Error Resume Next to check error info raised from a procedure in the same project, since extraneous data can be in the Err object if the called code uses On Error Resume Next, and does not do Err.Clear. I'll always use an On Error Goto trap instead.

A rule I already follow is...
3. Never include any lines in an On Error Resume Next block except the line you want to ignore errors from, and the lines to copy error information into variables to be subsequently examined by code outside the On Error Resume Next block.

Testing quandary

Fair warning: This is not explicitly topical to the group's chosen project or platform, but I think it raises a valid and interesting TDD question, so I'm posting it here for feedback.

I've been writing a fair bit of Python code today using TDD. I'm moving the data for a project from Access to XML, which means I won't be able to simply declare a field as Autonumber and forget about it. I need to generate ID numbers which are unique in a particular domain... and I can't for the life of me figure out how to write any meaningful tests!

Here's the spike I wrote (yes, I know it executes in polynomial time):

class IDGenerator:
    def __init__(self):
        self.ids = []

    def newID(self):
        while True:
            i = int(random.random() * (10 ** 8))
            if i not in self.ids:
                self.ids.append(i)
                return i

Any ideas for tests that would express the problem? (Or, for that matter, for an algorithm with linear scalability?)

Back in time Recap from Aug 14-20

This is an abbreviated recap since none of us really had time to post it while memory was totally fresh...

Nearly at release time, with the release being defined as a presentation to PAUG about our product and process, and having been completely stuck on the weekend session, Sam and I got together for an evening, and managed to get unstuck (see post “Unstuck”).

Given where we were at and the timeframe, we pretty much had to plan the enture PAUG presentation just before the presentation, but with the 3 of us working together with some notecards, we figured out what we wanted to say without much trouble. Our code was far enough along at this point that Sam was at least able to create a spike that demostrated actually generating expanded code from code with embedded macro definitions and expansion points, so we had something really worth showing.

I’d say we spent a little under 1/2 the time covering the product we’ve been producing, then covered the XP process for the remainder. There’s a lot to cover regarding the process, and there were a lot of questions, so we covered the planning part pretty thoroughly, then crammed in what we could of TDD, and gave some lip service to pair programming, etc. Finally, we made the pitch asking for more members to join the team.

The presentation seemed to be a big hit, and we had 3 people say they were definitely interested in working with our team.

Session recap part 3 Aug 22

Oh yeah,

This weekend was the end of an iteration (assuming no work during the coming week), and we've completed zero stories. That's not too terribly surprising given the presentation, the incorporation of 40% more team members, etc., but that leaves us with an average velocity of 1 since the point when we started over on our velocity measurement 4 weeks ago.

I assume we'll bring the velocity back up, but of course, XP is not about assuming, it's about measuring, and being honest. We'll make our new projections based on a velocity of 1, and adjust it again when - uh, if - the velocity goes up.

Session recap part 2 Aug 22

Jordan had to leave early, so first Jordan paired with Adnan working on the Simple Preview story which is a preview of what the processed code will look like once macro expansion has run. I was the odd-one-out working to create Basic Standard Error Macros that the eventual user can employ using the tool we’re writing. This story is not scheduled until the next iteration, but we had 2 cards scheduled, 2 pairs, and me, so that was the next thing to work on.

I also stuck my nose in on Jordan and Adnan to reiterate the advice Ward gave us last time we werking on a UI component which is to make the code do everything we can think of that the UI will need under TDD first, then implement the UI. Since Sam already had a successful spike expanding code into a new Access database project, the task looked trivial – when will we learn :)

Working on the error macros, I made some good progress. I came up with an idiom that provides a logical place for for user-defined cleanup code without the need for some kind of fancy user overridable code block, and allows the use of a simple parameter replacement to distinguish between the 2 main types of error handling code without needing macro groups, mutually exclusive seedings, etc.
Partly based on my nosey advice, Jordan and Adnan started working toward a test that would create an instance of a particular form, treating it just like any other class, pass it a database name and a module name, call a method to display the output, and check the contents of a text box on the form to see if it had the right text, and they made some progress in that direction.

Jordan had to leave, and since Sam and Kristie were just getting into a groove, I took Jordan’s place with Adnan. I must admit, I wasn’t pairing well - more like coding and telling Adnan what I was doing, though I did take some of his advice on how to proceed. In the future, I’ll try to give up the driver’s seat when I see this happening, and I suggest we all try the same thing to see if it helps.
When I sat down with the code for the Simple Preview, I realized that my advice had probably been wrong. What we really needed first was just a way to supply an Access database name, a procedure name, and get the expanded code text. We started to write that test, and I realized that there was not already a test to simply get the expanded text of a named module from an Access project. We commented out the test in progress, and wrote a test for parsing the module from the project by name. We then realized that, although our tests were opening Access projects to pass references to the tests, the metacode system had no ability to do that itself, so we needed that first.

At this point, I noticed that we had started a test, commented out to write a prerequisite test, then did it again with a pre-prerequisite task. I thought this seemed odd, so I consulted with Sam, and he thought it seemed like a perfectly good thing to do, so we kept going with it. We got the first test passing, then the next, then were back to the test we started with, then got stuck, and didn’t get unstuck until I was cleaning up myself 5 minutes after everyone else left, atwhich point, the test finally passed – sorry you weren’t there to see it go, Adnan.

The problem had been our confusion over where the macros get parsed out of the module. We had thought we needed to invoke ParseMetaCodeFromProject before trying to parse the text for a single module using ExpandModuleFromText, but ExpandModuleFromText actuallu calls ParseMetaCodeFromProject, and trying to do it twice gave us a collection key conflict. I think this should be refactored first thing next week because it’s counterintutive, and it means having to parse the entire project once for each module to be expanded. Also, clsMetacodeSystem is simply ending up doing too many unrelated things, and needs to be broken up.

At the very end of the session, we realized we still had loose ends to tie up. Code had not yet been integrated, wrap-up discussions had not taken place, etc. In the future, we’ll need to bring coding to a stop -before- everyone has to leave, so we can take care of these things. I was left to integrate the changes alone (which was absolutely trivial). I added the tests from the code Adnan and I were working on to the copy Kristie and Sam had been working on, hit compile, and copied code from the other project until the compile succeeded. After that, the tests ran and passed. It took about 10 minutes.

In doing the integration above, I ran into something interesting. Although Sam and Kristie had left their code passing all tests, it actually did not compile. This has to do with VBA’s “feature” that allows you to run partially compiled code, and compiles uncompiled code if/when it gets invoked. Since the code that would not compile was dead code, ready for removal, none of the tests complained.

A note regarding Sam’s concern about 2-week iterations: Just because we don’t have an iteration-planning meeting, doesn’t mean we shouldn’t have a status meeting, just that we would not do a full card spread, etc. I understand that it’s typical for XP teams to have weekly meetings within an iteration, and assess how things are going. At this point, if the iteration looks to be ahead or behind, the customer can decide what cards to push out or drag in, etc.

Partial Work Session Recap Aug 22

We had two new team members showing up this week (for a total of five -- maybe someday we can have an even number), so we took an extra hour or so for discussion at the beginning of the session to bring them up to speed on the project, talk about the planning process, and generally ask lots of questions. We did start getting into implementation discussions about specific cards, but this was quite worthwhile, since time spent thinking about any aspect of the project seemed to help people get into the proper headspace for working on it. (Also, I tried to gently play topic cop. Perhaps I need a hat, or maybe just an Armenian cucumber.)

During the initial discussion, Steve proposed extending our iteration time to two weeks in order to have some kind of velocity to measure. Given that this effectively means we have two-day iterations, this seemed perfectly reasonable. (Although, I'm not quite sure how this will work with such a long period between coding sessions. Having the initial meeting has been really useful at getting me back into the groove.)

Anyway, after wrapping up our discussion and a quick demo of VBLiteUnit, it was lunchtime already. When we got back, we split up... and as I only worked with Kristie today, Steve will have to post his recap of what everyone else was doing.

Kristie and I picked the "context parameters" story card and got to work. While this might not have been the ideal place to start a new member, she quickly got into it when we started refactoring the existing tests. Task #1 was to move all of our testing out to an external project. This allowed us to remove the last call to the evil "CreateModuleCallParserAndDestroyModule," and restored our ability to use the debugger (yaaaay!).

Cleaner tests in hand, we added much of the plumbing to the VbModuleLineReader class that will be necessary to finish this card. We wrote some code that was incomplete (i.e., it only worked on Subs and Functions, and not on Get/Set/Let statements). Knowing that, we got a little overconfident and tried to write a test at the end of the day that was too big. This got us considerably bogged down. Then, rolling this back turned into more of a pain, since the number of changes we'd made between the last green light and the time we decided to roll back exceeded the capacity of the Undo buffer in the VB editor. Source code control to the rescue... except that we weren't using it, because we don't have licenses to VSS and the ODE tools. Oops.

Perhaps another team task should be to implement a (relatively) simple source-code helper in Access which can transfer code from VBA modules to textfiles and vice versa, so we can use one of the various text-oriented source code control systems on it. (Or, one of us can do this between sessions in our copious spare time. Hm... perhaps I'll spike this if I have a free hour or two.)

When I left, Steve was working on integrating the various teams' work -- something else we'll have to learn how to do more efficiently and frequently throughout our workdays.

(Working with a new team member also raised two other issues for me: first, that we should agree on and document our code formatting standards -- indentation, capitalization, use of Hungarian notation, commenting, etc -- for hobgoblin's sake; and second, that it's amazing the kinds of things one learns from pairing. I got to teach another team member about Ctrl+Space, and learned a useful and really obvious trick for marking code for later deletion.)

One of the topics continually being raised is how Access makes it annoying and/or difficult to implement so many "best practices" like using source code control... or, come to think of it, handling errors. ;> Perhaps when we're done with this code generation tool, if we want to continue, we can continue developing other tools to make managing code in Access as much fun as developing database apps in it.

Prelude to the work recap

Hi all,

It was basically my turn to do the recap, but with the presentation itself, starting an 11-hour database recovery process at midnight of Wednedsay, helping my wife with her music project, updating my resume, etc., it has not been practical for me to do.

What I would say right now is that I think it would be logical to have each recap refer to an iteration, not just the weekend session. Thus, in a week like this one, where there was team activity scheduled during the week, the recap would follow the last activity for the week. With the larger team size, I also think we might end up changing to longer iterations soon, perhaps 2 or perhaps 3 weeks in length.

I'm still hoping to eventually do the recap for this week, though I wouldn't be able to do it until Sunday night - after the next session has also concluded. If no one else minds, I should still be able to accurately recall and reconstruct the events of the past week by then.

Welcome, PAUG members!

Greetings to the PAUG members who are coming here after the presentation! There's some stuff in the archives about how we started -- some of it will be redundant with what we discussed this evening, but some of it will provide useful background information.

If you'd like to volunteer and haven't already talked to one of us, click here to send me email via my website. (I wouldn't recommend posting your email address here because of spam harvesters.)

More news will be forthcoming, but not until I've had some sleep. (=

[EDIT: The archives are back where they're supposed to be. I hadn't tried to look through them since switching web hosts. Sorry!]

Unstuck

It feels odd to be posting this in advance of the session recap from last time (was I supposed to write that? Did we decide?), but here goes.

After the team spent two work sessions being utterly stuck on an odd bug that prevented virtually any forward progress, Steve and I got together to attack the problem again. Before Steve's arrival, I spent an hour playing with the code and worked up the following three problem statements:

1. Tests are not independent of one another. Uncommenting one test will make subsequent tests fail where they did not before.
2. Testing code has much duplication in supporting code (e.g., CreateModule/CreateModules, of which there are three separate copies). This isn't just a code smell, it's explicitly A Bad Thing.
3. Some tests are doing things (creating and destroying modules) that make it impossible to use the debugger to step through and figure out what's going on. While this might not actually help us find the problem, it's annoying, and the more I think about it, the more dynamic module manipulation begins to smell.

...as well as the following three strategies:

1. Leave this for later. I suspect this may be a side effect of something else we might discover by doing 2 and 3.
2. Eliminate the duplication by SLOWLY refactoring the testing code into a single procedure in a single location.
3. Change the testing method. Instead of creating and destroying modules on the fly, why not leave them in the project (perhaps with different prefixes), and write tests so that they can inspect only named modules? Tests written to work with an entire project can then be written in such a way that they only check collection counts, or check for one or two specified macros (this will help keep those tests maintainable as we continue modifying).

Proving the wisdom of pair programming, Steve actually convinced me that the simplest way to get around the practice of creating and destroying modules on the fly was to push the contents of those modules into projects in their own .MDBs and read them directly from there. This turned out to not only be easier than I thought it would be, but to make many of our tests considerably simpler.

Net result: we are now back to a green light. In fact, we're slightly ahead of where we were before, because while we were at it, Steve and I finished the next test in the list. Not sure exactly what we'll have for PAUG on Wednesday, but I feel much more confident in our ability to hack a demo together in the hour before we go on. (=

Context Swapping

I'm taking a last-minute look at the code before our session tomorrow (I guess you could say I'm JITing myself, but that sounds sort of naughty). Between my intensive cramming for a math test and my having missed a lot of work from the last session, I've sort of induced a case of Beginner's Mind with respect to this project -- which is a polite way of saying I don't remember what the heck we were thinking in some parts. (=

One thing is clear: we've got a few highly amusing and snarky comments in there, and I'm pretty sure I didn't write all of them! Another thing: it's not always immediately apparent where to begin when reading the code. I think this partly indicates that the refactoring isn't done, but from an API perspective, some of those method names are getting pretty long, and it's a little confusing to me. Maybe it's just that I should be going to sleep. Will take a fresh look at it tomorrow.

TDD on the day job

I've been thinking about how to introduce some TDD to my main paying client. The biggest issue that's coming to mind is how to convince them to deal with one more ActiveX DLL to install and maintain if the tests remain in the source code as they ought to do. They're having enough hassle already managing versions of their own library that they're finally moving shared code out into.

In the mean time, I was helping another programmer there in his struggles with that very same library. He had created a very useful abstraction that replaces a lot of existing duplicated, ugly code. His procedure basically takes a 1-cell wide, vertical range of Excel cells sorted by value, and returns a collection of subranges grouped by value.

He had made this procedure work fine in testing from VBA, but when he tried to put it in the library and call it from Excel, it had an off-by-one error that was really resisting debugging. I suggested we start over writing the procedure test-first. Since there is not test harness in place, I just started writing a test procedure that uses automation to set up and destroy an Excel worksheet (that was about 7 lines), and in the middle, I put code to write some hypothetical cell values in a column range, and made a test calling our as yet hypothetical procedure on a range starting from an empty cell, and checked that we got back an empty collection.

We kept writing tests, adding functionality, refactoring, etc., in standard TDD form until all the expected tests passed. The code we ended up with was a bit better structured, so if something went wrong, it might be easier to debug, and we had the tests to run against it from VB to make sure it still performed as specified before we tried running it in context. As it turned out, this code did not suffer the off-by-one problem, the tests did pass when runing them from VB, and the function worked when calling in place of the old function.

Finally - a documented, real-world (paid work) win for TDD by a member of our team.

VB Lite Unit is available for download on Sourceforge

I had no idea how much reading would be required to figure out how to put a file into a Sourceforge project for download. After that, I realized I'd have to spend some time putting the files into meaningful packages with readme files, etc., but...

I finally got the "VB Lite Unit" files into an official release, downloadable from the Sourceforge Web site. I also started a list of tasks there for work on future releases by myself or other contributors.

Aug 8 Work Session Recap

This was a half-day session (due to the Bridge Pedal), and also Sam couldn't make it.
Steve and I decided to "repay our programming debt" by focusing on refactoring. But first we needed to "refactor" our story cards, since we now had two sets: the 'originals' we'd been working on from the beginning, and the 'new ones' we made last week (because -somebody- forgot to bring them). We were going through the cards, getting rid of duplicates, and examining them within the context of trying to have a reasonably interesting (and working) demo for our August 18th demonstration at PAUG.
This was a great exercise for me personally, as it furthered my understanding of the "big picture". It was neat to see how the content and the order of the cards reflects the planning process:

• The first few cards deal with specific requirements needed to "lay the groundwork" for the whole system: 'Simple Preview' of expanded macros, 'Safe Expansion' of macros to their targets, and 'Auto Backup' of existing code


• Further out are some features that build upon this foundation and flesh it out a little: 'Basic Standard Error Macros' using dialog boxes, and 'Context Parameters' for things like automatically grabbing the name of the function that macro target lives in. We figured that having the basic error handling functionality would be necessary for the PAUG presentation since it would provide a concrete example of why and how you'd use this crazy contraption -- without it things would be a little too abstract for people to follow (well, it would be for me anyway).


• Towards the bottom of the stack we have ideas that are really just brainstorms. Honestly, I don't remember what most of them are, and some of them were over my head to begin with (Steve obviously has grand visions for this thing, and while I'm happy to come along, I don't quite understand their application yet). These are features that will *never* get implemented, but we have cards for them anyway because you know what your mother told you about never saying never (or more importantly, your client -- "We'll *never* need more than 4 digits for product codes..." Riiiiiight...)

After getting our story cards in order, we hit the keys for some "real" refactoring. During the week, Steve had put the unit testing harness into a separate component, so first we took that code out of our Macro project. After that, our strategy was to basically look for obvious code duplication. First stop: error handling. Since we were "holding on to" the errors raised by VBA for later use, we kept having to save the number and description in variables. We decided to put these into their own (reusable) data structure, so we made a class to hold them. This made our code look alot cleaner.
As we continued scanning the code for refactoring opportunities, we discussed our personal desires to move towards writing code that is as self-commenting as possible, and Steve had a great rule of thumb regarding this: if there's a comment in code, that's probably a good thing to break out into its own (well-named) function. We did this in several places, and the code kept getting cleaner.

After a few hours, we decided to work on the next story card, which was 'Simple Expansion' of Macro Targets (replacing a tag in code with the full definition for that tag found elsewhere in the project). Unfortunately, there was a problem: we accidentally copied the wrong project file from last week, and had been refactoring a slightly older version of the code! We really need to implement a version control system... But in the mean time we faced a dilemna: should we redo all of that refactoring on the newer (more functional) code, or should we try to copy that new functionality to the older (but nicely-refactored) code? I hate redoing work, but trying to copy/paste the new code might have broken something if we didn't get it exactly right. Fortunately we had the unit tests!! Because of the tests, we were able to confidently copy/paste the missing functionality back into our project, safe in the knowledge that everything would still work properly. Unit testing, where have you been all my life?!

As the end of the day drew near, we hit another snag -- one of the new tests we wrote wasn't passing, but we quickly found the cause (not properly wrapping 'regular' VB lines in XML tags so they could be safely passed to the DOM). However, when we fixed this problem, it seemed to cause another (completely unrelated) test to fail. We racked our brains trying to figure out how this one portion of the code could possibly affect the other, but we finally realized that it wasn't -- BOTH tests were failing, but only one was getting reported at a time. So we need to change the testing harness to display multiple failures.
I thought this might be problematic in the event that one failed test causes another failed test, but fortunately we have been keeping each test self-contained (not sharing data, re-initializing everything in between, etc.). I think this is one of the guidelines of Test-Driven Development, and now I see why.
Unfortunately, we ran out of time before we could fix the actual bug, but at least we isolated it so that we don't have to go on a wild goose chase next session.

I would say that the benefits of XP are dawning on me slowly but surely. It requires a lot of legwork in the beginning, some of which seems unneccessary or pedantic. But I think its major advantage isn't so much in writing the code you planned, but rather in coping with the inevitable problems you didn't plan.

VB Lite Unit on Sourceforge

I have successfully created a project repository on Sourceforge for the VB Lite Unit software this team has been using as our test harness. The URL is http://sourceforge.net/projects/vb-lite-unit/.

I have not yet posted any files to the repository, but I should have all the files up and available for download in the next day or 3. I hope to have the source files in CVS within a week or so.

The newbie's comments

Hi. I guess I'm not that much of a noob since I only came into this thing 2 weeks after it started. Nonetheless, my naivete has given me some unique insights into the pros and cons of the XP process as a whole.

1) Pro: As someone who's never even done unit testing before, words can't describe how great it feels when you make a rather complicated change to the code, then run the tests and see that 'All Tests Completed Successfully' message! All those hours spent worrying about uncaught bugs or unforseen problems, knowing that when you find one it only makes it worse because then you wonder about all the other things you may have forgotten to check. It sure is nice to have that weight off your shoulders!

2) Con: As nice as the unit tests are, I can't help but wonder if we're just pushing our problems 'up the stack', so to speak. With the unit tests, you don't have to worry so much about bugs you've missed, but instead you're wondering about what unit tests you've forgotten to write. That being said, it still seems like a better situation than coding without the tests. As Steve pointed out, once you write a test, it's there forever (whereas bugs have a tendency to keep cropping up after you've made more changes to the code).

3) Pro/Con: Working in pairs, planning out story/task cards, unit testing and prodigous refactoring have a strange effect on the way I think about the design of the project. Usually I start from the top -- coming up with a master plan -- and then try to fit all the details into this grand scheme. But with XP, I feel more disconnected from the overall design. This "modularity" can be good, since it facilitates working with others quite efficiently. But it can also be bad because alot of the time I'm not really sure where the piece of code I'm working on fits in to the big picture. Since you have unit tests, you don't have to worry about the pieces of code 'breaking', but I wonder if they are interconnected in an efficient manner. Another way to put it is that any great work of craftsmanship has some central theme(s) running through it from top to bottom in a very orthogonal way, and without this "top-down" planning (which you are supposed to avoid with XP), I worry that this quality will be lost. But perhaps I'm just not experienced enough with it to see things the right way. Or perhaps this doesn't matter as much as I think it does. Hmm...

All in all, I think XP is a great way to look at development. The jury is still out as to whether I'd want to use it for "real" work, but there is no question that forcing yourself to look at a recurring problem in a new way is quite refreshing and educational.

Finally, the Recap from Aug 1 Work Session

Before we started with the planning session, we did some catching up with Ward on how we fared guiding ourselves for the previous 2 weeks. We knew our planning process was having some difficulties, so Ward listened, offered suggestions, had us show him our custom-built test platform passing and failing a test. He said he now believes we are testing our code =).

He talked to us about how developers often overengineer rather than doing the simplest thing that could possibly work. The card we were stuck on was to make a simple list of macro definitions and calls, from where, and as an understandable list. We had actually finished code that could return a collection of macro definitions, all but finished code to return a collection of macro calls, and had almost finished a form with a tree view to display the results, so we were close. I pointed out that we had felt very close for mutliple weeks in a row, though, so history suggested we might stay almost done for quite some time!

Ward noted 2 things. First, we had envisioned an understndable list as something much more complex than necessary, and since Ward had been acting in the customer role for this story and had not been at the previous 2 sessions (we knew Ward would be available only sometimes), we had not had the on-site customer to ask what constitutes "understandable", and this contributed to our going overboard.

So - on to the planning process. -Somebody- forgot to bring back the cards, but this turned out to be no particular problem, and possibly a good thing. We will have to merge our new stories with our old ones next week, though. We decided to attack our zero-velocity problem by making smaller stories, and splitting our larger existing stories. We will not try to preserve old velocity information we did collect from our previous sessions using a different story size, but will simply start over with our new concept of a good story size. Our initial velocity guess that we're using to plan the rest of the release is 2 stories per week (1 day of work per week).

Even though we were almost done with it, in the spirit of smaller stories, and since so much time was spent reviewing process this week, we split the story of listing macros and targets into separate stories for listing macros and for listing targets. Also, as we were reconstructing our stories from memory (just enough to plan the day and roughly sketch out the release), we came up with a next story of Simple Macro Expand, and we realized that since this seemed like a risky operation (we're replacing an existing code module during the expansion), a story for backing up the original module would be good to add. Ward saw this as a sign to split the original story into a safe part, and a more risky part, so we made it into one story for a simple expansion preview, and another to actually apply the expansion to the module.

On to programming...

Next, we endeavored to finish up the story we've been on for so long. Ward encouraged us to do just a textual list, even though we have much work already done on a treeview. The treeview is not necessary now, and we can still use it as a prototype for later if we need it.

We had been thinking that the code we had to generate VBA Collections of macro data was all below the UI layer and subject to test, and the code to actually display a list from the collection data was part of the UI, not subject to test-first. Ward encouraged us to actually write tests for the expected text output for the list, then the UI does nothing but call that, and put the text in a textbox. This seemed pedantic even to us, having already bought into TDD, but in the process of doing it this way, it started to make sense. For instance, should the text have a line break after the last line, before the first line, or neither? This could make a difference to the appearance, and changes could mess up any calling code that wanted to add text ahead of or after the returned text. By writing the test first, we document the expected behavior and ensure that the behavior continues to follw the spec. over time.

So - we had the form displaying the list of macro definitions in no time, and added a second text box to the form, displaying the list of macro targets in not much more time at all. We again resisted an impulse to make this ever so slightly more complex than we knew it needed to be, and did not add a tab control and put the texboxes on tab pages. It's trivial to do, but it can be done later if the requirement manifests - first, just make sure the darn thing works and that the tests will prove it continues to work.

So next, we moved on to the simple macro expansion preview. Sam and Jordan worked on the code to replace the target tag contents, and expand the DOM-Munged data back into VB code after doing the replacements. I worked in customer-mode to come up with the syntax we'll need for inserting macro arguments and specifying argument values at the target points, then worked on a code line reader class that generates a collection of VB Code Line objects rather than the array of lines that is currently constructed before sucking the lines into the XML document format. This will be needed to allow recording special attributes such as procedure name and type that are available from the VB Editor (VBE) and that will need to be passed into the XML document to be usabled as context parameters duting expansion.

Sam and Jordan had to solve some very tricky issues with the usage of the MSXML API, but made good progress, and think they're close to having a working expansion preview. I got enough of a VB Line reader working to be ready to integrate. We did not have time to integrate my changes at the end of the session, so that will be the first thing to do next session.

We finished 2 stories today, so the average velocity, so far is 2 stories per week, though, our sample size (given that we're starting over on measurement) is too small to be meaningful as of yet.

Final notes:

We are seeing that we have -not- been doing a good job at one piece of the TDD process. After all tests pass, we're supposed to refactor. We haven't always been doing it, and we haven't always been doing enough of it when we do. Consequently, we now have an obvious technical debt to repay. We expect to spend most of the first 1/2 of next session refactoring, which will impact our velocity, but the cost only goes up the longer this is delayed, so we definitely need to take the time and catch up with this.

I've noticed that our stories have lots of sequential dependency, and this has made it hard for us to figure out what one person can do while the other 2 are pairing on the next story in the list. I had to work on something that was scheduled at a much lower priority in the list to have something to do while Sam and Jordan worked on the simple expand preview. We should try to figure out ways to re-slice some of the story space to have fewer dependencies between them.

Revised VB Lite Unit testing platform

Well, I stuck my foot in my mouth in a way, telling a manager on one of my jobs that I had a great Unit Testing platform for VB/VBA, then realized I don't have an easy enough way to to deploy it.

So - to live up to what I said, I spent a couple of hours last night turning the test platform into an ActiveX DLL. There were a few strange issues to work through, but now that it's done, the test code is totally self-contained. Just set a reference to the library, add test classes, and run them.

One of the issues I had is that I need some "static" public procedures, and the way to do that from within the VB environment for testing (a standard module) is not the same as how they are exposed to the client application (a class module that's GlobalMultiUse), so I had to make one wrap the other. Also, the only way to expose any kind of constant is via a Public Enum in a public class, so I had to convert my numeric constants into Enum members. This means that only integer numerics can be exposed constants, but fortunately, that's all I needed.

Finally, Debug.Print from within the library does nothing at run-time. It does not write to the client application's Debug window. This turned out to be a good excuse to send output to its own window with a nice caption, Copyright notice, etc. Any ideas for an icon?

Watch This Space for Work Session Recap

I'm in charge of the session recap this week, but I've been pretty backed up so far (see next post). I expect to get the recap up tonight.

Quick aside on refactoring tools

This isn't directly related to the project, but:

I'm working through Test Driven Development: By Example, which Steve lent me. Chapter 6 introduces a superclass and starts by refactoring a method from a subclass to the superclass, so I used the "Refactor | Pull up..." item on Eclipse's context menu rather than manually refactoring via the clipboard.

That was impressive. (I've just outed myself as a native VB programmer, haven't I? I don't care -- that was one seriously cool tool.)