Thursday, November 27, 2014

Functional Programming in Mainstream Languages, Part 2: Introduction to Higher-Order Functions in Java 8

(Jump to Table of Contents)
If immutability is the primary principle of functional programming, higher-order functions are its most important technique.  Higher-order functions, also called lambas, are now part of many mainstream languages.  In particular, they have been added to Java 8.  In this post I will show how higher-order functions can be expressed in Java 8.

I can't explain the details of the theory and techniques here; I highly recommend the wonderful book Structure and Interpretation of Computer Programs (2nd Edition) by Abelson and Sussman.  Most of the examples in this series of posts are taken from that book.

In functional programming, functions are first-class objects.  This means that functions can be passed as parameters to other functions, they can be returned from functions, they can be stored in data structures; in other words, anything you can do with any other object in the language, such as numbers or strings, you can do with functions.

Some languages fake it: for example, in C you can pass around function pointers, but not function objects.  The major difference is that function objects contain not only the code the function executes, but also its environment, meaning all bound variables known to the function.  This is particularly important (and powerful) when functions are nested inside other functions (or methods, or any other binding scope).

The inventors of LISP struggled with the correct semantics of first-class functions; they called this the "funarg problem."  It turns out that the power of functional programming requires lexical scope, which means that variable names must be resolved with respect to the nesting of functions in the code, rather than to the structure of the calling stack at runtime.  And in order for that to work correctly, it is impossible to allocate function parameters and other local variables on the program stack if those may be referred to in an internal function that is returned from the call.  This complicates the compiler, but is invisible to programmers using the language.

As a simple example, consider a function make-adder that accepts a single number, and returns a function that adds that number to its own argument.  For example, if make-adder is called with the value 1, it will return a function that can be called increment, since it adds one to whatever argument it is given.  Here is how to write this function in Scheme and in Java 8.

Scheme
(define make-adder (lambda (x) (lambda (y) (+ x y))) (define increment (make-adder 1)) (increment 4)
Java 8
IntFunction<IntUnaryOperator> makeAdder = x -> y -> x + y; IntUnaryOperator increment = makeAdder.apply(1); int res = increment.applyAsInt(4);

The result of the last expression (in either language) is 5.

The syntax is quite different, but the effect is the same.  Let's look at the functions first.  The Scheme syntax is a direct translation of the syntax used in the Lambda Calculus, which was invented by the great logician Alonzo Church, and which forms the theoretical basis for functional programming (in addition to its major importance in the theory of computer science).  In the lambda calculus, the make-adder function would be written like this: λx.λy.x+y.

This syntax describes a function of one argument whose name is x; this is indicated by the notation "λx.".  What follows the dot is the value of the function for a given value of x; in this case, this is another function, whose argument is called y.  The value returned by this function is the sum of x and y.

In Scheme, a function is denoted by the syntax (lambda (v1 v2 ...) body), where v1 v2 ... are the parameter names, and body is an expression representing the value that the function returns.

In Java 8, function syntax has the names of the parameters before the arrow (->), and the body follows the arrow.  It is possible to specify more than one argument, in which case the arguments need to be put in a parenthesized list.  It is also possible to specify the types of the arguments, but fortunately the designers of Java have finally realized what a burden repetitive types are on programs, and Java 7 introduced type inference into the compiler in order to eliminated some (but not enough) of this repetition.  Java 8 extends this mechanism also to lambdas.  In this case, the compiler can infer the types of the arguments from the type declared for the function makeAdder.  This type needs some explanation.

Java 8 has a new set of interfaces to describe various types of functions.  For example, Function<T,R> is the type of functions that take an argument of type T and return a result of type R.  However, there is a whole set of interfaces that cater for primitive types, in order to avoid boxing and unboxing.  (This primary mistake in the design of Java has been giving programmers grief since Java 1, and is unlikely to go away.)  So IntFunction<R> denotes functions that take an int and return a value of type R.

What is the return type of makeAdder?  It is a function that takes an int and returns an int; such functions are described by the interface IntUnaryOperator.  The corresponding interface for functions that take an object of type T and return an object of the same type is UnaryOperator<T>.

As I will discuss in a later post, typed functional languages such as ML and Haskell have a rich set of functional types and a concise yet very expressive type notation.  The type of makeAdder would be specified in such languages simply as int -> int -> int, a notation that is similar to the way functions are defined.  This one indicates a function that receives an int, and returns a function of type int -> int, that is, a function that takes an int and returns an int.  Such notations would make the burden of type specifications in Java much easier.

#functionalprogramming #java8

Sunday, September 21, 2014

Functional Programming in Mainstream Languages, Part 1: What and Why

(Jump to Table of Contents)
John Backus worked for IBM from 1950 until 1991 and was the principal developer of FORTRAN, one of the first high-level programming languages.  He is immortalized in the acronym BNF, which stands for "Backus-Naur Form" (or sometimes "Backus Normal Form").  In 1977 he received the ACM Turing Award (the "Nobel prize of Computer Science"); his award lecture was titled "Can Programming Be Liberated from the von Neumann Style? A Functional Style and Its Algebra of Programs."  This was in a way a reaction to FORTRAN, and describes the vision of functional programming.

The "von Neumann style" is a model of computation that is very close to the way computers actually work.  Its main component is the memory, which is composed of individually-addressable cells.  Each memory cell can be read and modified by the computer executing a program, which is also stored in memory.  This model was very useful for building early computers, and was naturally taken as the basis for the first high-level languages.  The closer the model of the language is to the hardware model, the easier it is to compile programs to machine instructions that the hardware can execute directly.

Backus, however, made the point that the language model can be more abstract than the hardware model.  While this puts a greater strain on compiler writers, it can free (or "liberate") programmers from a lot of the drudgery that is involved in programming directly to the hardware model.

The most basic principle of functional programming is the absence of any side effects.  This means that functional programming language do not have the concept of variables as places (representing memory cells) that can be changed.  Values in functional programs are immutable; that is, they can't change.  This concept extends to data structures as well; for example, lists, maps, and "objects" in functional languages can't be modified, in total or in part.  And it also extends to interactions of the program with the outside world.

This may seem like an impossible restriction; how can you describe a computation without changing values?  In fact, it is quite easy and natural, but it does take some getting used to, especially if you are used to programming in imperative languages.  These include most mainstream languages, in which variables and mutable data structures are the norm.

The major benefit of functional programming is its very simple computational model, which is very close to mathematical thinking.  Because values in a functional program can't change, functional programs enjoy the property of referential transparency.  This means that if an expression has some value, it will always have the same value.  (This is not true in imperative languages; just think of a parameterless function that returns pseudo-random numbers.)  This means that you can use the well-known substitution principle from mathematics: if a=b then you can replace a by b everywhere it appears.  This is not true in imperative languages, since even if a=b now, they may be different after any change that may affect one or both of the expressions.  The computational model of imperative languages therefore must incorporate the notion of time; every assertion may be true at one time and false in another.  This results in a much more complicated model.

Most programmers don't bother thinking about the computational model underlying their programming language, but it is crucial for writing, understanding, and reasoning about programs.  A simple model makes all these simpler.  Programming in a language that has a complex model is like walking around with a heavy weight on your back; you don't notice you're carrying it until it is suddenly gone.

Another important property of functional programs, which is becoming increasingly important, is that it is much easier to parallelize them.  Imperative programs suffer from problems such as race conditions and non-atomic updates, which are a result of side effects.  By eliminating side effects, functional programs get rid of these undesirable consequences as well.  One commercially-successful functional language that makes great use of this fact is Erlang.

The functional paradigm is Turing-complete; that is, it can express any computation that any conceivable machine can perform.  However, creating functional analogues of mutable data structures is complicated (although this work can mostly be relegated to library writers).  It is therefore unrealistic to expect most programmers to abandon mainstream languages in favor of functional languages.

Since functional programming is predicated on the lack of side effects, it seems to be inherently contradictory to imperative programming.  However, in the last decade or two, there is a growing movement to incorporate functional programming techniques into scripting and object-oriented languages, including JavaScript, Ruby, Scala, and, more recently, Java 8.  There is also much academic work on the integration of the functional and object-oriented paradigms; so much so, that one of the predictions of Murphy-Hill and Grossman in their paper "How Programming Languages Will Co-evolve with Software Engineering: A Bright Decade Ahead" (Future of Software Engineering, 2014) is the blurring between these paradigms (see my review of that paper).

How can these conflicting paradigms be reconciled?  The point is that even if you want to use side effects (including assignment and mutation of data structures), you don't have to go the whole hog and use them everywhere.  By making some of your data structures immutable, you will reap some of the benefits of functional programming.  Bertrand Meyer, one of the gurus of object-oriented programming, includes the Command-Query Separation principle in the chapter on designing class interfaces in his landmark book, Object-Oriented Software Construction (2nd Edition).  This principle requires methods to be either queries, which return information but don't mutate objects, or commands, which mutate objects but don't return any results, but not both.  This separation results in some measure of referential transparency, because it tends to reduce the number of mutations.

In addition to immutability, the most important aspects of functional programming that are also relevant to imperative languages are high-order functions and streams.  In following posts I will show examples of functional programming techniques, both in (fully or mostly) functional languages, as well as in mainstream languages including Java 8.

This series of posts has the following parts:

Introduction
Part 1: What and Why

Higher-order functions
Part 2: Java 8 (Introduction)
Part 3: Java 7 and Java 8
Part 4: JavaScript
Part 5: Scala
Part 6: Xtend
Part 7: Haskell

#functionalprogramming

Sunday, August 17, 2014

Review: How Programming Languages Will Co-evolve with Software Engineering

Until recently, programming languages have been developed with little consideration for programming environments and tools, with the notable exception of Smalltalk, which was co-developed with an elaborate environment from the start.  Murphy-Hill and Grossman note in their paper "How Programming Languages Will Co-evolve with Software Engineering: A Bright Decade Ahead" (Future of Software Engineering, 2014) that this situation is changing, and provide seven predictions on the convergence of programming languages and software engineering (or, more precisely, programming environments and tools) in the next decade.

They predict that future programming languages will be designed with integrated development environments (IDEs) and social networks in mind.  One example they give for the former is the addition of dot notation to F#, allowing a syntax such as x.Length in addition to the functional length(x).  (Note, though, that in F# these have different meanings.)  The authors claim that the primary benefit of this notation is the ability of the IDE to provide code-completion for method (or function) names when typing "x."; this facility doesn't (yet) exist for the functional notation.  I have my doubts about this being the major reason for the inclusion of members in F#, but it could certainly be a reason for introducing dot notation as syntactic sugar for function calls in new languages.

Another prediction is that source code will no longer be limited to a single sequence of characters, instead being replaced by multiple simultaneous views.  Multiple views are already available through the use of various tools; what I would call refactor-and-undo is probably the most widely used method, but there are various tools specifically targeted at showing multiple views of a program.  These multiple views can be stored together as the official representation of the program, instead of being generated from a single "ground truth" consisting of ASCII (or unicode) strings.

Murphy-Hill and Grossman expect that future language design will be influenced by controlled experiments on how programmers use language features and to what extent.  With the widespread availability of open-source projects, it is possible to study this rigorously, and select language features based on what really works "in the wild." This is already happening to some extent with existing languages, such as C#.  I would advocate some care when using this method, since we know that some programming features take a long time to achieve adoption; a case in point is functional-programming techniques (see the last prediction below).  We should be careful not to eliminate good ideas just because they don't receive a wide following in the short term.

Turning to other technologies, the next prediction is that formal verification will become a reality for developers, rather than being limited to a small group of highly experienced researchers.  This will require the development of proof assistants that are stronger and more usable than those available today.  Obviously, there is a lot of work to be done before the use of formal verification becomes widespread, but advances in this area provide some support for the authors' optimism.

Gradual typing is a technique that allows developers to write programs with little or no type information.  As the programs mature, types can be added in order to raise the level of confidence in the correctness of the program by making more properties statically checkable. Murphy-Hill and Grossman predict that gradual typing will become more widespread in new programming languages, and will expand to include properties that can be expressed as types but aren't available in popular contemporary languages.

Existing programming languages are being used in new domains, such as parallel and distributed programming and big data, even though these languages aren't particularly suited to these domains.  The success of Erlang is an example of how well a language built from the ground up to support distributed computation can do when applied appropriately. Murphy-Hill and Grossman predict that newer languages will focus more on new paradigms at the expense of traditional single-user (and often still single-threaded) applications.

This is related to the final prediction, which is that functional-progrmming concepts will become more widespread, and the distinction between functional and imperative (or object-oriented) programming will be more and more blurred.  This trend is already happening, with the introduction of function objects ("lambdas") into popular languages such as C++, Java, C#, and Ruby, and with mixed languages such as Scala.  First-class function objects are one primary feature of functional programming; the other is the use of immutable data, which is particularly important in distributed applications.

There is a recent (and not so recent) body of work that points in the direction of each of these predictions.  Unfortunately, there are many influences on the design of programming languages, and various features of currently-popular languages were not really designed but added without serious consideration.  I am therefore not quite as optimistic as the authors; however, the points they make are insightful and the paper is well-worth reading.  Be sure to put a link to the paper in your 2024 calendar, in order to see which of the predictions will be realized by then!

Thursday, June 5, 2014

Xtend, the Gentle and Functional Java

I didn't want to learn yet another programming language.  But I had to.

I needed to create an Eclipse editor for a domain-specific language, and I needed to customize content-assist and completion.  A friend recommended Xtext as a quick way to get an Eclipse editor with all the goodies up and running.  And it was quite easy to start with; I had to translate my ANTLR grammar into Xtext, which wasn't too painful, and within a day or so I had a working editor for my DSL.

The next step was to add custom content-assist to the editor.  This turned out to be relatively easy as well, except that the file Xtext generated for me was in this new language, Xtend.  So I had to learn the language.  Unlike quite a few other people I know, I first read the manual.  I admit I did this reluctantly, but as I kept reading I got more and more excited.

I've been using Java for many years, both in teaching and in developing applications.  It is certainly nicer than C++, and the refactoring support in Eclipse is wonderful, but there were other languages I used in the past that I enjoyed much more than Java.  Common Lisp, especially on the Lisp Machine, was a great language with a great environment.  Scheme was a special pleasure, being a very compact yet extremely powerful language.  In contrast, Java is very limiting and extremely verbose!

First, the substance.  I am a great fan of functional programming; it is a limiting yet at the same time liberating paradigm.  You don't get to make any assignments or other state changes; in return, the computation model is very simple and similar to standard mathematics.  Functional languages enjoy referential transparency, which means that the same expression in the same environment will always yield the same value.  In contrast, when side effects are possible, it is possible to call the same function or method multiple times and get different results each time, since something in the environment may have changed.  While object-oriented programming seems to be all about changing state, it is still possible to use functional-programming techniques to minimize state changes to those places where they are really useful.  Item 15 of Joshua Bloch's excellent book Effective Java (2nd Edition) is "Minimize mutabililty."  And Bertand Meyer, designer of the Eiffel programming language, expounds in his classic text Object-Oriented Software Construction (2nd Edition) on his Command-Query Separation Principle.  This principle advocates minimizing state changes, and using functional programming techniques inside object-oriented programs as much as possible.  The Scala language is a more recent attempt to reconcile the functional and object-oriented programming styles.  Java allows a functional programming style, but makes it unnecessarily difficult.  (A ray of hope, though: Java 8 is going to have lambda notation!)

And then, there is the form.  Java is obnoxiously verbose, especially in its requirements for types everywhere.  I'm not necessarily objecting to strong typing here, but even with strong typing there are many type inferences that the compiler can make.  For example, the fact that I have to write something like this is infuriating:

HashMap<String, List<String, Integer>> map = new HashMap<String, List<String, Integer>>();

One way of thinking about Xtend is as a simpler syntax for Java.  The Xtend compiler does extensive type inference, and most type specifications are optional.  But Xtend is much more than that.  It provides many conveniences, and is particularly well-geared to the use of functional techniques.  Functions ("lambda expressions") are an essential part of the language, and are a pleasure to use and read.  They simplify a lot of the boilerplate code needed in Java for callbacks of various kinds.  In addition, Xtend libraries supply many utilities in the functional style, and these look like an integral part of the language due to Xtend's extension capabilities, which allow external libraries to add methods that can be called as though they existed in the original Java classes.  Another great convenience is a much enhanced switch statements, which, among other things, can switch on types without requiring type casts.

At the same time, Xtend compiles directly into (readable) Java, not even into bytecode.  So it is 100% compatible with Java, and can be used in any combination with Java classes.  For example, Xtend classes can inherit from Java classes as well as the other way round.  Xtend is well-integrated with Eclipse, and offers most of the conveniences of Java development, including debugging the Xtext sources.  The major lack is in refactoring, although Rename, for example, is available using the Alt-Shift-R shortcut.

If you are stuck with Java but want a nicer syntax, I recommend that you take a good long look at Xtend.  You can start with the 50-second video on the Xtend website, then read the manual; surprisingly, it is quite readable!

#Xtend #FunctionalProgramming #Java

Monday, March 10, 2014

Review: Mars Code



Despite a few spectacular failures (such as the Mars Climate Orbiter and Mars Polar Lander), NASA is one of the top organizations worldwide in Systems and Software Engineering.  The Mars rover Opportunity has recently celebrated ten years of operation on Mars, and the newer Curiosity has been operating since August 2012, after a landing maneuver of unprecedented complexity.  This is quite a feat, considering the fact that the nearest repair technician is over 30 million miles away, and communications delay is about 15 minutes on average.  How do they do it?

The paper Mars Code by Gerard J. Holzmann (Communications of the ACM,  Feb. 2014) gives a rare glimpse into the process used in programming the Mars rovers.  It focuses on three methods used to reduce the risk of failure.  An important feature of all three methods is the use of automation.  The first consists of a set of risk-related coding rules, based on actual risks observed in previous missions.  For example, one rule requires the code to pass compilation and static analyzer checks without any warnings.  Another requires the code to have a minimum assertion density of 2%.  The assertions are enabled not only for testing, but also during the mission, when violations place the rover into a safe mode, during which failures can be diagnosed and fixed.  All the rules are checked automatically when the code is compiled. 

The second method consists of tool-based code review.  Four different static-analysis tools are used, with an automated system to manage the interaction between developers and reviewers in response to tool warnings.  In over 80% of the cases, developers agreed with the warnings and fixed the code without reviewer intervention.  An interesting observation Holzmann makes is that there was surprisingly little overlap between the between the warnings issued by the four tools. Apparently, each tool has its own areas of strength, and they are different enough that it is useful to use them all.

The third and strongest method is the use of a model checking tool for race conditions, one of the most difficult types of bugs to discover.  This cannot be done for the whole code base, but was used extensively in the Curiosity mission to verify critical multithreaded algorithms.   In some cases, the tool works directly from the C source code.  For larger subsystems, input models had to be prepared manually in the language of the model checker; this can reduce the input size by over one order of magnitude.

The challenges faced by NASA are obviously unique, and are very different from those involved in cloud-based mobile applications, to take a popular example.  There are many papers and books describing methodologies for developing such systems.  However, developing safety-critical systems for cars and aircraft, medical devices, and power grids, or financial systems that must be resilient to faults and deliberate attacks, requires reliability levels similar to those of NASA.  For developers of such systems, papers such as Holzmann’s should be required reading.

A related article you may want to read is They Write the Right Stuff, by Charles Fishman.  This article, from 1996, describes the culture of the group that wrote the software for the space shuttle, and how their development process led to the very high quality of their product.  For example, whenever a mistake was found in the code, it was traced to the corresponding problem in the process that allowed the mistake to be written and pass reviews and testing.  Then, the team investigated what other bugs could have been caused by the same process failure.  In this way, a bug in the control of the shuttle's robotic manipulator arm was traced back to a certain type of problem.  When investigating other effects of the same problem, a much more serious bug was found in the control of the high-gain antenna.  The bug was fixed before it had a chance to manifest itself.

It takes a lot of discipline and investment to work this way, but the result is a much better, more reliable product.  Sometimes it isn't all that important; other times, it's the difference between life and death.

Sunday, June 24, 2012

Design by Contract and Refactoring

I am a great believer both in refactoring and in design by contract.  How do these two work together?

First, contracts are a great help when refactoring.  The first thing you need to know when you refactor a piece of code is what it is supposed to do.  This tells you what you can and can't do with it, and, no less important, what would be useful to do with it.  If it's an arbitrary set of statements, say part of an existing method you are trying to extract into a new method, there's little to help beyond reading the code and trying to figure out what it does and how it fits with the rest of the code.  (Even then, there are tools that try to deduce a contract for an arbitrary piece of code; more on that in a later post).  But if it's a method, you should have more help.

If you have a good suite of unit tests, you can try to look at the tests of that method to figure out what it's supposed to do.  But it's much easier if you have a contract attached to the method; the contract should give you a lot of the information you need.  For example, suppose you want to move one or more methods and fields from one class to another (perhaps you are using Pull Up Method, Push Down Method, or just Move Method).  In that case, you should check the contracts of moved methods to see how they fit in their new class.  Are the invariants of the new class maintained by the moved methods?  Are the contracts of the methods still valid, taking inheritance into account?  Nasty bugs can result if the answers are negative.

On the other hand, refactoring may require corresponding changes in existing contracts, as well as the creation of brand-new contracts.  In other words, contracts need to be refactored together with the associated code.  This is an added burden, which may well be an obstacle to the use of the design-by-contract methodology; not only do I have to invest effort in creating the contracts in the first place, I also have to refactor them later.  Why should I bother?

There are several answers to this complaint.  First, consider the alternatives.  You may be flying blind, trusting only on your undocumented understanding of the code.  In that case, good luck to you; you'll need it.  If you follow an agile methodology, such as Extreme Programming, you should have an extensive suite of unit tests to help you refactor.  Unit tests are very vulnerable to change in the code, since they are attached to small units of code.  This means that any shift in responsibilities is likely to invalidate some tests, which will then have to be refactored or completely rewritten.  So there's always the need to refactor associated artifacts when you refactor the code.

Having contracts can significantly reduce the amount of detail in unit tests, and even the total number of unit tests.  This is due to the fact that a major part of the responsibility usually given to unit tests is now taken by the contract.  All that the tests need to do is exercise the system, but correctness checking (or a large part of it) is now done by checking the contracts.  So now you can have higher-level tests that exercise the system, instead of unit tests for each class and method.  For example, suppose you are implementing a cryptographic algorithm such as RSA.  A test that creates a random key, then encrypts a random data buffer, decrypts it, and checks that the result is the original data, is guaranteed to exercise almost all of your cryptographic code.  Moreover, this test can be used on many different implementations.  In contrast, the internals of the implementation can vary widely, since there are many ways to implement the large-number arithmetic operations required for RSA, and their efficiency will be different under different circumstances.  By having an application-level test augmented with contracts, the burden of refactoring tests is reduced to almost nil.

This still leaves the requirement of refactoring contracts, and the question of how much refactoring tools can help.  In order to understand that, it is necessary to examine the relationships between code refactorings and contracts.  On the simplest level, contracts are treated just like code. For example, when renaming a method, all references to it in the code must be appropriately modified; so must all references in assertions. Similarly, a method may be eliminated when it is not used anywhere, including in assertions.  Contract-aware tools should find these kinds of relationships easy to perform automatically.  Of the refactorings listed in Fowler's Refactoring book, 32% do not interact with contracts except possibly in this way.  These are mostly syntactic refactorings such as Remove Parameter or Rename Method, or those that eliminate classes or methods, such as Inline Class and Inline Method.

Some contracts affect the applicability of certain refactorings.  As mentioned above, a method should only be moved if it won't violate the invariant of its new class.  (Of course, it might be possible to modify that invariant to accomodate the moved method; this can be done in a separate step, before moving the method.)  13% of Fowler's refactorings have this nature.

Some refactorings require the creation of new or modified contracts.  The most extreme case is Extract Method, which creates a new method out of arbitrary code.  Discovering the contract for arbitrary code is impossible in general, although some tools can discover partial contracts.  Another example is Create Superclass, which can create contracts for new methods based on existing contracts in subclasses.  59% of Folwer's refactorings fall into this category (which includes 4% that are also included in the previous one).

Finally, some new refactorings can be defined to deal specifically with contracts; these include Pull Up Contract, Push Down Contract, Create Abstract Precondition, and Simplify Assertion.

In a future post I will discuss automation of contract-related refactorings and how refactoring tools can take some of the burden of the contracts.


Sunday, February 19, 2012

Refactoring

Change is a way of life in software development.  Requirements change because clients gain a better understanding of their needs (possibly due to experience with an existing version of the software), to take advantage of new business opportunities, or for external reasons such as new legislation that requires new types of auditing, or change in standards.  The environment may change in various ways; for example, moving from a private data center to a public cloud, operating systems may become obsolete, or mission-specific hardware may need to be upgraded.  Change is the greatest challenge to the software development process, and development methodologies employ various tactics to accommodate it.  The classic "waterfall" methodology tries to eliminate it, by spending a lot of time in the beginning of the project trying to get the requirements just right.  Unfortunately, even if this is extraordinarily successful, by the time the other steps in the process are done, the requirements are practically guaranteed to be out of date.

Agile methodologoies take the other extreme.  Significantly, the classic introduction to Extreme Programming (nicknamed XP) by Kent Beck is called Extreme Programming Explained: Embrace Change.  These methodologies accept that change is inevitable, and gear every activity to support it.  This can only work if it is possible to modify existing software to fit changing requirements, and this is only possible if the software is always well-designed.  Unfortunately, software tends to change away from a good design, because each change is usually done without much regard for the overall design.  As changes accumulate, it becomes harder and harder to make further changes, and the likelihood of introducing errors grows in an exponential manner.  Therefore, one of the basic tenets of agile methods is that developers must take the time to restore a clean design to their software every once in a while.  This activity may seem to be unproductive, since it doesn't result in any modifications to the external behavior of the software.  However, it enables further modifications, and is therefore essential.

The activity of changing the internal structure of the software in order to restore it to a well-designed form without affecting external behavior is called refactoring.  First introduced by Bill Opdyke in his 1992 PhD thesis, and popularized in the famous book Refactoring: Improving the Design of Existing Code, refactoring is an approach to software development as well as a set of "refactorings," each of which is a detailed list of instructions on how to perform structure-modifying but behavior-preserving changes in code in a reliable way.  Reliability is achieved by performing unit tests after each small change, in an attempt to guarantee that behavior is still preserved.  If some of the tests fail, it is easy to identify the reason, since it is rooted in the last small modification performed.

Of course, in order for this to work, it is necessary to have an extensive suite of unit tests, which will guarantee (with high probability) that any change that doesn't preserve functionality will be discovered immediately.  This is easier said than done.  A good set of unit tests may be considerably larger than the production code itself, and requires significant effort to create and maintain.  Most agilists consider this to be a necessary evil; I once heard a developer say he enjoys writing these tests, and even Kent Beck, the high priest of XP, expressed his surprise at that.

Still, developing without a good suite of tests is like driving blindfolded; you can't see where you're going and you're afraid to make any change at all.  In fact, Michael Feathers, in his book Working Effectively with Legacy Code, defines a legacy system to be one for which you don't have a good suite of unit tests.  And refactoring is a very effective and satisfying way to program.  I recall several experiences in which I started making some change in my code, which led to further changes, leading to a period of several days when I couldn't even compile my code, let alone run it.  This is a terrible feeling; you know you have problems in your code, but you have no way of testing it to find out what they are.  At the end of this process, when I got back to a running system, it indeed had problems, and it took a long time to find them all and fix them.  By restructuring the work into a series of refactorings, it would have been possible to test much more often, and thus find problems and fix them much earlier.

As I said, there can be no agile methodologies without refactoring.  However, refactoring is a useful activity regardless of which methodology you use (including no methodology at all).  It does come at a cost; you must have an extensive test suite, and refactoring itself takes time and effort.  I consider this cost to be well spent, since it saves much more effort down the line, when the inevitable requirement changes appear.  Hard as it is to convince programmers to invest in their future (see Programmers as Children), this is one of the important places to try.

As in other cases, tools are a great help in following the methodology, and therefore also in convincing programmers to use it.  Furthermore, good tools take away some of the burden of checking the correctness of the transformation.  Good refactoring support exists (mostly for Java) in Eclipse and other modern IDEs, and I urge every developer to make the maximum use of these.  When I modify my code, I make every effort to use the automated refactoring capabilities provided by the IDE, as well as the related source transformations and quick fixes.  I do this even when this forces me to make the changes in a certain way or order, just to take advantage of the automation (and related reliability) in the IDE

Unfortunately, all modern IDEs have bugs in their refactoring support, chiefly due to insufficient analysis.  For example, they might create variable bindings that shadow existing bindings incorrectly.  So you must be careful when using these tools, and be aware of their limitations.  In spite of these limitations, I find them extremely useful.  One of the current research interests of our group is automating more powerful refactorings and making them more reliable; more about that in a later post.